37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-10: prompts read on the GPU, faster than llama.cpp on the same GPU; Qwen3.5's recurrent layers on the GPU; the fast prompt on by default
camaurida0a99497b6a
diff --git a/ChangeLog.md b/ChangeLog.mdindex 286229b..7301576 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,17 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-10] - Prompts read on the GPU, faster than llama.cpp on the same GPU++- **The fast prompt, on by default.** A prompt's blocks go to the GPU's tiled products - Q4_K, Q5_K, Q6_K (plain and in planes), Q8_0, f32 and bf16 weights, a mixture's experts in one launch per layer, the GPU taking 60% of their tokens while the CPU computes the rest - and on the dense Qwen3 and Qwen3.5 models the whole block, every layer, goes to the GPU in one submission while the CPU rests. They round the sums group by group, so a prompt is no longer read to the bit as the CPU reads it; against the references the most likely token agrees as often as the exact way (within 2-3 of 256 on every model measured). A reply's tokens are as before. `JANAS_GPU_FAST_PROMPT=0` reads prompts the exact way.+- **Qwen3.5's recurrent layers on the GPU.** The Gated DeltaNet layers - convolution, recurrence (each row of the state kept in registers for the whole block, at sixteen bits between tokens as the CPU keeps it), gated norm - have shaders of their own, so Qwen3.5's prompts go to the GPU whole too; with an MTP block the hidden states come back for it (drafts kept 78-86% as before).+- **Faster than llama.cpp's Vulkan backend on the same integrated GPU**, reading 1,024 tokens: Qwen3.5-2B 441 tok/s against 323, Qwen3-4B 193 against 164, Qwen3.5-9B 111 against 78, Qwen3-30B-A3B 135 against 122, the four Gemma 4 models 65-191 against 46-150; writing 1.3 to 2.1 times as fast. The README's tables have the figures.+- **On a laptop, the CPU rests while the GPU computes.** CPU and GPU share one power limit: the CPU's threads spinning beside the GPU held it at half its clock. The pool's idle workers now sleep and the GPU's fence is waited on, with the fast prompt: Qwen3.5-2B +18%, Qwen3.5-9B +22%, Gemma 4 E4B +12%.+- **The tuner chooses better.** With the fast prompt a prompt starts on the performance cores' threads; the default competes from its first measurement (an alternative tried three times used to win even when slower: Qwen3.5-2B's prompts went to 20 threads, 272 tok/s, against 318 on 12); a chat's short blocks (17-63 tokens) are tuned apart from long prompts; and settled threads move when other threads win three tries in a row, since a laptop that warms up turns the first verdict round.+- **Larger prompt blocks where they pay**: 512 tokens on the Qwen3, Qwen3 MoE and Qwen3.5 models whose feed-forwards the GPU takes whole (Qwen3-30B-A3B reading 1,024 tokens: 108 -> 123 tok/s from that alone).+- **Fixed: the last layer's down matrix went to the CPU without a word** on several models: the GPU's Q6_K shaders read 4 bytes past a matrix and the expert arena's last slot had none. A guard page now follows it.+- **Fixed: a crash now and then while reading a mixture's first prompts** on the GPU, when the table of products' shapes filled up.+## [2026-10-09] - Qwen3-4B and Qwen3-30B-A3B also with an importance matrix- **`qwen3-4b-imatrix` and `qwen3-30b-a3b-imatrix`: bartowski's Q4_K_M beside Qwen's own, which stay the default.** Qwen's Q4_K_M files are quantized without an importance matrix, bartowski's with one. Against Qwen's own Q8_0 (400 tokens of llama.cpp's logits, most likely token agreeing): Qwen3-4B 381 against 364 on code and 351 against 329 on Italian, at the same size and speed; Qwen3-30B-A3B 388 against 376 and 367 against 335, the nearest of the three versions now in the catalog, 8-10% slower held in memory since it keeps more matrices in Q6_K. Each has its fast profile measured: `--head 4 --attention fast` (+5.7%) and `--head 4` (+3.2%). The three-bit Qwen3-30B-A3B also has one now: `--head 4` (+3.8%).diff --git a/INSTALL.md b/INSTALL.mdindex 6d0faad..530955b 100644--- a/INSTALL.md+++ b/INSTALL.md@@ -79,7 +79,7 @@ It takes a few minutes: it measures prompt processing and generation speed for e## Troubleshooting-- `JANAS_GPU=0` turns the GPU off; `JANAS_GPU_DEVICE=integrated` or `discrete` picks one when there are both.+- `JANAS_GPU=0` turns the GPU off; `JANAS_GPU_DEVICE=integrated` or `discrete` picks one when there are both; `JANAS_GPU_FAST_PROMPT=0` reads prompts the exact way, bit for bit as the CPU (the fast prompt, on by default, rounds them group by group on the GPU).- `JANAS_PREFETCH=0` never reads experts ahead (by default it turns itself on when the cache misses many experts a token).- `JANAS_ARENA_HUGE=1` asks the kernel for huge pages for the expert cache, as before 30 September 2026 (with transparent huge pages on `always` and their defrag on `madvise` it made the first writes to the cache slow).- `JANAS_KERNELS=avx2` uses the AVX2 kernels even where AVX-VNNI is available.diff --git a/README.md b/README.mdindex 0b5148a..b0b12e2 100644--- a/README.md+++ b/README.md@@ -65,29 +65,47 @@ Everything the engine does is chosen from what it measures, so another machine w- **`janas-server`:** the model behind an HTTP API that follows OpenAI's, so that clients written for it can use a model running on this machine: chat and completions with tools, JSON output held to a schema and log-probabilities, the Responses API with its conversations, embeddings and moderations, a chat page of its own to try it from a browser ([below](#using-it-over-http)), and code completion for VSCodium and VS Code ([docs/code-completion.md](docs/code-completion.md)).- **`janas-bench`:** measures the machine, fills its profile and writes a report.- **Faster replies** from drafts the model verifies, so the text is exactly the one it would have written: from the model's own multi-token prediction block where it has one, from the conversation where it does not, or from a small model of the same family given with `--draft`.-- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. Results are identical with and without it.+- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. A reply's tokens are identical with and without it; a prompt is read with the fast prompt (next).+- **The fast prompt**, on by default since 10 October 2026: a prompt's blocks go to the GPU's tiled products (Q4_K, Q5_K, Q6_K, Q8_0, the experts of a mixture in one launch per layer, f32 and bf16 weights), and on the dense Qwen3 and Qwen3.5 models the whole block - every layer, attention, Qwen3.5's recurrent layers - goes to the GPU in one submission while the CPU rests: on a laptop the CPU working beside the GPU takes the power the GPU would use. They round the sums group by group, so a prompt is not read to the bit as the CPU reads it; against the references the most likely token agrees as often as the exact way (within 2-3 of 256 on every model measured, the mean difference of the scores within 2%). On the development laptop Qwen3-4B reads the table's prompt (below) at 220 tok/s with it and at 85 without the GPU, and every model compared reads faster than llama.cpp does on the same GPU. `JANAS_GPU_FAST_PROMPT=0` reads them the exact way.- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even. `JANAS_PREFETCH_PREFILL=0` turns it off.- **Light experts not in memory are left out** when the expert cache holds less than a fifth of the experts: an expert the router weighs below 0.8/k of a token (0.95/k under a tenth), and that would have to be read from disk, is not read and not computed, and the token's other experts take its share — the cache-conditional routing of Skliar et al. (TMLR 2025). Until 8 October 2026 the engine dropped the lightest experts whether in memory or not (six of ten, or eight). Qwen3-Next-80B, four bits, 400 tokens against llama.cpp, most likely token agreeing and tokens/s: with a 2 GiB cache all ten 393 at 11.0, six 383 at 20.9, now 380 at 26.4; with 4 GiB 393 at 20.7, six 383 at 27.6, now 384 at 27.7, with the mean logit difference 0.36 against 0.53; on Italian prose with 8 GiB 382 at 18.7, eight 361 at 21.4, now 372 at 23.7. On Qwen3-30B-A3B and Qwen3.6-35B-A3B with caches of 9% and 18% of their experts (1.5-3.5 GiB), on code and Italian prose, it was faster than the old way every time, by 9-37% (Qwen3-30B-A3B with 3 GiB: 28.7 tokens/s against 24.4, 381 tokens agreeing of 400 against 377), with a mean logit difference as low or lower in seven of the eight; it agreed on fewer most likely tokens in two (Italian, 64 tokens: 47 against 54 on the 30B with 3 GiB). `/experts n` (or `0`) asks for a number of experts and turns it off.-- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 61 (48 until 9 October 2026, when its unpacking went through the stack), IQ3_S 25, against the 81 GB/s the memory gives; and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.+- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU) for a reply's tokens (and for a prompt with `JANAS_GPU_FAST_PROMPT=0`). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 61 (48 until 9 October 2026, when its unpacking went through the stack), IQ3_S 25, against the 81 GB/s the memory gives; and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.-On the machine above, with nothing else running, `janas-bench` reports this (measured before the changes of 6-7 October 2026: on Qwen3-30B-A3B in memory, `tests/bench_long` found them worth 4% reading a 1,900-token prompt and 1.5% writing after it):+On the machine above, with nothing else running, `janas-bench` reports this (measured on 10 October 2026, with the fast prompt the engine uses by default since that day):| Model | reading a prompt | writing a reply | with drafts, in a chat ||---|---:|---:|---:|-| Qwen3.5-2B (dense, 1.3 GB) | 215 tok/s | 49 tok/s | **75 tok/s** |-| Qwen3-4B (dense, 2.5 GB) | 93 tok/s | 28 tok/s | see `--draft` below |-| Qwen3.5-9B (dense, 5.7 GB) | 46 tok/s | 13 tok/s | **22 tok/s** |-| Qwen3-30B-A3B (18.6 GB) | 112 tok/s | 33 tok/s | — |-| Qwen3.6-35B-A3B (22.3 GB) | 73 tok/s | 23 tok/s | **36 tok/s** |-| Qwen3-Next-80B-A3B (48.4 GB) | 65 tok/s | 23 tok/s | **36 tok/s** |+| Qwen3.5-2B (dense, 1.3 GB) | 395 tok/s | 50 tok/s | **67 tok/s** |+| Qwen3-4B (dense, 2.5 GB) | 220 tok/s | 28 tok/s | see `--draft` below |+| Qwen3.5-9B (dense, 5.7 GB) | 103 tok/s | 14 tok/s | **24 tok/s** |+| Qwen3-30B-A3B (18.6 GB) | 133 tok/s | 34 tok/s | — |+| Qwen3.6-35B-A3B (22.3 GB) | 86 tok/s | 25 tok/s | **34 tok/s** |+| Qwen3-Next-80B-A3B (48.4 GB) | 64 tok/s | 22 tok/s | **37 tok/s** |+| Gemma-4-E2B-it (3.1 GB) | 211 tok/s | 38 tok/s | **52 tok/s** |+| Gemma-4-E4B-it (5.0 GB) | 133 tok/s | 20 tok/s | **32 tok/s** |+| Gemma-4-12B-it (dense, 7.1 GB) | 71 tok/s | 10 tok/s | **14 tok/s** |+| Gemma-4-26B-A4B-it (18.3 GB) | 90 tok/s | 18 tok/s | **26 tok/s** |-Every figure is the **mean of three rounds**, not a best round, of the configuration the engine itself chose for each kind of pass - the one a chat runs: here all the usable cores (performance and efficiency cores, not the low-power ones), with the integrated GPU only where it helped, which was reading prompts on the two largest mixtures. On another machine the engine may settle on fewer threads: the first passes of each kind try every core, one thread per performance core and the performance cores' threads, and keep the fastest (on an i9-14900HX every thread replied at less than half the speed of the performance cores' 16). The rounds run in alternating order so that the machine warming up and the cache filling weigh on each setting alike. Expect a few per cent either way between runs, and rather less than these on a first run, while the cache is still filling.+Every figure is the **mean of three rounds**, not a best round, of the configuration the engine itself chose for each kind of pass - the one a chat runs: a prompt of 256 tokens, then 64 written. Here the integrated GPU reads every prompt (the fast prompt, below), and the threads the engine settled on were the performance cores' 12 for a prompt and for a single token on nearly every model, all the usable cores (performance and efficiency, not the low-power ones) for a few tokens at once. On another machine the engine may settle on other threads: the first passes of each kind try every core, one thread per performance core and the performance cores' threads, keep the fastest, and later try the others now and then, so that a machine that warms up can change the verdict (on an i9-14900HX every thread replied at less than half the speed of the performance cores' 16). The rounds run in alternating order so that the machine warming up and the cache filling weigh on each setting alike. Expect a few per cent either way between runs, and rather less than these on a first run, while the cache is still filling. Without the GPU (`JANAS_GPU=0`) Qwen3-4B reads the same prompt at 85 tok/s and writes at 27.-The last column is the model's own prediction block guessing the next tokens and the model checking them, which is the same text at a higher rate — the whole of it is explained under [`--mtp`](#using-the-chat). It is measured on a chat reply: an ordinary question in English ("explain how bread is made at home"), 64 tokens of the answer, sampled as the chat samples. **What the block is worth depends on the text**: there the drafts survived 78-89% of the time; on Italian prose Qwen3.5-2B kept about a third of them, and gained next to nothing. Qwen3-30B-A3B has no such block, nor have the dense Qwen3; the dense Qwen3.5 have one in their original checkpoint, which `hf2jns_mtp` converts ([MODELS.md](MODELS.md#the-fingerprints)).+The last column is a draft guessing the next tokens and the model checking them, which is the same text at a higher rate — the whole of it is explained under [`--mtp`](#using-the-chat): the Qwen3.5 and Qwen3.6 models' and Qwen3-Next's own prediction block, Gemma 4's assistant (a small drafting model of its own). It is measured on a chat reply: an ordinary question in English ("explain how bread is made at home"), 64 tokens of the answer, sampled as the chat samples. **What the draft is worth depends on the text**: there the drafts survived 74-89% of the time on the Qwen models and Gemma's two larger ones, 63-72% on Gemma's E2B and E4B; on Italian prose Qwen3.5-2B kept about a third of them, and gained next to nothing. One round of 64 tokens is noisy here, a tenth either way between runs. Qwen3-30B-A3B has no such block, nor have the dense Qwen3; the dense Qwen3.5 have one in their original checkpoint, which `hf2jns_mtp` converts ([MODELS.md](MODELS.md#the-fingerprints)).**This column was wrong until 26 September 2026**, and higher: it measured the block continuing a paragraph that repeated itself, which a model copies and the block guesses almost every time - 107 tok/s for Qwen3.5-2B, 26 for Qwen3.5-9B. The MoE figures of the time (29 and 30) happened to be lower than today's; the ChangeLog has the details.-For a point of reference, llama.cpp (build `2b18470` of 18 September 2026) on the same Qwen3.5-9B file and the same machine, measured the same night with `llama-bench -p 256 -n 32` at 6, 12 and 20 threads, did best at 20: **32 tok/s** reading the prompt and **9.0 tok/s** writing. The two tools do not measure in exactly the same way (`llama-bench` writes without a prompt before it; `janas-bench` after 256 tokens of one), and it is one model on one machine.+For a point of reference, llama.cpp (build `2b18470` of 18 September 2026) with its Vulkan backend on the same integrated GPU, everything on it (`llama-bench -p 1024 -n 16 -ngl 99 -t 12`), against `janas-bench --prompt 1024 --gen 16` on the same files the same days (9 and 10 October 2026; llama.cpp's best configuration, Janas's own choice):++| Model | reading 1,024 tokens: Janas | llama.cpp | writing: Janas | llama.cpp |+|---|---:|---:|---:|---:|+| Qwen3.5-2B | 441 tok/s | 323 tok/s | 52 tok/s | 28 tok/s |+| Qwen3-4B | 193 tok/s | 164 tok/s | 27 tok/s | 18 tok/s |+| Qwen3.5-9B | 111 tok/s | 78 tok/s | 14 tok/s | 8.7 tok/s |+| Qwen3-30B-A3B | 135 tok/s | 122 tok/s | 32 tok/s | 15 tok/s |+| Gemma-4-E2B-it | 191 tok/s | 150 tok/s | 38 tok/s | 19 tok/s |+| Gemma-4-E4B-it | 122 tok/s | 109 tok/s | 20 tok/s | 15 tok/s |+| Gemma-4-12B-it | 65 tok/s | 46 tok/s | 9.9 tok/s | 7.1 tok/s |+| Gemma-4-26B-A4B-it | 84 tok/s | 76 tok/s | 18 tok/s | 12 tok/s |++Janas's figures are one round each; the writing column is without drafts on both sides. The two tools do not measure in exactly the same way (`llama-bench` writes without a prompt before it; `janas-bench` after the 1,024 tokens), and it is one machine. llama.cpp's own CPU backend did worse on this laptop than its Vulkan one at reading (Qwen3-4B 70 tok/s on 20 threads) and as well at writing.**Try it on an idle machine first.** The expert cache takes the memory that is free when the model is opened, and what is left of the model is read from disk while it answers, so everything else running takes its share. It degrades gently rather than breaking: measured with a browser and an editor open, the same model kept about three quarters of its speed and waited five times longer on the disk. But the first thing you see should be the machine's real speed, and any measurement you mean to send to others has to be taken with the machine to itself.@@ -617,7 +635,7 @@ The [hardware report form](https://github.com/prabanta-dev/janas/issues/new?temp**What is most wanted, in order.**1. **CPUs without AVX-VNNI**, and **AMD** — the arithmetic has a path for them and it is the least exercised of all.-2. **Discrete GPUs** (NVIDIA, AMD, Intel Arc). The GPU support is correct, bit for bit, and it pays where it was measured: on the development laptop's integrated GPU a dense model whose weights all fit in memory reads a prompt about 29% faster (Qwen3-4B, 83 to 107 tokens/s), and on an RTX 5080 (issue #15) Qwen3-4B read 13% faster and, with the GPU on single tokens too, replied 11-16% faster. Every other card is unknown. **Careful:** a GPU driver reset takes the desktop with it. Start with `janas-bench`, not with a long chat.+2. **Discrete GPUs** (NVIDIA, AMD, Intel Arc). The GPU support is correct - bit for bit on a reply's tokens, and on a prompt as close to the references as the CPU - and it pays where it was measured: on the development laptop's integrated GPU, with the fast prompt, Qwen3-4B reads a prompt 2.6 times faster than without the GPU (85 to 220 tokens/s; before the fast prompt, 83 to 107), and on an RTX 5080 (issue #15) Qwen3-4B read 13% faster and, with the GPU on single tokens too, replied 11-16% faster. Every other card is unknown, and the fast prompt has not run on a discrete one yet (`JANAS_GPU_FAST_PROMPT=0` turns it off if it misbehaves). **Careful:** a GPU driver reset takes the desktop with it. Start with `janas-bench`, not with a long chat.3. **Machines with 16 GB or less.** The engine is supposed to trade quality for memory on its own — fewer bits per weight, light experts not in memory left out — and the thresholds for that were measured on 32 GB.4. **Slower disks.** Everything assumes an NVMe SSD; on a SATA disk the engine should still work and simply wait more, but nobody has watched it do so.5. **Other models** of the supported families. A file that will not convert, or converts and then answers nonsense, is a useful bug.diff --git a/src/common/env_names.c b/src/common/env_names.cindex 1f63a6a..d132aa7 100644--- a/src/common/env_names.c+++ b/src/common/env_names.c@@ -39,6 +39,10 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY","JANAS_GPU_FULL_N","JANAS_GPU_KEEPALIVE_US","JANAS_GPU_MIN_N",+ "JANAS_GPU_FAST_PROMPT",+ "JANAS_GPU_ARENA_INPLACE",+ "JANAS_GPU_EXPERT_MIN",+ "JANAS_GPU_EXPERT_SHARE","JANAS_GPU_NO_ROW","JANAS_GPU_NOSPIN","JANAS_GPU_REPACK",diff --git a/src/llm/api.c b/src/llm/api.cindex 253bea9..3d41cba 100644--- a/src/llm/api.c+++ b/src/llm/api.c@@ -306,12 +306,8 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,uint32_t ring = 0;snprintf(key, sizeof(key), "%.32s.attention.sliding_window", arch);if (janas_jns_meta_u32(&j, key, &window) == 0 && window) {- const char *pb = getenv("JANAS_PREFILL_BLOCK"),- *rg = getenv("JANAS_SWA_RING");- long blk = pb ? atol(pb) : 256;- ring = janas_attn_ring_positions(window, blk < 64 ? 64- : blk > 4096 ? 4096- : (uint32_t)blk);+ const char *rg = getenv("JANAS_SWA_RING");+ ring = janas_attn_ring_positions(window, janas_llm_prefill_block(&j));if (ring >= p.n_ctx || (rg && atoi(rg) == 0))ring = 0;}@@ -388,8 +384,9 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,}snprintf(key, sizeof(key), "%.32s.expert_used_count", arch);janas_jns_meta_u32(&j, key, &k_used);- const char *pb = getenv("JANAS_PREFILL_BLOCK");- uint64_t blk = pb && atol(pb) > 256 ? (uint64_t)atol(pb) : 256;+ uint64_t blk = janas_llm_prefill_block(&j);+ if (blk < 256)+ blk = 256;uint64_t scratch =blk * (((uint64_t)k_used + 1) * (3ull * ff + embd) + 32ull * embd) *sizeof(float);@@ -688,12 +685,15 @@ int32_t janas_llm_describe(const janas_llm *llm, char *buf, int32_t cap,if (janas_llm_model_gpu(llm->m)) {/* which passes it is given, in the words /mode uses for them */int use = janas_llm_model_gpu_used(llm->m);+ if (janas_llm_model_gpu_prompts(llm->m))+ use |= 1 << 2; /* the fast prompt: every block */if (!use)gpu = ", GPU present (not used)";else {- const char *where[3] = {"single tokens", "draft checks", "blocks"};- char list[64] = "";- for (int k = 0; k < 3; k++)+ const char *where[4] = {"single tokens", "draft checks", "blocks",+ "short blocks"};+ char list[96] = "";+ for (int k = 0; k < 4; k++)if (use & (1 << k))snprintf(list + strlen(list), sizeof(list) - strlen(list),"%s%s", list[0] ? " and " : "", where[k]);diff --git a/src/llm/expert_cache.c b/src/llm/expert_cache.cindex a000171..39941de 100644--- a/src/llm/expert_cache.c+++ b/src/llm/expert_cache.c@@ -478,7 +478,7 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,char path[64];snprintf(path, sizeof(path), "/proc/self/fd/%d", j->fd);c->fd = open(path, O_RDONLY | O_DIRECT | O_CLOEXEC);- size_t arena_bytes = n_slots * slot_bytes;+ size_t arena_bytes = n_slots * slot_bytes + JANAS_ARENA_GUARD;c->arena = aligned_alloc(JANAS_JNS_ALIGN, arena_bytes);c->where = malloc(n_keys * sizeof(int32_t));c->key = malloc(n_slots * sizeof(int64_t));@@ -758,7 +758,14 @@ const uint8_t *janas_expert_cache_resident(struct janas_expert_cache *c,pthread_mutex_unlock(&c->api_lock);if (busy || !full)return NULL;- *bytes = (uint64_t)c->n_slots * c->slot_bytes;+ *bytes = (uint64_t)c->n_slots * c->slot_bytes + JANAS_ARENA_GUARD;+ return c->arena;+}++const uint8_t *janas_expert_cache_arena(const struct janas_expert_cache *c,+ uint64_t *bytes)+{+ *bytes = (uint64_t)c->n_slots * c->slot_bytes + JANAS_ARENA_GUARD;return c->arena;}diff --git a/src/llm/expert_cache.h b/src/llm/expert_cache.hindex b3aa9b2..7885173 100644--- a/src/llm/expert_cache.h+++ b/src/llm/expert_cache.h@@ -66,13 +66,21 @@ size_t janas_expert_cache_slots(const struct janas_expert_cache *c);*/const uint8_t *janas_expert_cache_resident(struct janas_expert_cache *c,uint64_t *bytes, uint64_t *loads);+/* The arena and its bytes, whatever it holds: for a GPU that reads the+ host's memory where it is, which sees each slot as it is loaded. The+ bytes given here and by _resident count a guard page past the last+ slot (JANAS_ARENA_GUARD): the GPU's shaders read a few bytes beyond a+ matrix (Q6_K: 4), and the last slot's had nowhere to go. */+#define JANAS_ARENA_GUARD 4096+const uint8_t *janas_expert_cache_arena(const struct janas_expert_cache *c,+ uint64_t *bytes);uint64_t janas_expert_cache_loads(const struct janas_expert_cache *c);/*- * The arena moved to arena (n_slots x slot bytes, owned by the caller, not- * freed here), its contents copied, the file read without O_DIRECT from- * then on: for memory a GPU's driver allocated, which it reads faster- * than imported pages. Only when nothing is being read (as _resident); 0,- * or -1 with nothing changed.+ * The arena moved to arena (n_slots x slot bytes and the guard, owned by the+ * caller, not freed here), its contents copied, the file read without O_DIRECT+ * from then on: for memory a GPU's driver allocated, which it reads faster than+ * imported pages. Only when nothing is being read (as _resident); 0, or -1 with+ * nothing changed.*/int janas_expert_cache_rebase(struct janas_expert_cache *c, uint8_t *arena);diff --git a/src/llm/gpu.c b/src/llm/gpu.cindex 5a3bc65..6de8e39 100644--- a/src/llm/gpu.c+++ b/src/llm/gpu.c@@ -26,6 +26,15 @@#include "b32_matvec.h" /* janas_spv_b32_matvec */#include "iq4xs_matvec.h" /* janas_spv_iq4xs_matvec */#include "q3k_matvec.h" /* janas_spv_q3k_matvec */+#include "q4k_mmq.h" /* janas_spv_q4k_mmq */+#include "q6kp_mmq.h" /* janas_spv_q6kp_mmq */+#include "q6k_mmq.h" /* janas_spv_q6k_mmq */+#include "q80_mmq.h" /* janas_spv_q80_mmq */+#include "q5k_mmq.h" /* janas_spv_q5k_mmq */+#include "dense_mm.h" /* janas_spv_dense_mm */+#include "q4k_mmid.h" /* janas_spv_q4k_mmid */+#include "q6kp_mmid.h" /* janas_spv_q6kp_mmid */+#include "q6k_mmid.h" /* janas_spv_q6k_mmid */#include "iq3s_matvec.h" /* janas_spv_iq3s_matvec */#include "expert_act.h" /* janas_spv_expert_act */#include "q4k_row.h" /* janas_spv_q4k_row */@@ -496,7 +505,37 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)&g->pipe3, 0)) != VK_SUCCESS ||(r = janas_g_make_pipeline(g, janas_spv_iq3s_matvec,sizeof(janas_spv_iq3s_matvec), &g->smi3,- &g->pipei3, 0)) != VK_SUCCESS) {+ &g->pipei3, 0)) != VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q4k_mmq, sizeof(janas_spv_q4k_mmq),+ &g->sm_mm, &g->pipe_mm, 0, NV_PER_GROUP, g->sg)) !=+ VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q6kp_mmq, sizeof(janas_spv_q6kp_mmq),+ &g->sm_mm6, &g->pipe_mm6, 0, NV_PER_GROUP, g->sg)) !=+ VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q6k_mmq, sizeof(janas_spv_q6k_mmq),+ &g->sm_mm6s, &g->pipe_mm6s, 0, NV_PER_GROUP,+ g->sg)) != VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q80_mmq, sizeof(janas_spv_q80_mmq),+ &g->sm_mm80, &g->pipe_mm80, 0, NV_PER_GROUP,+ g->sg)) != VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q5k_mmq, sizeof(janas_spv_q5k_mmq),+ &g->sm_mm5, &g->pipe_mm5, 0, NV_PER_GROUP, g->sg)) !=+ VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_dense_mm, sizeof(janas_spv_dense_mm),+ &g->sm_dm[0], &g->pipe_dm[0], 0, NV_PER_GROUP,+ g->sg)) != VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_dense_mm, sizeof(janas_spv_dense_mm),+ &g->sm_dm[1], &g->pipe_dm[1], 1, NV_PER_GROUP,+ g->sg)) != VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q4k_mmid, sizeof(janas_spv_q4k_mmid),+ &g->sm_id4, &g->pipe_id4, 0, NV_PER_GROUP, g->sg)) !=+ VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q6kp_mmid, sizeof(janas_spv_q6kp_mmid),+ &g->sm_id6, &g->pipe_id6, 0, NV_PER_GROUP, g->sg)) !=+ VK_SUCCESS ||+ (r = make_pipeline(g, janas_spv_q6k_mmid, sizeof(janas_spv_q6k_mmid),+ &g->sm_id6s, &g->pipe_id6s, 0, NV_PER_GROUP,+ g->sg)) != VK_SUCCESS) {set_err(err, err_len, "cannot create the shaders", r);goto fail;}@@ -602,7 +641,8 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)g->vkCreateFence(g->dev, &fci, NULL, &g->fence) != VK_SUCCESS ||janas_g_make_stage(g, &g->xs, X_STAGE_BYTES,!getenv("JANAS_GPU_XCOH")) != 0 ||- janas_g_make_stage(g, &g->ys, Y_STAGE_BYTES, 1) != 0) {+ janas_g_make_stage(g, &g->ys, Y_STAGE_BYTES, 1) != 0 ||+ janas_g_make_stage(g, &g->ts, 64 << 10, 1) != 0) {snprintf(err, err_len, "cannot create the GPU buffers");goto fail;}@@ -623,8 +663,15 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)g->full_n = e && atoi(e) > 0 ? (size_t)atoi(e) : (size_t)-1;e = getenv("JANAS_GPU_VERBOSE");g->verbose = e && atoi(e) > 0 ? atoi(e) : 0;- g->nospin = getenv("JANAS_GPU_NOSPIN") != NULL;- g->block_wait = getenv("JANAS_GPU_BLOCKWAIT") != NULL;+ /* the pool's idle workers asleep while the GPU computes, and a+ blocking wait for it: on with the fast prompt (=0 turns either off),+ where the GPU computes nearly everything and the CPU's spinning draws+ the power it would use (the laptop's PL4) - Qwen3.5-2B read a prompt+ at 389 tokens/s instead of 315 (each alone: 346, 317; 10 Oct 2026) */+ const char *ns = getenv("JANAS_GPU_NOSPIN"),+ *bw = getenv("JANAS_GPU_BLOCKWAIT");+ g->nospin = ns ? atoi(ns) != 0 : janas_gpu_fast_prompt();+ g->block_wait = bw ? atoi(bw) != 0 : janas_gpu_fast_prompt();return g;missing:@@ -667,8 +714,8 @@ void janas_gpu_destroy(struct janas_gpu *g)int waited;janas_gpu_finish(g, &waited);}- struct stage *st[2] = {&g->xs, &g->ys};- for (int i = 0; i < 2; i++) {+ struct stage *st[3] = {&g->xs, &g->ys, &g->ts};+ for (int i = 0; i < 3; i++) {if (st[i]->buf)g->vkDestroyBuffer(g->dev, st[i]->buf, NULL);if (st[i]->mem)@@ -723,6 +770,42 @@ void janas_gpu_destroy(struct janas_gpu *g)g->vkDestroyPipeline(g->dev, g->pipex4, NULL);if (g->pipe3)g->vkDestroyPipeline(g->dev, g->pipe3, NULL);+ if (g->pipe_mm)+ g->vkDestroyPipeline(g->dev, g->pipe_mm, NULL);+ if (g->pipe_mm6)+ g->vkDestroyPipeline(g->dev, g->pipe_mm6, NULL);+ if (g->pipe_mm6s)+ g->vkDestroyPipeline(g->dev, g->pipe_mm6s, NULL);+ if (g->pipe_mm80)+ g->vkDestroyPipeline(g->dev, g->pipe_mm80, NULL);+ if (g->pipe_mm5)+ g->vkDestroyPipeline(g->dev, g->pipe_mm5, NULL);+ if (g->sm_mm5)+ g->vkDestroyShaderModule(g->dev, g->sm_mm5, NULL);+ for (int i = 0; i < 2; i++) {+ if (g->pipe_dm[i])+ g->vkDestroyPipeline(g->dev, g->pipe_dm[i], NULL);+ if (g->sm_dm[i])+ g->vkDestroyShaderModule(g->dev, g->sm_dm[i], NULL);+ }+ for (int i = 0; i < 3; i++) {+ VkPipeline pp = i == 2 ? g->pipe_id6s+ : i ? g->pipe_id6+ : g->pipe_id4;+ VkShaderModule ss = i == 2 ? g->sm_id6s : i ? g->sm_id6 : g->sm_id4;+ if (pp)+ g->vkDestroyPipeline(g->dev, pp, NULL);+ if (ss)+ g->vkDestroyShaderModule(g->dev, ss, NULL);+ }+ if (g->sm_mm6s)+ g->vkDestroyShaderModule(g->dev, g->sm_mm6s, NULL);+ if (g->sm_mm80)+ g->vkDestroyShaderModule(g->dev, g->sm_mm80, NULL);+ if (g->sm_mm6)+ g->vkDestroyShaderModule(g->dev, g->sm_mm6, NULL);+ if (g->sm_mm)+ g->vkDestroyShaderModule(g->dev, g->sm_mm, NULL);if (g->pipei3)g->vkDestroyPipeline(g->dev, g->pipei3, NULL);if (g->sm3)@@ -1157,6 +1240,7 @@ int janas_gpu_begin(struct janas_gpu *g, const struct janas_matvec_task *tasks,if (g->vkBeginCommandBuffer(g->cb, &bb) != VK_SUCCESS)return -1;VkPipeline bound = VK_NULL_HANDLE;+ size_t recorded = 0; /* products to run, written directly or copied */for (size_t i = 0; i < n; i++) {if (gpu_rows[i] == 0)continue;@@ -1186,6 +1270,21 @@ int janas_gpu_begin(struct janas_gpu *g, const struct janas_matvec_task *tasks,x_used += bytes;n_xc++;}+ /* an output the GPU shares (the engine's buffers, imported at+ load): written there, rows as the CPU lays them out, no copy */+ size_t ystr = t->y_stride ? t->y_stride : t->rows;+ const struct region *yr = janas_g_find_region(+ g, t->y, ((nv - 1) * ystr + gpu_rows[i]) * sizeof(float));+ if (yr) {+ if (janas_g_record_task(+ g, &bound, t, gpu_rows[i], g->xs.addr + xc[k].off,+ yr->addr ++ (VkDeviceAddress)((const uint8_t *)t->y - yr->host),+ ystr) != 0)+ return -1;+ recorded++;+ continue;+ }size_t ybytes = gpu_rows[i] * nv * sizeof(float);if (g->n_pend == MAX_PENDING || y_used + ybytes > Y_STAGE_BYTES)return -1;@@ -1193,6 +1292,7 @@ int janas_gpu_begin(struct janas_gpu *g, const struct janas_matvec_task *tasks,g->xs.addr + xc[k].off, g->ys.addr + y_used,gpu_rows[i]) != 0)return -1;+ recorded++;g->pend[g->n_pend++] =(struct pending){.y = t->y,.rows = gpu_rows[i],@@ -1212,7 +1312,7 @@ int janas_gpu_begin(struct janas_gpu *g, const struct janas_matvec_task *tasks,if (g->vkEndCommandBuffer(g->cb) != VK_SUCCESS)return -1;g->y_used = y_used;- if (g->n_pend == 0)+ if (recorded == 0)return 0;if (!g->xs.coherent && x_used > 0) {VkDeviceSize sz = (x_used + g->atom - 1) / g->atom * g->atom;@@ -1266,6 +1366,45 @@ static int prod_index(int type)return -1;}+/*+ * The fast prompt (on unless JANAS_GPU_FAST_PROMPT=0, since 10 October+ * 2026): a prompt's blocks go to the tiled shaders (*_mmq.comp,+ * *_mmid.comp, dense_mm.comp) and, where they can, whole to the GPU in one+ * submission (gpu_token.c); they round group by group and so are not bit+ * for bit with the CPU. Qwen3-4B, 256 tokens against Qwen's Q8_0+ * (9 Oct 2026): the most likely token 234 of 256 on code with it and+ * without, 219 and 217 on Italian, mean |diff| 0.7234 against 0.7249 and+ * 0.5620 against 0.5637; the block 10% faster on the development laptop.+ */+static int mm_on(void);++int janas_gpu_fast_prompt(void)+{+ return mm_on();+}++void janas_gpu_set_only_fast(struct janas_gpu *g, int on)+{+ g->only_fast = on;+}++static int mm_on(void)+{+ static int on = -1;+ if (on < 0)+ on = !getenv("JANAS_GPU_FAST_PROMPT") ||+ atoi(getenv("JANAS_GPU_FAST_PROMPT")) > 0;+ return on;+}++/* A product the fast prompt's shaders take. */+static int fast_type(const struct janas_matvec_task *t)+{+ return t->type == JANAS_Q4_K || t->type == JANAS_Q6_K ||+ t->type == JANAS_Q8_0 || t->type == JANAS_Q5_K ||+ (t->type == JANAS_Q6_K_P && plane_level(t) == 3);+}+int janas_g_record_task(struct janas_gpu *g, VkPipeline *bound,const struct janas_matvec_task *t, size_t rows,VkDeviceAddress xa, VkDeviceAddress ya, size_t ys)@@ -1320,6 +1459,32 @@ int janas_g_record_task(struct janas_gpu *g, VkPipeline *bound,} else if (nv == 1 && pk >= 0 && g->pp1[pk]) {want = g->pp1[pk];nvg = 1;+ } else if (t->type == JANAS_Q6_K_P && plane_level(t) == 3 &&+ nv >= JANAS_GPU_FAST_N && g->pipe_mm6 && mm_on()) {+ want = g->pipe_mm6;+ nvg = 64;+ rows_wg = 64;+ } else if (t->type == JANAS_Q6_K && nv >= JANAS_GPU_FAST_N &&+ g->pipe_mm6s && mm_on()) {+ want = g->pipe_mm6s;+ nvg = 64;+ rows_wg = 64;+ } else if (t->type == JANAS_Q5_K && nv >= JANAS_GPU_FAST_N && g->pipe_mm5 &&+ mm_on()) {+ want = g->pipe_mm5;+ nvg = 64;+ rows_wg = 64;+ } else if (t->type == JANAS_Q8_0 && nv >= JANAS_GPU_FAST_N &&+ g->pipe_mm80 && mm_on()) {+ want = g->pipe_mm80;+ nvg = 64;+ rows_wg = 64;+ } else if (t->type == JANAS_Q4_K && nv >= JANAS_GPU_FAST_N && g->pipe_mm &&+ mm_on()) {+ /* the fast prompt: tiles of 64 rows by 64 vectors */+ want = g->pipe_mm;+ nvg = 64;+ rows_wg = 64;}if (want != *bound) {g->vkCmdBindPipeline(g->cb, VK_PIPELINE_BIND_POINT_COMPUTE, want);@@ -1540,9 +1705,324 @@ static struct share *find_share(struct janas_gpu *g, uint64_t key)return NULL;}+/*+ * The fast prompt for a mixture's experts: a group of products of one type+ * and shape (the experts of a block: some 16 tokens each of 256 on+ * average, from 1 to 228 seen on Qwen3-30B-A3B) in one dispatch (q4k_mmid.comp,+ * q6kp_mmid.comp), their activations - one array, the pairs of the block in+ * order - converted once, every output written where the engine reads it. Such+ * a group asks too much of janas_gpu_begin (more than MAX_GROUP products, more+ * than MAX_XCACHE activations): a block's experts never reached the GPU before.+ * The GPU takes a fixed share of the tokens (experts_pick), the CPU the+ * rest meanwhile. 0, or -1 if the group is not of that kind (the caller+ * goes on as before).+ */+/* the fewest tokens an expert needs to be given to the GPU, and the+ share of a group's tokens it is given (experts_pick) */+#define EXPERT_GPU_MIN 1+#define EXPERT_GPU_SHARE 0.6++/* Which experts of a group go to the GPU (on[i] = 1): those with the most+ tokens first, at least JANAS_GPU_EXPERT_MIN, until the GPU has+ JANAS_GPU_EXPERT_SHARE of the group's tokens; the CPU computes the+ others meanwhile. A fixed share, so that the results do not depend on+ timings. On Qwen3-30B-A3B, blocks of 256 tokens, development laptop+ (9 Oct 2026), token/s: 75 with the GPU taking the experts of 32 tokens+ and more, 82 at a share of 0.3, 93 at 0.5, 99 at 0.6, 93 at 0.7, 86 at+ 0.8; in blocks of 512 (janas_llm_prefill_block), a prompt of 1024+ tokens: 112.7 at 0.4, 122.7 at 0.5, 119.2 at 0.6, 112.2 at 0.7; with+ the shaders reading each word once (10 Oct): 124.8 at 0.5, 131.3-133.4+ at 0.6, 125.7 at 0.7. The largest token count, 0 if none. */+static size_t experts_pick(const struct janas_matvec_task *t, size_t n,+ unsigned char *on)+{+ static size_t min_nv;+ static double share;+ if (!min_nv) {+ const char *e = getenv("JANAS_GPU_EXPERT_MIN");+ min_nv = e && atoi(e) > 0 ? (size_t)atoi(e) : EXPERT_GPU_MIN;+ e = getenv("JANAS_GPU_EXPERT_SHARE");+ share = e ? atof(e) : EXPERT_GPU_SHARE;+ }+ size_t total = 0, given = 0, maxnv = 0;+ for (size_t i = 0; i < n; i++) {+ on[i] = 0;+ total += t[i].n_vec ? t[i].n_vec : 1;+ }+ for (;;) {+ size_t best = n, bnv = 0;+ for (size_t i = 0; i < n; i++) {+ size_t nv = t[i].n_vec ? t[i].n_vec : 1;+ if (!on[i] && nv >= min_nv && nv > bnv) {+ best = i;+ bnv = nv;+ }+ }+ if (best == n || (double)(given + bnv) > share * (double)total + 0.5)+ break;+ on[best] = 1;+ given += bnv;+ if (bnv > maxnv)+ maxnv = bnv;+ }+ return maxnv;+}++static int experts_id(struct janas_gpu *g, struct janas_pool *pool,+ const struct janas_matvec_task *t, size_t n)+{+ if (!mm_on() || g->busy || g->broken || n < 2 || !g->pipe_id4 ||+ n * 48 > g->ts.bytes || n > 2 * 128 + 8)+ return -1;+ int type = t[0].type;+ if (!(type == JANAS_Q4_K || type == JANAS_Q6_K ||+ (type == JANAS_Q6_K_P && plane_level(&t[0]) == 3)))+ return -1;+ size_t nb = t[0].cols / JANAS_QK, xs = t[0].x_stride ? t[0].x_stride : nb;+ const struct janas_block_q8k *x0 = t[0].x, *x1 = t[0].x;+ for (size_t i = 0; i < n; i++) {+ size_t nv = t[i].n_vec ? t[i].n_vec : 1;+ if (t[i].type != type || t[i].rows != t[0].rows ||+ t[i].cols != t[0].cols ||+ (t[i].x_stride ? t[i].x_stride : nb) != xs ||+ !janas_gpu_can(g, &t[i])) {+ if (g->verbose > 1)+ fprintf(stderr,+ "experts_id: product %zu of %zu not taken "+ "(type %d, nv %zu, can %d)\n",+ i, n, t[i].type, nv, janas_gpu_can(g, &t[i]));+ return -1;+ }+ if (t[i].x < x0)+ x0 = t[i].x;+ if (t[i].x + nv * xs > x1)+ x1 = t[i].x + nv * xs;+ }+ /* experts: tokens of their own each. A dense feed-forward's gate and up+ (one "expert" in the cache, Qwen3-4B) read the same activations, and+ go whole to the GPU as before - shared, the CPU got half of them and+ the 4B read a prompt at 111 tokens/s instead of 127 (9 Oct 2026) */+ int own = 0;+ for (size_t i = 1; i < n && !own; i++)+ own = t[i].x != t[0].x;+ if (!own)+ return -1;+ /* a prompt's block only: generating, each expert has a token or a+ few, and the GPU's share cost the 30B three quarters of its speed */+ size_t tokens = 0;+ for (size_t i = 0; i < n; i++)+ tokens += t[i].n_vec ? t[i].n_vec : 1;+ if (tokens < JANAS_GPU_FAST_N * 2)+ return -1;+ unsigned char on[2 * 128 + 8];+ size_t maxnv = experts_pick(t, n, on);+ if (maxnv == 0)+ return -1; /* no expert worth the GPU: the CPU's as before */+ /* the activations: one array of vectors xs blocks apart */+ size_t span = (size_t)(x1 - x0);+ if (span % xs != 0 || span / xs * nb * GPU_BLOCK > X_STAGE_BYTES)+ return -1;+ for (size_t i = 0; i < n; i++)+ if ((size_t)(t[i].x - x0) % xs != 0)+ return -1;+ /* the table: weights, activations, outputs (shared with the GPU) */+ uint32_t *tb = (uint32_t *)g->ts.map;+ size_t ng = 0;+ for (size_t i = 0; i < n; i++) {+ size_t nv = t[i].n_vec ? t[i].n_vec : 1;+ if (!on[i])+ continue;+ size_t ys = t[i].y_stride ? t[i].y_stride : t[i].rows;+ const struct region *wr =+ janas_g_find_region(g, t[i].w, read_bytes(&t[i], t[i].rows));+ const struct region *yr = janas_g_find_region(+ g, t[i].y, ((nv - 1) * ys + t[i].rows) * sizeof(float));+ if (!wr || !yr)+ return -1;+ uint32_t *e = tb + 12 * ng++;+ janas_g_push_addr(+ e,+ wr->addr + (VkDeviceAddress)((const uint8_t *)t[i].w - wr->host));+ janas_g_push_addr(e + 2,+ g->xs.addr + (VkDeviceAddress)((size_t)(t[i].x - x0) /+ xs * nb * GPU_BLOCK));+ janas_g_push_addr(+ e + 4,+ yr->addr + (VkDeviceAddress)((const uint8_t *)t[i].y - yr->host));+ e[6] = (uint32_t)nv;+ e[7] = (uint32_t)ys;+ e[8] = e[9] = e[10] = e[11] = 0;+ if (type == JANAS_Q6_K_P)+ for (int q = 0; q < 2; q++) {+ const struct region *pr =+ plane_region(g, t[i].plane[q], t[i].rows, nb);+ if (!pr)+ return -1;+ janas_g_push_addr(+ e + (q == 0 ? 8 : 10),+ pr->addr ++ (VkDeviceAddress)((const uint8_t *)t[i].plane[q] -+ pr->host));+ }+ }+ double tc = g->verbose ? now() : 0;+ /* the workgroups' tiles: (expert, first vector) for each 16 vectors of+ each expert, so that none is launched for nothing - with a grid as+ tall as the busiest expert's tiles, most of a block's experts (some+ 16 tokens each) had a dozen empty workgroups each */+ size_t nt = 0;+ for (size_t k = 0; k < ng; k++)+ for (uint32_t v = 0; v < tb[12 * k + 6]; v += 16)+ nt++;+ if ((3 * ng + nt) * 16 > g->ts.bytes)+ return -1;+ for (size_t k = 0, at = 3 * ng * 4; k < ng; k++)+ for (uint32_t v = 0; v < tb[12 * k + 6]; v += 16, at += 4) {+ tb[at] = (uint32_t)k;+ tb[at + 1] = v;+ tb[at + 2] = tb[at + 3] = 0;+ }+ convert_x(g->xs.map, x0, xs, span / xs, nb, janas_g_x_in_32(type));+ if (g->verbose)+ g->t_conv += now() - tc;+ struct stage *fl[2] = {&g->xs, &g->ts};+ size_t used[2] = {span / xs * nb * GPU_BLOCK, (3 * ng + nt) * 16};+ for (int k = 0; k < 2; k++)+ if (!fl[k]->coherent) {+ VkDeviceSize sz = (used[k] + g->atom - 1) / g->atom * g->atom;+ VkMappedMemoryRange mr = {+ .sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE,+ .memory = fl[k]->mem,+ .size = sz < fl[k]->bytes ? sz : VK_WHOLE_SIZE};+ g->vkFlushMappedMemoryRanges(g->dev, 1, &mr);+ }+ if (g->vkResetCommandBuffer(g->cb, 0) != VK_SUCCESS)+ return -1;+ VkCommandBufferBeginInfo bb = {+ .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,+ .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};+ if (g->vkBeginCommandBuffer(g->cb, &bb) != VK_SUCCESS)+ return -1;+ g->vkCmdBindPipeline(g->cb, VK_PIPELINE_BIND_POINT_COMPUTE,+ type == JANAS_Q4_K ? g->pipe_id4+ : type == JANAS_Q6_K ? g->pipe_id6s+ : g->pipe_id6);+ uint32_t pc[5];+ janas_g_push_addr(pc, g->ts.addr);+ pc[2] = (uint32_t)t[0].rows;+ pc[3] = (uint32_t)nb;+ pc[4] = (uint32_t)(3 * ng);+ g->vkCmdPushConstants(g->cb, g->pl, VK_SHADER_STAGE_COMPUTE_BIT, 0,+ sizeof(pc), pc);+ g->vkCmdDispatch(g->cb, (uint32_t)((t[0].rows + 63) / 64), 1, (uint32_t)nt);+ VkMemoryBarrier mb = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER,+ .srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT,+ .dstAccessMask = VK_ACCESS_HOST_READ_BIT};+ g->vkCmdPipelineBarrier(g->cb, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT,+ VK_PIPELINE_STAGE_HOST_BIT, 0, 1, &mb, 0, NULL, 0,+ NULL);+ if (g->vkEndCommandBuffer(g->cb) != VK_SUCCESS)+ return -1;+ g->n_pend = 0;+ g->y_used = 0;+ VkSubmitInfo sub = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO,+ .commandBufferCount = 1,+ .pCommandBuffers = &g->cb};+ pthread_mutex_lock(&g->qlock);+ VkResult sr = g->vkQueueSubmit(g->queue, 1, &sub, g->fence);+ pthread_mutex_unlock(&g->qlock);+ if (sr != VK_SUCCESS) {+ g->broken = 1;+ return -1;+ }+ g->busy = 1;+ /* the experts with few tokens on the CPU meanwhile */+ struct janas_matvec_task cpu[2 * 128 + 8];+ size_t nc = 0;+ for (size_t i = 0; i < n; i++)+ if (!on[i] && nc < sizeof(cpu) / sizeof(*cpu))+ cpu[nc++] = t[i];+ if (nc > 0)+ janas_matvec_q4k_group(pool, cpu, nc);+ int waited;+ if (janas_gpu_finish(g, &waited) != 0) {+ /* the GPU failed: all on the CPU (the outputs are overwritten) */+ janas_matvec_q4k_group(pool, t, n);+ }+ return 0;+}++int janas_gpu_dense_mm(struct janas_gpu *g, int bf16, const void *w,+ const float *x, float *y, size_t rows, size_t cols,+ size_t n, float scale)+{+ if (!g || g->busy || g->broken || !mm_on() || n < JANAS_GPU_FAST_N ||+ cols % 16 || !g->pipe_dm[bf16 != 0])+ return -1;+ const struct region *wr =+ janas_g_find_region(g, w, rows * cols * (bf16 ? 2 : 4));+ const struct region *xr = janas_g_find_region(g, x, n * cols * 4);+ const struct region *yr = janas_g_find_region(g, y, n * rows * 4);+ if (!wr || !xr || !yr)+ return -1;+ if (g->vkResetCommandBuffer(g->cb, 0) != VK_SUCCESS)+ return -1;+ VkCommandBufferBeginInfo bb = {+ .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,+ .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};+ if (g->vkBeginCommandBuffer(g->cb, &bb) != VK_SUCCESS)+ return -1;+ g->vkCmdBindPipeline(g->cb, VK_PIPELINE_BIND_POINT_COMPUTE,+ g->pipe_dm[bf16 != 0]);+ struct {+ uint32_t w[2], x[2], y[2];+ uint32_t rows, cols, nv, ys;+ float scale;+ } pc = {.rows = (uint32_t)rows,+ .cols = (uint32_t)cols,+ .nv = (uint32_t)n,+ .ys = (uint32_t)rows,+ .scale = scale};+ janas_g_push_addr(+ pc.w, wr->addr + (VkDeviceAddress)((const uint8_t *)w - wr->host));+ janas_g_push_addr(+ pc.x, xr->addr + (VkDeviceAddress)((const uint8_t *)x - xr->host));+ janas_g_push_addr(+ pc.y, yr->addr + (VkDeviceAddress)((const uint8_t *)y - yr->host));+ g->vkCmdPushConstants(g->cb, g->pl, VK_SHADER_STAGE_COMPUTE_BIT, 0,+ sizeof(pc), &pc);+ g->vkCmdDispatch(g->cb, (uint32_t)((rows + 63) / 64),+ (uint32_t)((n + 63) / 64), 1);+ VkMemoryBarrier mb = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER,+ .srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT,+ .dstAccessMask = VK_ACCESS_HOST_READ_BIT};+ g->vkCmdPipelineBarrier(g->cb, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT,+ VK_PIPELINE_STAGE_HOST_BIT, 0, 1, &mb, 0, NULL, 0,+ NULL);+ if (g->vkEndCommandBuffer(g->cb) != VK_SUCCESS)+ return -1;+ g->n_pend = 0;+ g->y_used = 0;+ VkSubmitInfo sub = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO,+ .commandBufferCount = 1,+ .pCommandBuffers = &g->cb};+ pthread_mutex_lock(&g->qlock);+ VkResult sr = g->vkQueueSubmit(g->queue, 1, &sub, g->fence);+ pthread_mutex_unlock(&g->qlock);+ if (sr != VK_SUCCESS) {+ g->broken = 1;+ return -1;+ }+ g->busy = 1;+ int waited;+ return janas_gpu_finish(g, &waited) == 0 ? 0 : -1;+}+void janas_gpu_matvec_group(struct janas_gpu *g, struct janas_pool *pool,const struct janas_matvec_task *tasks, size_t n){+ if (g && experts_id(g, pool, tasks, n) == 0)+ return;size_t nv = tasks[0].n_vec ? tasks[0].n_vec : 1;if (!g || g->broken || nv < g->min_n || n > MAX_GROUP) {janas_matvec_q4k_group(pool, tasks, n);@@ -1573,7 +2053,9 @@ void janas_gpu_matvec_group(struct janas_gpu *g, struct janas_pool *pool,fprintf(stderr, "\n");}}- if (sh && sh->off > 0) {+ /* the fast prompt: its products on the GPU, whole, always */+ int fast = mm_on() && nv >= JANAS_GPU_FAST_N;+ if (sh && sh->off > 0 && !fast) {/* the GPU was late even with the smallest share: CPU for a while */sh->off--;janas_matvec_q4k_group(pool, tasks, n);@@ -1584,7 +2066,10 @@ void janas_gpu_matvec_group(struct janas_gpu *g, struct janas_pool *pool,for (size_t i = 0; i < n; i++) {const struct janas_matvec_task *t = &tasks[i];size_t r = 0;- if (nv >= g->full_n && janas_gpu_can(g, t)) {+ if (g->only_fast && !(fast && fast_type(t))) {+ r = 0; /* the CPU's pass: only the fast prompt's products here */+ } else if ((nv >= g->full_n || (fast && fast_type(t))) &&+ janas_gpu_can(g, t)) {r = t->rows; /* the GPU alone */} else if (sh && t->rows >= 2 * ROWS_PER_GROUP && janas_gpu_can(g, t)) {r = (size_t)(sh->f * (float)t->rows) / ROWS_PER_GROUP *@@ -1654,6 +2139,8 @@ void janas_gpu_matvec_group(struct janas_gpu *g, struct janas_pool *pool,}return;}+ if (!sh)+ return; /* the table of shares full: nothing to learn from */if (g->verbose) {sh->t_begin += t1 - t0;sh->t_cpu += t2 - t1;@@ -1766,6 +2253,22 @@ size_t janas_gpu_min_n(const struct janas_gpu *g)return (size_t)-1;}+int janas_gpu_dense_mm(struct janas_gpu *g, int bf16, const void *w,+ const float *x, float *y, size_t rows, size_t cols,+ size_t n, float scale)+{+ (void)g;+ (void)bf16;+ (void)w;+ (void)x;+ (void)y;+ (void)rows;+ (void)cols;+ (void)n;+ (void)scale;+ return -1;+}+void janas_gpu_matvec_group(struct janas_gpu *g, struct janas_pool *pool,const struct janas_matvec_task *tasks, size_t n){diff --git a/src/llm/gpu.h b/src/llm/gpu.hindex a94b287..24985af 100644--- a/src/llm/gpu.h+++ b/src/llm/gpu.h@@ -103,6 +103,33 @@ int janas_gpu_finish(struct janas_gpu *g, int *waited);void janas_gpu_matvec_group(struct janas_gpu *g, struct janas_pool *pool,const struct janas_matvec_task *tasks, size_t n);+/*+ * The fast prompt's product of weights left as they are, f32 (bf16 0) or+ * bf16 (1), rows x cols, with n vectors of cols floats in x: y[v * rows + r]+ * = scale x row r . vector v, on the GPU alone, waited for. Only for n of+ * JANAS_GPU_FAST_N and more, cols a multiple of 16, and w, x and y in memory+ * the GPU holds; 0, or -1 with nothing written (the caller computes it).+ */+int janas_gpu_dense_mm(struct janas_gpu *g, int bf16, const void *w,+ const float *x, float *y, size_t rows, size_t cols,+ size_t n, float scale);++/*+ * Whether the fast prompt is on (unless JANAS_GPU_FAST_PROMPT=0): blocks of+ * JANAS_GPU_FAST_N vectors and more of the types it has shaders for go to+ * the GPU whole, rounded group by group - close to the CPU's results, not+ * bit for bit - and, so that a prompt always gives the same results, every+ * such block goes to the GPU, whatever the tuner would choose.+ */+/* 64: a mixture's experts get some 16 tokens each of a block of 256, too+ few to fill a tile of 64 - forced whole onto the GPU, Qwen3-30B-A3B read+ a prompt at 84 tokens/s instead of 97 (9 Oct 2026) */+#define JANAS_GPU_FAST_N 64+int janas_gpu_fast_prompt(void);+/* For the passes that follow: only the fast prompt's products on the GPU,+ the rest on the CPU (a pass the tuner gave the CPU alone). */+void janas_gpu_set_only_fast(struct janas_gpu *g, int on);+/* Keeps the GPU out of its sleep states (an empty dispatch every 0.5 ms,JANAS_GPU_KEEPALIVE_US) while on: for the passes that use it. */void janas_gpu_keep_awake(struct janas_gpu *g, int on);diff --git a/src/llm/gpu_impl.h b/src/llm/gpu_impl.hindex 7297739..07f841e 100644--- a/src/llm/gpu_impl.h+++ b/src/llm/gpu_impl.h@@ -26,10 +26,11 @@#include <unistd.h>#include <vulkan/vulkan.h>-#define MAX_REGIONS 32+#define MAX_REGIONS 48#define MAX_PENDING 64#define MAX_XCACHE 8-#define X_STAGE_BYTES ((size_t)8 << 20)+#define X_STAGE_BYTES \+ ((size_t)16 << 20) /* a 512-token block's experts: 10.5 MB */#define Y_STAGE_BYTES ((size_t)32 << 20)#define ROWS_PER_GROUP 64 /* local_size_x of the shader */#define NV_PER_GROUP 8 /* its NV */@@ -146,6 +147,14 @@ struct janas_gpu {/* blocks of 32: Q4_0, IQ4_NL, Q4_1, Q5_1 (b32_matvec's KIND) */VkPipeline pipe32[4];VkPipeline pipex4, pipe3, pipei3; /* IQ4_XS, Q3_K, IQ3_S */+ /* Q4_K on many vectors in tiles (q4k_mm.comp), MM_ROWS x MM_VECS */+ VkShaderModule sm_mm, sm_mm6, sm_mm6s, sm_mm80, sm_mm5, sm_dm[2];+ VkPipeline pipe_mm, pipe_mm6, pipe_mm6s, pipe_mm80, pipe_mm5, pipe_dm[2];+ int only_fast; /* janas_gpu_set_only_fast */+ /* a mixture's experts in one dispatch (experts_id): Q4_K, Q6_K_P */+ VkShaderModule sm_id4, sm_id6, sm_id6s;+ VkPipeline pipe_id4, pipe_id6, pipe_id6s;+ struct stage ts; /* their table, 3 uvec4 an expert *//* the product shaders made for a single vector (prod_index) */VkShaderModule pm1[N_PROD];VkPipeline pp1[N_PROD];diff --git a/src/llm/gpu_token.c b/src/llm/gpu_token.cindex 5aba459..e0853ec 100644--- a/src/llm/gpu_token.c+++ b/src/llm/gpu_token.c@@ -6,6 +6,7 @@*/#include "gpu_token.h"+#include <math.h>#include <stddef.h>#include <stdio.h>#include <stdlib.h>@@ -21,10 +22,15 @@#include "qk_rope.h" /* janas_spv_qk_rope */#include "attn_decode.h" /* janas_spv_attn_decode */#include "attn_merge.h" /* janas_spv_attn_merge */+#include "gdn_conv.h" /* janas_spv_gdn_conv */+#include "gdn_cstate.h" /* janas_spv_gdn_cstate */+#include "gdn_l2.h" /* janas_spv_gdn_l2 */+#include "gdn_rec.h" /* janas_spv_gdn_rec */+#include "gdn_gate.h" /* janas_spv_gdn_gate *//* The push constants of each shader, laid out as its GLSL block. */struct push_nq {- uint32_t x[2], add[2], w[2], xo[2], q256[2], q32[2];+ uint32_t x[2], add[2], w[2], xo[2], q256[2], q32[2], xn[2];uint32_t d;float eps;};@@ -43,8 +49,38 @@ struct push_at {};struct push_am {uint32_t part[2], q256[2], q32[2];- uint32_t hd, nch, ps, pad;+ uint32_t hd, nch, ps, gs;uint32_t pos[2]; /* 8-byte aligned, as the GLSL block puts it */+ uint32_t gate[2];+};+/* the Gated DeltaNet shaders (gdn_*.comp) */+struct push_gconv {+ uint32_t mixed[2], out[2], w[2], state[2], pos[2];+ uint32_t cc, d, hs, so;+};+struct push_gcst {+ uint32_t mixed[2], state[2], pos[2];+ uint32_t cc, d, hs, so, n;+};+struct push_gl2 {+ uint32_t x[2];+ uint32_t cc;+ float eps;+};+struct push_grec {+ uint32_t co[2], ba[2], att[2], state[2], pos[2], dt[2], a[2];+ uint32_t cc, nv, nkh, n, hs, so, vmap_mod;+ float scale;+};+struct push_ggate {+ uint32_t att[2], z[2], w[2], q256[2], q32[2];+ uint32_t d;+ float eps;+};+struct push_dm { /* gpu.c's dense_mm.comp */+ uint32_t w[2], x[2], y[2];+ uint32_t rows, cols, nv, ys;+ float scale;};/* The attention's positions a run (attn_decode.comp C), and the shapes itsshared memory takes: the query heads of a KV head times hd, and times a@@ -72,10 +108,11 @@ _Static_assert(sizeof(struct push_at) <= JANAS_G_PUSH_BYTES, "push size");_Static_assert(offsetof(struct push_qk, pos) % 8 == 0, "push layout");_Static_assert(offsetof(struct push_at, pos) % 8 == 0, "push layout");_Static_assert(offsetof(struct push_am, pos) % 8 == 0, "push layout");+_Static_assert(sizeof(struct push_grec) <= JANAS_G_PUSH_BYTES, "push size");/* The most tokens a block takes (the model's own blocks: model.hJANAS_LLM_MAX_BLOCK), and the memory the attention's runs may take forthem. */-#define TOK_MAX_BLOCK 64+#define TOK_MAX_BLOCK 256#define TOK_PART_BUDGET ((size_t)64 << 20)/* JANAS_GPU_TOKEN_PROFILE=1: the GPU's own timestamps after each phase of@@ -105,10 +142,13 @@ struct janas_gpu_tok {the position is read from the buffer at o_pos, so every block of ntokens submits the same commands */uint32_t nb;- VkCommandBuffer cb[TOK_MAX_BLOCK + 1][3];- int prof; /* timestamps in the commands being recorded (n = 1) */- VkQueryPool qp; /* timestamps, if profiling */- uint32_t n_ts; /* written this token */+ /* [n][all][hidden wanted] */+ VkCommandBuffer cb[TOK_MAX_BLOCK + 1][3][2];+ uint32_t cb_nch[TOK_MAX_BLOCK + 1][3]+ [2]; /* the runs each was recorded for */+ int prof; /* timestamps in the commands being recorded (n = 1) */+ VkQueryPool qp; /* timestamps, if profiling */+ uint32_t n_ts; /* written this token */uint8_t ph_of[MAX_TS]; /* the phase each timestamp closes */double ph_ns[PH_N];double wall_ns; /* submission to the fence, as the CPU sees it */@@ -116,15 +156,31 @@ struct janas_gpu_tok {uint64_t tokens;VkShaderModule sm_nq, sm_qk, sm_at, sm_am;VkPipeline p_nq, p_qk, p_at, p_am;+ /* qwen3.5's recurrent layers */+ VkShaderModule sm_gc, sm_gs, sm_gl, sm_gr, sm_gg;+ VkPipeline p_gc, p_gs, p_gl, p_gr, p_gg;+ size_t *par; /* per layer: its parameters' offsets in buf (PAR_*) */struct stage buf; /* every intermediate, one allocation *//* offsets into buf, each nb tokens' worth */- uint32_t nch; /* the attention's runs of positions, for the cache */- size_t part_tok; /* a token's runs, floats */+ uint32_t nch; /* the attention's runs of positions, for the cache */+ size_t part_floats; /* room for the runs of a block, floats */+ uint32_t rec_nch; /* the runs a token of the block being recorded has */size_t o_part;size_t o_x, o_x1, o_q256, o_q32, o_qkv, o_ywo, o_gu, o_h, o_yd, o_rope,o_logits, o_pos;+ /* qwen3.5: the normalized input (f32), q|k|v and z of the recurrent+ layers, their convolution's output, beta and alpha, their raw+ output, the query's gates, the final hidden states */+ size_t o_xnf, o_mix, o_z, o_conv, o_ba, o_att, o_gate, o_hid;};+/* A recurrent layer's parameters copied into buf: beta then alpha (2 nv x+ dm), the convolution (d_conv x conv_ch), dt, A (nv), the norm (ds) */+enum { PAR_BA, PAR_CONV, PAR_DT, PAR_A, PAR_NORM, PAR_N };++static void sync_range(struct janas_gpu *g, struct janas_gpu_tok *t, size_t off,+ size_t bytes, int flush);+int janas_gpu_shares_host(const struct janas_gpu *g){return g && !g->broken && !g->discrete && g->host_import &&@@ -164,13 +220,51 @@ static int check_model(const struct janas_gpu *g,md->n_head / md->n_kv * md->hd > ATT_MAXQ ||md->n_head / md->n_kv * ATT_RUN > ATT_MAXS)return fail(err, len, "a shape the token's shaders do not take");+ if (md->n_rec &&+ (md->ds != 128 || md->conv_ch % 256 || md->d_inner % 256 ||+ md->d_inner != md->nv * md->ds || md->nv > 64 || !md->nkh ||+ md->nv % md->nkh || md->d_conv < 2 || md->d_conv > 8 ||+ md->d_inner > md->dm * 4 ||+ md->conv_ch != 2 * md->nkh * md->ds + md->d_inner ||+ !addr_of(g, md->ssm_state,+ 2 * (size_t)md->n_rec * md->nv * md->ds * md->ds * 2) ||+ !addr_of(g, md->conv_state,+ 2 * (size_t)md->n_rec * (md->d_conv - 1) * md->conv_ch * 4)))+ return fail(err, len,+ "recurrent layers the token's shaders do not take");for (uint32_t l = 0; l < md->n_layer; l++) {const struct janas_gpu_tok_layer *ly = &md->layers[l];+ if (ly->rec) {+ const struct janas_matvec_task *r[6] = {&ly->ssm_qkv, &ly->ssm_z,+ &ly->ssm_out, &ly->gate,+ &ly->up, &ly->down};+ for (int i = 0; i < 6; i++)+ if (!janas_gpu_can(g, r[i])) {+ snprintf(err, len,+ "layer %u: a weight the GPU cannot compute "+ "(product %d, type %d)",+ l, i, r[i]->type);+ return -1;+ }+ if (janas_g_x_in_32(ly->down.type))+ return fail(err, len, "a down matrix in blocks of 32");+ if (!addr_of(g, ly->attn_norm, md->dm * 4) ||+ !addr_of(g, ly->ffn_norm, md->dm * 4))+ return fail(err, len, "a norm out of the GPU's reach");+ continue;+ }const struct janas_matvec_task *t[7] = {&ly->wq, &ly->wk, &ly->wv, &ly->wo, &ly->gate, &ly->up, &ly->down};for (int i = 0; i < 7; i++)- if (!janas_gpu_can(g, t[i]))- return fail(err, len, "a weight the GPU cannot compute");+ if (!janas_gpu_can(g, t[i])) {+ snprintf(err, len,+ "layer %u: a weight the GPU cannot compute (product "+ "%d, type %d)",+ l, i, t[i]->type);+ return -1;+ }+ if (md->gated_q && (ly->wq.type == JANAS_Q6_K_P || ly->wq.plane[0]))+ return fail(err, len, "a gated query in planes");if (janas_g_x_in_32(ly->down.type))return fail(err, len, "a down matrix in blocks of 32");size_t cache = (size_t)md->n_kv * md->cap * md->hd;@@ -211,11 +305,15 @@ struct janas_gpu_tok *janas_gpu_tok_create(struct janas_gpu *g,size_t at = 0;/* the attention's runs: per token, query head and run, hd sums, m andl; as many tokens a block as they leave room for */+ /* laid out for the runs the positions reached so far have, not for+ the whole cache's (janas_gpu_tok_block_at): a prompt's blocks stay+ long at any context - sized for the cache, 16384 positions left+ Qwen3-4B 30 tokens a block, too few for the fast prompt's tiles */t->nch = (md->cap + ATT_RUN - 1) / ATT_RUN;- t->part_tok = (size_t)md->n_head * t->nch * (md->hd + 2);- size_t nb = TOK_PART_BUDGET / (t->part_tok * 4);- t->nb = nb < 1 ? 1 : nb > TOK_MAX_BLOCK ? TOK_MAX_BLOCK : (uint32_t)nb;- nb = t->nb;+ size_t full = (size_t)md->n_head * t->nch * (md->hd + 2);+ t->part_floats = TOK_PART_BUDGET / 4 > full ? TOK_PART_BUDGET / 4 : full;+ t->nb = TOK_MAX_BLOCK;+ size_t nb = t->nb;t->o_x = take(&at, nb * md->dm * 4);t->o_x1 = take(&at, nb * md->dm * 4); /* the residual's other buffer */t->o_q256 = take(&at, nb * big / JANAS_QK * 320);@@ -228,7 +326,34 @@ struct janas_gpu_tok *janas_gpu_tok_create(struct janas_gpu *g,t->o_rope = take(&at, nb * md->n_rot * 4);t->o_logits = take(&at, nb * md->n_vocab * 4);t->o_pos = take(&at, 16);- t->o_part = take(&at, nb * t->part_tok * 4);+ t->o_part = take(&at, t->part_floats * 4);+ t->o_xnf = take(&at, nb * md->dm * 4);+ t->o_hid = take(&at, nb * md->dm * 4);+ if (md->gated_q)+ t->o_gate = take(&at, nb * qd * 4);+ if (md->n_rec) {+ t->o_mix = take(&at, nb * md->conv_ch * 4);+ t->o_z = take(&at, nb * md->d_inner * 4);+ t->o_conv = take(&at, nb * md->conv_ch * 4);+ t->o_ba = take(&at, nb * 2 * md->nv * 4);+ t->o_att = take(&at, nb * md->d_inner * 4);+ t->par = calloc((size_t)md->n_layer * PAR_N, sizeof(size_t));+ if (!t->par) {+ fail(err, err_len, "out of memory");+ janas_gpu_tok_destroy(g, t);+ return NULL;+ }+ for (uint32_t l = 0; l < md->n_layer; l++) {+ if (!md->layers[l].rec)+ continue;+ size_t *pr = t->par + (size_t)l * PAR_N;+ pr[PAR_BA] = take(&at, 2 * (size_t)md->nv * md->dm * 4);+ pr[PAR_CONV] = take(&at, (size_t)md->d_conv * md->conv_ch * 4);+ pr[PAR_DT] = take(&at, md->nv * 4);+ pr[PAR_A] = take(&at, md->nv * 4);+ pr[PAR_NORM] = take(&at, md->ds * 4);+ }+ }/* cached for the host (the logits are read on the CPU: 2.5 ms a tokenfrom uncached memory on the development laptop), flushed andinvalidated by hand where it is not coherent */@@ -243,11 +368,42 @@ struct janas_gpu_tok *janas_gpu_tok_create(struct janas_gpu *g,&t->p_at, md->n_head / md->n_kv, att_sg()) == 0 ||janas_g_make_pipeline(g, janas_spv_attn_merge,sizeof(janas_spv_attn_merge), &t->sm_am, &t->p_am,- 0) != VK_SUCCESS) {+ 0) != VK_SUCCESS ||+ (md->n_rec &&+ (janas_g_make_pipeline(g, janas_spv_gdn_conv,+ sizeof(janas_spv_gdn_conv), &t->sm_gc, &t->p_gc,+ 0) != VK_SUCCESS ||+ janas_g_make_pipeline(g, janas_spv_gdn_cstate,+ sizeof(janas_spv_gdn_cstate), &t->sm_gs,+ &t->p_gs, 0) != VK_SUCCESS ||+ janas_g_make_pipeline(g, janas_spv_gdn_l2, sizeof(janas_spv_gdn_l2),+ &t->sm_gl, &t->p_gl, 0) != VK_SUCCESS ||+ janas_g_make_pipeline(g, janas_spv_gdn_rec, sizeof(janas_spv_gdn_rec),+ &t->sm_gr, &t->p_gr, 0) != VK_SUCCESS ||+ janas_g_make_pipeline(g, janas_spv_gdn_gate,+ sizeof(janas_spv_gdn_gate), &t->sm_gg, &t->p_gg,+ 0) != VK_SUCCESS ||+ !g->pipe_dm[0]))) {fail(err, err_len, "cannot create the token's buffers or shaders");janas_gpu_tok_destroy(g, t);return NULL;}+ /* the recurrent layers' parameters, copied once */+ for (uint32_t l = 0; md->n_rec && l < md->n_layer; l++) {+ const struct janas_gpu_tok_layer *ly = &md->layers[l];+ if (!ly->rec)+ continue;+ const size_t *pr = t->par + (size_t)l * PAR_N;+ size_t nvd = (size_t)md->nv * md->dm * 4;+ memcpy(t->buf.map + pr[PAR_BA], ly->ssm_beta, nvd);+ memcpy(t->buf.map + pr[PAR_BA] + nvd, ly->ssm_alpha, nvd);+ memcpy(t->buf.map + pr[PAR_CONV], ly->conv_w,+ (size_t)md->d_conv * md->conv_ch * 4);+ memcpy(t->buf.map + pr[PAR_DT], ly->dt, md->nv * 4);+ memcpy(t->buf.map + pr[PAR_A], ly->ssm_a, md->nv * 4);+ memcpy(t->buf.map + pr[PAR_NORM], ly->ssm_norm, md->ds * 4);+ sync_range(g, t, pr[PAR_BA], pr[PAR_NORM] + md->ds * 4 - pr[PAR_BA], 1);+ }const char *pe = getenv("JANAS_GPU_TOKEN_PROFILE");if (pe && atoi(pe) > 0 && g->ts_period > 0) {VkQueryPoolCreateInfo qi = {@@ -285,11 +441,15 @@ void janas_gpu_tok_destroy(struct janas_gpu *g, struct janas_gpu_tok *t)}for (uint32_t n = 0; n <= TOK_MAX_BLOCK; n++)for (int a = 0; a < 3; a++)- if (t->cb[n][a])- g->vkFreeCommandBuffers(g->dev, g->cp, 1, &t->cb[n][a]);- VkPipeline p[4] = {t->p_nq, t->p_qk, t->p_at, t->p_am};- VkShaderModule s[4] = {t->sm_nq, t->sm_qk, t->sm_at, t->sm_am};- for (int i = 0; i < 4; i++) {+ for (int h = 0; h < 2; h++)+ if (t->cb[n][a][h])+ g->vkFreeCommandBuffers(g->dev, g->cp, 1, &t->cb[n][a][h]);+ VkPipeline p[9] = {t->p_nq, t->p_qk, t->p_at, t->p_am, t->p_gc,+ t->p_gs, t->p_gl, t->p_gr, t->p_gg};+ VkShaderModule s[9] = {t->sm_nq, t->sm_qk, t->sm_at, t->sm_am, t->sm_gc,+ t->sm_gs, t->sm_gl, t->sm_gr, t->sm_gg};+ free(t->par);+ for (int i = 0; i < 9; i++) {if (p[i])g->vkDestroyPipeline(g->dev, p[i], NULL);if (s[i])@@ -351,10 +511,11 @@ static void push(struct janas_gpu *g, const void *pc, size_t bytes)static void rec_norm_quant(struct janas_gpu *g, struct janas_gpu_tok *t,VkPipeline *bound, VkDeviceAddress x,VkDeviceAddress add, VkDeviceAddress w,- VkDeviceAddress xo, uint32_t d, float eps,- uint32_t n)+ VkDeviceAddress xo, VkDeviceAddress xn, uint32_t d,+ float eps, uint32_t n){struct push_nq pc = {.d = d, .eps = eps};+ janas_g_push_addr(pc.xn, xn);janas_g_push_addr(pc.x, x);janas_g_push_addr(pc.add, add);janas_g_push_addr(pc.w, w);@@ -389,9 +550,123 @@ static int rec_product(struct janas_gpu *g, struct janas_gpu_tok *t,comes from the buffer.A single token takes the layers' GPU copies (JANAS_GPU_REPACK), a blockthe weights as they are (the copies' shaders take one vector). */+/* The runs of ATT_RUN positions npos positions take, rounded up to a power+ of two (so that a prompt's blocks record their commands again only a few+ times), at most the cache's. */+static uint32_t runs_for(const struct janas_gpu_tok *t, uint32_t npos)+{+ uint32_t c = (npos + ATT_RUN - 1) / ATT_RUN, r = 1;+ while (r < c)+ r *= 2;+ return r < t->nch ? r : t->nch;+}++/* A token's runs, floats, at nch runs a head. */+static size_t part_tok(const struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md, uint32_t nch)+{+ (void)t;+ return (size_t)md->n_head * nch * (md->hd + 2);+}++/* A Gated DeltaNet layer (qwen3.5) on the block normalized in q256 / q32+ and XNF: its output into YWO, as model_rec.c's janas_m_rec_layer. */+static int rec_gdn(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly, uint32_t l,+ VkPipeline *bound, VkDeviceAddress YWO, uint32_t n)+{+ VkDeviceAddress B = t->buf.addr, MIX = B + t->o_mix, Z = B + t->o_z,+ CONV = B + t->o_conv, BA = B + t->o_ba, ATT = B + t->o_att,+ POS = B + t->o_pos;+ const size_t *pr = t->par + (size_t)l * PAR_N;+ uint32_t cc = md->conv_ch;+ /* q|k|v (the convolution's channels) and z */+ if (rec_product(g, t, bound, &ly->ssm_qkv, 0, MIX, n, cc) != 0 ||+ rec_product(g, t, bound, &ly->ssm_z, 0, Z, n, md->d_inner) != 0)+ return -1;+ /* beta then alpha, per token, from the normalized input at f32 */+ struct push_dm pd = {.rows = 2 * md->nv,+ .cols = md->dm,+ .nv = n,+ .ys = 2 * md->nv,+ .scale = 1.0f};+ janas_g_push_addr(pd.w, B + pr[PAR_BA]);+ janas_g_push_addr(pd.x, B + t->o_xnf);+ janas_g_push_addr(pd.y, BA);+ bind(g, bound, g->pipe_dm[0]);+ push(g, &pd, sizeof(pd));+ g->vkCmdDispatch(g->cb, (2 * md->nv + 63) / 64, (n + 63) / 64, 1);+ janas_g_compute_barrier(g);+ /* the convolution, then the state it leaves, the q and k heads' norms */+ uint32_t hs_c = md->n_rec * (md->d_conv - 1) * cc,+ so_c = ly->slot * (md->d_conv - 1) * cc;+ VkDeviceAddress CS = addr_of(g, md->conv_state, (size_t)hs_c * 8);+ struct push_gconv pc = {.cc = cc, .d = md->d_conv, .hs = hs_c, .so = so_c};+ janas_g_push_addr(pc.mixed, MIX);+ janas_g_push_addr(pc.out, CONV);+ janas_g_push_addr(pc.w, B + pr[PAR_CONV]);+ janas_g_push_addr(pc.state, CS);+ janas_g_push_addr(pc.pos, POS);+ bind(g, bound, t->p_gc);+ push(g, &pc, sizeof(pc));+ g->vkCmdDispatch(g->cb, cc / 256, n, 1);+ janas_g_compute_barrier(g);+ struct push_gcst ps = {+ .cc = cc, .d = md->d_conv, .hs = hs_c, .so = so_c, .n = n};+ janas_g_push_addr(ps.mixed, MIX);+ janas_g_push_addr(ps.state, CS);+ janas_g_push_addr(ps.pos, POS);+ bind(g, bound, t->p_gs);+ push(g, &ps, sizeof(ps));+ g->vkCmdDispatch(g->cb, cc / 256, 1, 1);+ struct push_gl2 pl = {.cc = cc, .eps = md->eps};+ janas_g_push_addr(pl.x, CONV);+ bind(g, bound, t->p_gl);+ push(g, &pl, sizeof(pl));+ g->vkCmdDispatch(g->cb, 2 * md->nkh, n, 1);+ janas_g_compute_barrier(g);+ /* the recurrence, then each head's norm, gate and quantization */+ uint32_t hs_s = md->n_rec * md->nv * md->ds * md->ds,+ so_s = ly->slot * md->nv * md->ds * md->ds;+ struct push_grec pg = {.cc = cc,+ .nv = md->nv,+ .nkh = md->nkh,+ .n = n,+ .hs = hs_s,+ .so = so_s,+ .vmap_mod = (uint32_t)md->vmap_mod,+ .scale = 1.0f / sqrtf((float)md->ds)};+ janas_g_push_addr(pg.co, CONV);+ janas_g_push_addr(pg.ba, BA);+ janas_g_push_addr(pg.att, ATT);+ janas_g_push_addr(pg.state, addr_of(g, md->ssm_state, (size_t)hs_s * 4));+ janas_g_push_addr(pg.pos, POS);+ janas_g_push_addr(pg.dt, B + pr[PAR_DT]);+ janas_g_push_addr(pg.a, B + pr[PAR_A]);+ bind(g, bound, t->p_gr);+ push(g, &pg, sizeof(pg));+ g->vkCmdDispatch(g->cb, md->nv * (md->ds / 16), 1, 1);+ janas_g_compute_barrier(g);+ struct push_ggate pq = {.d = md->d_inner, .eps = md->eps};+ janas_g_push_addr(pq.att, ATT);+ janas_g_push_addr(pq.z, Z);+ janas_g_push_addr(pq.w, B + pr[PAR_NORM]);+ janas_g_push_addr(pq.q256, B + t->o_q256);+ janas_g_push_addr(pq.q32, B + t->o_q32);+ bind(g, bound, t->p_gg);+ push(g, &pq, sizeof(pq));+ g->vkCmdDispatch(g->cb, md->d_inner / JANAS_QK, n, 1);+ janas_g_compute_barrier(g);+ if (rec_product(g, t, bound, &ly->ssm_out, 0, YWO, n, 0) != 0)+ return -1;+ janas_g_compute_barrier(g);+ return 0;+}+static int record(struct janas_gpu *g, struct janas_gpu_tok *t,const struct janas_gpu_tok_model *md, uint32_t n,- uint32_t first)+ uint32_t first, int hid){uint32_t qd = md->n_head * md->hd, kvd = md->n_kv * md->hd;size_t qs = qd + 2 * kvd;@@ -424,13 +699,35 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,/* the previous layer's feed-forward added, then this one's norm */rec_norm_quant(g, t, &bound, X, l ? YD : 0,addr_of(g, ly->attn_norm, md->dm * 4), l ? XN : 0,- md->dm, md->eps, n);+ ly->rec ? B + t->o_xnf : 0, md->dm, md->eps, n);if (l)SW = X, X = XN, XN = SW;stamp(g, t, PH_NORM);- /* q, k and v of each token side by side, qs floats a token */- if (rec_product(g, t, &bound, &ly->wq, 0, QKV, n, qs) != 0 ||- rec_product(g, t, &bound, &ly->wk, 0, QKV + (size_t)qd * 4, n,+ if (ly->rec) {+ if (rec_gdn(g, t, md, ly, l, &bound, YWO, n) != 0)+ return -1;+ goto ffn;+ }+ /* q, k and v of each token side by side, qs floats a token; a+ gated query (qwen3.5) head by head, its gates apart */+ if (md->gated_q) {+ size_t rb = (size_t)(md->dm / JANAS_QK) *+ janas_qtype_block_size(ly->wq.type);+ for (uint32_t h = 0; h < md->n_head; h++) {+ struct janas_matvec_task q = ly->wq, gt = ly->wq;+ q.rows = gt.rows = md->hd;+ q.w = (const uint8_t *)ly->wq.w + (size_t)2 * h * md->hd * rb;+ gt.w = (const uint8_t *)q.w + (size_t)md->hd * rb;+ if (rec_product(g, t, &bound, &q, 0,+ QKV + (size_t)h * md->hd * 4, n, qs) != 0 ||+ rec_product(g, t, &bound, >, 0,+ B + t->o_gate + (size_t)h * md->hd * 4, n,+ qd) != 0)+ return -1;+ }+ } else if (rec_product(g, t, &bound, &ly->wq, 0, QKV, n, qs) != 0)+ return -1;+ if (rec_product(g, t, &bound, &ly->wk, 0, QKV + (size_t)qd * 4, n,qs) != 0 ||rec_product(g, t, &bound, &ly->wv, 0, QKV + (size_t)(qd + kvd) * 4,n, qs) != 0)@@ -465,10 +762,10 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,.n_kv = md->n_kv,.hd = md->hd,.cap = md->cap,- .nch = t->nch,+ .nch = t->rec_nch,.scale = md->attn_scale,.qs = (uint32_t)qs,- .ps = (uint32_t)t->part_tok};+ .ps = (uint32_t)part_tok(t, md, t->rec_nch)};janas_g_push_addr(pa.pos, POS);janas_g_push_addr(pa.q, QKV);janas_g_push_addr(pa.kc, addr_of(g, ly->kc, cache));@@ -478,13 +775,16 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,janas_g_push_addr(pa.part, B + t->o_part);bind(g, &bound, t->p_at);push(g, &pa, sizeof(pa));- g->vkCmdDispatch(g->cb, md->n_kv * t->nch, n, 1);+ g->vkCmdDispatch(g->cb, md->n_kv * t->rec_nch, n, 1);janas_g_compute_barrier(g);stamp(g, t, PH_ATTN);/* the runs joined and the output quantized for wo, a workgroup ablock of 256 */- struct push_am pm = {- .hd = md->hd, .nch = t->nch, .ps = (uint32_t)t->part_tok};+ struct push_am pm = {.hd = md->hd,+ .nch = t->rec_nch,+ .ps = (uint32_t)part_tok(t, md, t->rec_nch),+ .gs = qd};+ janas_g_push_addr(pm.gate, md->gated_q ? B + t->o_gate : 0);janas_g_push_addr(pm.part, B + t->o_part);janas_g_push_addr(pm.q256, B + t->o_q256);janas_g_push_addr(pm.q32, B + t->o_q32);@@ -499,8 +799,9 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,return -1;janas_g_compute_barrier(g);stamp(g, t, PH_WO);+ ffn:rec_norm_quant(g, t, &bound, X, YWO,- addr_of(g, ly->ffn_norm, md->dm * 4), XN, md->dm,+ addr_of(g, ly->ffn_norm, md->dm * 4), XN, 0, md->dm,md->eps, n);SW = X, X = XN, XN = SW;stamp(g, t, PH_NORM);@@ -538,10 +839,12 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,/* the last feed-forward added, the final norm, the output head fortokens first .. n - 1 (none: the residual left in X, the next block'skeys and values already in the cache) */- if (first < n) {+ if (first < n || hid) {rec_norm_quant(g, t, &bound, X, YD,- addr_of(g, md->out_norm, md->dm * 4), 0, md->dm, md->eps,- n);+ addr_of(g, md->out_norm, md->dm * 4), 0,+ hid ? B + t->o_hid : 0, md->dm, md->eps, n);+ }+ if (first < n) {if (rec_product(g, t, &bound, output,(size_t)first * (md->dm / JANAS_QK) * 320,B + t->o_logits, n - first, 0) != 0)@@ -564,12 +867,27 @@ uint32_t janas_gpu_tok_max_block(const struct janas_gpu_tok *t)return t ? t->nb : 0;}+uint32_t janas_gpu_tok_block_at(const struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ uint32_t pos)+{+ if (!t)+ return 0;+ size_t n = t->part_floats / part_tok(t, md, runs_for(t, pos + t->nb));+ return n < 1 ? 1 : n > t->nb ? t->nb : (uint32_t)n;+}+int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,const struct janas_gpu_tok_model *md, const float *x,const float *cs, const float *sn, uint32_t pos,- uint32_t n, int all, float *logits)+ uint32_t n, int all, float *logits, float *hidden,+ uint32_t src, uint32_t dst){- if (!t || g->busy || g->broken || n < 1 || n > t->nb || pos + n > md->cap)+ if (!t || g->busy || g->broken || n < 1 || n > t->nb || pos + n > md->cap ||+ (md->n_rec && n + 1 < md->d_conv))+ return -1;+ uint32_t nch = runs_for(t, pos + n);+ if ((size_t)n * part_tok(t, md, nch) > t->part_floats)return -1;uint32_t half = md->n_rot / 2, first = all == 1 ? 0 : all == 2 ? n : n - 1;memcpy(t->buf.map + t->o_x, x, (size_t)n * md->dm * sizeof(float));@@ -579,11 +897,20 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,memcpy(r + half * sizeof(float), sn + (size_t)j * half,half * sizeof(float));}- memcpy(t->buf.map + t->o_pos, &pos, sizeof(pos));+ /* the position, then the halves of the recurrent state to read and to+ write, read by the shaders at run time */+ uint32_t pw[3] = {pos, src, dst};+ memcpy(t->buf.map + t->o_pos, pw, sizeof(pw));sync_range(g, t, t->o_x, (size_t)n * md->dm * sizeof(float), 1);sync_range(g, t, t->o_rope, (size_t)n * md->n_rot * sizeof(float), 1);- sync_range(g, t, t->o_pos, sizeof(pos), 1);- VkCommandBuffer *cb = &t->cb[n][all == 1 && n == 1 ? 0 : all];+ sync_range(g, t, t->o_pos, sizeof(pw), 1);+ int ai = all == 1 && n == 1 ? 0 : all, hi = hidden != NULL;+ VkCommandBuffer *cb = &t->cb[n][ai][hi];+ if (*cb && t->cb_nch[n][ai][hi] != nch) {+ /* recorded for other runs: again (a few times in a context) */+ g->vkFreeCommandBuffers(g->dev, g->cp, 1, cb);+ *cb = VK_NULL_HANDLE;+ }if (!*cb) {VkCommandBufferAllocateInfo ca = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO,@@ -596,7 +923,9 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,}VkCommandBuffer keep = g->cb;g->cb = *cb; /* the helpers record into g->cb */- int rc = record(g, t, md, n, first);+ t->rec_nch = nch;+ t->cb_nch[n][ai][hi] = nch;+ int rc = record(g, t, md, n, first, hi);g->cb = keep;if (rc != 0) {g->vkFreeCommandBuffers(g->dev, g->cp, 1, cb);@@ -625,6 +954,11 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,sync_range(g, t, t->o_logits, lb, 0);memcpy(logits, t->buf.map + t->o_logits, lb);}+ if (hidden) {+ size_t hb = (size_t)n * md->dm * 4;+ sync_range(g, t, t->o_hid, hb, 0);+ memcpy(hidden, t->buf.map + t->o_hid, hb);+ }struct timespec w2;clock_gettime(CLOCK_MONOTONIC, &w2);if (t->qp && n == 1 && t->n_ts > 1) {@@ -669,10 +1003,21 @@ uint32_t janas_gpu_tok_max_block(const struct janas_gpu_tok *t)return 0;}+uint32_t janas_gpu_tok_block_at(const struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ uint32_t pos)+{+ (void)t;+ (void)md;+ (void)pos;+ return 0;+}+int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,const struct janas_gpu_tok_model *md, const float *x,const float *cs, const float *sn, uint32_t pos,- uint32_t n, int all, float *logits)+ uint32_t n, int all, float *logits, float *hidden,+ uint32_t src, uint32_t dst){(void)n;(void)all;@@ -684,6 +1029,9 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,(void)sn;(void)pos;(void)logits;+ (void)hidden;+ (void)src;+ (void)dst;return -1;}diff --git a/src/llm/gpu_token.h b/src/llm/gpu_token.hindex 71c162c..3454c38 100644--- a/src/llm/gpu_token.h+++ b/src/llm/gpu_token.h@@ -7,12 +7,13 @@* (private to the library).** For now the dense attention models (qwen3's shape: q and k norms, NeoX- * RoPE, an eight-bit KV cache, SiLU feed-forward) on a GPU that shares the- * host's memory: the weights, the experts' arena and the KV cache are the- * CPU's own memory, imported, so the CPU's passes (a prompt in blocks) and- * the GPU's tokens read and write the same cache. The products are the- * CPU's bit for bit (their shaders); the norms and the attention sum in- * the GPU's order.+ * RoPE, an eight-bit KV cache, SiLU feed-forward), and qwen3.5's (Gated+ * DeltaNet layers between them, the query with an output gate) for blocks,+ * on a GPU that shares the host's memory: the weights, the experts' arena and+ * the KV cache are the CPU's own memory, imported, so the CPU's passes (a+ * prompt in blocks) and the GPU's tokens read and write the same cache. The+ * products are the CPU's bit for bit (their shaders); the norms and the+ * attention sum in the GPU's order.*/#ifndef JANAS_LLM_GPU_TOKEN_H#define JANAS_LLM_GPU_TOKEN_H@@ -30,6 +31,14 @@ struct janas_gpu_tok_layer {struct janas_matvec_task wq, wk, wv, wo, gate, up, down;int8_t *kc, *vc; /* the layer's KV cache: n_kv x cap x hd bytes */float *ks, *vs; /* its scales: n_kv x cap */+ /* a Gated DeltaNet layer (qwen3.5) in place of the attention: q|k|v and+ z products, beta then alpha (2 nv x dm, f32), the convolution+ (d_conv x conv_ch, transposed), dt and A (nv), the heads' norm (ds),+ the output product; its state slot */+ int rec;+ struct janas_matvec_task ssm_qkv, ssm_z, ssm_out;+ const float *ssm_beta, *ssm_alpha, *conv_w, *dt, *ssm_a, *ssm_norm;+ uint32_t slot;};struct janas_gpu_tok_model {@@ -43,6 +52,15 @@ struct janas_gpu_tok_model {are; NULL: layers and output */const struct janas_gpu_tok_layer *blayers;struct janas_matvec_task boutput;+ /* qwen3.5: wq's rows head by head, each head's q then its output gate;+ the recurrent layers' shapes and their states, two halves each (the+ model's double buffer): n_rec x nv x ds x ds sixteen-bit numbers and+ n_rec x (d_conv - 1) x conv_ch floats */+ int gated_q;+ uint32_t n_rec, ds, nv, nkh, d_conv, conv_ch, d_inner;+ int vmap_mod;+ uint16_t *ssm_state;+ float *conv_state;};struct janas_gpu_tok;@@ -56,20 +74,29 @@ void janas_gpu_tok_destroy(struct janas_gpu *g, struct janas_gpu_tok *t);/* The most tokens janas_gpu_tok_run takes at once (0 without t). */uint32_t janas_gpu_tok_max_block(const struct janas_gpu_tok *t);+/* ... starting at position pos: fewer as the context grows (the+ attention's runs take room). */+uint32_t janas_gpu_tok_block_at(const struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ uint32_t pos);/** A block of n tokens (1 .. janas_gpu_tok_max_block) whose embeddings are x* (n x dm floats), token j at position pos + j (its keys and values go to* row pos + j of the cache; positions 0 .. pos + j attended), cs and sn* their RoPE tables (n x n_rot / 2 each): the logits (n_vocab each) of- * the last token (all 0), of every token (1) or of none (2) into logits. 0, or- * -1 with nothing changed but the cache rows pos .. pos + n - 1 (the caller- * runs the block on the CPU then).+ * the last token (all 0), of every token (1) or of none (2) into logits;+ * with hidden, the final hidden states (n x dm: the last norm's output, an+ * MTP block's input) there. The recurrent layers read their state from+ * half src and write it to half dst (the same for a block's later pieces).+ * 0, or -1 with nothing changed but the cache rows pos .. pos + n - 1 (the+ * caller runs the block on the CPU then).*/int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,const struct janas_gpu_tok_model *md, const float *x,const float *cs, const float *sn, uint32_t pos,- uint32_t n, int all, float *logits);+ uint32_t n, int all, float *logits, float *hidden,+ uint32_t src, uint32_t dst);/* Whether the GPU reads the host's memory in place (an integrated GPU withthe host-memory import, not JANAS_GPU_COPY): what the token needs. */diff --git a/src/llm/model.c b/src/llm/model.cindex 8b67360..ba7237a 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -461,6 +461,11 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,where it can (model_gpu.c) */if (logits && janas_m_gpu_token(m, pos0, n, all_logits, logits) == 0) {m->ring_hi = pos0 + n;+ if (m->a->rec) { /* a long block (model_gpu.c): as at the end */+ m->base = 1 - m->src;+ m->base_pos = pos0 + n;+ m->log_n = 0;+ }return 0;}for (uint32_t l = 0; l < m->n_layer; l++) {@@ -549,8 +554,7 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,.cols = dm,.n_vec = n - first};if (logits) /* without: the hidden states alone, an embedding */- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &to, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &to, 1);if (logits && m->softcap > 0.0f) {struct tok_job sc = {.m = m, .n = n - first, .out = logits};janas_pool_run(m->compute, softcap_worker, &sc);@@ -674,8 +678,7 @@ int janas_llm_mtp_draft(struct janas_llm_model *m, const int32_t *tokens,.y = m->dh_logits,.rows = m->dh_n,.cols = m->d_model};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &t, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &t, 1);/* the best row and its softmax probability within the head */float *l = m->dh_logits, mx = l[0];uint32_t best = 0;@@ -736,8 +739,7 @@ int janas_llm_mtp_forward(struct janas_llm_model *m, const int32_t *tokens,.rows = dm,.cols = 2 * dm,.n_vec = n};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &tp, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &tp, 1);m->phase[JANAS_PH_QKV] += now() - ta;set_rope(m, n, pos0);for (uint32_t j = 0; j < n; j++) {@@ -773,8 +775,7 @@ int janas_llm_mtp_forward(struct janas_llm_model *m, const int32_t *tokens,.rows = m->n_vocab,.cols = dm,.n_vec = n - first};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &to, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &to, 1);m->phase[JANAS_PH_OUTPUT] += now() - td;return 0;}diff --git a/src/llm/model.h b/src/llm/model.hindex 74551c7..0c63e8f 100644--- a/src/llm/model.h+++ b/src/llm/model.h@@ -116,10 +116,20 @@ int janas_llm_model_settling(const struct janas_llm_model *m);taken back (a draft) is at most this long. */#define JANAS_LLM_MAX_BLOCK 64+/*+ * The prefill block of the model in j: JANAS_PREFILL_BLOCK, else 512 for a+ * model whose feed-forwards (its experts, or the one of a dense model in+ * the expert cache) the GPU's fast prompt takes whole (gate and up Q4_K,+ * down Q4_K or Q6_K, plain or in planes; janas_gpu_fast_prompt on), else+ * 256;+ * 64 .. 4096.+ */+uint32_t janas_llm_prefill_block(const struct janas_jns *j);+/** Largest block of tokens one forward call accepts: larger than* JANAS_LLM_MAX_BLOCK, so that a prefill reads each expert once for many- * tokens (256 unless JANAS_PREFILL_BLOCK says otherwise, 64 .. 4096). A call+ * tokens (janas_llm_prefill_block). A call* longer than JANAS_LLM_MAX_BLOCK cannot be taken back on a model with a* recurrent state: the next call starts where it ended.*/@@ -262,8 +272,12 @@ void janas_llm_model_tuning(const struct janas_llm_model *m, char *buf,* part worth telling anybody.*/int janas_llm_model_gpu_used(const struct janas_llm_model *m);+/* Whether the GPU reads the prompts' blocks whatever the tuner chooses+ (the fast prompt, outside eco mode). */+int janas_llm_model_gpu_prompts(const struct janas_llm_model *m);/* The compute threads of the configuration the tuner holds best for a- kind of pass (janas_tune_class: 0 one token, 2 a block); 0 unknown. */+ kind of pass (janas_tune_class: 0 one token, 1 a few, 2 a block, 3 a+ short one); 0 unknown. */int janas_llm_model_threads(const struct janas_llm_model *m, int cls);int janas_llm_model_candidates(const struct janas_llm_model *m, int *threads,diff --git a/src/llm/model_attn.c b/src/llm/model_attn.cindex 809d994..905333e 100644--- a/src/llm/model_attn.c+++ b/src/llm/model_attn.c@@ -249,8 +249,7 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,.rows = kvd,.cols = dm,.n_vec = n}};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, t, (size_t)nt);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, t, (size_t)nt);if (nt == 2) /* Gemma 4's full layers: V is K, before K's norm */memcpy(m->vv, m->kk, (size_t)n * kvd * sizeof(float));int8_t *kc = m->kcache + m->kv_off[ly->slot];@@ -303,8 +302,7 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,.rows = dm,.cols = qd,.n_vec = n};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &to, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &to, 1);if (ly->post_attn)janas_m_post_norm(m, f32(m, ly->post_attn), n);janas_m_residual(m, n);diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.cindex 71275d6..e15b419 100644--- a/src/llm/model_ffn.c+++ b/src/llm/model_ffn.c@@ -195,7 +195,12 @@ struct expert_run {uint32_t a, b;};-#define MAX_RUNS 65 /* a chunk of 64 routed experts and the shared one */+/* Experts are asked of the cache and computed a chunk at a time: 128, a+ layer of Qwen3-30B-A3B at once - in chunks of 64 the GPU got half a+ layer's experts per launch, and ran them at 340 GMAC/s instead of 420+ (9 Oct 2026). */+#define EXPERT_CHUNK 128+#define MAX_RUNS (EXPERT_CHUNK + 1) /* a chunk and the shared expert */#define EX_T(v, code) \do { \@@ -253,14 +258,12 @@ static void run_experts(struct janas_llm_model *m,}if (janas_m_exp_tracing())ex_setup += now() - t_enter;- EX_T(ex_t1, janas_gpu_matvec_group(- m->gpu_off || !m->gpu_use || m->arena_stale ? NULL : m->gpu,- m->compute, t1, 2 * (size_t)n_runs));+ EX_T(ex_t1, janas_gpu_matvec_group(janas_m_gpu(m, 1), m->compute, t1,+ 2 * (size_t)n_runs));struct act_job aj = {m, ng};EX_T(ex_act, janas_pool_run(m->compute, act_worker, &aj));- EX_T(ex_t2, janas_gpu_matvec_group(- m->gpu_off || !m->gpu_use || m->arena_stale ? NULL : m->gpu,- m->compute, t2, n_runs));+ EX_T(ex_t2,+ janas_gpu_matvec_group(janas_m_gpu(m, 1), m->compute, t2, n_runs));}int janas_m_route_rank(struct janas_llm_model *m, uint32_t l, uint32_t n,@@ -549,10 +552,10 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,/* experts in chunks: compute those in RAM while the others load; theshared expert, resident, goes with the first of them */- for (uint32_t c0 = 0; c0 < nd || (c0 == 0 && nt > np); c0 += 64) {- uint32_t cn = nd - c0 < 64 ? nd - c0 : 64;- const uint8_t *slots[64];- uint8_t ready[64];+ for (uint32_t c0 = 0; c0 < nd || (c0 == 0 && nt > np); c0 += EXPERT_CHUNK) {+ uint32_t cn = nd - c0 < EXPERT_CHUNK ? nd - c0 : EXPERT_CHUNK;+ const uint8_t *slots[EXPERT_CHUNK];+ uint8_t ready[EXPERT_CHUNK];if (cn) {int rc_b;EX_T(ex_cache, rc_b = janas_expert_cache_begin(@@ -596,7 +599,7 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,if (nr)run_experts(m, runs, nr);}- if (nd <= c0 + 64)+ if (nd <= c0 + EXPERT_CHUNK)break;}if (n < 2) {diff --git a/src/llm/model_gpu.c b/src/llm/model_gpu.cindex 7bfce4a..661ccd2 100644--- a/src/llm/model_gpu.c+++ b/src/llm/model_gpu.c@@ -5,9 +5,10 @@* (gpu_token.h): which models it takes, their layers described for it, the* KV cache shared with it, and the call from the forward pass.*- * For now opt-in (JANAS_GPU_TOKEN=1) and for the dense attention models of- * qwen3's shape on a GPU that reads the host's memory in place, with the- * feed-forward weights (the experts' arena) on it already.+ * For the dense attention models of qwen3's shape, and qwen3.5's (Gated+ * DeltaNet layers between them) for a prompt's blocks only, on a GPU that+ * reads the host's memory in place, with the feed-forward weights (the+ * experts' arena) on it already.*/#include "model.h"@@ -28,9 +29,12 @@ static const char *not_for_gpu(const struct janas_llm_model *m){if (!m->gpu || m->gpu_off || !janas_gpu_shares_host(m->gpu))return "no GPU reading the host's memory";- if (m->a->rec || m->a->swa || m->a->attn_gate || m->a->sandwich ||- m->a->gelu || m->kv16 || m->softcap > 0.0f || m->ple_dim || m->mtp ||- m->asst)+ /* qwen3.5: its recurrent layers and gated queries; an MTP block or a+ drafting model gets the final hidden states */+ int q35 = m->a->rec == JANAS_REC_Q35 && m->a->attn_gate;+ if ((m->a->rec && !q35) || (m->a->attn_gate && !q35) || m->a->swa ||+ m->a->sandwich || m->a->gelu || m->kv16 || m->softcap > 0.0f ||+ m->ple_dim)return "an architecture the token's shaders do not take yet";if (m->n_expert != 1 || !m->cache)return "not a dense model";@@ -38,8 +42,14 @@ static const char *not_for_gpu(const struct janas_llm_model *m)return "the feed-forward weights are not on the GPU";for (uint32_t l = 0; l < m->n_layer; l++) {const struct layer *ly = &m->layers[l];- if (ly->rec || ly->swa || !ly->wk || !ly->wv || !ly->q_norm ||- !ly->k_norm || ly->post_attn || ly->post_ffn || ly->pre_norm2 ||+ if (ly->rec) {+ if (!ly->ssm_qkv || !ly->ssm_z || !ly->ssm_out || !ly->ssm_beta ||+ !ly->ssm_alpha || !ly->conv_t || !ly->ssm_norm)+ return "a layer the token's shaders do not take yet";+ continue;+ }+ if (ly->swa || !ly->wk || !ly->wv || !ly->q_norm || !ly->k_norm ||+ ly->post_attn || ly->post_ffn || ly->pre_norm2 ||m->kv_ring[ly->slot] || ly->hd != m->head_dim ||ly->n_kv != m->n_head_kv || ly->n_rot != m->n_rot)return "a layer the token's shaders do not take yet";@@ -223,13 +233,27 @@ static int build(struct janas_llm_model *m, char *err, size_t len)return -1;}g->attn_norm = f32(m, ly->attn_norm);- g->q_norm = f32(m, ly->q_norm);- g->k_norm = f32(m, ly->k_norm);g->ffn_norm = f32(m, ly->ffn_norm);- g->wq = task_of(m, ly->wq, qd, dm);- g->wk = task_of(m, ly->wk, kvd, dm);- g->wv = task_of(m, ly->wv, kvd, dm);- g->wo = task_of(m, ly->wo, dm, qd);+ if (ly->rec) {+ g->rec = 1;+ g->slot = ly->slot;+ g->ssm_qkv = task_of(m, ly->ssm_qkv, m->conv_ch, dm);+ g->ssm_z = task_of(m, ly->ssm_z, m->d_inner, dm);+ g->ssm_out = task_of(m, ly->ssm_out, dm, m->d_inner);+ g->ssm_beta = ly->ssm_beta;+ g->ssm_alpha = ly->ssm_alpha;+ g->conv_w = ly->conv_t;+ g->dt = f32(m, ly->dt_bias);+ g->ssm_a = f32(m, ly->ssm_a);+ g->ssm_norm = f32(m, ly->ssm_norm);+ } else {+ g->q_norm = f32(m, ly->q_norm);+ g->k_norm = f32(m, ly->k_norm);+ g->wq = task_of(m, ly->wq, m->a->attn_gate ? 2 * qd : qd, dm);+ g->wk = task_of(m, ly->wk, kvd, dm);+ g->wv = task_of(m, ly->wv, kvd, dm);+ g->wo = task_of(m, ly->wo, dm, qd);+ }for (int p = 0; p < 3; p++) {const struct janas_jns_matrix *mx = &jl->m[p];struct janas_matvec_task t = {.type = (int)mx->type,@@ -248,10 +272,19 @@ static int build(struct janas_llm_model *m, char *err, size_t len)snprintf(err, len, "feed-forwards of different widths");return -1;}- g->kc = m->kcache + m->kv_off[ly->slot];- g->vc = m->vcache + m->kv_off[ly->slot];- g->ks = m->kscale + m->ks_off[ly->slot];- g->vs = m->vscale + m->ks_off[ly->slot];+ if (!ly->rec) {+ g->kc = m->kcache + m->kv_off[ly->slot];+ g->vc = m->vcache + m->kv_off[ly->slot];+ g->ks = m->kscale + m->ks_off[ly->slot];+ g->vs = m->vscale + m->ks_off[ly->slot];+ }+ }+ uint32_t a0 = 0; /* the first attention layer: the cache's capacity */+ while (a0 < m->n_layer && m->layers[a0].rec)+ a0++;+ if (a0 == m->n_layer) {+ snprintf(err, len, "no attention layer");+ return -1;}m->gtok_md = (struct janas_gpu_tok_model){.n_layer = m->n_layer,@@ -262,12 +295,23 @@ static int build(struct janas_llm_model *m, char *err, size_t len).n_rot = m->n_rot,.ff = ff,.n_vocab = m->n_vocab,- .cap = m->kv_cap[m->layers[0].slot],+ .cap = m->kv_cap[m->layers[a0].slot],.eps = m->eps,.attn_scale = 1.0f / sqrtf((float)hd),.layers = m->gtok_layers,.out_norm = f32(m, m->out_norm),- .output = task_of(m, m->output, m->n_vocab, dm)};+ .output = task_of(m, m->output, m->n_vocab, dm),+ .gated_q = m->a->attn_gate,+ .n_rec = m->a->rec ? m->n_rec : 0,+ .ds = m->ds,+ .nv = m->n_vh,+ .nkh = m->n_kh,+ .d_conv = m->d_conv,+ .conv_ch = m->conv_ch,+ .d_inner = m->d_inner,+ .vmap_mod = m->vmap_mod,+ .ssm_state = m->ssm_buf,+ .conv_state = m->conv_buf};free(m->gblk_layers);m->gblk_layers = NULL;if (getenv("JANAS_GPU_REPACK") && atoi(getenv("JANAS_GPU_REPACK")) > 0) {@@ -297,26 +341,36 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,{if (m->gtok_state < 0)return -1;- /* blocks of more than one token: opt-in for now (JANAS_GPU_BLOCK=1) */- static int blocks = -1;- if (blocks < 0)- blocks =- getenv("JANAS_GPU_BLOCK") && atoi(getenv("JANAS_GPU_BLOCK")) > 0;- if (n > 1 && !blocks)- return -1;- if (m->gtok_state == 0) {- const char *e = getenv("JANAS_GPU_TOKEN");- if (!e || atoi(e) <= 0) {- m->gtok_state = -1;- return -1;- }+ /* JANAS_GPU_TOKEN=1: tokens, and blocks with JANAS_GPU_BLOCK=1. With the+ fast prompt a prompt's blocks (from JANAS_GPU_FAST_N tokens) come+ here by themselves, JANAS_GPU_BLOCK=0 aside: the whole block in one+ submission, the CPU idle - Qwen3-4B read a prompt of 1024 tokens at+ 163 tokens/s instead of 148 (llama.cpp on the same GPU: 164), top-1+ against the Q8_0 reference 235/256 as on the CPU's path (9 Oct 2026).+ Not the tokens of a reply: one at a time the GPU is slower (20+ tokens/s against 27.6). */+ static int tok = -1, blocks = -1;+ if (tok < 0) {+ const char *e = getenv("JANAS_GPU_TOKEN"),+ *b = getenv("JANAS_GPU_BLOCK");+ tok = e && atoi(e) > 0;+ blocks = b ? atoi(b) > 0 : -1; /* -1: not said */}+ int prompt =+ janas_gpu_fast_prompt() && n >= JANAS_GPU_FAST_N && blocks != 0;+ if (!prompt && !(tok && (n == 1 || blocks > 0)))+ return -1;+ /* a recurrent model: blocks that need no log (longer than a draft's)+ and replay nothing - the state read whole from one half, written to+ the other, as forward() expects of such a block */+ if (m->a->rec && (n <= JANAS_LLM_MAX_BLOCK || m->replay))+ return -1;const char *why = not_for_gpu(m);if (why) {/* the arena may reach the GPU later: wait for it, say nothing */if (m->gtok_state == 0 && m->arena_gpu == 0 && m->gpu && !m->gpu_off)return -1;- if (m->gtok_state == 0)+ if (m->gtok_state == 0 && tok) /* asked for, not by itself */fprintf(stderr, "janas: no whole token on the GPU: %s\n", why);m->gtok_state = -1;return -1;@@ -325,7 +379,8 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,janas_expert_cache_loads(m->cache) != m->gtok_loads) {char err[160];if (build(m, err, sizeof(err)) != 0) {- fprintf(stderr, "janas: no whole token on the GPU: %s\n", err);+ if (tok)+ fprintf(stderr, "janas: no whole token on the GPU: %s\n", err);m->gtok_state = -1;return -1;}@@ -333,11 +388,20 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,}/* a longer block (a prompt, only its last logits wanted) in pieces:each piece's keys and values in the cache before the next */- uint32_t nb = janas_gpu_tok_max_block(m->gtok), half = m->n_rot / 2;- if (n > nb && all)+ uint32_t half = m->n_rot / 2;+ if (all && n > janas_gpu_tok_block_at(m->gtok, &m->gtok_md, pos))return -1;- for (uint32_t j = 0; j < n; j += nb) {- uint32_t k = n - j < nb ? n - j : nb;+ /* the recurrent state: the first piece from half src to the other,+ the later ones on that one in place; each piece at least the+ convolution's rows of state long. With an MTP block or a drafting+ model, every token's final hidden state */+ uint32_t src = m->src, dst = 1 - m->src, rows = m->a->rec ? m->d_conv : 1;+ float *hid = m->mtp || m->asst ? m->hid : NULL;+ for (uint32_t j = 0, k; j < n; j += k) {+ uint32_t nb = janas_gpu_tok_block_at(m->gtok, &m->gtok_md, pos + j);+ k = n - j < nb ? n - j : nb;+ if (n - j - k > 0 && n - j - k + 1 < rows)+ k -= rows; /* the last piece long enough */if (janas_gpu_tok_run(m->gpu, m->gtok, &m->gtok_md,m->x + (size_t)j * m->d_model,m->rope_cos + (size_t)j * half,@@ -345,7 +409,8 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,j + k < n ? 2: all ? 1: 0,- logits) != 0)+ logits, hid ? hid + (size_t)j * m->d_model : NULL,+ j ? dst : src, dst) != 0)return -1;}return 0;diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex affdaa3..a6cba0a 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -91,7 +91,8 @@ struct janas_llm_model {true, the experts' products stay on the CPU */int arena_gpu, arena_stale;uint64_t arena_loads;- char gpu_why[160]; /* no GPU: why (janas_llm_model_gpu_why) */+ int arena_in_place; /* read by the GPU where it is: never stale */+ char gpu_why[160]; /* no GPU: why (janas_llm_model_gpu_why) */struct janas_expert_cache *cache;const struct janas_arch *a; /* what this architecture has (arch.h) */@@ -268,6 +269,12 @@ struct janas_llm_model {is left out, not read (see skip_misses); 0: never. And how many were */float miss_skip;uint64_t skip_asked, skip_left;+ /* the products' output buffers, whole pages, shared with the GPU */+ struct janas_out_buf {+ float *p;+ size_t bytes;+ } out[16];+ int n_out;int warm_use; /* preload the most used experts, and count them */uint32_t *use_count; /* experts used this session, n_layer * n_expert */uint64_t head_before, head_after; /* output head requantized (head_bits) */@@ -429,4 +436,21 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,const struct janas_jns_layer *jl, uint32_t l, uint32_t n,int part);+/*+ * The GPU a product may use, or NULL: the tuner's choice for this pass+ * (gpu_use); with the fast prompt the GPU too where the tuner chose the+ * CPU alone, for the fast prompt's products only (janas_gpu_set_only_fast),+ * which must always run there to give the same results. arena: a product+ * of the experts' arena, kept off the GPU while its copy there is stale.+ */+static inline struct janas_gpu *janas_m_gpu(const struct janas_llm_model *m,+ int arena)+{+ if (!m->gpu || m->gpu_off || (arena && m->arena_stale))+ return NULL;+ if (m->gpu_use || (m->tuner.allow_gpu && janas_gpu_fast_prompt()))+ return m->gpu;+ return NULL;+}+#endifdiff --git a/src/llm/model_load.c b/src/llm/model_load.cindex 25a9ddd..b56f47b 100644--- a/src/llm/model_load.c+++ b/src/llm/model_load.c@@ -13,6 +13,7 @@#include <stdio.h>#include <stdlib.h>#include <string.h>+#include <sys/mman.h>#include <time.h>#include <unistd.h>@@ -570,6 +571,17 @@ int janas_m_init_draft_head(struct janas_llm_model *m, const uint8_t *src,return 0;}+/* A buffer a product writes: whole pages, so that the GPU can be given it+ (janas_gpu_import), listed for that. */+static float *out_alloc(struct janas_llm_model *m, size_t bytes)+{+ bytes = (bytes + 4095) & ~(size_t)4095;+ float *p = aligned_alloc(4096, bytes ? bytes : 4096);+ if (p && m->n_out < (int)(sizeof(m->out) / sizeof(*m->out)))+ m->out[m->n_out++] = (struct janas_out_buf){p, bytes};+ return p;+}+struct janas_llm_model *janas_llm_model_load(const char *path,const struct janas_llm_options *o,char *err, size_t err_len)@@ -805,13 +817,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,const char *kv = getenv("JANAS_KV");m->kv16 = kv ? atoi(kv) == 16 : m->a->kv16;}- {- const char *pb = getenv("JANAS_PREFILL_BLOCK");- long v = pb ? atol(pb) : 256;- m->blk = v < JANAS_LLM_MAX_BLOCK ? JANAS_LLM_MAX_BLOCK- : v > 4096 ? 4096- : (uint32_t)v;- }+ m->blk = janas_llm_prefill_block(&m->j);/* the sliding-window layers keep a ring of positions, not the wholecontext: the window and a prefill block (JANAS_SWA_RING=0 keepsevery position, as the full layers do) */@@ -874,11 +880,11 @@ struct janas_llm_model *janas_llm_model_load(const char *path,size_t B = m->blk, P = B * (m->k + 1);size_t inner = m->d_inner > qd ? m->d_inner : qd;size_t bigw = m->qkvz_dim > 2 * qd ? m->qkvz_dim : 2 * qd;- m->x = malloc(B * dm * sizeof(float));- m->xn = malloc(B * dm * sizeof(float));- m->q = malloc(B * qd * sizeof(float));- m->kk = malloc(B * kvd * sizeof(float));- m->vv = malloc(B * kvd * sizeof(float));+ m->x = out_alloc(m, B * dm * sizeof(float)); /* the GPU reads it */+ m->xn = out_alloc(m, B * dm * sizeof(float));+ m->q = out_alloc(m, B * qd * sizeof(float));+ m->kk = out_alloc(m, B * kvd * sizeof(float));+ m->vv = out_alloc(m, B * kvd * sizeof(float));m->att = malloc(B * inner * sizeof(float));m->router = malloc(B * m->n_expert * sizeof(float));m->rope_cos = malloc(B * m->n_rot / 2 * sizeof(float));@@ -895,8 +901,9 @@ struct janas_llm_model *janas_llm_model_load(const char *path,}if (m->ple_dim) {size_t W = (size_t)m->n_layer * m->ple_dim;- m->ple = malloc(B * W * sizeof(float));- m->ple_g = malloc(B * m->ple_dim * sizeof(float));+ /* where the GPU writes the branches' products (janas_gpu_dense_mm) */+ m->ple = out_alloc(m, B * W * sizeof(float));+ m->ple_g = out_alloc(m, B * m->ple_dim * sizeof(float));m->ple_row = malloc(2 * W * sizeof(float));m->ple_q = malloc(B * m->ple_dim / JANAS_QK * sizeof(*m->ple_q));if (!m->ple || !m->ple_g || !m->ple_row || !m->ple_q) {@@ -912,10 +919,10 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->glist = malloc(P * sizeof(uint32_t));m->count = malloc((m->n_expert + 1) * sizeof(uint32_t));m->pair_w = malloc(P * sizeof(float));- m->gate = malloc(P * ff * sizeof(float));- m->up = malloc(P * ff * sizeof(float));+ m->gate = out_alloc(m, P * ff * sizeof(float));+ m->up = out_alloc(m, P * ff * sizeof(float));m->h = malloc(P * ff * sizeof(float));- m->dout = malloc(P * dm * sizeof(float));+ m->dout = out_alloc(m, P * dm * sizeof(float));m->xg = malloc(P * dm / JANAS_QK * sizeof(*m->xg));m->hq = malloc(P * ff / JANAS_QK * sizeof(*m->hq));m->hlen = malloc(P * sizeof(uint32_t));@@ -1003,7 +1010,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,#endifif (m->a->attn_gate || m->a->rec) {/* the gated queries, and the DeltaNet's fused projections */- m->big = malloc(B * bigw * sizeof(float));+ m->big = out_alloc(m, B * bigw * sizeof(float));m->qgate = malloc(B * qd * sizeof(float));if (!m->big || !m->qgate) {snprintf(err, err_len, "out of memory");@@ -1011,10 +1018,17 @@ struct janas_llm_model *janas_llm_model_load(const char *path,}}if (m->a->rec) {- m->ssm_buf = calloc(2 * (size_t)m->n_rec * m->n_vh * m->ds * m->ds,- sizeof(uint16_t));- m->conv_buf = calloc(- 2 * (size_t)m->n_rec * m->conv_ch * (m->d_conv - 1), sizeof(float));+ /* the GPU reads and writes them too (gpu_token.c): imported */+ size_t sb = 2 * (size_t)m->n_rec * m->n_vh * m->ds * m->ds *+ sizeof(uint16_t),+ cb = 2 * (size_t)m->n_rec * m->conv_ch * (m->d_conv - 1) *+ sizeof(float);+ m->ssm_buf = (uint16_t *)out_alloc(m, sb);+ m->conv_buf = out_alloc(m, cb);+ if (m->ssm_buf)+ memset(m->ssm_buf, 0, sb);+ if (m->conv_buf)+ memset(m->conv_buf, 0, cb);m->ba = malloc(B * 2 * m->n_vh * sizeof(float));/* the rows of the replayed tokens (a logged call, at mostJANAS_LLM_MAX_BLOCK), then the block */@@ -1181,6 +1195,12 @@ struct janas_llm_model *janas_llm_model_load(const char *path,if (m->mtp_res && janas_gpu_import(m->gpu, m->mtp_res,m->mtp_j.h.resident_bytes) != 0)m->gpu_off = 1;+ /* the products' outputs too, so that the GPU writes them where the+ engine reads them: through a staging buffer they cost a copy on+ one thread, 0.4-2.8 ms a block of a layer on Qwen3-4B (9 Oct+ 2026). One that cannot be shared keeps the copy. */+ for (int i = 0; i < m->n_out; i++)+ janas_gpu_import(m->gpu, m->out[i].p, m->out[i].bytes);}janas_m_init_tuner(m);return m;@@ -1461,18 +1481,74 @@ int janas_m_gpu_import(struct janas_llm_model *m, const uint8_t *p,* others. Checked after each pass until then; one change of a slot later* (a new level of bits) and the GPU's copy is no longer used.*/+uint32_t janas_llm_prefill_block(const struct janas_jns *j)+{+ /* 512 where the GPU takes the feed-forwards whole: a mixture's experts+ (gpu.c experts_id) get some 32 tokens each, not 16, and fill its+ tiles better - Qwen3-30B-A3B read a prompt of 1024 tokens at 122.7+ tokens/s instead of 108.5 (GPU share of the experts' tokens 0.5;+ 0.4: 112.7, 0.6: 119.2; blocks of 1024: 101); a dense model's one+ "expert" too, Qwen3-4B 134.6 instead of 127.0. Not for the others,+ whose products the CPU shares: Gemma 4 26B-A4B (experts' down Q5_1,+ the rest Q8_0) 63 instead of 68 (9 Oct 2026). Qwen3.5 (qwen35, its+ recurrent layers' projections Q5_K): 2B 278.9 -> 318.6, 9B 80.3 ->+ 86.0, 0.8B 489.0 -> 493.5 (10 Oct 2026). */+ /* measured on these architectures only: Qwen3-Next's recurrent layers+ and the others keep the block they had */+ char arch[32] = "";+ janas_jns_meta_str(j, "general.architecture", arch, sizeof(arch));+ int fast = janas_gpu_fast_prompt() && j->h.n_expert >= 1 && j->layers &&+ (!strcmp(arch, "qwen3") || !strcmp(arch, "qwen3moe") ||+ !strcmp(arch, "qwen35"));+ for (uint32_t l = 0; fast && l < j->h.n_layer; l++) {+ const struct janas_jns_matrix *x = j->layers[l].m;+ fast = x[0].type == JANAS_Q4_K && x[1].type == JANAS_Q4_K &&+ (x[2].type == JANAS_Q4_K || x[2].type == JANAS_Q6_K ||+ x[2].type == JANAS_Q6_K_P);+ }+ const char *pb = getenv("JANAS_PREFILL_BLOCK");+ long v = pb ? atol(pb) : fast ? 512 : 256;+ return v < JANAS_LLM_MAX_BLOCK ? JANAS_LLM_MAX_BLOCK+ : v > 4096 ? 4096+ : (uint32_t)v;+}+void janas_m_arena_to_gpu(struct janas_llm_model *m){if (!m->gpu || m->gpu_off || !m->cache || m->arena_gpu < 0)return;if (m->arena_gpu == 1) {- if (!m->arena_stale &&+ if (!m->arena_stale && !m->arena_in_place &&janas_expert_cache_loads(m->cache) != m->arena_loads)m->arena_stale = 1;return;}- uint64_t bytes, loads;+ uint64_t bytes, loads = 0;const uint8_t *a = janas_expert_cache_resident(m->cache, &bytes, &loads);+ /*+ * A GPU that reads the host's memory in place (an integrated one) needs+ * no complete cache: it reads each slot where the CPU does, as loaded,+ * and never holds a stale copy. Waiting for every expert of every layer+ * keeps a mixture's experts off it for good - Qwen3-30B-A3B's warm-up+ * loads the 5,620 of its 6,144 its prompts use, and the rest never come+ * (9 Oct 2026). Only the fast prompt gives the GPU experts (gpu.c+ * experts_id), so only with it, or with JANAS_GPU_ARENA_INPLACE=1.+ */+ int in_place =+ !a && (janas_gpu_fast_prompt() || getenv("JANAS_GPU_ARENA_INPLACE")) &&+ janas_gpu_shares_host(m->gpu) && !getenv("JANAS_GPU_ALLOC") &&+ !getenv("JANAS_GPU_TOKEN");+ if (in_place) {+ a = janas_expert_cache_arena(m->cache, &bytes);+#ifdef MADV_POPULATE_WRITE+ /* every page the arena's own first: a slot never loaded is the+ kernel's shared zero page, and each load that replaces one+ invalidates the import, pinned again whole (4 GB) at the next+ submission - 1.6 s of submissions on a block of 256 tokens of+ Qwen3-30B-A3B instead of 0.1 (9 Oct 2026) */+ madvise((void *)a, bytes, MADV_POPULATE_WRITE);+#endif+ }if (!a)return;uint64_t slot = janas_expert_cache_slot_bytes(m->cache);@@ -1488,9 +1564,17 @@ void janas_m_arena_to_gpu(struct janas_llm_model *m)janas_gpu_flush(m->gpu, own, bytes);m->arena_gpu = 1;m->arena_loads = loads;+ m->arena_in_place = in_place;+ if (getenv("JANAS_GPU_VERBOSE"))+ fprintf(stderr, "arena: %.1f GB in the GPU driver's memory%s\n",+ bytes / 1e9, in_place ? " (in place)" : "");} else if (janas_m_gpu_import(m, a, bytes, slot) == 0) {m->arena_gpu = 1;m->arena_loads = loads;+ m->arena_in_place = in_place;+ if (getenv("JANAS_GPU_VERBOSE"))+ fprintf(stderr, "arena: %.1f GB imported%s\n", bytes / 1e9,+ in_place ? " (in place)" : "");} else {m->arena_gpu = -1; /* no room for it: the CPU goes on alone */}diff --git a/src/llm/model_ple.c b/src/llm/model_ple.cindex fec7739..2cf99d6 100644--- a/src/llm/model_ple.c+++ b/src/llm/model_ple.c@@ -261,6 +261,9 @@ static void product(struct janas_llm_model *m, const struct janas_jns_tensor *t,{uint32_t cols = (uint32_t)t->dims[0], rows = (uint32_t)t->dims[1];if (t->type == 0) {+ if (janas_gpu_dense_mm(janas_m_gpu(m, 0), 0, f32(m, t), x, y, rows,+ cols, n, 1.0f) == 0)+ return;struct f32_job b = {.w = f32(m, t), .x = x, .y = y, .rows = rows, .cols = cols, .n = n};atomic_init(&b.next, 0);@@ -277,8 +280,7 @@ static void product(struct janas_llm_model *m, const struct janas_jns_tensor *t,.rows = rows,.cols = cols,.n_vec = n};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &tk, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &tk, 1);}/*@@ -303,7 +305,8 @@ int janas_m_ple_inputs(struct janas_llm_model *m, const int32_t *tokens,atomic_init(&b.next, 0);if (n < 2)bf16_worker(&b, 0, 1);- else+ else if (janas_gpu_dense_mm(janas_m_gpu(m, 0), 1, b.w, b.x, b.y, b.rows,+ b.cols, n, b.scale) != 0)janas_pool_run(m->compute, bf16_worker, &b);const float *nw = f32(m, m->ple_norm);size_t row_bytes =diff --git a/src/llm/model_rec.c b/src/llm/model_rec.cindex 4285438..df24896 100644--- a/src/llm/model_rec.c+++ b/src/llm/model_rec.c@@ -269,8 +269,7 @@ void janas_m_rec_layer(struct janas_llm_model *m, const struct layer *ly,.cols = dm,.n_vec = n,.y_stride = m->qkvz_dim}};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, t, 2);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, t, 2);/* beta then alpha per head, as in the grouped layout's order */const float *wb = ly->ssm_beta, *wa = ly->ssm_alpha;for (uint32_t j = 0; j < n; j++)@@ -296,8 +295,7 @@ void janas_m_rec_layer(struct janas_llm_model *m, const struct layer *ly,.rows = 2 * nv,.cols = dm,.n_vec = n}};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, t, 2);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, t, 2);}/* per key head g the projection holds q, k, the values and the z gatesof its rf value heads, then per g: rf betas and rf alphas */@@ -402,8 +400,7 @@ void janas_m_rec_layer(struct janas_llm_model *m, const struct layer *ly,.rows = dm,.cols = m->d_inner,.n_vec = n};- janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, &to, 1);+ janas_gpu_matvec_group(janas_m_gpu(m, 0), m->compute, &to, 1);trace_row(m, "ssm_out", m->xn + (size_t)(n - 1) * dm, dm);for (uint32_t i = 0; i < n * dm; i++)m->x[i] += m->xn[i];diff --git a/src/llm/model_tune.c b/src/llm/model_tune.cindex 10a3944..e5918bf 100644--- a/src/llm/model_tune.c+++ b/src/llm/model_tune.c@@ -64,23 +64,34 @@ void janas_m_init_tuner(struct janas_llm_model *m)c[n++] =(struct janas_tune_cand){.threads = opts[i], .gpu = gpu};}- int def[JANAS_TUNE_CLASSES] = {0, 0, 0};+ int def[JANAS_TUNE_CLASSES] = {0, 0, 0, 0};for (int k = 0; k < n; k++) {if (!c[k].gpu && c[k].threads == m->all_threads)def[0] = k;if (!c[k].gpu && c[k].threads == block)- def[1] = def[2] = k;+ def[1] = def[2] = def[3] = k;}/* JANAS_GPU_ALWAYS=1: the GPU variant as the default for passes overseveral tokens (tests, and with JANAS_TUNE=0 a fixed GPU setup) */+ /* with the fast prompt the GPU computes nearly all of a block, and the+ CPU's threads mostly draw the power it would use (the laptop's PL4):+ blocks start on the performance cores' threads, not on all of them -+ of 12 models the fastest blocks were on 6 or 12 threads 11 times,+ Qwen3.5-2B 318 tokens/s on 6 and 309 on 12 against 272 on 20 (10 Oct+ 2026) */+ int fast = m->gpu && !m->gpu_off && janas_gpu_fast_prompt() && !bt;+ if (fast)+ for (int k = 0; k < n; k++)+ if (!c[k].gpu && c[k].threads == m->p_threads)+ def[2] = k;const char *ga = getenv("JANAS_GPU_ALWAYS");if (ga && atoi(ga) > 0)for (int k = 0; k < n; k++)if (c[k].gpu && c[k].threads == block)- def[1] = def[2] = k;+ def[1] = def[2] = def[3] = k;if (bt && block != c[def[1]].threads && n < JANAS_TUNE_MAX_CANDS) {c[n] = (struct janas_tune_cand){.threads = block, .gpu = 0};- def[1] = def[2] = n++;+ def[1] = def[2] = def[3] = n++;}janas_tuner_init(&m->tuner, c, n, def, (tn && atoi(tn) == 0) || bt);janas_tuner_allow_gpu(&m->tuner, 1);@@ -95,10 +106,10 @@ void janas_m_init_tuner(struct janas_llm_model *m)janas_cpu_name(cpu, sizeof(cpu));janas_jns_meta_str(&m->j, "general.name", name, sizeof(name));int w =- snprintf(m->tune_key, sizeof(m->tune_key), "v2|%s|%d/%d/%d|%s%s|%s|",+ snprintf(m->tune_key, sizeof(m->tune_key), "v2|%s|%d/%d/%d|%s%s%s|%s|",cpu, m->p_cores, m->p_threads, m->all_threads,m->gpu && !m->gpu_off ? janas_gpu_name(m->gpu) : "no GPU",- one ? " 1g" : "", name);+ one ? " 1g" : "", fast ? " f" : "", name);for (int k = 0; k < n && w > 0 && (size_t)w < sizeof(m->tune_key); k++)w += snprintf(m->tune_key + w, sizeof(m->tune_key) - (size_t)w, "%d%s,",c[k].threads, c[k].gpu ? "g" : "");@@ -113,8 +124,10 @@ void janas_m_apply_candidate(struct janas_llm_model *m, int cand)if (m->own_compute)janas_pool_set_active(m->compute, m->tuner.cand[cand].threads);m->gpu_use = m->tuner.cand[cand].gpu;- if (m->gpu)+ if (m->gpu) {janas_gpu_keep_awake(m->gpu, m->gpu_use);+ janas_gpu_set_only_fast(m->gpu, !m->gpu_use);+ }}void janas_llm_model_set_eco(struct janas_llm_model *m, int eco)@@ -135,6 +148,12 @@ int janas_llm_model_threads(const struct janas_llm_model *m, int cls)return m->tuner.cand[janas_tuner_best(&m->tuner, cls)].threads;}+int janas_llm_model_gpu_prompts(const struct janas_llm_model *m)+{+ return m->gpu && !m->gpu_off && m->tuner.allow_gpu &&+ janas_gpu_fast_prompt();+}+int janas_llm_model_gpu_used(const struct janas_llm_model *m){if (!m->gpu || m->gpu_off)diff --git a/src/llm/shaders/attn_merge.comp b/src/llm/shaders/attn_merge.compindex cb3d096..40ae7d9 100644--- a/src/llm/shaders/attn_merge.comp+++ b/src/llm/shaders/attn_merge.comp@@ -7,7 +7,9 @@// e % hd). The runs of attn_decode.comp joined in their order - the// largest maximum M, each run's sums and l scaled by e^(m - M), the sums// divided by the total - then quantized to Q8_K as norm_quant.comp does-// (q256, and q32 if set): no dispatch of its own between the two.+// (q256, and q32 if set): no dispatch of its own between the two. With+// gate set (qwen3.5's attention), each output times sigmoid of its gate+// (gs floats a token) before.#extension GL_EXT_buffer_reference : require#extension GL_KHR_shader_subgroup_basic : require#extension GL_KHR_shader_subgroup_clustered : require@@ -24,8 +26,9 @@ layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer RPlayout(push_constant) uniform P {uvec2 part, q256, q32;uint hd, nch, ps; /* ps: floats from a token's runs to the next's */- uint pad_;+ uint gs; /* the gate's floats a token */uvec2 pos;+ uvec2 gate;} p;shared float m8[32]; /* the block's maxima, eight values each */@@ -68,6 +71,10 @@ void main()a += PR.v[o + i] * f;}float v = a / L;+ if (set(p.gate)) {+ float gv = R(p.gate).v[j0 * p.gs + e];+ v *= 1.0 / (1.0 + exp(-gv));+ }/* the maxima of the groups of 32 and of the block */float mc = subgroupClusteredMax(abs(v), 8u);if (t % 8u == 0u)diff --git a/src/llm/shaders/dense_mm.comp b/src/llm/shaders/dense_mm.compnew file mode 100644index 0000000..d3f41af--- /dev/null+++ b/src/llm/shaders/dense_mm.comp@@ -0,0 +1,75 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for weights left as they are, f32 (KIND 0) or bf16 (1):+// the per-layer branches of Gemma 4's E2B and E4B (gpu.c+// janas_gpu_dense_mm). Tiles of 64 rows by 64 vectors, 16 columns staged+// per barrier, each invocation 4 rows by 4 vectors. y[v * ys + r] = scale x+// the dot product of row r with vector v (vectors cols floats apart). NOT+// bit for bit with the CPU: summed in another order.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 256) in;+layout(constant_id = 0) const uint NV = 8; // unused+layout(constant_id = 1) const uint SG = 8; // unused+layout(constant_id = 2) const uint KIND = 0;++const uint BM = 64u, BN = 64u, BK = 16u;++layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer F1 { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer FO { float v[]; };++layout(push_constant) uniform P { uvec2 w, x, y; uint rows, cols, nv, ys; float scale; } p;++shared float sa[BK * BM];+shared float sb[BK * BN];++// weight k of row r+float wt(uint r, uint k)+{+ uint i = r * p.cols + k;+ if (KIND == 0u)+ return F1(p.w).v[i];+ uint u = U1(p.w).v[i >> 1];+ return uintBitsToFloat((i & 1u) != 0u ? u & 0xffff0000u : u << 16);+}++void main()+{+ F1 X = F1(p.x);+ FO Y = FO(p.y);+ uint id = gl_LocalInvocationID.x;+ uint r0 = gl_WorkGroupID.x * BM, v0 = gl_WorkGroupID.y * BN;+ uint tr = id % 16u, tc = id / 16u;+ uint ld = id / 4u, kq = (id % 4u) * 4u; // staging: a row (vector), 4 columns+ uint rl = min(r0 + ld, p.rows - 1u), vl = min(v0 + ld, p.nv - 1u);+ float acc[16];+ [[unroll]] for (uint i = 0; i < 16u; i++) acc[i] = 0.0;+ for (uint k0 = 0; k0 < p.cols; k0 += BK) {+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ sa[(kq + i) * BM + ld] = wt(rl, k0 + kq + i);+ sb[(kq + i) * BN + ld] = X.v[vl * p.cols + k0 + kq + i];+ }+ barrier();+ [[unroll]] for (uint kk = 0; kk < BK; kk++) {+ float a[4], b[4];+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ a[i] = sa[kk * BM + tr * 4u + i];+ b[i] = sb[kk * BN + tc * 4u + i];+ }+ [[unroll]] for (uint i = 0; i < 4u; i++)+ [[unroll]] for (uint j = 0; j < 4u; j++)+ acc[i * 4u + j] += a[i] * b[j];+ }+ barrier();+ }+ [[unroll]] for (uint i = 0; i < 4u; i++)+ [[unroll]] for (uint j = 0; j < 4u; j++) {+ uint row = r0 + tr * 4u + i, v = v0 + tc * 4u + j;+ if (row < p.rows && v < p.nv)+ Y.v[v * p.ys + row] = acc[i * 4u + j] * p.scale;+ }+}diff --git a/src/llm/shaders/gdn_conv.comp b/src/llm/shaders/gdn_conv.compnew file mode 100644index 0000000..f5ac916--- /dev/null+++ b/src/llm/shaders/gdn_conv.comp@@ -0,0 +1,38 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// A Gated DeltaNet layer of the whole block on the GPU (gpu_token.c), its+// first step: the causal convolution of the q|k|v channels, then SiLU, as+// model_rec.c's conv_head does - out[t][c] = SiLU(sum over i < d of+// w[i][c] x in(t - d + 1 + i, c)), in(s) the block's row s (mixed) or, for+// s < 0, row d - 1 + s of the layer's convolution state (the half the+// position buffer names: its word 1). A thread a channel, a workgroup row+// of 256 channels per token.+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 256) in;++layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer R { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer W { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer RP { uint v[]; };++// mixed: n rows of cc; out: likewise; w: d rows of cc; state: two halves of+// hs floats, this layer's (d - 1) rows of cc at so in each+layout(push_constant) uniform P { uvec2 mixed, out_, w, state, pos; uint cc, d, hs, so; } p;++void main()+{+ uint c = gl_WorkGroupID.x * 256u + gl_LocalInvocationID.x, t = gl_WorkGroupID.y;+ if (c >= p.cc)+ return;+ R M = R(p.mixed), Wt = R(p.w), S = R(p.state);+ uint base = RP(p.pos).v[1] * p.hs + p.so;+ float s = 0.0;+ for (uint i = 0; i < p.d; i++) {+ int r = int(t) - int(p.d) + 1 + int(i);+ float x = r >= 0 ? M.v[uint(r) * p.cc + c] : S.v[base + uint(int(p.d) - 1 + r) * p.cc + c];+ s = i == 0u ? x * Wt.v[c] : fma(x, Wt.v[i * p.cc + c], s);+ }+ W(p.out_).v[t * p.cc + c] = s / (1.0 + exp(-s));+}diff --git a/src/llm/shaders/gdn_cstate.comp b/src/llm/shaders/gdn_cstate.compnew file mode 100644index 0000000..a59840f--- /dev/null+++ b/src/llm/shaders/gdn_cstate.comp@@ -0,0 +1,29 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// After gdn_conv.comp: the layer's convolution state for the next call,+// the block's last d - 1 rows of mixed (n >= d - 1), into the half the+// position buffer's word 2 names. Apart from gdn_conv.comp, which may still+// read the state when the two halves are one (a later piece of a block).+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 256) in;++layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer R { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) buffer S { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer RP { uint v[]; };++layout(push_constant) uniform P { uvec2 mixed, state, pos; uint cc, d, hs, so, n; } p;++void main()+{+ uint c = gl_WorkGroupID.x * 256u + gl_LocalInvocationID.x;+ if (c >= p.cc)+ return;+ R M = R(p.mixed);+ S St = S(p.state);+ uint base = RP(p.pos).v[2] * p.hs + p.so;+ for (uint k = 0; k + 1u < p.d; k++)+ St.v[base + k * p.cc + c] = M.v[(p.n - p.d + 1u + k) * p.cc + c];+}diff --git a/src/llm/shaders/gdn_gate.comp b/src/llm/shaders/gdn_gate.compnew file mode 100644index 0000000..f879459--- /dev/null+++ b/src/llm/shaders/gdn_gate.comp@@ -0,0 +1,109 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// A Gated DeltaNet layer's last step before its output product, as+// model_rec.c's gate_worker does: each head's raw output (ds = 128 numbers)+// normalized by its RMS and the weights w (ds), times SiLU(z), then+// quantized to Q8_K as norm_quant.comp does (q256, and q32 if set). A+// workgroup per block of 256 (two heads) and token (y), a number a thread.+#extension GL_EXT_buffer_reference : require+#extension GL_KHR_shader_subgroup_arithmetic : require+#extension GL_KHR_shader_subgroup_basic : require+#extension GL_KHR_shader_subgroup_clustered : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 256) in;++layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer F { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer U { uint v[]; };++// att, z: n rows of d floats; w: ds floats+layout(push_constant) uniform P { uvec2 att, z, w, q256, q32; uint d; float eps; } p;++shared float sq_[256];+shared float hs_[2];++shared float m8[32]; /* the block's maxima, eight values each */+shared int s8a[32], s8b[32]; /* and its quants' sums, eight each */++bool set(uvec2 a) { return a.x != 0u || a.y != 0u; }++float recip_det(float d)+{+ precise float y = uintBitsToFloat(0x7EF311C3u - floatBitsToUint(d));+ for (int i = 0; i < 3; i++)+ y = fma(y, fma(-d, y, 1.0), y);+ return y;+}++int q8(float v, float id)+{+ precise float q = roundEven(v * id);+ return int(clamp(q, -127.0, 127.0));+}++void main()+{+ uint t = gl_LocalInvocationID.x, b = gl_WorkGroupID.x, j = gl_WorkGroupID.y;+ uint e = 256u * b + t, o = j * p.d + e;+ U Q = U(p.q256), Q32 = U(p.q32);+ float a = F(p.att).v[o];+ sq_[t] = a * a;+ barrier();+ if (t < 2u) { /* each head's sum of squares, in order */+ float s = 0.0;+ for (uint i = 0; i < 128u; i++)+ s += sq_[128u * t + i];+ hs_[t] = s;+ }+ barrier();+ precise float sc = 1.0 / sqrt(hs_[t / 128u] / 128.0 + p.eps);+ precise float vs = a * sc;+ float z = F(p.z).v[o];+ float v = vs * F(p.w).v[e % 128u] * (z / (1.0 + exp(-z)));+ /* the maxima of the groups of 32 and of the block */+ float mc = subgroupClusteredMax(abs(v), 8u);+ if (t % 8u == 0u)+ m8[t / 8u] = mc;+ barrier();+ uint l = t / 32u;+ float m32 = max(max(m8[4u * l], m8[4u * l + 1u]), max(m8[4u * l + 2u], m8[4u * l + 3u]));+ float m256 = 0.0;+ for (uint k = 0; k < 32u; k++)+ m256 = max(m256, m8[k]);+ precise float dd = m256 * 0.007874015718698502;+ float id = dd > 0.0 ? recip_det(dd) : 0.0;+ precise float d32 = m32 * 0.007874015718698502;+ float id32 = d32 > 0.0 ? recip_det(d32) : 0.0;+ int qa = q8(v, id), qb = q8(v, id32);+ /* four quants a word, from four neighbouring threads */+ uint wa = subgroupClusteredOr((uint(qa) & 0xffu) << (8u * (t % 4u)), 4u);+ uint wb = subgroupClusteredOr((uint(qb) & 0xffu) << (8u * (t % 4u)), 4u);+ uint base = (j * gl_NumWorkGroups.x + b) * 80u; /* 320 bytes a block */+ if (t % 4u == 0u) {+ Q.v[base + t / 4u] = wa;+ if (set(p.q32))+ Q32.v[base + t / 4u] = wb;+ }+ /* the sums of 16 quants: two of eight each */+ int ca = subgroupClusteredAdd(qa, 8u), cb = subgroupClusteredAdd(qb, 8u);+ if (t % 8u == 0u) {+ s8a[t / 8u] = ca;+ s8b[t / 8u] = cb;+ }+ barrier();+ if (t < 8u) {+ int sa0 = s8a[4u * t] + s8a[4u * t + 1u], sa1 = s8a[4u * t + 2u] + s8a[4u * t + 3u];+ Q.v[base + 64u + t] = (uint(sa0) & 0xffffu) | (uint(sa1) << 16);+ if (set(p.q32)) {+ int sb0 = s8b[4u * t] + s8b[4u * t + 1u], sb1 = s8b[4u * t + 2u] + s8b[4u * t + 3u];+ Q32.v[base + 64u + t] = (uint(sb0) & 0xffffu) | (uint(sb1) << 16);+ precise float dt =+ max(max(m8[4u * t], m8[4u * t + 1u]), max(m8[4u * t + 2u], m8[4u * t + 3u])) *+ 0.007874015718698502;+ Q32.v[base + 72u + t] = floatBitsToUint(dt);+ }+ }+ if (t == 0u)+ Q.v[base + 72u] = floatBitsToUint(dd);+}diff --git a/src/llm/shaders/gdn_l2.comp b/src/llm/shaders/gdn_l2.compnew file mode 100644index 0000000..554e6fb--- /dev/null+++ b/src/llm/shaders/gdn_l2.comp@@ -0,0 +1,35 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// After gdn_conv.comp: each q and k head (the first 2 n_kh of the+// convolution's heads of ds = 128 channels) over its L2 norm, x / sqrt(sum+// x^2 + eps), in place. A workgroup per head (x) and token (y), a channel a+// thread.+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require+#extension GL_KHR_shader_subgroup_basic : require+#extension GL_KHR_shader_subgroup_arithmetic : require++layout(local_size_x = 128) in;++layout(buffer_reference, std430, buffer_reference_align = 4) buffer F { float v[]; };++layout(push_constant) uniform P { uvec2 x; uint cc; float eps; } p;++shared float sg_part[64];++void main()+{+ uint i = gl_LocalInvocationID.x, h = gl_WorkGroupID.x, t = gl_WorkGroupID.y;+ F X = F(p.x);+ uint o = t * p.cc + h * 128u + i;+ float v = X.v[o];+ float r = subgroupAdd(v * v);+ if (subgroupElect())+ sg_part[gl_SubgroupID] = r;+ barrier();+ float ss = sg_part[0];+ for (uint k = 1u; k < gl_NumSubgroups; k++)+ ss += sg_part[k];+ X.v[o] = v * (1.0 / sqrt(ss + p.eps));+}diff --git a/src/llm/shaders/gdn_rec.comp b/src/llm/shaders/gdn_rec.compnew file mode 100644index 0000000..0570639--- /dev/null+++ b/src/llm/shaders/gdn_rec.comp@@ -0,0 +1,82 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The Gated DeltaNet recurrence of a block, as deltanet.c's gdn_step_h does+// it token after token: per value head h and row r of its state M (ds = 128+// numbers, kept at sixteen bits between tokens), with the token's q and k+// (key head g), v and the head's beta and decay -+// x = M[r] decay, sk = x . k, d = (v[r] - sk) beta, y = x + d k,+// out[r] = (y . q) scale, M[r] = y at sixteen bits.+// beta = sigmoid(b), decay = exp(softplus(a + dt[h]) A[h]) from the raw b+// and a (ba: per token, the nv b then the nv a). The row stays in registers+// for the whole block: eight threads a row, sixteen numbers each, sixteen+// rows a workgroup. The state is read from the half the position buffer's+// word 1 names and written to the one its word 2 does.+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require+#extension GL_EXT_control_flow_attributes : require+#extension GL_KHR_shader_subgroup_basic : require+#extension GL_KHR_shader_subgroup_clustered : require++layout(local_size_x = 128) in;++layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer R { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer W { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) buffer U { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer RP { uint v[]; };++// co: n rows of cc (q heads, k heads at kd, v at 2 kd); ba: n rows of 2 nv;+// att: n rows of nv ds; state: two halves of hs sixteen-bit numbers, this+// layer's nv ds ds at so; dt, A: nv floats+layout(push_constant) uniform P {+ uvec2 co, ba, att, state, pos, dt, a;+ uint cc, nv, nkh, n, hs, so, vmap_mod;+ float scale;+} p;++void main()+{+ uint t = gl_LocalInvocationID.x, lane = t % 8u, rl = t / 8u;+ uint h = gl_WorkGroupID.x / 8u, r = (gl_WorkGroupID.x % 8u) * 16u + rl;+ uint kd = p.nkh * 128u, g = p.vmap_mod != 0u ? h % p.nkh : h / (p.nv / p.nkh);+ R CO = R(p.co), BA = R(p.ba);+ W O = W(p.att);+ U S = U(p.state);+ RP PS = RP(p.pos);+ /* this thread's sixteen numbers of the row: e0 .. e0 + 15 */+ uint e0 = lane * 16u;+ uint row = p.so + (h * 128u + r) * 128u + e0; /* sixteen-bit numbers */+ uint src = (PS.v[1] * p.hs + row) / 2u, dst = (PS.v[2] * p.hs + row) / 2u;+ float x[16];+ [[unroll]] for (uint i = 0; i < 8u; i++) {+ vec2 f = unpackHalf2x16(S.v[src + i]);+ x[2u * i] = f.x;+ x[2u * i + 1u] = f.y;+ }+ float dtv = R(p.dt).v[h], av = R(p.a).v[h];+ for (uint j = 0; j < p.n; j++) {+ float b = BA.v[j * 2u * p.nv + h], al = BA.v[j * 2u * p.nv + p.nv + h] + dtv;+ float beta = 1.0 / (1.0 + exp(-b));+ float sp = al > 20.0 ? al : log(1.0 + exp(al));+ float dec = exp(sp * av);+ uint qo = j * p.cc + g * 128u + e0, ko = qo + kd;+ float sk = 0.0;+ [[unroll]] for (uint i = 0; i < 16u; i++) {+ x[i] *= dec;+ sk = fma(x[i], CO.v[ko + i], sk);+ }+ sk = subgroupClusteredAdd(sk, 8u);+ float d = (CO.v[j * p.cc + 2u * kd + h * 128u + r] - sk) * beta;+ float sq = 0.0;+ [[unroll]] for (uint i = 0; i < 16u; i++) {+ float y = fma(d, CO.v[ko + i], x[i]);+ sq = fma(y, CO.v[qo + i], sq);+ x[i] = unpackHalf2x16(packHalf2x16(vec2(y, 0.0))).x;+ }+ sq = subgroupClusteredAdd(sq, 8u);+ if (lane == 0u)+ O.v[j * p.nv * 128u + h * 128u + r] = sq * p.scale;+ }+ [[unroll]] for (uint i = 0; i < 8u; i++)+ S.v[dst + i] = packHalf2x16(vec2(x[2u * i], x[2u * i + 1u]));+}diff --git a/src/llm/shaders/norm_quant.comp b/src/llm/shaders/norm_quant.compindex 49ff604..98b72a8 100644--- a/src/llm/shaders/norm_quant.comp+++ b/src/llm/shaders/norm_quant.comp@@ -5,7 +5,8 @@// residual, if add is set) stored at xo (if set), then v = rms_norm(x) * w// (if w is set; else v = x), quantized to Q8_K in the product shaders'// layouts - q256 (qs, bsums, d: janas_q8k_quantize_det) and q32 (the eight-// blocks of 32 with a scale each, if q32 is set). One workgroup per block+// blocks of 32 with a scale each, if q32 is set), and v itself at xn (if+// set: a recurrent layer's f32 products, an MTP block's hidden states). One workgroup per block// of 256 values: each one makes the whole vector's sum of squares, the// same operations in the same order, so the same bits, then normalizes and// quantizes its own block, a value a thread. x is only read: the residual@@ -25,7 +26,7 @@ layout(local_size_x = 256) in;layout(buffer_reference, std430, buffer_reference_align = 4) buffer F { float v[]; };layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer U { uint v[]; };-layout(push_constant) uniform P { uvec2 x, add, w, xo, q256, q32; uint d; float eps; } p;+layout(push_constant) uniform P { uvec2 x, add, w, xo, q256, q32, xn; uint d; float eps; } p;/* The workgroup's sum: each subgroup's in one operation, then thesubgroups' in a fixed order. */@@ -96,6 +97,8 @@ void main()precise float vs = xv * scale;v = vs * Wt.v[e];}+ if (set(p.xn))+ F(p.xn).v[xo0 + e] = v;/* the maxima of the groups of 32 and of the block */float mc = subgroupClusteredMax(abs(v), 8u);if (t % 8u == 0u)diff --git a/src/llm/shaders/q4k_mmid.comp b/src/llm/shaders/q4k_mmid.compnew file mode 100644index 0000000..65290ba--- /dev/null+++ b/src/llm/shaders/q4k_mmid.comp@@ -0,0 +1,125 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for a mixture's experts (gpu.c experts_id): every+// expert of a group in one dispatch, as llama.cpp's mul_mat_id, the+// workgroup's z the expert. Its weights, activations and outputs come from+// a table (P.t); an expert gets some 16 of a block's tokens, so the tiles+// are 64 rows by 16 vectors, each invocation 2 rows by 4. Otherwise as+// q4k_mmq.comp: groups of 32 staged 4 at a time, scales applied in float,+// NOT bit for bit with the CPU.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 16u, KS = 4u, WB = 9u, XB = 20u;++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++// t: the table, 3 uvec4 an expert: (w, x), (y, nv, ys), (p1, p0); then+// from uvec4 tiles on, one a workgroup's z: (expert, first vector)+layout(push_constant) uniform P { uvec2 t; uint rows, nb, tiles; } p;++struct ACache { uint qs[4]; vec2 dm; };+struct BCache { int qs[8]; vec2 ds; };+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++void main()+{+ U4 T = U4(p.t);+ uvec4 tl = T.v[p.tiles + gl_WorkGroupID.z];+ uint e = tl.x;+ uvec4 t0 = T.v[3u * e], t1 = T.v[3u * e + 1u];+ uint nv = t1.z, ys = t1.w;+ uint r0 = gl_WorkGroupID.x * BM, v0 = tl.y;+ U4 W4 = U4(t0.xy), X = U4(t0.zw);+ U1 W1 = U1(t0.xy);+ F Y = F(t1.xy);+ uint id = gl_LocalInvocationID.x;+ uint rq = (id % 32u) * 2u, vq = (id / 32u) * 4u;+ float sums[8];+ [[unroll]] for (uint i = 0; i < 8u; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // groups k and k + 1 are the low and high nibbles of the same+ // words: read once for both (they were read once each)+ [[unroll]] for (uint k = 0; k < KS; k += 2u) {+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint blk = (r * p.nb + b) * WB;+ uint wq = blk * 4u + 4u + (j / 2u) * 8u + 2u * iqs;+ uint w0 = W1.v[wq], w1 = W1.v[wq + 1u];+ sa[k * BM + rl + l].qs[iqs] = (w0 & 0x0f0f0f0fu) | ((w1 & 0x0f0f0f0fu) << 4);+ sa[(k + 1u) * BM + rl + l].qs[iqs] =+ ((w0 >> 4) & 0x0f0f0f0fu) | (((w1 >> 4) & 0x0f0f0f0fu) << 4);+ if (iqs == 0u) {+ uvec4 hd = W4.v[blk];+ vec2 dd = unpackHalf2x16(hd.x);+ [[unroll]] for (uint h = 0; h < 2u; h++) {+ uint jh = j + h, jj = jh % 4u;+ int sc = jh < 4u ? int(bitfieldExtract(hd.y, int(8u * jj), 6))+ : int(bitfieldExtract(hd.w, int(8u * jj), 4) |+ (bitfieldExtract(hd.y, int(8u * jj + 6u), 2) << 4));+ int mn = jh < 4u ? int(bitfieldExtract(hd.z, int(8u * jj), 6))+ : int(bitfieldExtract(hd.w, int(8u * jj + 4u), 4) |+ (bitfieldExtract(hd.z, int(8u * jj + 6u), 2) << 4));+ sa[(k + h) * BM + rl + l].dm = vec2(dd.x * float(sc), dd.y * float(mn));+ }+ }+ }+ }+ }+ {+ uint k = id / 32u, c = (id % 32u) / 2u, iqs = id % 2u;+ uint v = min(v0 + c, nv - 1u);+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint xo = (v * p.nb + b) * XB;+ uvec4 q = X.v[xo + 2u * j + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ float d8 = uintBitsToFloat(X.v[xo + 18u].x);+ uvec4 bs = X.v[xo + 16u + j / 4u];+ uint bw = j % 4u == 0u ? bs.x : j % 4u == 1u ? bs.y : j % 4u == 2u ? bs.z : bs.w;+ int s = bitfieldExtract(int(bw), 0, 16) + bitfieldExtract(int(bw), 16, 16);+ sb[k * BN + c].ds = vec2(d8, d8 * float(s));+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[2];+ [[unroll]] for (uint r = 0; r < 2u; r++)+ ca[r] = sa[k * BM + rq + r];+ [[unroll]] for (uint c = 0; c < 4u; c++) {+ BCache cb = sb[k * BN + vq + c];+ [[unroll]] for (uint r = 0; r < 2u; r++) {+ int q = 0;+ [[unroll]] for (uint i = 0; i < 8u; i++)+ q += dotPacked4x8EXT(int((ca[r].qs[i / 2u] >> ((i % 2u) * 4u)) & 0x0f0f0f0fu),+ cb.qs[i]);+ sums[c * 2u + r] += cb.ds.x * ca[r].dm.x * float(q) - ca[r].dm.y * cb.ds.y;+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint c = 0; c < 4u; c++)+ [[unroll]] for (uint r = 0; r < 2u; r++) {+ uint row = r0 + rq + r, v = v0 + vq + c;+ if (row < p.rows && v < nv)+ Y.v[v * ys + row] = sums[c * 2u + r];+ }+}diff --git a/src/llm/shaders/q4k_mmq.comp b/src/llm/shaders/q4k_mmq.compnew file mode 100644index 0000000..0fe8ade--- /dev/null+++ b/src/llm/shaders/q4k_mmq.comp@@ -0,0 +1,136 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt (JANAS_GPU_FAST_PROMPT): Q4_K weights times a block of+// Q8_K vectors in the tiling of llama.cpp's mul_mmq for K-quants on an+// integrated Intel GPU (its "medium" warptile): 128 invocations take 64+// rows by 64 vectors, each 2 rows by 16 vectors, the weights and+// activations of 4 groups of 32 staged per barrier, each group's scale and+// minimum applied in float. NOT bit for bit with the CPU: the sums are+// rounded group by group, not exactly per block; on the development+// laptop's GPU 1.5 times the exact product's speed on a prompt's block.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 64u, BK = 32u, KS = 4u;+const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;+const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8+const uint WSUBN = WN / WNITER; // 4+const uint WB = 9u, XB = 20u; // uvec4 a block: weights, activations++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++// w: the first row; x: vector v, block b at uvec4 (v * xs + b) * 20;+// y: row r of vector v at y[v * ys + r]+layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;++struct ACache { uint qs[4]; vec2 dm; }; // a group: nibbles 2 by 2, d*sc, dmin*m+struct BCache { int qs[8]; vec2 ds; }; // a group: quants, d8, d8 * their sum+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++void main()+{+ U4 W4 = U4(p.w), X = U4(p.x);+ U1 W1 = U1(p.w);+ F Y = F(p.y);+ uint id = gl_LocalInvocationID.x;+ uint r0 = gl_WorkGroupID.x * BM, v0 = gl_WorkGroupID.y * BN;+ uint warp = id / WARP, tiw = id % WARP;+ uint tiwr = tiw % (WM / TM), tiwc = tiw / (WM / TM);+ uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);+ float sums[WNITER * TN * TM];+ [[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ // weights: 4 invocations a row, each one word of the row's group+ // (2 x 4 nibbles), for 2 rows and 4 groups+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // groups k and k + 1 are the low and high nibbles of the same+ // words: read once for both (they were read once each)+ [[unroll]] for (uint k = 0; k < KS; k += 2u) {+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint blk = (r * p.nb + b) * WB;+ uint wq = blk * 4u + 4u + (j / 2u) * 8u + 2u * iqs;+ uint w0 = W1.v[wq], w1 = W1.v[wq + 1u];+ sa[k * BM + rl + l].qs[iqs] = (w0 & 0x0f0f0f0fu) | ((w1 & 0x0f0f0f0fu) << 4);+ sa[(k + 1u) * BM + rl + l].qs[iqs] =+ ((w0 >> 4) & 0x0f0f0f0fu) | (((w1 >> 4) & 0x0f0f0f0fu) << 4);+ if (iqs == 0u) {+ uvec4 hd = W4.v[blk];+ vec2 dd = unpackHalf2x16(hd.x);+ [[unroll]] for (uint h = 0; h < 2u; h++) {+ uint jh = j + h, jj = jh % 4u;+ int sc = jh < 4u ? int(bitfieldExtract(hd.y, int(8u * jj), 6))+ : int(bitfieldExtract(hd.w, int(8u * jj), 4) |+ (bitfieldExtract(hd.y, int(8u * jj + 6u), 2) << 4));+ int mn = jh < 4u ? int(bitfieldExtract(hd.z, int(8u * jj), 6))+ : int(bitfieldExtract(hd.w, int(8u * jj + 4u), 4) |+ (bitfieldExtract(hd.z, int(8u * jj + 6u), 2) << 4));+ sa[(k + h) * BM + rl + l].dm = vec2(dd.x * float(sc), dd.y * float(mn));+ }+ }+ }+ }+ }+ // activations: 2 invocations a vector, each half a group's quants+ {+ uint iqs = id % 2u, c = id / 2u;+ uint v = min(v0 + c, p.nv - 1u);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint xo = (v * p.xs + b) * XB;+ uvec4 q = X.v[xo + 2u * j + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ float d8 = uintBitsToFloat(X.v[xo + 18u].x);+ uvec4 bs = X.v[xo + 16u + j / 4u];+ uint bw = j % 4u == 0u ? bs.x : j % 4u == 1u ? bs.y : j % 4u == 2u ? bs.z : bs.w;+ int s = bitfieldExtract(int(bw), 0, 16) + bitfieldExtract(int(bw), 16, 16);+ sb[k * BN + c].ds = vec2(d8, d8 * float(s));+ }+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[TM];+ [[unroll]] for (uint cr = 0; cr < TM; cr++)+ ca[cr] = sa[k * BM + warp_r * WM + tiwr * TM + cr];+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ BCache cb = sb[k * BN + warp_c * WN + wsic * WSUBN + tiwc * TN + cc];+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ int q = 0;+ [[unroll]] for (uint i = 0; i < 8u; i++)+ q += dotPacked4x8EXT(int((ca[cr].qs[i / 2u] >> ((i % 2u) * 4u)) & 0x0f0f0f0fu),+ cb.qs[i]);+ sums[(wsic * TN + cc) * TM + cr] +=+ cb.ds.x * ca[cr].dm.x * float(q) - ca[cr].dm.y * cb.ds.y;+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++)+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint row = r0 + warp_r * WM + tiwr * TM + cr;+ uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ if (row < p.rows && v < p.nv)+ Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ }+}diff --git a/src/llm/shaders/q5k_mmq.comp b/src/llm/shaders/q5k_mmq.compnew file mode 100644index 0000000..49166d3--- /dev/null+++ b/src/llm/shaders/q5k_mmq.comp@@ -0,0 +1,135 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for Q5_K (Qwen3.5's recurrent layers' projections): as+// q4k_mmq.comp, each weight's fifth bit (bit j of qh[l] for weight l of+// group j) joined to its nibble, the group's 32 values staged as bytes. NOT+// bit for bit with the CPU: rounded group by group.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 64u, BK = 32u, KS = 4u;+const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;+const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8+const uint WSUBN = WN / WNITER; // 4+const uint WB = 11u, XB = 20u; // uvec4 a block: weights, activations++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++// w: the first row; x: vector v, block b at uvec4 (v * xs + b) * 20;+// y: row r of vector v at y[v * ys + r]+layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;++struct ACache { uint qs[8]; vec2 dm; }; // a group: its 32 values, d*sc, dmin*m+struct BCache { int qs[8]; vec2 ds; }; // a group: quants, d8, d8 * their sum+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++void main()+{+ U4 W4 = U4(p.w), X = U4(p.x);+ U1 W1 = U1(p.w);+ F Y = F(p.y);+ uint id = gl_LocalInvocationID.x;+ uint r0 = gl_WorkGroupID.x * BM, v0 = gl_WorkGroupID.y * BN;+ uint warp = id / WARP, tiw = id % WARP;+ uint tiwr = tiw % (WM / TM), tiwc = tiw / (WM / TM);+ uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);+ float sums[WNITER * TN * TM];+ [[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ // weights: 4 invocations a row, each one word of the row's group+ // (2 x 4 nibbles), for 2 rows and 4 groups+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // groups k and k + 1 are the low and high nibbles of the same+ // words: read once for both (they were read once each)+ [[unroll]] for (uint k = 0; k < KS; k += 2u) {+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint blk = (r * p.nb + b) * WB;+ uint wq = blk * 4u + 12u + (j / 2u) * 8u + 2u * iqs;+ uint wh = blk * 4u + 4u + 2u * iqs;+ [[unroll]] for (uint u = 0; u < 2u; u++) {+ uint w = W1.v[wq + u], h = W1.v[wh + u];+ sa[k * BM + rl + l].qs[2u * iqs + u] =+ (w & 0x0f0f0f0fu) | (((h >> j) & 0x01010101u) << 4);+ sa[(k + 1u) * BM + rl + l].qs[2u * iqs + u] =+ ((w >> 4) & 0x0f0f0f0fu) | (((h >> (j + 1u)) & 0x01010101u) << 4);+ }+ if (iqs == 0u) {+ uvec4 hd = W4.v[blk];+ vec2 dd = unpackHalf2x16(hd.x);+ [[unroll]] for (uint h = 0; h < 2u; h++) {+ uint jh = j + h, jj = jh % 4u;+ int sc = jh < 4u ? int(bitfieldExtract(hd.y, int(8u * jj), 6))+ : int(bitfieldExtract(hd.w, int(8u * jj), 4) |+ (bitfieldExtract(hd.y, int(8u * jj + 6u), 2) << 4));+ int mn = jh < 4u ? int(bitfieldExtract(hd.z, int(8u * jj), 6))+ : int(bitfieldExtract(hd.w, int(8u * jj + 4u), 4) |+ (bitfieldExtract(hd.z, int(8u * jj + 6u), 2) << 4));+ sa[(k + h) * BM + rl + l].dm = vec2(dd.x * float(sc), dd.y * float(mn));+ }+ }+ }+ }+ }+ // activations: 2 invocations a vector, each half a group's quants+ {+ uint iqs = id % 2u, c = id / 2u;+ uint v = min(v0 + c, p.nv - 1u);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint xo = (v * p.xs + b) * XB;+ uvec4 q = X.v[xo + 2u * j + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ float d8 = uintBitsToFloat(X.v[xo + 18u].x);+ uvec4 bs = X.v[xo + 16u + j / 4u];+ uint bw = j % 4u == 0u ? bs.x : j % 4u == 1u ? bs.y : j % 4u == 2u ? bs.z : bs.w;+ int s = bitfieldExtract(int(bw), 0, 16) + bitfieldExtract(int(bw), 16, 16);+ sb[k * BN + c].ds = vec2(d8, d8 * float(s));+ }+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[TM];+ [[unroll]] for (uint cr = 0; cr < TM; cr++)+ ca[cr] = sa[k * BM + warp_r * WM + tiwr * TM + cr];+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ BCache cb = sb[k * BN + warp_c * WN + wsic * WSUBN + tiwc * TN + cc];+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ int q = 0;+ [[unroll]] for (uint i = 0; i < 8u; i++)+ q += dotPacked4x8EXT(int(ca[cr].qs[i]), cb.qs[i]);+ sums[(wsic * TN + cc) * TM + cr] +=+ cb.ds.x * ca[cr].dm.x * float(q) - ca[cr].dm.y * cb.ds.y;+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++)+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint row = r0 + warp_r * WM + tiwr * TM + cr;+ uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ if (row < p.rows && v < p.nv)+ Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ }+}diff --git a/src/llm/shaders/q6k_matvec.comp b/src/llm/shaders/q6k_matvec.compindex 93a177b..326ccf0 100644--- a/src/llm/shaders/q6k_matvec.comp+++ b/src/llm/shaders/q6k_matvec.comp@@ -7,7 +7,9 @@// as the CPU kernels. Activations shared across the subgroup as in// q4k_matvec.comp. A Q6_K block is 210 bytes (ql 128, qh 64, scales 16,// d 2), so blocks start 2 bytes off a 4-byte boundary every other time:-// 32-bit words are read from the boundary below and realigned.+// a block is read as the 14 aligned uvec4 that cover it and realigned in+// registers. Read a 32-bit word at a time, it took 53 loads a block and+// ran at a quarter of Q4_K's speed (9 Oct 2026).#extension GL_EXT_integer_dot_product : require#extension GL_KHR_shader_subgroup_basic : require#extension GL_KHR_shader_subgroup_ballot : require@@ -30,7 +32,6 @@ uvec4 bc(uvec4 xr[NX], uint k) { return subgroupBroadcast(xr[k / SG], k % SG); }void main(){- U1 W = U1(p.w);U4 X = U4(p.x);F Y = F(p.y);uint lane = gl_SubgroupInvocationID;@@ -44,15 +45,29 @@ void main()acc[v] = 0.0;uint off = row * p.nb * 210u; /* bytes */for (uint b = 0; b < p.nb; b++, off += 210u) {- /* the block's 52 words (208 bytes) and d, realigned */- uint base = off >> 2, sh = (off & 3u) * 8u;+ /* the block's 52 words (208 bytes) and d, realigned: the 16-byte+ boundary below it, s bytes before (even: 210 and the rows' start+ are), and 14 uvec4 from there */+ uint lo = p.w.x + off;+ uvec2 at = uvec2(lo & ~15u, p.w.y + (lo < p.w.x ? 1u : 0u));+ uint s = lo & 15u;+ U4 B = U4(at);+ uint rw[57];+ for (uint k = 0; k < 14u; k++) {+ uvec4 q = B.v[k];+ rw[4u * k] = q.x;+ rw[4u * k + 1u] = q.y;+ rw[4u * k + 2u] = q.z;+ rw[4u * k + 3u] = q.w;+ }+ rw[56] = 0u;+ uint w0 = s >> 2;uint wd[53];for (uint k = 0; k < 53u; k++)- wd[k] = W.v[base + k];- if (sh != 0u)- for (uint k = 0; k < 52u; k++)- wd[k] = (wd[k] >> 16) | (wd[k + 1u] << 16);- float d = unpackHalf2x16(sh != 0u ? wd[52] >> 16 : wd[52] & 0xffffu).x;+ wd[k] = (s & 2u) != 0u+ ? (rw[w0 + k] >> 16) | (rw[w0 + k + 1u] << 16)+ : rw[w0 + k];+ float d = unpackHalf2x16(wd[52] & 0xffffu).x;int sc[16];for (uint k = 0; k < 4u; k++)for (uint j = 0; j < 4u; j++)diff --git a/src/llm/shaders/q6k_mmid.comp b/src/llm/shaders/q6k_mmid.compnew file mode 100644index 0000000..d55e9d1--- /dev/null+++ b/src/llm/shaders/q6k_mmid.comp@@ -0,0 +1,130 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for a mixture's experts whose down matrices are plain+// Q6_K (Qwen3-Coder-30B-A3B's): as q6kp_mmid.comp, the weights staged as+// q6k_mmq.comp does (the table's third uvec4 unused). NOT bit for bit with+// the CPU.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 16u, KS = 4u, XB = 20u;++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++// t: the table, 3 uvec4 an expert: (w, x), (y, nv, ys), (p1, p0); then+// from uvec4 tiles on, one a workgroup's z: (expert, first vector)+layout(push_constant) uniform P { uvec2 t; uint rows, nb, tiles; } p;++struct ACache { uint qs[8]; vec2 dsc; };+struct BCache { int qs[8]; vec2 d8s; float d8; };+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++// word i (0-52) of the Q6_K block at byte off from w, realigned+uint hword(U1 W, uint off, uint i)+{+ uint a = (off >> 2) + i;+ return (off & 2u) != 0u ? (W.v[a] >> 16) | (W.v[a + 1u] << 16) : W.v[a];+}++void main()+{+ U4 T = U4(p.t);+ uvec4 tl = T.v[p.tiles + gl_WorkGroupID.z];+ uint e = tl.x;+ uvec4 t0 = T.v[3u * e], t1 = T.v[3u * e + 1u];+ uint nv = t1.z, ys = t1.w;+ uint r0 = gl_WorkGroupID.x * BM, v0 = tl.y;+ U4 X = U4(t0.zw);+ U1 W = U1(t0.xy);+ F Y = F(t1.xy);+ uint id = gl_LocalInvocationID.x;+ uint rq = (id % 32u) * 2u, vq = (id / 32u) * 4u;+ float sums[8];+ [[unroll]] for (uint i = 0; i < 8u; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // the 4 groups staged share their high bits' word and two+ // words of low nibbles: each read once+ uint b = g0 / 8u, n = (g0 % 8u) / 4u;+ uint off = (r * p.nb + b) * 210u;+ [[unroll]] for (uint u = 0; u < 2u; u++) {+ uint kw = 2u * iqs + u;+ uint l0 = hword(W, off, 16u * n + kw), l1 = hword(W, off, 16u * n + 8u + kw);+ uint h = hword(W, off, 32u + 8u * n + kw);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint lw = (k & 1u) != 0u ? l1 : l0;+ sa[k * BM + rl + l].qs[kw] = ((lw >> ((k >> 1) * 4u)) & 0x0f0f0f0fu) |+ (((h >> (2u * k)) & 0x03030303u) << 4);+ }+ }+ if (iqs == 0u) {+ float d = unpackHalf2x16(hword(W, off, 52u) & 0xffffu).x;+ uint sw0 = hword(W, off, 48u + 2u * n), sw1 = hword(W, off, 49u + 2u * n);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint sw = k < 2u ? sw0 : sw1, sb0 = (2u * k) % 4u;+ int s0 = bitfieldExtract(int(sw), int(8u * sb0), 8);+ int s1 = bitfieldExtract(int(sw), int(8u * sb0 + 8u), 8);+ sa[k * BM + rl + l].dsc = vec2(d * float(s0), d * float(s1));+ }+ }+ }+ }+ {+ uint k = id / 32u, c = (id % 32u) / 2u, iqs = id % 2u;+ uint v = min(v0 + c, nv - 1u);+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint xo = (v * p.nb + b) * XB;+ uvec4 q = X.v[xo + 2u * j + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ uvec4 bs = X.v[xo + 16u + j / 4u];+ uint bw = j % 4u == 0u ? bs.x : j % 4u == 1u ? bs.y : j % 4u == 2u ? bs.z : bs.w;+ sb[k * BN + c].d8s = vec2(32.0 * float(bitfieldExtract(int(bw), 0, 16)),+ 32.0 * float(bitfieldExtract(int(bw), 16, 16)));+ sb[k * BN + c].d8 = uintBitsToFloat(X.v[xo + 18u].x);+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[2];+ [[unroll]] for (uint r = 0; r < 2u; r++)+ ca[r] = sa[k * BM + rq + r];+ [[unroll]] for (uint c = 0; c < 4u; c++) {+ BCache cb = sb[k * BN + vq + c];+ [[unroll]] for (uint r = 0; r < 2u; r++) {+ int q0 = 0, q1 = 0;+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ q0 += dotPacked4x8EXT(int(ca[r].qs[i]), cb.qs[i]);+ q1 += dotPacked4x8EXT(int(ca[r].qs[i + 4u]), cb.qs[i + 4u]);+ }+ sums[c * 2u + r] += cb.d8 * (ca[r].dsc.x * (float(q0) - cb.d8s.x) ++ ca[r].dsc.y * (float(q1) - cb.d8s.y));+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint c = 0; c < 4u; c++)+ [[unroll]] for (uint r = 0; r < 2u; r++) {+ uint row = r0 + rq + r, v = v0 + vq + c;+ if (row < p.rows && v < nv)+ Y.v[v * ys + row] = sums[c * 2u + r];+ }+}diff --git a/src/llm/shaders/q6k_mmq.comp b/src/llm/shaders/q6k_mmq.compnew file mode 100644index 0000000..eaf2d63--- /dev/null+++ b/src/llm/shaders/q6k_mmq.comp@@ -0,0 +1,140 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for Q6_K: the tiling of q4k_mmq.comp. A group of 32+// weights is two groups of 16 with a scale each; a weight's six bits are+// four of ql and two of qh, put together as q6k_matvec.comp does, from the+// 210-byte blocks read a word at a time and realigned; the -32 of Q6_K+// comes off with the activations' sums of 16. NOT bit for bit with the+// CPU: rounded group by group.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 64u, KS = 4u;+const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;+const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8+const uint WSUBN = WN / WNITER; // 4+const uint XB = 20u;++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;++struct ACache { uint qs[8]; vec2 dsc; }; // a group: six-bit values, d * scale of each 16+struct BCache { int qs[8]; vec2 d8s; float d8; }; // quants, 32 x the sums of 16, d8+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++// word i (0-52) of the block at byte off from w, realigned+uint hword(U1 W, uint off, uint i)+{+ uint a = (off >> 2) + i;+ return (off & 2u) != 0u ? (W.v[a] >> 16) | (W.v[a + 1u] << 16) : W.v[a];+}++void main()+{+ U4 X = U4(p.x);+ U1 W = U1(p.w);+ F Y = F(p.y);+ uint id = gl_LocalInvocationID.x;+ uint r0 = gl_WorkGroupID.x * BM, v0 = gl_WorkGroupID.y * BN;+ uint warp = id / WARP, tiw = id % WARP;+ uint tiwr = tiw % (WM / TM), tiwc = tiw / (WM / TM);+ uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);+ float sums[WNITER * TN * TM];+ [[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ // weights: 4 invocations a row, 2 words each of the group's 8+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // the 4 groups staged share their high bits' word and two+ // words of low nibbles: each read once (they were read once+ // per group, realigned: twice)+ uint b = g0 / 8u, n = (g0 % 8u) / 4u;+ uint off = (r * p.nb + b) * 210u;+ [[unroll]] for (uint t = 0; t < 2u; t++) {+ uint kw = 2u * iqs + t;+ uint l0 = hword(W, off, 16u * n + kw), l1 = hword(W, off, 16u * n + 8u + kw);+ uint h = hword(W, off, 32u + 8u * n + kw);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint lw = (k & 1u) != 0u ? l1 : l0;+ sa[k * BM + rl + l].qs[kw] = ((lw >> ((k >> 1) * 4u)) & 0x0f0f0f0fu) |+ (((h >> (2u * k)) & 0x03030303u) << 4);+ }+ }+ if (iqs == 0u) {+ float d = unpackHalf2x16(hword(W, off, 52u) & 0xffffu).x;+ uint sw0 = hword(W, off, 48u + 2u * n), sw1 = hword(W, off, 49u + 2u * n);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint sw = k < 2u ? sw0 : sw1, sb0 = (2u * k) % 4u;+ int s0 = bitfieldExtract(int(sw), int(8u * sb0), 8);+ int s1 = bitfieldExtract(int(sw), int(8u * sb0 + 8u), 8);+ sa[k * BM + rl + l].dsc = vec2(d * float(s0), d * float(s1));+ }+ }+ }+ }+ // activations: 2 invocations a vector, each half a group's quants+ {+ uint iqs = id % 2u, c = id / 2u;+ uint v = min(v0 + c, p.nv - 1u);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint g = g0 + k, b = g / 8u, G = g % 8u;+ uint xo = (v * p.xs + b) * XB;+ uvec4 q = X.v[xo + 2u * G + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ uvec4 bs = X.v[xo + 16u + G / 4u];+ uint bw = G % 4u == 0u ? bs.x : G % 4u == 1u ? bs.y : G % 4u == 2u ? bs.z : bs.w;+ sb[k * BN + c].d8s = vec2(32.0 * float(bitfieldExtract(int(bw), 0, 16)),+ 32.0 * float(bitfieldExtract(int(bw), 16, 16)));+ sb[k * BN + c].d8 = uintBitsToFloat(X.v[xo + 18u].x);+ }+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[TM];+ [[unroll]] for (uint cr = 0; cr < TM; cr++)+ ca[cr] = sa[k * BM + warp_r * WM + tiwr * TM + cr];+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ BCache cb = sb[k * BN + warp_c * WN + wsic * WSUBN + tiwc * TN + cc];+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ int q0 = 0, q1 = 0;+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ q0 += dotPacked4x8EXT(int(ca[cr].qs[i]), cb.qs[i]);+ q1 += dotPacked4x8EXT(int(ca[cr].qs[i + 4u]), cb.qs[i + 4u]);+ }+ sums[(wsic * TN + cc) * TM + cr] +=+ cb.d8 * (ca[cr].dsc.x * (float(q0) - cb.d8s.x) ++ ca[cr].dsc.y * (float(q1) - cb.d8s.y));+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++)+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint row = r0 + warp_r * WM + tiwr * TM + cr;+ uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ if (row < p.rows && v < p.nv)+ Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ }+}diff --git a/src/llm/shaders/q6kp_matvec.comp b/src/llm/shaders/q6kp_matvec.compindex dcac1bd..04722b2 100644--- a/src/llm/shaders/q6kp_matvec.comp+++ b/src/llm/shaders/q6kp_matvec.comp@@ -10,8 +10,9 @@// sum as sumi * 2^shk - c * off, the same integers and float operations as// the CPU: the results agree bit for bit. Activations shared across the// subgroup as in q4k_matvec.comp. The blocks are 82 bytes, so every other-// one starts 2 bytes off a 4-byte boundary: words are read from the-// boundary below and realigned, as in q6k_matvec.comp.+// one starts 2 bytes off a 4-byte boundary: a block is read as the 6+// aligned uvec4 that cover it and realigned, as in q6k_matvec.comp, and the+// bare planes (64 bytes a block, from a 4096-byte boundary) as 4 uvec4 each.#extension GL_EXT_integer_dot_product : require#extension GL_KHR_shader_subgroup_basic : require#extension GL_KHR_shader_subgroup_ballot : require@@ -36,7 +37,7 @@ uvec4 bc(uvec4 xr[NX], uint k) { return subgroupBroadcast(xr[k / SG], k % SG); }void main(){- U1 W = U1(p.w), P1 = U1(p.p1), P0 = U1(p.p0);+ U4 P1 = U4(p.p1), P0 = U4(p.p0);U4 X = U4(p.x);F Y = F(p.y);uint lane = gl_SubgroupInvocationID;@@ -51,25 +52,39 @@ void main()uint shk = p.level == 3u ? 1u : p.level == 2u ? 3u : 5u;int c = p.level == 3u ? 64 : p.level == 2u ? 61 : 49;uint off = row * p.nb * 82u; /* bytes */- uint pw = row * p.nb * 16u; /* words of the bare planes */- for (uint b = 0; b < p.nb; b++, off += 82u, pw += 16u) {- /* the block's 20 words (80 bytes) and d, realigned */- uint base = off >> 2, sh = (off & 3u) * 8u;+ uint pw = row * p.nb * 4u; /* uvec4 of the bare planes */+ for (uint b = 0; b < p.nb; b++, off += 82u, pw += 4u) {+ /* the block's 20 words (80 bytes) and d, realigned from the 16-byte+ boundary s bytes below it (s even) */+ uint lo = p.w.x + off;+ U4 B = U4(uvec2(lo & ~15u, p.w.y + (lo < p.w.x ? 1u : 0u)));+ uint s = lo & 15u;+ uint rw[25];+ for (uint k = 0; k < 6u; k++) {+ uvec4 q = B.v[k];+ rw[4u * k] = q.x;+ rw[4u * k + 1u] = q.y;+ rw[4u * k + 2u] = q.z;+ rw[4u * k + 3u] = q.w;+ }+ rw[24] = 0u;+ uint w0s = s >> 2;uint wd[21];for (uint k = 0; k < 21u; k++)- wd[k] = W.v[base + k];- if (sh != 0u)- for (uint k = 0; k < 20u; k++)- wd[k] = (wd[k] >> 16) | (wd[k + 1u] << 16);- float d = unpackHalf2x16(sh != 0u ? wd[20] >> 16 : wd[20] & 0xffffu).x;+ wd[k] = (s & 2u) != 0u+ ? (rw[w0s + k] >> 16) | (rw[w0s + k + 1u] << 16)+ : rw[w0s + k];+ float d = unpackHalf2x16(wd[20] & 0xffffu).x;int sc[16];for (uint k = 0; k < 4u; k++)for (uint j = 0; j < 4u; j++)sc[4u * k + j] = bitfieldExtract(int(wd[16u + k]), int(8u * j), 8);uint w1[16], w0[16];- for (uint k = 0; k < 16u; k++) {- w1[k] = p.level >= 2u ? P1.v[pw + k] : 0u;- w0[k] = p.level >= 3u ? P0.v[pw + k] : 0u;+ for (uint k = 0; k < 4u; k++) {+ uvec4 a1 = p.level >= 2u ? P1.v[pw + k] : uvec4(0u);+ uvec4 a0 = p.level >= 3u ? P0.v[pw + k] : uvec4(0u);+ w1[4u * k] = a1.x; w1[4u * k + 1u] = a1.y; w1[4u * k + 2u] = a1.z; w1[4u * k + 3u] = a1.w;+ w0[4u * k] = a0.x; w0[4u * k + 1u] = a0.y; w0[4u * k + 2u] = a0.z; w0[4u * k + 3u] = a0.w;}for (uint v = 0; v < NV; v++) {uvec4 xr[NX];diff --git a/src/llm/shaders/q6kp_mmid.comp b/src/llm/shaders/q6kp_mmid.compnew file mode 100644index 0000000..6df64ed--- /dev/null+++ b/src/llm/shaders/q6kp_mmid.comp@@ -0,0 +1,133 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for a mixture's experts whose down matrices are Q6_K in+// planes at level 3: as q4k_mmid.comp, the weights staged as q6kp_mmq.comp+// does (the table's third uvec4 the two bare planes). NOT bit for bit with+// the CPU.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 16u, KS = 4u, XB = 20u;++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++// t: the table, 3 uvec4 an expert: (w, x), (y, nv, ys), (p1, p0); then+// from uvec4 tiles on, one a workgroup's z: (expert, first vector)+layout(push_constant) uniform P { uvec2 t; uint rows, nb, tiles; } p;++struct ACache { uint qs[8]; vec2 dsc; };+struct BCache { int qs[8]; vec2 d8s; float d8; };+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++// word i (0-20) of the high plane block at byte off from w, realigned+uint hword(U1 W, uint off, uint i)+{+ uint a = (off >> 2) + i;+ return (off & 2u) != 0u ? (W.v[a] >> 16) | (W.v[a + 1u] << 16) : W.v[a];+}++void main()+{+ U4 T = U4(p.t);+ uvec4 tl = T.v[p.tiles + gl_WorkGroupID.z];+ uint e = tl.x;+ uvec4 t0 = T.v[3u * e], t1 = T.v[3u * e + 1u];+ uint nv = t1.z, ys = t1.w;+ uint r0 = gl_WorkGroupID.x * BM, v0 = tl.y;+ uvec4 t2 = T.v[3u * e + 2u];+ U4 X = U4(t0.zw);+ U1 W = U1(t0.xy), P1 = U1(t2.xy), P0 = U1(t2.zw);+ F Y = F(t1.xy);+ uint id = gl_LocalInvocationID.x;+ uint rq = (id % 32u) * 2u, vq = (id / 32u) * 4u;+ float sums[8];+ [[unroll]] for (uint i = 0; i < 8u; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // the 4 groups staged are those of one byte's 4 pairs of+ // bits: each word read once for all of them (it was read+ // once per group, the high plane's twice: realigned)+ uint b = g0 / 8u, n = (g0 % 8u) / 4u;+ uint off = (r * p.nb + b) * 82u;+ uint pw = (r * p.nb + b) * 16u;+ [[unroll]] for (uint u = 0; u < 2u; u++) {+ uint i = 8u * n + 2u * iqs + u;+ uint h = hword(W, off, i), a1 = P1.v[pw + i], a0 = P0.v[pw + i];+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint hsh = 2u * k;+ sa[k * BM + rl + l].qs[2u * iqs + u] =+ ((h >> hsh) & 0x03030303u) << 4 | ((a1 >> hsh) & 0x03030303u) << 2 |+ ((a0 >> hsh) & 0x03030303u);+ }+ }+ if (iqs == 0u) {+ float d = unpackHalf2x16(hword(W, off, 20u) & 0xffffu).x;+ uint sw0 = hword(W, off, 16u + 2u * n), sw1 = hword(W, off, 17u + 2u * n);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint sw = k < 2u ? sw0 : sw1, sb0 = (2u * k) % 4u;+ int s0 = bitfieldExtract(int(sw), int(8u * sb0), 8);+ int s1 = bitfieldExtract(int(sw), int(8u * sb0 + 8u), 8);+ sa[k * BM + rl + l].dsc = vec2(d * float(s0), d * float(s1));+ }+ }+ }+ }+ {+ uint k = id / 32u, c = (id % 32u) / 2u, iqs = id % 2u;+ uint v = min(v0 + c, nv - 1u);+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint xo = (v * p.nb + b) * XB;+ uvec4 q = X.v[xo + 2u * j + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ uvec4 bs = X.v[xo + 16u + j / 4u];+ uint bw = j % 4u == 0u ? bs.x : j % 4u == 1u ? bs.y : j % 4u == 2u ? bs.z : bs.w;+ sb[k * BN + c].d8s = vec2(32.0 * float(bitfieldExtract(int(bw), 0, 16)),+ 32.0 * float(bitfieldExtract(int(bw), 16, 16)));+ sb[k * BN + c].d8 = uintBitsToFloat(X.v[xo + 18u].x);+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[2];+ [[unroll]] for (uint r = 0; r < 2u; r++)+ ca[r] = sa[k * BM + rq + r];+ [[unroll]] for (uint c = 0; c < 4u; c++) {+ BCache cb = sb[k * BN + vq + c];+ [[unroll]] for (uint r = 0; r < 2u; r++) {+ int q0 = 0, q1 = 0;+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ q0 += dotPacked4x8EXT(int(ca[r].qs[i]), cb.qs[i]);+ q1 += dotPacked4x8EXT(int(ca[r].qs[i + 4u]), cb.qs[i + 4u]);+ }+ sums[c * 2u + r] += cb.d8 * (ca[r].dsc.x * (float(q0) - cb.d8s.x) ++ ca[r].dsc.y * (float(q1) - cb.d8s.y));+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint c = 0; c < 4u; c++)+ [[unroll]] for (uint r = 0; r < 2u; r++) {+ uint row = r0 + rq + r, v = v0 + vq + c;+ if (row < p.rows && v < nv)+ Y.v[v * ys + row] = sums[c * 2u + r];+ }+}diff --git a/src/llm/shaders/q6kp_mmq.comp b/src/llm/shaders/q6kp_mmq.compnew file mode 100644index 0000000..a7594ba--- /dev/null+++ b/src/llm/shaders/q6kp_mmq.comp@@ -0,0 +1,142 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for Q6_K in bit planes at level 3 (all three planes, the+// Q6_K weights themselves): the tiling of q4k_mmq.comp. A group of 32+// weights is two groups of 16 with a scale each; a weight's six bits are+// the high plane's two (in the row's 82-byte blocks, with the scales and+// d), then the two bare planes' (64 bytes a block each), and the -32 of+// Q6_K comes off with the activations' sums of 16. NOT bit for bit with+// the CPU: rounded group by group.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 64u, KS = 4u;+const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;+const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8+const uint WSUBN = WN / WNITER; // 4+const uint XB = 20u;++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys, level; uvec2 p1, p0; } p;++struct ACache { uint qs[8]; vec2 dsc; }; // a group: six-bit values, d * scale of each 16+struct BCache { int qs[8]; vec2 d8s; float d8; }; // quants, 32 x the sums of 16, d8+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++// word i (0-19) of the high plane block at byte off from w, realigned+uint hword(U1 W, uint off, uint i)+{+ uint a = (off >> 2) + i;+ return (off & 2u) != 0u ? (W.v[a] >> 16) | (W.v[a + 1u] << 16) : W.v[a];+}++void main()+{+ U4 X = U4(p.x);+ U1 W = U1(p.w), P1 = U1(p.p1), P0 = U1(p.p0);+ F Y = F(p.y);+ uint id = gl_LocalInvocationID.x;+ uint r0 = gl_WorkGroupID.x * BM, v0 = gl_WorkGroupID.y * BN;+ uint warp = id / WARP, tiw = id % WARP;+ uint tiwr = tiw % (WM / TM), tiwc = tiw / (WM / TM);+ uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);+ float sums[WNITER * TN * TM];+ [[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ // weights: 4 invocations a row, 2 words each of the group's 8+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ // the 4 groups staged are those of one byte's 4 pairs of+ // bits: each word read once for all of them (it was read+ // once per group, the high plane's twice: realigned)+ uint b = g0 / 8u, n = (g0 % 8u) / 4u;+ uint off = (r * p.nb + b) * 82u;+ uint pw = (r * p.nb + b) * 16u;+ [[unroll]] for (uint t = 0; t < 2u; t++) {+ uint i = 8u * n + 2u * iqs + t;+ uint h = hword(W, off, i), a1 = P1.v[pw + i], a0 = P0.v[pw + i];+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint hsh = 2u * k;+ sa[k * BM + rl + l].qs[2u * iqs + t] =+ ((h >> hsh) & 0x03030303u) << 4 | ((a1 >> hsh) & 0x03030303u) << 2 |+ ((a0 >> hsh) & 0x03030303u);+ }+ }+ if (iqs == 0u) {+ float d = unpackHalf2x16(hword(W, off, 20u) & 0xffffu).x;+ uint sw0 = hword(W, off, 16u + 2u * n), sw1 = hword(W, off, 17u + 2u * n);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint sw = k < 2u ? sw0 : sw1, sb0 = (2u * k) % 4u;+ int s0 = bitfieldExtract(int(sw), int(8u * sb0), 8);+ int s1 = bitfieldExtract(int(sw), int(8u * sb0 + 8u), 8);+ sa[k * BM + rl + l].dsc = vec2(d * float(s0), d * float(s1));+ }+ }+ }+ }+ // activations: 2 invocations a vector, each half a group's quants+ {+ uint iqs = id % 2u, c = id / 2u;+ uint v = min(v0 + c, p.nv - 1u);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint g = g0 + k, b = g / 8u, G = g % 8u;+ uint xo = (v * p.xs + b) * XB;+ uvec4 q = X.v[xo + 2u * G + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ uvec4 bs = X.v[xo + 16u + G / 4u];+ uint bw = G % 4u == 0u ? bs.x : G % 4u == 1u ? bs.y : G % 4u == 2u ? bs.z : bs.w;+ sb[k * BN + c].d8s = vec2(32.0 * float(bitfieldExtract(int(bw), 0, 16)),+ 32.0 * float(bitfieldExtract(int(bw), 16, 16)));+ sb[k * BN + c].d8 = uintBitsToFloat(X.v[xo + 18u].x);+ }+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[TM];+ [[unroll]] for (uint cr = 0; cr < TM; cr++)+ ca[cr] = sa[k * BM + warp_r * WM + tiwr * TM + cr];+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ BCache cb = sb[k * BN + warp_c * WN + wsic * WSUBN + tiwc * TN + cc];+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ int q0 = 0, q1 = 0;+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ q0 += dotPacked4x8EXT(int(ca[cr].qs[i]), cb.qs[i]);+ q1 += dotPacked4x8EXT(int(ca[cr].qs[i + 4u]), cb.qs[i + 4u]);+ }+ sums[(wsic * TN + cc) * TM + cr] +=+ cb.d8 * (ca[cr].dsc.x * (float(q0) - cb.d8s.x) ++ ca[cr].dsc.y * (float(q1) - cb.d8s.y));+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++)+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint row = r0 + warp_r * WM + tiwr * TM + cr;+ uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ if (row < p.rows && v < p.nv)+ Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ }+}diff --git a/src/llm/shaders/q80_mmq.comp b/src/llm/shaders/q80_mmq.compnew file mode 100644index 0000000..68af353--- /dev/null+++ b/src/llm/shaders/q80_mmq.comp@@ -0,0 +1,118 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// The fast prompt for Q8_0: the tiling of q4k_mmq.comp (64 rows by 64+// vectors, each invocation 2 rows by 16, groups of 32 staged 4 at a time),+// a group's weights the block's 32 bytes, its scale the block's d, the+// activations' the block of 32's own (convert_x's layout for blocks of 32:+// qs32, bsums32, d32). A block is 34 bytes, so its words are read+// realigned. NOT bit for bit with the CPU: rounded group by group.+#extension GL_EXT_control_flow_attributes : require+#extension GL_EXT_integer_dot_product : require+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 128) in;+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants+layout(constant_id = 1) const uint SG = 8;++const uint BM = 64u, BN = 64u, BK = 32u, KS = 4u;+const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;+const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8+const uint WSUBN = WN / WNITER; // 4+const uint XB = 20u; // uvec4 an activations' block++layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };++// w: the first row (rows of nb * 8 blocks of 34 bytes); x: vector v, block+// b at uvec4 (v * xs + b) * 20; y: row r of vector v at y[v * ys + r]+layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;++struct ACache { int qs[8]; float d; }; // a group: its 32 weights, d+struct BCache { int qs[8]; float d; }; // a group: quants, its d32+shared ACache sa[BM * KS];+shared BCache sb[BN * KS];++// word i of the 32-bit words from byte off of w, off even+uint hword(U1 W, uint off, uint i)+{+ uint a = (off >> 2) + i;+ return (off & 2u) != 0u ? (W.v[a] >> 16) | (W.v[a + 1u] << 16) : W.v[a];+}++void main()+{+ U4 X = U4(p.x);+ U1 W1 = U1(p.w);+ F Y = F(p.y);+ uint id = gl_LocalInvocationID.x;+ uint r0 = gl_WorkGroupID.x * BM, v0 = gl_WorkGroupID.y * BN;+ uint warp = id / WARP, tiw = id % WARP;+ uint tiwr = tiw % (WM / TM), tiwc = tiw / (WM / TM);+ uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);+ float sums[WNITER * TN * TM];+ [[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ uint ngroups = p.nb * 8u;+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {+ // weights: 4 invocations a row, 2 words each of the group's 8+ {+ uint iqs = id % 4u, rl = id / 4u;+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {+ uint r = min(r0 + rl + l, p.rows - 1u);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint off = (r * ngroups + g0 + k) * 34u;+ sa[k * BM + rl + l].qs[2u * iqs] = int(hword(W1, off + 2u, 2u * iqs));+ sa[k * BM + rl + l].qs[2u * iqs + 1u] = int(hword(W1, off + 2u, 2u * iqs + 1u));+ if (iqs == 0u)+ sa[k * BM + rl + l].d = unpackHalf2x16(hword(W1, off, 0u) & 0xffffu).x;+ }+ }+ }+ // activations: 2 invocations a vector, each half a group's quants+ {+ uint iqs = id % 2u, c = id / 2u;+ uint v = min(v0 + c, p.nv - 1u);+ [[unroll]] for (uint k = 0; k < KS; k++) {+ uint g = g0 + k, b = g / 8u, j = g % 8u;+ uint xo = (v * p.xs + b) * XB;+ uvec4 q = X.v[xo + 2u * j + iqs];+ sb[k * BN + c].qs[4u * iqs] = int(q.x);+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);+ if (iqs == 0u) {+ uvec4 dq = X.v[xo + 18u + j / 4u];+ uint dw = j % 4u == 0u ? dq.x : j % 4u == 1u ? dq.y : j % 4u == 2u ? dq.z : dq.w;+ sb[k * BN + c].d = uintBitsToFloat(dw);+ }+ }+ }+ barrier();+ for (uint k = 0; k < KS; k++) {+ ACache ca[TM];+ [[unroll]] for (uint cr = 0; cr < TM; cr++)+ ca[cr] = sa[k * BM + warp_r * WM + tiwr * TM + cr];+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ BCache cb = sb[k * BN + warp_c * WN + wsic * WSUBN + tiwc * TN + cc];+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ int q = 0;+ [[unroll]] for (uint i = 0; i < 8u; i++)+ q += dotPacked4x8EXT(ca[cr].qs[i], cb.qs[i]);+ sums[(wsic * TN + cc) * TM + cr] += ca[cr].d * cb.d * float(q);+ }+ }+ }+ barrier();+ }+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++)+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint row = r0 + warp_r * WM + tiwr * TM + cr;+ uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ if (row < p.rows && v < p.nv)+ Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ }+}diff --git a/src/llm/tuner.c b/src/llm/tuner.cindex e84d1d3..7157b24 100644--- a/src/llm/tuner.c+++ b/src/llm/tuner.c@@ -17,10 +17,14 @@#define TRIES 3 /* measurements of each candidate before choosing */#define RETRY_EVERY 128 /* passes between tries of an alternative, at first */#define RETRY_MAX 1024 /* at most, after tries that changed nothing */+#define RESETTLE_WINS 3 /* tries in a row won by other threads: theirs */int janas_tune_class(uint32_t n_tokens){- return n_tokens <= 1 ? 0 : (n_tokens <= 16 ? 1 : 2);+ return n_tokens <= 1 ? 0+ : n_tokens <= 16 ? 1+ : n_tokens >= JANAS_TUNE_BLOCK ? 2+ : 3;}void janas_tuner_init(struct janas_tuner *t, const struct janas_tune_cand *c,@@ -49,6 +53,15 @@ static int usable(const struct janas_tuner *t, int cls, int i)(!t->cls[cls].threads || t->cand[i].threads == t->cls[cls].threads);}+/* A candidate a try may run: as usable, but any threads - once they are+ settled, other threads are tried too, each try a fresh pass against the+ choice's recent mean, so that a machine that changes (warm, throttled)+ can move them */+static int retryable(const struct janas_tuner *t, int cls, int i)+{+ return !t->cand[i].gpu || (t->allow_gpu && (cls > 0 || t->gpu_one));+}+/* The default of a class: its own, or once the threads are settled thecandidate with those threads and no GPU */static int class_def(const struct janas_tuner *t, int cls)@@ -91,10 +104,14 @@ void janas_tuner_allow_gpu_one(struct janas_tuner *t, int allow)int janas_tuner_best(const struct janas_tuner *t, int cls){/* the default unless an alternative is clearly faster: measurements- are noisy, a near tie is not worth a change */+ are noisy, a near tie is not worth a change. The default's mean+ counts from its first measurement: waiting for all its tries let any+ alternative tried three times win, even a slower one - Qwen3.5-2B's+ blocks went to 20 threads (3.68 ms a token) over the default's 12+ (3.22, measured twice) (10 Oct 2026) */int def = class_def(t, cls), best = def;double bar =- t->cls[cls].tried[def] >= TRIES ? t->cls[cls].cost[def] * MARGIN : 1e30;+ t->cls[cls].tried[def] > 0 ? t->cls[cls].cost[def] * MARGIN : 1e30;for (int i = 0; i < t->n_cand; i++)if (i != def && usable(t, cls, i) && t->cls[cls].tried[i] >= TRIES &&t->cls[cls].cost[i] < bar) {@@ -143,7 +160,7 @@ int janas_tuner_pick(struct janas_tuner *t, int cls)if (p >= t->cls[cls].retry_at) {int old = -1;for (int i = 0; i < t->n_cand; i++)- if (i != best && usable(t, cls, i) &&+ if (i != best && retryable(t, cls, i) &&(old < 0 || t->cls[cls].last[i] < t->cls[cls].last[old]))old = i;t->cls[cls].retry_at = p + t->cls[cls].retry_gap;@@ -164,11 +181,33 @@ void janas_tuner_record(struct janas_tuner *t, int cls, int cand,/* a try that beats the choice clearly: tries often again, until theaverages settle which is better (one pass alone may be luck) */int rb = t->cls[cls].retry_best;- if (rb >= 0 && cand == t->cls[cls].retry_cand &&- sec_per_token < t->cls[cls].cost[rb] * MARGIN) {- t->cls[cls].retry_gap = RETRY_EVERY;- t->cls[cls].retry_at = t->cls[cls].passes + RETRY_EVERY;- t->cls[cls].retry_best = -1;+ if (rb >= 0 && cand == t->cls[cls].retry_cand) {+ if (sec_per_token < t->cls[cls].cost[rb] * MARGIN) {+ t->cls[cls].retry_gap = RETRY_EVERY;+ t->cls[cls].retry_at = t->cls[cls].passes + RETRY_EVERY;+ t->cls[cls].retry_best = -1;+ t->cls[cls].wins[cand]++;+ } else {+ t->cls[cls].wins[cand] = 0;+ }+ /*+ * Other threads, faster in try after try: theirs from now on. The+ * threads are settled by the first tries, and a laptop that warms+ * up turns the comparison round - Qwen3.5-2B settled on 20 threads+ * for its tokens, which then took 28.1 ms against 18.9 on 12 (10+ * Oct 2026). Their mean starts again from this pass: the old one+ * was made at another time.+ */+ int th = t->cls[cls].threads;+ if (th && t->cand[cand].threads != th &&+ t->cls[cls].wins[cand] >= RESETTLE_WINS) {+ t->cls[cls].threads = t->cand[cand].threads;+ t->cls[cls].wins[cand] = 0;+ t->cls[cls].cost[cand] = sec_per_token;+ t->cls[cls].tried[cand] = TRIES;+ t->cls[cls].last[cand] = t->cls[cls].passes;+ return;+ }}double *c = &t->cls[cls].cost[cand];uint32_t n = t->cls[cls].tried[cand];@@ -288,13 +327,16 @@ void janas_tuner_save(const struct janas_tuner *t, const char *key)void janas_tuner_report(const struct janas_tuner *t, char *buf, size_t len){- static const char *name[] = {"one token", "2-16 tokens", "blocks"};+ static const char *name[] = {"one token", "2-16 tokens", "blocks",+ "17-63 tokens"};+ static const int order[JANAS_TUNE_CLASSES] = {0, 1, 3, 2}; /* by size */size_t w = 0;buf[0] = 0;- for (int k = 0; k < JANAS_TUNE_CLASSES && w < len; k++) {+ for (int o = 0; o < JANAS_TUNE_CLASSES && w < len; o++) {+ int k = order[o];int b = janas_tuner_best(t, k);w += (size_t)snprintf(buf + w, len - w, "%s%s: %d threads%s",- k ? "; " : "", name[k], t->cand[b].threads,+ o ? "; " : "", name[k], t->cand[b].threads,t->cand[b].gpu ? " + GPU" : "");if (t->cls[k].tried[b] >= TRIES && w < len)w += (size_t)snprintf(buf + w, len - w, " (%.1f ms/token)",diff --git a/src/llm/tuner.h b/src/llm/tuner.hindex 6f0303f..d87af62 100644--- a/src/llm/tuner.h+++ b/src/llm/tuner.h@@ -8,12 +8,13 @@* runs anyway: for each kind of pass (one token, a few, a block) it tries* each candidate a few times, keeps the fastest, and now and then tries the* others again to follow the machine's state (heat, power source). The- * number of threads is settled once, by the first tries, and then kept:- * a pass costs more as the context grows, so a candidate tried again at- * long context against costs measured at short context loses unfairly -- * which moved chats onto fewer cores than they could use. Only whether the- * GPU takes part keeps being tried. The choices are saved per machine and- * model, so the next start begins from them.+ * number of threads is settled by the first tries, and moves only when+ * other threads win several tries in a row, each a fresh pass against the+ * choice's recent mean: a pass costs more as the context grows, so a+ * candidate's old mean, made at short context, must not decide - which+ * moved chats onto fewer cores than they could use - but a laptop that+ * warms up can turn the first tries' verdict round. The choices are saved+ * per machine and model, so the next start begins from them.*/#ifndef JANAS_LLM_TUNER_H#define JANAS_LLM_TUNER_H@@ -21,7 +22,13 @@#include <stddef.h>#include <stdint.h>-#define JANAS_TUNE_CLASSES 3 /* one token; 2-16 (verification); more */+/* one token; 2-16 (verification); blocks of JANAS_TUNE_BLOCK and more;+ 17 to JANAS_TUNE_BLOCK - 1 (a chat's short turns). The short blocks+ apart since the fast prompt (from JANAS_GPU_FAST_N = 64 tokens) puts+ the long ones nearly all on the GPU: together, their mean chose 20+ threads for Qwen3.5-2B's prompts, which ran faster on 12 (10 Oct 2026) */+#define JANAS_TUNE_CLASSES 4+#define JANAS_TUNE_BLOCK 64#define JANAS_TUNE_MAX_CANDS 8struct janas_tune_cand {@@ -40,6 +47,9 @@ struct janas_tuner {double cost[JANAS_TUNE_MAX_CANDS]; /* seconds per token, averaged */uint32_t tried[JANAS_TUNE_MAX_CANDS];uint32_t last[JANAS_TUNE_MAX_CANDS]; /* pass of the last try */+ /* tries in a row each candidate won against the choice (one with+ other threads takes the threads over at RESETTLE_WINS) */+ uint8_t wins[JANAS_TUNE_MAX_CANDS];uint32_t passes;int def; /* the default candidate *//* the next try of an alternative, the passes until the one after,diff --git a/tests/gpu_check.c b/tests/gpu_check.cindex 7f69c89..7a79b10 100644--- a/tests/gpu_check.c+++ b/tests/gpu_check.c@@ -31,6 +31,8 @@ static uint32_t rnd(void)int main(void){+ /* the exact path: the fast prompt (on by default) rounds otherwise */+ setenv("JANAS_GPU_FAST_PROMPT", "0", 1);/* GPU_CHECK_BIG_GB=n: n GiB of touched memory, like the engine'sexpert cache */const char *bg = getenv("GPU_CHECK_BIG_GB");@@ -46,7 +48,7 @@ int main(void)printf("gpu_check: no GPU (%s), skipped\n", err);return 0;}- enum { MAXR = 4096, MAXC = 4096, MAXV = 17 };+ enum { MAXR = 4096, MAXC = 4096, MAXV = 64 };size_t wbytes = (size_t)MAXR * (MAXC / JANAS_QK) * 144;wbytes = (wbytes + 4095) / 4096 * 4096;uint8_t *w = aligned_alloc(4096, wbytes);@@ -196,7 +198,7 @@ int main(void)const size_t shapes[][2] = {{4096, 2048}, {1000, 256}, {128, 4096},{2048, 512}, {4096, 4096}, {777, 1024},{1500, 768}}; /* odd blocks per row */- const size_t nvs[] = {1, 2, 3, 5, 8, 9, 16, 17};+ const size_t nvs[] = {1, 2, 3, 5, 8, 9, 16, 17, 33, 64};int bad = 0, runs = 0;/* 0 Q4_K, 1 Q6_K, 2-4 Q6_K_P with three, two and one planes, 5 Q5_K,6 Q8_0, 7-10 Q4_0, IQ4_NL, Q4_1, Q5_1, 11 IQ4_XS,diff --git a/tests/llm_eval.c b/tests/llm_eval.cindex 4bfafb1..437b340 100644--- a/tests/llm_eval.c+++ b/tests/llm_eval.c@@ -156,8 +156,8 @@ int main(int argc, char **argv)janas_llm_model_decode(m, tokens[p], (uint32_t)p, seq + p * nv);double pb0[JANAS_PH_COUNT], pb1[JANAS_PH_COUNT];janas_llm_model_phases(m, pb0);- double tb = now(), maxd = 0, sumd = 0;- size_t same = 0, top = 0;+ double tb = now(), maxd = 0, sumd = 0, rsum = 0;+ size_t same = 0, top = 0, rtop = 0;for (size_t p0 = 0; p0 < nn; p0 += (size_t)block) {uint32_t b =(uint32_t)(nn - p0 < (size_t)block ? nn - p0 : (size_t)block);@@ -178,6 +178,15 @@ int main(int argc, char **argv)}same += eq;top += argmax(a, nv) == argmax(c, nv);+ /* and against the reference, as the main loop compares */+ if (ref &&+ fseek(ref, (long)((p0 + j) * nv * sizeof(float)),+ SEEK_SET) == 0 &&+ fread(want, sizeof(float), nv, ref) == nv) {+ for (uint32_t i = 0; i < nv; i++)+ rsum += fabs((double)c[i] - want[i]);+ rtop += argmax(c, nv) == argmax(want, nv);+ }}}double tbs = now() - tb;@@ -190,6 +199,10 @@ int main(int argc, char **argv)printf("blocks of %d: most likely token as token by token at %zu/%zu, ""mean |diff| %.4f\n",block, top, nn, sumd / ((double)nn * nv));+ if (ref)+ printf("blocks of %d against the reference: top-1 %zu/%zu, mean "+ "|diff| %.4f\n",+ block, rtop, nn, rsum / ((double)nn * nv));free(seq);free(blk);}diff --git a/tests/test_expert_cache.c b/tests/test_expert_cache.cindex 146b54a..fd15289 100644--- a/tests/test_expert_cache.c+++ b/tests/test_expert_cache.c@@ -262,8 +262,8 @@ static void check_resident(struct janas_jns *j, struct janas_pool *io)CHECK(janas_expert_cache_finish(c) == 0, "finish failed");const uint8_t *a = janas_expert_cache_resident(c, &bytes, &loads);CHECK(a != NULL, "not resident with every expert loaded");- CHECK(bytes == (uint64_t)ALL * SLOT, "resident bytes %llu",- (unsigned long long)bytes);+ CHECK(bytes == (uint64_t)ALL * SLOT + JANAS_ARENA_GUARD,+ "resident bytes %llu", (unsigned long long)bytes);CHECK(loads == ALL && janas_expert_cache_loads(c) == ALL,"loads %llu, expected %d", (unsigned long long)loads, ALL);/* every expert is in the arena, where fetch says it is */diff --git a/tests/test_layout.c b/tests/test_layout.cindex 8058696..4059a0a 100644--- a/tests/test_layout.c+++ b/tests/test_layout.c@@ -8,6 +8,8 @@* model) and the file kept, in a directory of its own under ~/tmp.* The sources belong to the program, so they are compiled in here.*/+#define _GNU_SOURCE /* nftw */+#include <ftw.h>#include <stdio.h>#include <stdlib.h>#include <string.h>@@ -16,6 +18,15 @@#include "chat/glossary.c"#include "chat/layout.c"+static int rm_one(const char *path, const struct stat *st, int flag,+ struct FTW *ftw)+{+ (void)st;+ (void)flag;+ (void)ftw;+ return remove(path);+}+static int failures;#define CHECK(c, ...) \@@ -159,16 +170,15 @@ int main(void)coded_free(&c);}- /* tidy up: the one file and the two directories made here */+ /* tidy up: the layout the test kept is there, then the whole+ directory goes (the failed translation's mark too: left behind, it+ kept one directory per run in ~/tmp) */char path[1200];snprintf(path, sizeof path, "%s/janas/layouts/test.it.%016llx.txt", dir,(unsigned long long)glossary_mark("it", "test", mark(EN)));- CHECK(unlink(path) == 0, "no file at %s", path);- snprintf(path, sizeof path, "%s/janas/layouts", dir);- rmdir(path);- snprintf(path, sizeof path, "%s/janas", dir);- rmdir(path);- rmdir(dir);+ CHECK(access(path, F_OK) == 0, "no file at %s", path);+ nftw(dir, rm_one, 16, FTW_DEPTH | FTW_PHYS);+ CHECK(access(dir, F_OK) != 0, "%s left behind", dir);if (failures) {printf("test_layout: %d failures\n", failures);diff --git a/tests/test_tuner.c b/tests/test_tuner.cindex 0fcb105..cd71290 100644--- a/tests/test_tuner.c+++ b/tests/test_tuner.c@@ -165,21 +165,36 @@ int main(void)/* the threads are settled by the first tries and kept: a pass costingmore as the context grows must not hand the choice to a candidatewhose cost was measured at short context (a chat drifting onto- fewer cores). 20 threads win the first tries; then every pass of- theirs costs more, the others are never measured again */+ fewer cores). 20 threads win the first tries; then every pass, of+ any candidate, costs more: the other threads, tried now and then+ with fresh passes, never win */janas_tuner_init(&t, cand, 4, def, 0);janas_tuner_allow_gpu(&t, 0);run(&t, (const double[]){1.0, 1.2, 0.8, 0.9}, 40);CHECK(janas_tuner_best(&t, 2) == 2, "settled best %d, want 2",janas_tuner_best(&t, 2));+ int others = 0;for (int i = 0; i < 5000; i++) {int c = janas_tuner_pick(&t, 2);- CHECK(t.cand[c].threads == 20, "pass on %d threads after settling",- t.cand[c].threads);+ others += t.cand[c].threads != 20;janas_tuner_record(&t, 2, c, 0.8 + i * 0.001);}CHECK(janas_tuner_best(&t, 2) == 2, "drifted: best %d, want 2",janas_tuner_best(&t, 2));+ CHECK(others > 0 && others < 50, "%d passes on other threads in 5000",+ others);++ /* the machine changes after the threads are settled (a laptop that+ warms up): 12 threads become clearly faster than the 20 chosen, and+ tries won in a row hand the threads over */+ janas_tuner_init(&t, cand, 4, def, 0);+ janas_tuner_allow_gpu(&t, 0);+ run(&t, (const double[]){1.0, 1.2, 0.8, 0.9}, 40);+ CHECK(janas_tuner_best(&t, 2) == 2, "settled best %d, want 2",+ janas_tuner_best(&t, 2));+ run(&t, (const double[]){1.0, 0.6, 0.9, 0.9}, 5000);+ CHECK(janas_tuner_best(&t, 2) == 1, "warmed up: best %d, want 1",+ janas_tuner_best(&t, 2));/* forced: always that candidate */janas_tuner_force(&v, 1);for (int i = 0; i < 10; i++)