Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-08: models by name with measured fast profiles and a measure on the machine; the output head at four bits on request; faster kernels and attention

camaurib04ce09d1b60
diff --git a/ChangeLog.md b/ChangeLog.md
index 43ee506..918b485 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,63 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-10-08] - Every model's fast profile, measured
+
+- **Every lever that trades a little accuracy for speed was tried on every model of the catalog, alone and in every combination**: `--head 4`, `--attention fast`, and `--bits 4` where the experts are in planes. A lever is in a model's `fast` profile when it adds 2% or more: Qwen3-0.6B `--head 4 --attention fast` (+14.3%), Qwen3-4B the same (+6.8%), Qwen3.5-0.8B `--attention fast` (+2.0%), Qwen3.5-2B and 9B `--head 4` (+10.2%, +5.1%), Qwen3-30B-A3B all three (+10.0%), Qwen3.6-35B-A3B, Qwen3-Next-80B-A3B and Qwen3-Coder-Next `--head 4 --bits 4` (+6.5%, +8.6%, +10.5%). The agreement with llama.cpp of each is in the README. Gemma 4 has none: sixteen-bit keys, heads not six-bit, experts not in planes. Oracles of llama.cpp for the four Gemma 4 models were made for it (micro-batches of 8).
+- Measured on the way: for a model larger than its cache, a benchmark that generates its own text cannot compare levers, since each lever changes the greedy text and with it the experts read from disk (Qwen3.6-35B-A3B: 92.5-96% of the experts from memory, each lever slower alone and all three 7.9% faster); those three models were measured on a fixed text instead.
+- **The measure on this machine uses three replies of different kinds**, not one: on Gemma 4 E2B a single reply drafted at 38.6 tokens/s in one run and 46.9 in the next.
+- **A model of the catalog takes its prediction file from the catalog.** One merely found beside it may belong to another model of the same architecture: Qwen3-Coder-Next took Qwen3-Next-80B-A3B's MTP block and drafted at 25.9 tokens/s where the conversation's drafts gave 28.8.
+
+## [2026-10-08] - The chat measures a model on the machine it runs on
+
+- **The first time a model starts in a terminal, janas-chat offers to measure it**: the model opened each way that can differ on this machine - drafts from its prediction file or draft, from the conversation, or none; in the `fast` profile also the head as in the file - one reply to warm it and the same reply measured at temperature 0. The fastest way, if it beats the starting one by 3% or more, is kept in `~/.cache/janas/tune-<name>-<profile>.conf`, a layer between the catalog and the user's files, used only for levers the user did not set; a "no" is kept as well. `--tune` measures again, `--no-tune` leaves it out. Development laptop: Qwen3-4B 33.1 tokens/s with its draft against 29.2-29.5 otherwise, Qwen3.5-2B 77.4 with its MTP block against 41.9 without drafts and 60.6 from the conversation.
+- **A model freed gives its caller back the CPUs it had.** Opening a model pins the calling thread to one CPU; a second model opened afterwards in the same process saw that one CPU and ran on a single thread ("up to 1 compute threads"): the measure found it, opening the model three times. The caller's CPUs are now kept at load and restored when the model is freed.
+- Without drafting (`--no-spec`), the chat no longer loads a prediction file or draft it would not use.
+
+## [2026-10-08] - The chat takes a model by name
+
+- **`janas-chat qwen3-30b-a3b`**: a name of the catalog instead of a path, looked for in `~/.local/share/janas/models` (or `JANAS_MODELS`, or the new `--models <dir>`); a model not there yet is downloaded and converted through `janas-get`, with its prediction file or draft, once the user says yes. With no model, the chat lists the catalog, marks what is there, and asks which one. The first download proposes the folder and takes another typed one, saying how the chat will find it again. A path works as before, and a file of the catalog given by path gets its profile too.
+- **Two profiles per model, and settings in layers.** `--profile quality` (the default: the answers the file gives) or `--profile fast` (the catalog's measured levers on top, today `--head 4` on six models). Each layer over the one before: the catalog's profile, `chat.conf`, the model's own `~/.config/janas/models/<name>.conf`, the command line. `/save-config`, `/load-config` and `/del-config` work on the model's file, or with `global` on `chat.conf`; nothing writes the catalog. A draft named by the catalog is taken by itself (Qwen3-4B: "drafts from qwen3-0.6b").
+
+## [2026-10-08] - One catalog of models, and a home for them
+
+- **The catalog of models moved from `janas-get` to the code every program shares** (`src/common/catalog.c`), so that the chat can use it too. Each model may now name a **draft** (Qwen3-4B: Qwen3-0.6B, new in the catalog and in MODELS.md, from unsloth's repository since the Qwen team's no longer has a Q4_K_M) and a **"fast" profile** of chat options measured to make it faster at a small cost to its answers (`--head 4` on Qwen3-0.6B, Qwen3-4B, Qwen3.5-2B, 9B, Qwen3-30B-A3B and Qwen3-Next-80B-A3B; not on Qwen3.5-0.8B, which with it answered "3" to "2+3" in a chat). `janas-get qwen3-4b` brings its draft with it, unless `--no-mtp`.
+- **The models now go to `~/.local/share/janas/models`** (`$XDG_DATA_HOME/janas/models`), or to `JANAS_MODELS`, instead of a `models` folder under whatever directory `janas-get` was started from; `--dir` still chooses any other.
+
+## [2026-10-08] - The output head at four bits, on request
+
+- **`janas-chat --head 4` (and `head_bits` in `janas_llm_params`, `JANAS_HEAD_BITS=4`) requantizes a six-bit output head to four bits when the model opens.** The head is the largest matrix read whole for every token - a seventh of Qwen3.5-9B's - and most files keep it in Q6_K. 128 tokens after a prompt of 1,900: Qwen3.5-9B 13.35-13.36 tokens/s where 12.76-12.77 (+4.6%), Qwen3-Next-80B-A3B with a 16 GiB cache 25.0-25.3 where 24.4-24.5 (+2.8%); prompts unchanged. The answers move a little away from the original weights: against llama.cpp on 400 tokens, the most likely token agreed 372 times where 383 on Qwen3.5-9B (Italian prose), 372 where 382 on Qwen3-Next-80B-A3B, 387 where 392 on Qwen3-30B-A3B (C source). The default is unchanged: the head as the file has it.
+- On the small Qwen, whose head is their token embeddings, the table is requantized for both uses: three runs each with nothing else running: Qwen3-0.6B 107.0-107.7 tokens/s where 100.6-101.9 (+6%), Qwen3.5-0.8B 113.5-114.7 where 104.2-105.8 (+9%), Qwen3.5-2B 54.4-54.9 where 50.0-50.4 (+9%), Qwen3-4B 24.0-24.1 where 23.2-23.3 (+3.4%); the answers move more than on the larger models (Qwen3.5-2B, most likely token 352 where 376 against llama.cpp).
+- Only the head: requantizing every six-bit matrix (attention, shared experts too) cost more agreement (361 where 383 on Qwen3.5-9B) for next to no more speed there. Opening costs a second or two more (Qwen3.5-9B: 2.5 s where 0.7), with a new `janas_q4k_quantize_fast` that takes each sub-block's range as it is instead of searching 49 shrunken ones: nine times faster, and on this head the mean logit difference 0.157 where 0.154. `include/janas/llm.bi` has the new field, checked against the C layout with fbc.
+
+## [2026-10-07] - Attention merges its pieces in the same round
+
+- **The thread that finishes the last chunk of a group of query heads merges that group at once**, instead of the pool starting a third round for the merge after everybody has finished. The other threads go on with their chunks meanwhile, and a token's attention costs one wait for all the threads less. Only where a group's merge is small - decoding and the check of a few drafts, or blocks at a short context; a block of 64 tokens past about 8,000 positions keeps the separate round, spread over all the threads, which there was faster. Each output is computed exactly as before, so the results are the same to the bit (`test_attention`'s checksum unchanged with the merge always in the round, never, and as chosen). `tests/bench_attn`, a block of 64 tokens at 8,000 positions: 26.0-26.3 ms per layer where 28.0-28.5 (Qwen3-30B-A3B's geometry), 26.1-26.7 where 27.1-28.0 (Qwen3-Next's). Qwen3-30B-A3B in memory, 128 tokens after a prompt of 1,900: attention 6.58-6.62 ms a token where 6.90-6.92, 27.4-27.6 tokens/s where 27.3-27.4. `JANAS_ATTN_FUSE=0` keeps the separate round.
+
+## [2026-10-07] - The chat redraws only what changed
+
+- **While a reply streams in, `janas-chat` sends the terminal only the rows that changed.** It redrew the whole conversation window for every piece of text, 8 KB a token on a 160 x 40 terminal; now each row is laid out into the bytes it is drawn with, a row the terminal already shows is not sent again, and when the text wraps the terminal moves the rows up itself (a newline on the last row of the scrolling region) and only the new ones are drawn. A clear of the screen, a change of its size and every redraw other than a reply's piece still draw everything. In a pseudo-terminal, Qwen3-0.6B answering the same question: 2.0 KB a token where 8.1, and 102 tokens/s where 80 (three rounds each, 96-110 against 78-83), the chat's own drawing no longer holding the model up. Checked by feeding the window 3,000 pieces of text at 50 x 24, 80 x 40 and 34 x 20 and rebuilding both screens with a terminal emulator (pyte): the same characters and colours after every piece.
+- Found on the way: a row could come out one column too wide - the space it was broken after left at its end - and the terminal, wrapping it, moved the whole window up one row until the next redraw. A row is now drawn no further than the window's edge.
+
+## [2026-10-07] - A row costs one call
+
+- **The products over one vector loop over their rows beside the kernels.** Each row of a matrix-vector product went through `janas_dot`, which chose the kernel by the weights' type, then a wrapper and a function pointer, and the caller worked out the task's sizes again for every row; on the short rows of a mixture of experts (2,048 columns, eight blocks) that was 6% of a token's cycles in a profile. Now the sizes are worked out once for each run of rows a thread claims, the type is chosen once, and each row is one direct call to the same kernel as before - the logits of Qwen3-30B-A3B are the same to the bit. Qwen3-30B-A3B in memory, 128 tokens after a prompt of 1,900: 29.4-29.6 tokens/s where 28.4 (+3.8%), the experts' phase 15.9 ms a token where 16.9, prompts unchanged.
+
+## [2026-10-07] - The fast attention weighs the values with integers too
+
+- **On its sixteen-bit path, attention now weighs the values with sixteen-bit integers as well as computing the scores so.** A chunk's weights go to 0-32767 with a scale of their own, two positions a word, and the products with the eight-bit values are summed exactly in integers, sixteen to an instruction where a float multiply-add does eight. The path is the one the engine takes past 16,384 tokens of context or on request (`JANAS_ATTN=int16`); the exact path, the default below that, is unchanged. `tests/bench_attn`, Qwen3-30B-A3B's geometry on 12 threads, a token's attention per layer: 2,048 positions 0.096 ms where 0.107, 8,000 0.325 where 0.354, 32,000 1.34 where 1.42 (6-11%).
+- The weights' rounding adds a little error: against a double-precision reference on random scores, the worst output is now 2-6e-4 off where it was below 1e-4, still under the 4e-3 the eight-bit keys and values carry, and `test_attention`'s threshold for this path is 1e-3. On the model it does not show: Qwen3-30B-A3B against llama.cpp, 400 tokens of C, mean logit difference 0.2865 (0.2893 with float weights, 0.2871 exact), top token 390/400 (388, 392). The scalar, AVX2 and AVX-VNNI versions give the same bits, and blocks the same bits as token by token.
+
+## [2026-10-07] - Attention's scores two heads at a time
+
+- **The scores of attention are computed for two query heads at a time, the keys four at a time**: one key read serves two products, and the eight running sums, two queries and a key fit in the processor's sixteen vector registers. Each sum adds the same products in the same order, so the scores are the same to the bit (`test_attention`'s checksum unchanged on AVX2 and AVX-VNNI). `tests/bench_attn`, Qwen3-30B-A3B's geometry on 12 threads: a token's attention 4-5% faster per layer at 512, 2,048 and 8,000 positions, a block of 64 tokens 5-8%. In the engine, Qwen3-30B-A3B in memory after a prompt of 1,900 tokens: attention 6.28-6.31 ms a token where 6.50-6.53, the prompt read at 70.9-71.0 tokens/s where 70.0-70.1; the tokens written move within the noise of the rest.
+- Measured on the way: with the exact arithmetic, attention is bound by the multiply-adds, not by the memory - at 8,000 positions a layer reads its keys and values at 20 GB/s, and twelve threads go four times as fast as one, about as many as the six performance cores they run on.
+
+## [2026-10-07] - Weights asked for ahead of the product
+
+- **Every kernel asks for its weights 2 KB ahead**, one prefetch per cache line, when it multiplies one or two vectors. The hardware prefetcher alone kept too few reads in flight for the types that compute more per byte: on 12 threads of the development laptop Q4_K read 69 GB/s where Q8_0, which asks for its bytes faster, read 84. Now, measured by `tests/bench_kernels` on one vector, Q4_K reads 83 GB/s, Q5_K 80 (68), Q6_K 85 (73), Q8_0 88 (84), Q4_1 74 (67), Q4_0 and IQ4_NL 70 (65), IQ4_XS 64 (59); on two vectors, the check of one draft, Q6_K 70 (56) and Q8_0 78 (69). Q3_K and IQ3_S, bound by their arithmetic, are unchanged, and so are products of three vectors or more, where the prefetches would only cost instructions.
+- Inside the engine the gain is smaller, because its matrices were already read at 68-74 GB/s, nearer the memory's speed than one huge matrix in the benchmark: the output head, a single large Q6_K matrix, takes 11% less; Qwen3.6-35B-A3B writes 25.1-25.6 tokens/s where it wrote 24.3-24.5 (+3.4%), Qwen3-30B-A3B 30.9-31.0 where 30.6-30.9 (+0.8%), prompts unchanged. Results the same to the bit (`test_quant` on AVX2 and AVX-VNNI).
+
## [2026-10-07] - Small gains, kept: blocks of tokens and long contexts
- **The activations' sums of 32 are made once.** The Q4_K and Q5_K products over several vectors - a prompt read in blocks, the drafts checked - weigh the sums of each 32 activations by the block's mins, and added them up again for every row of every matrix; the activation's quantization now leaves them beside the sums of 16 (16 bytes more a block of 256). 5-6% on those products.
diff --git a/MODELS.md b/MODELS.md
index 62d9178..5df4380 100644
--- a/MODELS.md
+++ b/MODELS.md
@@ -79,6 +79,13 @@ fadc3e5f8d42bf7e894a785b05082e47daee4df26680389817e2093056f088ad Qwen3-Coder-30
bee7d25509e911a195cbf05a157e69bb05a62d2241cca2c4fdcfef269dcb2bdd qwen3-coder-30b-a3b-q4km.jns (gguf2jns)
```
+**Qwen3-0.6B**, from [`unsloth/Qwen3-0.6B-GGUF`](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) (the Qwen team's own repository no longer has a Q4_K_M), dense, and the model that guesses Qwen3-4B's tokens (`--draft`). Converted in one step; two conversions give the same bytes.
+
+```
+ac2d97712095a558e31573f62f466a3f9d93990898b0ec79d7c974c1780d524a Qwen3-0.6B-Q4_K_M.gguf
+9174da760919b16b9f3a4ae8738c2d53605412f40bb2c0fe68d60174bf8a4fb5 qwen3-0.6b-q4km.jns (gguf2jns)
+```
+
**Qwen3-4B**, from [`Qwen/Qwen3-4B-GGUF`](https://huggingface.co/Qwen/Qwen3-4B-GGUF). This one is dense: it has no experts to route, so its feed-forward becomes the single slot of each layer, and half its layers keep a `down` matrix in Q4_K, which has no planes to cut.
```
diff --git a/README.md b/README.md
index cac371b..13dc8d3 100644
--- a/README.md
+++ b/README.md
@@ -114,12 +114,12 @@ sudo pacman -S vulkan-headers glslang vulkan-tools # the same, Arch
## Getting a model
-**The short way: `janas-get`.** One command downloads a model from Hugging Face, checks the SHA-256 of every file against [MODELS.md](MODELS.md), converts it and makes its prediction file beside it (an MTP block, or Gemma 4's assistant), which the chat then finds by itself:
+**The short way: `janas-get`.** One command downloads a model from Hugging Face, checks the SHA-256 of every file against [MODELS.md](MODELS.md), converts it and makes its prediction file beside it (an MTP block, Gemma 4's assistant, or for Qwen3-4B the small Qwen3-0.6B that guesses its tokens), which the chat then finds by itself. The models go to `~/.local/share/janas/models`, or wherever `JANAS_MODELS` or `--dir` says:
```sh
bin/x86_64-linux/janas-get list # the models it knows
-bin/x86_64-linux/janas-get gemma-4-e4b # into models/
-bin/x86_64-linux/janas-chat models/gemma-4-e4b-it-q4km.jns
+bin/x86_64-linux/janas-get gemma-4-e4b # into ~/.local/share/janas/models
+bin/x86_64-linux/janas-chat ~/.local/share/janas/models/gemma-4-e4b-it-q4km.jns
```
A download stopped halfway resumes where it stopped, a file already there is checked rather than fetched again, and the GGUF goes once converted (`--keep` keeps it). Any other GGUF: `janas-get hf:<owner>/<repo>/<file.gguf>`, checked against the SHA-256 Hugging Face lists for it. The memory column of `janas-get list` is the smallest machine each model is meant for. The rest of this section is the same by hand.
@@ -228,11 +228,36 @@ The first start reads the model's resident weights (a couple of GB) and then fil
## Using the chat
```
-janas-chat <model.jns> [options]
+janas-chat [model] [options]
```
+The model is a `.jns` file, or the name of a model of the catalog (`janas-get list` shows them): `janas-chat qwen3-30b-a3b` looks for it in `~/.local/share/janas/models` (or `JANAS_MODELS`, or `--models`), and if it is not there yet offers to download and convert it, with its prediction file or draft, through `janas-get`. With no model at all the chat lists the catalog, marks what is already there, and asks which one. The first download proposes the folder and takes another if you type one; janas-chat then finds the models there only when told (`--models` or `JANAS_MODELS`).
+
+The settings come in layers, each over the one before: the model's profile in the catalog, the measure of the model on this machine, the chat's own file (`~/.config/janas/chat.conf`), the model's file (`~/.config/janas/models/<name>.conf`), the command line. `/save-config` writes the model's file (`/save-config global` the chat's), `/del-config` deletes it, and the catalog itself is never written. A model of the catalog has two profiles: `quality`, the default, where the answers are the ones the file gives; and `fast` (`--profile fast`), which adds the levers measured to make that model faster at a small, measured cost to its answers (see below). A model whose catalog entry names a draft gets it by itself: Qwen3-4B guesses with Qwen3-0.6B.
+
+**The fast profiles, as measured.** Every lever that trades a little accuracy for speed — `--head 4`, `--attention fast`, `--bits 4` where the experts are in planes — was tried on every model of the catalog, alone and in every combination, on 8 October 2026 on the development laptop; a lever is in a model's profile when it adds 2% or more. Speed: 128 tokens after a prompt of 1,900 (`tests/bench_long`, two runs each); for the three models larger than their 16 GiB cache, a fixed text read token by token instead (`tests/llm_eval`), because each lever changes the text a greedy reply takes and with it the experts read from disk. Agreement: the most likely token against llama.cpp, Italian prose / C source, as in the file → with the profile.
+
+| Model | Fast profile | Faster by | Agreement |
+|---|---|---|---|
+| Qwen3-0.6B | `--head 4 --attention fast` | 14.3% | 345 → 331 / 377 → 368 of 400 |
+| Qwen3-4B | `--head 4 --attention fast` | 6.8% | 379 → 369 / 397 → 388 |
+| Qwen3.5-0.8B | `--attention fast` | 2.0% | 387 → 386 / 386 → 387 |
+| Qwen3.5-2B | `--head 4` | 10.2% | 376 → 352 / 388 → 375 |
+| Qwen3.5-9B | `--head 4` | 5.1% | 383 → 372 / 393 → 391 |
+| Qwen3-30B-A3B | `--head 4 --attention fast --bits 4` | 10.0% | 57 → 55 of 64 / 392 → 385 |
+| Qwen3.6-35B-A3B | `--head 4 --bits 4` | 6.5% | 60 → 55 of 64 / 196 → 194 of 200 |
+| Qwen3-Next-80B-A3B | `--head 4 --bits 4` | 8.6% | 386 → 372 / 397 → 392 |
+| Qwen3-Coder-Next | `--head 4 --bits 4` | 10.5% | — / 193 → 196 of 200 |
+
+Qwen3.5-0.8B does not take `--head 4`: with it, it answered "3" to "2+3". Gemma 4 has no fast profile: its keys are sixteen-bit, where `--attention fast` has nothing to do, its heads are not six-bit and its experts are not in planes. `--attention fast` pays where attention is a larger share of a token at 1,900 positions (Qwen3-0.6B +7.5%, Qwen3-30B-A3B +4.0%); on the hybrid Qwen3.5, Qwen3.6 and Qwen3-Next it adds 2% or less.
+
+**The measure on this machine.** What makes a model fastest depends on the machine too, so the first time a model starts in a terminal the chat offers to measure it: the model opened each way that can differ — drafts from its prediction file or draft model, from the conversation, or none; in the `fast` profile also the output head as in the file — one reply to warm it and the same reply measured at temperature 0, so every way writes the same text. A way other than the one the model starts with has to be at least 3% faster to be kept (one reply is measured, and a couple of per cent is its noise). The result goes to `~/.cache/janas/tune-<name>-<profile>.conf`, with the speeds as a comment, and is used for the levers you did not set yourself; a "no" is kept too, so the question comes once. `--tune` measures again, `--no-tune` neither asks nor uses it. On the development laptop: Qwen3-4B 33.1 tokens/s with Qwen3-0.6B's drafts, 29.2 without drafts, 29.5 with drafts from the conversation; Qwen3.5-2B 77.4 with its MTP block, 41.9 without drafts, 60.6 from the conversation, and with the head at four bits 77.4 against 75.6 at six; Qwen3-0.6B within 1% either way, so it starts as it would have.
+
| Option | What it does |
|---|---|
+| `--profile <quality\|fast>` | the catalog's profile for the model (default `quality`) |
+| `--models <dir>` | where the models are (default: `JANAS_MODELS`, or `~/.local/share/janas/models`) |
+| `--tune`, `--no-tune` | measure again how the model runs fastest on this machine; or neither ask nor use that measure |
| `--mtp <file>` | the model's multi-token prediction file (an MTP block, or Gemma 4's assistant): faster replies, same text; without it, the one beside the model that fits it |
| `--no-mtp` | no prediction file, not even one found beside the model |
| `--draft <file>` | a small model of the same family to guess the next tokens: faster replies, same text (see below) |
@@ -241,6 +266,7 @@ janas-chat <model.jns> [options]
| `--reserve <GiB>` | when the cache is automatic, the memory left to other programs (default: a fifth of the machine's); more keeps a busy desktop out of swap, at the price of a smaller cache |
| `--bits <2\|4\|6>` | bits per weight of the experts' down matrix (default: the memory decides) |
| `--attention <exact\|fast>` | how the attention scores are computed (default: fast above 16384 tokens of context) |
+| `--head <4\|6>` | bits per weight of the output head: `4` requantizes a six-bit head at start, a few per cent faster and a little less exact (default: `6`, as in the file; see below) |
| `--no-preload` | do not fill the expert cache at start from this machine's profile |
| `--no-recap` | when the context fills, drop the oldest turns without summing them up |
| `--system <text>` | the system message (`""` for none; the default gives the assistant its name, Janas, names the model it thinks with, and says who wrote the engine) |
@@ -260,7 +286,9 @@ Where a model has no prediction block the guesses are copied from the conversati
**On sampling.** The defaults are the ones Qwen recommends for its own models: temperature 0.7, top-k 20, top-p 0.8. They are narrower than they look. With top-p 0.8, on Qwen3-4B, **196 of 256 tokens had a single candidate left after the cut** - nothing to draw, so the reply is the one the model would have given greedily, and two questions that mean the same thing get the same words. Starting with `--top-p 0.95` brings that to 142 of 256. It is not a fault and not a setting this project chose: it is what those numbers do to a model that is sure of itself.
-**On `--attention`.** At a long context most of a token goes into attention, and most of that into the scores: one dot product per position, per head. The `fast` way reads the query as sixteen-bit integers with a scale of its own, so the dot product becomes an exact integer sum — the machine does sixteen of those products at a time instead of eight, and the scores come out a fifth to a quarter quicker. What it costs is the query's quantization, a relative 3e-5, sixty times less than the eight-bit keys and values already carry; on four hundred tokens against llama.cpp the most likely token agreed 394 times out of 400 on C source against 393 the exact way, and 377 against 382 on Italian prose. The engine asks for it by itself only when you asked for more than 16384 tokens of context, where the gain is real and the conversation will get there; `--attention exact` and `--attention fast` settle it either way, and so does `JANAS_ATTN=float|int16`.
+**On `--attention`.** At a long context most of a token goes into attention, and most of that into the scores: one dot product per position, per head. The `fast` way reads the query as sixteen-bit integers with a scale of its own, so the dot product becomes an exact integer sum — the machine does sixteen of those products at a time instead of eight, and the scores come out a fifth to a quarter quicker. What it costs is the query's quantization, a relative 3e-5, sixty times less than the eight-bit keys and values already carry; on four hundred tokens against llama.cpp the most likely token agreed 394 times out of 400 on C source against 393 the exact way, and 377 against 382 on Italian prose. Since 7 October 2026 the `fast` way also weighs the values with sixteen-bit integers, a further 6-11% off a layer's attention (`tests/bench_attn`, Qwen3-30B-A3B's geometry, 2,048 to 32,000 positions); on Qwen3-30B-A3B, 400 tokens of C against llama.cpp, the most likely token agreed 390 times out of 400 against 388 with the scores alone and 392 the exact way, with the same mean logit difference (0.2865, 0.2893, 0.2871); the figures before this sentence are for the scores alone. The engine asks for it by itself only when you asked for more than 16384 tokens of context, where the gain is real and the conversation will get there; `--attention exact` and `--attention fast` settle it either way, and so does `JANAS_ATTN=float|int16`.
+
+**On `--head`.** The output head - the matrix that turns the model's state into a score for every word of the vocabulary - is the largest one read whole for every token: on Qwen3.5-9B a seventh of a token's time. Most files keep it at six bits per weight (Q6_K); `--head 4` requantizes it to four (Q4_K) when the model opens, a second or two, and leaves everything else as it is. Measured on 7 October 2026, 128 tokens after a prompt of 1,900: Qwen3.5-9B writes 13.35-13.36 tokens/s where 12.76-12.77 (+4.6%, the head 7.8 ms a token where 11.0), Qwen3-Next-80B-A3B with a 16 GiB cache 25.0-25.3 where 24.4-24.5 (+2.8%); prompts are read at the same speed. The price is in the answers: against llama.cpp on 400 tokens, the most likely token agreed 372 times where 383 on Qwen3.5-9B (Italian prose) and 391 where 393 (C source), 372 where 382 and 392 where 393 on Qwen3-Next-80B-A3B, 387 where 392 on Qwen3-30B-A3B (C source), and the mean difference of the scores grows by a quarter to two thirds. Requantizing the other six-bit matrices as well (attention, the shared experts) cost more agreement for next to no speed, so the option stops at the head. The files of Qwen3.5-9B, Qwen3.8-27B, Qwen3-30B-A3B, Qwen3-Coder-30B-A3B, Qwen3.6-35B-A3B and Qwen3-Next-80B-A3B have a six-bit head of their own; those of Qwen3-0.6B, Qwen3-4B, Qwen3.5-0.8B and Qwen3.5-2B use their six-bit token embeddings as the head, and there the table is requantized for both uses (the input embeddings then read at four bits, as the larger models' files already keep them): three runs each with nothing else running, Qwen3-0.6B 107.0-107.7 tokens/s where 100.6-101.9 (+6%), Qwen3.5-0.8B 113.5-114.7 where 104.2-105.8 (+9%), Qwen3.5-2B 54.4-54.9 where 50.0-50.4 (+9%), Qwen3-4B 24.0-24.1 where 23.2-23.3 (+3.4%). Here the answers move more: 352 where 376 on Qwen3.5-2B (prose), 363 where 387 on Qwen3.5-0.8B, 366 where 379 on Qwen3-4B, 327 where 345 on Qwen3-0.6B. Gemma 4's embeddings are not six-bit and are left alone. `JANAS_HEAD_BITS=4` does the same from the environment.
In the chat, a line starting with `/` is a command, and it turns grey-blue as soon as it names one the chat really has:
diff --git a/include/janas/llm.bi b/include/janas/llm.bi
index d370314..8de94e4 100644
--- a/include/janas/llm.bi
+++ b/include/janas/llm.bi
@@ -62,6 +62,8 @@ type janas_llm_params
expert_bits as long '' 2, 4 or 6; 0: the engine chooses
attn_scores as long '' JANAS_LLM_ATTN_*; 0: the engine chooses
reserve_bytes as ulongint '' memory left to other programs (0: a fifth)
+ head_bits as long '' 4: a six-bit output head read at four
+ '' (faster, less exact); 0 or 6: as in the file
end type
type janas_llm_chat_params
diff --git a/include/janas/llm.h b/include/janas/llm.h
index e905c75..ca4f8c4 100644
--- a/include/janas/llm.h
+++ b/include/janas/llm.h
@@ -103,16 +103,30 @@ struct janas_llm_params {
2 GiB (default). More keeps a busy desktop out of swap, at the price
of a smaller cache on a model larger than the memory */
uint64_t reserve_bytes;
+ /* bits per weight of the output head, the largest matrix read whole at
+ every token. 4: when the file holds it at six bits (Q6_K), it is
+ requantized to four (Q4_K) as the model opens - a second or two - for
+ a few per cent more tokens a second and answers a little less close
+ to the original weights. On 400 tokens against llama.cpp the most
+ likely token agreed 372 times where 383 on Qwen3.5-9B (Italian prose;
+ 4.6% faster), 372 where 382 on Qwen3-Next-80B-A3B (Italian prose;
+ 2.8% faster), 387 where 392 on Qwen3-30B-A3B (C source). Where the
+ head is the token embeddings, as in the small Qwen, the table is
+ requantized for both: Qwen3.5-2B 9% faster, 352 where 376. 0 or 6:
+ as in the file (default) */
+ int32_t head_bits;
};
/*
* How the attention scores are computed. EXACT reads the query as it is and
* sums in floating point. FAST reads it as sixteen-bit integers with a scale
* of its own: the sum is then an exact integer one, which the machine does
- * about twice as fast, and the only error left is the query's quantization -
- * a relative 3e-5, against the 4e-3 the eight-bit keys and values already
- * carry. AUTO takes FAST when the machine has the arithmetic for it and the
- * caller asked for more than the usual 16384 tokens of context, where
+ * about twice as fast, and the error it adds is the query's quantization - a
+ * relative 3e-5, against the 4e-3 the eight-bit keys and values already
+ * carry. FAST also weighs the values with sixteen-bit integers, whose
+ * rounding adds at most a few 1e-4 to an output (no difference measured on
+ * a model's logits). AUTO takes FAST when the machine has the arithmetic for it
+ * and the caller asked for more than the usual 16384 tokens of context, where
* attention is the greater part of a token; below that it takes EXACT, since
* a conversation that stays short would pay the quantization for a gain it
* never sees. The environment variable JANAS_ATTN (float, int16, auto)
diff --git a/src/chat/edit.h b/src/chat/edit.h
index 7335979..9de9cb7 100644
--- a/src/chat/edit.h
+++ b/src/chat/edit.h
@@ -341,6 +341,7 @@ static int line_read(struct janas_term *t, const char *prompt, char *buf,
pos = len;
} else if (c == 12) { /* Ctrl-L: a clean window */
printf("\033[2J\033[H");
+ t->cleared++;
term_region(t);
printf("\0337");
} else if (c == 27) { /* a key that arrives as an escape sequence */
diff --git a/src/chat/main.c b/src/chat/main.c
index 54d13a4..7e1f6b3 100644
--- a/src/chat/main.c
+++ b/src/chat/main.c
@@ -37,6 +37,8 @@
#include "janas/llm.h"
#include "markdown.h"
#include "mcp_chat.h"
+#include "models.h"
+#include "tune.h"
#include "scroll.h"
#include "term.h"
@@ -236,7 +238,19 @@ static void usage(void)
{
fprintf(
stderr,
- "usage: janas-chat <model.jns> [options]\n"
+ "usage: janas-chat [model] [options]\n"
+ " model a .jns file, or a name of the catalog (none: the\n"
+ " list, to choose one; a model not here yet is\n"
+ " downloaded and converted, after asking)\n"
+ " --profile <quality|fast>\n"
+ " quality (default): answers as the file gives\n"
+ " them; fast: also what was measured faster on\n"
+ " this model at a small cost to its answers\n"
+ " --models <dir> where the models are (default: JANAS_MODELS,\n"
+ " or ~/.local/share/janas/models)\n"
+ " --tune measure again how this model runs fastest on\n"
+ " this machine (asked once, on its first start)\n"
+ " --no-tune neither ask nor use that measure\n"
" --mtp <file> multi-token prediction file: an MTP block or\n"
" Gemma 4's assistant (faster replies); without\n"
" it, the one beside the model that fits it\n"
@@ -252,6 +266,10 @@ static void usage(void)
" --bits <2|4|6> bits per weight of the experts' down matrix\n"
" (files that hold it in planes; default: the\n"
" memory decides)\n"
+ " --head <4|6> bits per weight of the output head: 4\n"
+ " requantizes a six-bit one at start, 2-5%%\n"
+ " faster and a little less exact (default: 6,\n"
+ " as in the file)\n"
" --attention <exact|fast>\n"
" the attention scores: the query as it is, or\n"
" to sixteen bits with an exact integer sum,\n"
@@ -334,10 +352,13 @@ static const struct {
{"/mcp auto [on|off]",
"run the tools the model calls without asking first (off by default: "
"every call is shown and waits for y, n or a - always that tool)"},
- {"/save-config",
- "keep these settings for the next start (asks to overwrite)"},
- {"/load-config", "read them back, losing the current ones (asks first)"},
- {"/del-config", "forget them: the next start uses the chat's own"},
+ {"/save-config [global]",
+ "keep these settings for this model's next start (asks to overwrite); "
+ "global: for every model"},
+ {"/load-config [global]",
+ "read them back, losing the current ones (asks first)"},
+ {"/del-config [global]",
+ "forget them: the next start uses the catalog's and the chat's own"},
{"/quit", "exit (also Ctrl-D)"},
{NULL, "A backslash and Enter add a line to the message (the backslash "
"is not sent); Enter alone sends it. Ctrl-C stops a reply."},
@@ -799,19 +820,23 @@ struct chat_opts { /* where the words land: main keeps them, this names them */
int *no_mtp;
const char *mtp_found; /* the MTP file found, not asked for: unsaved */
int *all_tools;
+ int *profile; /* 1: the catalog's fast profile */
+ int *tune; /* 1: measure again (--tune), -1: never (--no-tune) */
};
-/* every option, for the nearest to one mistyped: the first twelve stand
+/* every option, for the nearest to one mistyped: the first N_ALONE stand
alone, the others take a value */
+#define N_ALONE 14
static const char *const OPTIONS[] = {
"--no-spec", "--stats", "--no-markdown", "--no-gpu",
"--no-preload", "--no-recap", "--no-mtp", "--no-mcp",
"--no-services", "--all-tools", "--mcp-auto", "--mcp-instructions",
- "--mtp", "--draft", "--think", "--mode",
- "--ctx", "--cache", "--reserve", "--bits",
- "--attention", "--system", "--mcp-config", "--progress",
- "--temp", "--top-k", "--top-p", "--min-p",
- "--seed", "--max"};
+ "--tune", "--no-tune", "--mtp", "--draft",
+ "--think", "--mode", "--ctx", "--cache",
+ "--reserve", "--bits", "--attention", "--system",
+ "--mcp-config", "--progress", "--temp", "--top-k",
+ "--top-p", "--min-p", "--seed", "--max",
+ "--head", "--profile", "--models"};
#define N_OPTIONS (sizeof OPTIONS / sizeof *OPTIONS)
/* v of option a as a whole number or a number within limits; on a
@@ -856,6 +881,8 @@ static int take_options(int n, char **w, struct chat_opts *o, char *why,
static const char *const MODES[] = {"auto", "eco", "max"};
static const char *const ATTN[] = {"auto", "exact", "fast"};
static const char *const BITS[] = {"2", "4", "6"};
+ static const char *const HEAD[] = {"4", "6"};
+ static const char *const PROFILES[] = {"quality", "fast"};
for (int i = 0; i < n; i++) {
const char *a = w[i], *v = i + 1 < n ? w[i + 1] : NULL;
long long x;
@@ -909,8 +936,12 @@ static int take_options(int n, char **w, struct chat_opts *o, char *why,
*o->mcp_instructions = 1;
continue;
}
+ if (strcmp(a, "--tune") == 0 || strcmp(a, "--no-tune") == 0) {
+ *o->tune = a[2] == 't' ? 1 : -1;
+ continue;
+ }
int valued = 0;
- for (size_t j = 12; j < N_OPTIONS && !valued; j++)
+ for (size_t j = N_ALONE; j < N_OPTIONS && !valued; j++)
valued = strcmp(a, OPTIONS[j]) == 0;
if (!valued) {
janas_unknown_word(why, why_len, a, "an option", OPTIONS,
@@ -952,6 +983,16 @@ static int take_options(int n, char **w, struct chat_opts *o, char *why,
if ((k = opt_choice(a, v, BITS, 3, why, why_len)) < 0)
return -1;
o->mp->expert_bits = 2 + 2 * k;
+ } else if (strcmp(a, "--profile") == 0) {
+ if ((k = opt_choice(a, v, PROFILES, 2, why, why_len)) < 0)
+ return -1;
+ *o->profile = k;
+ } else if (strcmp(a, "--models") == 0) {
+ setenv("JANAS_MODELS", v, 1); /* read when the model is found */
+ } else if (strcmp(a, "--head") == 0) {
+ if ((k = opt_choice(a, v, HEAD, 2, why, why_len)) < 0)
+ return -1;
+ o->mp->head_bits = k == 0 ? 4 : 0;
} else if (strcmp(a, "--attention") == 0) {
if ((k = opt_choice(a, v, ATTN, 3, why, why_len)) < 0)
return -1;
@@ -1023,7 +1064,9 @@ static int ask_yes(const char *question)
return yes;
}
-static int conf_path(char *buf, size_t n, int create)
+/* The settings file: the chat's own (key NULL), or a model's - key its
+ catalog name or its file's - in models/ beside it. */
+static int conf_path(char *buf, size_t n, int create, const char *key)
{
const char *xdg = getenv("XDG_CONFIG_HOME"), *home = getenv("HOME");
char dir[768];
@@ -1042,8 +1085,16 @@ static int conf_path(char *buf, size_t n, int create)
mkdir(up, 0700);
}
mkdir(dir, 0700);
+ if (key) {
+ char md[800];
+ snprintf(md, sizeof(md), "%s/models", dir);
+ mkdir(md, 0700);
+ }
}
- snprintf(buf, n, "%s/chat.conf", dir);
+ if (key)
+ snprintf(buf, n, "%s/models/%s.conf", dir, key);
+ else
+ snprintf(buf, n, "%s/chat.conf", dir);
return 0;
}
@@ -1124,6 +1175,8 @@ static void conf_write(const char *path, const struct chat_opts *o)
fprintf(f, "--reserve %.2f\n", (double)m->reserve_bytes / (1 << 30));
if (m->expert_bits)
fprintf(f, "--bits %d\n", m->expert_bits);
+ if (m->head_bits == 4)
+ fprintf(f, "--head 4\n");
if (m->attn_scores == JANAS_LLM_ATTN_EXACT)
fprintf(f, "--attention exact\n");
else if (m->attn_scores == JANAS_LLM_ATTN_FAST)
@@ -1158,6 +1211,8 @@ static void conf_write(const char *path, const struct chat_opts *o)
fprintf(f, "--mcp-auto\n");
if (*o->mcp_instructions)
fprintf(f, "--mcp-instructions\n");
+ if (*o->profile)
+ fprintf(f, "--profile fast\n");
fclose(f);
term_printf(&term, "written to %s (the system message is not kept)\n",
path);
@@ -1494,10 +1549,14 @@ int main(int argc, char **argv)
/* wcwidth() answers for the locale's character set, and in the C locale
it answers -1 to everything past ASCII: ask for the user's own */
setlocale(LC_CTYPE, "");
- if (argc < 2 || argv[1][0] == '-') {
- usage();
- return 2;
- }
+ for (int i = 1; i < argc; i++)
+ if (strcmp(argv[i], "-h") == 0 || strcmp(argv[i], "--help") == 0) {
+ usage();
+ return 0;
+ }
+ /* the model first, if given: a path or a name of the catalog */
+ const char *marg = argc >= 2 && argv[1][0] != '-' ? argv[1] : NULL;
+ int oi = marg ? 2 : 1;
if (janas_llm_abi_version() != JANAS_LLM_ABI_VERSION) {
fprintf(stderr, "janas-chat: library ABI %d, expected %d\n",
janas_llm_abi_version(), JANAS_LLM_ABI_VERSION);
@@ -1520,31 +1579,87 @@ int main(int argc, char **argv)
int gpu = 1; /* the GPU may be given work, if there is one */
const char *mcp_config = NULL; /* NULL: the default file */
int mcp = 1, mcp_auto = 0, mcp_instructions = 0, services = 1, no_mtp = 0;
- int all_tools = 0;
+ int all_tools = 0, profile = 0, tune = 0;
static char mtp_found[4096];
struct chat_opts o = {
- &mp, &cp, &system_arg, &stats, &markdown,
- &gpu, &mcp_config, &mcp, &mcp_auto, &mcp_instructions,
- &services, &no_mtp, mtp_found, &all_tools};
- /* the file first, the command line after it: what is typed wins */
+ &mp, &cp, &system_arg, &stats, &markdown,
+ &gpu, &mcp_config, &mcp, &mcp_auto, &mcp_instructions,
+ &services, &no_mtp, mtp_found, &all_tools, &profile,
+ &tune};
+ /*
+ * The settings in layers, each over the one before: the catalog's
+ * profile for the model, the chat's own file, the model's file, the
+ * command line. Only /save-config writes, and only the model's file
+ * (or, asked, the chat's): the catalog is never changed.
+ */
char cpath[1024];
- char *conf_text = NULL;
- if (conf_path(cpath, sizeof(cpath), 0) == 0) {
- int cn = 0;
- char **cw = conf_read(cpath, &cn, &conf_text);
- if (cw) {
- char why[400];
- int bad = take_options(cn, cw, &o, why, sizeof why) != 0;
- free(cw);
- if (bad) {
- fprintf(stderr, "janas-chat: %s: %s\n", cpath, why);
- free(conf_text);
+ char *conf_text = NULL, *mconf_text = NULL;
+ int cn = 0, mn = 0;
+ char **cw = conf_path(cpath, sizeof(cpath), 0, NULL) == 0
+ ? conf_read(cpath, &cn, &conf_text)
+ : NULL;
+ const char *where = chat_words_last(argc - oi, argv + oi, "--models");
+ if (!where)
+ where = chat_words_last(cn, cw, "--models");
+ if (where)
+ setenv("JANAS_MODELS", where, 1);
+ struct chat_model cm;
+ if (chat_model_resolve(marg, &cm) != 0)
+ return 1;
+ char mpath[1024];
+ char **mw = conf_path(mpath, sizeof(mpath), 0, cm.key) == 0
+ ? conf_read(mpath, &mn, &mconf_text)
+ : NULL;
+ const char *pv = chat_words_last(argc - oi, argv + oi, "--profile");
+ if (!pv)
+ pv = chat_words_last(mn, mw, "--profile");
+ if (!pv)
+ pv = chat_words_last(cn, cw, "--profile");
+ char why[400];
+ if (pv && strcmp(pv, "fast") == 0) {
+ const char *fw = cm.entry ? cm.entry->fast : NULL;
+ if (!fw)
+ fprintf(stderr, "profile fast: nothing measured faster on this "
+ "model yet, so the same as quality\n");
+ else {
+ /* the catalog's words, split on spaces */
+ static char fbuf[256];
+ char *pw[16];
+ int pn = 0;
+ snprintf(fbuf, sizeof(fbuf), "%s", fw);
+ for (char *t = strtok(fbuf, " "); t && pn < 16;
+ t = strtok(NULL, " "))
+ pw[pn++] = t;
+ if (take_options(pn, pw, &o, why, sizeof why) != 0) {
+ fprintf(stderr, "janas-chat: catalog: %s\n", why);
return 2;
}
+ fprintf(stderr, "profile fast: %s\n", fw);
}
}
- char why[400];
- if (take_options(argc - 2, argv + 2, &o, why, sizeof why) != 0) {
+ /* the levers the user set, in any file or typed: the measure leaves
+ them alone */
+ static const char *const SPEC_OPTS[] = {"--mtp", "--draft", "--no-mtp",
+ "--no-spec", NULL};
+ static const char *const HEAD_OPTS[] = {"--head", NULL};
+ int spec_user = chat_words_any(cn, cw, SPEC_OPTS) ||
+ chat_words_any(mn, mw, SPEC_OPTS) ||
+ chat_words_any(argc - oi, argv + oi, SPEC_OPTS);
+ int head_user = chat_words_any(cn, cw, HEAD_OPTS) ||
+ chat_words_any(mn, mw, HEAD_OPTS) ||
+ chat_words_any(argc - oi, argv + oi, HEAD_OPTS);
+ const char *bad_in = NULL;
+ if (cw && take_options(cn, cw, &o, why, sizeof why) != 0)
+ bad_in = cpath;
+ else if (mw && take_options(mn, mw, &o, why, sizeof why) != 0)
+ bad_in = mpath;
+ free(cw);
+ free(mw);
+ if (bad_in) {
+ fprintf(stderr, "janas-chat: %s: %s\n", bad_in, why);
+ return 2;
+ }
+ if (take_options(argc - oi, argv + oi, &o, why, sizeof why) != 0) {
fprintf(stderr, "janas-chat: %s\n", why);
fprintf(stderr, "janas-chat --help lists the options\n");
free(conf_text);
@@ -1559,9 +1674,79 @@ int main(int argc, char **argv)
int tty = term.tty;
/* the model's MTP block or assistant, when not named: the one beside
it that fits (--no-mtp: none) */
- if (!mp.mtp_path && !no_mtp &&
- janas_llm_find_mtp(argv[1], mtp_found, sizeof(mtp_found)) ==
- JANAS_LLM_OK) {
+ /* the measure of this model on this machine: a layer of its own over
+ the catalog's, for the levers the user left alone */
+ struct chat_tune_dims dims = {
+ .spec = !spec_user,
+ .head = profile && cm.entry && cm.entry->fast &&
+ strstr(cm.entry->fast, "--head 4") && !head_user};
+ char tpath[2300];
+ if (tune >= 0 && (dims.spec || dims.head) &&
+ chat_tune_path(tpath, sizeof(tpath), cm.key, profile) == 0) {
+ int have = access(tpath, F_OK) == 0;
+ int go = tune == 1;
+ if (!go && !have && isatty(STDIN_FILENO) && isatty(STDERR_FILENO)) {
+ go = chat_ask_yes("Measure how this model runs fastest on this "
+ "machine (a reply or two each way, a minute "
+ "or so on a large model)?");
+ if (!go) { /* asked once: the answer is kept */
+ FILE *f = fopen(tpath, "w");
+ if (f) {
+ fprintf(f, "# janas-chat: not measured (declined); "
+ "janas-chat --tune measures it\n");
+ fclose(f);
+ }
+ }
+ }
+ if (go) {
+ struct janas_llm_params probe = mp;
+ static char probe_mtp[4096], probe_draft[4096];
+ if (!probe.mtp_path && !probe.draft_path && !no_mtp &&
+ !(probe.draft_path = chat_model_draft(&cm, probe_draft,
+ sizeof(probe_draft))))
+ probe.mtp_path =
+ chat_model_mtp(&cm, probe_mtp, sizeof(probe_mtp));
+ chat_tune_run(cm.path, &probe, &cp, gpu, dims, tpath);
+ }
+ int tn = 0;
+ char *ttext = NULL; /* kept: the words point into it */
+ char **tw = conf_read(tpath, &tn, &ttext);
+ char *keep[16];
+ int kn = 0;
+ for (int i = 0; tw && i < tn && kn < 15; i++) {
+ int head = strcmp(tw[i], "--head") == 0;
+ if (head ? !dims.head : !dims.spec) {
+ i += head; /* skip the value too */
+ continue;
+ }
+ keep[kn++] = tw[i];
+ if (head && i + 1 < tn)
+ keep[kn++] = tw[++i];
+ }
+ if (kn) {
+ if (take_options(kn, keep, &o, why, sizeof why) != 0)
+ fprintf(stderr, "janas-chat: %s: %s\n", tpath, why);
+ else {
+ fprintf(stderr, "measured on this machine:");
+ for (int i = 0; i < kn; i++)
+ fprintf(stderr, " %s", keep[i]);
+ fprintf(stderr, " (--no-tune: without)\n");
+ }
+ }
+ free(tw);
+ }
+ static char draft_found[4096];
+ /* no drafting, no prediction file or draft to load */
+ if (!cp.speculate)
+ no_mtp = 1;
+ if (!mp.mtp_path && !mp.draft_path && !no_mtp &&
+ (mp.draft_path =
+ chat_model_draft(&cm, draft_found, sizeof(draft_found))))
+ fprintf(stderr,
+ "drafts from %s, as the catalog says (--no-mtp: without)\n",
+ cm.entry->draft);
+ if (!mp.mtp_path && !mp.draft_path && !no_mtp &&
+ chat_model_mtp(&cm, mtp_found, sizeof(mtp_found))) {
mp.mtp_path = mtp_found;
const char *b = strrchr(mtp_found, '/');
fprintf(stderr,
@@ -1569,12 +1754,12 @@ int main(int argc, char **argv)
"(--no-mtp: without)\n",
b ? b + 1 : mtp_found);
}
- fprintf(stderr, "loading %s ...\n", argv[1]);
+ fprintf(stderr, "loading %s ...\n", cm.path);
janas_llm *llm = NULL;
janas_llm_chat *c = NULL; /* only written when the open succeeded, which
a compiler looking across the whole program
cannot always see */
- if (janas_llm_open(argv[1], &mp, &llm) != JANAS_LLM_OK ||
+ if (janas_llm_open(cm.path, &mp, &llm) != JANAS_LLM_OK ||
janas_llm_chat_create(llm, &cp, &c) != JANAS_LLM_OK) {
fprintf(stderr, "janas-chat: %s\n", janas_llm_last_error());
return 1;
@@ -1620,7 +1805,7 @@ int main(int argc, char **argv)
char name[96] = "";
janas_llm_name(llm, name, sizeof(name), &dl);
if (term.tty) {
- term_banner(&term, "Janas-Chat", name[0] ? name : argv[1],
+ term_banner(&term, "Janas-Chat", name[0] ? name : cm.path,
"/help for the commands, /quit to leave");
term_printf(&term, "%s%s%s\n\n", T_DIM(&term), desc, T_RESET(&term));
char hp[512];
@@ -1680,7 +1865,8 @@ int main(int argc, char **argv)
fputs(page, stdout);
free(page);
} else if (strcmp(msg, "/save-config") == 0) {
- if (conf_path(cpath, sizeof(cpath), 1) != 0)
+ const char *key = strcmp(arg, "global") == 0 ? NULL : cm.key;
+ if (conf_path(cpath, sizeof(cpath), 1, key) != 0)
term_printf(&term, "nowhere to write it\n");
/* only when there is something to lose: asking about an
empty place is noise, and noise is what makes people
@@ -1689,15 +1875,16 @@ int main(int argc, char **argv)
ask_yes("overwrite the saved settings?"))
conf_write(cpath, &o);
} else if (strcmp(msg, "/del-config") == 0) {
- if (conf_path(cpath, sizeof(cpath), 0) != 0)
+ const char *key = strcmp(arg, "global") == 0 ? NULL : cm.key;
+ if (conf_path(cpath, sizeof(cpath), 0, key) != 0)
term_printf(&term, "nowhere to look for it\n");
else if (access(cpath, F_OK) != 0)
term_printf(&term, "there is no %s\n", cpath);
else if (ask_yes("delete the saved settings?")) {
if (remove(cpath) == 0)
term_printf(&term,
- "%s deleted: the next start uses the "
- "chat's own settings\n",
+ "%s deleted: the next start goes without "
+ "it\n",
cpath);
else
term_printf(&term, "cannot delete %s\n", cpath);
@@ -1708,7 +1895,8 @@ int main(int argc, char **argv)
struct janas_llm_params was = mp;
int cn = 0;
char *text = NULL;
- char **cw = conf_path(cpath, sizeof(cpath), 0) == 0
+ const char *key = strcmp(arg, "global") == 0 ? NULL : cm.key;
+ char **cw = conf_path(cpath, sizeof(cpath), 0, key) == 0
? conf_read(cpath, &cn, &text)
: NULL;
if (!cw) {
@@ -1730,6 +1918,7 @@ int main(int argc, char **argv)
if (was.n_ctx != mp.n_ctx ||
was.cache_bytes != mp.cache_bytes ||
was.expert_bits != mp.expert_bits ||
+ was.head_bits != mp.head_bits ||
was.attn_scores != mp.attn_scores ||
was.mtp_path != mp.mtp_path)
term_printf(&term,
@@ -1766,7 +1955,7 @@ int main(int argc, char **argv)
} else if (strcmp(msg, "/reset") == 0) {
restart(c, system);
if (term.tty) { /* a clean window, as at the start */
- term_banner(&term, "Janas-Chat", name[0] ? name : argv[1],
+ term_banner(&term, "Janas-Chat", name[0] ? name : cm.path,
"/help for the commands, /quit to leave");
footer(llm, c, NULL);
} else
diff --git a/src/chat/models.c b/src/chat/models.c
new file mode 100644
index 0000000..6e0ca62
--- /dev/null
+++ b/src/chat/models.c
@@ -0,0 +1,267 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * models.c - janas-chat's models by name (see models.h).
+ */
+#define _GNU_SOURCE
+#include "models.h"
+
+#include <spawn.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "janas/llm.h"
+
+extern char **environ;
+
+static int is_file(const char *path)
+{
+ struct stat st;
+ return stat(path, &st) == 0 && S_ISREG(st.st_mode);
+}
+
+static int is_dir(const char *path)
+{
+ struct stat st;
+ return stat(path, &st) == 0 && S_ISDIR(st.st_mode);
+}
+
+/* A line from the user, without its newline; NULL at the end of input. */
+static char *ask_line(const char *question, char *buf, size_t len)
+{
+ fputs(question, stderr);
+ fflush(stderr);
+ if (!fgets(buf, (int)len, stdin))
+ return NULL;
+ buf[strcspn(buf, "\n")] = 0;
+ return buf;
+}
+
+int chat_ask_yes(const char *question)
+{
+ char b[64];
+ fprintf(stderr, "%s [y/N] ", question);
+ fflush(stderr);
+ if (!fgets(b, sizeof(b), stdin))
+ return 0;
+ return b[0] == 'y' || b[0] == 'Y';
+}
+
+static void set_key(struct chat_model *cm, const char *base)
+{
+ snprintf(cm->key, sizeof(cm->key), "%s", base);
+ size_t n = strlen(cm->key);
+ if (n > 4 && strcmp(cm->key + n - 4, ".jns") == 0)
+ cm->key[n - 4] = 0;
+}
+
+/* The catalog's chat models, numbered from 1, with what is already here. */
+static void list(const char *dir)
+{
+ fprintf(stderr, "The models janas-chat knows (in %s):\n\n", dir);
+ int k = 0;
+ for (int i = 0; i < janas_catalog_n; i++) {
+ const struct janas_model_entry *m = &janas_catalog[i];
+ if (m->not_chat)
+ continue;
+ char p[4200];
+ snprintf(p, sizeof(p), "%s/%s", dir, m->out);
+ fprintf(stderr,
+ "%2d. %s%s\n %s; %.1f GB to download, %d GB of "
+ "memory or more\n",
+ ++k, m->name, is_file(p) ? " (here)" : "", m->what, m->gguf_gb,
+ m->ram_gb);
+ }
+ fputc('\n', stderr);
+}
+
+/* The k-th chat model of the list, from 1. */
+static const struct janas_model_entry *nth(int k)
+{
+ for (int i = 0; i < janas_catalog_n; i++)
+ if (!janas_catalog[i].not_chat && --k == 0)
+ return &janas_catalog[i];
+ return NULL;
+}
+
+/* janas-get, beside this program, for the model into dir; 0 if it made it */
+static int fetch(const struct janas_model_entry *m, const char *dir)
+{
+ char self[4096], tool[4200];
+ ssize_t n = readlink("/proc/self/exe", self, sizeof(self) - 1);
+ if (n <= 0)
+ return -1;
+ self[n] = 0;
+ char *slash = strrchr(self, '/');
+ if (!slash)
+ return -1;
+ *slash = 0;
+ snprintf(tool, sizeof(tool), "%s/janas-get", self);
+ char *argv[] = {"janas-get", (char *)m->name, "--dir", (char *)dir, NULL};
+ pid_t pid;
+ if (posix_spawn(&pid, tool, NULL, NULL, argv, environ) != 0) {
+ fprintf(stderr, "janas-chat: cannot start %s\n", tool);
+ return -1;
+ }
+ int st = 0;
+ while (waitpid(pid, &st, 0) < 0)
+ ;
+ return WIFEXITED(st) && WEXITSTATUS(st) == 0 ? 0 : -1;
+}
+
+/*
+ * The folder for a first download: the proposed one, or another the user
+ * types - and then a word on how janas-chat will find it again, since only
+ * JANAS_MODELS or --models tell it.
+ */
+static int choose_dir(char *dir, size_t len)
+{
+ const char *e = getenv("JANAS_MODELS");
+ if ((e && *e) || is_dir(dir) || !isatty(STDIN_FILENO))
+ return 0;
+ char b[4096];
+ fprintf(stderr, "The models will go to %s.\n", dir);
+ if (!ask_line("Press Enter to accept, or type another folder: ", b,
+ sizeof(b)))
+ return -1;
+ if (b[0]) {
+ snprintf(dir, len, "%s", b);
+ fprintf(stderr,
+ "Note: janas-chat looks in %s only when told - start it with "
+ "--models %s, or set JANAS_MODELS=%s.\n",
+ dir, dir, dir);
+ }
+ return 0;
+}
+
+int chat_model_resolve(const char *arg, struct chat_model *cm)
+{
+ memset(cm, 0, sizeof(*cm));
+ if (arg && is_file(arg)) {
+ snprintf(cm->path, sizeof(cm->path), "%s", arg);
+ snprintf(cm->dir, sizeof(cm->dir), "%s", arg);
+ char *slash = strrchr(cm->dir, '/');
+ const char *base = slash ? slash + 1 : arg;
+ if (slash)
+ *slash = 0;
+ else
+ snprintf(cm->dir, sizeof(cm->dir), ".");
+ for (int i = 0; i < janas_catalog_n; i++)
+ if (strcmp(janas_catalog[i].out, base) == 0)
+ cm->entry = &janas_catalog[i];
+ set_key(cm, cm->entry ? cm->entry->name : base);
+ return 0;
+ }
+ if (arg && (strchr(arg, '/') || strstr(arg, ".jns"))) {
+ fprintf(stderr, "janas-chat: there is no file %s\n", arg);
+ return -1;
+ }
+ if (janas_models_dir(cm->dir, sizeof(cm->dir)) != 0) {
+ fprintf(stderr, "janas-chat: no home directory for the models: give "
+ "a model's path\n");
+ return -1;
+ }
+ const struct janas_model_entry *m = NULL;
+ if (arg) {
+ m = janas_catalog_find(arg);
+ if (!m || m->not_chat) {
+ fprintf(stderr,
+ "janas-chat: no model called %s (janas-chat with no "
+ "model lists them)\n",
+ arg);
+ return -1;
+ }
+ } else {
+ if (!isatty(STDIN_FILENO)) {
+ fprintf(stderr, "janas-chat: no model given\n");
+ return -1;
+ }
+ list(cm->dir);
+ char b[128];
+ while (!m) {
+ if (!ask_line("Which one (number or name, Enter to leave)? ", b,
+ sizeof(b)) ||
+ !b[0])
+ return -1;
+ int k = atoi(b);
+ m = k > 0 ? nth(k) : janas_catalog_find(b);
+ if (m && m->not_chat)
+ m = NULL;
+ if (!m)
+ fprintf(stderr, "not in the list: %s\n", b);
+ }
+ }
+ cm->entry = m;
+ set_key(cm, m->name);
+ snprintf(cm->path, sizeof(cm->path), "%s/%s", cm->dir, m->out);
+ if (is_file(cm->path))
+ return 0;
+ if (!isatty(STDIN_FILENO)) {
+ fprintf(stderr, "janas-chat: %s is not here yet: janas-get %s\n",
+ m->name, m->name);
+ return -1;
+ }
+ if (choose_dir(cm->dir, sizeof(cm->dir)) != 0)
+ return -1;
+ snprintf(cm->path, sizeof(cm->path), "%s/%s", cm->dir, m->out);
+ if (is_file(cm->path))
+ return 0;
+ char q[512];
+ snprintf(q, sizeof(q),
+ "%s is not here: download it (%.1f GB, and about %.0f GB free "
+ "while it is converted) with its prediction file?",
+ m->name, m->gguf_gb, m->gguf_gb * 2.0 + 0.5);
+ if (!chat_ask_yes(q))
+ return -1;
+ if (fetch(m, cm->dir) != 0 || !is_file(cm->path)) {
+ fprintf(stderr, "janas-chat: %s was not made\n", cm->path);
+ return -1;
+ }
+ return 0;
+}
+
+const char *chat_model_draft(const struct chat_model *cm, char *buf, size_t len)
+{
+ const struct janas_model_entry *d =
+ cm->entry && cm->entry->draft ? janas_catalog_find(cm->entry->draft)
+ : NULL;
+ if (!d)
+ return NULL;
+ snprintf(buf, len, "%s/%s", cm->dir, d->out);
+ return is_file(buf) ? buf : NULL;
+}
+
+int chat_words_any(int n, char *const *w, const char *const *opts)
+{
+ for (int i = 0; i < n; i++)
+ for (int k = 0; opts[k]; k++)
+ if (strcmp(w[i], opts[k]) == 0)
+ return 1;
+ return 0;
+}
+
+const char *chat_model_mtp(const struct chat_model *cm, char *buf,
+ size_t len)
+{
+ if (cm->entry) {
+ if (cm->entry->extra == JANAS_EXTRA_NONE || !cm->entry->extra_out)
+ return NULL;
+ snprintf(buf, len, "%s/%s", cm->dir, cm->entry->extra_out);
+ return is_file(buf) ? buf : NULL;
+ }
+ return janas_llm_find_mtp(cm->path, buf, len) == JANAS_LLM_OK ? buf
+ : NULL;
+}
+
+const char *chat_words_last(int n, char *const *w, const char *opt)
+{
+ const char *v = NULL;
+ for (int i = 0; i + 1 < n; i++)
+ if (strcmp(w[i], opt) == 0)
+ v = w[i + 1];
+ return v;
+}
diff --git a/src/chat/models.h b/src/chat/models.h
new file mode 100644
index 0000000..22a30a0
--- /dev/null
+++ b/src/chat/models.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * models.h - janas-chat's models by name: a path, a name of the catalog
+ * (src/common/catalog.c), or none, and then a list to choose from. A model
+ * of the catalog that is not there yet is downloaded and converted by
+ * janas-get, with its prediction file or draft, once the user says so.
+ */
+#ifndef JANAS_CHAT_MODELS_H
+#define JANAS_CHAT_MODELS_H
+
+#include <stddef.h>
+
+#include "common/catalog.h"
+
+struct chat_model {
+ char path[4400]; /* the .jns to open */
+ char dir[4096]; /* the folder it is in */
+ char key[128]; /* its settings file: the catalog name, or the file's */
+ const struct janas_model_entry *entry; /* NULL: not in the catalog */
+};
+
+/* arg: a path, a catalog name, or NULL for the list. 0, or -1 with the
+ reason printed. */
+int chat_model_resolve(const char *arg, struct chat_model *cm);
+
+/* The draft the catalog names for the model, if its file is beside it:
+ the path in buf, or NULL. */
+const char *chat_model_draft(const struct chat_model *cm, char *buf,
+ size_t len);
+
+/* The model's prediction file: for a model of the catalog the one the
+ catalog names, beside it (none if it names none: a file that merely fits
+ may be another model's - Qwen3-Coder-Next took Qwen3-Next-80B's, and
+ drafted slower than from the conversation); otherwise the one
+ janas_llm_find_mtp finds beside it. The path in buf, or NULL. */
+const char *chat_model_mtp(const struct chat_model *cm, char *buf,
+ size_t len);
+
+/* The value of the last opt among n words, or NULL. */
+const char *chat_words_last(int n, char *const *w, const char *opt);
+
+/* 1 if one of the n words is one of opts (NULL-terminated). */
+int chat_words_any(int n, char *const *w, const char *const *opts);
+
+/* A question answered y or n on standard input (before the chat's own
+ terminal is set up); anything but y is no. */
+int chat_ask_yes(const char *question);
+
+#endif
diff --git a/src/chat/scroll.h b/src/chat/scroll.h
index 85fee3d..90b1b60 100644
--- a/src/chat/scroll.h
+++ b/src/chat/scroll.h
@@ -39,6 +39,14 @@ struct janas_scroll {
char sgr[SCROLL_SGR_MAX];
size_t sgr_n;
int back; /* rows above the end the window sits at; 0: at the end */
+ /* the bytes each row of the window was last drawn with, for the partial
+ redraw of scroll_sink: trusted while the window's size and the
+ terminal's count of clears are those of that drawing */
+ char **shown;
+ size_t *shown_n;
+ int shown_view, shown_cols; /* 0: nothing known */
+ int shown_full; /* every row of the window had text */
+ unsigned shown_cleared;
};
static void scroll_init(struct janas_scroll *s)
@@ -52,6 +60,10 @@ static void scroll_free(struct janas_scroll *s)
free(s->line[i]);
free(s->line);
free(s->pend);
+ for (int r = 0; r < s->shown_view; r++)
+ free(s->shown[r]);
+ free(s->shown);
+ free(s->shown_n);
memset(s, 0, sizeof(*s));
}
@@ -221,20 +233,44 @@ static int sc_collect(const struct janas_scroll *s, int cols, int back,
return got;
}
+/* A growing buffer of bytes. */
+struct sc_buf {
+ char *b;
+ size_t n, cap;
+};
+
+static void sc_put(struct sc_buf *o, const char *p, size_t n)
+{
+ if (o->n + n > o->cap) {
+ size_t cap = o->cap ? o->cap * 2 : 256;
+ while (cap < o->n + n)
+ cap *= 2;
+ char *b = realloc(o->b, cap);
+ if (!b)
+ return;
+ o->b = b;
+ o->cap = cap;
+ }
+ memcpy(o->b + o->n, p, n);
+ o->n += n;
+}
+
/*
* One row, with the colours it inherits: a row that starts in the middle of
* a written line would otherwise lose the escape that coloured it, so the
* escapes before it are sent again, and nothing else of what they preceded.
+ * Into o, which then holds exactly what the row is drawn with.
*/
-static void sc_put_row(const char *text, const struct sc_row *r)
+static void sc_put_row(struct sc_buf *o, const char *text,
+ const struct sc_row *r, int cols)
{
size_t n = strlen(text);
- if (r->hang > 0)
- printf("%*s", r->hang, "");
+ for (int k = 0; k < r->hang; k++)
+ sc_put(o, " ", 1);
for (size_t i = 0; i < (size_t)r->off && i < n;) {
size_t e = sc_esc(text + i, n - i);
if (e) {
- fwrite(text + i, 1, e, stdout);
+ sc_put(o, text + i, e);
i += e;
} else {
size_t bytes;
@@ -242,15 +278,46 @@ static void sc_put_row(const char *text, const struct sc_row *r)
i += bytes ? bytes : 1;
}
}
+ /* no further than the window's edge: a row is wrapped to fit, but the
+ space it was broken after can stay at its end, one column past the
+ edge, where the terminal would wrap it again - on the region's last
+ row by moving the whole window up one */
size_t from = (size_t)r->off;
size_t to = r->end < 0 ? n : (size_t)r->end;
- if (from < to)
- fwrite(text + from, 1, to - from, stdout);
+ int used = r->hang;
+ for (size_t i = from; i < to;) {
+ size_t e = sc_esc(text + i, to - i);
+ if (e) {
+ sc_put(o, text + i, e);
+ i += e;
+ continue;
+ }
+ size_t bytes;
+ int w = term_char_cols(text + i, to - i, &bytes);
+ if (!bytes)
+ bytes = 1;
+ if (used + w <= cols) {
+ sc_put(o, text + i, bytes);
+ used += w;
+ }
+ i += bytes;
+ }
}
-/* The window, drawn over the scrolling region. Returns the columns the
- bottom row holds, which is where the next text goes. */
-static int scroll_draw(struct janas_term *t, const struct janas_scroll *s)
+/*
+ * The window, drawn over the scrolling region. Returns the columns the
+ * bottom row holds, which is where the next text goes. With `partial`, the
+ * rows the terminal already shows are left alone: while a reply streams in,
+ * a piece of it changes the row being written and, when the text wraps, moves
+ * every row up. Redrawing the whole window for each piece sent 12 KB a token
+ * to the terminal and cost Konsole 3% of the reply's speed (2 Oct 2026). Each
+ * row is laid out into the bytes it is drawn with; a row whose bytes are
+ * those it was drawn with last time is not sent again, and when the rows
+ * match those shown k rows lower the terminal moves them itself - a newline
+ * on the region's last row - and only the rows that are new are drawn.
+ */
+static int scroll_draw_rows(struct janas_term *t, struct janas_scroll *s,
+ int partial)
{
if (!t->tty)
return 0;
@@ -258,8 +325,12 @@ static int scroll_draw(struct janas_term *t, const struct janas_scroll *s)
if (view < 1)
view = 1;
struct sc_row *rows = malloc((size_t)view * sizeof(*rows));
- if (!rows)
+ struct sc_buf *now = calloc((size_t)view, sizeof(*now));
+ if (!rows || !now) {
+ free(rows);
+ free(now);
return 0;
+ }
int got = sc_collect(s, t->cols, s->back, view, rows);
/*
* A window that is not full fills from the top, as a terminal does: the
@@ -267,15 +338,81 @@ static int scroll_draw(struct janas_term *t, const struct janas_scroll *s)
* under it, not pushed to the bottom of the screen.
*/
int bottom = got < view ? got : view;
- printf("\0337"); /* the box and the footer keep their cursor */
for (int r = 1; r <= view; r++) {
- printf("\033[%d;1H\033[K\033[0m", r);
int i = bottom - r; /* rows[0] is the newest, and sits at bottom */
if (i >= 0 && i < got)
- sc_put_row(sc_text(s, rows[i].line), &rows[i]);
+ sc_put_row(&now[r - 1], sc_text(s, rows[i].line), &rows[i],
+ t->cols);
+ }
+ int known = partial && s->shown && s->shown_view == view &&
+ s->shown_cols == t->cols && s->shown_cleared == t->cleared;
+ if (!known) { /* everything drawn, and remembered from here */
+ for (int r = 0; r < s->shown_view; r++)
+ free(s->shown[r]);
+ free(s->shown);
+ free(s->shown_n);
+ s->shown = calloc((size_t)view, sizeof(*s->shown));
+ s->shown_n = calloc((size_t)view, sizeof(*s->shown_n));
+ s->shown_view = s->shown && s->shown_n ? view : 0;
+ s->shown_cols = t->cols;
+ s->shown_cleared = t->cleared;
+ }
+ printf("\0337"); /* the box and the footer keep their cursor */
+ /* a window that was not full held its rows at the top, with room
+ below: a move is a move only from full to full */
+ if (known && got >= view && s->shown_full) {
+ int best = 0, most = -1;
+ for (int k = 0; k <= 4 && k < view; k++) {
+ int match = 0;
+ for (int r = 1; r + k <= view; r++)
+ match +=
+ s->shown[r + k - 1] &&
+ s->shown_n[r + k - 1] == now[r - 1].n &&
+ !memcmp(s->shown[r + k - 1], now[r - 1].b, now[r - 1].n);
+ if (match > most) { /* ties: the smaller move */
+ most = match;
+ best = k;
+ }
+ }
+ if (best > 0) {
+ /* the region again, should anything have reset it: a newline
+ outside one would scroll the box and the footer away too */
+ printf("\033[1;%dr\033[%d;1H", view, view);
+ for (int k = 0; k < best; k++)
+ putchar('\n');
+ for (int r = 0; r < best; r++)
+ free(s->shown[r]);
+ memmove(s->shown, s->shown + best,
+ (size_t)(view - best) * sizeof(*s->shown));
+ memmove(s->shown_n, s->shown_n + best,
+ (size_t)(view - best) * sizeof(*s->shown_n));
+ for (int r = view - best; r < view; r++) {
+ s->shown[r] = NULL; /* blank now: a row unknown */
+ s->shown_n[r] = 0;
+ }
+ }
+ }
+ for (int r = 1; r <= view; r++) {
+ struct sc_buf *o = &now[r - 1];
+ if (known && s->shown[r - 1] && s->shown_n[r - 1] == o->n &&
+ !memcmp(s->shown[r - 1], o->b, o->n))
+ continue; /* the terminal shows these bytes already */
+ printf("\033[%d;1H\033[K\033[0m", r);
+ if (o->n)
+ fwrite(o->b, 1, o->n, stdout);
+ if (s->shown_view) {
+ free(s->shown[r - 1]);
+ s->shown[r - 1] = o->b ? o->b : calloc(1, 1);
+ s->shown_n[r - 1] = o->n;
+ o->b = NULL; /* kept */
+ }
}
+ for (int r = 0; r < view; r++)
+ free(now[r].b);
+ free(now);
printf("\033[0m\0338");
fflush(stdout);
+ s->shown_full = got >= view;
int cols = 0;
if (got > 0) { /* the bottom row: what is written there already */
const char *text = sc_text(s, rows[0].line);
@@ -297,12 +434,17 @@ static int scroll_draw(struct janas_term *t, const struct janas_scroll *s)
return cols;
}
+static int scroll_draw(struct janas_term *t, struct janas_scroll *s)
+{
+ return scroll_draw_rows(t, s, 0);
+}
+
/*
* The same, and the place the conversation writes from put back where the
* window now ends: the scrolling region moves when the box grows, and the
* cursor the conversation had saved may be under it by then.
*/
-static void scroll_anchor(struct janas_term *t, const struct janas_scroll *s)
+static void scroll_anchor(struct janas_term *t, struct janas_scroll *s)
{
int cols = scroll_draw(t, s);
if (!t->tty)
@@ -375,7 +517,7 @@ static void scroll_sink(void *arg, const char *text, size_t n)
* made narrower.
*/
if (s->back == 0 && s->t)
- scroll_draw(s->t, s);
+ scroll_draw_rows(s->t, s, 1);
}
static void scroll_attach(struct janas_term *t, struct janas_scroll *s)
diff --git a/src/chat/term.h b/src/chat/term.h
index fcc7e65..9c8fb50 100644
--- a/src/chat/term.h
+++ b/src/chat/term.h
@@ -42,6 +42,9 @@ struct janas_term {
through it, they are drawn where they are and kept nowhere. */
void (*sink)(void *, const char *, size_t);
void *sink_arg;
+ /* how many times the screen has been cleared: the store's partial
+ redraw (scroll.h) trusts what it drew only while this stays put */
+ unsigned cleared;
struct termios saved;
};
@@ -808,6 +811,7 @@ static void term_pager(struct janas_term *t, const char *text)
term_raw(t, 1);
for (;;) {
printf("\033[2J\033[H");
+ t->cleared++;
for (int i = 0; i < page && top + i < n; i++)
printf("%.*s%s\n", row[top + i].len, row[top + i].s, T_RESET(t));
/* the way out, said as fully as the window lets it be said */
@@ -865,6 +869,7 @@ static void term_page_begin(struct janas_term *t)
{
if (t->tty) {
printf("\033[?1049h\033[r\033[2J\033[H");
+ t->cleared++;
fflush(stdout);
}
}
@@ -1010,6 +1015,7 @@ static void term_banner(struct janas_term *t, const char *product,
const char *text[2] = {product, line3};
const char *face[2] = {T_BOLD(t), T_DIM(t)};
printf("\033[2J\033[H"); /* the cursor at the top: the banner goes there */
+ t->cleared++;
term_region(t);
printf("\033[1;1H");
term_beside(t, text, face, 2, t->rows - t->footer_rows - 1, 0);
diff --git a/src/chat/tune.c b/src/chat/tune.c
new file mode 100644
index 0000000..6d0958b
--- /dev/null
+++ b/src/chat/tune.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * tune.c - janas-chat's measure of a model on this machine (see tune.h).
+ *
+ * Each way is a model opened anew - the prediction file and the head's bits
+ * are taken when it opens - one reply to warm its caches, then three
+ * replies of different kinds measured together: output tokens over output
+ * seconds, at temperature 0, so every way writes the same text. Drafting off is measured on the first
+ * opening, since it only takes a parameter of the chat.
+ */
+#include "tune.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <time.h>
+
+#define TUNE_REPLY 128
+/* a way other than the one the model starts with has to be this much
+ faster: one reply is measured, and a couple of per cent is its noise
+ (Qwen3-0.6B: 154.6 and 154.7 drafting, 157.2 and 156.3 without) */
+#define TUNE_MARGIN 1.03
+/* three kinds of reply - prose, code, a list - since how far ahead the
+ drafts are right depends on the text: one reply alone swung by a fifth
+ between two runs (Gemma 4 E2B with its assistant: 38.6, then 46.9) */
+static const char *const PROMPTS[] = {
+ "Explain in about one hundred words how a bicycle stays upright.",
+ "Write a C function that reverses a singly linked list, with a short "
+ "comment.",
+ "List ten Italian cities with one line about each."};
+#define N_PROMPTS (sizeof PROMPTS / sizeof *PROMPTS)
+
+int chat_tune_path(char *buf, size_t len, const char *key, int fast)
+{
+ const char *x = getenv("XDG_CACHE_HOME"), *h = getenv("HOME");
+ char dir[2048];
+ if (x && *x)
+ snprintf(dir, sizeof(dir), "%s/janas", x);
+ else if (h && *h)
+ snprintf(dir, sizeof(dir), "%s/.cache/janas", h);
+ else
+ return -1;
+ int n = snprintf(buf, len, "%s/tune-%s-%s.conf", dir, key,
+ fast ? "fast" : "quality");
+ return n > 0 && (size_t)n < len ? 0 : -1;
+}
+
+/* One reply to prompt from a fresh conversation: its output tokens and
+ seconds added to *tok and *sec. 0, or -1. */
+static int reply1(janas_llm_chat *c, const char *prompt, double *tok,
+ double *sec)
+{
+ if (janas_llm_chat_reset(c) != JANAS_LLM_OK ||
+ janas_llm_chat_send(c, prompt, -1) != JANAS_LLM_OK)
+ return -1;
+ char buf[256];
+ int32_t len, r;
+ while ((r = janas_llm_chat_next(c, buf, sizeof(buf), &len)) == JANAS_LLM_OK)
+ ;
+ struct janas_llm_chat_stats s;
+ memset(&s, 0, sizeof(s));
+ s.size = sizeof(s);
+ if (r != JANAS_LLM_DONE || janas_llm_chat_stats(c, &s) != JANAS_LLM_OK ||
+ s.output_seconds <= 0 || s.output_tokens < 8)
+ return -1;
+ *tok += s.output_tokens;
+ *sec += s.output_seconds;
+ return 0;
+}
+
+/* The replies to every prompt: tokens a second over all of them, or -1. */
+static double reply(janas_llm_chat *c)
+{
+ double tok = 0, sec = 0;
+ for (size_t i = 0; i < N_PROMPTS; i++)
+ if (reply1(c, PROMPTS[i], &tok, &sec) != 0)
+ return -1;
+ return tok / sec;
+}
+
+/* One way: opened, warmed, measured drafting (*on) and, if off is given,
+ without drafting. 0, or -1. */
+static int measure(const char *path, const struct janas_llm_params *mp,
+ const struct janas_llm_chat_params *cp, int gpu, double *on,
+ double *off)
+{
+ janas_llm *llm = NULL;
+ janas_llm_chat *c = NULL;
+ struct janas_llm_params p = *mp;
+ p.no_preload = 1; /* the same start for every way */
+ struct janas_llm_chat_params q = *cp;
+ q.temperature = 0;
+ q.max_reply = TUNE_REPLY;
+ q.thinking = 0;
+ q.speculate = 1;
+ if (janas_llm_open(path, &p, &llm) != JANAS_LLM_OK) {
+ fprintf(stderr, "tuning: %s\n", janas_llm_last_error());
+ return -1;
+ }
+ if (!gpu)
+ janas_llm_set_gpu(llm, 0);
+ int rc = -1;
+ double wt = 0, ws = 0; /* the first prompt once, to warm the caches */
+ if (janas_llm_chat_create(llm, &q, &c) == JANAS_LLM_OK &&
+ reply1(c, PROMPTS[0], &wt, &ws) == 0 && (*on = reply(c)) > 0) {
+ rc = 0;
+ if (off) {
+ q.speculate = 0;
+ if (janas_llm_chat_set_params(c, &q) != JANAS_LLM_OK ||
+ (*off = reply(c)) <= 0)
+ rc = -1;
+ }
+ }
+ janas_llm_chat_destroy(c);
+ janas_llm_close(llm);
+ return rc;
+}
+
+int chat_tune_run(const char *path, const struct janas_llm_params *mp,
+ const struct janas_llm_chat_params *cp, int gpu,
+ struct chat_tune_dims dims, const char *file)
+{
+ const char *src = mp->draft_path ? "a second model"
+ : mp->mtp_path ? "the prediction file"
+ : "the model or the conversation";
+ double on = 0, off = 0, conv = 0;
+ fprintf(stderr, "tuning: drafts from %s...\n", src);
+ if (measure(path, mp, cp, gpu, &on, dims.spec ? &off : NULL) != 0)
+ return -1;
+ fprintf(stderr, "tuning: %.1f tokens/s drafting", on);
+ if (dims.spec)
+ fprintf(stderr, ", %.1f without drafts", off);
+ fputc('\n', stderr);
+ /* the drafts copied from the conversation, where a file or a second
+ model gives them now */
+ int alt = dims.spec && (mp->mtp_path || mp->draft_path);
+ if (alt) {
+ struct janas_llm_params p = *mp;
+ p.mtp_path = NULL;
+ p.draft_path = NULL;
+ fprintf(stderr, "tuning: drafts from the conversation...\n");
+ if (measure(path, &p, cp, gpu, &conv, NULL) != 0)
+ conv = 0;
+ else
+ fprintf(stderr, "tuning: %.1f tokens/s\n", conv);
+ }
+ /* the winner: 0 as it starts, 1 without drafts, 2 from the
+ conversation */
+ int best = 0;
+ double speed = on;
+ if (dims.spec && off > on * TUNE_MARGIN) {
+ best = 1;
+ speed = off;
+ }
+ if (conv > on * TUNE_MARGIN && conv > speed) {
+ best = 2;
+ speed = conv;
+ }
+ int head6 = 0;
+ double h6 = 0;
+ if (dims.head) {
+ struct janas_llm_params p = *mp;
+ p.head_bits = 6;
+ if (best == 2)
+ p.mtp_path = p.draft_path = NULL;
+ struct janas_llm_chat_params q = *cp;
+ if (best == 1)
+ q.speculate = 0;
+ fprintf(stderr, "tuning: the output head as in the file...\n");
+ if (measure(path, &p, &q, gpu, &h6, NULL) == 0) {
+ fprintf(stderr, "tuning: %.1f tokens/s\n", h6);
+ head6 = h6 > speed * TUNE_MARGIN;
+ }
+ }
+ char dir[2048];
+ snprintf(dir, sizeof(dir), "%s", file);
+ char *slash = strrchr(dir, '/');
+ if (slash) {
+ *slash = 0;
+ mkdir(dir, 0755);
+ }
+ FILE *f = fopen(file, "w");
+ if (!f) {
+ fprintf(stderr, "tuning: cannot write %s\n", file);
+ return -1;
+ }
+ char when[32];
+ time_t t = time(NULL);
+ strftime(when, sizeof(when), "%Y-%m-%d %H:%M", localtime(&t));
+ fprintf(f,
+ "# janas-chat: measured on this machine, %s, %zu replies of up "
+ "to %d tokens\n# at temperature 0 each way (tokens/s): drafting from "
+ "%s %.1f",
+ when, N_PROMPTS, TUNE_REPLY, src, on);
+ if (dims.spec)
+ fprintf(f, ", without drafts %.1f", off);
+ if (alt)
+ fprintf(f, ", drafts from the conversation %.1f", conv);
+ if (dims.head)
+ fprintf(f, ", output head as in the file %.1f", h6);
+ fprintf(f, "\n# janas-chat --tune measures again; delete the file to "
+ "forget it\n");
+ if (best == 1)
+ fprintf(f, "--no-spec\n");
+ if (best == 2)
+ fprintf(f, "--no-mtp\n");
+ if (head6)
+ fprintf(f, "--head 6\n");
+ fclose(f);
+ fprintf(stderr, "tuning: kept in %s\n", file);
+ return 0;
+}
diff --git a/src/chat/tune.h b/src/chat/tune.h
new file mode 100644
index 0000000..f74c8de
--- /dev/null
+++ b/src/chat/tune.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * tune.h - janas-chat's measure of a model on this machine: the levers that
+ * depend on it (where the drafts come from, whether to draft at all, the
+ * output head's bits in the fast profile) tried on one fixed reply each, the
+ * fastest kept in a file of its own - a layer between the catalog's profile
+ * and the user's settings, never written by /save-config.
+ */
+#ifndef JANAS_CHAT_TUNE_H
+#define JANAS_CHAT_TUNE_H
+
+#include <stddef.h>
+
+#include "janas/llm.h"
+
+/* ~/.cache/janas/tune-<key>-<quality|fast>.conf; 0, or -1 without a home */
+int chat_tune_path(char *buf, size_t len, const char *key, int fast);
+
+/* What may be tried: the drafts' source and drafting itself, the head. */
+struct chat_tune_dims {
+ int spec; /* the user set none of --mtp, --draft, --no-mtp, --no-spec */
+ int head; /* the fast profile asks for --head 4 and the user did not */
+};
+
+/*
+ * Measures the model at path with mp and cp as they would start (mp with
+ * the prediction file or draft already found), the levers of dims each
+ * way, and writes the winning words to file. gpu 0: the GPU is kept out.
+ * 0, or -1 if nothing could be measured.
+ */
+int chat_tune_run(const char *path, const struct janas_llm_params *mp,
+ const struct janas_llm_chat_params *cp, int gpu,
+ struct chat_tune_dims dims, const char *file);
+
+#endif
diff --git a/src/get/catalog.c b/src/common/catalog.c
similarity index 71%
rename from src/get/catalog.c
rename to src/common/catalog.c
index 66a103e..6d533c9 100644
--- a/src/get/catalog.c
+++ b/src/common/catalog.c
@@ -1,15 +1,39 @@
/* SPDX-License-Identifier: GPL-3.0-or-later
Copyright (C) 2026 Maurizio Cammalleri */
/*
- * catalog.c - the models janas-get knows (see catalog.h). Every SHA-256
- * here is one of MODELS.md; a model added here is added there too, and to
- * janas-try.sh's levels through this tool.
+ * catalog.c - the models Janas knows (see catalog.h). Every SHA-256 here
+ * is one of MODELS.md; a model added here is added there too, and to
+ * janas-try.sh's levels through janas-get. A "fast" profile holds the
+ * levers measured on that model, every combination of them, to make it
+ * faster by 2% or more each (8 Oct 2026, README "The fast profiles, as
+ * measured"; the models larger than the cache on the same text, since each
+ * lever changes the text a greedy reply takes and with it the experts read from
+ * disk). Not
+ * --head 4 on Qwen3.5-0.8B, which then answered "3" to "2+3"; nothing on
+ * Gemma 4, whose keys are sixteen-bit, heads not six-bit and experts not
+ * in planes.
*/
+#include <errno.h>
+#include <stdio.h>
+#include <stdlib.h>
#include <string.h>
+#include <sys/stat.h>
-#include "catalog.h"
+#include "common/catalog.h"
-const struct get_model get_catalog[] = {
+const struct janas_model_entry janas_catalog[] = {
+ {.name = "qwen3-0.6b",
+ .what = "Qwen3-0.6B, dense - also the draft of Qwen3-4B",
+ .repo = "unsloth/Qwen3-0.6B-GGUF",
+ .parts = {"Qwen3-0.6B-Q4_K_M.gguf"},
+ .part_sha =
+ {"ac2d97712095a558e31573f62f466a3f9d93990898b0ec79d7c974c1780d524a"},
+ .gguf_gb = 0.40,
+ .out = "qwen3-0.6b-q4km.jns",
+ .flat_sha =
+ "9174da760919b16b9f3a4ae8738c2d53605412f40bb2c0fe68d60174bf8a4fb5",
+ .ram_gb = 2,
+ .fast = "--head 4 --attention fast"},
{.name = "qwen3-4b",
.what = "Qwen3-4B, dense - the quick start",
.repo = "Qwen/Qwen3-4B-GGUF",
@@ -22,7 +46,9 @@ const struct get_model get_catalog[] = {
"e8ed44bdd3c612ce3ec1204a4b9db671115efab4a2ada384c70cf6492d430291",
.jns_sha =
"1cbd8ebfdf14aee05777f5278e655382f040cdedc21d282e93806fc8cf279fc5",
- .ram_gb = 8},
+ .ram_gb = 8,
+ .draft = "qwen3-0.6b",
+ .fast = "--head 4 --attention fast"},
{.name = "qwen3.5-0.8b",
.what = "Qwen3.5-0.8B, dense",
.repo = "unsloth/Qwen3.5-0.8B-GGUF",
@@ -33,7 +59,8 @@ const struct get_model get_catalog[] = {
.out = "qwen3.5-0.8b-q4km.jns",
.flat_sha =
"75a3b62c4a10cad87a7ce64bf89485cfcc564fa949f2c5af8c7051615dc26124",
- .ram_gb = 2},
+ .ram_gb = 2,
+ .fast = "--attention fast"},
{.name = "qwen3.5-2b",
.what = "Qwen3.5-2B, dense, with its MTP block",
.repo = "unsloth/Qwen3.5-2B-GGUF",
@@ -44,12 +71,13 @@ const struct get_model get_catalog[] = {
.out = "qwen3.5-2b-q4km.jns",
.flat_sha =
"286118ea3ad1903890a593582e22bed82859bf7c38b0ff9ed043050e5bd572a6",
- .extra = EXTRA_HF2JNS,
+ .extra = JANAS_EXTRA_HF2JNS,
.extra_src = "hf:Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc",
.extra_out = "qwen3.5-2b-mtp-q4k.jns",
.extra_sha =
"19c2d7eba1800acb10a06d7b38709ec92776e852cb2f25032ddd8977d17f2b05",
- .ram_gb = 4},
+ .ram_gb = 4,
+ .fast = "--head 4"},
{.name = "qwen3.5-9b",
.what = "Qwen3.5-9B, dense, with its MTP block",
.repo = "unsloth/Qwen3.5-9B-GGUF",
@@ -60,12 +88,13 @@ const struct get_model get_catalog[] = {
.out = "qwen3.5-9b-q4km.jns",
.flat_sha =
"e40ec6c58ef2390c7a4ce8c9b7b6b384ed4e55c9980f646d1c98e83e2c018115",
- .extra = EXTRA_HF2JNS,
+ .extra = JANAS_EXTRA_HF2JNS,
.extra_src = "hf:Qwen/Qwen3.5-9B@c202236235762e1c871ad0ccb60c8ee5ba337b9a",
.extra_out = "qwen3.5-9b-mtp-q4k.jns",
.extra_sha =
"67f31f57dbc78ca8c02ef5e3aa137c5df3836a3decda20b6ea057d004089c361",
- .ram_gb = 8},
+ .ram_gb = 8,
+ .fast = "--head 4"},
{.name = "qwen3-30b-a3b",
.what = "Qwen3-30B-A3B, mixture of experts",
.repo = "Qwen/Qwen3-30B-A3B-GGUF",
@@ -78,7 +107,8 @@ const struct get_model get_catalog[] = {
"6906de51f2923b3be3a83eb040eaaeb090cba3389e9217648abe08bd678376e9",
.jns_sha =
"3fa0185b7eecd43e222d420f68aae28a86b3b90392fd7e4052b5cd337318e895",
- .ram_gb = 16},
+ .ram_gb = 16,
+ .fast = "--head 4 --attention fast --bits 4"},
{.name = "qwen3.6-35b-a3b",
.what = "Qwen3.6-35B-A3B, mixture of experts, MTP inside",
.repo = "bartowski/Qwen_Qwen3.6-35B-A3B-GGUF",
@@ -91,7 +121,8 @@ const struct get_model get_catalog[] = {
"31772037300f332d712f4ef8dd5e7251c7cd76cf6b9dab33b089394b7c079fb2",
.jns_sha =
"8feaeb66d93593b9876b564ca3878f2a9252e46b8d4e2d06caa4e19d43dc6464",
- .ram_gb = 16},
+ .ram_gb = 16,
+ .fast = "--head 4 --bits 4"},
{.name = "qwen3-next-80b-a3b",
.what = "Qwen3-Next-80B-A3B-Instruct, with its MTP block",
.repo = "Qwen/Qwen3-Next-80B-A3B-Instruct-GGUF",
@@ -104,12 +135,13 @@ const struct get_model get_catalog[] = {
"fa9a50dd910de4e78064ecff17aef43286694277eea3a89a57e9921ca2bfd2da",
.jns_sha =
"fa3255c73106391e9cb5a97efd30f77a928fb4518d40060f795a8cff6656daed",
- .extra = EXTRA_HF2JNS,
+ .extra = JANAS_EXTRA_HF2JNS,
.extra_src = "hf:Qwen/Qwen3-Next-80B-A3B-Instruct",
.extra_out = "qwen3-next-80b-a3b-mtp-q4k.jns",
.extra_sha =
"5c9bc4cb6f739193fac5e4daa30c922f73255acf41443ecfc0123ef084584559",
- .ram_gb = 32},
+ .ram_gb = 32,
+ .fast = "--head 4 --bits 4"},
{.name = "qwen3-coder-next",
.what = "Qwen3-Coder-Next, mixture of experts, for code",
.repo = "Qwen/Qwen3-Coder-Next-GGUF",
@@ -129,7 +161,8 @@ const struct get_model get_catalog[] = {
"5912d964247002473eff1811058b4ffadf7b9bb47ae336c486cdf2964de68382",
.jns_sha =
"0276b0697a05075fec8ab076a757205537804ff41dd7e9dadace7c6993763563",
- .ram_gb = 32},
+ .ram_gb = 32,
+ .fast = "--head 4 --bits 4"},
{.name = "qwen3-embedding-0.6b",
.what = "Qwen3-Embedding-0.6B, for janas-server --embedding-model",
.repo = "Qwen/Qwen3-Embedding-0.6B-GGUF",
@@ -140,7 +173,8 @@ const struct get_model get_catalog[] = {
.out = "qwen3-embedding-0.6b-q8.jns",
.flat_sha =
"1e8eb8cbeba7a1a657354d259c5730bb79f6087160a9ca8da901bd08bdbe943f",
- .ram_gb = 2},
+ .ram_gb = 2,
+ .not_chat = 1},
{.name = "gemma-4-e2b",
.what = "Gemma-4-E2B-it, with its assistant",
.repo = "unsloth/gemma-4-E2B-it-GGUF",
@@ -151,7 +185,7 @@ const struct get_model get_catalog[] = {
.out = "gemma-4-e2b-it-q4km.jns",
.flat_sha =
"1f18e08878df41f377d563b38048d41841c7973ddea845cd9672fcf23cbd864b",
- .extra = EXTRA_GGUF,
+ .extra = JANAS_EXTRA_GGUF,
.extra_src = "mtp-gemma-4-E2B-it.gguf",
.extra_src_sha =
"9eba819938efccfd6044f8af84e3bbfddc639a2bcf32ebc36420e6a649191919",
@@ -169,7 +203,7 @@ const struct get_model get_catalog[] = {
.out = "gemma-4-e4b-it-q4km.jns",
.flat_sha =
"7ec828e9bdc1ada2eda907e93b87072f3180f2e8cba5ccedcd360d1a53858675",
- .extra = EXTRA_GGUF,
+ .extra = JANAS_EXTRA_GGUF,
.extra_src = "mtp-gemma-4-E4B-it.gguf",
.extra_src_sha =
"b6a723115efa510d3b3215db1e26790dae84cd08c2134a764f3d194f1f0c3376",
@@ -187,7 +221,7 @@ const struct get_model get_catalog[] = {
.out = "gemma-4-12b-it-q4km.jns",
.flat_sha =
"cae09e04598891dea6e9cd93fb496fbde08d822ac35de4ff45dbfab1ea25c4dd",
- .extra = EXTRA_GGUF,
+ .extra = JANAS_EXTRA_GGUF,
.extra_src = "mtp-gemma-4-12b-it.gguf",
.extra_src_sha =
"145db9094bc0f85f1701e255a2ed216dcc9800fc8bc8631ad00905b456bd451b",
@@ -205,7 +239,7 @@ const struct get_model get_catalog[] = {
.out = "gemma-4-26b-a4b-it-udq4km.jns",
.flat_sha =
"35c3845bbc21de52bf630d3c7455b4bb5379c34487b5925e020ec2e573931666",
- .extra = EXTRA_GGUF,
+ .extra = JANAS_EXTRA_GGUF,
.extra_src = "mtp-gemma-4-26B-A4B-it.gguf",
.extra_src_sha =
"6326fb9f5e487aa8dcdd313a091e3c67724cb2a666ec3b7d2895b5b26d93ed1b",
@@ -215,12 +249,48 @@ const struct get_model get_catalog[] = {
.ram_gb = 16},
};
-const int get_catalog_n = (int)(sizeof(get_catalog) / sizeof(get_catalog[0]));
+const int janas_catalog_n =
+ (int)(sizeof(janas_catalog) / sizeof(janas_catalog[0]));
-const struct get_model *get_find(const char *name)
+const struct janas_model_entry *janas_catalog_find(const char *name)
{
- for (int i = 0; i < get_catalog_n; i++)
- if (strcmp(get_catalog[i].name, name) == 0)
- return &get_catalog[i];
+ for (int i = 0; i < janas_catalog_n; i++)
+ if (strcmp(janas_catalog[i].name, name) == 0)
+ return &janas_catalog[i];
return NULL;
}
+
+int janas_models_dir(char *buf, size_t len)
+{
+ const char *e = getenv("JANAS_MODELS"), *x = getenv("XDG_DATA_HOME"),
+ *h = getenv("HOME");
+ int n;
+ if (e && *e)
+ n = snprintf(buf, len, "%s", e);
+ else if (x && *x)
+ n = snprintf(buf, len, "%s/janas/models", x);
+ else if (h && *h)
+ n = snprintf(buf, len, "%s/.local/share/janas/models", h);
+ else
+ return -1;
+ return n > 0 && (size_t)n < len ? 0 : -1;
+}
+
+int janas_mkdirs(const char *dir)
+{
+ char p[4096];
+ int n = snprintf(p, sizeof(p), "%s", dir);
+ if (n <= 0 || (size_t)n >= sizeof(p))
+ return -1;
+ for (char *c = p + 1; *c; c++)
+ if (*c == '/') {
+ *c = 0;
+ if (mkdir(p, 0755) != 0 && errno != EEXIST)
+ return -1;
+ *c = '/';
+ }
+ if (mkdir(p, 0755) != 0 && errno != EEXIST)
+ return -1;
+ struct stat st;
+ return stat(p, &st) == 0 && S_ISDIR(st.st_mode) ? 0 : -1;
+}
diff --git a/src/common/catalog.h b/src/common/catalog.h
new file mode 100644
index 0000000..f9d741c
--- /dev/null
+++ b/src/common/catalog.h
@@ -0,0 +1,66 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * catalog.h - the models Janas knows: where each comes from, the SHA-256 of
+ * every file on the way, as MODELS.md lists them, and how janas-chat runs
+ * it fastest. janas-get downloads and converts them; janas-chat lists them,
+ * takes one by name and starts it with its profile.
+ */
+#ifndef JANAS_COMMON_CATALOG_H
+#define JANAS_COMMON_CATALOG_H
+
+#include <stddef.h>
+
+#define JANAS_CATALOG_PARTS 4
+
+/* how the model's multi-token prediction file is made, if it has one */
+enum janas_model_extra {
+ JANAS_EXTRA_NONE,
+ JANAS_EXTRA_GGUF, /* a GGUF of its own (Gemma 4's assistant), gguf2jns */
+ JANAS_EXTRA_HF2JNS /* the block of the original checkpoint, hf2jns_mtp hf:
+ */
+};
+
+struct janas_model_entry {
+ const char *name; /* what janas-get takes */
+ const char *what; /* a line for the list */
+ const char *repo; /* the GGUF's repository */
+ const char *parts[JANAS_CATALOG_PARTS]; /* its file(s), the first given to
+ gguf2jns */
+ const char *part_sha[JANAS_CATALOG_PARTS];
+ double gguf_gb; /* download */
+ const char *out; /* the .jns, beside the downloads */
+ const char *flat_sha; /* what gguf2jns writes (the .jns itself when
+ there are no planes) */
+ const char *jns_sha; /* what jns_planes writes, NULL: one step */
+ enum janas_model_extra extra;
+ const char *extra_src; /* GGUF file of repo, or hf:repo[@rev] */
+ const char *extra_src_sha; /* of that GGUF (JANAS_EXTRA_GGUF) */
+ const char *extra_out;
+ const char *extra_sha;
+ int ram_gb; /* the smallest machine it is meant for */
+ int not_chat; /* 1: not a chat model (embeddings) */
+ const char *draft; /* the catalog model that guesses its tokens, when
+ that beats its own prediction file */
+ /* janas-chat's options for the "fast" profile, on top of the default
+ ("quality") one: the levers measured to make this model faster at a
+ small, measured cost to its answers (README, ChangeLog) */
+ const char *fast;
+};
+
+extern const struct janas_model_entry janas_catalog[];
+extern const int janas_catalog_n;
+
+const struct janas_model_entry *janas_catalog_find(const char *name);
+
+/*
+ * Where the models are: JANAS_MODELS if set, otherwise
+ * $XDG_DATA_HOME/janas/models, otherwise ~/.local/share/janas/models.
+ * Returns 0, or -1 if there is no home to put it in.
+ */
+int janas_models_dir(char *buf, size_t len);
+
+/* Makes dir and its parents (mode 0755). 0 if it is there afterwards. */
+int janas_mkdirs(const char *dir);
+
+#endif
diff --git a/src/common/env_names.c b/src/common/env_names.c
index fbe5afb..913643d 100644
--- a/src/common/env_names.c
+++ b/src/common/env_names.c
@@ -17,6 +17,7 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY",
"JANAS_API_KEY",
"JANAS_ARENA_HUGE",
"JANAS_ATTN",
+ "JANAS_ATTN_FUSE",
"JANAS_ATTN_SKIPSTAT",
"JANAS_AVIATIONSTACK_KEY",
"JANAS_BLOCK_THREADS",
@@ -46,11 +47,13 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY",
"JANAS_GPU_TOKEN_PROFILE",
"JANAS_GPU_VERBOSE",
"JANAS_GPU_XCOH",
+ "JANAS_HEAD_BITS",
"JANAS_KERNELS",
"JANAS_KV",
"JANAS_LOCATION",
"JANAS_METRICS",
"JANAS_METRICS_EVERY",
+ "JANAS_MODELS",
"JANAS_NOMINATIM_URL",
"JANAS_OSRM_URL",
"JANAS_OVERPASS_URL",
@@ -76,9 +79,6 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY",
const size_t JANAS_N_ENV_NAMES =
sizeof JANAS_ENV_NAMES / sizeof *JANAS_ENV_NAMES;
-/* read by Janas's own scripts, not by its programs */
-static const char *const SCRIPTS[] = {"JANAS_MODELS"};
-
int janas_env_check(const char *prog)
{
int n = 0;
@@ -94,8 +94,6 @@ int janas_env_check(const char *prog)
int known = 0;
for (size_t i = 0; i < JANAS_N_ENV_NAMES && !known; i++)
known = strcmp(name, JANAS_ENV_NAMES[i]) == 0;
- for (size_t i = 0; i < sizeof SCRIPTS / sizeof *SCRIPTS && !known; i++)
- known = strcmp(name, SCRIPTS[i]) == 0;
if (known)
continue;
char why[240];
diff --git a/src/get/catalog.h b/src/get/catalog.h
deleted file mode 100644
index 8de7fc6..0000000
--- a/src/get/catalog.h
+++ /dev/null
@@ -1,44 +0,0 @@
-/* SPDX-License-Identifier: GPL-3.0-or-later
- Copyright (C) 2026 Maurizio Cammalleri */
-/*
- * catalog.h - the models janas-get knows: where each comes from, and the
- * SHA-256 of every file on the way, as MODELS.md lists them.
- */
-#ifndef JANAS_GET_CATALOG_H
-#define JANAS_GET_CATALOG_H
-
-#define GET_MAX_PARTS 4
-
-/* how the model's multi-token prediction file is made, if it has one */
-enum get_extra {
- EXTRA_NONE,
- EXTRA_GGUF, /* a GGUF of its own (Gemma 4's assistant), gguf2jns */
- EXTRA_HF2JNS /* the block of the original checkpoint, hf2jns_mtp hf: */
-};
-
-struct get_model {
- const char *name; /* what janas-get takes */
- const char *what; /* a line for the list */
- const char *repo; /* the GGUF's repository */
- const char *parts[GET_MAX_PARTS]; /* its file(s), the first given to
- gguf2jns */
- const char *part_sha[GET_MAX_PARTS];
- double gguf_gb; /* download */
- const char *out; /* the .jns, beside the downloads */
- const char *flat_sha; /* what gguf2jns writes (the .jns itself when
- there are no planes) */
- const char *jns_sha; /* what jns_planes writes, NULL: one step */
- enum get_extra extra;
- const char *extra_src; /* GGUF file of repo, or hf:repo[@rev] */
- const char *extra_src_sha; /* of that GGUF (EXTRA_GGUF) */
- const char *extra_out;
- const char *extra_sha;
- int ram_gb; /* the smallest machine it is meant for */
-};
-
-extern const struct get_model get_catalog[];
-extern const int get_catalog_n;
-
-const struct get_model *get_find(const char *name);
-
-#endif
diff --git a/src/get/main.c b/src/get/main.c
index b8f413f..683e61a 100644
--- a/src/get/main.c
+++ b/src/get/main.c
@@ -9,6 +9,10 @@
* janas-get <model> [--dir DIR] [--keep] [--no-mtp]
* janas-get hf:<owner>/<repo>/<file.gguf> [--dir DIR] [--keep]
*
+ * The models go where janas-chat looks for them (janas_models_dir: the
+ * environment's JANAS_MODELS, or ~/.local/share/janas/models), unless
+ * --dir says otherwise. A model with a draft in the catalog brings it.
+ *
* A download stopped halfway resumes where it stopped; a file already
* there is checked, not fetched again. The GGUF goes once converted,
* unless --keep.
@@ -19,7 +23,7 @@
#include <sys/stat.h>
#include <sys/statvfs.h>
-#include "catalog.h"
+#include "common/catalog.h"
#include "common/env_names.h"
#include "common/sysinfo.h"
#include "steps.h"
@@ -31,16 +35,17 @@ static void usage(void)
" janas-get <model> [--dir DIR] [--keep] [--no-mtp]\n"
" janas-get hf:<owner>/<repo>/<file.gguf> [--dir DIR] "
"[--keep]\n"
- " --dir DIR where the models go (default: models)\n"
+ " --dir DIR where the models go (default: JANAS_MODELS, or\n"
+ " ~/.local/share/janas/models)\n"
" --keep keep the downloaded GGUF once converted\n"
- " --no-mtp without the model's prediction file\n");
+ " --no-mtp without the model's prediction file or draft\n");
}
static void list(void)
{
printf("%-22s %8s %7s %s\n", "model", "download", "memory", "");
- for (int i = 0; i < get_catalog_n; i++) {
- const struct get_model *m = &get_catalog[i];
+ for (int i = 0; i < janas_catalog_n; i++) {
+ const struct janas_model_entry *m = &janas_catalog[i];
printf("%-22s %6.1f GB %4d GB %s\n", m->name, m->gguf_gb, m->ram_gb,
m->what);
}
@@ -78,8 +83,8 @@ static int convert(const char *first, const char *out)
return get_run("gguf2jns", argv) == 0 ? 0 : -1;
}
-static int from_catalog(const struct get_model *m, const char *dir, int keep,
- int no_mtp)
+static int from_catalog(const struct janas_model_entry *m, const char *dir,
+ int keep, int no_mtp)
{
char out[4096], first[4096], flat[4200];
snprintf(out, sizeof(out), "%s/%s", dir, m->out);
@@ -108,7 +113,7 @@ static int from_catalog(const struct get_model *m, const char *dir, int keep,
return 1;
}
printf("== %s: downloading\n", m->name);
- for (int p = 0; p < GET_MAX_PARTS && m->parts[p]; p++)
+ for (int p = 0; p < JANAS_CATALOG_PARTS && m->parts[p]; p++)
if (get_fetch(m->repo, m->parts[p], dir, m->part_sha[p]) != 0)
return 1;
snprintf(flat, sizeof(flat), "%s.flat", out);
@@ -117,7 +122,7 @@ static int from_catalog(const struct get_model *m, const char *dir, int keep,
check(conv, m->flat_sha, "gguf2jns") != 0)
return 1;
if (!keep)
- for (int p = 0; p < GET_MAX_PARTS && m->parts[p]; p++) {
+ for (int p = 0; p < JANAS_CATALOG_PARTS && m->parts[p]; p++) {
char part[4096];
get_local(part, sizeof(part), dir, m->parts[p]);
remove(part);
@@ -132,12 +137,12 @@ static int from_catalog(const struct get_model *m, const char *dir, int keep,
remove(flat);
}
}
- if (m->extra != EXTRA_NONE && !no_mtp) {
+ if (m->extra != JANAS_EXTRA_NONE && !no_mtp) {
char xout[4096];
snprintf(xout, sizeof(xout), "%s/%s", dir, m->extra_out);
if (exists(xout) && get_same(xout, m->extra_sha) == 1) {
printf("%s: already here, fingerprint checked\n", xout);
- } else if (m->extra == EXTRA_GGUF) {
+ } else if (m->extra == JANAS_EXTRA_GGUF) {
char src[4096];
printf("== its assistant\n");
if (get_fetch(m->repo, m->extra_src, dir, m->extra_src_sha) != 0)
@@ -158,7 +163,6 @@ static int from_catalog(const struct get_model *m, const char *dir, int keep,
return 1;
}
}
- printf("\nReady: %s\nChat with it: janas-chat %s\n", out, out);
return 0;
}
@@ -207,7 +211,8 @@ int main(int argc, char **argv)
usage();
return 2;
}
- const char *dir = "models";
+ static char home_dir[4096];
+ const char *dir = NULL;
int keep = 0, no_mtp = 0;
for (int i = 2; i < argc; i++) {
if (strcmp(argv[i], "--dir") == 0 && i + 1 < argc)
@@ -229,14 +234,34 @@ int main(int argc, char **argv)
usage();
return 0;
}
- mkdir(dir, 0755);
+ if (!dir) {
+ if (janas_models_dir(home_dir, sizeof(home_dir)) != 0) {
+ fprintf(stderr, "janas-get: no home directory: use --dir\n");
+ return 2;
+ }
+ dir = home_dir;
+ }
+ if (janas_mkdirs(dir) != 0) {
+ fprintf(stderr, "janas-get: cannot make %s\n", dir);
+ return 1;
+ }
if (strncmp(argv[1], "hf:", 3) == 0)
return from_hf(argv[1], dir, keep);
- const struct get_model *m = get_find(argv[1]);
+ const struct janas_model_entry *m = janas_catalog_find(argv[1]);
if (!m) {
fprintf(stderr, "janas-get: no model called %s (janas-get list)\n",
argv[1]);
return 2;
}
- return from_catalog(m, dir, keep, no_mtp);
+ int r = from_catalog(m, dir, keep, no_mtp);
+ const struct janas_model_entry *d =
+ m->draft && !no_mtp ? janas_catalog_find(m->draft) : NULL;
+ if (r == 0 && d) {
+ printf("== its draft, %s\n", d->name);
+ r = from_catalog(d, dir, keep, 1);
+ }
+ if (r == 0)
+ printf("\nReady: %s/%s\nChat with it: janas-chat %s\n", dir, m->out,
+ m->name);
+ return r;
}
diff --git a/src/llm/api.c b/src/llm/api.c
index 292f0f9..16b8117 100644
--- a/src/llm/api.c
+++ b/src/llm/api.c
@@ -444,7 +444,8 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
.use_gpu = p.mode != JANAS_LLM_MODE_ECO,
.warm = !p.no_preload,
.expert_bits = p.expert_bits,
- .attn_scores = p.attn_scores};
+ .attn_scores = p.attn_scores,
+ .head_bits = p.head_bits};
llm->m = janas_llm_model_load(path, &mo, err, sizeof(err));
if (!llm->m) {
janas_tokenizer_destroy(llm->tok);
@@ -493,7 +494,7 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
if (warm_n)
snprintf(warm, sizeof(warm), ", %llu experts loading in the background",
(unsigned long long)warm_n);
- char bits[96] = "";
+ char bits[128] = "";
int nb = janas_llm_model_expert_bits(llm->m);
uint32_t used = janas_llm_model_experts(llm->m, NULL);
int w = 0;
@@ -503,8 +504,14 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
: "");
if (janas_llm_model_experts_auto(llm->m) && w >= 0 &&
(size_t)w < sizeof(bits))
+ w += snprintf(bits + w, sizeof(bits) - (size_t)w,
+ ", %u experts per token (memory)", used);
+ uint64_t d0, d1;
+ janas_llm_model_head_requant(llm->m, &d0, &d1);
+ if (d0 && w >= 0 && (size_t)w < sizeof(bits))
snprintf(bits + w, sizeof(bits) - (size_t)w,
- ", %u experts per token (memory)", used);
+ ", output head at 4 bits (%.0f -> %.0f MB)", d0 / 1e6,
+ d1 / 1e6);
llm->cache_bytes = cache;
llm->reserve_bytes = reserve;
llm->kv_token = kv_token;
diff --git a/src/llm/attention.c b/src/llm/attention.c
index 7bf33c4..f3a2044 100644
--- a/src/llm/attention.c
+++ b/src/llm/attention.c
@@ -37,12 +37,13 @@ struct janas_attn {
int16_t *qi; /* q to 16 bits: MAX_BLOCK x n_head x hd */
float *qs; /* its scale, one per token and head */
int n_threads;
- int scores; /* JANAS_ATTN_FLOAT or JANAS_ATTN_INT16, resolved */
- int scores_auto; /* 1: the engine chose it */
- float scale; /* of the scores: 1 / sqrt(head_dim) unless set */
- int kv16; /* keys and values of sixteen bits, not eight */
- int ring; /* position p in row p mod n_ctx (with a window) */
- uint32_t window; /* a token sees the last window positions; 0: all */
+ int scores; /* JANAS_ATTN_FLOAT or JANAS_ATTN_INT16, resolved */
+ int scores_auto; /* 1: the engine chose it */
+ float scale; /* of the scores: 1 / sqrt(head_dim) unless set */
+ int kv16; /* keys and values of sixteen bits, not eight */
+ int ring; /* position p in row p mod n_ctx (with a window) */
+ uint32_t window; /* a token sees the last window positions; 0: all */
+ atomic_uint *done; /* chunks finished, per (KV head, subgroup) */
};
struct attn_job {
@@ -55,6 +56,7 @@ struct attn_job {
float *out;
uint32_t n, pos0, n_chunks, split;
uint32_t c0; /* the first chunk any token of the call sees */
+ int fuse; /* 1: the last chunk of a subgroup merges it */
atomic_uint next; /* next item to claim */
};
@@ -123,6 +125,54 @@ static void att_values_ref(const float *const *w, const int8_t *V, uint32_t nt,
}
}
+/*
+ * The scores' sixteen-bit path weighs the values with sixteen-bit integers
+ * too: a slot's weights (each between 0 and its largest, the value's scale
+ * folded in) to 0 .. 32767 with one scale, rounded to nearest as the query
+ * is, then packed in place two positions a word at the row's start, an odd
+ * last position paired with 0. The scale goes to the row's last float,
+ * which the 128 words never reach. The rounding of lrintf and of
+ * _mm256_cvtps_epi32 is the same (to nearest, the default mode), so both
+ * give the same integers. The error is the query's, a relative 3e-5,
+ * against the 4e-3 of the eight-bit values.
+ */
+static void att_wquant_tail(float *row, uint32_t t, uint32_t nt, float inv)
+{
+ int32_t *wp = (int32_t *)row;
+ for (; t < nt; t += 2) {
+ int32_t lo = (int32_t)lrintf(row[t] * inv);
+ int32_t hi = t + 1 < nt ? (int32_t)lrintf(row[t + 1] * inv) : 0;
+ wp[t / 2] = (int32_t)((uint32_t)lo | (uint32_t)hi << 16);
+ }
+}
+
+static void att_wquant_ref(float *row, uint32_t nt)
+{
+ float mx = 0.0f;
+ for (uint32_t t = 0; t < nt; t++)
+ if (row[t] > mx)
+ mx = row[t];
+ float inv = mx > 0.0f ? 32767.0f / mx : 0.0f;
+ att_wquant_tail(row, 0, nt, inv);
+ row[ATT_CHUNK - 1] = mx / 32767.0f;
+}
+
+static void att_values_i16_ref(const int32_t *const *wp, const float *wsc,
+ const int8_t *V, uint32_t nt, uint32_t hd,
+ float *const *out, int ns)
+{
+ for (int s = 0; s < ns; s++)
+ for (uint32_t i = 0; i < hd; i++) {
+ int32_t acc = 0;
+ for (uint32_t t = 0; t < nt; t++) {
+ int32_t w = (int16_t)(uint16_t)((uint32_t)wp[s][t / 2] >>
+ (16 * (t % 2)));
+ acc += w * V[(size_t)t * hd + i];
+ }
+ out[s][i] = (float)acc * wsc[s];
+ }
+}
+
/* The same over keys and values of sixteen bits. */
static void att_scores_f32_ref16(const float *const *q, const int16_t *K,
const float *ks, uint32_t nt, uint32_t hd,
@@ -153,6 +203,34 @@ static void att_values_ref16(const float *const *w, const int16_t *V,
#if defined(__x86_64__)
#define AVX2 __attribute__((target("avx2,fma,f16c")))
+/* Eight rows' sums reduced together, each the same tree as one row alone
+ below - the halves, then lanes 0 + 2 and 1 + 3, then the two - and
+ scaled: the scores of rows t .. t + 7 of one slot. */
+static inline AVX2 __attribute__((always_inline)) void
+f32_reduce8(const __m256 *acc, float scale, __m256 ksv, __m256i order,
+ float *out)
+{
+ /* each row's halves added: rows r and r + 1 in one register */
+ __m256 h[4];
+ _Pragma("GCC unroll 4") for (int r = 0; r < 4; r++) h[r] =
+ _mm256_add_ps(_mm256_permute2f128_ps(acc[2 * r], acc[2 * r + 1], 0x20),
+ _mm256_permute2f128_ps(acc[2 * r], acc[2 * r + 1], 0x31));
+ /* lanes 0 + 2 and 1 + 3 of each row's four */
+ __m256 g01 =
+ _mm256_add_ps(_mm256_shuffle_ps(h[0], h[1], _MM_SHUFFLE(1, 0, 1, 0)),
+ _mm256_shuffle_ps(h[0], h[1], _MM_SHUFFLE(3, 2, 3, 2)));
+ __m256 g23 =
+ _mm256_add_ps(_mm256_shuffle_ps(h[2], h[3], _MM_SHUFFLE(1, 0, 1, 0)),
+ _mm256_shuffle_ps(h[2], h[3], _MM_SHUFFLE(3, 2, 3, 2)));
+ /* then the two: rows 0 2 4 6 | 1 3 5 7, put in order */
+ __m256 sum =
+ _mm256_add_ps(_mm256_shuffle_ps(g01, g23, _MM_SHUFFLE(2, 0, 2, 0)),
+ _mm256_shuffle_ps(g01, g23, _MM_SHUFFLE(3, 1, 3, 1)));
+ sum = _mm256_permutevar8x32_ps(sum, order);
+ _mm256_storeu_ps(
+ out, _mm256_mul_ps(_mm256_mul_ps(sum, _mm256_set1_ps(scale)), ksv));
+}
+
/* The scores over the query as it is: one key row converted for all the
slots, one FMA per slot and eight weights. */
static inline AVX2 __attribute__((always_inline)) void
@@ -163,12 +241,12 @@ f32_body(const float *const *q, const int8_t *K, const float *ks, uint32_t nt,
uint32_t t = 0;
/*
* Eight rows at a time, from four slots on: the rows converted once into
- * a buffer, then for each slot one accumulator a row and the eight sums
- * reduced together. The reduction of one row alone below cost as many
- * instructions as its products, on the same ports; done for eight it
- * costs a third of one. Each row's sum is the same tree as below - the
- * halves, then lanes 0 + 2 and 1 + 3, then the two - so the scores are
- * the same to the bit.
+ * a buffer, then for each slot one accumulator a row (two slots and four
+ * rows at a time, below) and the eight sums reduced together. The reduction
+ * of one row alone below cost as many instructions as its products, on the
+ * same ports; done for eight it costs a third of one. Each row's sum is the
+ * same tree as below - the halves, then lanes 0 + 2 and 1 + 3, then the two
+ * - so the scores are the same to the bit.
*/
if (ns >= 4 && hd <= 256 && hd % 8 == 0) {
float kb[8 * 256] __attribute__((aligned(32)));
@@ -187,39 +265,37 @@ f32_body(const float *const *q, const int8_t *K, const float *ks, uint32_t nt,
_mm256_cvtepi32_ps(_mm256_cvtepi8_epi32(_mm_loadl_epi64(
(const __m128i *)(K + (size_t)(t + r) * hd + i)))));
const __m256 ksv = _mm256_loadu_ps(ks + t);
- for (int s = 0; s < ns; s++) {
- __m256 acc[8];
- _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++) acc[r] =
- _mm256_setzero_ps();
- for (uint32_t i = 0; i < hd; i += 8) {
- __m256 qv = _mm256_loadu_ps(q[s] + i);
- _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++) acc[r] =
- _mm256_fmadd_ps(qv, _mm256_load_ps(kb + r * hd + i),
- acc[r]);
+ /* two slots at a time, the rows four at a time: one key load
+ serves two FMAs, and eight accumulators, two queries and a
+ key fit in the sixteen registers (5-7% over one slot and
+ eight rows, 7 Oct 2026). Each accumulator sums the same
+ products in the same order. */
+ _Pragma("GCC unroll 4") for (int s = 0; s < ns; s += 2)
+ {
+ __m256 a0[8], a1[8];
+ _Pragma("GCC unroll 2") for (int h = 0; h < 8; h += 4)
+ {
+ __m256 b0[4], b1[4];
+ _Pragma("GCC unroll 4") for (int r = 0; r < 4; r++) b0[r] =
+ b1[r] = _mm256_setzero_ps();
+ for (uint32_t i = 0; i < hd; i += 8) {
+ __m256 q0 = _mm256_loadu_ps(q[s] + i);
+ __m256 q1 = _mm256_loadu_ps(q[s + 1] + i);
+ _Pragma("GCC unroll 4") for (int r = 0; r < 4; r++)
+ {
+ __m256 k = _mm256_load_ps(kb + (h + r) * hd + i);
+ b0[r] = _mm256_fmadd_ps(q0, k, b0[r]);
+ b1[r] = _mm256_fmadd_ps(q1, k, b1[r]);
+ }
+ }
+ _Pragma("GCC unroll 4") for (int r = 0; r < 4; r++)
+ {
+ a0[h + r] = b0[r];
+ a1[h + r] = b1[r];
+ }
}
- /* each row's halves added: rows r and r + 1 in one register */
- __m256 h[4];
- _Pragma("GCC unroll 4") for (int r = 0; r < 4; r++) h[r] =
- _mm256_add_ps(_mm256_permute2f128_ps(acc[2 * r],
- acc[2 * r + 1], 0x20),
- _mm256_permute2f128_ps(acc[2 * r],
- acc[2 * r + 1], 0x31));
- /* lanes 0 + 2 and 1 + 3 of each row's four */
- __m256 g01 = _mm256_add_ps(
- _mm256_shuffle_ps(h[0], h[1], _MM_SHUFFLE(1, 0, 1, 0)),
- _mm256_shuffle_ps(h[0], h[1], _MM_SHUFFLE(3, 2, 3, 2)));
- __m256 g23 = _mm256_add_ps(
- _mm256_shuffle_ps(h[2], h[3], _MM_SHUFFLE(1, 0, 1, 0)),
- _mm256_shuffle_ps(h[2], h[3], _MM_SHUFFLE(3, 2, 3, 2)));
- /* then the two: rows 0 2 4 6 | 1 3 5 7, put in order */
- __m256 sum = _mm256_add_ps(
- _mm256_shuffle_ps(g01, g23, _MM_SHUFFLE(2, 0, 2, 0)),
- _mm256_shuffle_ps(g01, g23, _MM_SHUFFLE(3, 1, 3, 1)));
- sum = _mm256_permutevar8x32_ps(sum, order);
- _mm256_storeu_ps(
- sc[s] + t,
- _mm256_mul_ps(_mm256_mul_ps(sum, _mm256_set1_ps(scale)),
- ksv));
+ f32_reduce8(a0, scale, ksv, order, sc[s] + t);
+ f32_reduce8(a1, scale, ksv, order, sc[s + 1] + t);
}
}
}
@@ -290,6 +366,38 @@ static AVX2 void att_scores_f32_avx2(const float *const *q, const int8_t *K,
#undef SCORES_TARGET
#undef SCORES_ACC
+/* att_wquant_ref sixteen weights a step: the same largest, the same
+ products and roundings, the same words. */
+static AVX2 void att_wquant_avx2(float *row, uint32_t nt)
+{
+ __m256 m8 = _mm256_setzero_ps();
+ uint32_t t = 0;
+ for (; t + 8 <= nt; t += 8)
+ m8 = _mm256_max_ps(m8, _mm256_loadu_ps(row + t));
+ float m[8], mx = 0.0f;
+ _mm256_storeu_ps(m, m8);
+ for (int l = 0; l < 8; l++)
+ if (m[l] > mx)
+ mx = m[l];
+ for (; t < nt; t++)
+ if (row[t] > mx)
+ mx = row[t];
+ float inv = mx > 0.0f ? 32767.0f / mx : 0.0f;
+ const __m256 iv = _mm256_set1_ps(inv);
+ int32_t *wp = (int32_t *)row;
+ /* words 8u .. 8u + 7 from floats 16u .. 16u + 15, read before written */
+ for (t = 0; t + 16 <= nt; t += 16) {
+ __m256i a =
+ _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(row + t), iv));
+ __m256i b =
+ _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(row + t + 8), iv));
+ __m256i p = _mm256_permute4x64_epi64(_mm256_packs_epi32(a, b), 0xd8);
+ _mm256_storeu_si256((__m256i *)(wp + t / 2), p);
+ }
+ att_wquant_tail(row, t, nt, inv);
+ row[ATT_CHUNK - 1] = mx / 32767.0f;
+}
+
static inline AVX2 __attribute__((always_inline)) void
values_body(const float *const *w, const int8_t *V, uint32_t nt, uint32_t hd,
float *const *out, const int ns)
@@ -490,164 +598,175 @@ att_weights_avx2(float *row, uint32_t nt, uint32_t vis, float *mx)
}
#endif
-static void attn_worker(void *arg, int tid, int n_threads)
+/* One item: chunk c of KV head kvh for one subgroup of its query heads. */
+static void attn_item(struct attn_job *a, uint32_t it, float *sc, int avx,
+ int vnni)
{
- (void)n_threads;
- struct attn_job *a = arg;
struct janas_attn *A = a->A;
uint32_t hd = A->hd;
uint32_t group = A->n_head / A->n_kv, gs = group / a->split;
uint32_t span = a->n_chunks - a->c0; /* chunks c0 .. n_chunks - 1 */
- uint32_t items = A->n_kv * span * a->split;
float scale = A->scale;
- float *sc = A->scr + (size_t)tid * JANAS_ATTN_MAX_BLOCK * group * ATT_CHUNK;
-#if defined(__x86_64__)
- int avx = __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma") &&
- __builtin_cpu_supports("f16c");
- int vnni = avx && janas_use_vnni();
+#if !defined(__x86_64__)
+ (void)avx;
+ (void)vnni;
#endif
- for (;;) {
- /* items claimed in order: a thread that finishes early takes more */
- uint32_t it =
- atomic_fetch_add_explicit(&a->next, 1, memory_order_relaxed);
- if (it >= items)
- break;
- uint32_t sub = it % a->split, c = a->c0 + it / a->split % span;
- uint32_t kvh = it / a->split / span, g0 = sub * gs;
- uint32_t t0 = c * ATT_CHUNK, t1 = t0 + ATT_CHUNK;
- if (t1 > a->pos0 + a->n)
- t1 = a->pos0 + a->n;
- uint32_t nt = t1 - t0;
- /* rows of one or two bytes a number; in a ring, the chunk's rows
- are contiguous all the same (n_ctx is a multiple of a chunk) */
- size_t r0 = (size_t)kvh * A->n_ctx + (A->ring ? t0 % A->n_ctx : t0);
- size_t row0 = (r0 * hd) << A->kv16;
- const int8_t *K = a->k + row0;
- const int8_t *V = a->v + row0;
- const float *ks = a->ks + r0;
- const float *vs = a->vs + r0;
- /* token j sees positions up to pos0 + j: tokens before jlo see
- nothing of this chunk */
- uint32_t jlo = t0 > a->pos0 ? t0 - a->pos0 : 0;
- for (uint32_t j = 0; j < jlo && j < a->n; j++)
- for (uint32_t g = g0; g < g0 + gs; g++) {
- float *pm = att_partial(A, kvh, c, j, g);
- pm[0] = -INFINITY;
- pm[1] = 0.0f;
- memset(pm + 2, 0, hd * sizeof(float));
- }
- if (jlo >= a->n)
- continue;
- /* slot = (j - jlo) * gs + g, taken 8, 4, 2 or 1 at a time */
- uint32_t ns = (a->n - jlo) * gs, step;
- for (uint32_t s0 = 0; s0 < ns; s0 += step) {
- step = ns - s0 >= 8 ? 8 : ns - s0 >= 4 ? 4 : ns - s0 >= 2 ? 2 : 1;
- const int16_t *qi[8];
- const float *qf[8];
- float qsc[8], *srow[8];
- int fast = A->scores == JANAS_ATTN_INT16 && !A->kv16;
- for (uint32_t s = 0; s < step; s++) {
- uint32_t sl = s0 + s, j = jlo + sl / gs, g = g0 + sl % gs;
- size_t h = (size_t)j * A->n_head + kvh * group + g;
- if (fast) {
- qi[s] = a->qi + h * hd;
- qsc[s] = a->qs[h];
- } else {
- qf[s] = a->q + h * hd;
- }
- srow[s] = sc + (size_t)sl * ATT_CHUNK;
+ uint32_t sub = it % a->split, c = a->c0 + it / a->split % span;
+ uint32_t kvh = it / a->split / span, g0 = sub * gs;
+ uint32_t t0 = c * ATT_CHUNK, t1 = t0 + ATT_CHUNK;
+ if (t1 > a->pos0 + a->n)
+ t1 = a->pos0 + a->n;
+ uint32_t nt = t1 - t0;
+ /* sixteen-bit weights for the values too (att_wquant_ref) */
+ const int fastv = A->scores == JANAS_ATTN_INT16 && !A->kv16;
+ /* rows of one or two bytes a number; in a ring, the chunk's rows
+ are contiguous all the same (n_ctx is a multiple of a chunk) */
+ size_t r0 = (size_t)kvh * A->n_ctx + (A->ring ? t0 % A->n_ctx : t0);
+ size_t row0 = (r0 * hd) << A->kv16;
+ const int8_t *K = a->k + row0;
+ const int8_t *V = a->v + row0;
+ const float *ks = a->ks + r0;
+ const float *vs = a->vs + r0;
+ /* token j sees positions up to pos0 + j: tokens before jlo see
+ nothing of this chunk */
+ uint32_t jlo = t0 > a->pos0 ? t0 - a->pos0 : 0;
+ for (uint32_t j = 0; j < jlo && j < a->n; j++)
+ for (uint32_t g = g0; g < g0 + gs; g++) {
+ float *pm = att_partial(A, kvh, c, j, g);
+ pm[0] = -INFINITY;
+ pm[1] = 0.0f;
+ memset(pm + 2, 0, hd * sizeof(float));
+ }
+ if (jlo >= a->n)
+ return;
+ /* slot = (j - jlo) * gs + g, taken 8, 4, 2 or 1 at a time */
+ uint32_t ns = (a->n - jlo) * gs, step;
+ for (uint32_t s0 = 0; s0 < ns; s0 += step) {
+ step = ns - s0 >= 8 ? 8 : ns - s0 >= 4 ? 4 : ns - s0 >= 2 ? 2 : 1;
+ const int16_t *qi[8];
+ const float *qf[8];
+ float qsc[8], *srow[8];
+ int fast = A->scores == JANAS_ATTN_INT16 && !A->kv16;
+ for (uint32_t s = 0; s < step; s++) {
+ uint32_t sl = s0 + s, j = jlo + sl / gs, g = g0 + sl % gs;
+ size_t h = (size_t)j * A->n_head + kvh * group + g;
+ if (fast) {
+ qi[s] = a->qi + h * hd;
+ qsc[s] = a->qs[h];
+ } else {
+ qf[s] = a->q + h * hd;
}
- const int8_t *pf = s0 ? NULL : V;
- if (A->kv16) {
+ srow[s] = sc + (size_t)sl * ATT_CHUNK;
+ }
+ const int8_t *pf = s0 ? NULL : V;
+ if (A->kv16) {
#if defined(__x86_64__)
- if (avx)
- att_scores_f32_avx2_16(qf, (const int16_t *)K, ks, nt, hd,
- scale, srow, (const int16_t *)pf,
- (int)step);
- else
+ if (avx)
+ att_scores_f32_avx2_16(qf, (const int16_t *)K, ks, nt, hd,
+ scale, srow, (const int16_t *)pf,
+ (int)step);
+ else
#endif
- att_scores_f32_ref16(qf, (const int16_t *)K, ks, nt, hd,
- scale, srow, (int)step);
- } else if (fast) {
+ att_scores_f32_ref16(qf, (const int16_t *)K, ks, nt, hd, scale,
+ srow, (int)step);
+ } else if (fast) {
#if defined(__x86_64__)
- if (vnni)
- att_scores_i16_vnni(qi, K, ks, nt, hd, qsc, scale, srow, pf,
- (int)step);
- else if (avx)
- att_scores_i16_avx2(qi, K, ks, nt, hd, qsc, scale, srow, pf,
- (int)step);
- else
+ if (vnni)
+ att_scores_i16_vnni(qi, K, ks, nt, hd, qsc, scale, srow, pf,
+ (int)step);
+ else if (avx)
+ att_scores_i16_avx2(qi, K, ks, nt, hd, qsc, scale, srow, pf,
+ (int)step);
+ else
#endif
- att_scores_i16_ref(qi, K, ks, nt, hd, qsc, scale, srow,
- (int)step);
- } else {
+ att_scores_i16_ref(qi, K, ks, nt, hd, qsc, scale, srow,
+ (int)step);
+ } else {
#if defined(__x86_64__)
- if (avx)
- att_scores_f32_avx2(qf, K, ks, nt, hd, scale, srow, pf,
- (int)step);
- else
+ if (avx)
+ att_scores_f32_avx2(qf, K, ks, nt, hd, scale, srow, pf,
+ (int)step);
+ else
#endif
- att_scores_f32_ref(qf, K, ks, nt, hd, scale, srow,
- (int)step);
- }
+ att_scores_f32_ref(qf, K, ks, nt, hd, scale, srow, (int)step);
}
- /* per slot: largest visible score, weights and their sum */
- for (uint32_t sl = 0; sl < ns; sl++) {
- uint32_t j = jlo + sl / gs;
- uint32_t vis = a->pos0 + j + 1 - t0; /* visible positions */
- float *pm = att_partial(A, kvh, c, j, g0 + sl % gs);
- if (A->window && a->pos0 + j + 1 > A->window) {
- /* a sliding window: the positions before its start weigh
- nothing, and a chunk wholly before it nothing at all */
- uint32_t lo = a->pos0 + j + 1 - A->window;
- float *row = sc + (size_t)sl * ATT_CHUNK;
- if (lo >= t1) {
- pm[0] = -INFINITY;
- pm[1] = 0.0f;
- memset(row, 0, nt * sizeof(float));
- continue;
- }
- for (uint32_t t = t0; t < lo; t++)
- row[t - t0] = -INFINITY;
+ }
+ /* per slot: largest visible score, weights and their sum */
+ for (uint32_t sl = 0; sl < ns; sl++) {
+ uint32_t j = jlo + sl / gs;
+ uint32_t vis = a->pos0 + j + 1 - t0; /* visible positions */
+ float *pm = att_partial(A, kvh, c, j, g0 + sl % gs);
+ if (A->window && a->pos0 + j + 1 > A->window) {
+ /* a sliding window: the positions before its start weigh
+ nothing, and a chunk wholly before it nothing at all */
+ uint32_t lo = a->pos0 + j + 1 - A->window;
+ float *row = sc + (size_t)sl * ATT_CHUNK;
+ if (lo >= t1) {
+ pm[0] = -INFINITY;
+ pm[1] = 0.0f;
+ memset(row, 0, nt * sizeof(float));
+ continue;
}
+ for (uint32_t t = t0; t < lo; t++)
+ row[t - t0] = -INFINITY;
+ }
+#if defined(__x86_64__)
+ if (avx)
+ pm[1] = att_weights_avx2(sc + (size_t)sl * ATT_CHUNK, nt,
+ vis < nt ? vis : nt, pm);
+ else
+#endif
+ pm[1] = att_weights_ref(sc + (size_t)sl * ATT_CHUNK, nt,
+ vis < nt ? vis : nt, pm);
+ /* the scale of a position belongs to its value: folded into the
+ weight, it costs one multiply per position instead of one per
+ number read. The sum in pm[1] stays what it was. */
+ float *row = sc + (size_t)sl * ATT_CHUNK;
+ for (uint32_t t = 0; t < nt; t++)
+ row[t] *= vs[t];
+ if (fastv) {
#if defined(__x86_64__)
if (avx)
- pm[1] = att_weights_avx2(sc + (size_t)sl * ATT_CHUNK, nt,
- vis < nt ? vis : nt, pm);
+ att_wquant_avx2(row, nt);
else
#endif
- pm[1] = att_weights_ref(sc + (size_t)sl * ATT_CHUNK, nt,
- vis < nt ? vis : nt, pm);
- /* the scale of a position belongs to its value: folded into the
- weight, it costs one multiply per position instead of one per
- number read. The sum in pm[1] stays what it was. */
- float *row = sc + (size_t)sl * ATT_CHUNK;
- for (uint32_t t = 0; t < nt; t++)
- row[t] *= vs[t];
+ att_wquant_ref(row, nt);
}
- for (uint32_t s0 = 0; s0 < ns; s0 += step) {
- step = ns - s0 >= 8 ? 8 : ns - s0 >= 4 ? 4 : ns - s0 >= 2 ? 2 : 1;
- const float *w[8];
- float *out[8];
- for (uint32_t s = 0; s < step; s++) {
- uint32_t sl = s0 + s;
- w[s] = sc + (size_t)sl * ATT_CHUNK;
- out[s] =
- att_partial(A, kvh, c, jlo + sl / gs, g0 + sl % gs) + 2;
- }
+ }
+ for (uint32_t s0 = 0; s0 < ns; s0 += step) {
+ step = ns - s0 >= 8 ? 8 : ns - s0 >= 4 ? 4 : ns - s0 >= 2 ? 2 : 1;
+ const float *w[8];
+ const int32_t *wp[8];
+ float *out[8], wsc[8];
+ for (uint32_t s = 0; s < step; s++) {
+ uint32_t sl = s0 + s;
+ w[s] = sc + (size_t)sl * ATT_CHUNK;
+ wp[s] = (const int32_t *)w[s];
+ wsc[s] = w[s][ATT_CHUNK - 1];
+ out[s] = att_partial(A, kvh, c, jlo + sl / gs, g0 + sl % gs) + 2;
+ }
+ if (fastv) {
#if defined(__x86_64__)
- if (avx && A->kv16)
- att_values_avx2_16(w, (const int16_t *)V, nt, hd, out,
- (int)step);
+ if (vnni)
+ att_values_i16_vnni(wp, wsc, V, nt, hd, out, (int)step);
else if (avx)
- att_values_avx2(w, V, nt, hd, out, (int)step);
+ att_values_i16_avx2(wp, wsc, V, nt, hd, out, (int)step);
else
#endif
- if (A->kv16)
- att_values_ref16(w, (const int16_t *)V, nt, hd, out, (int)step);
- else
- att_values_ref(w, V, nt, hd, out, (int)step);
+ att_values_i16_ref(wp, wsc, V, nt, hd, out, (int)step);
+ continue;
}
+#if defined(__x86_64__)
+ if (avx && A->kv16)
+ att_values_avx2_16(w, (const int16_t *)V, nt, hd, out, (int)step);
+ else if (avx)
+ att_values_avx2(w, V, nt, hd, out, (int)step);
+ else
+#endif
+ if (A->kv16)
+ att_values_ref16(w, (const int16_t *)V, nt, hd, out, (int)step);
+ else
+ att_values_ref(w, V, nt, hd, out, (int)step);
}
}
@@ -708,50 +827,93 @@ static void skip_report(void)
}
}
-/* Merge of the partials, item = (token j, query head h). */
-static void merge_worker(void *arg, int tid, int n_threads)
+/* Merge of the partials of token j, query head h. */
+static void merge_one(struct attn_job *a, uint32_t j, uint32_t h)
{
- struct attn_job *a = arg;
struct janas_attn *A = a->A;
uint32_t hd = A->hd, qd = A->n_head * hd;
- uint32_t group = A->n_head / A->n_kv, items = a->n * A->n_head;
- for (uint32_t it = (uint32_t)tid; it < items; it += (uint32_t)n_threads) {
- uint32_t j = it / A->n_head, h = it % A->n_head;
- uint32_t kvh = h / group, g = h % group;
- uint32_t nc = (a->pos0 + j + ATT_CHUNK) / ATT_CHUNK;
- float mx = -INFINITY, sum = 0.0f;
+ uint32_t group = A->n_head / A->n_kv;
+ uint32_t kvh = h / group, g = h % group;
+ uint32_t nc = (a->pos0 + j + ATT_CHUNK) / ATT_CHUNK;
+ float mx = -INFINITY, sum = 0.0f;
+ for (uint32_t c = a->c0; c < nc; c++) {
+ float v = att_partial(A, kvh, c, j, g)[0];
+ if (v > mx)
+ mx = v;
+ }
+ if (skip_stat > 0) {
+ uint64_t k[3] = {0};
for (uint32_t c = a->c0; c < nc; c++) {
float v = att_partial(A, kvh, c, j, g)[0];
- if (v > mx)
- mx = v;
+ k[0] += v < mx - 8.0f;
+ k[1] += v < mx - 16.0f;
+ k[2] += v < mx - 32.0f;
}
- if (skip_stat > 0) {
- uint64_t k[3] = {0};
- for (uint32_t c = a->c0; c < nc; c++) {
- float v = att_partial(A, kvh, c, j, g)[0];
- k[0] += v < mx - 8.0f;
- k[1] += v < mx - 16.0f;
- k[2] += v < mx - 32.0f;
- }
- for (int w = 0; w < 2; w++) {
- if (w == 1 && a->pos0 + j < 16384)
- break;
- atomic_fetch_add(&skip_n[w], nc);
- for (int t = 0; t < 3; t++)
- atomic_fetch_add(&skip_k[w][t], k[t]);
- }
+ for (int w = 0; w < 2; w++) {
+ if (w == 1 && a->pos0 + j < 16384)
+ break;
+ atomic_fetch_add(&skip_n[w], nc);
+ for (int t = 0; t < 3; t++)
+ atomic_fetch_add(&skip_k[w][t], k[t]);
}
- float *out = a->out + (size_t)j * qd + (size_t)h * hd;
- memset(out, 0, hd * sizeof(float));
- for (uint32_t c = a->c0; c < nc; c++) {
- const float *pm = att_partial(A, kvh, c, j, g);
- float w = expf(pm[0] - mx);
- sum += w * pm[1];
- janas_axpy_f32(out, w, pm + 2, hd);
- }
- float inv = 1.0f / sum;
- for (uint32_t i = 0; i < hd; i++)
- out[i] *= inv;
+ }
+ float *out = a->out + (size_t)j * qd + (size_t)h * hd;
+ memset(out, 0, hd * sizeof(float));
+ for (uint32_t c = a->c0; c < nc; c++) {
+ const float *pm = att_partial(A, kvh, c, j, g);
+ float w = expf(pm[0] - mx);
+ sum += w * pm[1];
+ janas_axpy_f32(out, w, pm + 2, hd);
+ }
+ float inv = 1.0f / sum;
+ for (uint32_t i = 0; i < hd; i++)
+ out[i] *= inv;
+}
+
+/* Merge of the partials, item = (token j, query head h). */
+static void merge_worker(void *arg, int tid, int n_threads)
+{
+ struct attn_job *a = arg;
+ uint32_t n_head = a->A->n_head, items = a->n * n_head;
+ for (uint32_t it = (uint32_t)tid; it < items; it += (uint32_t)n_threads)
+ merge_one(a, it / n_head, it % n_head);
+}
+
+static void attn_worker(void *arg, int tid, int n_threads)
+{
+ (void)n_threads;
+ struct attn_job *a = arg;
+ struct janas_attn *A = a->A;
+ uint32_t group = A->n_head / A->n_kv, gs = group / a->split;
+ uint32_t span = a->n_chunks - a->c0;
+ uint32_t items = A->n_kv * span * a->split;
+ float *sc = A->scr + (size_t)tid * JANAS_ATTN_MAX_BLOCK * group * ATT_CHUNK;
+ int avx = 0, vnni = 0;
+#if defined(__x86_64__)
+ avx = __builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma") &&
+ __builtin_cpu_supports("f16c");
+ vnni = avx && janas_use_vnni();
+#endif
+ for (;;) {
+ /* items claimed in order: a thread that finishes early takes more */
+ uint32_t it =
+ atomic_fetch_add_explicit(&a->next, 1, memory_order_relaxed);
+ if (it >= items)
+ break;
+ attn_item(a, it, sc, avx, vnni);
+ if (!a->fuse)
+ continue;
+ /* the thread that finishes a subgroup's last chunk merges it while
+ the others still work, and the pool needs no third round */
+ uint32_t sub = it % a->split, kvh = it / a->split / span;
+ if (atomic_fetch_add_explicit(&A->done[kvh * a->split + sub], 1,
+ memory_order_acq_rel) +
+ 1 <
+ span)
+ continue;
+ for (uint32_t j = 0; j < a->n; j++)
+ for (uint32_t g = sub * gs; g < (sub + 1) * gs; g++)
+ merge_one(a, j, kvh * group + g);
}
}
@@ -815,12 +977,13 @@ struct janas_attn *janas_attn_create(uint32_t n_head, uint32_t n_head_kv,
A->part = malloc(janas_attn_part_bytes(n_head, n_head_kv, head_dim, n_ctx));
A->scr = malloc((size_t)n_threads * (JANAS_ATTN_MAX_BLOCK * group + 2) *
ATT_CHUNK * sizeof(float));
+ A->done = malloc(n_head * sizeof(*A->done));
if (A->scores == JANAS_ATTN_INT16) {
A->qi = malloc((size_t)JANAS_ATTN_MAX_BLOCK * n_head * head_dim *
sizeof(int16_t));
A->qs = malloc((size_t)JANAS_ATTN_MAX_BLOCK * n_head * sizeof(float));
}
- if (!A->part || !A->scr ||
+ if (!A->part || !A->scr || !A->done ||
(A->scores == JANAS_ATTN_INT16 && (!A->qi || !A->qs))) {
janas_attn_destroy(A);
return NULL;
@@ -876,6 +1039,7 @@ void janas_attn_destroy(struct janas_attn *A)
free(A->scr);
free(A->qi);
free(A->qs);
+ free(A->done);
free(A);
}
@@ -915,6 +1079,20 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,
.split = split};
job.qi = A->qi;
job.qs = A->qs;
+ /* merged inside the attention round when a subgroup's merge is small
+ (decoding, a few drafts): a long block keeps its own round, spread
+ over all the threads. 30B at 1900 positions: attention 6.9 -> 6.6
+ ms a token (7 Oct 2026). JANAS_ATTN_FUSE=0 never, =N up to N
+ partials a subgroup */
+ static int fuse_max = -1;
+ if (fuse_max < 0) {
+ const char *e = getenv("JANAS_ATTN_FUSE");
+ fuse_max = e ? atoi(e) : 16384;
+ }
+ job.fuse = n * (group / split) * (n_chunks - c0) <= (uint32_t)fuse_max;
+ if (job.fuse)
+ for (uint32_t i = 0; i < A->n_kv * split; i++)
+ atomic_store_explicit(&A->done[i], 0, memory_order_relaxed);
atomic_init(&job.next, 0);
if (skip_stat < 0) {
const char *e = getenv("JANAS_ATTN_SKIPSTAT");
@@ -925,5 +1103,6 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,
if (A->scores == JANAS_ATTN_INT16 && !A->kv16)
janas_pool_run(pool, quant_worker, &job);
janas_pool_run(pool, attn_worker, &job);
- janas_pool_run(pool, merge_worker, &job);
+ if (!job.fuse)
+ janas_pool_run(pool, merge_worker, &job);
}
diff --git a/src/llm/attention.h b/src/llm/attention.h
index 7872075..a5a0a1f 100644
--- a/src/llm/attention.h
+++ b/src/llm/attention.h
@@ -29,7 +29,9 @@
* floating point. INT16 reads it as sixteen-bit integers with a scale of its
* own per token and head: the dot product is then an exact integer sum, which
* the machine does sixteen products at a time instead of eight - about twice
- * as fast - at the cost of the query's quantization, a relative 3e-5. AUTO
+ * as fast - at the cost of the query's quantization, a relative 3e-5; and it
+ * weighs the values with sixteen-bit integers too, a chunk's weights with one
+ * scale (attention.c, att_wquant_ref): a few 1e-4 on an output at most. AUTO
* takes INT16 when the instructions for it are there and the caller asked
* for a context longer than the usual one (more than 16384 positions), where
* attention is the greater part of a token; FLOAT otherwise, since below that
diff --git a/src/llm/attention_scores.h b/src/llm/attention_scores.h
index 146b1fc..b6b10dc 100644
--- a/src/llm/attention_scores.h
+++ b/src/llm/attention_scores.h
@@ -5,7 +5,8 @@
* (private to attention.c, which includes it once per instruction set).
*
* The query is read as 16-bit integers with a scale of its own, the keys are
- * the eight-bit ones of the cache: the dot product is an exact integer sum,
+ * the eight-bit ones of the cache (and, further down, the weights of the
+ * values the same way): the dot product is an exact integer sum,
* and the float arithmetic that follows is one multiply per score. The
* includer defines:
* SCORES(name) the name with the instruction-set suffix
@@ -97,3 +98,60 @@ SCORES(att_scores)(const int16_t *const *q, const int8_t *K, const float *ks,
SCORES(scores_body)(q, K, ks, nt, hd, qsc, scale, sc, pf, 1);
}
}
+
+/*
+ * The values weighted by sixteen-bit weights, for the scores' sixteen-bit
+ * path: wp[s] holds a slot's weights two positions a word (the even one in
+ * the low half), wsc[s] their scale. Two value rows interleaved byte by byte
+ * and widened give each lane a pair of numbers, one per position, which
+ * SCORES_ACC multiplies by the pair of weights and adds: an exact integer
+ * sum per number (at most 256 positions of 32767 x 127: below 2^31), then
+ * one multiply. vpdpwssd does sixteen products an instruction where the
+ * float FMA does eight.
+ */
+static inline SCORES_TARGET __attribute__((always_inline)) void
+SCORES(values_body)(const int32_t *const *wp, const float *wsc, const int8_t *V,
+ uint32_t nt, uint32_t hd, float *const *out, const int ns)
+{
+ const uint32_t np = (nt + 1) / 2;
+ for (uint32_t i = 0; i < hd; i += 8) {
+ __m256i acc[8];
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =
+ _mm256_setzero_si256();
+ for (uint32_t p = 0; p < np; p++) {
+ __m128i r0 = _mm_loadl_epi64(
+ (const __m128i *)(V + (size_t)(2 * p) * hd + i));
+ __m128i r1 =
+ 2 * p + 1 < nt
+ ? _mm_loadl_epi64(
+ (const __m128i *)(V + (size_t)(2 * p + 1) * hd + i))
+ : _mm_setzero_si128();
+ __m256i v = _mm256_cvtepi8_epi16(_mm_unpacklo_epi8(r0, r1));
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =
+ SCORES_ACC(acc[s], v, _mm256_set1_epi32(wp[s][p]));
+ }
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) _mm256_storeu_ps(
+ out[s] + i,
+ _mm256_mul_ps(_mm256_cvtepi32_ps(acc[s]), _mm256_set1_ps(wsc[s])));
+ }
+}
+
+static SCORES_TARGET void SCORES(att_values)(const int32_t *const *wp,
+ const float *wsc, const int8_t *V,
+ uint32_t nt, uint32_t hd,
+ float *const *out, int ns)
+{
+ switch (ns) {
+ case 8:
+ SCORES(values_body)(wp, wsc, V, nt, hd, out, 8);
+ break;
+ case 4:
+ SCORES(values_body)(wp, wsc, V, nt, hd, out, 4);
+ break;
+ case 2:
+ SCORES(values_body)(wp, wsc, V, nt, hd, out, 2);
+ break;
+ default:
+ SCORES(values_body)(wp, wsc, V, nt, hd, out, 1);
+ }
+}
diff --git a/src/llm/chat.h b/src/llm/chat.h
index 56dffd6..58f84d3 100644
--- a/src/llm/chat.h
+++ b/src/llm/chat.h
@@ -57,7 +57,7 @@ struct janas_llm {
whether the GPU is given work is a thing that changes */
uint64_t cache_bytes, reserve_bytes;
int n_compute;
- char arch[32], bits[96], warm[64];
+ char arch[32], bits[128], warm[64];
char name[96];
int32_t mode;
int gpu_wanted; /* the caller's say, on top of what the mode decides */
diff --git a/src/llm/matvec.c b/src/llm/matvec.c
index 635ff80..1031791 100644
--- a/src/llm/matvec.c
+++ b/src/llm/matvec.c
@@ -103,30 +103,44 @@ static void group_worker(void *arg, int tid, int n_threads)
if (g0 >= total)
break;
/* a thread's claims grow, whatever the size: the task index only
- goes forward */
- for (size_t g = g0; g < g1; g++) {
- while (job->first_row[t + 1] <= g)
+ goes forward. A claim may span tasks: each part with its task's
+ sizes worked out once, and on one vector all its rows in one
+ call (janas_dot_rows) */
+ while (g0 < g1) {
+ while (job->first_row[t + 1] <= g0)
t++;
const struct janas_matvec_task *k = &job->tasks[t];
- size_t r = g - job->first_row[t];
+ size_t r0 = g0 - job->first_row[t];
+ size_t end =
+ job->first_row[t + 1] < g1 ? job->first_row[t + 1] : g1;
+ size_t nr = end - g0;
size_t nb = k->cols / JANAS_QK;
size_t row_bytes = nb * janas_qtype_block_size(k->type);
- const void *row = (const char *)k->w + r * row_bytes;
+ const char *w0 = (const char *)k->w + r0 * row_bytes;
size_t nv = k->n_vec ? k->n_vec : 1;
size_t xs = k->x_stride ? k->x_stride : nb;
size_t ys = k->y_stride ? k->y_stride : k->rows;
if (k->type == JANAS_Q6_K_P) {
size_t pb = nb * JANAS_Q6KP_PLANE;
- const uint8_t *p1 =
- k->plane[0] ? (const uint8_t *)k->plane[0] + r * pb : NULL;
- const uint8_t *p0 =
- k->plane[1] ? (const uint8_t *)k->plane[1] + r * pb : NULL;
- janas_q6kp_dot_multi(row, p1, p0, k->x, xs, nv, nb, k->y + r,
- ys);
- } else if (nv == 1)
- k->y[r] = janas_dot(k->type, row, k->x, nb);
- else
- janas_dot_multi(k->type, row, k->x, xs, nv, nb, k->y + r, ys);
+ for (size_t r = r0; r < r0 + nr; r++) {
+ const void *row = (const char *)k->w + r * row_bytes;
+ const uint8_t *p1 =
+ k->plane[0] ? (const uint8_t *)k->plane[0] + r * pb
+ : NULL;
+ const uint8_t *p0 =
+ k->plane[1] ? (const uint8_t *)k->plane[1] + r * pb
+ : NULL;
+ janas_q6kp_dot_multi(row, p1, p0, k->x, xs, nv, nb,
+ k->y + r, ys);
+ }
+ } else if (nv == 1) {
+ janas_dot_rows(k->type, w0, row_bytes, k->x, nb, nr, k->y + r0);
+ } else {
+ for (size_t i = 0; i < nr; i++)
+ janas_dot_multi(k->type, w0 + i * row_bytes, k->x, xs, nv,
+ nb, k->y + r0 + i, ys);
+ }
+ g0 = end;
}
}
}
diff --git a/src/llm/model.h b/src/llm/model.h
index 6c82539..54b118e 100644
--- a/src/llm/model.h
+++ b/src/llm/model.h
@@ -49,6 +49,10 @@ struct janas_llm_options {
choice to the engine: every bit when the experts fit
in cache_bytes, four otherwise (environment
JANAS_EXPERT_BITS overrides) */
+ int head_bits; /* 4: the output head, if the file holds it at six bits
+ (Q6_K), requantized to four (Q4_K) at load - faster, a
+ little less accurate; 0 or 6: as in the file
+ (environment JANAS_HEAD_BITS overrides) */
};
struct janas_llm_model;
@@ -79,6 +83,10 @@ int janas_llm_model_cache_file(const struct janas_llm_model *m,
whether that was chosen by the engine from the memory available. */
int janas_llm_model_expert_bits(const struct janas_llm_model *m);
int janas_llm_model_expert_bits_auto(const struct janas_llm_model *m);
+/* The bytes of the output head requantized at load (head_bits 4), before
+ and after; 0 and 0 if it was not. */
+void janas_llm_model_head_requant(const struct janas_llm_model *m,
+ uint64_t *before, uint64_t *after);
/* How the attention scores are computed (janas_attn_scores), and whether the
engine chose it. */
int janas_llm_model_attn_scores(const struct janas_llm_model *m);
diff --git a/src/llm/model_impl.h b/src/llm/model_impl.h
index 9b4b23e..7d7d8c9 100644
--- a/src/llm/model_impl.h
+++ b/src/llm/model_impl.h
@@ -266,6 +266,11 @@ struct janas_llm_model {
int k_auto; /* and so was the number of experts per token */
int warm_use; /* preload the most used experts, and count them */
uint32_t *use_count; /* experts used this session, n_layer * n_expert */
+ uint64_t head_before, head_after; /* output head requantized (head_bits) */
+ /* the caller's CPUs before the load pinned it (a cpu_set_t), given
+ back when the model is freed */
+ unsigned char caller_cpus[128];
+ int caller_pinned;
janas_llm_route_fn route_fn; /* janas_llm_model_route_observer */
void *route_ctx;
janas_llm_route_fn pred_fn; /* janas_llm_model_route_predictor */
diff --git a/src/llm/model_load.c b/src/llm/model_load.c
index 4dcbc9a..1ff5607 100644
--- a/src/llm/model_load.c
+++ b/src/llm/model_load.c
@@ -8,6 +8,8 @@
#include "model.h"
#include <math.h>
+#include <pthread.h>
+#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -173,6 +175,84 @@ static int rows_f32(struct janas_llm_model *m, const struct janas_jns *j,
return 0;
}
+/*
+ * head_bits 4: the output head, when the file holds it at six bits (Q6_K),
+ * requantized to Q4_K once, at load, before anything takes its type: the
+ * largest matrix read whole at every token, a seventh of Qwen3.5-9B's
+ * token. Into a buffer of its own, then copied where it was: the smaller
+ * copy fits there. Only the head: on Qwen3.5-9B, 400 tokens of Italian
+ * prose against llama.cpp, the most likely token agreed 375 times where
+ * 383 with the head requantized, 368 with every other six-bit matrix -
+ * attention, the shared experts - which gave next to no speed (7 Oct 2026).
+ * Where the head is the token embeddings (Qwen3-0.6B and 4B, Qwen3.5-0.8B
+ * and 2B) the table is requantized in place, so the embeddings read at four
+ * bits too, as in the larger models' files: Qwen3.5-2B +9%, the most
+ * likely token 352 where 376 (8 Oct 2026).
+ */
+struct rq_job {
+ const uint8_t *src;
+ uint8_t *dst;
+ size_t first, count; /* blocks of JANAS_QK weights */
+};
+
+static void *rq_worker(void *arg)
+{
+ struct rq_job *r = arg;
+ float buf[JANAS_QK * 16];
+ size_t sb = janas_qtype_block_size(JANAS_Q6_K),
+ db = janas_qtype_block_size(JANAS_Q4_K);
+ for (size_t b = r->first; b < r->first + r->count; b += 16) {
+ size_t nb = r->first + r->count - b < 16 ? r->first + r->count - b : 16;
+ janas_dequantize(JANAS_Q6_K, r->src + b * sb, buf, nb * JANAS_QK);
+ janas_q4k_quantize_fast(
+ buf, (struct janas_block_q4k *)(r->dst + b * db), nb * JANAS_QK);
+ }
+ return NULL;
+}
+
+static void requant_head(struct janas_llm_model *m)
+{
+ /* tied embeddings (the small Qwen): the table is the head */
+ struct janas_jns_tensor *t =
+ (struct janas_jns_tensor *)janas_jns_tensor(&m->j, "output.weight");
+ if (!t)
+ t = (struct janas_jns_tensor *)janas_jns_tensor(&m->j,
+ "token_embd.weight");
+ if (!t || t->type != JANAS_Q6_K || t->dims[0] % JANAS_QK ||
+ t->offset < m->j.h.resident_offset ||
+ t->offset + t->bytes > m->j.h.resident_offset + m->j.h.resident_bytes)
+ return;
+ enum { NT = 16 };
+ long cpus = sysconf(_SC_NPROCESSORS_ONLN);
+ int nt = cpus < 1 ? 1 : cpus > NT ? NT : (int)cpus;
+ size_t nb = t->bytes / janas_qtype_block_size(JANAS_Q6_K);
+ uint8_t *src = (uint8_t *)data(m, t);
+ uint8_t *dst = malloc(nb * janas_qtype_block_size(JANAS_Q4_K));
+ if (!dst)
+ return;
+ pthread_t th[NT];
+ struct rq_job jb[NT];
+ int started = 0;
+ for (int k = 0; k < nt; k++) {
+ size_t a = nb * k / nt, b = nb * (k + 1) / nt;
+ jb[k] =
+ (struct rq_job){.src = src, .dst = dst, .first = a, .count = b - a};
+ if (pthread_create(&th[k], NULL, rq_worker, &jb[k]) == 0)
+ started |= 1 << k;
+ else
+ rq_worker(&jb[k]);
+ }
+ for (int k = 0; k < nt; k++)
+ if (started & 1 << k)
+ pthread_join(th[k], NULL);
+ m->head_before = t->bytes;
+ t->type = JANAS_Q4_K;
+ t->bytes = nb * janas_qtype_block_size(JANAS_Q4_K);
+ memcpy(src, dst, t->bytes);
+ m->head_after = t->bytes;
+ free(dst);
+}
+
static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,
uint32_t l, struct layer *ly, char *err, size_t err_len)
{
@@ -637,6 +717,9 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
}
done += (uint64_t)r;
}
+ const char *hbe = getenv("JANAS_HEAD_BITS");
+ if ((hbe ? atoi(hbe) : o->head_bits) == 4 && !m->resident_gpu)
+ requant_head(m);
uint32_t dm = m->d_model, qd = m->n_head * m->hd_max, kvd = m->kvd_max,
ff = m->d_ff;
@@ -994,8 +1077,15 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
err_len) != 0 ||
janas_m_assist_head(m, dvocab) != 0))
goto fail;
- if (pin >= 0)
+ /* the caller pinned for the time the model lives, and given its CPUs
+ back when it goes: a model opened after another in the same process
+ (the chat measuring, a reload) would otherwise see one CPU */
+ _Static_assert(sizeof(cpu_set_t) <= sizeof(m->caller_cpus), "cpu set");
+ if (pin >= 0) {
+ m->caller_pinned = sched_getaffinity(0, sizeof(cpu_set_t),
+ (cpu_set_t *)m->caller_cpus) == 0;
janas_pin_current_thread(pin);
+ }
m->cache = janas_expert_cache_create(&m->j, o->cache_bytes, m->exp_level,
m->io, err, err_len);
if (!m->cache)
@@ -1099,6 +1189,8 @@ void janas_llm_model_free(struct janas_llm_model *m)
janas_m_exp_report();
if (!m)
return;
+ if (m->caller_pinned)
+ sched_setaffinity(0, sizeof(cpu_set_t), (cpu_set_t *)m->caller_cpus);
if (m->tune_key[0])
janas_tuner_save(&m->tuner, m->tune_key);
if (m->use_count) {
@@ -1277,6 +1369,13 @@ int janas_llm_model_expert_bits_auto(const struct janas_llm_model *m)
return m->exp_auto;
}
+void janas_llm_model_head_requant(const struct janas_llm_model *m,
+ uint64_t *before, uint64_t *after)
+{
+ *before = m->head_before;
+ *after = m->head_after;
+}
+
int janas_llm_model_experts_auto(const struct janas_llm_model *m)
{
return m->k_auto;
diff --git a/src/llm/quant.c b/src/llm/quant.c
index 83ee20c..17d3e64 100644
--- a/src/llm/quant.c
+++ b/src/llm/quant.c
@@ -319,10 +319,12 @@ void janas_q4k_dequantize(const struct janas_block_q4k *x, float *y, size_t n)
/*
* Affine fit of one sub-block: x ~ a * q - b with q in 0..15, a >= 0 and
* b >= 0 (Q4_K stores the offset as a subtracted min). Candidate ranges
- * shrink the data range from either end; for each, q is rounded and (a, b)
- * refit by least squares. Returns the squared error of the best candidate.
+ * shrink the data range from either end by 0 to shrink 64ths; for each, q
+ * is rounded and (a, b) refit by least squares. Returns the squared error
+ * of the best candidate.
*/
-static double q4k_fit_sub(const float *x, int n, float *a_out, float *b_out)
+static double q4k_fit_sub(const float *x, int n, int shrink, float *a_out,
+ float *b_out)
{
float lo = 0.0f, hi = x[0];
for (int i = 0; i < n; i++) {
@@ -339,8 +341,8 @@ static double q4k_fit_sub(const float *x, int n, float *a_out, float *b_out)
return 0.0;
double best = INFINITY;
float step = (hi - lo) / 64.0f;
- for (int i = 0; i <= 6; i++)
- for (int j = 0; j <= 6; j++) {
+ for (int i = 0; i <= shrink; i++)
+ for (int j = 0; j <= shrink; j++) {
float l = lo + i * step, h = hi - j * step;
if (h <= l)
continue;
@@ -382,12 +384,13 @@ static double q4k_fit_sub(const float *x, int n, float *a_out, float *b_out)
return best;
}
-void janas_q4k_quantize(const float *x, struct janas_block_q4k *y, size_t n)
+static void q4k_quantize(const float *x, struct janas_block_q4k *y, size_t n,
+ int shrink)
{
for (size_t b = 0; b < n / JANAS_QK; b++, x += JANAS_QK) {
float a[8], mn[8], amax = 0.0f, bmax = 0.0f;
for (int s = 0; s < 8; s++) {
- q4k_fit_sub(x + 32 * s, 32, &a[s], &mn[s]);
+ q4k_fit_sub(x + 32 * s, 32, shrink, &a[s], &mn[s]);
if (a[s] > amax)
amax = a[s];
if (mn[s] > bmax)
@@ -459,6 +462,17 @@ void janas_q4k_quantize(const float *x, struct janas_block_q4k *y, size_t n)
}
}
+void janas_q4k_quantize(const float *x, struct janas_block_q4k *y, size_t n)
+{
+ q4k_quantize(x, y, n, 6);
+}
+
+void janas_q4k_quantize_fast(const float *x, struct janas_block_q4k *y,
+ size_t n)
+{
+ q4k_quantize(x, y, n, 0);
+}
+
float janas_q4k_dot_ref(const struct janas_block_q4k *w,
const struct janas_block_q8k *x, size_t nb)
{
@@ -1428,6 +1442,61 @@ void janas_axpy_f32(float *y, float a, const float *x, size_t n)
axpy_f32_ref(y, a, x, n);
}
+void janas_dot_rows(int type, const void *w, size_t row_bytes,
+ const struct janas_block_q8k *x, size_t nb, size_t rows,
+ float *y)
+{
+#if defined(__x86_64__)
+ if (use_avx2_fma && __builtin_cpu_supports("f16c")) {
+ typedef void (*rows_fn)(const void *, size_t,
+ const struct janas_block_q8k *, size_t, size_t,
+ float *);
+ rows_fn f = NULL;
+ switch (type) {
+ case JANAS_Q4_K:
+ f = use_vnni ? q4k_rows_vnni : q4k_rows_avx2;
+ break;
+ case JANAS_Q6_K:
+ f = use_vnni ? q6k_rows_vnni : q6k_rows_avx2;
+ break;
+ case JANAS_Q5_K:
+ f = use_vnni ? q5k_rows_vnni : q5k_rows_avx2;
+ break;
+ case JANAS_Q8_0:
+ f = use_vnni ? q8_0_rows_vnni : q8_0_rows_avx2;
+ break;
+ case JANAS_IQ4_NL:
+ f = use_vnni ? iq4nl_rows_vnni : iq4nl_rows_avx2;
+ break;
+ case JANAS_IQ4_XS:
+ f = use_vnni ? iq4xs_rows_vnni : iq4xs_rows_avx2;
+ break;
+ case JANAS_Q4_0:
+ f = use_vnni ? q4_0_rows_vnni : q4_0_rows_avx2;
+ break;
+ case JANAS_Q5_1:
+ f = use_vnni ? q5_1_rows_vnni : q5_1_rows_avx2;
+ break;
+ case JANAS_Q4_1:
+ f = use_vnni ? q4_1_rows_vnni : q4_1_rows_avx2;
+ break;
+ case JANAS_Q3_K:
+ f = use_vnni ? q3k_rows_vnni : q3k_rows_avx2;
+ break;
+ case JANAS_IQ3_S:
+ f = use_vnni ? iq3s_rows_vnni : iq3s_rows_avx2;
+ break;
+ }
+ if (f) {
+ f(w, row_bytes, x, nb, rows, y);
+ return;
+ }
+ }
+#endif
+ for (size_t r = 0; r < rows; r++)
+ y[r] = janas_dot(type, (const char *)w + r * row_bytes, x, nb);
+}
+
void janas_dot_multi(int type, const void *w, const struct janas_block_q8k *x,
size_t x_stride, size_t nv, size_t nb, float *out,
size_t out_stride)
diff --git a/src/llm/quant.h b/src/llm/quant.h
index cc8b658..62cfddc 100644
--- a/src/llm/quant.h
+++ b/src/llm/quant.h
@@ -223,6 +223,15 @@ float janas_dot(int type, const void *w, const struct janas_block_q8k *x,
*/
void janas_dequantize(int type, const void *w, float *y, size_t n);
+/*
+ * The dot products of `rows` consecutive weight rows (row r at w + r *
+ * row_bytes) with one vector: y[r]. The same bits as janas_dot row by row,
+ * with the type chosen once.
+ */
+void janas_dot_rows(int type, const void *w, size_t row_bytes,
+ const struct janas_block_q8k *x, size_t nb, size_t rows,
+ float *y);
+
/*
* The dot products of one weight row with nv vectors (vector v at x + v *
* x_stride blocks), result v at out[v * out_stride]. Bit-identical to calling
@@ -254,6 +263,12 @@ void janas_q8k_quantize_det(const float *x, struct janas_block_q8k *y,
* 6-bit sub-block scales. For offline conversion, not for the hot path.
*/
void janas_q4k_quantize(const float *x, struct janas_block_q4k *y, size_t n);
+/* The same with each sub-block's range taken as it is, not searched: nine
+ times faster, for requantizing at load (Qwen3.5-9B's output head, a
+ billion weights: 2.5 s against 22 s to open), and on that head the mean
+ logit difference against llama.cpp 0.157 where 0.154 (7 Oct 2026). */
+void janas_q4k_quantize_fast(const float *x, struct janas_block_q4k *y,
+ size_t n);
/* Dequantizes n weights (n multiple of JANAS_QK). */
void janas_q4k_dequantize(const struct janas_block_q4k *x, float *y, size_t n);
diff --git a/src/llm/quant_dotn.h b/src/llm/quant_dotn.h
index a0228a2..a351865 100644
--- a/src/llm/quant_dotn.h
+++ b/src/llm/quant_dotn.h
@@ -38,6 +38,28 @@ DOTN(hsum128_epi32)(__m128i s)
return _mm_cvtsi128_si32(s);
}
+/*
+ * The weights asked for ahead of the product, PF_AHEAD bytes on, into L1:
+ * one prefetch per cache line of what a step of the loop reads, on one or
+ * two vectors (on more the product is bound by its arithmetic, and the
+ * extra instructions cost a little). The hardware prefetcher alone kept
+ * too few lines in flight: on 12 threads Q4_K read 70 GB/s where Q8_0,
+ * which asks for its bytes faster, read 84; with 2 KB ahead Q4_K read 82
+ * (1, 4 and 8 KB gave 81-82, and 4-8 KB slowed a single core down: 7 Oct
+ * 2026). Rows follow one another in memory, so past a row's end the lines
+ * are the next row's.
+ */
+#ifndef PF_AHEAD
+#define PF_AHEAD 2048
+#endif
+static inline DOTN_TARGET __attribute__((always_inline)) void
+DOTN(prefetch_ahead)(const void *at, size_t size)
+{
+ const char *p = (const char *)at + PF_AHEAD;
+ for (size_t i = 0; i < size; i += 64)
+ _mm_prefetch(p + i, _MM_HINT_T0);
+}
+
/*
* Q4_K: the per-block integer-sum scheme. For each block and vector the
* integer sums are exact: isum = sum of scale x (weight . activation) over
@@ -60,6 +82,8 @@ DOTN(q4k_dotn_body)(const struct janas_block_q4k *w,
accm4n = _mm_setzero_ps();
__m256 acc8 = _mm256_setzero_ps(), accm8 = _mm256_setzero_ps();
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
uint32_t u[4];
unpack_scales_u32(w[b].scales, u);
__m128i sm = _mm_loadu_si128((const __m128i *)u);
@@ -192,6 +216,8 @@ DOTN(q5k_dotn_body)(const struct janas_block_q5k *w,
__m128 acc4 = _mm_setzero_ps();
__m256 acc8 = _mm256_setzero_ps(), accm8 = _mm256_setzero_ps();
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
uint32_t u[4];
unpack_scales_u32(w[b].scales, u);
__m128i sm = _mm_loadu_si128((const __m128i *)u);
@@ -321,6 +347,8 @@ DOTN(q8_0_dotn_body)(const struct janas_block_q8_0 *w,
_Pragma("GCC unroll 8") for (int v = 0; v < nv; v++) acc[v] =
_mm256_setzero_ps();
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(w + 8 * b, 8 * sizeof(*w));
const struct janas_block_q8_0 *wb = w + 8 * b;
__m256 dk = _mm256_cvtph_ps(_mm_setr_epi16(
(short)wb[0].d, (short)wb[1].d, (short)wb[2].d, (short)wb[3].d,
@@ -390,6 +418,8 @@ DOTN(q6k_dotn_body)(const struct janas_block_q6k *w,
__m128 acc4 = _mm_setzero_ps();
__m256 acc8 = _mm256_setzero_ps();
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
const int8_t *sc = w[b].scales;
/* the sixteen scales once, as int16, and the two halves spread over
both lanes so that one shuffle builds a quarter's pair */
@@ -690,6 +720,8 @@ DOTN(nib_dotn_body)(const struct janas_block_iq4nl *w,
_Pragma("GCC unroll 8") for (int v = 0; v < nv; v++) acc[v] =
_mm256_setzero_ps();
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(w + 8 * b, 8 * sizeof(*w));
const struct janas_block_iq4nl *wb = w + 8 * b;
__m256 dk = _mm256_cvtph_ps(_mm_setr_epi16(
(short)wb[0].d, (short)wb[1].d, (short)wb[2].d, (short)wb[3].d,
@@ -747,6 +779,8 @@ DOTN(iq4xs_dotn_body)(const struct janas_block_iq4xs *w,
const __m256i levels = DOTN(iq4_levels)();
float acc[8] = {0};
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
__m256i q[8], aq[8], sc[8];
_Pragma("GCC unroll 8") for (int g = 0; g < 8; g++)
{
@@ -893,6 +927,15 @@ DOTN(q6kp_dotn_body)(const struct janas_block_q6kp *w, const uint8_t *p1,
__m128 acc4 = _mm_setzero_ps();
__m256 acc8 = _mm256_setzero_ps();
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2) {
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
+ if (level >= 2)
+ DOTN(prefetch_ahead)(p1 + b * JANAS_Q6KP_PLANE,
+ JANAS_Q6KP_PLANE);
+ if (level >= 3)
+ DOTN(prefetch_ahead)(p0 + b * JANAS_Q6KP_PLANE,
+ JANAS_Q6KP_PLANE);
+ }
const int8_t *sc = w[b].scales;
/* the sixteen scales once, as int16, and the two halves spread over
both lanes so that one shuffle builds a quarter's pair */
@@ -1141,6 +1184,8 @@ DOTN(dm_dotn_body)(const uint8_t *w, const size_t size, const int five,
accm[v] = _mm256_setzero_ps();
}
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(w + 8 * b * size, 8 * size);
const uint8_t *wb = w + 8 * b * size;
uint16_t d[8], m[8];
_Pragma("GCC unroll 8") for (int k = 0; k < 8; k++)
@@ -1314,6 +1359,8 @@ DOTN(q3k_dotn_body)(const struct janas_block_q3k *w,
const __m256i three = _mm256_set1_epi8(3), four = _mm256_set1_epi8(4);
float acc[8] = {0};
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
__m256i u[8], sc16[8];
__m256i hm = _mm256_loadu_si256((const __m256i *)w[b].hmask);
_Pragma("GCC unroll 2") for (int n = 0; n < 2; n++)
@@ -1424,6 +1471,8 @@ DOTN(iq3s_dotn_body)(const struct janas_block_iq3s *w,
const __m256i one = _mm256_set1_epi8(1);
float acc[8] = {0};
for (size_t b = 0; b < nb; b++) {
+ if (nv <= 2)
+ DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));
__m256i lv[8], sg[8], sc[8];
_Pragma("GCC unroll 8") for (int g = 0; g < 8; g++)
{
@@ -1501,3 +1550,34 @@ static DOTN_TARGET float DOTN(iq3s_dot)(const struct janas_block_iq3s *w,
DOTN(iq3s_dotn_body)(w, x, 0, nb, &r, 1);
return r;
}
+
+/*
+ * Rows after rows of one type, one vector: the loop of a matrix-vector
+ * product here, beside the kernels, so that a row costs one direct call.
+ * Called row by row through janas_dot, each row chose its type, went
+ * through a wrapper and a function pointer, and matvec.c worked out its
+ * task's sizes again: 6% of the cycles of a token of Qwen3-30B-A3B (7 Oct
+ * 2026), whose rows are short. Each row is the same function as before, so
+ * the same bits.
+ */
+#define DOTN_ROWS(name, T) \
+ static DOTN_TARGET void DOTN(name##_rows)( \
+ const void *w, size_t row_bytes, const struct janas_block_q8k *x, \
+ size_t nb, size_t rows, float *y) \
+ { \
+ for (size_t r = 0; r < rows; r++) \
+ y[r] = DOTN(name##_dot)( \
+ (const T *)((const char *)w + r * row_bytes), x, nb); \
+ }
+DOTN_ROWS(q4k, struct janas_block_q4k)
+DOTN_ROWS(q6k, struct janas_block_q6k)
+DOTN_ROWS(q5k, struct janas_block_q5k)
+DOTN_ROWS(q8_0, struct janas_block_q8_0)
+DOTN_ROWS(iq4nl, struct janas_block_iq4nl)
+DOTN_ROWS(iq4xs, struct janas_block_iq4xs)
+DOTN_ROWS(q4_0, struct janas_block_q4_0)
+DOTN_ROWS(q5_1, struct janas_block_q5_1)
+DOTN_ROWS(q4_1, struct janas_block_q4_1)
+DOTN_ROWS(q3k, struct janas_block_q3k)
+DOTN_ROWS(iq3s, struct janas_block_iq3s)
+#undef DOTN_ROWS
diff --git a/tests/test_attention.c b/tests/test_attention.c
index 7aad910..25f9a7c 100644
--- a/tests/test_attention.c
+++ b/tests/test_attention.c
@@ -5,11 +5,17 @@
* softmax over the same eight-bit keys and values, and blocks against
* token-by-token calls (bit for bit).
*
- * The engine reads the query as sixteen-bit integers, so the scores rest on
- * an exact integer sum and the only error left is that quantization: a
- * relative 3e-5, which on these outputs shows as about 7e-5 absolute. The
- * eight-bit keys and values the reference shares already carry a relative
- * 4e-3, sixty times more, so the threshold below is set at 1e-4.
+ * The exact path sums in floating point: its threshold is 1e-5. The fast
+ * one reads the query as sixteen-bit integers, so the scores rest on an
+ * exact integer sum and the error there is that quantization, a relative
+ * 3e-5; and it weighs the values with sixteen-bit integers too (7 Oct 2026),
+ * whose rounding, half a step of the largest weight of a chunk, adds up over
+ * its positions: with the random scores here, whose weights crowd on a few
+ * positions, the worst output is 2-6e-4 off. The eight-bit keys and values
+ * the reference shares already carry a relative 4e-3, so the threshold of
+ * the fast path is 1e-3; on Qwen3-30B-A3B against llama.cpp (400 tokens of
+ * C) the fast path gave a mean logit difference of 0.2865 against 0.2893
+ * with float weights and 0.2871 exact.
*
* The checksum printed at the end covers every output of the run: the AVX2
* and the AVX-VNNI paths must print the same one (JANAS_KERNELS=avx2 asks
@@ -134,10 +140,10 @@ static void run(struct janas_pool *pool, uint32_t n_head, uint32_t n_kv,
"heads %u/%u, pos0 %u, n %u: block differs from token by "
"token",
n_head, n_kv, pos0, n);
- /* the exact query keeps the float sum's own error; the sixteen-bit one
- adds its quantization, a relative 3e-5 on the scores */
+ /* the exact path keeps the float sum's own error; the fast one adds the
+ sixteen-bit query's and weights' quantization (see the top) */
int fast = janas_attn_scores_mode(a) == JANAS_ATTN_INT16;
- double limit = fast ? 1e-4 : 1e-5;
+ double limit = fast ? 1e-3 : 1e-5;
CHECK(worst < limit, "heads %u/%u, pos0 %u, n %u, %s: error %g", n_head,
n_kv, pos0, n, fast ? "fast" : "exact", worst);
printf("heads %2u/%u dim %3u, positions %4u..%4u, %-5s: error %.2e, "
diff --git a/tests/test_get_catalog.c b/tests/test_get_catalog.c
index 69e98cf..5656f9c 100644
--- a/tests/test_get_catalog.c
+++ b/tests/test_get_catalog.c
@@ -11,7 +11,7 @@
#include <stdlib.h>
#include <string.h>
-#include "../src/get/catalog.c"
+#include "common/catalog.h"
static int failures;
@@ -58,26 +58,33 @@ int main(void)
"repository's root)\n");
return 1;
}
- for (int i = 0; i < get_catalog_n; i++) {
- const struct get_model *m = &get_catalog[i];
- for (int p = 0; p < GET_MAX_PARTS && m->parts[p]; p++)
+ for (int i = 0; i < janas_catalog_n; i++) {
+ const struct janas_model_entry *m = &janas_catalog[i];
+ for (int p = 0; p < JANAS_CATALOG_PARTS && m->parts[p]; p++)
digest(models, m->name, m->part_sha[p]);
digest(models, m->name, m->flat_sha);
digest(models, m->name, m->jns_sha);
digest(models, m->name, m->extra_src_sha);
digest(models, m->name, m->extra_sha);
- if (!m->flat_sha || (m->extra != EXTRA_NONE && !m->extra_sha)) {
+ if (!m->flat_sha || (m->extra != JANAS_EXTRA_NONE && !m->extra_sha)) {
printf("FAIL %s: a fingerprint missing\n", m->name);
failures++;
}
+ const struct janas_model_entry *d =
+ m->draft ? janas_catalog_find(m->draft) : NULL;
+ if (m->draft && (!d || d->not_chat || d == m)) {
+ printf("FAIL %s: draft %s is not a chat model of the catalog\n",
+ m->name, m->draft);
+ failures++;
+ }
for (int k = 0; k < i; k++)
- if (strcmp(get_catalog[k].name, m->name) == 0) {
+ if (strcmp(janas_catalog[k].name, m->name) == 0) {
printf("FAIL %s: twice in the catalog\n", m->name);
failures++;
}
}
free(models);
printf("test_get_catalog: %s (%d models)\n", failures ? "FAILED" : "ok",
- get_catalog_n);
+ janas_catalog_n);
return failures ? 1 : 0;
}
diff --git a/tests/test_quant.c b/tests/test_quant.c
index 760aba5..1056354 100644
--- a/tests/test_quant.c
+++ b/tests/test_quant.c
@@ -371,14 +371,16 @@ static void test_matvec_multi(void)
/*
* Q4_K quantization: round trip on Gaussian-like and heavy-tailed weights.
* The error relative to the weights' spread must stay near what 4 bits with
- * 6-bit sub-block scales allow, and below a plain min/max rounding.
+ * 6-bit sub-block scales allow, and below a plain min/max rounding - also
+ * for the fast way, which takes each sub-block's range as it is.
*/
static void test_quantize_q4k(void)
{
enum { N = 256 * 64 };
float *x = malloc(N * sizeof(float)), *y = malloc(N * sizeof(float));
struct janas_block_q4k *q = malloc(N / JANAS_QK * sizeof(*q));
- for (int kind = 0; kind < 2; kind++) {
+ for (int k2 = 0; k2 < 4; k2++) {
+ int kind = k2 & 1, fast = k2 >> 1;
double var = 0, err = 0, naive = 0;
for (int i = 0; i < N; i++) {
/* sum of uniforms ~ Gaussian; its cube is heavy-tailed */
@@ -386,7 +388,10 @@ static void test_quantize_q4k(void)
x[i] = 0.02f * (kind ? g * g * g : g);
var += (double)x[i] * x[i];
}
- janas_q4k_quantize(x, q, N);
+ if (fast)
+ janas_q4k_quantize_fast(x, q, N);
+ else
+ janas_q4k_quantize(x, q, N);
janas_q4k_dequantize(q, y, N);
for (int i = 0; i < N; i++)
err += ((double)y[i] - x[i]) * (y[i] - x[i]);
@@ -404,12 +409,13 @@ static void test_quantize_q4k(void)
}
}
double rel = sqrt(err / var), rel_naive = sqrt(naive / var);
- printf("q4k quantize, %s weights: relative rms error %.4f (min/max "
+ printf("q4k quantize%s, %s weights: relative rms error %.4f (min/max "
"rounding %.4f)\n",
- kind ? "heavy-tailed" : "Gaussian", rel, rel_naive);
+ fast ? " fast" : "", kind ? "heavy-tailed" : "Gaussian", rel,
+ rel_naive);
CHECK(rel < (kind ? 0.20 : 0.11) && rel < rel_naive,
- "q4k quantize %d: relative error %.4f, naive %.4f", kind, rel,
- rel_naive);
+ "q4k quantize %d%s: relative error %.4f, naive %.4f", kind,
+ fast ? " fast" : "", rel, rel_naive);
}
free(x);
free(y);