Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-27 (2): the README and docs/features.md with Gemma 4 whole on the GPU, Llama 3 and janas-nsm; Llama 3.2 (Meta) and DeepNSM's adapter (CC BY-NC-SA) among the models' credits

camauri4d492fc02c09
diff --git a/README.md b/README.md
index b2e36ce..b409077 100644
--- a/README.md
+++ b/README.md
@@ -52,6 +52,8 @@ Janas-LLM's figures are the best of the configurations `janas-bench` measures, a
The longer the context, the wider the gap: 2.2 times as fast writing at 1,024 tokens, 6.9 times at 16,384. Both slow down as the context grows, and on both the attention is what grows: at 32,000 tokens it takes 59 of the 93 ms Janas-LLM spends on a token written, and the 32,000 tokens of prompt take 15 minutes to read. Janas-LLM's peak memory went from 2.6 GiB at 1,024 tokens to 4.9 at 32,000. The conditions are the same wherever possible; where not: Janas-LLM keeps the keys and values at eight bits, llama.cpp at sixteen (its default); llama.cpp was not run at 32,000 tokens (almost another hour). How it was measured: [below](#how-the-figures-were-measured).
+**New: Gemma 4 whole on the GPU, Llama 3, and `janas-nsm`.** Gemma 4's four models - 12B, the 26B mixture, E2B and E4B - now run every layer of a token on the GPU, with the GPU's norms and quantizations giving the CPU's bits: Gemma-4-12B agrees with llama.cpp at 367 of 400 most likely tokens on the GPU, at 362 on the CPU. Llama 3 models run too, a LoRA adapter merged into one by `tools/hf2gguf`: DeepNSM-1B (Llama-3.2-1B with the adapter of [arXiv 2505.11764](https://arxiv.org/abs/2505.11764)) writes its answers at 57.8 tok/s on the laptop's CPU, where llama.cpp writes 37.9 to 39.8 on the same file. `janas-nsm` asks it for a word's explication in the Natural Semantic Metalanguage - Goddard and Wierzbicka's sixty-odd semantic primes, the words every language has - and counts it as the paper does; it can also restrict the model's output to those words alone. A research tool, and a sign of how far this project means to go: [docs/tools.md](docs/tools.md#janas-nsm).
+
**Try it** on Linux x86-64 (Ubuntu 22.04, Debian 12 and later):
```sh
@@ -223,7 +225,7 @@ Janas is designed and written by **Maurizio "camauri" Cammalleri** ([LinkedIn](h
It stands on the work of others, and says so:
-- **The models** it runs are their authors': Qwen3, Qwen3.5, Qwen3.6 and Qwen3-Next by the Qwen team of Alibaba Cloud, Gemma 4 by Google; the GGUF quantizations by Qwen, bartowski and unsloth ([MODELS.md](MODELS.md) has every source).
+- **The models** it runs are their authors': Qwen3, Qwen3.5, Qwen3.6 and Qwen3-Next by the Qwen team of Alibaba Cloud, Gemma 4 by Google, Llama 3.2 by Meta (under the Llama 3.2 Community License); DeepNSM's adapter by the authors of [arXiv 2505.11764](https://arxiv.org/abs/2505.11764) (CC BY-NC-SA: research, not commercial use); the GGUF quantizations by Qwen, bartowski and unsloth ([MODELS.md](MODELS.md) has every source).
- **llama.cpp and ggml** are the reference the engine is checked against (logits, tokens, speed); none of their code is in Janas. The one thing taken from ggml is data that defines a format: the IQ3_S grid, in `src/llm/iq3s_grid.h`, under ggml's MIT licence.
- **The Unicode Character Database** gives the tokenizer its tables (`tools/gen_unicode`), and **OpenAI's OpenAPI specification** gives `janas-server` its route table (`tools/openapi_routes`).
- **GNU libmicrohttpd, yyjson and Mbed TLS** are compiled into the programs named below.
diff --git a/docs/features.md b/docs/features.md
index 865272a..d7e5f6b 100644
--- a/docs/features.md
+++ b/docs/features.md
@@ -1,7 +1,8 @@
# What works now
- **Models:** Qwen3.5 and Qwen3.6 (`qwen35moe`, e.g. Qwen3.6-35B-A3B), Qwen3-Next (`qwen3next`, e.g. Qwen3-Next-80B-A3B-Instruct and Qwen3-Coder-Next), Qwen3 MoE (`qwen3moe`, e.g. Qwen3-30B-A3B), the **dense** Qwen3.5 models (`qwen35`) and the **dense** Qwen3 models (`qwen3`), from the usual **Q4_K_M** GGUF files, or **Q3_K**, **IQ3_S**, **IQ4_NL**, **IQ4_XS**, **Q4_0**, **Q4_1** and **Q5_1** ones, and unsloth's "UD" mixes of them (checked on Qwen3.5-0.8B and 9B against llama.cpp), and the "MXFP4_MOE" files, whose experts are MXFP4 (checked on Qwen3.6-35B-A3B against llama.cpp on the same file: 199 of 200 most likely tokens agree). A dense feed-forward is a mixture with one expert and no router, so the same path serves both and there is no second engine to keep in step. Of the dense ones **Qwen3.5-0.8B, 2B and 9B, Qwen3-4B and Qwen3-0.6B are the ones that have been run here**; the Qwen3.5 three were checked against llama.cpp on the same files. **Nearly everything here was measured on Qwen3-Next-80B-A3B-Instruct**: it is the one the engine was shaped around, and the one whose numbers are quoted below. The others have been converted, checked against a reference and used, but far less.
-- **Gemma 4** (`gemma4`): **Gemma-4-12B-it** (dense) and **Gemma-4-26B-A4B-it** (a mixture of 128 experts beside a dense feed-forward) convert and chat, with their reasoning on or off (`--think`); the prompts are, token for token, the ones the model's own chat template writes, and a full context slides as it does for the other models. Its sliding-window layers keep only the last positions they can see, so a long context costs mostly the full layers: for the 12B about 16 KB a token and half a gigabyte fixed, instead of 175 KB a token. Keys and values are kept at sixteen bits (Gemma is far more sensitive to them than Qwen). Tools work in Gemma's own notation - declarations, calls and the tools' answers inside the model's turn, token for token as its template writes them. The small **Gemma-4-E2B-it** and **E4B-it** run too, with chat and tools: their per-layer embeddings - a table as large as half of E2B's file - are read from the file a row at a time and never loaded, so E2B ran here in 1.5 GB of memory (a 3.1 GB file) and E4B in 3.1 GB (5.0 GB). Each of the four has an **assistant**, a small drafting model of its own that reads the main model's keys and values, taken as its `--mtp` file: on an Italian chat reply of 110 tokens, greedy, it took E2B from 34.7 tokens/s to 51.0, E4B from 18.3 to 26.3, the 12B from 8.5 to 12.9 and the 26B from 16.9 to 24.5 on the development laptop, the same tokens with and without it. Against llama.cpp on 200 tokens of prose the most likely token agrees at 160 positions on the 12B (mean logit difference 0.84) and at 160 on the 26B (0.97); on another 200-token text at 191 on E2B (0.45) and 195 on E4B (0.28). Against an exact float64 computation over the same weights (a NumPy forward pass written to check them), on the prose: the 12B agrees at 167 positions (llama.cpp 173), the 26B at 156 (llama.cpp 160), with mean errors of 0.75 and 0.87 (llama.cpp 0.67 and 0.90). The larger two turn small numerical differences into large ones, and llama.cpp is not a steady reference for them: against itself with an eight-bit cache it agrees at 153 positions on the 12B, and with a 32-bit cache and no flash attention, at 112. The reference has to be computed in batches of at most 32 tokens: with larger ones llama.cpp's logits for the early positions change (it looks ahead), for a reason not yet found.
+- **Gemma 4** (`gemma4`): **Gemma-4-12B-it** (dense) and **Gemma-4-26B-A4B-it** (a mixture of 128 experts beside a dense feed-forward) convert and chat, with their reasoning on or off (`--think`); the prompts are, token for token, the ones the model's own chat template writes, and a full context slides as it does for the other models. Its sliding-window layers keep only the last positions they can see, so a long context costs mostly the full layers: for the 12B about 16 KB a token and half a gigabyte fixed, instead of 175 KB a token. Keys and values are kept at sixteen bits (Gemma is far more sensitive to them than Qwen). Tools work in Gemma's own notation - declarations, calls and the tools' answers inside the model's turn, token for token as its template writes them. The small **Gemma-4-E2B-it** and **E4B-it** run too, with chat and tools: their per-layer embeddings - a table as large as half of E2B's file - are read from the file a row at a time and never loaded, so E2B ran here in 1.5 GB of memory (a 3.1 GB file) and E4B in 3.1 GB (5.0 GB). Each of the four has an **assistant**, a small drafting model of its own that reads the main model's keys and values, taken as its `--mtp` file: on an Italian chat reply of 110 tokens, greedy, it took E2B from 34.7 tokens/s to 51.0, E4B from 18.3 to 26.3, the 12B from 8.5 to 12.9 and the 26B from 16.9 to 24.5 on the development laptop, the same tokens with and without it. Against llama.cpp on 200 tokens of prose the most likely token agrees at 160 positions on the 12B (mean logit difference 0.84) and at 160 on the 26B (0.97); on another 200-token text at 191 on E2B (0.45) and 195 on E4B (0.28). Against an exact float64 computation over the same weights (a NumPy forward pass written to check them), on the prose: the 12B agrees at 167 positions (llama.cpp 173), the 26B at 156 (llama.cpp 160), with mean errors of 0.75 and 0.87 (llama.cpp 0.67 and 0.90). The larger two turn small numerical differences into large ones, and llama.cpp is not a steady reference for them: against itself with an eight-bit cache it agrees at 153 positions on the 12B, and with a 32-bit cache and no flash attention, at 112. The reference has to be computed in batches of at most 32 tokens: with larger ones llama.cpp's logits for the early positions change (it looks ahead), for a reason not yet found. Since 26 October 2026 all four run **whole on the GPU** too - the sliding-window layers' rings, the full layers' heads of 512, the sixteen-bit keys and values, the sandwich norms, the per-layer embeddings' branch, the 26B's two branches and its Q5_1 experts - with the GPU's norms and quantizations giving the CPU's bits: against llama.cpp on 400 tokens the 12B agrees at 367 positions on the GPU (the CPU 362), the 26B at 321 (321), E2B at 386 (391) and E4B at 392 (394).
+- **Llama 3** (`llama`, since 25 October 2026): Llama 3 checkpoints from Hugging Face, with a PEFT LoRA adapter merged into the weights if one is given, written as a GGUF by `tools/hf2gguf` that llama.cpp's own tools quantize and `gguf2jns` converts. Checked on DeepNSM-1B (Llama-3.2-1B with the adapter of arXiv 2505.11764): the merged weights the same to the bit as an independent merge, the tokens the same as llama.cpp's, the most likely token as llama.cpp's at 466 of 477 positions in Q8_0; on the CPU and the GPU.
- **Qwen3.8-Flash-Next** (`qwen4exp`, since 22 October 2026): a mixture of 512 experts and 177 billion parameters, from ISTA-DASLab's two-bit GGUF (GSQ-RCO, Q2_0, 66.4 GB), on the CPU, the GPU and the NPU. Its residual runs in four streams (hyper-connections), its n-gram embeddings are read from the file a row at a time, and past 2,051 positions its attention is sparse, an indexer choosing the blocks each token reads. Against llama.cpp on the same file, 2,200 tokens of prose: the most likely token agrees at 2,171 positions (mean logit difference 0.134), and at all 148 positions read by the sparse attention. On 8,192 tokens, its last 512 positions (an Italian table of links and numbers) agree at 482, mean difference 0.191, and the same 512 tokens read after only 512 others agree at 482 too (0.178): the long context costs no agreement. Its multi-token prediction block comes from unsloth's GGUF of the block ([models](models.md)): its draft is the model's own next token at 130 of 158 positions of code, and on a chat reply on the CPU 29 of 40 drafts were kept, the text the same with drafts and without. The chat writes, token for token, the prompt the model's own template writes - the reasoning effort's words at the top of the system message, the earlier replies with their reasoning - and calls tools in its XML notation. `JANAS_HC_MIX=q8` rewrites its mixes of the streams - 1.26 GB of BF16 matrices read at every token, a third of what a token reads besides the experts - as Q8_0 when the model opens, half the bytes, their inputs kept at sixteen bits: against llama.cpp on 400 tokens of C the most likely token agreed at 392 positions (BF16: 390), on 400 of Italian prose at 364 (367), the mean logit difference 0.138 and 0.173 (0.135, 0.166); with the inputs quantized to eight bits as the quantized products take them, 388 and 359, and weights at four bits, 382 and 349. Those products run on the CPU, and in a prompt's blocks on the GPU (below). On the CPU (Core Ultra 9 185H, 12 threads, a 12 GiB expert cache, 1,024 tokens of prose and then 200) a reply runs at 11.9 tokens/s with them and at 10.95 with the BF16 mixes, the prompt at 24.0 either way; the default stays BF16. `--head 4` (`JANAS_HEAD_BITS=4`) takes its output head from Q5_K to Q4_K, 437 MB a token to 358, at a price in quality as on the other models: 387 and 351 positions (0.184, 0.234). A prompt's blocks run whole on a GPU that shares the host's memory (`JANAS_GPU_MOE=1` where the cache holds fewer than half the experts), its sparse attention too, choosing the same blocks as the CPU: on the last 512 of 8,192 tokens the most likely token agreed with llama.cpp at 481 positions (the CPU 482). Against llama.cpp, blocks of 256 tokens of C agree at 247 of 256 positions (the GPU's products beside the CPU: 248), of Italian prose at 230 (235 to 238; the CPU alone 235) with the lowest divergence of the three (KL 0.0172, against 0.0176 and 0.0186); its speed is not measured yet. The weights are under the Qwen Community License 1.0 ([MODELS.md](../MODELS.md#what-this-project-redistributes)); `janas-get qwen3.8-flash-next` downloads it converted, with its MTP block, from [`prabanta-dev/Qwen3.8-Flash-Next-JNS`](https://huggingface.co/prabanta-dev/Qwen3.8-Flash-Next-JNS) (68.4 GB, under the same licence); with `--from-gguf` it downloads ISTA's GGUF and converts it.
- **Reasoning models:** the reasoning is shown apart, in grey, and can be turned off.
- **`janas-chat`:** a terminal chat with a line editor of its own, history, colours, a status line and a context that slides instead of ending.