37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
What works now
- Models: Qwen3.5 and Qwen3.6 (
qwen35moe, e.g. Qwen3.6-35B-A3B), Qwen3-Next (qwen3next, e.g. Qwen3-Next-80B-A3B-Instruct and Qwen3-Coder-Next), Qwen3 MoE (qwen3moe, e.g. Qwen3-30B-A3B), the dense Qwen3.5 models (qwen35) and the dense Qwen3 models (qwen3), from the usual Q4_K_M GGUF files, or Q3_K, IQ3_S, IQ4_NL, IQ4_XS, Q4_0, Q4_1 and Q5_1 ones, and unsloth's "UD" mixes of them (checked on Qwen3.5-0.8B and 9B against llama.cpp), and the "MXFP4_MOE" files, whose experts are MXFP4 (checked on Qwen3.6-35B-A3B against llama.cpp on the same file: 199 of 200 most likely tokens agree). A dense feed-forward is a mixture with one expert and no router, so the same path serves both and there is no second engine to keep in step. Of the dense ones Qwen3.5-0.8B, 2B and 9B, Qwen3-4B and Qwen3-0.6B are the ones that have been run here; the Qwen3.5 three were checked against llama.cpp on the same files. Nearly everything here was measured on Qwen3-Next-80B-A3B-Instruct: it is the one the engine was shaped around, and the one whose numbers are quoted below. The others have been converted, checked against a reference and used, but far less. - Gemma 4 (
gemma4): Gemma-4-12B-it (dense) and Gemma-4-26B-A4B-it (a mixture of 128 experts beside a dense feed-forward) convert and chat, with their reasoning on or off (--think); the prompts are, token for token, the ones the model's own chat template writes, and a full context slides as it does for the other models. Its sliding-window layers keep only the last positions they can see, so a long context costs mostly the full layers: for the 12B about 16 KB a token and half a gigabyte fixed, instead of 175 KB a token. Keys and values are kept at sixteen bits (Gemma is far more sensitive to them than Qwen). Tools work in Gemma's own notation - declarations, calls and the tools' answers inside the model's turn, token for token as its template writes them. The small Gemma-4-E2B-it and E4B-it run too, with chat and tools: their per-layer embeddings - a table as large as half of E2B's file - are read from the file a row at a time and never loaded, so E2B ran here in 1.5 GB of memory (a 3.1 GB file) and E4B in 3.1 GB (5.0 GB). Each of the four has an assistant, a small drafting model of its own that reads the main model's keys and values, taken as its--mtpfile: on an Italian chat reply of 110 tokens, greedy, it took E2B from 34.7 tokens/s to 51.0, E4B from 18.3 to 26.3, the 12B from 8.5 to 12.9 and the 26B from 16.9 to 24.5 on the development laptop, the same tokens with and without it. Against llama.cpp on 200 tokens of prose the most likely token agrees at 160 positions on the 12B (mean logit difference 0.84) and at 160 on the 26B (0.97); on another 200-token text at 191 on E2B (0.45) and 195 on E4B (0.28). Against an exact float64 computation over the same weights (a NumPy forward pass written to check them), on the prose: the 12B agrees at 167 positions (llama.cpp 173), the 26B at 156 (llama.cpp 160), with mean errors of 0.75 and 0.87 (llama.cpp 0.67 and 0.90). The larger two turn small numerical differences into large ones, and llama.cpp is not a steady reference for them: against itself with an eight-bit cache it agrees at 153 positions on the 12B, and with a 32-bit cache and no flash attention, at 112. The reference has to be computed in batches of at most 32 tokens: with larger ones llama.cpp's logits for the early positions change (it looks ahead), for a reason not yet found. Since 26 October 2026 all four run whole on the GPU too - the sliding-window layers' rings, the full layers' heads of 512, the sixteen-bit keys and values, the sandwich norms, the per-layer embeddings' branch, the 26B's two branches and its Q5_1 experts - with the GPU's norms and quantizations giving the CPU's bits: against llama.cpp on 400 tokens the 12B agrees at 367 positions on the GPU (the CPU 362), the 26B at 321 (321), E2B at 386 (391) and E4B at 392 (394). - Llama 3 (
llama, since 25 October 2026): Llama 3 checkpoints from Hugging Face, with a PEFT LoRA adapter merged into the weights if one is given, written as a GGUF bytools/hf2ggufthat llama.cpp's own tools quantize andgguf2jnsconverts. Checked on DeepNSM-1B (Llama-3.2-1B with the adapter of arXiv 2505.11764): the merged weights the same to the bit as an independent merge, the tokens the same as llama.cpp's, the most likely token as llama.cpp's at 466 of 477 positions in Q8_0; on the CPU and the GPU. - NVIDIA Nemotron 3.5 Bolt (
nemotron_h_moe, since 28 October 2026, on thenemotronbranch until it is merged): Nemotron-3.5-Bolt-74B-A9B from NVIDIA's own 24GB GGUF (22.0 GB: its experts in IQ2_XXS, trained at two bits, the Mamba layers in NVFP4, the embedding and the output head in Q8_0). Its 88 layers each do one thing with a norm and a residual of their own: 40 Mamba-2 mixers, 40 mixtures and 8 attentions without RoPE. The mixtures route by sigmoids with a bias for the choice, each layer with its own number of experts (up to 496, 9 a token), and their experts work in a latent space of 1,024 (the input brought down before them, their weighted sum brought up after), relu^2 without a gate, beside a shared expert. Against llama.cpp on the same file, 400 tokens of C: the most likely token agrees at 395 positions (mean logit difference 0.183); its tokenizer (Mistral's Tekken) gives llama.cpp's tokens on seven texts. On the development laptop's CPU, the same file and 12 threads, it writes at 7.0 tokens/s where llama.cpp writes at 5.92, and reads a prompt of 512 tokens at 15.1 tokens/s (16.5 without the first read of the experts from disk) where llama.cpp reads at 16.7; with all 20 usable threads (janas-bench) 20.2 and 8.3. A prompt's blocks give the GPU its products, NVFP4 included (the same bits as the CPU), and the NPU the products of the weights that stay in memory (on the laptop a little slower than the CPU: the engine leaves it off there); its whole pass on the GPU is the next step. - Qwen3.8-Flash-Next (
qwen4exp, since 22 October 2026): a mixture of 512 experts and 177 billion parameters, from ISTA-DASLab's two-bit GGUF (GSQ-RCO, Q2_0, 66.4 GB), on the CPU, the GPU and the NPU. Its residual runs in four streams (hyper-connections), its n-gram embeddings are read from the file a row at a time, and past 2,051 positions its attention is sparse, an indexer choosing the blocks each token reads. Against llama.cpp on the same file, 2,200 tokens of prose: the most likely token agrees at 2,171 positions (mean logit difference 0.134), and at all 148 positions read by the sparse attention. On 8,192 tokens, its last 512 positions (an Italian table of links and numbers) agree at 482, mean difference 0.191, and the same 512 tokens read after only 512 others agree at 482 too (0.178): the long context costs no agreement. Its multi-token prediction block comes from unsloth's GGUF of the block (models): its draft is the model's own next token at 130 of 158 positions of code, and on a chat reply on the CPU 29 of 40 drafts were kept, the text the same with drafts and without. The chat writes, token for token, the prompt the model's own template writes - the reasoning effort's words at the top of the system message, the earlier replies with their reasoning - and calls tools in its XML notation.JANAS_HC_MIX=q8rewrites its mixes of the streams - 1.26 GB of BF16 matrices read at every token, a third of what a token reads besides the experts - as Q8_0 when the model opens, half the bytes, their inputs kept at sixteen bits: against llama.cpp on 400 tokens of C the most likely token agreed at 392 positions (BF16: 390), on 400 of Italian prose at 364 (367), the mean logit difference 0.138 and 0.173 (0.135, 0.166); with the inputs quantized to eight bits as the quantized products take them, 388 and 359, and weights at four bits, 382 and 349. Those products run on the CPU, and in a prompt's blocks on the GPU (below). On the CPU (Core Ultra 9 185H, 12 threads, a 12 GiB expert cache, 1,024 tokens of prose and then 200) a reply runs at 11.9 tokens/s with them and at 10.95 with the BF16 mixes, the prompt at 24.0 either way; the default stays BF16.--head 4(JANAS_HEAD_BITS=4) takes its output head from Q5_K to Q4_K, 437 MB a token to 358, at a price in quality as on the other models: 387 and 351 positions (0.184, 0.234). A prompt's blocks run whole on a GPU that shares the host's memory (JANAS_GPU_MOE=1where the cache holds fewer than half the experts), its sparse attention too, choosing the same blocks as the CPU: on the last 512 of 8,192 tokens the most likely token agreed with llama.cpp at 481 positions (the CPU 482). Against llama.cpp, blocks of 256 tokens of C agree at 247 of 256 positions (the GPU's products beside the CPU: 248), of Italian prose at 230 (235 to 238; the CPU alone 235) with the lowest divergence of the three (KL 0.0172, against 0.0176 and 0.0186); its speed is not measured yet. The weights are under the Qwen Community License 1.0 (MODELS.md);janas-get qwen3.8-flash-nextdownloads it converted, with its MTP block, fromprabanta-dev/Qwen3.8-Flash-Next-JNS(68.4 GB, under the same licence); with--from-ggufit downloads ISTA's GGUF and converts it. - Reasoning models: the reasoning is shown apart, in grey, and can be turned off.
janas-chat: a terminal chat with a line editor of its own, history, colours, a status line and a context that slides instead of ending.libjanas_llm: a C library (include/janas/llm.h) and a FreeBASIC binding (include/janas/llm.bi).- Tools from MCP servers in
janas-chat: the servers configured as the other clients configure them, each call shown and confirmed before it runs (below). The client is a library of its own,libjanas_mcp(include/janas/mcp.h,include/janas/mcp.bi). janas-mcp: the other way round, the model offered as tools to MCP clients - a question to it, the embedding of a text (below).janas-flights: where a flight is and what flies over a sea or a place, as MCP tools, from the ADS-B receivers of the community, and today's schedules with a free AviationStack key: the first of Janas's services.janas-weather: the weather now, with what the nearest station measured, the forecast, the sea and its waves, and the warnings in force (in Italy the Civil Protection's levels), from free sources with no key.janas-maps: roads and places: a route by car, bike or on foot, step by step in your language, journeys by public transport, what is near (fuel, a pharmacy, a cash machine...), where an address is; OpenStreetMap's data through free services with no key.janas-prices: prices and finance, for information: currencies at the ECB's rates, crypto-assets, shares (with a free key of yours), inflation, central banks' rates, electricity where it is open data, the cheapest fuel near a place in Italy, France, Spain and Austria.janas-github: a project on GitHub: its issues and pull requests (the new ones since you last asked), its latest releases, its recent commits, what it is; it only reads, with the gh command's token when you have one.janas-git: the git repositories of your computer: what has changed, the commits, the branches, a diff; and a commit, pull, push or switch of branch, each shown whole and asked of you first; nothing that throws work away.janas-quakes: the earthquakes: the latest near a place or where you are, the strongest in the world, one in detail, from INGV (Italy and its seas) and the USGS.janas-system: this computer: how it is (memory, processor, disks, battery, temperatures), what fills a directory, the programs running, the errors of its logs, the updates, the network; it only reads, and none of its tools reaches the network.janas-wiki: Wikipedia when you ask for it: its search, a page shown to you as it is (the introduction, then the section asked for) while the model reads only a note of it, and the pages about what lies around a place.- The services, each with its page - what it answers, its sources, its keys, how to use it in Claude Code and other clients: src/services.
janas-server: the model behind an HTTP API that follows OpenAI's, so that clients written for it can use a model running on this machine: chat and completions with tools, JSON output held to a schema and log-probabilities, the Responses API with its conversations, embeddings and moderations, a chat page of its own to try it from a browser (below), and code completion for VSCodium and VS Code (docs/code-completion.md).janas-bench: measures the machine, fills its profile and writes a report.- Faster replies from drafts the model verifies, so the text is exactly the one it would have written: from the model's own multi-token prediction block where it has one, from the conversation where it does not, or from a small model of the same family given with
--draft. - Optional GPU (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. A reply's tokens are identical with and without it; a prompt is read with the fast prompt (next).
- The fast prompt, on by default since 10 October 2026: a prompt's blocks go to the GPU's tiled products (Q4_K, Q5_K, Q6_K, Q8_0, Q3_K, IQ4_XS, IQ3_S, IQ4_NL, MXFP4, the experts of a mixture in one launch per layer - Q4_K, Q6_K, MXFP4 and, since 21 October 2026, Q3_K - f32 and bf16 weights), and on the dense Qwen3 and Qwen3.5 models the whole block - every layer, attention, Qwen3.5's recurrent layers - goes to the GPU in one submission while the CPU rests, and on the Qwen3 and Qwen3.6 mixtures a layer a submission, the CPU only sorting the router's choice and reading the next layer's experts ahead (where the expert cache holds half the experts or more;
JANAS_GPU_MOE=0or1decides instead): on a laptop the CPU working beside the GPU takes the power the GPU would use. They round the sums group by group, so a prompt is not read to the bit as the CPU reads it. Against the references the most likely token agreed with the exact way's within 2-3 of 256 positions on the models measured on 10 October 2026, the mean difference of the scores within 2%; on 21 October, Qwen3-30B-A3B against llama.cpp's scores of Qwen's Q8_0, on Italian prose and code, within 5 either way (unsloth's 3-bit file 216 and 246 against the exact way's 221 and 245, Qwen's Q4_K_M 210 and 239 against 207 and 244) and the mean difference within 3%: 8 to 27 of a text's 256 positions change their most likely token with the rounding, and they fall on either side. On the development laptop Qwen3-4B read a prompt of 256 tokens at 220 tok/s with it and at 85 without the GPU (10 October 2026), and on 21 October every model compared read a prompt faster than llama.cpp did on the same GPU (the table).JANAS_GPU_FAST_PROMPT=0reads them the exact way. On a mixture larger than its expert cache the fast prompt is also not repeatable: a group of experts goes to the GPU's rounded products only when all of them are in the memory given to the GPU, which changes as the cache fills, so the same prompt read twice can have some groups rounded and others exact, and a reply that follows it can differ from one run to the next (Qwen3.8-Flash-Next on the development laptop, 22 October 2026: the reply to the same question changed after a dozen tokens).JANAS_GPU_REPEATABLE=1keeps every product of a mixture's experts on the CPU, exact, and in a prompt's blocks gives the GPU only the fast prompt's products: the same request then gives the same reply from one run to the next, to the last digit of every token's probability, and the same reply with drafts and without (Qwen3.8-Flash-Next on the development laptop, 22 October 2026), at the price of the experts' share of the GPU's speed (not measured yet). The difference, in short: by default, the fastest prompt, and on a mixture larger than its cache replies that can vary slightly from run to run, with drafts or without them; withJANAS_GPU_REPEATABLE=1, the same reply every run, the experts computed on the CPU; withJANAS_GPU_FAST_PROMPT=0, the same results as the CPU alone, to the bit. On the CPU alone drafts never change the text. Since that day the chains of drafts the GPU verifies also stay off the experts' rounded products in one launch (until then a chain of 13 tokens or more on a mixture could take them). - Intel's NPU (the Core Ultra's, on Linux; since 21 October 2026) reads a prompt's blocks of 64 tokens and more. A dense model whose layers fit in its memory, nine in ten at least, goes to it layer by layer, whole - Qwen3, Qwen3.5 and Gemma 4's dense models - and other models give it the products of the weights that stay in memory; a reply's tokens stay with the CPU. Janas-LLM writes the NPU's graphs itself, in OpenVINO's IR format, and opens Intel's Level Zero driver when a model opens: nothing is linked against it. The engine takes the NPU by itself where it reads faster and its answers measured as close to llama.cpp's as the CPU's (the Qwen models, and Gemma 4's whose layers go whole); where the expert cache cannot hold all of a model's feed-forward weights (the larger mixtures, Qwen3.8-27B) it stays off unless asked, its copies' memory being the cache's. On the development laptop, 1,024 tokens, with it and without: Qwen3-0.6B 1,421 and 974 tok/s, Qwen3.5-0.8B 1,403 and 950, Qwen3.5-2B 729 and 477, Qwen3-4B 333 and 201, Gemma-4-E2B 508 and 196, Gemma-4-E4B 245 and 129; on the mixtures and the dense models too large for it the GPU's fast prompt reads faster there. Its first blocks in a session wait while it is made ready, 7.5 to 53 s, and the chat's status line says so. How it works and what it needs: the README.
- What runs where, by hand:
--prompt-on,--drafts-onand--reply-on(auto,cpu,gpu,npu,npu+gpu, or a list the engine chooses from) injanas-chat,janas-serverandjanas-bench(docs/chat.md). - Experts read ahead when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even.
JANAS_PREFETCH_PREFILL=0turns it off. - Light experts not in memory are left out when the expert cache holds less than a fifth of the experts: an expert the router weighs below 0.8/k of a token (0.95/k under a tenth), and that would have to be read from disk, is not read and not computed, and the token's other experts take its share — the cache-conditional routing of Skliar et al. (TMLR 2025). Until 8 October 2026 the engine dropped the lightest experts whether in memory or not (six of ten, or eight). Qwen3-Next-80B, four bits, 400 tokens against llama.cpp, most likely token agreeing and tokens/s: with a 2 GiB cache all ten 393 at 11.0, six 383 at 20.9, now 380 at 26.4; with 4 GiB 393 at 20.7, six 383 at 27.6, now 384 at 27.7, with the mean logit difference 0.36 against 0.53; on Italian prose with 8 GiB 382 at 18.7, eight 361 at 21.4, now 372 at 23.7. On Qwen3-30B-A3B and Qwen3.6-35B-A3B with caches of 9% and 18% of their experts (1.5-3.5 GiB), on code and Italian prose, it was faster than the old way every time, by 9-37% (Qwen3-30B-A3B with 3 GiB: 28.7 tokens/s against 24.4, 381 tokens agreeing of 400 against 377), with a mean logit difference as low or lower in seven of the eight; it agreed on fewer most likely tokens in two (Italian, 64 tokens: 47 against 54 on the 30B with 3 GiB).
/experts n(or0) asks for a number of experts and turns it off. - Every weight type near the memory's speed, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU) for a reply's tokens (and for a prompt with
JANAS_GPU_FAST_PROMPT=0).tests/bench_kernelsmeasures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 61 (48 until 9 October 2026, when its unpacking went through the stack), IQ3_S 25, against the 81 GB/s the memory gives (MXFP4, since 14 October 2026, at 66 GB/s where IQ4_NL reads at 73 in the same run: 4% fewer weights a second, a block of it being 17 bytes to IQ4_NL's 18); and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.