Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

9.6 KB
Raw Blame

Janas for Windows x86-64

Qwen3-Next-80B, a 48 GB model, on a laptop with 32 GB of memory: 26 tok/s writing a reply, where llama.cpp manages 3.0, and 68 tok/s reading a prompt, against 18. Janas-LLM read and wrote faster than llama.cpp on every model compared on that laptop (Debian 13 Linux with kernel 6.12.111, Intel Core Ultra 9 185H, its integrated Arc GPU with Mesa 26.1.6, 32 GB of DDR5-5600 memory, an NVMe SSD), under the same conditions wherever possible, each in its best configuration without the NPU, which this package does not use, measured on 21 October 2026:

Model reading 1,024 tokens: Janas-LLM llama.cpp writing: Janas-LLM llama.cpp
Qwen3-Next-80B-A3B (48.4 GB, more than the memory) 68.4 tok/s 17.7 tok/s 26.4 tok/s 3.0 tok/s
Qwen3-Coder-Next (48.4 GB, more than the memory) 71.4 tok/s 14.0 tok/s 26.5 tok/s 2.9 tok/s
Qwen3.6-35B-A3B 136 tok/s 81.8 tok/s 28.0 tok/s 12.1 tok/s
Qwen3-30B-A3B 133 tok/s 122 tok/s† 33.3 tok/s 15.0 tok/s†
Qwen3-30B-A3B, bartowski's Q4_K_M (importance matrix) 132 tok/s 125 tok/s 32.1 tok/s 15.9 tok/s
Qwen3-30B-A3B, unsloth's UD-Q3_K_XL (3 bits) 120 tok/s 114 tok/s 35.3 tok/s 11.3 tok/s
Gemma-4-26B-A4B-it 85.0 tok/s 78.0 tok/s 18.9 tok/s 12.6 tok/s
Qwen3.8-27B (dense, 17.6 GB) 31.4 tok/s 25.5 tok/s 4.4 tok/s 2.1 tok/s
Gemma-4-12B-it 65.7 tok/s 47.6 tok/s 10.0 tok/s 7.4 tok/s
Qwen3.5-9B 122 tok/s 80.3 tok/s 14.5 tok/s 8.7 tok/s
Gemma-4-E4B-it 129 tok/s 110 tok/s 19.8 tok/s 15.3 tok/s
Qwen3-4B 201 tok/s 172 tok/s 27.7 tok/s 17.9 tok/s
Qwen3-4B, bartowski's Q4_K_M (importance matrix) 202 tok/s 173 tok/s 27.8 tok/s 18.2 tok/s
Gemma-4-E2B-it 196 tok/s 150 tok/s 36.9 tok/s 18.9 tok/s
Qwen3.5-2B 477 tok/s 314 tok/s 53.3 tok/s 27.5 tok/s
Qwen3.5-0.8B 950 tok/s 514 tok/s 108 tok/s 31.1 tok/s
Qwen3-0.6B 974 tok/s 710 tok/s 129 tok/s 46.0 tok/s

Janas-LLM's figures are the best of the configurations janas-bench measures without the NPU, the GPU reading a prompt's blocks; llama.cpp ran with its Vulkan backend on the same GPU, the API Janas-LLM uses (on the two models larger than the memory it wrote with its CPU backend, faster there); the conditions are the same wherever the two engines allow it, and the main README says where they are not. † llama.cpp's figures of 11 October. On another machine the figures will be other ones: janas-bench measures yours (below). How they were taken, and everything else: https://github.com/prabanta-dev/janas.

Long contexts: at 8,192 tokens Janas-LLM writes a reply at 18.2 tok/s, where llama.cpp manages 4.9, and reads the prompt at 74 tok/s, against 20 - both keeping keys and values at eight bits. Qwen3-4B Q4_K_M on the same laptop (measured on Linux), the CPU alone on both sides (12 threads), the same file, 26-27 October 2026; a prompt read from the start, then 64 tokens written after it:

Context reading the prompt: Janas-LLM llama.cpp writing after it: Janas-LLM llama.cpp
1,024 tokens 118 tok/s 54 27.6 tok/s 12.1
4,096 96 32 22.0 7.5
8,192 74 20 18.2 4.9
16,384 56 under 9.1 15.0 -
32,000 35 - 10.7 -

The longer the context, the wider the gap: 2.3 times as fast writing at 1,024 tokens, 3.7 times at 8,192. Both slow down as the context grows, and on both the attention is what grows: at 32,000 tokens it takes 59 of the 93 ms Janas-LLM spends on a token written, and the 32,000 tokens of prompt take 15 minutes to read. Janas-LLM's peak memory went from 2.6 GiB at 1,024 tokens to 4.9 at 32,000. The same conditions on both sides: keys and values at eight bits (llama.cpp's q8_0 cache, with its flash attention, which q8_0 values need). llama.cpp had not finished reading 16,384 tokens after 30 minutes, so its writing there was not measured, nor anything at 32,000. With its default sixteen-bit cache llama.cpp read faster (67, 55, 44 and 31 tok/s at 1,024 to 16,384 tokens) and wrote about as fast (12.5, 7.7, 4.6 and 2.2). How it was measured: the README on GitHub.

New: Gemma 4 whole on the GPU, Llama 3, and janas-nsm. Gemma 4's four models - 12B, the 26B mixture, E2B and E4B - now run every layer of a token on the GPU, with the GPU's norms and quantizations giving the CPU's bits: Gemma-4-12B agrees with llama.cpp at 367 of 400 most likely tokens on the GPU, at 362 on the CPU. Llama 3 models run too, a LoRA adapter merged into one by tools/hf2gguf: DeepNSM-1B (Llama-3.2-1B with the adapter of arXiv 2505.11764) writes its answers at 57.8 tok/s on the laptop's CPU, where llama.cpp writes 37.9 to 39.8 on the same file. janas-nsm asks it for a word's explication in the Natural Semantic Metalanguage - Goddard and Wierzbicka's sixty-odd semantic primes, the words every language has - and counts it as the paper does; it can also restrict the model's output to those words alone. A research tool, and a sign of how far this project means to go: docs/tools.md.

Janas-LLM, the inference engine of Janas, runs large language models on ordinary computers, with or without a GPU, mixtures of experts larger than the memory among them: the weights stay on disk, the experts each token needs are read from the SSD and the ones that keep coming back stay in memory. It is written in C and needs nothing but the system's C library; a GPU is used through its Vulkan driver when there is one.

The figures above were measured on Linux. On the same laptop under Windows 11 (build 26300, 12 October 2026), this package wrote as fast (Qwen3-4B 25.6 tok/s, Qwen3-30B-A3B 34.2), read a prompt as fast with the mixture (Qwen3-30B-A3B 132 tok/s) and faster with the dense model (Qwen3-4B 346 tok/s, against 193 on Linux that week), whose whole blocks go to the GPU through Intel's Windows driver.

Start

In a terminal (Windows Terminal, PowerShell or the Command Prompt), in the folder where the package was unpacked:

.\janas-get list
.\janas-get qwen3-4b
.\janas-chat qwen3-4b
.\janas-bench qwen3-4b

The first says which models it knows, their size and the memory they want; the second downloads one, checked, into %LOCALAPPDATA%\janas\models; the third talks to it; the fourth measures how fast this machine runs it and writes a report to send back.

.\janas-chat with no model shows the list and downloads the one you choose, after asking. The models come already converted from Hugging Face (prabanta-dev), each file checked against the fingerprints of MODELS.md.

What is here

janas-chat a chat in the terminal: line editor, history, the reasoning shown apart, a context that slides instead of ending, the tools of MCP servers and of Janas's own services
janas-server the model served over HTTP with an API that follows OpenAI's: chat completions (tools, JSON schemas), completions, responses, embeddings, and llama.cpp's route for filling in code in editors
janas-get downloads a model of the catalog, checked; --from-gguf converts the original GGUF here instead
janas-bench measures what this machine does with a model, every configuration the engine can use, and fills the profile it starts from next time
janas-mcp the model as an MCP server for other clients (Claude Code, an editor): a question to it, the embedding of a text
gguf2jns converts a GGUF file (Q4_K_M and the other types MODELS.md lists) to the .jns file Janas reads
jns_planes cuts the experts of a converted mixture into bit planes, so that fewer bits can be read when the memory is short
hf2jns_mtp makes a model's multi-token prediction block from its original checkpoint
jns_check what a .jns file says about itself: architecture, sizes, types, licence
vkcheck whether this computer's GPU can run Janas's GPU backend, and why not: what the backend needs with the driver's answer, then three short tests on the GPU
services/ Janas's services, which bring data of the world to a model: flights, weather, maps, prices, earthquakes, Wikipedia, GitHub, git, this computer. janas-chat starts them by itself; any other MCP client can use them. See services/README.md
lib/, include/ libjanas_llm.dll and libjanas_mcp.dll, their import libraries for MinGW (.dll.a), their C headers and FreeBASIC bindings, for programs of your own; the programs above do not need them

Needs

  • Windows 10 (version 1903 or later) or Windows 11, on x86-64. Nothing to install: the programs use only what Windows has.
  • An NVMe SSD for the models larger than the memory.
  • A GPU is optional: one with a Vulkan driver (the drivers of Intel, AMD and NVIDIA include it) reads the prompts faster.

What Janas keeps goes to %LOCALAPPDATA%\janas (the models, what it learns about the machine) and %APPDATA%\janas (the settings, the MCP servers); the variables JANAS_MODELS and XDG_DATA_HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME move them elsewhere.

It is an experiment, developed and measured on one Linux machine: an Intel CPU with AVX2 and AVX-VNNI. The Windows build is new and has run on that one machine only; the report of janas-bench is the most useful thing to send back (https://github.com/prabanta-dev/janas/issues).

Licence

Janas is under the GNU General Public License, version 3 or later: LICENSE. The models are not part of it and keep their own licences: MODELS.md.