37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
Installing Janas
Janas-LLM runs large mixture-of-experts language models on ordinary computers: the model's weights stay on disk and the experts each token needs are streamed from an NVMe SSD, with the useful ones kept in memory. This file explains how to build it, prepare a model, chat with it and measure your machine.
Requirements
- Linux on x86-64, or Windows 10 (1903 or later) and 11 on x86-64: the binary package of each release, or a build made on Linux with MinGW-w64 (below).
- CPU: any x86-64 runs it; AVX2 with FMA and F16C (Intel since 2013, AMD since 2015) is needed for good speed, and AVX-VNNI is used when present.
- Memory: at least 16 GB; 32 GB or more for large models (Qwen3-Next-80B-A3B: about 2 GB of resident weights, plus as much expert cache as the memory allows).
- Disk: the model on an NVMe SSD (the experts are read directly from it while generating).
- GPU (optional): an integrated or discrete GPU with a Vulkan 1.3 driver. The engine works fully without one; it measures whether the GPU helps on your machine and uses it only then.
- NPU (optional, Linux): the NPU of an Intel Core Ultra, with Intel's NPU driver (below). It reads prompts; the engine measures it as it does the GPU.
Building
On Debian or Ubuntu:
sudo apt install build-essential
# optional, for GPU support:
sudo apt install libvulkan-dev glslang-tools vulkan-tools
On Fedora: sudo dnf install gcc make and, for the GPU, vulkan-loader-devel glslang vulkan-tools. On Arch: sudo pacman -S base-devel and vulkan-headers vulkan-icd-loader glslang vulkan-tools.
The shaders need glslang 15.2 or later (glslangValidator --version). Ubuntu 24.04's glslang-tools is 15.1, too old: the build then goes on without the GPU and says why. A newer one is the prebuilt release from https://github.com/KhronosGroup/glslang/releases (glslang-16.6.0-linux-x86_64-release.tar.gz, the one Janas's own packages are built with): unpack it and put its bin/ first in the PATH before building.
The GPU also needs its Vulkan driver: Mesa (mesa-vulkan-drivers on Debian/Ubuntu) for Intel and AMD, the proprietary driver for NVIDIA. vulkaninfo --summary should list your GPU with Vulkan 1.3 or later. When the engine takes none, janas-bench and the model's description say why: a build without GPU support (glslangValidator missing or older than 15.2: see above, and build again), no Vulkan loader, a GPU whose driver lacks something (named), or the GPU turned off.
The NPU needs nothing at build time: the engine opens Intel's Level Zero driver when a model opens, if it is installed, and goes on without it otherwise. To use it, install the intel-level-zero-npu and intel-driver-compiler-npu packages of a release of linux-npu-driver (1.38.0 was tried, with its packages for Ubuntu 24.04 on Debian 13) and a Level Zero loader, libze1 (1.32.0 was tried), and add your user to the render group (sudo usermod -aG render $USER, then log in again), which may open the NPU's device. janas-bench names the NPU on its NPU: line, or says why there is none to use. JANAS_NPU_LOADER names a loader outside the library path, such as one unpacked without installing (dpkg -x, with LD_LIBRARY_PATH pointing at the driver's libraries); JANAS_NPU=0 keeps the engine off the NPU. Only the first generation's NPU (Meteor Lake) has been tried.
Then, in the source directory:
./build.sh
Everything goes to bin/x86_64-linux/: janas-chat, janas-server, janas-mcp, janas-bench, the libraries libjanas_llm.so (C API in include/janas/llm.h) and libjanas_mcp.so (the MCP client, include/janas/mcp.h) and the conversion tools. Without glslangValidator the build still succeeds, without GPU support. ./build.sh release test also runs the tests, and ./build.sh --help lists the other variants — a debug build, and the three sanitizers asan, ubsan and tsan — with what each one catches.
For Windows
On Linux, with MinGW-w64 (sudo apt install gcc-mingw-w64-x86-64 on Debian or Ubuntu), the same sources build for Windows x86-64:
JANAS_OS=windows ./build.sh # into bin/x86_64-windows/: .exe, the two .dll
JANAS_OS=windows ./build.sh release test # the tests, under Wine
The GPU's support comes in when glslangValidator and the Vulkan headers are there, as for Linux. The programs need nothing but Windows's own DLLs. On Windows what Janas keeps goes to %LOCALAPPDATA%\janas (the models, what it learns about the machine) and %APPDATA%\janas (the settings, the MCP servers).
Preparing a model
Janas reads its own file format (.jns), converted from a GGUF file. Supported now: Qwen3.5 and Qwen3.6 (qwen35moe, e.g. Qwen3.6-35B-A3B), Qwen3-Next (qwen3next, e.g. Qwen3-Next-80B-A3B-Instruct and Qwen3-Coder-Next) and Qwen3 MoE (qwen3moe, e.g. Qwen3-30B-A3B), in the Q4_K, Q5_K, Q6_K and Q8_0 formats (so the usual Q4_K_M files).
Keep models in models/ next to the sources: the tools take a path, so anywhere works, but that is where the examples here and in the models' page put them. That page also lists, for each model, which Hugging Face repository to take the GGUF from and how large it is.
Download a Q4_K_M GGUF of the model from Hugging Face, into
models/gguf/.Convert it with the converter the build made (with a model split into several files, give the first):
bin/x86_64-linux/gguf2jns models/gguf/Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf \ models/qwen3-next.jnsYou need room for both while it runs: about twice the model.
Optional, for Qwen3-Next: its multi-token prediction block makes replies faster (drafts of the next tokens, verified by the model: the text is the same). GGUF files leave that block out, so take it from the original checkpoint, Qwen/Qwen3-Next-80B-A3B-Instruct, whose last shard holds it:
bin/x86_64-linux/hf2jns_mtp models/qwen3-next.jns \ models/hf/model-00041-of-00041.safetensors models/qwen3-next-mtp.jns
Qwen3.5 and Qwen3.6 keep their prediction block inside the model file: it is converted and used with no further step.
Keep the .jns files on the NVMe SSD: the experts are read from there while the model answers, and a slower disk is felt at every token.
Chatting
bin/x86_64-linux/janas-chat qwen3-next.jns --mtp qwen3-next-mtp.jns
Type a message and press Enter; a backslash and Enter add a line to it instead of sending it, and the backslash is not part of what the model reads. Commands start with /: /help lists them, /stats shows the speed of every reply, /mode the power mode (auto, eco, max) and what the engine chose, /think turns reasoning on or off for models that reason (their reasoning is printed in grey), /experts n sets how many experts each token uses (fewer: faster replies, a little less accurate). Ctrl-C stops a reply, Ctrl-D quits. janas-chat with no arguments lists the options (context length, memory for the expert cache, sampling).
The engine tunes itself while it runs: it tries the configurations your machine offers (how many threads, whether the GPU takes part) on real work and keeps the fastest. The first minutes of use are the learning phase; the choices are kept in ~/.cache/janas/, per machine and model. Afterwards it still tries an alternative now and then, to follow the machine's state (heat, power source), and ever more rarely while the alternatives keep losing; the number of threads, settled by the first tries, moves when other threads win several tries in a row.
Measuring your machine
bin/x86_64-linux/janas-bench qwen3-next-80b-a3b
The model is a name of the catalog, downloaded after asking if it is not there yet, with its prediction file; a .jns file works too, and with no model it shows the list. It takes a few minutes: it measures prompt processing and generation speed for every configuration, fills the tuning profile (so chatting starts tuned) and writes a report, janas-bench-<date>.txt, in the current directory. The report lists your processor, memory, GPU, the speeds and the choices: send it along with any feedback.
Troubleshooting
JANAS_GPU=0turns the GPU off;JANAS_GPU_DEVICE=integratedordiscretepicks one when there are both;JANAS_GPU_FAST_PROMPT=0reads prompts the exact way, bit for bit as the CPU (the fast prompt, on by default, rounds them group by group on the GPU).JANAS_GPU_BLOCK=0keeps the dense models' prompts off the whole-block path (every layer of a block in one submission to the GPU) while leaving the fast prompt's products on;JANAS_PREFILL_BLOCK=nsets the tokens of a prompt's block (256 by default, 512 where the GPU takes the feed-forwards whole).JANAS_PREFETCH=0never reads experts ahead (by default it turns itself on when the cache misses many experts a token).JANAS_ARENA_HUGE=1asks the kernel for huge pages for the expert cache, as before 30 September 2026 (with transparent huge pages onalwaysand their defrag onmadviseit made the first writes to the cache slow).JANAS_KERNELS=avx2uses the AVX2 kernels even where AVX-VNNI is available.JANAS_TUNE=0turns self-tuning off (fixed defaults); delete~/.cache/janas/tuning.txtto start tuning from scratch.JANAS_TUNE_THREADS=0keeps every usable thread instead of letting the first passes choose between every thread, one per performance core and the performance cores' threads; the choice is kept for that machine and model, and moves only when other threads win several tries in a row.- The expert cache takes the free memory, leaving a fifth of the machine's memory to the rest of the system;
--cache <GiB>injanas-chatsets it.