37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Commits on main
Release 2026-10-30 (2): fma() rounded once on the GPUs whose driver splits it (RADV before RDNA2, issue #48) - a probe when the GPU opens, and shaders whose fma goes through float64; float conversions held exact in the shaders
faae71d camauri committed 20 days ago
Release 2026-10-30: the ONNX converters (silero2jns, wespeaker2jns) build under gcc's thread sanitizer - their sub-buffers start empty
cc96c1d camauri committed 20 days ago
Release 2026-10-29 (2): test_server_files no longer depends on how fast the disk writes - its expiring file is added after the listings (it expired before them on Windows' runners)
84a65f5 camauri committed 21 days ago
Release 2026-10-27 (5): gpu_check no longer forces on the GPU a product the engine leaves to the CPU - weights off a 4-byte boundary, which AMD's driver reads wrong (issue #46)
463c48c camauri committed 22 days ago
Release 2026-10-27 (4): the long contexts against llama.cpp at the same eight-bit KV cache (q8_0): at 8,192 tokens 18.2 against 4.9 tok/s writing, 74 against 20 reading
f085859 camauri committed 23 days ago
Release 2026-10-27 (3): the packages' READMEs with the long contexts and Gemma 4 on the GPU, Llama 3 and janas-nsm
ea67777 camauri committed 23 days ago
Release 2026-10-27 (2): the README and docs/features.md with Gemma 4 whole on the GPU, Llama 3 and janas-nsm; Llama 3.2 (Meta) and DeepNSM's adapter (CC BY-NC-SA) among the models' credits
4d492fc camauri committed 23 days ago
Release 2026-10-27: Gemma 4 on the GPU's whole pass, long contexts, Llama 3 and janas-nsm
fc2f11d camauri committed 23 days ago
Release 2026-10-17 (2): test_jmath gives far angles' sines a margin - Windows's C library reduces them with fewer digits of pi, and the Windows package was not built
7b12692 camauri committed 1 month ago
Release 2026-10-17: the same answer at temperature 0 whatever the C library and the compiler - exp, log, pow, sin, cos and tanh of the engine's own, no multiply and add fused by the compiler (issue #38); test_quant says whether it compared the AVX2 SiLU (issue #26)
4f00ce5 camauri committed 1 month ago
Develop 2026-10-17: docs/reports.md, every tester's report in one table, by date
2d69686 camauri committed 1 month ago
Release 2026-10-14: MXFP4 on the GPU, and a mixture's experts on an integrated GPU again when its cache holds more than 4 GiB of them
afb874e camauri committed 1 month ago
Develop 2026-10-14 (5): the ChangeLog with the MXFP4_MOE file's speed against the Q4_K_M
2bfaf2d camauri committed 1 month ago
Develop 2026-10-14 (4): an integrated GPU no longer kept awake for a reply's single tokens (issue #16)
e4691cd camauri committed 1 month ago
Develop 2026-10-14 (3): the MXFP4_MOE files run - MXFP4 experts read as they are, output heads in BF16 or F16 read as Q8_0
a234d57 camauri committed 1 month ago