37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-06: Q3_K, IQ4_XS and IQ3_S weights read at full speed; bench_kernels
camauri7197a2ea0489
diff --git a/ChangeLog.md b/ChangeLog.mdindex 50ad4ed..455e460 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,12 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-06] - Q3_K, IQ4_XS and IQ3_S weights read at full speed++- **The products of Q3_K, IQ4_XS and IQ3_S weights were six to thirteen times slower than they should have been.** Their kernels converted each block's 16-bit scale by calling a function, inside the vector loop, where the others use the processor's own conversion; now they do too, with the same result to the bit. On one core, a 4096 x 2048 matrix: Q3_K 3.38 to 0.43 ms, IQ4_XS 3.26 to 0.25, IQ3_S 3.45 to 0.56. Unsloth's UD files mix these types in: Qwen3.8-27B UD-Q4_K_XL, 3.3 GB of whose 17.5 are of them, writes 3.6 tokens/s where it wrote 1.7-1.8, and reads a prompt at 8.2 tokens/s where it read 6.4.+- **`tests/bench_kernels`**: every weight type's product on one core and on many, over 1 to 8 vectors, in GB/s of weights, to set beside the memory's bandwidth. On the development machine, one vector on 12 threads: Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the 4- and 5-bit types 61-68, Q3_K 33 and IQ3_S 25 - those two still bound by their arithmetic.+- **Where the time of a token goes, against the bandwidth** (bytes each phase reads, over its time): on Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B and the 27B, every phase that reads weights runs at 60-70 GB/s, 59-62 on average - three quarters of the 81 GB/s the memory gives, nine tenths of what the kernels do alone.+## [2026-10-06] - A long prompt reads the next layer's experts ahead- **The prefill reads ahead too.** Block by block, the next layer's router is applied to the whole block before the current layer's experts, and the experts most of its tokens will want are read from the disk while the current layer computes; until now each layer waited for the disk chunk by chunk. On after a block whose layers missed more than 32 experts each, off where the experts fit. Qwen3-Next-80B-A3B, 4,096 tokens, every bit: 36.8 to 42.0 tokens/s with a 4 GiB cache (time waited for the disk 40.8 to 20.7 s), 48.5 to 50.0 with 16 GiB, the 256 tokens written after it 21.8 to 21.5 tokens/s; Qwen3-30B-A3B in a cache that holds it, even. The cost is the router applied once more, about 1.1 ms a token. `JANAS_PREFETCH_PREFILL=0` never, `=2` always. The idea is Strata's (github.com/Niko1221/Strata), which streams the next layer's experts to its graphics card.diff --git a/README.md b/README.mdindex c43a248..99a13ff 100644--- a/README.md+++ b/README.md@@ -546,7 +546,7 @@ Close what you can before running it: it measures the machine you give it, and a| `tools/gen_geo` | the services' tables of airports, cities and seas, from OurAirports, GeoNames and Natural Earth || `mcp_fake_server` | an MCP server over stdio in either era of the protocol, for trying a client without anybody else's server || `llm_eval` | compares the engine's logits against a reference dump |-| `bench_gemm`, `bench_attn`, `bench_long`, ... | the pieces measured on their own |+| `bench_kernels`, `bench_gemm`, `bench_attn`, `bench_long`, ... | the pieces measured on their own |They are built into `bin/x86_64-linux/` along with everything else.diff --git a/src/llm/quant_dotn.h b/src/llm/quant_dotn.hindex 6312488..71915b7 100644--- a/src/llm/quant_dotn.h+++ b/src/llm/quant_dotn.h@@ -738,7 +738,7 @@ DOTN(iq4xs_dotn_body)(const struct janas_block_iq4xs *w,aq[g] = _mm256_abs_epi8(q[g]);sc[g] = _mm256_set1_epi16((short)iq4xs_scale(&w[b], g));}- float d = janas_fp16_to_fp32(w[b].d);+ float d = _cvtsh_ss(w[b].d);_Pragma("GCC unroll 8") for (int v = 0; v < nv; v++){const struct janas_block_q8k *xb = x + (size_t)v * xs + b;@@ -1299,7 +1299,7 @@ DOTN(q3k_dotn_body)(const struct janas_block_q3k *w,_mm_set1_epi16(sc[2 * k + 1]));__m256i scall =_mm256_cvtepi8_epi16(_mm_loadu_si128((const __m128i *)sc));- float d = janas_fp16_to_fp32(w[b].d);+ float d = _cvtsh_ss(w[b].d);_Pragma("GCC unroll 8") for (int v = 0; v < nv; v++){const struct janas_block_q8k *xb = x + (size_t)v * xs + b;@@ -1397,7 +1397,7 @@ DOTN(iq3s_dotn_body)(const struct janas_block_iq3s *w,sg[g] = _mm256_or_si256(m, one); /* -1 where the sign bit is set */sc[g] = _mm256_set1_epi16((short)iq3s_scale(&w[b], g));}- float d = janas_fp16_to_fp32(w[b].d);+ float d = _cvtsh_ss(w[b].d);_Pragma("GCC unroll 8") for (int v = 0; v < nv; v++){const struct janas_block_q8k *xb = x + (size_t)v * xs + b;diff --git a/tests/bench_kernels.c b/tests/bench_kernels.cnew file mode 100644index 0000000..02623e6--- /dev/null+++ b/tests/bench_kernels.c@@ -0,0 +1,127 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * bench_kernels.c - every weight type's product, on one core and on many,+ * over 1 to 8 vectors, in GB/s of weights read: to set beside the memory's+ * bandwidth (about 81 GB/s on the development machine). A product of one+ * vector reads each weight once and does little with it, so a kernel well+ * below the others on one core is spending its time computing, not waiting+ * for memory - which is how the Q3_K, IQ4_XS and IQ3_S kernels were found+ * to convert their scale through a function call (6 Oct 2026).+ *+ * Usage: bench_kernels [MiB [threads [first_cpu]]]+ * Defaults: 256 MiB of weights per type, 12 threads on CPUs 0-11; the+ * single core is first_cpu (default 2).+ * The weights are random bytes below 0x7c, so that every fp16 scale is a+ * finite number; the speed of a kernel does not depend on the values.+ */+#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <time.h>++#include "common/pool.h"+#include "llm/matvec.h"+#include "llm/quant.h"++static double now(void)+{+ struct timespec ts;+ clock_gettime(CLOCK_MONOTONIC, &ts);+ return ts.tv_sec + ts.tv_nsec * 1e-9;+}++static const struct {+ const char *name;+ int type;+} TYPES[] = {+ {"Q4_K", JANAS_Q4_K}, {"Q5_K", JANAS_Q5_K}, {"Q6_K", JANAS_Q6_K},+ {"Q3_K", JANAS_Q3_K}, {"Q8_0", JANAS_Q8_0}, {"Q4_0", JANAS_Q4_0},+ {"Q4_1", JANAS_Q4_1}, {"Q5_1", JANAS_Q5_1}, {"IQ4_NL", JANAS_IQ4_NL},+ {"IQ4_XS", JANAS_IQ4_XS}, {"IQ3_S", JANAS_IQ3_S}};++/* Seconds per product, the best of passes run for at least 0.3 s. */+static double timed(struct janas_pool *pool, struct janas_matvec_task *t)+{+ double best = 1e9, start = now();+ int runs = 0;+ while (runs < 3 || now() - start < 0.3) {+ double t0 = now();+ janas_matvec_q4k_group(pool, t, 1);+ double dt = now() - t0;+ if (dt < best)+ best = dt;+ runs++;+ }+ return best;+}++int main(int argc, char **argv)+{+ size_t mib = argc > 1 ? (size_t)atol(argv[1]) : 256;+ int nt = argc > 2 ? atoi(argv[2]) : 12;+ int one = argc > 3 ? atoi(argv[3]) : 2;+ if (mib < 1 || nt < 1 || nt > 256 || one < 0) {+ fprintf(stderr, "usage: bench_kernels [MiB [threads [first_cpu]]]\n");+ return 2;+ }+ int cpus[256];+ for (int i = 0; i < nt; i++)+ cpus[i] = i;+ struct janas_pool *many = janas_pool_create(nt, cpus);+ struct janas_pool *single = janas_pool_create(1, &one);+ if (!many || !single) {+ fprintf(stderr, "bench_kernels: no thread pool\n");+ return 1;+ }+ const size_t cols = 2048, nb = cols / JANAS_QK, nvs[] = {1, 2, 4, 8};+ struct janas_block_q8k *x = calloc(8 * nb, sizeof(*x));+ float *v = malloc(8 * cols * sizeof(float));+ srand(1);+ for (size_t i = 0; i < 8 * cols; i++)+ v[i] = (float)rand() / RAND_MAX - 0.5f;+ for (int k = 0; k < 8; k++)+ janas_q8k_quantize(v + k * cols, x + k * nb, cols);+ printf("GB/s of weights, %zu MiB a type, 2048 columns; 1 core (CPU %d) "+ "and %d threads\n",+ mib, one, nt);+ printf("%-7s %8s %8s %8s %8s | %8s %8s %8s %8s\n", "type", "1x1", "1x2",+ "1x4", "1x8", "Nx1", "Nx2", "Nx4", "Nx8");+ for (size_t k = 0; k < sizeof TYPES / sizeof *TYPES; k++) {+ size_t bs = janas_qtype_block_size(TYPES[k].type);+ size_t rows = (mib << 20) / (nb * bs), bytes = rows * nb * bs;+ unsigned char *w = malloc(bytes);+ float *y = malloc(8 * rows * sizeof(float));+ if (!w || !y) {+ fprintf(stderr, "bench_kernels: out of memory\n");+ return 1;+ }+ for (size_t i = 0; i < bytes; i++)+ w[i] = (unsigned char)(rand() % 0x7c);+ printf("%-7s", TYPES[k].name);+ for (int p = 0; p < 2; p++) {+ for (size_t i = 0; i < 4; i++) {+ struct janas_matvec_task t = {.type = TYPES[k].type,+ .w = w,+ .x = x,+ .y = y,+ .rows = rows,+ .cols = cols,+ .n_vec = nvs[i],+ .x_stride = nb,+ .y_stride = rows};+ double s = timed(p ? many : single, &t);+ printf(" %8.1f", bytes / s / 1e9);+ }+ printf(p ? "\n" : " |");+ }+ fflush(stdout);+ free(w);+ free(y);+ }+ free(x);+ free(v);+ janas_pool_destroy(single);+ janas_pool_destroy(many);+ return 0;+}