37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-07: small gains kept - prompts in blocks, drafts checked, long contexts; the README on the kernels
camaurife33e68bdfb7
diff --git a/ChangeLog.md b/ChangeLog.mdindex 827fd38..43ee506 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,13 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-07] - Small gains, kept: blocks of tokens and long contexts++- **The activations' sums of 32 are made once.** The Q4_K and Q5_K products over several vectors - a prompt read in blocks, the drafts checked - weigh the sums of each 32 activations by the block's mins, and added them up again for every row of every matrix; the activation's quantization now leaves them beside the sums of 16 (16 bytes more a block of 256). 5-6% on those products.+- **Three or four vectors reduced in four lanes**, where they were reduced in eight, half of them empty: the size of the check of a few drafts. 2% on 12 threads, 5% on one core.+- **Attention's scores eight keys at a time.** The sum of each key's products was reduced on its own, with as many instructions as the products and on the same ports; eight are reduced together now, in the same order of additions, so the scores are the same to the bit. 5-8% off the attention of a layer at 8,000 positions.+- Together, on Qwen3-30B-A3B in memory, a prompt of 1,900 tokens: read at 70.1-70.5 tokens/s where it was 67.4-67.5 (+4%), the 128 tokens after it 25.8-26.0 where 25.4-25.5 (+1.5%), attention 6.5 ms a token where 6.9. Every result the same to the bit (`test_quant` on AVX2 and AVX-VNNI, `test_attention`'s checksum unchanged).+## [2026-10-06] - Q3_K, IQ4_XS and IQ3_S weights read at full speed- **The products of Q3_K, IQ4_XS and IQ3_S weights were six to thirteen times slower than they should have been.** Their kernels converted each block's 16-bit scale by calling a function, inside the vector loop, where the others use the processor's own conversion; now they do too, with the same result to the bit. On one core, a 4096 x 2048 matrix: Q3_K 3.38 to 0.43 ms, IQ4_XS 3.26 to 0.25, IQ3_S 3.45 to 0.56. Unsloth's UD files mix these types in: Qwen3.8-27B UD-Q4_K_XL, 3.3 GB of whose 17.5 are of them, writes 3.6 tokens/s where it wrote 1.7-1.8, and reads a prompt at 8.2 tokens/s where it read 6.4.diff --git a/README.md b/README.mdindex 99a13ff..cac371b 100644--- a/README.md+++ b/README.md@@ -67,8 +67,9 @@ Everything the engine does is chosen from what it measures, so another machine w- **Faster replies** from drafts the model verifies, so the text is exactly the one it would have written: from the model's own multi-token prediction block where it has one, from the conversation where it does not, or from a small model of the same family given with `--draft`.- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. Results are identical with and without it.- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even. `JANAS_PREFETCH_PREFILL=0` turns it off.+- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 48, IQ3_S 25, against the 81 GB/s the memory gives; and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.-On the machine above, with nothing else running, `janas-bench` reports this:+On the machine above, with nothing else running, `janas-bench` reports this (measured before the changes of 6-7 October 2026: on Qwen3-30B-A3B in memory, `tests/bench_long` found them worth 4% reading a 1,900-token prompt and 1.5% writing after it):| Model | reading a prompt | writing a reply | with drafts, in a chat ||---|---:|---:|---:|diff --git a/src/llm/attention.c b/src/llm/attention.cindex 416ec0f..7bf33c4 100644--- a/src/llm/attention.c+++ b/src/llm/attention.c@@ -160,7 +160,70 @@ f32_body(const float *const *q, const int8_t *K, const float *ks, uint32_t nt,uint32_t hd, float scale, float *const *sc, const int8_t *pf,const int ns){- for (uint32_t t = 0; t < nt; t++) {+ uint32_t t = 0;+ /*+ * Eight rows at a time, from four slots on: the rows converted once into+ * a buffer, then for each slot one accumulator a row and the eight sums+ * reduced together. The reduction of one row alone below cost as many+ * instructions as its products, on the same ports; done for eight it+ * costs a third of one. Each row's sum is the same tree as below - the+ * halves, then lanes 0 + 2 and 1 + 3, then the two - so the scores are+ * the same to the bit.+ */+ if (ns >= 4 && hd <= 256 && hd % 8 == 0) {+ float kb[8 * 256] __attribute__((aligned(32)));+ const __m256i order = _mm256_setr_epi32(0, 4, 1, 5, 2, 6, 3, 7);+ for (; t + 8 <= nt; t += 8) {+ if (pf)+ for (uint32_t r = 0; r < 8; r++)+ for (uint32_t i = 0; i < hd; i += 32)+ _mm_prefetch(+ (const char *)(pf + (size_t)(t + r) * hd + i),+ _MM_HINT_T1);+ for (uint32_t r = 0; r < 8; r++)+ for (uint32_t i = 0; i < hd; i += 8)+ _mm256_store_ps(+ kb + r * hd + i,+ _mm256_cvtepi32_ps(_mm256_cvtepi8_epi32(_mm_loadl_epi64(+ (const __m128i *)(K + (size_t)(t + r) * hd + i)))));+ const __m256 ksv = _mm256_loadu_ps(ks + t);+ for (int s = 0; s < ns; s++) {+ __m256 acc[8];+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++) acc[r] =+ _mm256_setzero_ps();+ for (uint32_t i = 0; i < hd; i += 8) {+ __m256 qv = _mm256_loadu_ps(q[s] + i);+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++) acc[r] =+ _mm256_fmadd_ps(qv, _mm256_load_ps(kb + r * hd + i),+ acc[r]);+ }+ /* each row's halves added: rows r and r + 1 in one register */+ __m256 h[4];+ _Pragma("GCC unroll 4") for (int r = 0; r < 4; r++) h[r] =+ _mm256_add_ps(_mm256_permute2f128_ps(acc[2 * r],+ acc[2 * r + 1], 0x20),+ _mm256_permute2f128_ps(acc[2 * r],+ acc[2 * r + 1], 0x31));+ /* lanes 0 + 2 and 1 + 3 of each row's four */+ __m256 g01 = _mm256_add_ps(+ _mm256_shuffle_ps(h[0], h[1], _MM_SHUFFLE(1, 0, 1, 0)),+ _mm256_shuffle_ps(h[0], h[1], _MM_SHUFFLE(3, 2, 3, 2)));+ __m256 g23 = _mm256_add_ps(+ _mm256_shuffle_ps(h[2], h[3], _MM_SHUFFLE(1, 0, 1, 0)),+ _mm256_shuffle_ps(h[2], h[3], _MM_SHUFFLE(3, 2, 3, 2)));+ /* then the two: rows 0 2 4 6 | 1 3 5 7, put in order */+ __m256 sum = _mm256_add_ps(+ _mm256_shuffle_ps(g01, g23, _MM_SHUFFLE(2, 0, 2, 0)),+ _mm256_shuffle_ps(g01, g23, _MM_SHUFFLE(3, 1, 3, 1)));+ sum = _mm256_permutevar8x32_ps(sum, order);+ _mm256_storeu_ps(+ sc[s] + t,+ _mm256_mul_ps(_mm256_mul_ps(sum, _mm256_set1_ps(scale)),+ ksv));+ }+ }+ }+ for (; t < nt; t++) {const int8_t *k = K + (size_t)t * hd;/* the values pass reads V by columns, which the hardware prefetcherdoes not follow: bring its rows into L2 in order, now */diff --git a/src/llm/quant.c b/src/llm/quant.cindex a3a9abb..83ee20c 100644--- a/src/llm/quant.c+++ b/src/llm/quant.c@@ -124,6 +124,9 @@ static void q8k_quantize_ref(const float *x, struct janas_block_q8k *y,s += y[b].qs[g * 16 + i];y[b].bsums[g] = (int16_t)s;}+ for (int k = 0; k < JANAS_QK / 32; k++)+ y[b].bsums2[k] =+ (int16_t)(y[b].bsums[2 * k] + y[b].bsums[2 * k + 1]);q8_32_ref(x, &y[b], 0);}}@@ -149,6 +152,9 @@ static void q8k_quantize_det_ref(const float *x, struct janas_block_q8k *y,s += y[b].qs[g * 16 + i];y[b].bsums[g] = (int16_t)s;}+ for (int k = 0; k < JANAS_QK / 32; k++)+ y[b].bsums2[k] =+ (int16_t)(y[b].bsums[2 * k] + y[b].bsums[2 * k + 1]);q8_32_ref(x, &y[b], 1);}}@@ -225,6 +231,9 @@ __attribute__((target("avx2,fma"))) static voidq8k_block_avx2(const float *x, struct janas_block_q8k *y, float id, int det){q8_groups_avx2(x, y->qs, y->bsums, JANAS_QK / 16, id);+ __m128i lo = _mm_loadu_si128((const __m128i *)y->bsums),+ hi = _mm_loadu_si128((const __m128i *)(y->bsums + 8));+ _mm_storeu_si128((__m128i *)y->bsums2, _mm_hadd_epi16(lo, hi));for (int k = 0; k < 8; k++) {float amax = q8_amax32(x + 32 * k);float d = det ? amax * 0.007874015718698502f : amax / 127.0f;diff --git a/src/llm/quant.h b/src/llm/quant.hindex 67b4783..cc8b658 100644--- a/src/llm/quant.h+++ b/src/llm/quant.h@@ -166,6 +166,9 @@ struct janas_block_q8k {float d32[8];int8_t qs32[JANAS_QK];int16_t bsums32[JANAS_QK / 16];+ /* bsums two by two: the sums of 32, which the Q4_K and Q5_K products+ weigh by their mins, made once here instead of for every row */+ int16_t bsums2[JANAS_QK / 32];};_Static_assert(sizeof(struct janas_block_q4k) == 144, "q4k block size");@@ -180,7 +183,7 @@ _Static_assert(sizeof(struct janas_block_q5_1) == 24, "q5_1 block size");_Static_assert(sizeof(struct janas_block_q4_1) == 20, "q4_1 block size");_Static_assert(sizeof(struct janas_block_q3k) == 110, "q3k block size");_Static_assert(sizeof(struct janas_block_iq3s) == 110, "iq3_s block size");-_Static_assert(sizeof(struct janas_block_q8k) == 612, "q8k block size");+_Static_assert(sizeof(struct janas_block_q8k) == 628, "q8k block size");/* Quantization types, numbered as in GGUF (ggml_type). */enum janas_qtype {diff --git a/src/llm/quant_dotn.h b/src/llm/quant_dotn.hindex 03d7cc5..a0228a2 100644--- a/src/llm/quant_dotn.h+++ b/src/llm/quant_dotn.h@@ -54,8 +54,10 @@ DOTN(q4k_dotn_body)(const struct janas_block_q4k *w,float *out, const int nv){const __m256i low4 = _mm256_set1_epi8(0x0f);- /* up to 2 vectors: lanes acc0, acc1, accm0, accm1 */- __m128 acc4 = _mm_setzero_ps();+ /* up to 2 vectors: lanes acc0, acc1, accm0, accm1; 3 or 4: one lane+ each in acc4n and accm4n */+ __m128 acc4 = _mm_setzero_ps(), acc4n = _mm_setzero_ps(),+ accm4n = _mm_setzero_ps();__m256 acc8 = _mm256_setzero_ps(), accm8 = _mm256_setzero_ps();for (size_t b = 0; b < nb; b++) {uint32_t u[4];@@ -105,10 +107,8 @@ DOTN(q4k_dotn_body)(const struct janas_block_q4k *w,pm[v] = _mm_setzero_si128();if (v < nv) {const struct janas_block_q8k *xb = x + (size_t)v * xs + b;- __m128i bs_lo = _mm_loadu_si128((const __m128i *)xb->bsums);- __m128i bs_hi =- _mm_loadu_si128((const __m128i *)(xb->bsums + 8));- pm[v] = _mm_madd_epi16(mins16, _mm_hadd_epi16(bs_lo, bs_hi));+ pm[v] = _mm_madd_epi16(+ mins16, _mm_loadu_si128((const __m128i *)xb->bsums2));}}if (nv <= 2) {@@ -124,6 +124,20 @@ DOTN(q4k_dotn_body)(const struct janas_block_q4k *w,__m128 d8 = _mm_setr_ps(d80, d81, d80, d81);__m128 dd = _mm_setr_ps(dw, dw, dmw, dmw);acc4 = _mm_fmadd_ps(_mm_mul_ps(dd, d8), _mm_cvtepi32_ps(t), acc4);+ } else if (nv <= 4) {+ /* the sums of four vectors, one lane each: half the reduction */+ __m256i t = _mm256_hadd_epi32(_mm256_hadd_epi32(sumi[0], sumi[1]),+ _mm256_hadd_epi32(sumi[2], sumi[3]));+ __m128i is4 = _mm_add_epi32(_mm256_castsi256_si128(t),+ _mm256_extracti128_si256(t, 1));+ __m128i im4 = _mm_hadd_epi32(_mm_hadd_epi32(pm[0], pm[1]),+ _mm_hadd_epi32(pm[2], pm[3]));+ __m128 d8 = _mm_setr_ps(x[b].d, x[xs + b].d, x[2 * xs + b].d,+ nv == 4 ? x[3 * xs + b].d : 0.0f);+ acc4n = _mm_fmadd_ps(_mm_mul_ps(_mm_set1_ps(dw), d8),+ _mm_cvtepi32_ps(is4), acc4n);+ accm4n = _mm_fmadd_ps(_mm_mul_ps(_mm_set1_ps(dmw), d8),+ _mm_cvtepi32_ps(im4), accm4n);} else {/* the sums of the eight vectors, one lane each */__m256i s01 = _mm256_hadd_epi32(sumi[0], sumi[1]);@@ -156,6 +170,10 @@ DOTN(q4k_dotn_body)(const struct janas_block_q4k *w,out[0] = r[0] - r[2];if (nv == 2)out[1] = r[1] - r[3];+ } else if (nv <= 4) {+ float r[4];+ _mm_storeu_ps(r, _mm_sub_ps(acc4n, accm4n));+ _Pragma("GCC unroll 4") for (int v = 0; v < nv; v++) out[v] = r[v];} else {float r[8];_mm256_storeu_ps(r, _mm256_sub_ps(acc8, accm8));@@ -231,10 +249,8 @@ DOTN(q5k_dotn_body)(const struct janas_block_q5k *w,pm[v] = _mm_setzero_si128();if (v < nv) {const struct janas_block_q8k *xb = x + (size_t)v * xs + b;- __m128i bs_lo = _mm_loadu_si128((const __m128i *)xb->bsums);- __m128i bs_hi =- _mm_loadu_si128((const __m128i *)(xb->bsums + 8));- pm[v] = _mm_madd_epi16(mins16, _mm_hadd_epi16(bs_lo, bs_hi));+ pm[v] = _mm_madd_epi16(+ mins16, _mm_loadu_si128((const __m128i *)xb->bsums2));}}if (nv <= 2) {diff --git a/tests/test_quant.c b/tests/test_quant.cindex c72c2a6..760aba5 100644--- a/tests/test_quant.c+++ b/tests/test_quant.c@@ -903,6 +903,9 @@ static void q8k_ref(const float *x, struct janas_block_q8k *y, size_t n,s += y[b].qs[g * 16 + i];y[b].bsums[g] = (int16_t)s;}+ for (int k = 0; k < JANAS_QK / 32; k++)+ y[b].bsums2[k] =+ (int16_t)(y[b].bsums[2 * k] + y[b].bsums[2 * k + 1]);for (int k = 0; k < 8; k++) {float m = 0.0f;for (int i = 0; i < 32; i++)