Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Develop 2026-10-25 (2): the CPU's tiles transposed lighter - the next chunk's weights prefetched while a chunk's tiles compute, the transposition's loops unrolled, the sums of a prompt's block of 256 tokens at once (each chunk transposed once a block); on 32 vectors Q4_K 8.4 to 9.5 GB/s, Q3_K 5.6 to 6.4, IQ4_XS 7.8 to 9.0; a CPU prompt: Qwen3-30B-A3B UD-Q3_K_XL 116-118 to 121-123 tokens/s, Qwen3-4B Q4_K_M 115 to 121; the same bits

camauri7eb9dfe7ff0d
diff --git a/ChangeLog.md b/ChangeLog.md
index 2781f11..118ed5a 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -9,6 +9,7 @@
- **A mixture larger than the card: a tier of experts in its memory, the CPU computing the others**: on a card with memory of its own a mixture's whole pass no longer needs every expert copied to the card. It keeps a tier of slots in the card's memory - the room it has left, 85% of its memory less what Janas took and 1 GiB for the rest (`JANAS_GPU_TIER_MB=<n>` asks for n MiB), in pieces of 1 GiB that any driver allocates - and between two layers, where the CPU already sorts the router's choice, each chosen expert is either in the tier, or read into it in place of the one used least among sixteen (never one this layer counts on), or computed by the CPU from the layer's normalized input into its own rows, which the GPU then sums with the others in the router's order (`janas_m_expert_rows`, the CPU's expert kernels). Only the experts not in the tier are asked of the cache, so a tier hit never reads the disk. The arena is not copied to such a card any more, so the tier has its memory (before, a cache holding every expert was copied whole, and a copy that did not fit could leave pieces behind). `JANAS_GPU_TIER=0` turns the tier off. Checked as a card of its own on the development laptop's Arc, functional runs only: Qwen3-30B-A3B UD-Q3_K_XL with a tier of 1 GiB (480 of its 6,144 experts) and of 300 MB (140, a slot given up at nearly every layer) - sensible text, every pass whole on the GPU, no error under the validation layer with synchronization checks. The text is not the same to the bit with every tier: the two tiers' first 15 tokens are the same, then they part, as the run with drafts parts from the one without after 8-9 tokens - an expert computed by the CPU or by the GPU rounds differently (the GPU's expert products sum in their own order), and which computes it depends on what the tier holds.
- **Every fma() of the shaders one operation, as on the CPU: a mixture's tier no longer changes its answer**: glslang decorates a `precise` variable's multiplies and adds NoContraction but not its `fma()` calls, and GLSL.std.450's Fma without it may be done as a multiply and an add, two roundings - the Arc's driver did split some: the experts' activation (`expert_act.comp`, its e^x and reciprocal) came out a unit in the last place off the CPU's in about 1.5% of its values, which moved a block's scale in one expert in eighty. The shaders' SPIR-V now goes through `tools/spv_fma.c`, which decorates every Fma (built by `build.sh` with the build machine's compiler, `HOSTCC`, and writing the C array that `--vn` wrote). With that, and the CPU's experts quantizing the GPU's input with the same deterministic operations (`janas_q8k_quantize_det`), an expert computed by the CPU gives the GPU's bits: Qwen3-30B-A3B's logits on 160 tokens are the same to the bit with a tier of 300 MB, 1 GiB and every expert. Found with a probe that recomputed each GPU expert on the CPU (input, gate and up the same to the bit, the activation not). Every block's experts now go through the matvecs by default (`JANAS_GPU_MVID=16` as before: a prompt's blocks in `*_mmid`'s tiles), Q6_K_P's too (`q6kp_matvec.comp` with `ID`, its planes and level from the table): a prompt's GPU experts went through tiles that are not the CPU's bits and its other experts through the CPU, so the tokens with drafts and without parted after some ten (Qwen3-30B-A3B with a tier of 300 MB at the 11th, Qwen3.6-35B-A3B with one of 1 GiB at the 40th); now they are the same, 48 of 48 on both. The prompt as a card of its own on the laptop's Arc: 7% faster with every expert in the tier, 3% slower with 300 MB.
- **A mixture's whole pass on a card of its own 2.2 times as fast on a single token**: measured as a card of its own on the laptop's Arc (Qwen3-30B-A3B UD-Q3_K_XL, every expert in the tier, 160 tokens of fixed text one at a time, the second half): from 4.7 to 10.5 tokens/s; with a tier of 300 MB, the other experts on the CPU, from 3.1 to 8.5; Qwen3.6-35B-A3B Q4_K_M with a tier of 1 GiB 7.0 tokens/s. Four causes, found with the GPU's own timestamps: (1) between two layers the CPU's plan was read and written in six or seven copies of their own, each a submission waited for - now the choice of experts (and, with a tier, the experts' input) goes to the mirror at the end of a layer's commands and the tables come back at the start of the next one's, and the experts read into the tier and the CPU's rows go in one submission together (`janas_g_copies`); (2) the router's product went through `dense_mm`'s tiles of 64 rows by 64 vectors, three workgroups on a token - now `dense_mv.comp`, a workgroup a row, on up to 8 vectors (also the DeltaNet's beta and alpha, and qwen4exp's mixes, these not tried: the whole pass refuses them on a card of its own); (3) `moe_topk.comp` had one invocation a token search 128 logits nine times - now a workgroup a token, each search a reduction; the router from 1.07 to 0.11 ms a layer; (4) the experts' products on a few vectors went through `*_mmid`'s tiles of 16 vectors, nearly empty on a token (9 GB/s) - now through the matvec shaders of Q3_K, Q4_K and Q6_K with a constant `ID` that reads the weights, inputs and outputs from the same table (blocks of 16 tokens or fewer, `JANAS_GPU_MVID=<n>`, 0 never): gate and up from 55 to 26 ms a token. Every step checked: logits the same to the bit before and after the copies were moved; the same agreement with the CPU (145-150 of 160 tokens' most likely token, as Qwen3-4B's whole pass, 154); no error under the validation layer with synchronization checks; prompts as fast or faster (512 tokens 111.9 to 112.9 tokens/s, 120.7 with every block through the matvecs). With a tier holding every expert the tokens are the same with drafts and without; with a partial one they part after some tokens, as before: the CPU's experts do not round as the GPU's (open).
+- **The tiles' transposition lighter: its weights asked for ahead, its loops unrolled, a block's 256 tokens at once**: the transposition of a chunk of eight rows waited for memory (62% of its time on its loads, perf, Q4_K on 24 vectors) while the tiles that followed left memory idle: the next chunk's bytes are now prefetched while a chunk's tiles compute (`gemm_ahead`, not Q8_0's: 3-4% slower there), the transposition's loops are unrolled so its registers no longer go through the stack, and the tiles hold the sums of 256 tokens, a prompt's block on the CPU, not 64 (32 for the types in blocks of 32), so each chunk is transposed once a block instead of four to eight times. On 32 vectors (12 threads) Q4_K 8.4 to 9.5 GB/s, Q3_K 5.6 to 6.4, IQ4_XS 7.8 to 9.0, Q5_K 10.5 to 11.4; on 256 6-15%. A prompt on the CPU with the experts in memory: Qwen3-30B-A3B UD-Q3_K_XL (1024 tokens) 116.4-117.7 to 121.4-123.4 tokens/s, Qwen3-4B Q4_K_M (512) 115.2 to 121.2-121.4. The same bits (`test_quant`'s rows to 300 vectors, past a block of 256; a mutation of the blocks' index caught). `bench_kernels` takes up to 512 vectors.
- **Q2_K and the IQ types with grids as tiles** (IQ3_S, IQ3_XXS, IQ2_XXS, IQ2_XS, IQ2_S, IQ1_S, IQ1_M): through Q6_K's tiles, a group of 16 weights a scale; the IQ types' weights as signed bytes (level times sign; with AVX-VNNI plus 128, the offset taken off with the scales' sums of activations, with AVX2 by sign and abs), their scales 1 + 2 s, the block's d with its factor; Q2_K's values 0 to 3, its mins' sums apart. The same bits (`test_quant`'s 11 rows to 65 vectors with all eight, AVX2 and AVX-VNNI; mutations of the offset, of abs and of Q2_K's dmin caught). From 12 vectors; on 64 (12 threads) 1.6 to 2.6 times as fast: IQ1_M 0.6 to 1.6 GB/s with AVX-VNNI, IQ3_XXS 1.2 to 2.8, Q2_K 1.5 to 2.6; with AVX2 1.5 to 2.2 times.
- **Q2_0 as tiles, and rows that end inside a group of 256 by rows**: Q2_0's two-bit values (the weight plus 1) go through the tiles of the types in blocks of 32, a block of 64 two of them with its d in both, from 24 vectors with AVX-VNNI (on 64, 12 threads, 1.7 to 2.1 GB/s); with AVX2 its products on reordered activations stay, the tiles never faster. Rows that end inside a group of 256 - Qwen3.8-Flash-Next's down experts, 640 columns - no longer go one by one on many vectors: the whole groups of all the rows, then the tails padded 16 rows at a time, both through the rows' products and their tiles (`janas_dot_multi_rows_n`); the down matrix (2560 rows of 640, one core) 1.5 times as fast on 8 vectors, 2.4 on 64 with AVX-VNNI, 1.7 with AVX2. The same bits (`test_row_tails` on 33 vectors and 37 rows; mutations of Q2_0's offset and of the tails' sum caught).
- **IQ4_XS as tiles too**: its signed values from the IQ4 table transposed, a signed scale a 32-weight group; with AVX-VNNI the values plus 128 by vpdpbusd, a multiply by the scales, 128 times the scales' sums of activations taken off; with AVX2 sign and abs, the scales by madd; one fma a block as before, the same bits. From 12 vectors; on 64 with AVX-VNNI 2.4 times as fast (12 threads), with AVX2 39%. Qwen3.8-27B UD-Q4_K_XL (IQ4_XS where its rows are many), prompt of 256 tokens with every tile type, AVX-VNNI: 9.0 to 15.6 tokens/s. The tiles' scales are made with vectors (an 8 x 8 transpose of the rows' scale pairs): Q3_K's tiles 19% faster on 24 vectors.
diff --git a/src/llm/quant_dotn.h b/src/llm/quant_dotn.h
index 95022a3..067e80f 100644
--- a/src/llm/quant_dotn.h
+++ b/src/llm/quant_dotn.h
@@ -68,6 +68,9 @@ DOTN(hsum128_epi32)(__m128i s)
#ifndef PF_AHEAD
#define PF_AHEAD 2048
#endif
+#ifndef PF_GEMM_HINT
+#define PF_GEMM_HINT _MM_HINT_T0
+#endif
static inline DOTN_TARGET __attribute__((always_inline)) void
DOTN(prefetch_ahead)(const void *at, size_t size)
{
@@ -76,6 +79,34 @@ DOTN(prefetch_ahead)(const void *at, size_t size)
_mm_prefetch(p + i, _MM_HINT_T0);
}
+/*
+ * The tiles' weights asked for ahead: after a chunk of eight rows is
+ * transposed, the next chunk's bytes (the same rows' next blocks, or the
+ * next eight rows' first ones) are fetched while this one's tiles compute.
+ * Without it the transposition waited for memory and the tiles left it idle
+ * in turn: 62% of the transposition's time on its loads (perf, Q4_K on 24
+ * vectors). gb is the bytes of a group of 256 weights, chunk the groups
+ * transposed at a time.
+ */
+static inline DOTN_TARGET __attribute__((always_inline)) void
+DOTN(gemm_ahead)(const void *w, size_t row_bytes, size_t gb, size_t g,
+ size_t rows, size_t nb, size_t b0, size_t chunk)
+{
+ b0 += chunk;
+ if (b0 >= nb) {
+ b0 = 0;
+ g += 8;
+ if (g + 8 > rows)
+ return;
+ }
+ size_t len = (nb - b0 < chunk ? nb - b0 : chunk) * gb;
+ for (int j = 0; j < 8; j++) {
+ const char *p = (const char *)w + (g + j) * row_bytes + b0 * gb;
+ for (size_t i = 0; i < len; i += 64)
+ _mm_prefetch(p + i, PF_GEMM_HINT);
+ }
+}
+
/*
* Q4_K: the per-block integer-sum scheme. For each block and vector the
* integer sums are exact: isum = sum of scale x (weight . activation) over
@@ -796,7 +827,10 @@ K_MULTI_ROWS(q5k, struct janas_block_q5k)
#ifndef Q4KG_DEFINED
#define Q4KG_DEFINED
#define Q4KG_CHUNK 16 /* blocks transposed at a time: 23 KB */
-#define Q4KG_TOK 64 /* tokens whose sums are held at a time */
+/* tokens whose sums are held at a time: a prompt's block of 256 at once,
+ its rows transposed once (with 64 four times: bench_kernels on 256
+ vectors, 12 threads, Q5_K 1.6 to 1.75 GB/s, IQ4_XS 1.25 to 1.45) */
+#define Q4KG_TOK 256
struct q4kg_blk {
uint32_t qs[4][8][8]; /* [piece][dword][row] */
int16_t sc[8][16]; /* [sub-block][2 row + i]: its scale, twice */
@@ -869,20 +903,22 @@ DOTN(q4kg_fill)(const struct janas_block_q4k *const *w, size_t b0, int n,
struct q4kg_blk *k = bk + i;
for (int c = 0; c < 4; c++) {
__m256i r[8], t[8], u[8];
- for (int j = 0; j < 8; j++)
- r[j] = _mm256_loadu_si256(
- (const __m256i *)(w[j][b0 + i].qs + 32 * c));
- for (int j = 0; j < 8; j += 2) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++) r[j] =
+ _mm256_loadu_si256((const __m256i *)(w[j][b0 + i].qs + 32 * c));
+ _Pragma("GCC unroll 4") for (int j = 0; j < 8; j += 2)
+ {
t[j] = _mm256_unpacklo_epi32(r[j], r[j + 1]);
t[j + 1] = _mm256_unpackhi_epi32(r[j], r[j + 1]);
}
- for (int j = 0; j < 8; j += 4) {
+ _Pragma("GCC unroll 2") for (int j = 0; j < 8; j += 4)
+ {
u[j] = _mm256_unpacklo_epi64(t[j], t[j + 2]);
u[j + 1] = _mm256_unpackhi_epi64(t[j], t[j + 2]);
u[j + 2] = _mm256_unpacklo_epi64(t[j + 1], t[j + 3]);
u[j + 3] = _mm256_unpackhi_epi64(t[j + 1], t[j + 3]);
}
- for (int q = 0; q < 4; q++) {
+ _Pragma("GCC unroll 4") for (int q = 0; q < 4; q++)
+ {
_mm256_storeu_si256(
(__m256i *)k->qs[c][q],
_mm256_permute2x128_si256(u[q], u[q + 4], 0x20));
@@ -1028,6 +1064,8 @@ static DOTN_TARGET void DOTN(q4k_gemm)(const void *w, size_t row_bytes,
for (size_t b0 = 0; b0 < nb; b0 += Q4KG_CHUNK) {
int n = nb - b0 < Q4KG_CHUNK ? (int)(nb - b0) : Q4KG_CHUNK;
DOTN(q4kg_fill)(wr, b0, n, bk);
+ DOTN(gemm_ahead)(w, row_bytes, sizeof(struct janas_block_q4k),
+ g, rows, nb, b0, Q4KG_CHUNK);
size_t t0 = 0;
for (; t0 + Q4KG_T <= nt; t0 += Q4KG_T)
DOTN(q4kg_tile)(bk, n, b0, x + (v0 + t0) * xs, xs,
@@ -1515,17 +1553,20 @@ static inline DOTN_TARGET __attribute__((always_inline)) void
DOTN(gemm_t8)(__m256i r[8], uint32_t (*qs)[8])
{
__m256i t[8], u[8];
- for (int j = 0; j < 8; j += 2) {
+ _Pragma("GCC unroll 4") for (int j = 0; j < 8; j += 2)
+ {
t[j] = _mm256_unpacklo_epi32(r[j], r[j + 1]);
t[j + 1] = _mm256_unpackhi_epi32(r[j], r[j + 1]);
}
- for (int j = 0; j < 8; j += 4) {
+ _Pragma("GCC unroll 2") for (int j = 0; j < 8; j += 4)
+ {
u[j] = _mm256_unpacklo_epi64(t[j], t[j + 2]);
u[j + 1] = _mm256_unpackhi_epi64(t[j], t[j + 2]);
u[j + 2] = _mm256_unpacklo_epi64(t[j + 1], t[j + 3]);
u[j + 3] = _mm256_unpackhi_epi64(t[j + 1], t[j + 3]);
}
- for (int q = 0; q < 4; q++) {
+ _Pragma("GCC unroll 4") for (int q = 0; q < 4; q++)
+ {
_mm256_storeu_si256((__m256i *)qs[q],
_mm256_permute2x128_si256(u[q], u[q + 4], 0x20));
_mm256_storeu_si256((__m256i *)qs[q + 4],
@@ -1603,7 +1644,8 @@ DOTN(q6kg_fill_q6k)(const struct janas_block_q6k *const *w, size_t b0, int n,
struct q6kg_blk *k = bk + i;
for (int hn = 0; hn < 2; hn++) {
__m256i q[4][8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
const struct janas_block_q6k *bl = &w[j][b0 + i];
__m256i ql0 =
_mm256_loadu_si256((const __m256i *)(bl->ql + 64 * hn));
@@ -1631,7 +1673,8 @@ DOTN(q6kg_fill_q6k)(const struct janas_block_q6k *const *w, size_t b0, int n,
DOTN(gemm_t8)(q[part], k->qs + 8 * (4 * hn + part));
}
__m128i sc[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
sc[j] = _mm_loadu_si128((const __m128i *)w[j][b0 + i].scales);
k->d[j] = _cvtsh_ss(w[j][b0 + i].d);
}
@@ -1652,7 +1695,8 @@ DOTN(q6kg_fill_q6kp)(const struct janas_block_q6kp *const *w,
size_t b = b0 + (size_t)i;
for (int hn = 0; hn < 2; hn++) {
__m256i q[4][8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
__m256i h =
_mm256_loadu_si256((const __m256i *)(w[j][b].q + 32 * hn));
__m256i m1 =
@@ -1684,7 +1728,8 @@ DOTN(q6kg_fill_q6kp)(const struct janas_block_q6kp *const *w,
DOTN(gemm_t8)(q[part], k->qs + 8 * (4 * hn + part));
}
__m128i sc[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
sc[j] = _mm_loadu_si128((const __m128i *)w[j][b].scales);
k->d[j] = _cvtsh_ss(w[j][b].d) * 0.5f;
}
@@ -1705,7 +1750,8 @@ DOTN(q6kg_fill_q3k)(const struct janas_block_q3k *const *w, size_t b0, int n,
for (int kk = 0; kk < 8; kk++) {
const int hn = kk / 4, j = kk % 4;
__m256i u[8];
- for (int r = 0; r < 8; r++) {
+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++)
+ {
__m256i qs =
_mm256_loadu_si256((const __m256i *)(w[r][b].qs + 32 * hn));
__m256i hm = _mm256_loadu_si256((const __m256i *)w[r][b].hmask);
@@ -1719,7 +1765,8 @@ DOTN(q6kg_fill_q3k)(const struct janas_block_q3k *const *w, size_t b0, int n,
DOTN(gemm_t8)(u, k->qs + 8 * kk);
}
__m128i sc[8];
- for (int r = 0; r < 8; r++) {
+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++)
+ {
sc[r] = DOTN(q3k_scales16)(w[r][b].scales);
k->d[r] = _cvtsh_ss(w[r][b].d);
}
@@ -1905,6 +1952,11 @@ DOTN(q6k_gemm_body)(const void *w, size_t row_bytes, const uint8_t *p1,
DOTN(q6kg_fill_q6kp)(
(const struct janas_block_q6kp *const *)wr, r1, r0, b0,
n, bk, L);
+ DOTN(gemm_ahead)(w, row_bytes,
+ L == 0 ? sizeof(struct janas_block_q6k)
+ : L == 4 ? sizeof(struct janas_block_q3k)
+ : row_bytes / nb,
+ g, rows, nb, b0, Q6KG_CHUNK);
DOTN(q6kg_tiles)(bk, n, b0, x + v0 * xs, xs, nt, af, af, L);
}
for (size_t t = 0; t < nt; t++)
@@ -2073,7 +2125,8 @@ DOTN(iqkg_fill)(const int TY, const uint8_t *const *w, size_t b0, int n,
const __m256i three = _mm256_set1_epi8(3);
for (int kk = 0; kk < 8; kk++) {
__m256i u[8];
- for (int r = 0; r < 8; r++) {
+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++)
+ {
const struct janas_block_q2k *bl =
(const struct janas_block_q2k *)(w[r] + b * size);
__m256i q = _mm256_loadu_si256(
@@ -2084,7 +2137,8 @@ DOTN(iqkg_fill)(const int TY, const uint8_t *const *w, size_t b0, int n,
DOTN(gemm_t8)(u, k->qs + 8 * kk);
}
__m128i sc[8], mn[8];
- for (int r = 0; r < 8; r++) {
+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++)
+ {
const struct janas_block_q2k *bl =
(const struct janas_block_q2k *)(w[r] + b * size);
__m128i m8 = _mm_loadu_si128((const __m128i *)bl->scales);
@@ -2099,7 +2153,8 @@ DOTN(iqkg_fill)(const int TY, const uint8_t *const *w, size_t b0, int n,
int8_t s[8][16];
for (int kk = 0; kk < 8; kk++) {
__m256i u[8];
- for (int r = 0; r < 8; r++) {
+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++)
+ {
u[r] = DOTN(iqkg_group)(TY, w[r] + b * size, kk, s[r]);
#ifdef DOTN_HAS_VNNI
u[r] = _mm256_xor_si256(u[r], _mm256_set1_epi8((char)0x80));
@@ -2108,7 +2163,8 @@ DOTN(iqkg_fill)(const int TY, const uint8_t *const *w, size_t b0, int n,
DOTN(gemm_t8)(u, k->qs + 8 * kk);
}
__m128i sc[8];
- for (int r = 0; r < 8; r++) {
+ _Pragma("GCC unroll 8") for (int r = 0; r < 8; r++)
+ {
sc[r] = _mm_loadu_si128((const __m128i *)s[r]);
k->d[r] = DOTN(iqkg_d)(TY, w[r] + b * size);
}
@@ -2136,6 +2192,8 @@ DOTN(iqk_gemm_t)(const int TY, const void *w, size_t row_bytes,
for (size_t b0 = 0; b0 < nb; b0 += Q6KG_CHUNK) {
int n = nb - b0 < Q6KG_CHUNK ? (int)(nb - b0) : Q6KG_CHUNK;
DOTN(iqkg_fill)(TY, wr, b0, n, bk);
+ DOTN(gemm_ahead)(w, row_bytes, DOTN(iqkg_size)(TY), g, rows, nb,
+ b0, Q6KG_CHUNK);
DOTN(q6kg_tiles)(bk, n, b0, x + v0 * xs, xs, nt, af, am, L);
}
for (size_t t = 0; t < nt; t++)
@@ -2200,7 +2258,8 @@ DOTN(q5kg_fill)(const struct janas_block_q5k *const *w, size_t b0, int n,
size_t b = b0 + (size_t)i;
for (int c = 0; c < 4; c++) {
__m256i lo[8], hi[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
__m256i q4 =
_mm256_loadu_si256((const __m256i *)(w[j][b].qs + 32 * c));
__m256i qh = _mm256_loadu_si256((const __m256i *)w[j][b].qh);
@@ -2219,7 +2278,8 @@ DOTN(q5kg_fill)(const struct janas_block_q5k *const *w, size_t b0, int n,
DOTN(gemm_t8)(hi, k->qs + 8 * (2 * c + 1));
}
__m128i sv[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
const struct janas_block_q5k *blk = &w[j][b];
uint32_t u[4];
unpack_scales_u32(blk->scales, u);
@@ -2335,6 +2395,8 @@ static DOTN_TARGET void DOTN(q5k_gemm)(const void *w, size_t row_bytes,
for (size_t b0 = 0; b0 < nb; b0 += Q5KG_CHUNK) {
int n = nb - b0 < Q5KG_CHUNK ? (int)(nb - b0) : Q5KG_CHUNK;
DOTN(q5kg_fill)(wr, b0, n, bk);
+ DOTN(gemm_ahead)(w, row_bytes, sizeof(struct janas_block_q5k),
+ g, rows, nb, b0, Q5KG_CHUNK);
const struct janas_block_q8k *xv = x + v0 * xs;
size_t t0 = 0;
for (; t0 + Q4KG_T <= nt; t0 += Q4KG_T)
@@ -2476,7 +2538,8 @@ DOTN(b32g_fill)(const int TY, const uint8_t *const *w, size_t b0, int n,
/* a block of 64: positions 2 kb2 and 2 kb2 + 1 */
for (int kb2 = 0; kb2 < 4; kb2++) {
__m256i lo[8], hi[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
const struct janas_block_q2_0 *blk =
(const struct janas_block_q2_0 *)w[j] + (b0 + i) * 4 +
kb2;
@@ -2490,7 +2553,8 @@ DOTN(b32g_fill)(const int TY, const uint8_t *const *w, size_t b0, int n,
}
for (int kb = 0; kb < 8; kb++) {
__m256i r[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
const uint8_t *blk = w[j] + ((b0 + i) * 8 + kb) * size;
r[j] = DOTN(b32g_unpack)(TY, blk);
if (TY == JANAS_MXFP4) {
@@ -2632,7 +2696,9 @@ DOTN(b32g_hsum8)(const float *a)
_mm256_add_ps(_mm256_add_ps(p1, p5), _mm256_add_ps(p3, p7)));
}
-#define B32G_TOK 32 /* tokens whose sums are held at a time: 16 KB */
+/* tokens whose sums are held at a time, as Q4KG_TOK: 64 KB (with 32, Q8_0
+ on 256 vectors 2.0 against 2.3 GB/s) */
+#define B32G_TOK 256
static inline DOTN_TARGET __attribute__((always_inline)) void
DOTN(b32_gemm_t)(const int TY, const void *w, size_t row_bytes,
@@ -2654,6 +2720,13 @@ DOTN(b32_gemm_t)(const int TY, const void *w, size_t row_bytes,
for (size_t b0 = 0; b0 < nb; b0 += B32G_CHUNK) {
int n = nb - b0 < B32G_CHUNK ? (int)(nb - b0) : B32G_CHUNK;
DOTN(b32g_fill)(TY, wr, b0, n, bk);
+ /* not Q8_0's: 3-4% slower on 24-32 vectors */
+ if (TY != JANAS_Q8_0)
+ DOTN(gemm_ahead)(w, row_bytes,
+ TY == JANAS_Q2_0
+ ? 4 * sizeof(struct janas_block_q2_0)
+ : 8 * DOTN(b32g_size)(TY),
+ g, rows, nb, b0, B32G_CHUNK);
const struct janas_block_q8k *xv = x + v0 * xs;
size_t t0 = 0;
for (; t0 + 4 <= nt; t0 += 4)
@@ -2725,7 +2798,8 @@ DOTN(iq4xsg_fill)(const struct janas_block_iq4xs *const *w, size_t b0, int n,
size_t b = b0 + (size_t)i;
for (int g = 0; g < 8; g++) {
__m256i r[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
r[j] = DOTN(iq4_unpack32)(w[j][b].qs + 16 * g, levels);
#ifdef DOTN_HAS_VNNI
r[j] = _mm256_xor_si256(r[j], _mm256_set1_epi8((char)0x80));
@@ -2734,7 +2808,8 @@ DOTN(iq4xsg_fill)(const struct janas_block_iq4xs *const *w, size_t b0, int n,
DOTN(gemm_t8)(r, k->qs + 8 * g);
}
__m128i sv[8];
- for (int j = 0; j < 8; j++) {
+ _Pragma("GCC unroll 8") for (int j = 0; j < 8; j++)
+ {
int8_t ls[16];
for (int g = 0; g < 8; g++)
ls[g] = ls[8 + g] = (int8_t)iq4xs_scale(&w[j][b], g);
@@ -2841,6 +2916,8 @@ static DOTN_TARGET void DOTN(iq4xs_gemm)(const void *w, size_t row_bytes,
for (size_t b0 = 0; b0 < nb; b0 += Q5KG_CHUNK) {
int n = nb - b0 < Q5KG_CHUNK ? (int)(nb - b0) : Q5KG_CHUNK;
DOTN(iq4xsg_fill)(wr, b0, n, bk);
+ DOTN(gemm_ahead)(w, row_bytes, sizeof(struct janas_block_iq4xs),
+ g, rows, nb, b0, Q5KG_CHUNK);
const struct janas_block_q8k *xv = x + v0 * xs;
size_t t0 = 0;
for (; t0 + Q4KG_T <= nt; t0 += Q4KG_T)
diff --git a/tests/bench_kernels.c b/tests/bench_kernels.c
index edf303b..e2e7737 100644
--- a/tests/bench_kernels.c
+++ b/tests/bench_kernels.c
@@ -15,9 +15,9 @@
* ordinary memory aligned to 2 MB and collapsed into huge pages: the CPU's
* speed on each, against malloc's (13 Oct 2026).
* BENCH_TYPES=Q4_K,Q2_0 measures the types named alone; BENCH_NV=8,16,64
- * the vectors a product takes (1,2,4,8 by default, 64 at most): where the
- * prompt's tiles (JANAS_CPU_TILES) pay against the rows' products.
- * Defaults: 256 MiB of weights per type, 12 threads on CPUs 0-11; the
+ * the vectors a product takes (1,2,4,8 by default, 512 at most: a prompt's
+ * block): where the prompt's tiles (JANAS_CPU_TILES) pay against the rows'
+ * products. Defaults: 256 MiB of weights per type, 12 threads on CPUs 0-11; the
* single core is first_cpu (default 2).
* The weights are random bytes below 0x7c, so that every fp16 scale is a
* finite number; the speed of a kernel does not depend on the values.
@@ -142,7 +142,7 @@ int main(int argc, char **argv)
fprintf(stderr, "bench_kernels: no thread pool\n");
return 1;
}
- enum { MAX_NV = 64 };
+ enum { MAX_NV = 512 };
size_t nvs[8] = {1, 2, 4, 8}, n_nv = 4;
const char *ev = getenv("BENCH_NV");
if (ev) {
@@ -157,7 +157,7 @@ int main(int argc, char **argv)
break;
}
if (!n_nv) {
- fprintf(stderr, "bench_kernels: BENCH_NV lists 1 to 64\n");
+ fprintf(stderr, "bench_kernels: BENCH_NV lists 1 to 512\n");
return 1;
}
}
diff --git a/tests/test_quant.c b/tests/test_quant.c
index 4d512c8..2d9e522 100644
--- a/tests/test_quant.c
+++ b/tests/test_quant.c
@@ -1140,8 +1140,9 @@ static void random_blocks(int type, unsigned char *w, size_t n)
static void test_multi_rows_long(void)
{
/* 11 rows: Q4_K's tiles take eight (q4k_gemm), the products the
- rest; 65 vectors: past the tiles' 64 tokens at a time */
- enum { ROWS = 11, NV = 65 };
+ rest; up to 65 vectors, then 300: past the tiles' 256 tokens at a
+ time */
+ enum { ROWS = 11, NV = 300 };
static const size_t lens[] = {2, 48, 130};
static const int types[] = {
JANAS_Q2_0, JANAS_Q4_K, JANAS_Q5_K, JANAS_Q6_K, JANAS_Q8_0,
@@ -1211,7 +1212,10 @@ static void test_multi_rows_long(void)
xf[i] = rndf(-1.0f, 1.0f);
janas_q8k_quantize(xf, x + v * nb, n);
}
- for (int nv = 1; nv <= NV; nv += nv < 9 ? 1 : 8) {
+ for (int nv = 1; nv <= NV; nv = nv < 9 ? nv + 1
+ : nv < 65 ? nv + 8
+ : nv == 65 ? NV
+ : NV + 1) {
janas_dot_multi_rows(t, w, rb, x, nb, (size_t)nv, nb, ROWS, out,
ROWS);
for (int v = 0; v < nv; v++)