Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-14: MXFP4 on the GPU, and a mixture's experts on an integrated GPU again when its cache holds more than 4 GiB of them

camauriafb874e4da84
diff --git a/ChangeLog.md b/ChangeLog.md
index 9ef8ceb..82dc908 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,12 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-10-14] - MXFP4 on the GPU, and the experts of a large cache back on it
+
+- **A mixture's experts reach the GPU again when the expert cache holds more than 4 GiB of them.** Since the release of 13 October the memory the GPU reads is given to it in regions that start on 2 MB boundaries, each overlapping the one before, and the overlap was not counted in the 4 GiB a region may take: past the first, a region could come to a little more, which the GPU refused, and the expert cache was then not given to it at all. On an integrated GPU a mixture's experts stayed on the CPU, in the fast prompt and in the block read whole on the GPU, whenever the cache held more than about 4 GiB of experts, depending on the size of an expert: Qwen3.6-35B-A3B's MXFP4_MOE file beyond 4 GiB, its Q4_K_M at six bits beyond 8 GiB. Each region counts its overlap now, and `tests/test_import_regions` checks them.
+- **MXFP4 on the GPU**, which only the CPU read since this morning's MXFP4 products: the products of one vector to a few, to the same bits as the CPU; the fast prompt's tiles; and a mixture's MXFP4 experts in one launch per layer, the layer read whole on the GPU. noctrex's Qwen3.6-35B-A3B-MXFP4_MOE read in blocks of 100 with the GPU, against llama.cpp on the same file: 200 of 200 most likely tokens agree, mean logit difference 0.106 (on the CPU, token by token: 199, 0.105).
+
+
## [2026-10-14] - An integrated GPU no longer kept awake for single tokens
- **The GPU is kept awake only for the passes it takes.** On an integrated GPU a reply's single tokens stay on the CPU, but a configuration with the GPU kept it awake for them too - an empty dispatch every half millisecond - and on a laptop the awake GPU takes the power the cores need: janas-bench's "+ GPU" rows wrote a reply slower than the CPU alone, 10.5 tokens/s against 17.3 on a Radeon 780M (issue #16, reported by dlahoza), 2-5% slower on the development laptop's Arc. Now they write as fast (Qwen3-4B, 20, 12 and 6 threads: 28.5/28.1, 28.0/28.0 and 26.3/26.3 tokens/s with "+ GPU", 28.5/28.3, 28.1/28.0 and 26.3/26.3 without, two alternated runs). A chat's own choices were not affected: the tuner gives single tokens to the GPU only where it takes them, a discrete GPU. janas-bench's note says that the single tokens are the CPU's in every row where so.
diff --git a/docs/features.md b/docs/features.md
index 0f8ca3c..36b26c9 100644
--- a/docs/features.md
+++ b/docs/features.md
@@ -21,7 +21,7 @@
- **`janas-bench`:** measures the machine, fills its profile and writes a report.
- **Faster replies** from drafts the model verifies, so the text is exactly the one it would have written: from the model's own multi-token prediction block where it has one, from the conversation where it does not, or from a small model of the same family given with `--draft`.
- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. A reply's tokens are identical with and without it; a prompt is read with the fast prompt (next).
-- **The fast prompt**, on by default since 10 October 2026: a prompt's blocks go to the GPU's tiled products (Q4_K, Q5_K, Q6_K, Q8_0, Q3_K, IQ4_XS, IQ3_S, IQ4_NL, the experts of a mixture in one launch per layer, f32 and bf16 weights), and on the dense Qwen3 and Qwen3.5 models the whole block - every layer, attention, Qwen3.5's recurrent layers - goes to the GPU in one submission while the CPU rests, and on the Qwen3 and Qwen3.6 mixtures a layer a submission, the CPU only sorting the router's choice and reading the next layer's experts ahead (where the expert cache holds half the experts or more; `JANAS_GPU_MOE=0` or `1` decides instead): on a laptop the CPU working beside the GPU takes the power the GPU would use. They round the sums group by group, so a prompt is not read to the bit as the CPU reads it; against the references the most likely token agrees as often as the exact way (within 2-3 of 256 on every model measured, the mean difference of the scores within 2%). On the development laptop Qwen3-4B reads the table's prompt (below) at 220 tok/s with it and at 85 without the GPU, and every model compared but one reads faster than llama.cpp does on the same GPU (the exception: unsloth's 3-bit Qwen3-30B, whose Q3_K experts stay on the CPU). `JANAS_GPU_FAST_PROMPT=0` reads them the exact way.
+- **The fast prompt**, on by default since 10 October 2026: a prompt's blocks go to the GPU's tiled products (Q4_K, Q5_K, Q6_K, Q8_0, Q3_K, IQ4_XS, IQ3_S, IQ4_NL, MXFP4, the experts of a mixture in one launch per layer, f32 and bf16 weights), and on the dense Qwen3 and Qwen3.5 models the whole block - every layer, attention, Qwen3.5's recurrent layers - goes to the GPU in one submission while the CPU rests, and on the Qwen3 and Qwen3.6 mixtures a layer a submission, the CPU only sorting the router's choice and reading the next layer's experts ahead (where the expert cache holds half the experts or more; `JANAS_GPU_MOE=0` or `1` decides instead): on a laptop the CPU working beside the GPU takes the power the GPU would use. They round the sums group by group, so a prompt is not read to the bit as the CPU reads it; against the references the most likely token agrees as often as the exact way (within 2-3 of 256 on every model measured, the mean difference of the scores within 2%). On the development laptop Qwen3-4B reads the table's prompt (below) at 220 tok/s with it and at 85 without the GPU, and every model compared but one reads faster than llama.cpp does on the same GPU (the exception: unsloth's 3-bit Qwen3-30B, whose Q3_K experts stay on the CPU). `JANAS_GPU_FAST_PROMPT=0` reads them the exact way.
- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even. `JANAS_PREFETCH_PREFILL=0` turns it off.
- **Light experts not in memory are left out** when the expert cache holds less than a fifth of the experts: an expert the router weighs below 0.8/k of a token (0.95/k under a tenth), and that would have to be read from disk, is not read and not computed, and the token's other experts take its share — the cache-conditional routing of Skliar et al. (TMLR 2025). Until 8 October 2026 the engine dropped the lightest experts whether in memory or not (six of ten, or eight). Qwen3-Next-80B, four bits, 400 tokens against llama.cpp, most likely token agreeing and tokens/s: with a 2 GiB cache all ten 393 at 11.0, six 383 at 20.9, now 380 at 26.4; with 4 GiB 393 at 20.7, six 383 at 27.6, now 384 at 27.7, with the mean logit difference 0.36 against 0.53; on Italian prose with 8 GiB 382 at 18.7, eight 361 at 21.4, now 372 at 23.7. On Qwen3-30B-A3B and Qwen3.6-35B-A3B with caches of 9% and 18% of their experts (1.5-3.5 GiB), on code and Italian prose, it was faster than the old way every time, by 9-37% (Qwen3-30B-A3B with 3 GiB: 28.7 tokens/s against 24.4, 381 tokens agreeing of 400 against 377), with a mean logit difference as low or lower in seven of the eight; it agreed on fewer most likely tokens in two (Italian, 64 tokens: 47 against 54 on the 30B with 3 GiB). `/experts n` (or `0`) asks for a number of experts and turns it off.
- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU) for a reply's tokens (and for a prompt with `JANAS_GPU_FAST_PROMPT=0`). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 61 (48 until 9 October 2026, when its unpacking went through the stack), IQ3_S 25, against the 81 GB/s the memory gives (MXFP4, since 14 October 2026, at 66 GB/s where IQ4_NL reads at 73 in the same run: 4% fewer weights a second, a block of it being 17 bytes to IQ4_NL's 18); and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.
diff --git a/src/llm/gpu.c b/src/llm/gpu.c
index 857e393..021a534 100644
--- a/src/llm/gpu.c
+++ b/src/llm/gpu.c
@@ -33,6 +33,7 @@
#include "iq4xs_mmq.h" /* janas_spv_iq4xs_mmq */
#include "iq3s_mmq.h" /* janas_spv_iq3s_mmq */
#include "iq4nl_mmq.h" /* janas_spv_iq4nl_mmq */
+#include "mxfp4_mmid.h" /* janas_spv_mxfp4_mmid */
#include "q80_mmq.h" /* janas_spv_q80_mmq */
#include "q5k_mmq.h" /* janas_spv_q5k_mmq */
#include "dense_mm.h" /* janas_spv_dense_mm */
@@ -537,7 +538,10 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)
&g->pipe80, 0)) != VK_SUCCESS ||
(r = janas_g_make_pipeline(g, janas_spv_expert_act,
sizeof(janas_spv_expert_act), &g->sma,
- &g->pipea, 0)) != VK_SUCCESS) {
+ &g->pipea, 0)) != VK_SUCCESS ||
+ (r = janas_g_make_pipeline(g, janas_spv_expert_act,
+ sizeof(janas_spv_expert_act), &g->sma32,
+ &g->pipea32, 1)) != VK_SUCCESS) {
set_err(err, err_len, "cannot create the shaders", r);
goto fail;
}
@@ -571,6 +575,9 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)
(r = make_pipeline(g, janas_spv_iq4nl_mmq, sizeof(janas_spv_iq4nl_mmq),
&g->sm_mmn4, &g->pipe_mmn4, 0, NV_PER_GROUP,
g->sg)) != VK_SUCCESS ||
+ (r = make_pipeline(g, janas_spv_iq4nl_mmq, sizeof(janas_spv_iq4nl_mmq),
+ &g->sm_mmmx, &g->pipe_mmmx, 1, NV_PER_GROUP,
+ g->sg)) != VK_SUCCESS ||
(r = make_pipeline(g, janas_spv_q80_mmq, sizeof(janas_spv_q80_mmq),
&g->sm_mm80, &g->pipe_mm80, 0, NV_PER_GROUP,
g->sg)) != VK_SUCCESS ||
@@ -591,7 +598,10 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)
VK_SUCCESS ||
(r = make_pipeline(g, janas_spv_q6k_mmid, sizeof(janas_spv_q6k_mmid),
&g->sm_id6s, &g->pipe_id6s, 0, NV_PER_GROUP,
- g->sg)) != VK_SUCCESS) {
+ g->sg)) != VK_SUCCESS ||
+ (r = make_pipeline(
+ g, janas_spv_mxfp4_mmid, sizeof(janas_spv_mxfp4_mmid), &g->sm_idmx,
+ &g->pipe_idmx, 0, NV_PER_GROUP, g->sg)) != VK_SUCCESS) {
set_err(err, err_len, "cannot create the shaders", r);
goto fail;
}
@@ -613,7 +623,8 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)
{janas_spv_b32_matvec, sizeof(janas_spv_b32_matvec), 3},
{janas_spv_iq4xs_matvec, sizeof(janas_spv_iq4xs_matvec), 0},
{janas_spv_q3k_matvec, sizeof(janas_spv_q3k_matvec), 0},
- {janas_spv_iq3s_matvec, sizeof(janas_spv_iq3s_matvec), 0}};
+ {janas_spv_iq3s_matvec, sizeof(janas_spv_iq3s_matvec), 0},
+ {janas_spv_b32_matvec, sizeof(janas_spv_b32_matvec), 4}};
for (int i = 0; i < N_PROD; i++)
if ((r = janas_g_make_pipeline_nv(g, one[i].code, one[i].bytes,
&g->pm1[i], &g->pp1[i],
@@ -701,7 +712,7 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)
goto fail;
}
}
- for (uint32_t k = 0; k < 4; k++)
+ for (uint32_t k = 0; k < N_B32; k++)
if ((r = janas_g_make_pipeline(
g, janas_spv_b32_matvec, sizeof(janas_spv_b32_matvec),
&g->sm32[k], &g->pipe32[k], k)) != VK_SUCCESS) {
@@ -898,6 +909,8 @@ void janas_gpu_destroy(struct janas_gpu *g)
g->vkDestroyPipeline(g->dev, g->pipe_mmi3, NULL);
if (g->pipe_mmn4)
g->vkDestroyPipeline(g->dev, g->pipe_mmn4, NULL);
+ if (g->pipe_mmmx)
+ g->vkDestroyPipeline(g->dev, g->pipe_mmmx, NULL);
if (g->pipe_mm80)
g->vkDestroyPipeline(g->dev, g->pipe_mm80, NULL);
if (g->pipe_mm5)
@@ -910,11 +923,15 @@ void janas_gpu_destroy(struct janas_gpu *g)
if (g->sm_dm[i])
g->vkDestroyShaderModule(g->dev, g->sm_dm[i], NULL);
}
- for (int i = 0; i < 3; i++) {
- VkPipeline pp = i == 2 ? g->pipe_id6s
- : i ? g->pipe_id6
- : g->pipe_id4;
- VkShaderModule ss = i == 2 ? g->sm_id6s : i ? g->sm_id6 : g->sm_id4;
+ for (int i = 0; i < 4; i++) {
+ VkPipeline pp = i == 3 ? g->pipe_idmx
+ : i == 2 ? g->pipe_id6s
+ : i ? g->pipe_id6
+ : g->pipe_id4;
+ VkShaderModule ss = i == 3 ? g->sm_idmx
+ : i == 2 ? g->sm_id6s
+ : i ? g->sm_id6
+ : g->sm_id4;
if (pp)
g->vkDestroyPipeline(g->dev, pp, NULL);
if (ss)
@@ -930,6 +947,8 @@ void janas_gpu_destroy(struct janas_gpu *g)
g->vkDestroyShaderModule(g->dev, g->sm_mmi3, NULL);
if (g->sm_mmn4)
g->vkDestroyShaderModule(g->dev, g->sm_mmn4, NULL);
+ if (g->sm_mmmx)
+ g->vkDestroyShaderModule(g->dev, g->sm_mmmx, NULL);
if (g->sm_mm80)
g->vkDestroyShaderModule(g->dev, g->sm_mm80, NULL);
if (g->sm_mm6)
@@ -944,7 +963,7 @@ void janas_gpu_destroy(struct janas_gpu *g)
g->vkDestroyShaderModule(g->dev, g->smi3, NULL);
if (g->smx4)
g->vkDestroyShaderModule(g->dev, g->smx4, NULL);
- for (int k = 0; k < 4; k++) {
+ for (int k = 0; k < N_B32; k++) {
if (g->pipe32[k])
g->vkDestroyPipeline(g->dev, g->pipe32[k], NULL);
if (g->sm32[k])
@@ -954,6 +973,10 @@ void janas_gpu_destroy(struct janas_gpu *g)
g->vkDestroyPipeline(g->dev, g->pipea, NULL);
if (g->sma)
g->vkDestroyShaderModule(g->dev, g->sma, NULL);
+ if (g->pipea32)
+ g->vkDestroyPipeline(g->dev, g->pipea32, NULL);
+ if (g->sma32)
+ g->vkDestroyShaderModule(g->dev, g->sma32, NULL);
if (g->sm6)
g->vkDestroyShaderModule(g->dev, g->sm6, NULL);
if (g->sm6p)
@@ -1343,6 +1366,7 @@ static int b32_kind(int type)
: type == JANAS_IQ4_NL ? 1
: type == JANAS_Q4_1 ? 2
: type == JANAS_Q5_1 ? 3
+ : type == JANAS_MXFP4 ? 4
: -1;
}
@@ -1488,14 +1512,14 @@ void janas_g_push_addr(uint32_t out[2], VkDeviceAddress a)
}
/* The index of a type's product shader in the single-vector table, -1 if
- none: Q4_K, Q6_K, Q6_K_P, Q5_K, Q8_0, the four of b32 (Q4_0, IQ4_NL,
- Q4_1, Q5_1), IQ4_XS, Q3_K, IQ3_S. */
+ none: Q4_K, Q6_K, Q6_K_P, Q5_K, Q8_0, four of b32 (Q4_0, IQ4_NL, Q4_1,
+ Q5_1), IQ4_XS, Q3_K, IQ3_S, and b32's MXFP4. */
static int prod_index(int type)
{
- static const int types[N_PROD] = {JANAS_Q4_K, JANAS_Q6_K, JANAS_Q6_K_P,
- JANAS_Q5_K, JANAS_Q8_0, JANAS_Q4_0,
- JANAS_IQ4_NL, JANAS_Q4_1, JANAS_Q5_1,
- JANAS_IQ4_XS, JANAS_Q3_K, JANAS_IQ3_S};
+ static const int types[N_PROD] = {
+ JANAS_Q4_K, JANAS_Q6_K, JANAS_Q6_K_P, JANAS_Q5_K, JANAS_Q8_0,
+ JANAS_Q4_0, JANAS_IQ4_NL, JANAS_Q4_1, JANAS_Q5_1, JANAS_IQ4_XS,
+ JANAS_Q3_K, JANAS_IQ3_S, JANAS_MXFP4};
for (int i = 0; i < N_PROD; i++)
if (types[i] == type)
return i;
@@ -1586,7 +1610,7 @@ static int vectors_first(const struct janas_gpu *g, VkPipeline p)
p == g->pipe_mv || p == g->pipe_mvd || p == g->pipe_mvd6 ||
p == g->pipe_mvd5 || p == g->pipe_mvdx4)
return 1;
- for (int k = 0; k < 4; k++)
+ for (int k = 0; k < N_B32; k++)
if (p == g->pipe32[k])
return 1;
for (int i = 0; i < N_PROD; i++)
@@ -1602,6 +1626,7 @@ static int fast_type(const struct janas_matvec_task *t)
t->type == JANAS_Q8_0 || t->type == JANAS_Q5_K ||
t->type == JANAS_Q3_K || t->type == JANAS_IQ4_XS ||
t->type == JANAS_IQ3_S || t->type == JANAS_IQ4_NL ||
+ t->type == JANAS_MXFP4 ||
(t->type == JANAS_Q6_K_P && plane_level(t) == 3);
}
@@ -1705,6 +1730,11 @@ int janas_g_record_task(struct janas_gpu *g, VkPipeline *bound,
want = g->pipe_mmn4;
nvg = 64;
rows_wg = 64;
+ } else if (t->type == JANAS_MXFP4 && nv >= JANAS_GPU_FAST_N &&
+ g->pipe_mmmx && mm_on()) {
+ want = g->pipe_mmmx;
+ nvg = 64;
+ rows_wg = 64;
} else if (t->type == JANAS_Q5_K && nv >= JANAS_GPU_FAST_N && g->pipe_mm5 &&
mm_on()) {
want = g->pipe_mm5;
@@ -2008,16 +2038,30 @@ static size_t experts_pick(const struct janas_matvec_task *t, size_t n,
return maxnv;
}
+/* A group experts_id leaves to the CPU, and why, with JANAS_GPU_VERBOSE=2:
+ -1. */
+static int experts_off(const struct janas_gpu *g, size_t n, int type,
+ const char *why)
+{
+ if (g->verbose > 1)
+ fprintf(stderr,
+ "experts_id: a group of %zu (type %d first) not taken: %s\n", n,
+ type, why);
+ return -1;
+}
+
static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
const struct janas_matvec_task *t, size_t n)
{
- if (!mm_on() || g->busy || g->broken || n < 2 || !g->pipe_id4 ||
- n * 48 > g->ts.bytes || n > 2 * 128 + 8)
+ if (!mm_on() || g->busy || g->broken || n < 2 || !g->pipe_id4)
return -1;
int type = t[0].type;
+ if (n * 48 > g->ts.bytes || n > 2 * 128 + 8)
+ return experts_off(g, n, type, "more products than its table");
if (!(type == JANAS_Q4_K || type == JANAS_Q6_K ||
+ (type == JANAS_MXFP4 && g->pipe_idmx) ||
(type == JANAS_Q6_K_P && plane_level(&t[0]) == 3)))
- return -1;
+ return experts_off(g, n, type, "a type without its shader");
size_t nb = t[0].cols / JANAS_QK, xs = t[0].x_stride ? t[0].x_stride : nb;
const struct janas_block_q8k *x0 = t[0].x, *x1 = t[0].x;
for (size_t i = 0; i < n; i++) {
@@ -2046,7 +2090,7 @@ static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
for (size_t i = 1; i < n && !own; i++)
own = t[i].x != t[0].x;
if (!own)
- return -1;
+ return experts_off(g, n, type, "no expert with tokens of its own");
/* a prompt's block only: generating, each expert has a token or a
few, and the GPU's share cost the 30B three quarters of its speed */
size_t tokens = 0;
@@ -2056,15 +2100,15 @@ static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
return -1;
unsigned char on[2 * 128 + 8];
size_t maxnv = experts_pick(t, n, on);
- if (maxnv == 0)
- return -1; /* no expert worth the GPU: the CPU's as before */
+ if (maxnv == 0) /* no expert worth the GPU: the CPU's as before */
+ return experts_off(g, n, type, "no expert worth the GPU");
/* the activations: one array of vectors xs blocks apart */
size_t span = (size_t)(x1 - x0);
if (span % xs != 0 || span / xs * nb * GPU_BLOCK > X_STAGE_BYTES)
- return -1;
+ return experts_off(g, n, type, "activations beyond the stage");
for (size_t i = 0; i < n; i++)
if ((size_t)(t[i].x - x0) % xs != 0)
- return -1;
+ return experts_off(g, n, type, "activations off the stride");
/* the table: weights, activations, outputs (shared with the GPU) */
uint32_t *tb = (uint32_t *)g->ts.map;
size_t ng = 0;
@@ -2078,7 +2122,7 @@ static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
const struct region *yr = janas_g_find_region(
g, t[i].y, ((nv - 1) * ys + t[i].rows) * sizeof(float));
if (!wr || !yr || !yr->host_shared) /* outputs written in place */
- return -1;
+ return experts_off(g, n, type, "weights or outputs out of reach");
uint32_t *e = tb + 12 * ng++;
janas_g_push_addr(
e,
@@ -2097,7 +2141,7 @@ static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
const struct region *pr =
plane_region(g, t[i].plane[q], t[i].rows, nb);
if (!pr)
- return -1;
+ return experts_off(g, n, type, "a plane out of reach");
janas_g_push_addr(
e + (q == 0 ? 8 : 10),
pr->addr +
@@ -2115,7 +2159,7 @@ static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
for (uint32_t v = 0; v < tb[12 * k + 6]; v += 16)
nt++;
if ((3 * ng + nt) * 16 > g->ts.bytes)
- return -1;
+ return experts_off(g, n, type, "more tiles than its table");
for (size_t k = 0, at = 3 * ng * 4; k < ng; k++)
for (uint32_t v = 0; v < tb[12 * k + 6]; v += 16, at += 4) {
tb[at] = (uint32_t)k;
@@ -2144,9 +2188,10 @@ static int experts_id(struct janas_gpu *g, struct janas_pool *pool,
if (g->vkBeginCommandBuffer(g->cb, &bb) != VK_SUCCESS)
return -1;
g->vkCmdBindPipeline(g->cb, VK_PIPELINE_BIND_POINT_COMPUTE,
- type == JANAS_Q4_K ? g->pipe_id4
- : type == JANAS_Q6_K ? g->pipe_id6s
- : g->pipe_id6);
+ type == JANAS_Q4_K ? g->pipe_id4
+ : type == JANAS_Q6_K ? g->pipe_id6s
+ : type == JANAS_MXFP4 ? g->pipe_idmx
+ : g->pipe_id6);
uint32_t pc[5];
janas_g_push_addr(pc, g->ts.addr);
pc[2] = (uint32_t)t[0].rows;
diff --git a/src/llm/gpu_impl.h b/src/llm/gpu_impl.h
index 31309ea..61549ee 100644
--- a/src/llm/gpu_impl.h
+++ b/src/llm/gpu_impl.h
@@ -16,7 +16,9 @@
/* The push constants a shader may take: every Vulkan device gives 128. */
#define JANAS_G_PUSH_BYTES 128
/* the kinds of product shader (gpu.c's prod_index) */
-#define N_PROD 12
+#define N_PROD 13
+/* the kinds of b32_matvec.comp (gpu.c's b32_kind) */
+#define N_B32 5
#define VK_NO_PROTOTYPES
#include <dlfcn.h>
@@ -145,26 +147,31 @@ struct janas_gpu {
VkPhysicalDevice pd;
VkDevice dev;
VkQueue queue;
- VkShaderModule sm, sm6, sm6p, sm5, sm80, sm32[4], smx4, sm3, smi3, sma;
+ VkShaderModule sm, sm6, sm6p, sm5, sm80, sm32[N_B32], smx4, sm3, smi3, sma,
+ sma32;
VkPipelineLayout pl;
VkPipeline pipe, pipe6, pipe6p, pipe5, pipe80, pipea; /* Q4_K, Q6_K,
Q6_K_P, Q5_K,
Q8_0, experts'
activation */
- /* blocks of 32: Q4_0, IQ4_NL, Q4_1, Q5_1 (b32_matvec's KIND) */
- VkPipeline pipe32[4];
+ /* the experts' activation in blocks of 32 (expert_act.comp's KIND 1),
+ for a down matrix in blocks of 32 (MXFP4) */
+ VkPipeline pipea32;
+ /* blocks of 32: Q4_0, IQ4_NL, Q4_1, Q5_1, MXFP4 (b32_matvec's KIND) */
+ VkPipeline pipe32[N_B32];
VkPipeline pipex4, pipe3, pipei3; /* IQ4_XS, Q3_K, IQ3_S */
/* Q4_K on many vectors in tiles (q4k_mm.comp), MM_ROWS x MM_VECS */
VkShaderModule sm_mm, sm_mm6, sm_mm6s, sm_mm80, sm_mm5, sm_mm3, sm_mmx4,
- sm_mmi3, sm_mmn4,
- sm_dm[2]; /* sm_mmx4 IQ4_XS, mmi3 IQ3_S, mmn4 IQ4_NL */
+ sm_mmi3, sm_mmn4, sm_mmmx,
+ sm_dm[2]; /* sm_mmx4 IQ4_XS, mmi3 IQ3_S, mmn4 IQ4_NL, mmmx MXFP4 */
VkPipeline pipe_mm, pipe_mm6, pipe_mm6s, pipe_mm80, pipe_mm5, pipe_mm3,
- pipe_mmx4, pipe_mmi3, pipe_mmn4, pipe_dm[2];
+ pipe_mmx4, pipe_mmi3, pipe_mmn4, pipe_mmmx, pipe_dm[2];
int only_fast; /* janas_gpu_set_only_fast */
uint32_t nvg; /* vectors a product's workgroup takes (NV_PER_GROUP) */
- /* a mixture's experts in one dispatch (experts_id): Q4_K, Q6_K_P */
- VkShaderModule sm_id4, sm_id6, sm_id6s;
- VkPipeline pipe_id4, pipe_id6, pipe_id6s;
+ /* a mixture's experts in one dispatch (experts_id): Q4_K, Q6_K_P,
+ Q6_K, MXFP4 */
+ VkShaderModule sm_id4, sm_id6, sm_id6s, sm_idmx;
+ VkPipeline pipe_id4, pipe_id6, pipe_id6s, pipe_idmx;
struct stage ts; /* their table, 3 uvec4 an expert */
/* the product shaders made for a single vector (prod_index) */
VkShaderModule pm1[N_PROD];
diff --git a/src/llm/gpu_token.c b/src/llm/gpu_token.c
index a1a0563..589cd2c 100644
--- a/src/llm/gpu_token.c
+++ b/src/llm/gpu_token.c
@@ -94,9 +94,11 @@ static const char *check_ffn(const struct janas_gpu *g,
return "a feed-forward weight the GPU cannot compute";
return NULL;
}
- int dt = ly->down.type;
- if (ly->gate.type != JANAS_Q4_K || ly->up.type != JANAS_Q4_K ||
- !(dt == JANAS_Q4_K || dt == JANAS_Q6_K || dt == JANAS_Q6_K_P) ||
+ int gt = ly->gate.type, dt = ly->down.type;
+ if (!(gt == JANAS_Q4_K || (gt == JANAS_MXFP4 && g->pipe_idmx)) ||
+ ly->up.type != gt ||
+ !(dt == JANAS_Q4_K || dt == JANAS_Q6_K || dt == JANAS_Q6_K_P ||
+ (dt == JANAS_MXFP4 && g->pipe_idmx)) ||
ly->gate.rows != md->eff || ly->up.rows != md->eff ||
ly->gate.cols != md->dm || ly->up.cols != md->dm ||
ly->down.rows != md->dm || ly->down.cols != md->eff)
diff --git a/src/llm/gpu_token_moe.c b/src/llm/gpu_token_moe.c
index f95279c..4d6707c 100644
--- a/src/llm/gpu_token_moe.c
+++ b/src/llm/gpu_token_moe.c
@@ -70,7 +70,9 @@ int janas_gt_rec_router(struct janas_gpu *g, struct janas_gpu_tok *t,
/* A mixture's experts on the block whose router chose them (plan wrote
their tables and sizes): the activations gathered in the experts'
order, gate and up, the activation, down, then each token's weighted
- sum (and the shared expert) into YD. */
+ sum (and the shared expert) into YD. Experts in blocks of 32 (MXFP4)
+ read the activations in that layout: the block's q32, and the
+ activation quantized into it (expert_act.comp's KIND 1). */
void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md,
const struct janas_gpu_tok_layer *ly,
@@ -78,8 +80,10 @@ void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
{
VkDeviceAddress B = t->buf.addr;
uint32_t np = n * md->k, nbd = md->dm / JANAS_QK, nbf = md->eff / JANAS_QK;
+ int dt = ly->down.type;
struct push_ga pg = {.nb = nbd};
- janas_g_push_addr(pg.src, B + t->o_q256);
+ janas_g_push_addr(
+ pg.src, B + (janas_g_x_in_32(ly->gate.type) ? t->o_q32 : t->o_q256));
janas_g_push_addr(pg.dst, B + t->o_xg);
janas_g_push_addr(pg.tok, B + t->o_gt);
janas_gt_bind(g, bound, t->p_ga);
@@ -88,7 +92,8 @@ void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
janas_g_compute_barrier(g);
struct push_id pi = {.rows = md->eff, .nb = nbd, .tiles = t->tiles1};
janas_g_push_addr(pi.t, B + t->o_tab1);
- janas_gt_bind(g, bound, g->pipe_id4);
+ janas_gt_bind(g, bound,
+ ly->gate.type == JANAS_MXFP4 ? g->pipe_idmx : g->pipe_id4);
janas_gt_push(g, &pi, sizeof(pi));
g->vkCmdDispatchIndirect(g->cb, t->buf.buf, t->o_ind);
janas_g_compute_barrier(g);
@@ -96,16 +101,17 @@ void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
janas_g_push_addr(ac.g, B + t->o_egu);
janas_g_push_addr(ac.u, B + t->o_egu + (size_t)np * md->eff * 4);
janas_g_push_addr(ac.x, B + t->o_eh);
- janas_gt_bind(g, bound, g->pipea);
+ janas_gt_bind(g, bound, janas_g_x_in_32(dt) ? g->pipea32 : g->pipea);
janas_gt_push(g, &ac, sizeof(ac));
g->vkCmdDispatch(g->cb, np * nbf, 1, 1);
janas_g_compute_barrier(g);
pi = (struct push_id){.rows = md->dm, .nb = nbf, .tiles = t->tiles2};
janas_g_push_addr(pi.t, B + t->o_tab2);
janas_gt_bind(g, bound,
- ly->down.type == JANAS_Q4_K ? g->pipe_id4
- : ly->down.type == JANAS_Q6_K ? g->pipe_id6s
- : g->pipe_id6);
+ dt == JANAS_Q4_K ? g->pipe_id4
+ : dt == JANAS_Q6_K ? g->pipe_id6s
+ : dt == JANAS_MXFP4 ? g->pipe_idmx
+ : g->pipe_id6);
janas_gt_push(g, &pi, sizeof(pi));
g->vkCmdDispatchIndirect(g->cb, t->buf.buf, t->o_ind + 16);
janas_g_compute_barrier(g);
@@ -180,7 +186,7 @@ static int plan(struct janas_gpu *g, struct janas_gpu_tok *t,
uint32_t *t1 = (uint32_t *)(t->buf.map + t->o_tab1),
*t2 = (uint32_t *)(t->buf.map + t->o_tab2);
uint32_t n1 = 0, n2 = 0; /* tiles */
- size_t gb = (size_t)md->eff * nbd * janas_qtype_block_size(JANAS_Q4_K),
+ size_t gb = (size_t)md->eff * nbd * janas_qtype_block_size(ly->gate.type),
db =
(size_t)md->dm * nbf * janas_qtype_block_size(ly->down.type) + 4,
pb = (size_t)md->dm * nbf * JANAS_Q6KP_PLANE;
diff --git a/src/llm/model_impl.h b/src/llm/model_impl.h
index da044b8..879a083 100644
--- a/src/llm/model_impl.h
+++ b/src/llm/model_impl.h
@@ -389,6 +389,15 @@ void janas_m_init_tuner(struct janas_llm_model *m);
/* Memory the GPU will import collapsed into 2 MB pages (Linux; see
model_load.c), before the import pins it. */
void janas_m_huge_pages(void *p, uint64_t bytes);
+/* The longest region of memory the GPU imports (gpu.c janas_gpu_import). */
+#define JANAS_M_REGION_MAX ((uint64_t)4 << 30)
+/* The next region the GPU imports of bytes, from at: its first byte into
+ *from (at, or the 2 MB boundary before it where huge: memory allocated on
+ one), and its end, returned: bytes, or past at a multiple of unit (unit
+ 0: the farthest it may reach, for the caller to cut), never more than
+ JANAS_M_REGION_MAX from *from. */
+uint64_t janas_m_import_next(uint64_t at, uint64_t bytes, uint64_t unit,
+ int huge, uint64_t *from);
int janas_m_gpu_import(struct janas_llm_model *m, const uint8_t *p,
uint64_t bytes, uint64_t unit);
void janas_m_arena_to_gpu(struct janas_llm_model *m);
diff --git a/src/llm/model_load.c b/src/llm/model_load.c
index 5d997ce..080b43e 100644
--- a/src/llm/model_load.c
+++ b/src/llm/model_load.c
@@ -1550,8 +1550,6 @@ uint64_t janas_llm_model_warm(const struct janas_llm_model *m, double *seconds)
/* The GPU region of 4 GB at the most: cut at a multiple of unit (an
expert slot), or for the resident region (unit 0) where a tensor
begins, so that no tensor straddles two and each stays in reach */
-#define GPU_REGION_MAX ((uint64_t)4 << 30)
-
/*
* Memory the GPU imports, collapsed into huge pages first: the GPU's own
* page tables map it in 2 MB pages only where it is on them and the region
@@ -1598,38 +1596,53 @@ void janas_m_huge_pages(void *p, uint64_t bytes)
#endif
}
-static uint64_t resident_cut(const struct janas_llm_model *m, uint64_t from)
+/* The last tensor boundary of the resident memory past from and at most
+ room further (from + room if none). */
+static uint64_t resident_cut(const struct janas_llm_model *m, uint64_t from,
+ uint64_t room)
{
uint64_t base = m->j.h.resident_offset, best = from;
for (uint64_t i = 0; i < m->j.h.n_tensor; i++) {
uint64_t at = m->j.tensors[i].offset - base;
- if (at > from && at <= from + GPU_REGION_MAX && at > best)
+ if (at > from && at <= from + room && at > best)
best = at;
}
- return best > from ? best : from + GPU_REGION_MAX;
+ return best > from ? best : from + room;
+}
+
+/*
+ * Each region from a 2 MB boundary (the memory is allocated on one),
+ * overlapping the one before by what it takes: the GPU maps only such
+ * regions in huge pages (janas_m_huge_pages). The overlap counts in the
+ * region's length: from 13 October 2026, when the regions began on 2 MB
+ * boundaries, until 14 October it did not, and a region past the first
+ * could come to more than the 4 GiB the GPU takes - Qwen3.6-35B-A3B's
+ * MXFP4_MOE file, slots of 1,671,168 bytes, the arena's second region 4 GiB
+ * and 1.9 MB (its Q4_K_M at six bits the third): refused, and the experts
+ * of a cache holding more than 4 GiB (8 GiB) stayed off the GPU.
+ */
+uint64_t janas_m_import_next(uint64_t at, uint64_t bytes, uint64_t unit,
+ int huge, uint64_t *from)
+{
+ *from = huge ? at / JANAS_HUGE_PAGE * JANAS_HUGE_PAGE : at;
+ uint64_t room = JANAS_M_REGION_MAX - (at - *from);
+ if (bytes - at <= room)
+ return bytes;
+ return unit && room >= unit ? at + room / unit * unit : at + room;
}
int janas_m_gpu_import(struct janas_llm_model *m, const uint8_t *p,
uint64_t bytes, uint64_t unit)
{
+ int huge =
+ ((uintptr_t)p & (JANAS_HUGE_PAGE - 1)) == 0 &&
+ !(getenv("JANAS_GPU_HUGE") && atoi(getenv("JANAS_GPU_HUGE")) == 0);
for (uint64_t at = 0; at < bytes;) {
- uint64_t end = at + GPU_REGION_MAX;
- if (end >= bytes)
- end = bytes;
- else if (unit)
- end = at + GPU_REGION_MAX / unit * unit;
- else
- end = resident_cut(m, at);
- if (end > bytes)
- end = bytes;
- /* each region from a 2 MB boundary (the memory is allocated on
- one), overlapping the one before by what it takes: the GPU maps
- only such regions in huge pages (janas_m_huge_pages); whole
- pages, the last one's length rounded up (the allocations are) */
- uint64_t from = at;
- if (((uintptr_t)p & (JANAS_HUGE_PAGE - 1)) == 0 &&
- !(getenv("JANAS_GPU_HUGE") && atoi(getenv("JANAS_GPU_HUGE")) == 0))
- from = at / JANAS_HUGE_PAGE * JANAS_HUGE_PAGE;
+ uint64_t from, end = janas_m_import_next(at, bytes, unit, huge, &from);
+ if (!unit && end < bytes)
+ end = resident_cut(m, at, end - at);
+ /* whole pages, the last one's length rounded up (the allocations
+ are) */
uint64_t len = (end - from + 4095) & ~(uint64_t)4095;
if (janas_gpu_import(m->gpu, p + from, len) != 0)
return -1;
diff --git a/src/llm/shaders/b32_matvec.comp b/src/llm/shaders/b32_matvec.comp
index 36f9b76..32566e0 100644
--- a/src/llm/shaders/b32_matvec.comp
+++ b/src/llm/shaders/b32_matvec.comp
@@ -4,7 +4,10 @@
// Weights in blocks of 32 with four- or five-bit values, KIND chosen when
// the pipeline is made: 0 Q4_0 (q - 8), 1 IQ4_NL (a table of sixteen
// levels), 2 Q4_1 and 3 Q5_1 (d q + m, Q5_1 with a fifth bit per weight in
-// qh). As the CPU computes them (nib_dot_ref and dm_dot_ref in quant.c):
+// qh), 4 MXFP4 (17 bytes a block: its E8M0 byte, the power of two
+// 2^(e - 128) for a scale, then the nibbles, through MXFP4's table of 0 to
+// 12 and their negatives). As the CPU computes them (nib_dot_ref,
+// janas_mxfp4_dot_ref and dm_dot_ref in quant.c):
// each block's integer dot in one fma into the accumulator of its position
// in the group of eight, the minimum of Q4_1/Q5_1 (m times the block's
// activation sum) in accumulators of its own, the eight added at the end as
@@ -13,7 +16,10 @@
// The activations come in the 32-block layout (20 uvec4 a Q8_K block: qs32
// in 16, bsums32 in 2, d32 in 2), shared across the subgroup as in
// q4k_matvec.comp. The low nibbles of a block's 16 bytes are weights 0-15,
-// the high ones 16-31.
+// the high ones 16-31. MXFP4's blocks start at any byte, so its words are
+// realigned by the byte, the others' by the half word. An E8M0 byte of 0 or
+// 1 is a subnormal scale, which a GPU may flush to zero where the CPU does
+// not: such blocks (weights under 2^-125) differ from the CPU's.
#extension GL_EXT_integer_dot_product : require
#extension GL_KHR_shader_subgroup_basic : require
#extension GL_KHR_shader_subgroup_ballot : require
@@ -26,9 +32,10 @@ layout(constant_id = 1) const uint SG = 8;
layout(constant_id = 2) const uint KIND = 0;
const uint NX = (20u + SG - 1u) / SG;
/* bytes of a block, and of a group of eight in words */
-const uint S = KIND == 3u ? 24u : KIND == 2u ? 20u : 18u;
+const uint S = KIND == 4u ? 17u : KIND == 3u ? 24u : KIND == 2u ? 20u : 18u;
const uint GW = 2u * S;
-const bool DM = KIND >= 2u;
+const bool DM = KIND == 2u || KIND == 3u;
+const bool TAB = KIND == 1u || KIND == 4u; /* levels through a table */
layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };
layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };
@@ -38,6 +45,13 @@ layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;
const int IQ4[16] = int[16](-127, -104, -83, -65, -49, -35, -22, -10,
1, 13, 25, 38, 53, 69, 89, 113);
+const int MX[16] = int[16](0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12);
+
+/* an E8M0 byte as MXFP4's scale, 2^(e - 128) (quant.c janas_e8m0_half) */
+float e8m0_half(uint e)
+{
+ return uintBitsToFloat(e < 2u ? 0x00200000u << e : (e - 1u) << 23);
+}
int dp(uint a, uint b) { return dotPacked4x8EXT(int(a), int(b)); }
int dp4(uvec4 a, uvec4 b) { return dp(a.x, b.x) + dp(a.y, b.y) + dp(a.z, b.z) + dp(a.w, b.w); }
@@ -53,7 +67,7 @@ uint spread5(uint b)
varying index is far slower on some GPUs (iq4xs_matvec.comp) */
shared uint T[16];
-/* each byte of w through the IQ4_NL table */
+/* each byte of w through the table (IQ4_NL's or MXFP4's) */
uint iq4(uint w)
{
return T[w & 15u] | (T[(w >> 8) & 15u] << 8) | (T[(w >> 16) & 15u] << 16) |
@@ -68,7 +82,8 @@ float hsum8(float a[8])
void main()
{
if (gl_LocalInvocationID.x < 16u)
- T[gl_LocalInvocationID.x] = uint(IQ4[gl_LocalInvocationID.x]) & 0xffu;
+ T[gl_LocalInvocationID.x] =
+ uint(KIND == 4u ? MX[gl_LocalInvocationID.x] : IQ4[gl_LocalInvocationID.x]) & 0xffu;
barrier();
U1 W = U1(p.w);
U4 X = U4(p.x);
@@ -99,17 +114,22 @@ void main()
for (uint k = 0; k < 8u; k++) {
uint B = S * k; /* the block's first byte */
uint hw = wg[B >> 2];
- vec2 dm = unpackHalf2x16((B & 2u) != 0u ? (hw >> 16) | (wg[(B >> 2) + 1u] << 16) : hw);
- d[k] = dm.x;
- m[k] = dm.y;
- uint qb = B + S - 16u, wi = qb >> 2;
+ if (KIND == 4u) {
+ d[k] = e8m0_half((hw >> (8u * (B & 3u))) & 0xffu);
+ m[k] = 0.0;
+ } else {
+ vec2 dm = unpackHalf2x16((B & 2u) != 0u ? (hw >> 16) | (wg[(B >> 2) + 1u] << 16) : hw);
+ d[k] = dm.x;
+ m[k] = dm.y;
+ }
+ uint qb = B + S - 16u, wi = qb >> 2, sh = 8u * (qb & 3u);
uint q[4];
for (uint i = 0; i < 4u; i++)
- q[i] = (qb & 2u) != 0u ? (wg[wi + i] >> 16) | (wg[wi + i + 1u] << 16) : wg[wi + i];
+ q[i] = sh != 0u ? (wg[wi + i] >> sh) | (wg[wi + i + 1u] << (32u - sh)) : wg[wi + i];
uint qh = KIND == 3u ? wg[(B + 4u) >> 2] : 0u;
for (uint i = 0; i < 4u; i++) {
uint l = q[i] & 0x0f0f0f0fu, h = (q[i] >> 4) & 0x0f0f0f0fu;
- if (KIND == 1u) {
+ if (TAB) {
l = iq4(l);
h = iq4(h);
} else if (KIND == 3u) {
diff --git a/src/llm/shaders/expert_act.comp b/src/llm/shaders/expert_act.comp
index fb0d40b..86ef659 100644
--- a/src/llm/shaders/expert_act.comp
+++ b/src/llm/shaders/expert_act.comp
@@ -6,14 +6,20 @@
// janas_q8k_quantize_det): h = SiLU(gate) * up with the deterministic e^x
// and reciprocal, then Q8_K quantization of h, written in the layout the
// product shaders read (320 bytes a block: 256 quants, 16 pair sums, d).
+// KIND 1, for a down matrix in blocks of 32 (MXFP4): the layout of the
+// blocks of 32 instead (norm_quant.comp's q32: eight groups of 32 quants,
+// each with its own scale, the sums of 16, the eight scales), as
+// janas_q8k_quantize_det makes qs32, bsums32 and d32.
// One workgroup per 256-value block; gate and up of block b of item i at
// g[(i * nb + b) * 256 ...], u likewise; the output block at x.
#extension GL_EXT_buffer_reference : require
#extension GL_KHR_shader_subgroup_basic : require
#extension GL_KHR_shader_subgroup_arithmetic : require
+#extension GL_KHR_shader_subgroup_clustered : require
#extension GL_EXT_buffer_reference_uvec2 : require
layout(local_size_x = 256) in;
+layout(constant_id = 2) const uint KIND = 0;
layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer F { float v[]; };
layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer U { uint v[]; };
@@ -38,6 +44,8 @@ float wg_max(float s)
}
shared int q_s[256];
+shared float m8[32]; /* KIND 1: the maxima of eight values */
+shared int s8[32]; /* ... and the sums of their quants */
float exp_det(float x)
{
@@ -79,6 +87,42 @@ void main()
precise float s = recip_det(1.0 + e);
precise float gs = gv * s;
precise float h = gs * uv;
+ if (KIND == 1u) {
+ /* each group of 32 its largest magnitude and scale */
+ float mc = subgroupClusteredMax(abs(h), 8u);
+ if (t % 8u == 0u)
+ m8[t / 8u] = mc;
+ barrier();
+ uint l = t / 32u;
+ float m32 = max(max(m8[4u * l], m8[4u * l + 1u]), max(m8[4u * l + 2u], m8[4u * l + 3u]));
+ precise float d32 = m32 * 0.007874015718698502;
+ precise float id32 = d32 > 0.0 ? recip_det(d32) : 0.0;
+ precise float q32 = roundEven(h * id32);
+ int q = int(clamp(q32, -127.0, 127.0));
+ q_s[t] = q;
+ int cs = subgroupClusteredAdd(q, 8u);
+ if (t % 8u == 0u)
+ s8[t / 8u] = cs;
+ barrier();
+ uint o32 = blk * 80u;
+ if (t < 64u)
+ X.v[o32 + t] = uint(q_s[4u * t] & 0xff) | (uint(q_s[4u * t + 1u] & 0xff) << 8) |
+ (uint(q_s[4u * t + 2u] & 0xff) << 16) |
+ (uint(q_s[4u * t + 3u] & 0xff) << 24);
+ else if (t < 72u) {
+ /* two sums of 16 quants a word */
+ uint k = t - 64u;
+ int s0 = s8[4u * k] + s8[4u * k + 1u], s1 = s8[4u * k + 2u] + s8[4u * k + 3u];
+ X.v[o32 + t] = uint(s0 & 0xffff) | (uint(s1 & 0xffff) << 16);
+ } else if (t < 80u) {
+ uint k = t - 72u;
+ precise float dk =
+ max(max(m8[4u * k], m8[4u * k + 1u]), max(m8[4u * k + 2u], m8[4u * k + 3u])) *
+ 0.007874015718698502;
+ X.v[o32 + t] = floatBitsToUint(dk);
+ }
+ return;
+ }
/* the block's largest magnitude (max is exact in any order) */
float amax = wg_max(abs(h));
precise float d = amax * 0.007874015718698502;
diff --git a/src/llm/shaders/iq4nl_mmq.comp b/src/llm/shaders/iq4nl_mmq.comp
index 57135e4..fe50b50 100644
--- a/src/llm/shaders/iq4nl_mmq.comp
+++ b/src/llm/shaders/iq4nl_mmq.comp
@@ -1,11 +1,13 @@
/* SPDX-License-Identifier: GPL-3.0-or-later
Copyright (C) 2026 Maurizio Cammalleri */
#version 450
-// The fast prompt for IQ4_NL: q80_mmq.comp's tiling and activations (blocks
-// of 32), a group's weights its 16 bytes of nibbles through the sixteen
-// signed levels of IQ4_NL (weights 0-15 the low nibbles, 16-31 the high),
-// its scale the block's d. A block is 18 bytes, so its words are read
-// realigned. NOT bit for bit with the CPU: rounded group by group.
+// The fast prompt for IQ4_NL (KIND 0) and MXFP4 (KIND 1): q80_mmq.comp's
+// tiling and activations (blocks of 32), a group's weights its 16 bytes of
+// nibbles through the sixteen signed levels of its table (weights 0-15 the
+// low nibbles, 16-31 the high), its scale the block's d (IQ4_NL) or the
+// power of two of its E8M0 byte (MXFP4, quant.c janas_e8m0_half). A block
+// is 18 bytes (IQ4_NL) or 17 (MXFP4), so its words are read realigned.
+// NOT bit for bit with the CPU: rounded group by group.
#extension GL_EXT_control_flow_attributes : require
#extension GL_EXT_integer_dot_product : require
#extension GL_EXT_buffer_reference : require
@@ -14,6 +16,8 @@
layout(local_size_x = 128) in;
layout(constant_id = 0) const uint NV = 8; // unused: the table's constants
layout(constant_id = 1) const uint SG = 8;
+layout(constant_id = 2) const uint KIND = 0;
+const uint S = KIND == 1u ? 17u : 18u; // bytes a block
const uint BM = 64u, BN = 64u, BK = 32u, KS = 4u;
const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;
@@ -25,7 +29,7 @@ layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4
layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };
layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };
-// w: the first row (rows of nb * 8 blocks of 34 bytes); x: vector v, block
+// w: the first row (rows of nb * 8 blocks of S bytes); x: vector v, block
// b at uvec4 (v * xs + b) * 20; y: row r of vector v at y[v * ys + r]
layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;
@@ -34,22 +38,34 @@ struct BCache { int qs[8]; float d; }; // a group: quants, its d32
shared ACache sa[BM * KS];
shared BCache sb[BN * KS];
-// word i of the 32-bit words from byte off of w, off even
-uint hword(U1 W, uint off, uint i)
+// word i of the 32-bit words from byte off of w
+uint bword(U1 W, uint off, uint i)
{
- uint a = (off >> 2) + i;
- return (off & 2u) != 0u ? (W.v[a] >> 16) | (W.v[a + 1u] << 16) : W.v[a];
+ uint a = (off >> 2) + i, sh = 8u * (off & 3u);
+ return sh != 0u ? (W.v[a] >> sh) | (W.v[a + 1u] << (32u - sh)) : W.v[a];
}
const int IQ4[16] = int[16](-127, -104, -83, -65, -49, -35, -22, -10,
1, 13, 25, 38, 53, 69, 89, 113);
+const int MX[16] = int[16](0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12);
-/* each byte of w (a nibble) through the IQ4_NL table */
+int lvl(uint i) { return KIND == 1u ? MX[i] : IQ4[i]; }
+
+/* each byte of w (a nibble) through the table */
uint iq4(uint w)
{
- return (uint(IQ4[w & 15u]) & 0xffu) | ((uint(IQ4[(w >> 8) & 15u]) & 0xffu) << 8) |
- ((uint(IQ4[(w >> 16) & 15u]) & 0xffu) << 16) |
- ((uint(IQ4[(w >> 24) & 15u]) & 0xffu) << 24);
+ return (uint(lvl(w & 15u)) & 0xffu) | ((uint(lvl((w >> 8) & 15u)) & 0xffu) << 8) |
+ ((uint(lvl((w >> 16) & 15u)) & 0xffu) << 16) |
+ ((uint(lvl((w >> 24) & 15u)) & 0xffu) << 24);
+}
+
+/* the scale of the block at byte off */
+float scale(U1 W, uint off)
+{
+ uint w = bword(W, off, 0u);
+ return KIND == 1u ? uintBitsToFloat((w & 0xffu) < 2u ? 0x00200000u << (w & 0xffu)
+ : ((w & 0xffu) - 1u) << 23)
+ : unpackHalf2x16(w & 0xffffu).x;
}
void main()
@@ -72,14 +88,14 @@ void main()
[[unroll]] for (uint l = 0; l < BM; l += 32u) {
uint r = min(r0 + rl + l, p.rows - 1u);
[[unroll]] for (uint k = 0; k < KS; k++) {
- uint off = (r * ngroups + g0 + k) * 18u;
+ uint off = (r * ngroups + g0 + k) * S;
[[unroll]] for (uint t = 0; t < 2u; t++) {
- uint kw = 2u * iqs + t, q = hword(W1, off + 2u, kw & 3u);
+ uint kw = 2u * iqs + t, q = bword(W1, off + S - 16u, kw & 3u);
sa[k * BM + rl + l].qs[kw] =
int(iq4(kw < 4u ? q & 0x0f0f0f0fu : (q >> 4) & 0x0f0f0f0fu));
}
if (iqs == 0u)
- sa[k * BM + rl + l].d = unpackHalf2x16(hword(W1, off, 0u) & 0xffffu).x;
+ sa[k * BM + rl + l].d = scale(W1, off);
}
}
}
diff --git a/src/llm/shaders/mxfp4_mmid.comp b/src/llm/shaders/mxfp4_mmid.comp
new file mode 100644
index 0000000..448c455
--- /dev/null
+++ b/src/llm/shaders/mxfp4_mmid.comp
@@ -0,0 +1,139 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+#version 450
+// The fast prompt for a mixture's experts in MXFP4 (gpu.c experts_id, and
+// the whole token's mixture, gpu_token_moe.c): q4k_mmid.comp's table and
+// tiles (64 rows by 16 vectors, each invocation 2 rows by 4, groups of 32
+// staged 4 at a time), the weights read as iq4nl_mmq.comp reads MXFP4 (17
+// bytes a block: its E8M0 byte, then 16 of nibbles through MXFP4's levels,
+// 0 to 12 and their negatives, the power of two 2^(e - 128) for a scale),
+// the activations in blocks of 32, each group with its own scale. NOT bit
+// for bit with the CPU: rounded group by group.
+#extension GL_EXT_control_flow_attributes : require
+#extension GL_EXT_integer_dot_product : require
+#extension GL_EXT_buffer_reference : require
+#extension GL_EXT_buffer_reference_uvec2 : require
+
+layout(local_size_x = 128) in;
+layout(constant_id = 0) const uint NV = 8; // unused: the table's constants
+layout(constant_id = 1) const uint SG = 8;
+
+const uint BM = 64u, BN = 16u, KS = 4u, S = 17u, XB = 20u;
+
+layout(buffer_reference, std430, buffer_reference_align = 16) readonly buffer U4 { uvec4 v[]; };
+layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };
+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F { float v[]; };
+
+// t: the table, 3 uvec4 an expert: (w, x), (y, nv, ys), (unused); then
+// from uvec4 tiles on, one a workgroup's z: (expert, first vector)
+layout(push_constant) uniform P { uvec2 t; uint rows, nb, tiles; } p;
+
+struct ACache { int qs[8]; float d; }; // a group: its 32 levels, its scale
+struct BCache { int qs[8]; float d; }; // a group: quants, its d32
+shared ACache sa[BM * KS];
+shared BCache sb[BN * KS];
+
+const int MX[16] = int[16](0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12);
+
+/* the levels in the workgroup's shared memory: a constant array read at a
+ varying index is far slower on some GPUs (iq4xs_matvec.comp) */
+shared uint T[16];
+
+/* each byte of w (a nibble) through the table */
+uint lv(uint w)
+{
+ return T[w & 15u] | (T[(w >> 8) & 15u] << 8) | (T[(w >> 16) & 15u] << 16) |
+ (T[(w >> 24) & 15u] << 24);
+}
+
+// word i of the 32-bit words from byte off of w
+uint bword(U1 W, uint off, uint i)
+{
+ uint a = (off >> 2) + i, sh = 8u * (off & 3u);
+ return sh != 0u ? (W.v[a] >> sh) | (W.v[a + 1u] << (32u - sh)) : W.v[a];
+}
+
+/* an E8M0 byte as MXFP4's scale, 2^(e - 128) (quant.c janas_e8m0_half) */
+float e8m0_half(uint e)
+{
+ return uintBitsToFloat(e < 2u ? 0x00200000u << e : (e - 1u) << 23);
+}
+
+void main()
+{
+ if (gl_LocalInvocationID.x < 16u)
+ T[gl_LocalInvocationID.x] = uint(MX[gl_LocalInvocationID.x]) & 0xffu;
+ barrier();
+ U4 TB = U4(p.t);
+ uvec4 tl = TB.v[p.tiles + gl_WorkGroupID.z];
+ uint e = tl.x;
+ uvec4 t0 = TB.v[3u * e], t1 = TB.v[3u * e + 1u];
+ uint nv = t1.z, ys = t1.w;
+ uint r0 = gl_WorkGroupID.x * BM, v0 = tl.y;
+ U4 X = U4(t0.zw);
+ U1 W1 = U1(t0.xy);
+ F Y = F(t1.xy);
+ uint id = gl_LocalInvocationID.x;
+ uint rq = (id % 32u) * 2u, vq = (id / 32u) * 4u;
+ float sums[8];
+ [[unroll]] for (uint i = 0; i < 8u; i++) sums[i] = 0.0;
+ uint ngroups = p.nb * 8u;
+ for (uint g0 = 0; g0 < ngroups; g0 += KS) {
+ // weights: 4 invocations a row, 2 words each of the group's 8
+ {
+ uint iqs = id % 4u, rl = id / 4u;
+ [[unroll]] for (uint l = 0; l < BM; l += 32u) {
+ uint r = min(r0 + rl + l, p.rows - 1u);
+ [[unroll]] for (uint k = 0; k < KS; k++) {
+ uint off = (r * ngroups + g0 + k) * S;
+ [[unroll]] for (uint t = 0; t < 2u; t++) {
+ uint kw = 2u * iqs + t, q = bword(W1, off + 1u, kw & 3u);
+ sa[k * BM + rl + l].qs[kw] =
+ int(lv(kw < 4u ? q & 0x0f0f0f0fu : (q >> 4) & 0x0f0f0f0fu));
+ }
+ if (iqs == 0u)
+ sa[k * BM + rl + l].d = e8m0_half(bword(W1, off, 0u) & 0xffu);
+ }
+ }
+ }
+ // activations: 2 invocations a vector, each half a group's quants
+ {
+ uint k = id / 32u, c = (id % 32u) / 2u, iqs = id % 2u;
+ uint v = min(v0 + c, nv - 1u);
+ uint g = g0 + k, b = g / 8u, j = g % 8u;
+ uint xo = (v * p.nb + b) * XB;
+ uvec4 q = X.v[xo + 2u * j + iqs];
+ sb[k * BN + c].qs[4u * iqs] = int(q.x);
+ sb[k * BN + c].qs[4u * iqs + 1u] = int(q.y);
+ sb[k * BN + c].qs[4u * iqs + 2u] = int(q.z);
+ sb[k * BN + c].qs[4u * iqs + 3u] = int(q.w);
+ if (iqs == 0u) {
+ uvec4 dq = X.v[xo + 18u + j / 4u];
+ uint dw = j % 4u == 0u ? dq.x : j % 4u == 1u ? dq.y : j % 4u == 2u ? dq.z : dq.w;
+ sb[k * BN + c].d = uintBitsToFloat(dw);
+ }
+ }
+ barrier();
+ for (uint k = 0; k < KS; k++) {
+ ACache ca[2];
+ [[unroll]] for (uint r = 0; r < 2u; r++)
+ ca[r] = sa[k * BM + rq + r];
+ [[unroll]] for (uint c = 0; c < 4u; c++) {
+ BCache cb = sb[k * BN + vq + c];
+ [[unroll]] for (uint r = 0; r < 2u; r++) {
+ int q = 0;
+ [[unroll]] for (uint i = 0; i < 8u; i++)
+ q += dotPacked4x8EXT(ca[r].qs[i], cb.qs[i]);
+ sums[c * 2u + r] += ca[r].d * cb.d * float(q);
+ }
+ }
+ }
+ barrier();
+ }
+ [[unroll]] for (uint c = 0; c < 4u; c++)
+ [[unroll]] for (uint r = 0; r < 2u; r++) {
+ uint row = r0 + rq + r, v = v0 + vq + c;
+ if (row < p.rows && v < nv)
+ Y.v[v * ys + row] = sums[c * 2u + r];
+ }
+}
diff --git a/tests/bench_gemm.c b/tests/bench_gemm.c
index 6ddc905..dfc8a83 100644
--- a/tests/bench_gemm.c
+++ b/tests/bench_gemm.c
@@ -9,7 +9,8 @@
* the figure of merit is multiply-accumulates per second.
*
* Usage: bench_gemm [rows cols
- * q4k|q5k|q6k|q3k|q80|iq4xs|iq3s|iq4nl|q6kp1|q6kp2|q6kp3 threads [gpu|gpufew]]
+ * q4k|q5k|q6k|q3k|q80|iq4xs|iq3s|iq4nl|mxfp4|q6kp1|q6kp2|q6kp3 threads
+ * [gpu|gpufew]]
* q6kpL is Q6_K in bit planes with L of the three planes in memory.
* Defaults: 12288 x 2048 Q4_K (Qwen3-Next ssm_in), 12 threads on CPUs 0-11.
* gpu: the products of a prompt's blocks on the GPU (the fast prompt's
@@ -101,6 +102,8 @@ int main(int argc, char **argv)
type = JANAS_IQ3_S;
else if (!strcmp(kind, "iq4nl"))
type = JANAS_IQ4_NL;
+ else if (!strcmp(kind, "mxfp4"))
+ type = JANAS_MXFP4;
else if (!strncmp(kind, "q6kp", 4)) {
type = JANAS_Q6_K_P;
level = kind[4] ? kind[4] - '0' : 3;
@@ -153,10 +156,13 @@ int main(int argc, char **argv)
for (size_t i = 0; i < wbytes; i++)
w[i] = (uint8_t)rand();
/* blocks of 32 weights: eight to a block of 256 */
- for (size_t b = 0;
- b < rows * nb * (type == JANAS_IQ4_NL || type == JANAS_Q8_0 ? 8 : 1);
- b++) {
- if (type == JANAS_Q8_0) {
+ size_t per =
+ type == JANAS_IQ4_NL || type == JANAS_Q8_0 || type == JANAS_MXFP4 ? 8
+ : 1;
+ for (size_t b = 0; b < rows * nb * per; b++) {
+ if (type == JANAS_MXFP4) {
+ ((struct janas_block_mxfp4 *)w)[b].e = 118; /* 2^-10 */
+ } else if (type == JANAS_Q8_0) {
((struct janas_block_q8_0 *)w)[b].d = janas_fp32_to_fp16(0.001f);
} else if (type == JANAS_Q5_K) {
((struct janas_block_q5k *)w)[b].d = janas_fp32_to_fp16(0.001f);
diff --git a/tests/gpu_check.c b/tests/gpu_check.c
index 7a79b10..c21ed7d 100644
--- a/tests/gpu_check.c
+++ b/tests/gpu_check.c
@@ -3,7 +3,7 @@
/*
* gpu_check.c - products split between the GPU and the CPU must equal the
* CPU's bit for bit: random Q4_K, Q5_K, Q6_K, Q6_K_P (bit planes, at its
- * three levels), Q8_0 and the blocks of 32 (Q4_0, IQ4_NL, Q4_1, Q5_1)
+ * three levels), Q8_0 and the blocks of 32 (Q4_0, IQ4_NL, Q4_1, Q5_1, MXFP4)
* matrices in shared memory, and the GPU's aligned copy of Q6_K (on the GPU
* alone, against the CPU's Q6_K), several shapes, vector
* counts and strides. Skips (exit 0) when no GPU is usable.
@@ -115,10 +115,12 @@ int main(void)
for (size_t b = 0; b < w8bytes / 34; b++)
w8b[b].d = janas_fp32_to_fp16(0.0001f + (rnd() % 1000) * 1e-6f);
/* blocks of 32 with four or five bits: Q4_0 and IQ4_NL (18 bytes: d,
- 16 of nibbles), Q4_1 (20: d, m, ...), Q5_1 (24: d, m, qh, ...);
- random bytes with d (and m) set to small scales */
- static const int t32[4] = {JANAS_Q4_0, JANAS_IQ4_NL, JANAS_Q4_1,
- JANAS_Q5_1};
+ 16 of nibbles), Q4_1 (20: d, m, ...), Q5_1 (24: d, m, qh, ...), MXFP4
+ (17: the E8M0 byte, 16 of nibbles); random bytes with d (and m) set
+ to small scales, MXFP4's byte to powers of two from 2^-10 to 2^-3 (a
+ subnormal one, under 2^-125, a GPU may flush) */
+ static const int t32[5] = {JANAS_Q4_0, JANAS_IQ4_NL, JANAS_Q4_1, JANAS_Q5_1,
+ JANAS_MXFP4};
/* Q3_K (d at byte 108) and IQ3_S (d at 0): 110-byte blocks */
uint8_t *w3[2];
size_t w3bytes = (size_t)MAXR * (MAXC / JANAS_QK) * 110 + 4096;
@@ -142,16 +144,20 @@ int main(void)
uint16_t d = janas_fp32_to_fp16(0.00005f + (rnd() % 1000) * 1e-7f);
memcpy(wx + b, &d, 2);
}
- static const size_t s32[4] = {18, 18, 20, 24};
- uint8_t *w32[4];
- size_t w32bytes[4];
- for (int k = 0; k < 4; k++) {
+ static const size_t s32[5] = {18, 18, 20, 24, 17};
+ uint8_t *w32[5];
+ size_t w32bytes[5];
+ for (int k = 0; k < 5; k++) {
w32bytes[k] = (size_t)MAXR * (MAXC / JANAS_QK) * 8 * s32[k];
w32bytes[k] = (w32bytes[k] + 4095) / 4096 * 4096;
w32[k] = aligned_alloc(4096, w32bytes[k]);
for (size_t i = 0; i < w32bytes[k]; i++)
w32[k][i] = (uint8_t)rnd();
for (size_t b = 0; b + s32[k] <= w32bytes[k]; b += s32[k]) {
+ if (t32[k] == JANAS_MXFP4) {
+ w32[k][b] = (uint8_t)(118 + rnd() % 8);
+ continue;
+ }
uint16_t d = janas_fp32_to_fp16(0.0005f + (rnd() % 1000) * 1e-6f);
memcpy(w32[k] + b, &d, 2);
if (s32[k] > 18) {
@@ -171,6 +177,7 @@ int main(void)
janas_gpu_import(g, w32[1], w32bytes[1]) != 0 ||
janas_gpu_import(g, w32[2], w32bytes[2]) != 0 ||
janas_gpu_import(g, w32[3], w32bytes[3]) != 0 ||
+ janas_gpu_import(g, w32[4], w32bytes[4]) != 0 ||
janas_gpu_import(g, wx, wxbytes) != 0 ||
janas_gpu_import(g, w3[0], w3bytes) != 0 ||
janas_gpu_import(g, w3[1], w3bytes) != 0) {
@@ -202,12 +209,12 @@ int main(void)
int bad = 0, runs = 0;
/* 0 Q4_K, 1 Q6_K, 2-4 Q6_K_P with three, two and one planes, 5 Q5_K,
6 Q8_0, 7-10 Q4_0, IQ4_NL, Q4_1, Q5_1, 11 IQ4_XS,
- 12 Q3_K, 13 IQ3_S, 14 the aligned Q6_K */
+ 12 Q3_K, 13 IQ3_S, 14 the aligned Q6_K, 15 MXFP4 */
static const char *names[] = {"Q4_K", "Q6_K", "Q6_K_P/3", "Q6_K_P/2",
"Q6_K_P/1", "Q5_K", "Q8_0", "Q4_0",
"IQ4_NL", "Q4_1", "Q5_1", "IQ4_XS",
- "Q3_K", "IQ3_S", "Q6_K_A"};
- for (int q6 = 0; q6 < 15; q6++)
+ "Q3_K", "IQ3_S", "Q6_K_A", "MXFP4"};
+ for (int q6 = 0; q6 < 16; q6++)
for (size_t s = 0; s < sizeof(shapes) / sizeof(shapes[0]); s++)
for (size_t k = 0; k < sizeof(nvs) / sizeof(nvs[0]); k++)
for (int strided = 0; strided < 2; strided++) {
@@ -216,7 +223,8 @@ int main(void)
size_t skip = s % 2; /* a row further in, every other
shape */
struct janas_matvec_task t = {
- .type = q6 == 14 ? JANAS_Q6_K
+ .type = q6 == 15 ? JANAS_MXFP4
+ : q6 == 14 ? JANAS_Q6_K
: q6 == 13 ? JANAS_IQ3_S
: q6 == 12 ? JANAS_Q3_K
: q6 == 11 ? JANAS_IQ4_XS
@@ -226,7 +234,8 @@ int main(void)
: q6 >= 2 ? JANAS_Q6_K_P
: q6 == 1 ? JANAS_Q6_K
: JANAS_Q4_K,
- .w = q6 == 14 ? w6 + skip * 210 * nb
+ .w = q6 == 15 ? w32[4] + skip * 8 * s32[4] * nb
+ : q6 == 14 ? w6 + skip * 210 * nb
: q6 >= 12 ? w3[q6 - 12] + skip * 110 * nb
: q6 == 11 ? wx + skip * 136 * nb
: q6 >= 7
@@ -321,7 +330,7 @@ int main(void)
free(wp);
free(w5);
free(w8);
- for (int k = 0; k < 4; k++)
+ for (int k = 0; k < 5; k++)
free(w32[k]);
free(wx);
free(w3[0]);
diff --git a/tests/test_import_regions.c b/tests/test_import_regions.c
new file mode 100644
index 0000000..14edc82
--- /dev/null
+++ b/tests/test_import_regions.c
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * test_import_regions.c - the regions in which the GPU imports an expert
+ * cache's arena (model_load.c janas_m_import_next): each starts on a 2 MB
+ * boundary, overlapping the one before, and must still be at most the 4 GiB
+ * the GPU takes from its first byte, end on a slot, and leave no gap. From
+ * 13 to 14 October 2026 the overlap was not counted, and Qwen3.6-35B-A3B's
+ * MXFP4 experts (slots of 1,671,168 bytes) got a second region of 4 GiB and
+ * 1.9 MB, which the GPU refused. No GPU needed.
+ */
+#include <stdio.h>
+
+#include "llm/model_impl.h"
+
+static int failures;
+
+static void check(uint64_t unit, uint64_t bytes, int huge)
+{
+ const uint64_t mb2 = (uint64_t)2 << 20;
+ uint64_t at = 0;
+ int n = 0;
+ while (at < bytes) {
+ uint64_t from, end = janas_m_import_next(at, bytes, unit, huge, &from);
+ int bad = from > at || end <= at || end > bytes ||
+ end - from > JANAS_M_REGION_MAX ||
+ (huge && from % mb2 != 0) || (!huge && from != at) ||
+ (end != bytes && (end - at) % unit != 0);
+ if (bad) {
+ printf("FAIL: unit %llu, %llu bytes, huge %d: region %d "
+ "[%llu, %llu) from %llu\n",
+ (unsigned long long)unit, (unsigned long long)bytes, huge, n,
+ (unsigned long long)at, (unsigned long long)end,
+ (unsigned long long)from);
+ failures++;
+ return;
+ }
+ at = end;
+ n++;
+ }
+}
+
+int main(void)
+{
+ /* the slots of Qwen3.6-35B-A3B MXFP4_MOE, of its Q4_K_M at six bits,
+ of a page, of 2 MB and around it, of a dense layer's feed-forward */
+ static const uint64_t units[] = {1671168, 2039808, 4096,
+ 2097152, 2101248, 2093056,
+ 157286400, 1179648, 3014656};
+ for (size_t u = 0; u < sizeof(units) / sizeof(units[0]); u++)
+ for (uint64_t slots = 1; slots <= 40000; slots = slots * 3 + 7)
+ for (int huge = 0; huge < 2; huge++)
+ check(units[u], slots * units[u], huge);
+ /* the arena of the 35B's MXFP4 experts, every slot of its 40 layers */
+ check(1671168, (uint64_t)40 * 256 * 1671168, 1);
+ printf("test_import_regions: %s\n", failures ? "FAILED" : "ok");
+ return failures ? 1 : 0;
+}