Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Develop 2026-10-25 (3): a mixture larger than the card's tier, the CPU's experts computed while the GPU computes its own - the plan's last copies in the next segment's submission, a segment in two submissions (the GPU's products not waited for, the CPU's experts meanwhile, then their rows and the rest); Qwen3-30B-A3B as a card of its own, a tier of 1 GiB: 9.0 to 9.9 tokens/s, 2 GiB 8.6 to 9.3; the same logits to the bit

camauria2e7bcfd15fb
diff --git a/ChangeLog.md b/ChangeLog.md
index 118ed5a..c3ec83d 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -4,6 +4,7 @@
## [2026-10-24] - A card with memory of its own runs the whole model
+- **A mixture larger than the card's tier: the CPU's experts computed while the GPU computes its own, a round trip less a layer**: where the tier does not hold every expert a pass goes in a submission a layer, the CPU's plan between. The plan's last copies (experts read into the tier, the CPU's rows) went in a submission of their own, waited for: now in the next segment's submission, after a barrier. And a segment is two submissions: the GPU's products of the experts, not waited for, while the CPU computes the experts left to it; then their rows' copies and the rest of the layer, from the experts' sum, waited for (`cpu_part`; one submission as before where the CPU has none). Qwen3-30B-A3B UD-Q3_K_XL as a card of its own on the laptop's Arc, a tier of 1 GiB, 480 tokens one at a time: the CPU's plans from 43.8 to 12.5 ms a pass, the time between segments from 51 to 29, the second half from 9.0 to 9.9 tokens/s. The same logits to the bit; with drafts the same tokens as without, prompt blocks included, and no error of the validation layer, synchronization included. `JANAS_GPU_TOKEN_PROFILE=1` prints a pass's experts in the tier, read into it and computed by the CPU, and the CPU's and the cache's time.
- **Every pass whole on a discrete GPU**: on a card with memory of its own the model's passes - a prompt's blocks, a chain of drafts to verify, single tokens - go whole to the GPU, one submission each, where before every product's rows were shared with the CPU and the activations crossed the bus at each one (an Arc B580 read a prompt at 15.8 tokens/s, issue #43). The token's state - each attention layer's KV rows and scales, the Gated DeltaNet states and their log - is copied to the card's memory and has one owner at a time: the GPU after its pass, the CPU after its own, copied over only when the passes change hands (`janas_gpu_sync`), only the rows the GPU's layers use, so that an MTP block's and Gemma 4's assistant's own stay the CPU's. `JANAS_GPU_WHOLE=0` shares the products as before. Checked as a card of its own on the development laptop's Arc (`JANAS_GPU_COPY=1`), functional runs only: Qwen3.5-2B, Qwen3-4B and Qwen3.5-9B greedy, with and without drafts the same tokens, the text sensible; Qwen3.5-2B's 64 tokens 141 single tokens, 27 draft checks and 2 blocks whole on the GPU, the state sent three times and never read back; a prompt on the GPU and its tokens on the CPU (`JANAS_GPU_WHOLE=0`) right too. As on an integrated GPU, the whole pass's norms and attention sum in the GPU's order, so the text is not the CPU's to the bit. - **The whole pass's working buffers in the card's own memory**: on a card with memory of its own every intermediate of a pass is in its memory, where the shaders read and write it at its speed, and only what the host writes or reads - a block's inputs and position, the parameters once, the logits and hidden states - goes through a host-visible mirror (`janas_gt_sync_range` copies those ranges); before, every layer's activations crossed the bus. The same tokens to the bit as with the buffer in host memory (Qwen3.5-2B, Qwen3-4B, Qwen3.5-9B, as a card of its own; a prompt on the GPU and its tokens on the CPU too). Mixtures too: their routing, read and written by the host between layers, goes through the mirror as the rest; Qwen3-30B-A3B's prompt blocks whole on the GPU gave the same tokens through the mirror as without, also under the validation layer (`JANAS_GPU_TOK_MIRROR=1` puts the buffer in the GPU's memory behind a mirror on an integrated GPU too, to test it with the weights read in place; the verbose line says where the buffer is). `JANAS_GPU_TOK_HOSTBUF=1` keeps the host-visible buffer for any model. The staging buffers and a card's copies of the weights now declare the copies they take part in (transfer usage), which a driver may enforce. Not yet: mixtures larger than the card, Qwen3.8-Flash-Next on a card of its own.
- **A dense model's pass in one submission with its copies**: the copies of a pass's inputs (the embeddings, the rotations, the position words) to the card and of its logits and hidden states back to the mirror are recorded in the pass's own commands, with their barriers, instead of four submissions of their own around it, each waited for; the same tokens to the bit (Qwen3.5-2B with and without MTP drafts, Qwen3-4B against the host-visible buffer). Under the Khronos validation layer with synchronization checks, gpu_check on every type and Qwen3.5-2B's generation with drafts, on the shared memory and as a card of its own: no error.
- **A mixture larger than the card: a tier of experts in its memory, the CPU computing the others**: on a card with memory of its own a mixture's whole pass no longer needs every expert copied to the card. It keeps a tier of slots in the card's memory - the room it has left, 85% of its memory less what Janas took and 1 GiB for the rest (`JANAS_GPU_TIER_MB=<n>` asks for n MiB), in pieces of 1 GiB that any driver allocates - and between two layers, where the CPU already sorts the router's choice, each chosen expert is either in the tier, or read into it in place of the one used least among sixteen (never one this layer counts on), or computed by the CPU from the layer's normalized input into its own rows, which the GPU then sums with the others in the router's order (`janas_m_expert_rows`, the CPU's expert kernels). Only the experts not in the tier are asked of the cache, so a tier hit never reads the disk. The arena is not copied to such a card any more, so the tier has its memory (before, a cache holding every expert was copied whole, and a copy that did not fit could leave pieces behind). `JANAS_GPU_TIER=0` turns the tier off. Checked as a card of its own on the development laptop's Arc, functional runs only: Qwen3-30B-A3B UD-Q3_K_XL with a tier of 1 GiB (480 of its 6,144 experts) and of 300 MB (140, a slot given up at nearly every layer) - sensible text, every pass whole on the GPU, no error under the validation layer with synchronization checks. The text is not the same to the bit with every tier: the two tiers' first 15 tokens are the same, then they part, as the run with drafts parts from the one without after 8-9 tokens - an expert computed by the CPU or by the GPU rounds differently (the GPU's expert products sum in their own order), and which computes it depends on what the tier holds.
diff --git a/src/llm/gpu.c b/src/llm/gpu.c
index c1ea063..ef3a8ca 100644
--- a/src/llm/gpu.c
+++ b/src/llm/gpu.c
@@ -1454,14 +1454,12 @@ int janas_g_make_local(struct janas_gpu *g, struct stage *s, size_t bytes)
return 0;
}
-int janas_g_copies(struct janas_gpu *g, const struct janas_g_cp *c, uint32_t n)
+int janas_g_record_copies(struct janas_gpu *g, const struct janas_g_cp *c,
+ uint32_t n, int barrier)
{
VkCommandBufferBeginInfo bb = {
.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};
- VkSubmitInfo sub = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO,
- .commandBufferCount = 1,
- .pCommandBuffers = &g->cb};
int ok = g->vkResetCommandBuffer(g->cb, 0) == VK_SUCCESS &&
g->vkBeginCommandBuffer(g->cb, &bb) == VK_SUCCESS;
for (uint32_t i = 0; ok && i < n; i++) {
@@ -1469,7 +1467,31 @@ int janas_g_copies(struct janas_gpu *g, const struct janas_g_cp *c, uint32_t n)
.srcOffset = c[i].so, .dstOffset = c[i].dof, .size = c[i].bytes};
g->vkCmdCopyBuffer(g->cb, c[i].src, c[i].dst, 1, &cp);
}
- ok = ok && g->vkEndCommandBuffer(g->cb) == VK_SUCCESS;
+ if (ok && barrier) {
+ /* what the commands submitted after it read: shaders, copies and
+ indirect dispatches */
+ VkMemoryBarrier mb = {
+ .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER,
+ .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT,
+ .dstAccessMask =
+ VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT |
+ VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_TRANSFER_WRITE_BIT |
+ VK_ACCESS_INDIRECT_COMMAND_READ_BIT};
+ g->vkCmdPipelineBarrier(g->cb, VK_PIPELINE_STAGE_TRANSFER_BIT,
+ VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT |
+ VK_PIPELINE_STAGE_TRANSFER_BIT |
+ VK_PIPELINE_STAGE_DRAW_INDIRECT_BIT,
+ 0, 1, &mb, 0, NULL, 0, NULL);
+ }
+ return ok && g->vkEndCommandBuffer(g->cb) == VK_SUCCESS ? 0 : -1;
+}
+
+int janas_g_copies(struct janas_gpu *g, const struct janas_g_cp *c, uint32_t n)
+{
+ VkSubmitInfo sub = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO,
+ .commandBufferCount = 1,
+ .pCommandBuffers = &g->cb};
+ int ok = janas_g_record_copies(g, c, n, 0) == 0;
pthread_mutex_lock(&g->qlock);
ok = ok && g->vkQueueSubmit(g->queue, 1, &sub, g->fence) == VK_SUCCESS;
pthread_mutex_unlock(&g->qlock);
diff --git a/src/llm/gpu_impl.h b/src/llm/gpu_impl.h
index cb29921..3c978d1 100644
--- a/src/llm/gpu_impl.h
+++ b/src/llm/gpu_impl.h
@@ -330,6 +330,10 @@ struct janas_g_cp {
};
/* Copies c[0..n) in one submission and waits: 0, or -1. */
int janas_g_copies(struct janas_gpu *g, const struct janas_g_cp *c, uint32_t n);
+/* The same recorded in g->cb, not submitted: with barrier, followed by a
+ barrier before what reads them in the same submission. 0, or -1. */
+int janas_g_record_copies(struct janas_gpu *g, const struct janas_g_cp *c,
+ uint32_t n, int barrier);
/* Forgets the last region imported, at host (the check's own weights). */
void janas_g_drop_region(struct janas_gpu *g, const void *host);
diff --git a/src/llm/gpu_token.c b/src/llm/gpu_token.c
index 64350d3..7b5b63d 100644
--- a/src/llm/gpu_token.c
+++ b/src/llm/gpu_token.c
@@ -655,6 +655,15 @@ void janas_gpu_tok_destroy(struct janas_gpu *g, struct janas_gpu_tok *t)
if (t->plan_ns > 0)
fprintf(stderr, " (the CPU's plans %.3f ms a pass)\n",
t->plan_ns / t->tokens * 1e-6);
+ if (t->n_slot)
+ fprintf(stderr,
+ " (a pass's experts: %.1f in the tier, %.1f read into "
+ "it, %.1f by the CPU in %.3f ms; the cache %.3f ms)\n",
+ (double)t->tier_hits / t->tokens,
+ (double)t->tier_loads / t->tokens,
+ (double)t->cpu_experts / t->tokens,
+ t->cpu_ns / t->tokens * 1e-6,
+ t->fetch_ns / t->tokens * 1e-6);
for (int i = 0; i < PH_N; i++)
fprintf(stderr, " %-14s %7.3f ms %5.1f%%\n", ph_name[i],
t->ph_ns[i] / t->tokens * 1e-6,
@@ -790,7 +799,7 @@ static int record_hc(struct janas_gpu *g, struct janas_gpu_tok *t,
if (l == 0) {
janas_gt_rec_hc_combine(g, t, md, bound, B + t->o_x, 1, n);
} else {
- janas_gt_rec_experts(g, t, md, &layers[l - 1], bound, YD, n);
+ janas_gt_rec_experts(g, t, md, &layers[l - 1], bound, YD, n, 0);
janas_gt_rec_hc_combine(g, t, md, bound, YD, 0, n);
}
if (l == md->ng_layer && janas_gt_rec_ng(g, t, md, ly, bound, n) != 0)
@@ -832,7 +841,7 @@ static int record_hc(struct janas_gpu *g, struct janas_gpu_tok *t,
return -1;
}
if (seg < 0 || seg == (int)nl) {
- janas_gt_rec_experts(g, t, md, &layers[nl - 1], bound, YD, n);
+ janas_gt_rec_experts(g, t, md, &layers[nl - 1], bound, YD, n, 0);
janas_gt_rec_hc_combine(g, t, md, bound, YD, 0, n);
/* the streams themselves are the hidden states (an MTP block's
input): run copies them from o_res */
@@ -858,10 +867,12 @@ static int record_hc(struct janas_gpu *g, struct janas_gpu_tok *t,
A mixture's block goes in n_layer + 1 submissions, seg the one recorded
(-1: a dense model's, all of it): submission s takes layer s - 1's
experts, then layer s up to its router's choice, the last one the final
- norm and the output head. */
+ norm and the output head. part 1 records only segment seg's products of
+ the experts, part 2 the rest from their sum on (0 all of it): the CPU
+ computes its experts between the two. */
static int record(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md, uint32_t n,
- uint32_t first, int hid, int seg)
+ uint32_t first, int hid, int seg, int part)
{
const struct janas_gpu_tok_layer *layers =
n > 1 && md->blayers ? md->blayers : md->layers;
@@ -886,7 +897,7 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,
t->n_ts = 0;
g->vkCmdResetQueryPool(g->cb, t->qp, 0, MAX_TS);
janas_gt_stamp(g, t, PH_N); /* the start */
- } else if (t->qp) {
+ } else if (t->qp && part != 2) {
janas_gt_stamp(g, t, PH_BETWEEN);
}
/* on a card with memory of its own: the pass's inputs from the mirror,
@@ -914,7 +925,7 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,
VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 1, &ib,
0, NULL, 0, NULL);
}
- if (seg > 0 && md->n_expert) {
+ if (seg > 0 && md->n_expert && part != 2) {
/* the previous layer's plan (gpu_token_moe.c): its rows' tokens, its
pairs' rows, its tables and dispatches' sizes, laid out one after
the other; and the experts read into the tier, or the CPU's rows,
@@ -947,8 +958,11 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_layer *ly = &layers[l];
int on = seg < 0 || seg == (int)l;
/* a mixture: the previous layer's experts first, into YD */
- if (on && l && md->n_expert)
- janas_gt_rec_experts(g, t, md, &layers[l - 1], &bound, YD, n);
+ if (on && l && md->n_expert) {
+ janas_gt_rec_experts(g, t, md, &layers[l - 1], &bound, YD, n, part);
+ if (part == 1)
+ goto end;
+ }
/* the previous layer's feed-forward added, then this one's norm */
if (on) {
janas_gt_rec_norm_quant(
@@ -983,8 +997,12 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,
tokens first .. n - 1 (none: the residual left in X, the next block's
keys and values already in the cache) */
if (!md->hc && (seg < 0 || seg == (int)nl)) {
- if (md->n_expert)
- janas_gt_rec_experts(g, t, md, &layers[nl - 1], &bound, YD, n);
+ if (md->n_expert) {
+ janas_gt_rec_experts(g, t, md, &layers[nl - 1], &bound, YD, n,
+ part);
+ if (part == 1)
+ goto end;
+ }
if (first < n || hid) {
janas_gt_rec_norm_quant(
g, t, &bound, X, YD,
@@ -1044,6 +1062,7 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,
}
g->vkCmdPipelineBarrier(g->cb, from, VK_PIPELINE_STAGE_HOST_BIT, 0, 1, &mb,
0, NULL, 0, NULL);
+end:
if (g->vkEndCommandBuffer(g->cb) != VK_SUCCESS)
return -1;
return 0;
@@ -1069,7 +1088,8 @@ uint32_t janas_gpu_tok_block_at(const struct janas_gpu_tok *t,
block of n tokens recorded into it: 0, or -1 with *cb NULL. */
int janas_gt_make_cb(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md, VkCommandBuffer *cb,
- uint32_t n, uint32_t first, int hi, uint32_t nch, int seg)
+ uint32_t n, uint32_t first, int hi, uint32_t nch, int seg,
+ int part)
{
VkCommandBufferAllocateInfo ca = {
.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO,
@@ -1083,7 +1103,7 @@ int janas_gt_make_cb(struct janas_gpu *g, struct janas_gpu_tok *t,
VkCommandBuffer keep = g->cb;
g->cb = *cb; /* the helpers record into g->cb */
t->rec_nch = nch;
- int rc = record(g, t, md, n, first, hi, seg);
+ int rc = record(g, t, md, n, first, hi, seg, part);
g->cb = keep;
if (rc != 0) {
g->vkFreeCommandBuffers(g->dev, g->cp, 1, cb);
@@ -1095,9 +1115,42 @@ int janas_gt_make_cb(struct janas_gpu *g, struct janas_gpu_tok *t,
int janas_gt_submit_wait(struct janas_gpu *g, const VkCommandBuffer *cb)
{
+ return janas_gt_submit_wait2(g, VK_NULL_HANDLE, cb);
+}
+
+int janas_gt_submit_list(struct janas_gpu *g, const VkCommandBuffer *cbs,
+ uint32_t n, int wait)
+{
+ VkCommandBuffer l[4];
+ uint32_t c = 0;
+ for (uint32_t i = 0; i < n && c < 4; i++)
+ if (cbs[i])
+ l[c++] = cbs[i];
+ VkSubmitInfo sub = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO,
+ .commandBufferCount = c,
+ .pCommandBuffers = l};
+ pthread_mutex_lock(&g->qlock);
+ VkResult r =
+ g->vkQueueSubmit(g->queue, 1, &sub, wait ? g->fence : VK_NULL_HANDLE);
+ pthread_mutex_unlock(&g->qlock);
+ if (r == VK_SUCCESS && wait)
+ r = g->vkWaitForFences(g->dev, 1, &g->fence, VK_TRUE, UINT64_MAX);
+ if (r != VK_SUCCESS) {
+ g->broken = 1;
+ return -1;
+ }
+ if (wait)
+ g->vkResetFences(g->dev, 1, &g->fence);
+ return 0;
+}
+
+int janas_gt_submit_wait2(struct janas_gpu *g, VkCommandBuffer pre,
+ const VkCommandBuffer *cb)
+{
+ VkCommandBuffer two[2] = {pre, *cb};
VkSubmitInfo sub = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO,
- .commandBufferCount = 1,
- .pCommandBuffers = cb};
+ .commandBufferCount = pre ? 2 : 1,
+ .pCommandBuffers = pre ? two : cb};
pthread_mutex_lock(&g->qlock);
VkResult r = g->vkQueueSubmit(g->queue, 1, &sub, g->fence);
pthread_mutex_unlock(&g->qlock);
@@ -1187,7 +1240,7 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,
*cb = VK_NULL_HANDLE;
}
if (!*cb) {
- if (janas_gt_make_cb(g, t, md, cb, n, first, hi, nch, -1) != 0)
+ if (janas_gt_make_cb(g, t, md, cb, n, first, hi, nch, -1, 0) != 0)
return -1;
t->cb_nch[n][ai][hi] = nch;
}
diff --git a/src/llm/gpu_token_impl.h b/src/llm/gpu_token_impl.h
index 5dc0fc5..4e95ca4 100644
--- a/src/llm/gpu_token_impl.h
+++ b/src/llm/gpu_token_impl.h
@@ -245,6 +245,8 @@ struct janas_gpu_tok {
double wall_ns; /* submission to the fence, as the CPU sees it */
double copy_ns; /* the logits copied out */
double plan_ns; /* a mixture's plans, on the CPU */
+ double cpu_ns; /* of which its experts computed by the CPU */
+ double fetch_ns; /* and the experts asked of the cache */
uint64_t tokens; /* with profiling: passes measured */
uint64_t pass_tokens; /* ... and their tokens */
VkShaderModule sm_nq, sm_qk, sm_at, sm_am;
@@ -283,6 +285,13 @@ struct janas_gpu_tok {
tiles2 on), the two dispatches' sizes */
VkCommandBuffer *seg_cb[TOK_MAX_BLOCK + 1][3][2];
uint32_t seg_nch[TOK_MAX_BLOCK + 1][3][2];
+ /* a plan's last copies recorded in g->cb, for the next segment's
+ submission */
+ int cp_pending;
+ /* a plan's experts left to the CPU (gpu_token_moe.c cpu_part), and the
+ command buffer of their rows' copies */
+ uint32_t pend_nc;
+ VkCommandBuffer cb_rows;
VkShaderModule sm_tk, sm_ga, sm_cb, sm_sm;
VkPipeline p_tk, p_ga, p_cb, p_sm;
size_t o_r, o_pe, o_pw, o_shw, o_sh, o_xg, o_egu, o_eh, o_dout, o_gt, o_inv,
@@ -373,8 +382,15 @@ void janas_gt_sync_range(struct janas_gpu *g, struct janas_gpu_tok *t,
NULL. */
int janas_gt_make_cb(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md, VkCommandBuffer *cb,
- uint32_t n, uint32_t first, int hi, uint32_t nch, int seg);
+ uint32_t n, uint32_t first, int hi, uint32_t nch, int seg,
+ int part);
int janas_gt_submit_wait(struct janas_gpu *g, const VkCommandBuffer *cb);
+/* The same with pre (or VK_NULL_HANDLE) submitted before it, together */
+int janas_gt_submit_wait2(struct janas_gpu *g, VkCommandBuffer pre,
+ const VkCommandBuffer *cb);
+/* cbs[0..n) in one submission, NULL ones left out; waited for if wait */
+int janas_gt_submit_list(struct janas_gpu *g, const VkCommandBuffer *cbs,
+ uint32_t n, int wait);
/* gpu_token_rec.c: the commands, recorded into g->cb */
void janas_gt_stamp(struct janas_gpu *g, struct janas_gpu_tok *t, int ph);
@@ -455,10 +471,12 @@ int janas_gt_rec_router(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md,
const struct janas_gpu_tok_layer *ly, uint32_t l,
VkPipeline *bound, uint32_t n);
+/* part 0 all of it, 1 the products, 2 the experts' sum */
void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md,
const struct janas_gpu_tok_layer *ly,
- VkPipeline *bound, VkDeviceAddress YD, uint32_t n);
+ VkPipeline *bound, VkDeviceAddress YD, uint32_t n,
+ int part);
int janas_gt_tier_create(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md);
void janas_gt_tier_destroy(struct janas_gpu *g, struct janas_gpu_tok *t);
diff --git a/src/llm/gpu_token_moe.c b/src/llm/gpu_token_moe.c
index fafe5d1..1bec0a3 100644
--- a/src/llm/gpu_token_moe.c
+++ b/src/llm/gpu_token_moe.c
@@ -16,14 +16,14 @@
#if defined(JANAS_GPU_SPV) && __has_include(<vulkan/vulkan.h>)
-/* A mixture's submissions for one block shape, freed (n_layer + 1 of
- them, the array too). */
+/* A mixture's submissions for one block shape, freed (two for each of the
+ n_layer + 1 segments, its products and the rest, the array too). */
void janas_gt_free_segments(struct janas_gpu *g, struct janas_gpu_tok *t,
VkCommandBuffer **seg)
{
if (!*seg)
return;
- for (uint32_t s = 0; s < t->n_seg; s++)
+ for (uint32_t s = 0; s < 2 * t->n_seg; s++)
if ((*seg)[s])
g->vkFreeCommandBuffers(g->dev, g->cp, 1, &(*seg)[s]);
free(*seg);
@@ -117,11 +117,14 @@ static void experts_product(struct janas_gpu *g, struct janas_gpu_tok *t,
void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
const struct janas_gpu_tok_model *md,
const struct janas_gpu_tok_layer *ly,
- VkPipeline *bound, VkDeviceAddress YD, uint32_t n)
+ VkPipeline *bound, VkDeviceAddress YD, uint32_t n,
+ int part)
{
VkDeviceAddress B = t->buf.addr;
uint32_t np = n * md->k, nbd = md->dm / JANAS_QK, nbf = t->effp / JANAS_QK;
int dt = ly->down.type, gtp = ly->gate.type;
+ if (part == 2)
+ goto sum;
struct push_ga pg = {.nb = nbd};
janas_g_push_addr(
pg.src, B + (janas_g_x_in_32(ly->gate.type) ? t->o_q32 : t->o_q256));
@@ -160,6 +163,9 @@ void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,
md->eff / 32, t->o_ind + 16);
janas_g_compute_barrier(g);
janas_gt_stamp(g, t, PH_DOWN);
+ if (part == 1)
+ return;
+sum:;
struct push_cb pc = {
.dm = md->dm, .k = md->k, .has_sh = ly->sh_gate_w.type != 0};
janas_g_push_addr(pc.dout, B + t->o_dout);
@@ -545,8 +551,16 @@ static int plan(struct janas_gpu *g, struct janas_gpu_tok *t,
t->miss[nm++] = e;
}
}
+ struct timespec f0, f1;
+ if (t->qp)
+ clock_gettime(CLOCK_MONOTONIC, &f0);
if (nm && md->fetch(md->ctx, l, t->miss, nm, t->cpu_slots) != 0)
return -1;
+ if (t->qp) {
+ clock_gettime(CLOCK_MONOTONIC, &f1);
+ t->fetch_ns += (double)(f1.tv_sec - f0.tv_sec) * 1e9 +
+ (double)(f1.tv_nsec - f0.tv_nsec);
+ }
/* misses into the tier where it gives up an expert used less;
the rest to the CPU */
int32_t *dst = (int32_t *)t->fill; /* free until the sort below */
@@ -643,32 +657,92 @@ static int plan(struct janas_gpu *g, struct janas_gpu_tok *t,
/* o_gt to o_ind are one range (gpu_token.c's layout), copied by the
next submission */
host_side(g, t, t->o_gt, t->o_ind + 32 - t->o_gt, 1);
- if (nc) {
- /* the CPU's experts, from the layer's normalized input (copied to
- the mirror by the last submission), into their rows */
- for (uint32_t i = 0; i < nc; i++)
- t->ra[i] = cnt[t->on_cpu[i]], t->rb[i] = cnt[t->on_cpu[i] + 1];
- host_side(g, t, t->o_xnf, (size_t)n * md->dm * 4, 0);
- if (md->cpu_rows(md->ctx, l, n, (const float *)(t->buf.map + t->o_xnf),
- nc, t->on_cpu, t->cpu_slots, t->ra, t->rb, gt,
- (float *)(t->buf.map + t->o_dout)) != 0)
+ /* the last experts read into the tier, in the next segment's
+ submission: a submission of their own, waited for, cost a round trip
+ a layer; the CPU's experts left for cpu_part */
+ if (ncp && janas_g_record_copies(g, t->cps, (uint32_t)ncp, 1) != 0)
+ return -1;
+ t->cp_pending = ncp != 0;
+ t->pend_nc = nc;
+ return 0;
+}
+
+/*
+ * The experts plan left to the CPU (t->pend_nc), from layer l's normalized
+ * input (copied to the mirror by the last submission) into their rows,
+ * while the GPU computes its own; their copies to the card's memory
+ * recorded in t->cb_rows, after a barrier on all that went before, for the
+ * submission of the rest of the segment. 0, or -1.
+ */
+static int cpu_part(struct janas_gpu *g, struct janas_gpu_tok *t,
+ const struct janas_gpu_tok_model *md, uint32_t l,
+ uint32_t n)
+{
+ uint32_t nc = t->pend_nc, *cnt = t->cnt,
+ *gt = (uint32_t *)(t->buf.map + t->o_gt), ncp = 0;
+ for (uint32_t i = 0; i < nc; i++)
+ t->ra[i] = cnt[t->on_cpu[i]], t->rb[i] = cnt[t->on_cpu[i] + 1];
+ host_side(g, t, t->o_xnf, (size_t)n * md->dm * 4, 0);
+ struct timespec c0, c1;
+ if (t->qp)
+ clock_gettime(CLOCK_MONOTONIC, &c0);
+ if (md->cpu_rows(md->ctx, l, n, (const float *)(t->buf.map + t->o_xnf), nc,
+ t->on_cpu, t->cpu_slots, t->ra, t->rb, gt,
+ (float *)(t->buf.map + t->o_dout)) != 0)
+ return -1;
+ if (t->qp) {
+ clock_gettime(CLOCK_MONOTONIC, &c1);
+ t->cpu_ns += (double)(c1.tv_sec - c0.tv_sec) * 1e9 +
+ (double)(c1.tv_nsec - c0.tv_nsec);
+ }
+ for (uint32_t i = 0; i < nc; i++) {
+ size_t o = t->o_dout + (size_t)t->ra[i] * md->dm * 4,
+ b = (size_t)(t->rb[i] - t->ra[i]) * md->dm * 4;
+ host_side(g, t, o, b, 1);
+ if (t->local)
+ t->cps[ncp++] = (struct janas_g_cp){.src = t->mirror.buf,
+ .dst = t->buf.buf,
+ .so = o,
+ .dof = o,
+ .bytes = b};
+ }
+ if (!t->cb_rows) {
+ VkCommandBufferAllocateInfo ca = {
+ .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO,
+ .commandPool = g->cp,
+ .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY,
+ .commandBufferCount = 1};
+ if (g->vkAllocateCommandBuffers(g->dev, &ca, &t->cb_rows) !=
+ VK_SUCCESS) {
+ t->cb_rows = VK_NULL_HANDLE;
return -1;
- for (uint32_t i = 0; i < nc; i++) {
- size_t o = t->o_dout + (size_t)t->ra[i] * md->dm * 4,
- b = (size_t)(t->rb[i] - t->ra[i]) * md->dm * 4;
- host_side(g, t, o, b, 1);
- if (t->local)
- t->cps[ncp++] = (struct janas_g_cp){.src = t->mirror.buf,
- .dst = t->buf.buf,
- .so = o,
- .dof = o,
- .bytes = b};
}
}
- /* the last experts read into the tier and the CPU's rows, together */
- if (ncp && janas_g_copies(g, t->cps, (uint32_t)ncp) != 0)
+ VkCommandBufferBeginInfo bb = {
+ .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
+ .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};
+ if (g->vkResetCommandBuffer(t->cb_rows, 0) != VK_SUCCESS ||
+ g->vkBeginCommandBuffer(t->cb_rows, &bb) != VK_SUCCESS)
return -1;
- return 0;
+ /* after the products submitted before, without a wait between */
+ VkMemoryBarrier all = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER,
+ .srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT,
+ .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT |
+ VK_ACCESS_MEMORY_WRITE_BIT};
+ g->vkCmdPipelineBarrier(t->cb_rows, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT,
+ VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, 1, &all, 0,
+ NULL, 0, NULL);
+ for (uint32_t i = 0; i < ncp; i++) {
+ VkBufferCopy cp = {.srcOffset = t->cps[i].so,
+ .dstOffset = t->cps[i].dof,
+ .size = t->cps[i].bytes};
+ g->vkCmdCopyBuffer(t->cb_rows, t->cps[i].src, t->cps[i].dst, 1, &cp);
+ }
+ if (ncp)
+ g->vkCmdPipelineBarrier(t->cb_rows, VK_PIPELINE_STAGE_TRANSFER_BIT,
+ VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, 1, &all,
+ 0, NULL, 0, NULL);
+ return g->vkEndCommandBuffer(t->cb_rows) == VK_SUCCESS ? 0 : -1;
}
/* A mixture's block: its n_layer + 1 submissions, each recorded once for
@@ -681,7 +755,7 @@ int janas_gt_run_segments(struct janas_gpu *g, struct janas_gpu_tok *t,
if (*seg && t->seg_nch[n][ai][hi] != nch)
janas_gt_free_segments(g, t, seg); /* recorded for other runs: again */
if (!*seg) {
- *seg = calloc(t->n_seg, sizeof(**seg));
+ *seg = calloc(2 * t->n_seg, sizeof(**seg));
if (!*seg)
return -1;
t->seg_nch[n][ai][hi] = nch;
@@ -689,6 +763,7 @@ int janas_gt_run_segments(struct janas_gpu *g, struct janas_gpu_tok *t,
/* a block whose tokens choose some 4 times each expert or more: those
few that no token takes are under 2% (e^-4) */
int ahead = md->prefetch && (size_t)n * md->k >= 4 * (size_t)md->n_expert;
+ t->cp_pending = 0, t->pend_nc = 0;
for (uint32_t s = 0; s < t->n_seg; s++) {
struct timespec p0, p1;
if (t->qp)
@@ -708,11 +783,35 @@ int janas_gt_run_segments(struct janas_gpu *g, struct janas_gpu_tok *t,
no faster (10 Oct 2026) */
if (ahead && s + 1 < t->n_seg)
md->prefetch(md->ctx, s);
- if (!(*seg)[s] && janas_gt_make_cb(g, t, md, &(*seg)[s], n, first, hi,
- nch, (int)s) != 0)
- return -1;
- if (janas_gt_submit_wait(g, &(*seg)[s]) != 0)
+ /* past the first: the segment's products of the experts, then the
+ rest - the CPU computing its experts between the two
+ submissions, the GPU its own (qwen4exp's streams whole) */
+ int split = s && !md->hc;
+ VkCommandBuffer *a = &(*seg)[2 * s], *b = &(*seg)[2 * s + 1];
+ if ((!*a && janas_gt_make_cb(g, t, md, a, n, first, hi, nch, (int)s,
+ split) != 0) ||
+ (split && !*b &&
+ janas_gt_make_cb(g, t, md, b, n, first, hi, nch, (int)s, 2) != 0))
return -1;
+ VkCommandBuffer pre = t->cp_pending ? g->cb : VK_NULL_HANDLE;
+ t->cp_pending = 0;
+ if (!split) {
+ if (janas_gt_submit_wait2(g, pre, a) != 0)
+ return -1;
+ } else if (!t->pend_nc) {
+ VkCommandBuffer l3[3] = {pre, *a, *b};
+ if (janas_gt_submit_list(g, l3, 3, 1) != 0)
+ return -1;
+ } else {
+ VkCommandBuffer l2[2] = {pre, *a};
+ if (janas_gt_submit_list(g, l2, 2, 0) != 0 ||
+ cpu_part(g, t, md, s - 1, n) != 0)
+ return -1;
+ VkCommandBuffer r2[2] = {t->cb_rows, *b};
+ if (janas_gt_submit_list(g, r2, 2, 1) != 0)
+ return -1;
+ }
+ t->pend_nc = 0;
}
return 0;
}