Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

tier: fit the small-allocation footprint to the measured cudaMalloc curve

kreuzzelg40ff64502264
diff --git a/c/qwen36_tier.c b/c/qwen36_tier.c
index 2e610715..0aae56bb 100644
--- a/c/qwen36_tier.c
+++ b/c/qwen36_tier.c
@@ -456,10 +456,19 @@ int qt_init_fp8(int nl, int ne, int D, int Ih, int cap, int topk, const float *e
/* VRAM an allocation of `bytes` really occupies (cudaMalloc granularity,
* see the exp_bytes comment in qt_init). */
static size_t dev_alloc_footprint(size_t bytes){
- const size_t MiB = 1048576u;
- if(bytes <= 10240u) return 10240u;
- if(bytes <= MiB) return MiB;
- return (bytes + 2*MiB - 1) / (2*MiB) * (2*MiB);
+ /* measured with cudaMemGetInfo over 256 allocations each (driver 5xx):
+ * 400 B, 3 KiB, 4 KiB -> 8 KiB 10 KiB -> 16 KiB 16..64 KiB -> exact
+ * 96 KiB -> 104 KiB 384 KiB -> 416 KiB 768 KiB -> 1 MiB 1 MiB -> 1 MiB
+ * 1.5 MiB -> 2 MiB 3 MiB -> 4 MiB
+ * i.e. above 1 MiB multiples of 2 MiB, above 512 KiB one 1 MiB page, and
+ * below that roughly the size plus a sixteenth, in 8 KiB steps, 8 KiB
+ * minimum. The small-size rule is a fit, slightly conservative. */
+ const size_t KiB = 1024u, MiB = 1048576u;
+ if(bytes > MiB) return (bytes + 2*MiB - 1) / (2*MiB) * (2*MiB);
+ if(bytes > 512*KiB) return MiB;
+ size_t b = bytes + bytes/16;
+ if(b < 8*KiB) b = 8*KiB;
+ return (b + 8*KiB - 1) / (8*KiB) * (8*KiB);
}
int qt_init(int nl, int ne, int D, int Ih, int cap, int topk, int expert_gs,
@@ -544,9 +553,9 @@ int qt_init(int nl, int ne, int D, int Ih, int cap, int topk, int expert_gs,
G.sc_d = expert_gs ? (size_t)D * ((Ih + expert_gs - 1)/expert_gs) : (size_t)D;
}
/* Charge what the device allocator takes, not what the bytes measure:
- * cudaMalloc rounds an allocation above 1 MiB up to a multiple of 2 MiB
- * and one at or below 1 MiB up to 1 MiB (measured with cudaMemGetInfo,
- * driver 5xx; 1,638,400 B -> 2 MiB, 819,200 B -> 1 MiB, 400 B -> 10 KiB).
+ * cudaMalloc rounds an allocation above 1 MiB up to a multiple of 2 MiB,
+ * one above 512 KiB up to 1 MiB, and small ones to 8 KiB steps
+ * (dev_alloc_footprint has the measured table).
* An expert is three weight allocations plus three scale allocations.
* Charged by payload, the fp8 Qwen3.8 expert (3 x 1.56 MiB) looked like
* 4.69 MiB and took 6.03 MiB: the budget filled the card to the last