Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: the shipped expert profiles, and what they do to a first run
JustVuggd1e29696b731
diff --git a/docs/qwen36.md b/docs/qwen36.mdindex 3a31475b..7e1af17a 100644--- a/docs/qwen36.md+++ b/docs/qwen36.md@@ -95,7 +95,8 @@ trunk on the CPU anywhere and `COLI_VK_DENSE=1` puts it on the device anywhere;This engine kept no expert history before; with the tier on it keeps route_trace.h's`.coli_usage` beside the container (`COLI_USAGE` moves it), saved at the end of-every run and serve turn, and fills the tier from it at the next start. With+every run and serve turn, and fills the tier from it at the next start. `coli+setup` gives Qwen3.6-35B-A3B a starting one ([vulkan.md](vulkan.md#the-routed-expert-tier-vk_tierc)). With`COLI_CUDA=1` as well, the CUDA tier wins.On an integrated Radeon 780M, with the int4 gs64 container at cap 64 and the trunkdiff --git a/docs/qwen38.md b/docs/qwen38.mdindex e377ef66..1ad0e73e 100644--- a/docs/qwen38.md+++ b/docs/qwen38.md@@ -367,7 +367,8 @@ alternating.In a `VK=1` build, `COLI_VULKAN=1` puts the trunk on the Vulkan device (as before)and the routed experts of the model's layers on the shared Vulkan expert tier(`c/vk_tier.c`, [vulkan.md](vulkan.md#the-routed-expert-tier-vk_tierc)): a cache of-experts in device memory, filled at startup from `.coli_usage` and adapted while+experts in device memory, filled at startup from `.coli_usage` (`coli setup` ships a+starting one) and adapted whileyou chat, whose experts the device computes while the CPU computes the rest of thestep. It takes every expert form this engine reads: the int4-g64 sidecar's planarrecords (as int4 groups of 64 on the device), the release's FP8 with the 128x128diff --git a/docs/vulkan.md b/docs/vulkan.mdindex a825f173..6127d7ad 100644--- a/docs/vulkan.md+++ b/docs/vulkan.md@@ -396,6 +396,17 @@ of them on the device the way a GPU-equipped PC should use its card:than 1.25 x their heat + 4). A promotion copies the expert's bytes once on the engine thread(at most `COLI_VK_TIER_RATE` per token, 16) and an uploader thread writes it tothe device; it serves from the next layer step on.+- **A fresh install starts warm.** A model downloaded a minute ago has no history,+ so its first runs would fill the tier one routing at a time. `coli setup` copies a+ starting history into the model folder when it has none, for the catalog models+ that ship one (`c/profiles/<catalog id>.coli_usage`: Qwen3.6-35B-A3B and Qwen3.8+ Flash Next). It comes from a calibration session (16 prompts: chat in five+ languages, code, reasoning, JSON), and the engine adds every run's routing to it,+ so your own use takes over; an existing history is never replaced. Measured on a+ Radeon 780M with Qwen3.6-35B-A3B, two prompts outside the calibration set, 128+ tokens, two rounds each: the first run went from 5.9-6.2 tok/s without it to+ 10.9-11.1 with it, the device serving 92-94% of the routed experts instead of+ 15-16%, for a warm start of about 3 s.- **The device and the CPU compute at the same time.** For each MoE layer step therouted (row, expert) pairs whose expert is resident go to the device as ONEsubmit that nobody waits for: per expert, its rows run gate+up and the activation