Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Commits on main
vulkan: int3-g64 (fmt=5) decode in all three shaders + tier plumbing
eab1593 Steve Markgraf committed 4 months ago
docs: Vulkan backend (vulkan.md, ENVIRONMENT.md table, README pointers)
1b0a89c Steve Markgraf committed 4 months ago
VK attn: require the f32 KV cache (Lc/Rc non-NULL) before mirroring
125e9a9 Steve Markgraf committed 4 months ago
VK: default COLI_VK_EXPERTS to 320 (sweep optimum on a 16 GB card)
4285198 Steve Markgraf committed 4 months ago
VK tier fill: expert_load returns 0 on success β the transient-load path was inverted
3a6f0ba Steve Markgraf committed 4 months ago
VK tier: registry-served experts skip RAM slots, disk loads, and prefetch entirely
7c4d7af Steve Markgraf committed 4 months ago
VK: pinned+async expert tier β heat-ranked startup residency, GPU batch overlapped with CPU rows
dd024ec Steve Markgraf committed 4 months ago
VK: fix o-proj OOB staging (correctness), fuse absorb+o and shared expert (1 submit each)
363c376 Steve Markgraf committed 4 months ago
glm: COLI_VK_EXPERTS=0 disables the VK expert tier (experts stay on the parallel CPU loop)
936d13e Steve Markgraf committed 4 months ago
glm: VK MLA absorb attention core (COLI_VK_ATTN) β the last CPU compute piece on Vulkan-only machines
559383d Steve Markgraf committed 4 months ago
glm: VK dense offload (COLI_VK_DENSE) β attention projections + shared expert on Vulkan
7c2f45f Steve Markgraf committed 4 months ago
glm: wire the Vulkan expert tier into moe() (COLI_VULKAN=1)
22eb4e9 Steve Markgraf committed 4 months ago
Makefile: VK=1 target builds the Vulkan backend + SPIR-V shaders
811c427 Steve Markgraf committed 4 months ago
Vulkan backend: cached readback buffers -> per-call expert_group beats ROCm
49c84d6 Steve Markgraf committed 4 months ago
Vulkan backend: full coli_vk_expert_group (gate_up->down, on-device, one submit)
aac6355 Steve Markgraf committed 4 months ago
Vulkan backend: fused gate+up+silu kernel -> beats production ROCm expert
e354ded Steve Markgraf committed 4 months ago
Vulkan backend: subgroup shader + overhead caching + batched probe
3e41761 Steve Markgraf committed 4 months ago
Vulkan compute backend (from PR #84), M1: builds + correct on RX 9070
38af749 Steve Markgraf committed 4 months ago
Merge pull request #701 from JustVugg/feat/inkling-int4-dense-and-cache-fix
df82781 Vincenzo Fornaro committed 3 months ago
fix(inkling): expert cache served the wrong experts; int4 dense container; --chat
5f75d67 JustVugg committed 3 months ago
Merge pull request #694 from terrizoaguimor/expert-gb-warning
1532bc7 Vincenzo Fornaro committed 3 months ago
Merge pull request #698 from ariannamethod/expert-page-cache
72f2851 Vincenzo Fornaro committed 3 months ago
olmoe: keep expert pages in the page cache (EXPERT_DROP=1 restores the old drop)
86a7683 Arianna Method committed 3 months ago
kimi_k3: --chat β K3's XTML chat format with the preserved-thinking channel
eb3fbbc Steve Markgraf committed 3 months ago