Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Commits on main
server: clamp max_tokens to the server cap instead of rejecting it (fixes #260)
da260c3 JustVugg committed 4 months ago
coli: consume the engine's TIERS line at startup so it doesn't leak into the first answer
b796108 Skeldoor committed 4 months ago
glm.c/coli: Ctrl-C soft-stops the current turn instead of killing the engine
3745cc6 Skeldoor committed 4 months ago
glm.c: 4-accumulator NEON SDOT for int8/int4 expert dots (2.4x/core on Apple Silicon)
af421d6 Skeldoor committed 4 months ago
Merge pull request #246 from woolcoxm/feat/diskio-kv-batching
4d04383 Vincenzo committed 4 months ago
windows: PIPE default ON + compat_fadvise WILLNEED cache-warmer (pread path)
25219de woolcoxm committed 4 months ago
serve mux: keep ragged decode batches off the fused Metal kernels
0811730 monotophic committed 4 months ago
tools: E8 lattice quantization for the ablation harness β int{2,3,4,8}[-gN][-e8|-e8u][-rot] (#81)
0189a9f ZacharyZcR committed 4 months ago
experiment: miss-aware EXPERT_BUDGET β keep all cache hits, only drop misses
dd0d606 woolcoxm committed 4 months ago
experiment: EXPERT_BUDGET=N β cap distinct experts/layer, up to 1.8x faster decode
413b370 woolcoxm committed 4 months ago
diskio: KV write batching + persistent handle; generators: unfuse experts
71c262c woolcoxm committed 4 months ago
windows-dev: CUDA build script, expert budget + grouped quant research
e71d4fb woolcoxm committed 4 months ago
test-models: add --fp8 emission for the FP8->int4 converter test path
2c6946c woolcoxm committed 4 months ago
diskio: research on disk I/O minimization for MoE expert streaming
21eb86a woolcoxm committed 4 months ago
Windows-dev: grouped quantization + mixed precision + expert budget + download tool
69f65d5 woolcoxm committed 4 months ago
serve: print the KV prefix-reuse diagnostic in mux mode too β [API] KV slot line existed only in the legacy \x02PROMPT path (#153)
d04d99e ZacharyZcR committed 4 months ago
diskio: KV write batching + persistent handle; generators: unfuse experts
e2ad6c7 woolcoxm committed 4 months ago
Merge pull request #242 from woolcoxm/feat/grouped-quant-fmt4
3fd47b7 Vincenzo committed 4 months ago
tools: one expert atlas, not two β retire expert_atlas.py, analyze.py gains --web for the dashboard
e7cb7df ZacharyZcR committed 4 months ago