Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Commits on main
ci: GitHub Actions for engine, web UI, Python tests, and CUDA syntax check
58ead39 ZacharyZcR committed 4 months ago
README: GB10 (DGX Spark) row (#136) β 0.50 tok/s warm, matmul-bound, unified memory; confirms #76 fix on aarch64 (TF 32/32 + greedy 20/20 on NEON and CUDA sm_121)
6d3ed7e JustVugg committed 4 months ago
README: native-Windows warm-cache row (#128, 0.03->0.5 tok/s) β honest label per #131 (pipe+RAM fixes, no GPU)
086b2df JustVugg committed 4 months ago
Windows native port: serve-mode pipe fix + RAM detection + POSIX guards, AVX-VNNI kernel, gated CUDA DLL (#131, fixes #123)
2319b94 woolcoxm committed 4 months ago
Makefile: detect Windows from $(OS)=Windows_NT before uname so make works from native PowerShell/CMD (#129)
afc259c bopof committed 4 months ago
docs: Metal M5 Max perf report β OMP hot-team spin (#77) throttles Apple GPU, NO_OMP+PIPE recovers to 2.24 tok/s (#116)
5dd7503 RDouglas committed 4 months ago
quant_ablation: add rotation preconditioning (-rot, QuaRot/QuIP#) and int3 schemes (#132, #81)
f8c0552 ZacharyZcR committed 4 months ago
olmoe: reject quant_bits outside 2..8 (fixes degenerate output, #134) + correct ref.json to OLMoE-0125-Instruct oracle (#133)
1f0e1b7 JustVugg committed 4 months ago
README: benchmark rows for Gen5-NVMe 9950X3D2+5090 (1.23 tok/s, #120) and first Strix Halo datapoint (1.10 tok/s, #124)
4418e16 JustVugg committed 4 months ago
coli: honor --gpu/--vram without --auto-tier, fail-fast on CPU-only binary (#125, fixes #121)
5f49a5a ZacharyZcR committed 4 months ago
attention: size per-thread score buffer by the batch's true max nt, not Tk+1 β fixes serve heap-overflow (#117)
d439ac8 Dennis Paul committed 4 months ago
openai server: type tool arguments from schema + implement tool_choice, 9 new tool-calling tests (#118)
15bf8ca Dennis Paul committed 4 months ago
expert dot: AVX-512 int4βfloat accumulator, runtime-switchable (I4_ACC512), +7% CPU-heavy decode, more accurate than AVX2 order (#95)
4b1d0e3 ZacharyZcR committed 4 months ago
tools: quant_ablation.py β engine-free A/B of any quantization scheme vs fp16 (fake-quant, isolates weight error) (#108, #115)
97c756a Dennis Paul committed 4 months ago
Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)
cbd5990 ZacharyZcR committed 4 months ago
README: first native-Windows datapoint (#113, i5-12600K MinGW, MTP int8 57% β port validated)
cb53589 JustVugg committed 4 months ago
Merge main into dev: quality bench result + M4 Pro row + bench deps (#107,#108)
05df180 JustVugg committed 4 months ago
README: first quality benchmark result (#108, 62.5% with confound caveats + OLMoE A/B as decisive test), bench pip deps (tokenizers datasets), M4 Pro Metal row (#107)
5254470 JustVugg committed 4 months ago
Metal backend (Apple Silicon): batched experts + fused attention on GPU, unified-memory zero-copy, gated behind COLI_METAL β 2.06 tok/s M5 Max (#72, #87, #103)
3716e40 Code Arranger committed 4 months ago
attention: heap per-thread score buffers β fix stack overflow past 8192 ctx on non-DSA snapshots (both absorb + dense paths, oracle-exact) (#110)
342ceaf John Balis committed 4 months ago
README: EPYC 7443 430GB datapoint (#104) β hit 98%, disk eliminated, RAM-bandwidth+matmul bound; evidence lower-bit quant helps RAM-bound hosts too
a78a06f JustVugg committed 4 months ago
README: M5 Max Metal 2.06 tok/s β new record, coherent output, on the pre-rebase branch (#103)
a9f3f35 JustVugg committed 4 months ago
Merge main into dev: generalized reproducibility note (#100)
8ce825f JustVugg committed 4 months ago
README: generalize the reproducibility note (#100) β quantized-kernel rounding forks greedy across MTP/CUDA/batched, not just MTP; exact-mode recipe DRAFT=0 IDOT=0 COLI_CUDA=0
92011de JustVugg committed 4 months ago