Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Commits on main
README: corrected 9800X3D benchmark row (#101) β 10.5 GB/s VHDX still disk-bound, CUDA tier ~0% when AVX-512 CPU matches the 5090; CUDA value is CPU-dependent
2baac89 JustVugg committed 4 months ago
VSX integer-dot kernels for POWER8 (12.8x int8 / 7.6x int4 over scalar, #ifdef-gated, x86 path untouched) (#98)
bc6bc9c AutoJanitor committed 4 months ago
perf(pilot): PILOT_REAL real cross-layer expert loads, independent from PIPE pool (data-driven: two pools beat layered/unified on both disk- and matmul-bound hosts) (#78)
9bba681 Rodolfo Hansen committed 4 months ago
Merge main into dev: README honesty fixes (#100,#101) + #96 tool calling + #97 ppc64le (landed on main by mistake, syncing dev)
69e3a3a JustVugg committed 4 months ago
OpenAI tool calling for GLM-5.2: gated tool declaration + parse_tool_calls, non-tool path preserved, opt-in salvage (#96)
0f9f99c RDouglas committed 4 months ago
Makefile: Linux PowerPC (ppc64le) build support β -mcpu, scalar fallback, x86 path untouched (#97)
6aaa8fc AutoJanitor committed 4 months ago
README honesty fixes: MTP not byte-identical to greedy (shape-dependent kernels, #100); VHDX ceiling was this drive not the virtualization layer (#101, community measured 10.5 GB/s)
f1fbbca JustVugg committed 4 months ago
docs: GLM-5.2 6x RTX 5090 full-residency experiment β 6.84 tok/s, honest dead-ends log (#94)
3fb5a00 ZacharyZcR committed 4 months ago
docs: adopt dev integration branch β PRs target dev, batched into stable main (ZacharyZcR's suggestion)
f38e74e JustVugg committed 4 months ago
README: Metal backend M5 Max benchmark β 1.83 tok/s, fastest datapoint yet (#72, #87)
e945b43 JustVugg committed 4 months ago
README: correct M5 Max disk number (14.2 GB/s was cache-influenced ~4 GB/s real) + iobench cold-read caveat, macOS F_NOCACHE limitation (#86)
928fd67 JustVugg committed 4 months ago
perf(moe): overlap NVMe expert loads with matmul via opt-in PIPE I/O pool (default off, oracle-exact) (#79)
03d9a23 Rodolfo Hansen committed 4 months ago
perf(omp): keep OpenMP worker team hot across tiny per-expert matmul regions (seed + re-exec once, fully overridable) (#77)
704ed49 Rodolfo Hansen committed 4 months ago
Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)
2416bc9 Sidd committed 4 months ago
Guard against running the tiny oracle against a real model (#76)
6e7aa6f JustVugg committed 4 months ago
Tauri desktop shell (reuses web/ frontend, connects to OpenAI-compatible endpoint, isolated in desktop/) (#24)
5198fee ZacharyZcR committed 4 months ago
Web UI: runtime console + KV session panel, with runtime/storage libs and tests (#32)
57730f6 ZacharyZcR committed 4 months ago
Add flake.nix for reproducible NixOS builds (engine + converter python env) (#66)
d7ffdc4 Maikel Frias Mosquea committed 4 months ago
Grammar-forced speculative drafts: GBNF grammar as a third draft source, guaranteed-accepted forced spans, lossless + opt-in (#48, #70)
cec7d6b Fabio Rovai committed 4 months ago
iobench: Windows correctness β aligned free, 64-bit totals, full-range random offsets (also improves Linux offset coverage) (#64)
1bab324 nalepy committed 4 months ago
coli doctor: read-only setup/health diagnostics (path, config, shards, disk, RAM budget, placement) (#33)
8f5f3e3 ZacharyZcR committed 4 months ago
setup.sh: detect libomp by dylib presence, not brew prefix (false-positive fix by fabio-rovai, #47)
1bdaeee JustVugg committed 4 months ago
Windows 11 native port, phase 1: MinGW-w64 static build, compat shims, setup + docs (#40)
89d95fc nalepy committed 4 months ago
Reduce allocation overhead in quantized matmul and MoE routing (thread-local scratch, trimmed temporaries) (#43)
7d8d5f3 0xhis committed 4 months ago
README: Strix Halo warm numbers (#39) β 0.16 -> 0.40 tok/s in five runs, learned pin 47.6 GB, fastest non-Apple datapoint
a5fc89e JustVugg committed 4 months ago
README: Strix Halo + Optane 905p datapoint (#39) β first Ryzen AI Max+, Optane latency invisible to 19MB-chunk reads
ae8975a JustVugg committed 4 months ago
README: 9950X disk-swap datapoint (#31) β same machine, P3 vs 9100 PRO: x5.8 disk = x2.9 tokens, profile flips 66% disk -> 57% matmul
c809bc8 JustVugg committed 4 months ago
Isolated sequence KV contexts: KVState extraction, KV_SLOTS serve contexts with per-slot persistence, budget-aware pool accounting (#29)
9e6e1ac ZacharyZcR committed 4 months ago
Bounded FIFO inference scheduling: admission queue, OpenAI-shaped 429/503, queued-disconnect detection (#28)
76e858a ZacharyZcR committed 4 months ago
coli plan: read-only Disk/RAM/VRAM placement planner (mirrors engine budget math, versioned --json, --auto-tier) (#27)
3cbc52a ZacharyZcR committed 4 months ago