Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
Commits on main
REPIN follows live session heat (decaying heat map; .coli_usage stays the persistent signal) + disk→RAM→VRAM promotion (#26)
1453dab ZacharyZcR committed 4 months ago
OpenAI-compatible HTTP API: stdlib-only gateway over SERVE with KV prefix reuse across stateless requests (#21)
3e4d08b ZacharyZcR committed 4 months ago
Harden against hostile model files: config dim validation at parse + overflow-safe falloc (PR #25 report, done right)
ea5f6fb JustVugg committed 4 months ago
README: catch up with the last two days — DSA is done (was 'in progress'), KV persistence, PILOT, web/ UI section + repo layout, honest line count
1a909f3 JustVugg committed 4 months ago
Organize project tools and local workflows: c/tools, c/scripts, c/tests, root Makefile (flat C core untouched) (#22)
13e8f09 ZacharyZcR committed 4 months ago
Web interface (React/TS, shadcn, pure OpenAI-API client) under web/ (#23)
a2942b2 ZacharyZcR committed 4 months ago
KV-to-disk: conversations reopen WARM across engine restarts (.coli_kv, KVSAVE=0 opts out)
8f99e12 JustVugg committed 4 months ago
Local project checks and contribution templates: make check, dependency-free C/python tests, issue forms + blank issues, CONTRIBUTING (#20)
8a2e443 ZacharyZcR committed 4 months ago
README: Framework 13 warm numbers (0.37 tok/s, hit 66%, MTP 52% — #12), int8-MTP mirror clone link (#2), Epyc row column fix
9911199 JustVugg committed 4 months ago
LOOKA: routing predictability counters; PILOT: router-lookahead disk prefetch (async I/O thread); cap auto-raise to RAM budget (#12)
0778942 JustVugg committed 4 months ago
Tiered CUDA acceleration for routed experts (opt-in, CPU default untouched) + REPLAY fixture harness (#16)
57706a0 ZacharyZcR committed 4 months ago
README: community benchmark — Epyc 9654 ES, PCIe3 NVMe (0.31 tok/s, hit 35%) (#17)
4ea9ddc adapt-L committed 4 months ago
README: accurate SSD note (reads don't wear the drive; swap + thermals are the real concerns)
ed3916b JustVugg committed 4 months ago
MTP head is int8 by default (int4 head: 0-4% acceptance, unusable — measured in #8); README: honest MTP numbers + M5 Max community datapoint (#4, #5)
c035f41 JustVugg committed 4 months ago
RFC: live re-pin of the hot-store between turns (REPIN=n, opt-in, default off) (#11)
ab69508 RDouglas committed 4 months ago
IDOT: NEON single-token int4 gate (g_i4s/I4S) — SDOT makes S=1 decode pay on Apple Silicon (+14%, expert-matmul -16%) (#10)
3daceaf RDouglas committed 4 months ago
IDOT: AVX-512 VNNI kernel paths + batch-size gate for single-row int4 (+20% decode on Xeon) (#9)
8040dac Dog279 committed 4 months ago
PIN: mlock pinned experts into physical RAM (macOS: stop the compressor evicting the hot-store) (#7)
cfd16b4 RDouglas committed 4 months ago
converter: multi-stream, Range-resume shard downloader (2-4x faster, no lost bytes) (#6)
e74eb07 RDouglas committed 4 months ago
README: add measured community benchmark (WSL2, 24 GB RAM — issue #2) (#3)
996ae0e DatPat committed 4 months ago
coli chat: streaming markdown renderer — clean chat, no raw markers
2bd886e JustVugg committed 4 months ago
Revert "coli: multi-model registry + picker (coli models / chat menu); README: MoE-only model roadmap (gpt-oss-120b next)"
492c3b6 JustVugg committed 4 months ago
coli: multi-model registry + picker (coli models / chat menu); README: MoE-only model roadmap (gpt-oss-120b next)
289befb JustVugg committed 4 months ago
coli bench: self-contained (venv python + auto-fetch datasets); README calls for a quality run on faster hardware
b71d388 JustVugg committed 4 months ago
DSA lightning indexer: full implementation with auto-detection
193d2ce JustVugg committed 4 months ago
macOS/Apple Silicon port: NEON kernels + platform shim (PR #1 by RDouglasSharp)
dd8c907 RDouglas committed 4 months ago
README: today's engine work (integer kernels, MLA absorption, readahead, sampling, DSA status)
8d7fbef JustVugg committed 4 months ago
async I/O: WILLNEED readahead of the next expert block while computing the current one
7a5ffd1 JustVugg committed 4 months ago