Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
Commits on main
build: portable build_cuda.bat with MSVC/CUDA/arch auto-detection
857f725 Marco Godoy committed 4 months ago
COLI_GROUP_ASYNC: don't recompute a device whose async group already succeeded
aabb4eb 刘宣岐 committed 4 months ago
EXPERT_BUDGET: never trim a position to zero experts; compensate weights without norm_topk
1cc12c0 刘宣岐 committed 4 months ago
moe: route the eight hot-path allocations through a checked alloc
a13652e 刘宣岐 committed 4 months ago
router: guard best==-1 in top-K selection (NaN logits caused OOB read/writes)
6dc1ee0 刘宣岐 committed 4 months ago
pipe2: download the resident residual before every exit from residency
9da4c70 刘宣岐 committed 4 months ago
cuda: DSA sparse decode on GPU via gather staging (COLI_DSA_GATHER=1)
56b2ef9 ZacharyZcR committed 4 months ago
cuda: fuse silu into the dual hidden kernels' epilogue — one less launch and no up[] round-trip per group call
a03c79e ZacharyZcR committed 4 months ago
prefill: COLI_PREFILL_CHUNK=N — slice long prompts, bound activation memory to chunk size
8c00e0e ZacharyZcR committed 4 months ago
serve: cross-slot KV prefix adoption (COLI_KV_SHARE=1) — shared-prefix TTFT 50.1s -> 1.7s
8b3f06c ZacharyZcR committed 4 months ago
tools: make convert_olmoe_merged.py disk/RAM-safe (streaming, resumable)
ee0b40d BE committed 4 months ago
olmoe: add an AVX2 int8 dot-product path to matmul_q (x86 had none)
2c4e9de BE committed 4 months ago
test: fix CUDA env test for colibri rename + CPU-only CI
6a92177 Hermes Quant Auditor committed 4 months ago
openai_server: serve the inkling engine (ARCH detect + reasoning channel)
aa188fe Nicholas Beerbower committed 4 months ago
inkling: serve mode (openai_server.py engine protocol) + sampling
2354540 Nicholas Beerbower committed 4 months ago
inkling: docs/inkling.md + honest usage-save reporting
6ebac54 Nicholas Beerbower committed 4 months ago
inkling: -f promptfile mode for usage warmup (one load, many prompts)
deb3bd0 Nicholas Beerbower committed 4 months ago
inkling: default PIN_N to cap/2 (measured: pinning is the whole ballgame)
d2fc927 Nicholas Beerbower committed 4 months ago
inkling: CUDA tier — bf16 residents in VRAM (A6000), +28% decode, best prefill
a34edb2 Nicholas Beerbower committed 4 months ago
inkling: fuse gate+up matmuls, OMP hot-thread re-exec (neutral on decode)
524f2a4 Nicholas Beerbower committed 4 months ago