Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Commits on main
kimi_k3: chunked prefill (bit-identical, 2.6x) + AVX2 KDA state sweeps
60945c6 Steve Markgraf committed 3 months ago
kimi_k3: pipeline expert loads under compute + K3_IDOT + K3_TOPP
998c2f9 Steve Markgraf committed 3 months ago
quant.h: matmul_mxfp4_i8 β int8-activation MXFP4 matmul (integer dots)
d6cc710 Steve Markgraf committed 3 months ago
kimi_k3: K3_LOGITS β dump f32 logits per prefill position
e3a0e78 Steve Markgraf committed 3 months ago
kimi_k3: saturate the NVMe β parallel O_DIRECT expert reads + RAM fixes
cfd6ac2 Steve Markgraf committed 3 months ago
tools: K3 tokenizer generator, numpy layer reference, container repack
8a2eccb Steve Markgraf committed 3 months ago
kimi_k3: Kimi K3 engine (KDA + gated NoPE MLA + AttnRes + LatentMoE)
7fbc5cf Steve Markgraf committed 3 months ago
tok.h: Kimi pre-tokenizer family + rank-BPE mode for tiktoken-derived vocabs
f605a39 Steve Markgraf committed 3 months ago
quant.h: MXFP4 matmul kernel (e2m1 nibbles + ue8m0 per-32 scales), scalar + AVX2
2f38e26 Steve Markgraf committed 3 months ago
feat(cuda): say it once, loudly, when resident tensors fall back to CPU (#687)
f41e2c0 terrizoaguimor committed 3 months ago
docs(corpus-draft): state the scope of the +22% and cross-link #689
68082da JustVugg committed 3 months ago
Merge pull request #690 from terrizoaguimor/corpus-drafts
52dadb7 Vincenzo Fornaro committed 3 months ago
Merge pull request #692 from ZacharyZcR/fix/685-ragged-attn-g64
720e280 Vincenzo Fornaro committed 3 months ago
Merge pull request #691 from terrizoaguimor/release-host-default
3f85dfa Vincenzo Fornaro committed 3 months ago
fix(cuda): release host copies of the VRAM tier on single-GPU large-PIN_GB hosts (#686)
e6456e6 terrizoaguimor committed 3 months ago
feat(spec): frozen-corpus draft source (COLI_DRAFT_CORPUS)
da7e972 terrizoaguimor committed 3 months ago
Merge remote-tracking branch 'origin/main' into fix/deprecate-convert-olmoe
6c46357 Zixin Zhou committed 3 months ago
Merge pull request #681 from MrMetric/fix/hip-non-wmma-archs
0e825da Vincenzo Fornaro committed 3 months ago
fix(coli): accept CUDA_EXPERT_GB=auto in the launcher's resource request
5c97607 Mr. Metric committed 3 months ago
fix(hip): support GPUs without matrix cores (gfx9xx, gfx101x/103x)
0ff5bb8 Mr. Metric committed 3 months ago
Merge pull request #679 from JustVugg/fix/678-mux-spec-stop-midturn
c67dbab Vincenzo Fornaro committed 3 months ago
fix(serve): honor STOP/CANCEL mid-turn in single-slot speculative decode (#678)
65d44ba JustVugg committed 3 months ago
Merge pull request #671 from monotophic/entropy/rans-codec-e1
a3a5a75 Vincenzo Fornaro committed 3 months ago
Merge pull request #674 from ZacharyZcR/docs/experimental-inference-engine
b95fc97 Vincenzo Fornaro committed 3 months ago
Merge pull request #649 from tasoo-oos/fix/coli-allowed-host-wiring
659dbd6 Vincenzo Fornaro committed 3 months ago
Merge pull request #666 from RDouglasSharp/parallel-convert-workers
96fd82b Vincenzo Fornaro committed 3 months ago
Merge pull request #668 from khalilswdp/fix/win-test-and-warnings
c616ea6 Vincenzo Fornaro committed 3 months ago
Merge origin/main into dev: sync website/favicon + #642 docs
cd1b5e0 JustVugg committed 3 months ago