Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
Commits on main
diag: say why a turn did not reuse, and capture stderr reliably
c7f7f8c JustVugg committed 3 months ago
fix(inkling): grow the KV instead of restarting it, or reuse never fires
b4c4f61 JustVugg committed 3 months ago
fix(test): detect the CUDA recipe by fact, not by error message
64ee54e JustVugg committed 3 months ago
test(inkling): give the tiny fixture a byte-level tokenizer for serve mode
120e52f JustVugg committed 3 months ago
test(inkling): prove prefix reuse is token-identical, end to end, in CI
c3e251e JustVugg committed 3 months ago
docs(experiments): the engine's own profiler, and the DRAFT re-test
2ade553 Nicolas committed 3 months ago
feat(inkling): reuse the attention state a previous turn already built
639e3c3 JustVugg committed 3 months ago
fix(test): skip the CUDA=1 recipe assertions where CUDA=1 is rejected
73da98f JustVugg committed 3 months ago
fix(783): doctor inspects the model's own engine; CUDA=1 stops decorating kimi_k3
a1f9e8e JustVugg committed 3 months ago
cuda: refuse non-per-row formats in the expert-group fallbacks
d9ef4ca kreuzzelg committed 3 months ago
cuda: dispatch grouped int4 (fmt=4) in the async expert-group path
59b303a kreuzzelg committed 3 months ago
some edits in mission, introducing inference multitiering concept
374577d marco guardigli committed 3 months ago
st: document st_read_raw's caller contract, and add the capped sibling
42ac78d Mario Gutierrez committed 3 months ago
st: document st_read_raw's caller contract, and add the capped sibling
de2b317 Mario Gutierrez committed 3 months ago
docs(experiments): fold the retractions in, add the controlled measurements
0436bd2 Nicolas committed 3 months ago
metal: GPU attention for S>4 prefill (single command buffer, opt-in)
1709feb Doug Sharp committed 4 months ago
Merge pull request #780 from ThefloorMiner/feat/usage-decay
72ddb67 Vincenzo Fornaro committed 3 months ago
Merge pull request #771 from ThefloorMiner/docs/prefill-chunk-and-cuda-dense
a1725b3 Vincenzo Fornaro committed 3 months ago
Merge pull request #770 from ThefloorMiner/fix/plan-warns-when-vram-is-already-in-use
b50621e Vincenzo Fornaro committed 3 months ago
Merge pull request #761 from terrizoaguimor/docs/serve-carries-the-dashboard
fddb30a Vincenzo Fornaro committed 3 months ago
Merge pull request #779 from JustVugg/feat/fmt8-ue8m0
fd5d72e Vincenzo Fornaro committed 3 months ago
Merge pull request #778 from JustVugg/feat/st-fp8-dtypes
57bf04e Vincenzo Fornaro committed 3 months ago
feat(usage): COLI_USAGE_DECAY — give the learned expert history a half-life
4134347 Nicolas committed 3 months ago
st.h: index F8_E4M3, F8_E8M0 and I64, and refuse them in the float readers
2171adf JustVugg committed 3 months ago
[email protected] added a small clarification for windows invocation from powershell
ec52bc3 marco guardigli committed 3 months ago
docs(env): document COLI_PREFILL_CHUNK, and what CUDA_DENSE is worth
a25539e ThefloorMiner committed 3 months ago