Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
inkling: docs/inkling.md + honest usage-save reporting
Nicholas Beerbower6ebac549633d
diff --git a/c/inkling.c b/c/inkling.cindex 848dab29..5f5fa692 100644--- a/c/inkling.c+++ b/c/inkling.c@@ -794,25 +794,25 @@ static void pins_load(Model *m, const char *snap) {* glm's .coli_usage — copy it aside if you need a stable ranking).* USAGE_SAVE=0 skips the rewrite (e.g. benchmark loops that would skew the* ranking); PIN=off also implies no save (that run never seeded counts). */-static void usage_save(Model *m, const char *snap) {+static int usage_save(Model *m, const char *snap) {Cfg *c = &m->c; int E = c->n_experts;char up[2048], tp[2060];const char *env = getenv("PIN");const char *sv = getenv("USAGE_SAVE");- if (sv && *sv == '0') return;- if (env && (!strcmp(env, "off") || !strcmp(env, "0"))) return;+ if (sv && *sv == '0') return 0;+ if (env && (!strcmp(env, "off") || !strcmp(env, "0"))) return 0;if (env) snprintf(up, sizeof(up), "%s", env);else snprintf(up, sizeof(up), "%s/.coli_usage", snap);snprintf(tp, sizeof(tp), "%s.tmp", up);FILE *f = fopen(tp, "wb");- if (!f) return;+ if (!f) return 0;uint32_t hdr[3] = { 0x31554B49u, (uint32_t)c->n_layers, (uint32_t)E };fwrite(hdr, 4, 3, f);uint32_t *zero = calloc(E, 4);for (int i = 0; i < c->n_layers; i++)fwrite(m->eusage[i] ? m->eusage[i] : zero, 4, E, f);free(zero); fclose(f);- rename(tp, up);+ return rename(tp, up) == 0;}/* ---------- attention (GQA + sliding/global + relative bias + K/V sconv) ---------- */@@ -1198,11 +1198,12 @@ int main(int argc, char **argv) {}fclose(pf);}- usage_save(&m, snap);+ int saved = usage_save(&m, snap);double tot = m.hits + m.miss;- printf("[cache] hit %.1f%% (%llu hit / %llu load) | usage history saved\n",+ printf("[cache] hit %.1f%% (%llu hit / %llu load)%s\n",tot ? 100.0*m.hits/tot : 0.0,- (unsigned long long)m.hits, (unsigned long long)m.miss);+ (unsigned long long)m.hits, (unsigned long long)m.miss,+ saved ? " | usage history saved" : "");return 0;}diff --git a/docs/inkling.md b/docs/inkling.mdnew file mode 100644index 00000000..f86028a6--- /dev/null+++ b/docs/inkling.md@@ -0,0 +1,95 @@+# Inkling (Thinking Machines 975B MoE) on colibri++`c/inkling.c` runs [Thinking Machines Inkling](https://huggingface.co/thinkingmachines/Inkling)+(975B total / 41B active, Apache 2.0) with colibri's expert-streaming approach:+dense weights resident (RAM or VRAM), routed experts streamed from disk with an+LRU + pinned cache. Text-only; vision/audio encoders and the MTP head are not loaded.++## Quickstart++Pre-converted weights (int4 experts + bf16 residents, ~469 GiB):++```sh+hf download nbeerbower/Inkling-colibri-int4 --local-dir ~/Models/inkling_i4+```++or convert the original bf16 checkpoint yourself (shard-resumable; `--watch`+converts while the download is still running):++```sh+python3 c/tools/convert_inkling_int4.py --indir <bf16-checkpoint> --outdir ~/Models/inkling_i4+```++Build and run:++```sh+make -C c inkling # pure CPU (dependency-free, like glm)+make -C c inkling CUDA=1 # + bf16 residents in VRAM (needs ~37 GB free)++SNAP=~/Models/inkling_i4 ./c/inkling -p "The capital of France is" -n 64+```++Requirements: ~120 GB RAM (CPU build keeps ~86 GB of bf16 residents in RAM;+the CUDA build moves them to VRAM and uses the freed RAM for a larger expert+cache), NVMe storage for the snapshot.++## Modes++| Invocation | What it does |+|---|---|+| `-p "text" [-n N]` | streaming greedy generation (stops at eos or N tokens) |+| `-f prompts.txt [-n N]` | one prompt per line (`#` comments skipped), single model load, state reset between prompts — the cache-warming workflow below |+| `[cap] [bits] [ref.json]` | token-exact oracle harness against a `tools/make_tiny_inkling.py` fixture (CI-style validation) |++## Cache warming (same idea as glm's `.coli_usage`)++Expert selections are counted per `(layer, expert)` and written to+`SNAP/.coli_usage` after each generation run. On startup the top `PIN_N`+experts per layer are pinned (non-evictable, loaded in one parallel burst).+Counts **accumulate across runs**, so the ranking converges toward your real+workload — pins trained on a single prompt overfit badly (see the benchmark+table), which is why a diverse warmup matters:++```sh+SNAP=~/Models/inkling_i4 ./c/inkling -f warmup_prompts.txt -n 32+```++| Env | Effect |+|---|---|+| `PIN=off` | disable warming entirely: no seeding, no pins, no history rewrite |+| `PIN=<path>` | alternate history file |+| `PIN_N=<n>` | pins per layer (default `cap/2`; 0 = seed ranking only) |+| `USAGE_SAVE=0` | don't rewrite the history (benchmark runs) |+| `NOGPU=1` / `GPU_DEV=<n>` | disable CUDA / select device |+| `IDOT=0` | byte-exact scalar int kernels (debugging) |+| First positional arg | expert-cache cap per layer (`0` = auto-size from free RAM) |++## Performance (975B, Ryzen 9 7900 / 24t, 187 GB DDR5, RTX A6000, NVMe)++24-token greedy decode, 5-token prompt, commit-tagged runs, single run each.+"Trained" = usage history built from the same prompt; "novel" = never-seen prompt.++| Configuration | Prefill | Decode | Cache hit |+|---|---|---|---|+| Stage A (plain LRU, serial I/O, CPU) | 150.2 s | 0.06 tok/s | ~0% |+| + packed-int4 cache, parallel fills, pins (CPU) | 21.1 s | 0.25 tok/s | 81.5% |+| + CUDA resident tier (A6000) | 18.4 s | 0.32 tok/s | 83.6% |+| + deep pins, trained prompt (`PIN_N=64`) | 1.9 s | 2.51 tok/s | 100.0% |+| deep pins, novel prompt (overfit pins) | 33.8 s | 0.17 tok/s | 79.8% |+| **steady state: 11-prompt diverse history, default pins, novel prompt** ¹ | **35.4 s** | **0.25 tok/s** | **82.2%** |++¹ 48-token generation, 13-token prompt. With only a small warmup corpus the+ranking is barely ahead of plain LRU; hit rate (and therefore decode speed)+grows toward the trained-prompt number as real-use history accumulates.++Phase profile at high hit rates: ~90% CPU expert matmul — the next lever is+expert compute on the GPU, not more I/O work.++## Validation++Every mode is token-exact against HF transformers on a tiny random-init oracle+(`c/tools/make_tiny_inkling.py`): f32, int4-container (VNNI and `IDOT=0`+scalar), bf16 residents on CPU, and bf16 residents through the CUDA kernel.+The tokenizer (o200k family, auto-detected by `tok.h`) encodes 357/357 test+strings identically to HF `tokenizers`. The converter round-trips a fabricated+TML-layout checkpoint back through the engine exactly (`--selftest-e2e`).