Run frontier MoE models on hardware you already own โ pure C, zero deps, experts streamed from disk. Tiny engine, immense model. ๐ฆ
C
docs: clarify multi-disk streaming gains in README
Stamina9a9649c099736
diff --git a/README.md b/README.mdindex d8460f65..9de4d876 100644--- a/README.md+++ b/README.md@@ -1,726 +1,735 @@-<p align="center">- <img src="assets/colibri-logo.svg" width="560" alt="colibrรฌ โ tiny engine, immense model">-</p>--<p align="center">- <a href="https://justvugg.github.io/colibri"><img src="https://img.shields.io/badge/website-justvugg.github.io%2Fcolibri-1f6feb" alt="Website"></a>- <a href="https://github.com/JustVugg/colibri/releases"><img src="https://img.shields.io/github/v/release/JustVugg/colibri?color=2ea043" alt="Latest release"></a>-</p>--<p align="center">- <a href="https://justvugg.github.io/colibri"><b>Website</b></a> ยท- <a href="https://discord.gg/RXV83nSZdk"><b>Discord</b></a> ยท- English ยท <a href="README.zh-CN.md">็ฎไฝไธญๆ</a> ยท <a href="README.zh-TW.md">็น้ซไธญๆ</a> ยท <a href="README.it.md">Italiano</a>-</p>--**Tiny engine, immense model.** Run **frontier MoE models โ 744B to 2.8T-parameters** โ on consumer and heterogeneous hardware, in pure C with zero-engine dependencies, by treating storage, RAM, and VRAM as a single inference-hierarchy (AI memory multitiering).--Nine families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, with-vision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, with vision),-**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) and-**OLMoE** (7B) โ-one C file each, the same `coli chat` / `coli serve` / `coli web` front end.-[Full roster โ](#other-supported-models)--> **Colibrรฌ is an inference engine you can run today, and an open research-> platform.** Its primary goal is to pursue inference-side performance across-> the entire software/hardware boundary โ model formats, memory hierarchy,-> storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU-> overlap โ so large models depend less on scarce hardware and cost less to run.--Colibrรฌ treats VRAM, RAM, and storage as a single multitier hierarchy, and it is-deliberately a place to test aggressive systems ideas โ so there is **no SLA on-speed, and a hard guarantee on semantics**: experiments must earn their place-through reproducible end-to-end measurements, and the default policy **never-silently changes model precision or router semantics**. Insufficient fast memory-may reduce speed; it must not quietly redefine the model.--```-$ ./coli chat- ๐ฆ colibri v1.11.0 โ GLM-5.2 ยท 744B MoE ยท int4 ยท streaming CPU- โ ready in 32s ยท resident 9.9 GB- โบ ciao!- โ Ciao! ๐ Come posso aiutarti oggi?-```--## See it running--<p align="center">- <img src="docs/media/colibri-dashboard.png" width="900" alt="colibrรฌ web dashboard โ live metrics, hardware panel, expert tiers">-</p>-<p align="center"><em>The web dashboard (<code>./coli web</code>): a 744B model at <strong>4 tok/s, TTFT 1.6 s, disk 0</strong> โ-full expert residency on 6ร RTX 5090, with live token metrics, the per-turn time breakdown,-the VRAM/RAM/disk tier bar and the live mini-brain in the corner.</em></p>--<p align="center">- <img src="docs/media/colibri-brain.png" width="900" alt="the Brain page โ 19,456 experts as a live cortex">-</p>-<p align="center"><em>The <strong>Brain</strong> page: all 19,456 experts as a living cortex โ colour is the storage tier,-brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's-<a href="https://github.com/JustVugg/colibri/issues/175">measured topic affinity</a>.</em></p>--<p align="center">- <img src="docs/media/colibri-atlas.png" width="900" alt="the Atlas page โ the measured expert atlas as a 3-D galaxy">-</p>-<p align="center"><em>The <strong>Atlas</strong> page: the <a href="https://github.com/JustVugg/colibri/issues/175">measured expert atlas</a>-as a 3-D galaxy โ 13,260 characterised experts, 1,041 replicated specialists clustering by topic-(poetry, law, Chinese, SQLโฆ). Position is measured routing affinity, not a learned embedding. Drag to spin.</em></p>--## The research mission--With Colibrรฌ, private frontier model access is not limited by availability of hyperscaler-class hardware.--With its multitiering features Colibrรฌ **removes proprietary hardware dependencies aggressively-optimizing functional inference engine pipelines**.--Our operational mission includes changing how weights are represented and moved, deciding what-lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing-launch and synchronization overhead, exploiting sparsity and reuse, and testing-new decoding algorithms. Nothing is protected merely because it is conventional;-nothing is adopted merely because a microbenchmark looks fast. The deciding-result is end-to-end inference on real machines, with correctness and quality-measured alongside throughput, latency, memory, and cost.--The practical consequence is **accessibility**: run a 744B-parameter model on-hardware you already own, watch every expert fire in real time, and change the-code that does it. Not renting intelligence behind an API โ *holding* it:-probing it, measuring it, improving it. The engine is deliberately small enough-that the next useful optimization can come from anyone willing to measure it.--## Core techniques and measured findings--- **One hierarchy, not limited by tier capacity.** VRAM, RAM, and NVMe are placement- tiers for the same weights; limited fast memory changes speed, not model semantics.-- **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learned- pinned hot-store, and one-layer-ahead prefetch instead of loading every expert.- It wins on repeatable workloads; history can overfit, and lookahead can lose on- some hosts, so both remain measurable policies rather than promises.-- **I/O is part of the engine.** Batched expert unions, overlapped reads and- compute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path- rather than pretending storage latency is free. `O_DIRECT` is drive-dependent,- and dual-SSD still needs broader end-to-end community A/Bs.-- **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full- expert residency share one runtime and can be combined according to the machine;- the profitable combination depends on compute, bandwidth, residency, and workload.-- **Compressed state without a different model.** Token-exact forward validation,- 57ร smaller MLA KV state, persistent warm conversations, and faithful DSA keep- optimization tied to correctness. These are memory, latency, and correctness- properties โ not a blanket throughput claim.-- **Speculation that must earn its keep.** Native MTP and grammar-forced drafts- are measured end to end and can be disabled when acceptance does not repay verification.--## Open hypotheses, experiments, and how to help--Colibrรฌ treats an optimization as a hypothesis until a controlled end-to-end A/B-shows otherwise. These are the main questions now:--| hypothesis | evidence so far | experiment still needed |-|---|---|---|-| Routing history can place experts better than plain LRU | learned pins improve repeated workloads, but can overfit a prompt | held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads |-| Multiple SSDs can turn independent bandwidth into decode speed | weighted mirror/split routing is implemented and validated; the bandwidth model is sound | cold-cache one-drive vs two-drive GLM-5.2 runs on real, independent controllers |-| A hardware-aware planner can approach each machine's best configuration automatically | RAM/VRAM budgets and several backends are detected today | compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems |-| Lossless or quality-bounded representations can reduce weight movement enough to matter | format and quantization ablations exist, with correctness/quality gates | reproduce quality, bytes moved, latency, and cost per useful token together โ not compression ratio alone |-| Routing-aware speculation can pay before near-full residency | MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit | map the break-even surface across acceptance, expert hit rate, batch union, and draft depth |-| CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck | CUDA and Metal wins exist, but fast CPUs and low residency can erase them | per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines |--Want to help? Pick one row and publish the negative results too. Record the-hardware, commit, model/container, exact command, prompt, cache state, throughput,-TTFT, expert hit rate, bytes read, and quality check; change one variable, repeat-the run, and attach raw logs. Start with-[CONTRIBUTING.md](CONTRIBUTING.md), compare against-[the benchmark protocol](docs/benchmarking.md), then-[open an experiment issue](https://github.com/JustVugg/colibri/issues/new).-A well-controlled failure is more valuable here than an unexplained fast number.--## The idea--A 744B Mixture-of-Experts model activates only ~40B parameters per token โ and-only ~11 GB of those change from token to token (the routed experts):--<p align="center">- <img src="docs/media/sparse.png" width="880" alt="only ~5.4% of parameters are active per token">-</p>--So the model doesn't need to *fit* in fast memory โ it needs to be **placed**:--- the **dense part** (attention, shared experts, embeddings โ ~17B params) stays- **resident in RAM at int4** (~9.9 GB);-- the **19,456 routed experts** (75 MoE layers ร 256 + the MTP head, ~19 MB each- at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a- per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.--Think of the core algorithm as **a JIT, but for weights**. A compiler JIT never-compiles the whole program โ it watches what actually runs and compiles the hot-paths, just in time. colibrรฌ makes the same bet about a 744B parameter space:-parameters are not resident state to be held, they are **data to be staged**-across a heterogeneous storage hierarchy (VRAM / RAM / NVMe), exactly when the-router proves they are needed. Measured routing heat decides which experts earn-which tier, the router runs a layer ahead so prefetch hides the staging latency,-and โ like a JIT โ the engine learns your workload: the more you run, the hotter-the right experts get. It works because routing has measurable structure (see-the [expert atlas](https://github.com/JustVugg/colibri/issues/175)) โ and-structure is cacheable.--The engine is a single C file (`c/colibri.c`) plus small headers. No BLAS, no-Python at runtime, no GPU required.--### Local cluster mode--The coordinator keeps token generation, routing, and KV state local while-disk-backed expert workers execute routed FFNs on other Macs. A layer's routed-batch-union is sent as one persistent TCP request, so a token does not incur one-round trip per expert.--Start the optional registration service:--```bash-./coli cluster coordinator --host 0.0.0.0 --port 8765-```--On each worker, with the same converted model available locally:--```bash-./coli cluster worker --model /nvme/glm52_i4 --port 9100 \- --coordinator http://COORDINATOR:8765 --advertise-host WORKER_IP-```--Run the coordinator with discovery, or provide `--cluster-workers-HOST:PORT,...` for a static setup:--```bash-./coli serve --model /nvme/glm52_i4 \- --cluster-coordinator http://127.0.0.1:8765-```--The transport is disabled unless workers are configured, so the existing-single-machine path remains unchanged. Dense-layer sharding and browser/WebGPU-workers are separate follow-up seams.--## How it works--### The per-token path--<p align="center">- <img src="docs/media/token-path.png" width="880" alt="route โ union โ place โ overlap โ learn">-</p>--Every layer of every token walks the same five steps. The design goal is that-**placement only ever decides speed** โ the router's decisions and the weights'-precision are the same whether an expert answered from VRAM or from disk.--### One memory hierarchy instead of one memory requirement--<p align="center">- <img src="docs/media/tiers.png" width="880" alt="VRAM / RAM / NVMe three-tier expert residency">-</p>--### Dual-SSD: two copies of the model, twice the read bandwidth--Decode is disk-bound on most machines, and expert reads are read-only โ so if you have a **second SSD**, put a full copy of the model on it and let the engine stream from both drives at once:--```bash-COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat-COLI_DISK_WEIGHTS=9,3 ... # optional: primary,mirror bandwidth ratio (else measured at startup)-```--Each expert is routed to one drive by a deterministic hash, weighted by the two drives' measured (or declared) bandwidth, so readahead/PILOT prefetch and the demand read always hit the same drive and nothing is cached twice. The aggregate bandwidth is the sum of both drives โ a 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone, and the OMP-parallel pin/warmup load streams from both. Details worth knowing:--- the mirror is **validated at startup** (per-file size + safetensors header must be byte-identical to the primary); divergent or missing files silently stay on the primary, so a **partial mirror is fine** โ a smaller second SSD holding only some shards still helps;-- the mirror is **never written**: `.coli_usage`, `.coli_kv` and all sidecars stay on the primary;-- a read error on the mirror falls back to the primary (one warning, no crash), so unplugging the second drive mid-run degrades instead of killing the server;-- routing never changes tokens โ both copies are byte-identical, and the per-run `MIRROR:` stats line shows GB served per drive.--The same engine spans the whole range: on a 25 GB laptop everything streams from-disk (slow but correct); on a large host the entire expert set becomes resident-(`CUDA_EXPERT_GB=auto PIN_GB=all`) and disk drops out of the decode path-entirely. Between the tiers sits a **learning cache**: the engine records which-experts *your* workload routes to (`.coli_usage`, updated every turn) and pins-the hottest ones automatically โ colibrรฌ literally gets faster the more you use-it. On multi-socket hosts, `COLI_NUMA=1` interleaves the resident weights across-memory controllers ([#82](https://github.com/JustVugg/colibri/issues/82)).--For a second drive that cannot hold the whole model, Colibri can rank a partial-mirror from the expert history it already learns. Run a few representative-prompts first so `.coli_usage` reflects the workload, then plan, stage, and-verify the mirror:--```bash-./c/coli mirror plan --model /fast/glm52_i4 --mirror /second/glm52_i4 \- --budget-gib 200 --reserve-gib 20-./c/coli mirror stage --model /fast/glm52_i4 --mirror /second/glm52_i4 \- --budget-gib 200 --reserve-gib 20-./c/coli mirror verify --model /fast/glm52_i4 --mirror /second/glm52_i4-```--The planner reads safetensors headers directly, follows split-model directories-from `COLI_MODEL_DIRS`, and prioritizes shards that can serve the hottest routed-experts. Staging never changes the primary model: it copies through temporary-files, preserves the requested free-space reserve, verifies every shard with-SHA-256, never deletes an existing mirror shard, and atomically publishes a-receipt only after the selected mirror is ready.--### Never wait for the disk twice--Misses are expensive, so the engine spends most of its cleverness avoiding and-overlapping them: each expert's three matrices are stored adjacent and read in-one `pread`; a bounded async I/O pool (`PIPE=1`, default) loads missing experts-while resident ones compute; batched positions read each unique expert once-(**batch-union**); and a router-lookahead thread (`PILOT=1`) prefetches the next-layer's experts โ routing is measurably **71.6% predictable one layer ahead**.-On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream-on-device across layers so the CPU expert loop runs uninterrupted; on Apple-Silicon an experimental [Metal backend](docs/metal.md) does the batched expert-math on the unified-memory GPU; and a [Vulkan backend](docs/vulkan.md) brings-the expert tier, dense projections, and the MLA attention core to any GPU with-a Vulkan 1.2 driver โ including AMD cards via Mesa/RADV (the only backend for-cards the vendor stacks no longer support, like the RX 580, and competitive-with ROCm on RDNA4 โ see [the benchmarking notes](docs/vulkan.md)).--> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is-> often a large win on drives with DRAM cache and bandwidth headroom (+34%-> decode measured with `PIPE=1` on a Blackwell/Windows box; 4.25โ9.69 GB/s in-> iobench on a GB10) โ but it is drive-dependent: QLC/DRAM-less or virtualised-> disks can be neutral to negative. Try it first; keep what your hardware-> rewards.--### Faithful model, compressed state--The forward pass is validated against a `transformers` oracle (teacher-forcing-typically 30-32/32; two tiny-oracle positions are floating-point near-ties and-toolchain-dependent). MLA attention stores a compressed KV state โ 576-floats/token instead of 32,768 (**57ร smaller**) โ and persists it across-restarts (`.coli_kv`): conversations reopen warm with zero re-prefill,-byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's-lightning indexer) is implemented faithfully and validated by forcing full-key-selection to reproduce dense attention exactly.--### Speculative decoding, honestly--GLM-5.2's native MTP head drafts tokens that the main model verifies in one-batched forward โ 2.2โ2.8 tokens/forward when it pays. Two hard-won rules ship-as defaults: the MTP head must be **int8** (int4 heads collapse to 0โ4%-acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)), and draft and-verify must compute **the same function** โ `SPEC_PIN=1` pins both to one-kernel family ([#163](https://github.com/JustVugg/colibri/issues/163) is the-full forensic story). Grammar-forced drafts-([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) add ~free acceptance on-constrained JSON output. Whether speculation is a net win depends on your-cache temperature โ measure, and use `DRAFT=0` when it doesn't pay.--## What it achieves--<p align="center">- <img src="docs/media/ladder.png" width="880" alt="measured decode speed by hardware class">-</p>--Same engine, same int4 container โ the hardware only changes where the experts-live. Highlights from the [full benchmark tables](docs/benchmarks.md):--- **6ร RTX 5090, full residency:** 5.8โ6.8 tok/s decode, TTFT ~13 s- ([experiment log](docs/experiments/glm52-6x5090-2026-07-12.md));-- **128 GB CPU-only desktop:** ~1.8 tok/s warm ([#200](https://github.com/JustVugg/colibri/issues/200));-- **single RTX 5070 Ti laptop-class box:** 1.07 tok/s via the GPU-resident- pipeline ([#273](https://github.com/JustVugg/colibri/issues/273));-- **25 GB dev box:** 0.05โ0.1 tok/s cold โ the proven floor where this project- started, and still the honest baseline.--Quality is measured, not assumed: the int4 container's quantization cost and the-scale-granularity/rotation ablations live in-[docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) and-[#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81).--## Get started--You need two things: **the program** (a few hundred KB) and **the model**-(372 GB). Step-by-step for every platform in the-[Quick Start guide](docs/quickstart.md).--### 1. Get colibri--**Download a prebuilt release** โ Linux, macOS and Windows, no compiler needed.-Take the archive for your platform from-[Releases](https://github.com/JustVugg/colibri/releases) and unpack it:--```bash-mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri-python3 coli info # engine ready โ-```--Inside you get the engine (`colibri`, `colibri.exe` on Windows), the `coli`-launcher and its Python helpers. Nothing to rename or configure โ `coli` finds-the engine next to itself. You only need-[Python 3](https://www.python.org/downloads/) installed: the launcher and the-API gateway are Python scripts, while the engine itself is pure C with zero-dependencies.--**Or build from source** โ needs `gcc` (or clang) with OpenMP:--```bash-git clone https://github.com/JustVugg/colibri && cd colibri/c-./setup.sh # checks gcc/OpenMP, builds, self-tests-```--Want `coli` on your PATH? From a checkout, `pip install -e .` registers it (the-engine still lives in `c/` โ an editable install from the clone, not a wheel).--### 2. Get the model--A pre-converted **GLM-5.2 int4** container is on Hugging Face โ use the-**group-scaled (gs64)** build with the **int8 MTP head**. It is about **372 GB**,-so put it on a disk with the room, ideally a fast one:--**https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp**--**GLM-5.3** is the same family and loads with the same engine. It has its own-container, also group-scaled (gs64), about **419 GB**. It ships **without** the-MTP head, so speculative decoding stays off:--**https://huggingface.co/Justvugg/GLM-5.3-colibri-int4-g64**--> โ ๏ธ Use the **gs64** container above, not the older per-row int4 mirrors-> (`mateogrgic/โฆ`, `jlnsrk/โฆ`): those measure ~9pp worse on quality and are the-> root cause of the original think-mode loops and never-terminating generations-> in [#455](https://github.com/JustVugg/colibri/issues/455). The gs64 container-> fixed those controlled per-row A/Bs, but it is not a general repetition or-> EOS-starvation guard. The MTP head must also be **int8, not int4**-> (int4 โ 0% draft acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)):-> `ls -l <model>/out-mtp-*` โ int8 (correct) is `3527131672 / 5366238584 / 1065950496`-> as three files, or a single `out-mtp-00000.safetensors` of `9959321520` bytes-> (the current upload of the recommended container ships it as one file: same-> int8 tensors, 777 of them at one byte per element).--Or convert from the FP8 source yourself โ one resumable command that never needs-the full 756 GB on disk at once:--```bash-./coli convert --model /nvme/glm52_i4 # download+convert shard by shard (python, one-time)-```--#### Other supported models--GLM-5.2 is the reference model, but the same streaming approach runs six more-families. Each is a **sibling engine** โ one C file, its own architecture, the same-`coli chat` / `coli serve` / `coli web` front end (the launcher picks the binary from-the model's `config.json`):--> **What each one needs.** These differ a lot, and reading two of them together-> has confused people into thinking the requirements contradict each other-> ([#191](https://github.com/JustVugg/colibri/issues/191)). They do not โ they-> are different models. **None of them needs a GPU.**->-> | Model | Disk for the weights | RAM | GPU |-> |---|---|---|---|-> | **OLMoE** | ~7 GB (int8 container) | 8 GB | not needed |-> | **GLM-5.2/5.3** | ~372 GB (5.2) / ~419 GB (5.3) | 16 GB min, 24 GB comfortable | not needed |-> | **GLM-5.3-Flash** | ~195 GB converted | 25 GB (12 GB weights at int4 + expert cache) | not needed |-> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |-> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |-> | **DeepSeek V4 Flash** | ~167 GB (REAP 150B: ~85 GB) | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |-> | **Qwen3.8-Flash-Next** | ~185.5 GB (official FP8 checkpoint) | 16 GB min, 24 GB comfortable at the default context | not supported; CPU only |-> | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU |->-> A GPU only ever makes it faster. Speed is set by your disk, because the experts-> are streamed from it โ expect a fraction of a token per second on a slow drive-> and a few per second on a fast one with the cache warm.--| Family | Total / active | Weights | Build | Docs |-|---|---|---|---|---|-| **GLM-5.2/5.3** | 744B / 40B | [`mastouri/โฆ-int4-g64-with-int8-mtp`](https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp) (372 GB) or [`Justvugg/GLM-5.3-colibri-int4-g64`](https://huggingface.co/Justvugg/GLM-5.3-colibri-int4-g64) (419 GB) | `make -C c glm` | this page |-| **Inkling** (Thinking Machines) | 975B / 41B | [`nbeerbower/Inkling-colibri-int4`](https://huggingface.co/nbeerbower/Inkling-colibri-int4) (469 GB) | `make -C c inkling` | [inkling.md](docs/inkling.md) |-| **GLM-5.3-Flash** (Z.ai) | 321B / 40B | [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) โ converted to **int4-gs64** routed experts, dense stays BF16 and the precision is a load-time choice; vision included | `make -C c glm53` | [glm53-flash.md](docs/glm53-flash.md) |-| **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) โ original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) |-| **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint โ routed experts stay **native fp4**, dense stays fp8-e4m3; the **REAP-pruned 150B** ([`puwaer/DeepSeek-V4-Flash-0731-reap-150b`](https://huggingface.co/puwaer/DeepSeek-V4-Flash-0731-reap-150b), 85 GB, 132 of 256 experts) loads with the same engine and no conversion | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |-| **DeepSeek V4.1 Flash** | 552B / 16B | official checkpoint, **no conversion**: experts are already fp4, dense is fp8-e4m3. 203 GB of it is an n-gram memory read from disk a few hundred bytes at a time, and the routed experts cost **4.5 GB per token** against GLM-5.2's 12.7. Vision, tool calling and the DSpark draft head are all on | `make -C c deepseek_v41` | [deepseek-v41.md](docs/deepseek-v41.md) |-| **Qwen3.8-Flash-Next** (Alibaba) | 125B + 51B n-gram / 6B | [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) โ original checkpoint; PLE stays pageable and experts stay **native block-FP8** | `make -C c qwen38` (CPU only) | [qwen38.md](docs/qwen38.md) |-| **Qwen3.6** (Alibaba) | 35B / 3B | [`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64) (~20 GB, **recommended**) โ hybrid Gated Attention + Gated DeltaNet | `make -C c qwen36` (`CUDA=1` for the VRAM expert tier) | [qwen36.md](docs/qwen36.md) |-| **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` โ **int8** container, ~7 GB | `make -C c olmoe` | โ |--Qwen3.6 ships three pre-converted containers: **int4-gs64** (recommended โ measured-cosine to the int8 anchor 0.98777 โ 0.99313 and KL 0.109 โ 0.080 against per-row, i.e.-~44% less quantization error), [int4 per-row](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4)-as the A/B baseline, and [KAT-Coder v2.5](https://huggingface.co/Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64),-which the same engine runs unchanged โ any architecture-identical checkpoint works-without a code path of its own. With `CUDA=1` the VRAM expert tier measured-**1.44 โ 10.05 tok/s (7.0ร) on two 8 GB cards**, output bit-identical to the CPU path.--Kimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight from-the original Hugging Face shards, and the bf16 dense set is quantized at load time.-Long agent sessions can opt into recurrent-state checkpoints (`COLI_K3_CKPT=N`-slots in RAM, or parked on disk with `COLI_K3_CKPT_DIR`): an edited or follow-up-prompt restores the deepest surviving checkpoint and re-prefills only the tail,-instead of replaying the whole conversation through the SSM layers. On Vulkan-hosts `K3_VK_UP=auto` sizes the expert tier upload from measured bandwidth. The-engine's KDA and MLA paths are validated token-exact in CI against the vendor-implementation.--Inkling ships int4 experts but **bf16 dense weights** (49.4 GB resident); on a host-that cannot hold those, [inkling.md](docs/inkling.md) has a one-pass tool that brings-the dense set to 15.3 GB and lets the 975B run on a 25 GB box โ with the honest-trade-off written down.--### 3. Run it--```bash-COLI_MODEL=/nvme/glm52_i4 ./coli chat # RAM budget, cache and MTP auto-detected-COLI_MODEL=/nvme/glm52_i4 ./coli plan # inspect the planned VRAM/RAM/disk placement-COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only readiness check-COLI_MODEL=/nvme/glm52_i4 ./coli doctor --deep # strict tensors/shards/index/mirror preflight-COLI_MODEL=/nvme/glm52_i4 ./coli tune # measure and save this machine's fastest safe execution profile-./coli web --model /nvme/glm52_i4 # API + dashboard, and opens a browser-./coli serve --model /nvme/glm52_i4 # API + dashboard, no browser (headless)-```--On Windows a release archive ships `coli.cmd`: double-click it for the quick-start, or run `coli.cmd chat --model D:\glm52_i4` from cmd or PowerShell.-From a source checkout the same commands work with `python coli chat --model-D:\glm52_i4`. The `.exe` files are the engines, not the launcher: started on-their own they have no model to load and exit immediately.-The engine at runtime is pure C โ python is only used by the one-time converter-and the optional API gateway.--#### The same commands run any of the models--`coli` reads the model's `config.json`, picks the matching engine binary, and-renders that family's chat template โ so **nothing about the command line-changes between models**. Build the engine you want once, then just point-`COLI_MODEL` at the right directory:--```bash-make -C c glm # GLM-5.2-make -C c inkling # Inkling-make -C c kimi_k3 # Kimi K3--COLI_MODEL=/nvme/glm52_i4 ./coli chat # TUI-COLI_MODEL=/nvme/inkling_i4 ./coli chat-COLI_MODEL=/nvme/kimi_k3 ./coli chat--./coli web --model /nvme/inkling_i4 # API + dashboard, opens a browser-./coli web --model /nvme/kimi_k3-./coli serve --model /nvme/inkling_i4 # API + dashboard, no browser-```--For the non-GLM engines `coli chat` starts the gateway locally and attaches the-TUI to it, so the TUI, the API and the dashboard all go through the same-arch-aware chat template โ you never have to pass the template yourself.--Two things that differ per model, both documented in the per-model page:--- **Inkling on a RAM-tight host** needs the int4 dense container and a small- expert cache: `./coli chat --model /nvme/inkling_i4 --cap 2`- (see [inkling.md](docs/inkling.md) โ the default `--cap 8` wants ~14 GB of- cache on top of the resident set).-- **Kimi K3** streams its MXFP4 experts from the original checkpoint, so there- is nothing to convert โ but the snapshot is ~1.6 TB- (see [kimi_k3.md](docs/kimi_k3.md)).--### 4. Go deeper--| topic | doc |-|---|---|-| Benchmarks, community datapoints, quality measurements | [docs/benchmarks.md](docs/benchmarks.md) |-| Reproducible benchmark protocol and minimum report | [docs/benchmarking.md](docs/benchmarking.md) |-| Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) |-| Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) |-| CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) |-| Vulkan backend (any GPU: AMD via RADV, incl. cards ROCm dropped) | [docs/vulkan.md](docs/vulkan.md) |-| Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) |-| OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) |-| Experimental layer-segment embedding ABI | [docs/segment-runtime.md](docs/segment-runtime.md) |-| Experimental tokenizer/embedding/head Edge ABI | [docs/edge-runtime.md](docs/edge-runtime.md) |-| Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) |-| Environment variable inventory | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) |--## DeepSeek V4--**DeepSeek V4 Flash** streams the official checkpoint with no conversion: routed-experts stay **native fp4**, the dense set stays **fp8-e4m3** with UE8M0 block-scales. MLA + DSA sparse attention, 43 layers, 256 routed experts plus one-shared, top-6. Supported on x86-64/aarch64 Linux and Windows/MSYS2 (CPU), with-an optional CUDA tier (Windows runtime DLL; Linux `CUDA=1` direct link,-verified under WSL2) that keeps every stage CPU-canonical and falls back per stage.--```bash-cd c-make deepseek-v4-python ./coli chat --model /path/to/DeepSeek-V4-Flash --ram 32-# also: coli run / coli serve / coli web-# Windows CUDA tier: make cuda-dsv4-dll CUDA_ARCH=portable (+ make cuda-dsv4-dg-dll on RTX 50)-```--Two opt-in GPU levers are new and looking for community numbers, both default-off and byte-identical when unset: `DSV4_HYBRID=1` splits VRAM-tier misses-between the GPU fill branch and the CPU branch using bandwidths measured at-runtime, and `COLI_CUDA_MOE_DOUBLE=1` (on top of `COLI_CUDA_MOE_BATCH=1`)-prefetches the next layer's full expert set into a second VRAM bank while the-current layer computes, falling back to the single bank when VRAM is short.-The CUDA tier also runs on Pascal and Turing cards now (GTX 10 / RTX 20-series): build with `CUDA_ARCH=portable-pre-ampere NO_TC=1`.--Greedy decode and one KV slot. Tool calling is wired through the HTTP gateway-with V4's native prompt and DSML call blocks; grammar is not supported. See the-[per-engine API matrix](docs/api.md#tool-calling-support). Prefix checkpoints-(in memory + on disk) make agent sessions and follow-up turns start in seconds-after the first prefill of a system prompt. Measured on an RTX 5080 + 2 NVMe:-3324-token prefill 90 s, 8.3k-token first turn ~4 min once, later-sessions/turns 6-9 s, decode ~1.6 tok/s at 3k context โ see-[docs/deepseek-v4.md](docs/deepseek-v4.md).--**Give it RAM.** 43 ร 256 routed experts are ~137 GiB on disk and a token-touches 301 of them, so the expert cache hit rate is what sets tok/s โ `--ram`-is the single most valuable knob, and it changes speed only, never output.--**Speculative drafting exists and is off.** DSpark's markov drafter and full MTP-are both implemented and verified: a draft can save forward passes but never-change a token, because every accepted token is still the target's own argmax.-Measured on real multi-turn chat, they accepted 1 in 15 and 10 in 24, and the-rejected-suffix replay of this engine's recurrent attention state cost more than-the drafts saved โ one 14-token answer took 495 seconds. So `V4_DRAFT` and-`V4_MTP` default to `0` and the code stays, with the numbers beside it, for-whoever retries this on faster storage.--See [docs/deepseek-v4.md](docs/deepseek-v4.md) for the CUDA tier (build, DLL-selection, GPU coverage), the environment reference, performance numbers,-checkpoint validation, and the generated tiny independent oracle.--## What's next--- **Inference-systems research is the product.** The current hierarchy is LRU +- a learned pin set; active work spans model formats, compression, placement,- scheduling, I/O, CPU/GPU kernels, heterogeneous overlap, KV state, and- routing-aware speculation. The objective is lower hardware requirements and- lower cost per useful token. Everything lands the way this project works:- measured end to end, reviewed, and developed in the open.-- **More open models.** The tiering algorithm is model-agnostic: any MoE with- routed experts can be staged the same way. Nine families run today (GLM-5.2,- GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next,- Qwen3.6, OLMoE); further open-weight families โ **MiniMax** among the- candidates โ earn an engine the way the first eight did: when someone- measures one end to end.--## Supporting the project--colibrรฌ started as a one-person project on a 12-core laptop with 25 GB of RAM;-today its numbers come from a community of real machines. If it's useful to you:--- โญ star the repo and share it;-- ๐ open issues with benchmark numbers from your hardware โ datapoints move- this project more than anything else;-- ๐ฌ join the [Discord community](https://discord.gg/RXV83nSZdk) to discuss- experiments, hardware results, and research directions;-- ๐ฌ reach out via GitHub issues to sponsor development or donate hardware.--## Repo layout--```-Makefile root build/check entry point-c/-โโโ colibri.c GLM-5.2 engine (make glm)-โโโ inkling.c Inkling engine (make inkling)-โโโ kimi_k3.c Kimi K3 engine (make kimi_k3)-โโโ deepseek_v4.c DeepSeek V4 Flash engine (make deepseek-v4)-โโโ qwen38.c Qwen3.8-Flash-Next text engine (make qwen38)-โโโ qwen36.c Qwen3.6 engine (make qwen36)-โโโ olmoe.c OLMoE engine (make olmoe)-โ-โโโ st.h safetensors index and range reads-โโโ quant.h canonical container decoders-โโโ expert_ffn.h routed-expert FFN kernel shared by the MoE engines (planar int4, layer runner)-โโโ tok.h, json.h tokenizer and JSON parser-โโโ compat.h Windows/macOS shims (POSIX names, one place)-โโโ expert_store.h streaming expert cache-โโโ route_trace.h routing telemetry and .coli_usage, engine-agnostic-โโโ kv_prefix.h KV prefix reuse across turns-โ-โโโ backend_cuda.* optional CUDA tier (CUDA=1)-โโโ backend_metal.* optional Metal tier (METAL=1)-โโโ backend_vulkan.* optional Vulkan tier (VULKAN=1)-โ-โโโ Makefile build and local checks-โโโ coli user-facing CLI-โโโ openai_server.py OpenAI-compatible HTTP gateway-โโโ resource_plan.py RAM/VRAM planner behind `coli plan` and `coli doctor`-โโโ tools/ offline conversion, fixtures and benchmarks-โโโ scripts/ long-running conversion helpers-โโโ tests/ dependency-free C and Python tests-web/ browser UI (pure OpenAI-API client)-desktop/ Tauri v2 desktop shell wrapping the web UI-docker/ container images-docs/ reference docs, experiments, media-```--**One `.c` per model family, over shared single headers.** An engine owns its-architecture and nothing else; anything two engines both need โ the safetensors-reader, the container decoders, the tokenizer, the expert cache โ lives in a-header they both include, so a fix reaches all of them at once. That rule is not-decorative: the defects that keep recurring here are the ones where a mechanism-landed in one engine and never reached its siblings.--From the repository root, `make`, `make check` and `make clean` delegate to the-engine Makefile.--## Why "colibrรฌ"--The hummingbird weighs a few grams, hovers in place, and visits a thousand-flowers a day. This engine keeps a 744-billion-parameter giant alive on-hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.--## Acknowledgements--colibrรฌ is an engine; the minds it runs are a gift. Thank you to the teams-releasing frontier-class weights in the open โ **Z.ai** (GLM), **Moonshot AI**-(Kimi), **Alibaba Qwen**, **MiniMax**, and **Allen AI** (OLMoE) โ and to every-contributor who benchmarked, bisected, replicated an atlas run, or sent a patch.-This project is proof of what open weights make possible.--The project's expert placement, compression, and routing experiments also build-on ideas and evidence from the following open research and systems work:--- [REAP](https://github.com/CerebrasResearch/reap) and- [EASY-EP](https://github.com/RUCAIBox/EASYEP) for output-aware and- domain-specific expert importance.-- [SERE](https://github.com/JL-Cheng/SERE) for similarity-based expert- re-routing, and [ReMoE](https://github.com/BUAA-OSCAR/ReMoE) for- cache-locality-aware router fine-tuning.-- [MC-SMoE](https://github.com/UNITES-Lab/MC-SMoE) for routing-guided expert- merging and compression.-- [MoBE](https://github.com/inclusionAI/MoBE) and- [Dยฒ-MoE](https://github.com/lliai/D2MoE) for shared expert bases and- low-rank expert deltas.-- [HybriMoE](https://github.com/PKU-SEC-Lab/HybriMoE) for hybrid CPU/GPU expert- scheduling, [ScMoE](https://arxiv.org/abs/2404.05019) for overlapping expert- communication with computation, and- [OD-MoE](https://arxiv.org/abs/2512.03927) for distributed on-demand expert- loading.-- [vLLM](https://github.com/vllm-project/vllm),- [llama.cpp](https://github.com/ggml-org/llama.cpp), and- [kTransformers](https://github.com/kvcache-ai/ktransformers) for the open- inference systems and expert-offload work that make comparisons reproducible.--The engine also stands on concrete engineering work, not only ideas. Each of-these is used or reimplemented in the tree today:--- [safetensors](https://github.com/huggingface/safetensors) โ the container- every engine reads (`c/st.h`), including its fp8 and I64 dtypes.-- [tiktoken](https://github.com/openai/tiktoken) โ `c/tok.h` reimplements its- `byte_pair_encode` exactly, merging the adjacent pair whose concatenation has- the lowest vocab id, so a tiktoken-derived vocabulary needs no merges list.-- [llama.cpp](https://github.com/ggml-org/llama.cpp) โ the GBNF grammar subset- in `c/grammar.h` follows its syntax and its set-of-stacks PDA, and the Metal- path borrows its `newBufferWithBytesNoCopy` residency trick.-- [vLLM](https://github.com/vllm-project/vllm) โ the reference for output- semantics the engine matches position by position (e.g. where the final norm- lands relative to the LM head).-- [transformers](https://github.com/huggingface/transformers) โ the oracle:- CI reproduces a random-init model token for token against it.-- [DietGPU](https://github.com/facebookresearch/dietgpu) โ the GPU ANS codec- behind the experimental compressed expert tier (`COLI_ANS`).-- [rocWMMA](https://github.com/ROCm/rocWMMA) โ the HIP backend maps CUDA's- `nvcuda::wmma` fragment/mma_sync API onto it (`c/backend_gpu_compat.h`), which- is what lets one .cu source compile for both vendors.--## License--Apache 2.0, Copyright 2026 Vincenzo Fornaro. See [LICENSE](LICENSE) and [NOTICE](NOTICE). GLM-5.2 weights are released by Z.ai under MIT.+<p align="center">+ <img src="assets/colibri-logo.svg" width="560" alt="colibrรฌ โ tiny engine, immense model">+</p>++<p align="center">+ <a href="https://justvugg.github.io/colibri"><img src="https://img.shields.io/badge/website-justvugg.github.io%2Fcolibri-1f6feb" alt="Website"></a>+ <a href="https://github.com/JustVugg/colibri/releases"><img src="https://img.shields.io/github/v/release/JustVugg/colibri?color=2ea043" alt="Latest release"></a>+</p>++<p align="center">+ <a href="https://justvugg.github.io/colibri"><b>Website</b></a> ยท+ <a href="https://discord.gg/RXV83nSZdk"><b>Discord</b></a> ยท+ English ยท <a href="README.zh-CN.md">็ฎไฝไธญๆ</a> ยท <a href="README.zh-TW.md">็น้ซไธญๆ</a> ยท <a href="README.it.md">Italiano</a>+</p>++**Tiny engine, immense model.** Run **frontier MoE models โ 744B to 2.8T+parameters** โ on consumer and heterogeneous hardware, in pure C with zero+engine dependencies, by treating storage, RAM, and VRAM as a single inference+hierarchy (AI memory multitiering).++Nine families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, with+vision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, with vision),+**Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) and+**OLMoE** (7B) โ+one C file each, the same `coli chat` / `coli serve` / `coli web` front end.+[Full roster โ](#other-supported-models)++> **Colibrรฌ is an inference engine you can run today, and an open research+> platform.** Its primary goal is to pursue inference-side performance across+> the entire software/hardware boundary โ model formats, memory hierarchy,+> storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU+> overlap โ so large models depend less on scarce hardware and cost less to run.++Colibrรฌ treats VRAM, RAM, and storage as a single multitier hierarchy, and it is+deliberately a place to test aggressive systems ideas โ so there is **no SLA on+speed, and a hard guarantee on semantics**: experiments must earn their place+through reproducible end-to-end measurements, and the default policy **never+silently changes model precision or router semantics**. Insufficient fast memory+may reduce speed; it must not quietly redefine the model.++```+$ ./coli chat+ ๐ฆ colibri v1.11.0 โ GLM-5.2 ยท 744B MoE ยท int4 ยท streaming CPU+ โ ready in 32s ยท resident 9.9 GB+ โบ ciao!+ โ Ciao! ๐ Come posso aiutarti oggi?+```++## See it running++<p align="center">+ <img src="docs/media/colibri-dashboard.png" width="900" alt="colibrรฌ web dashboard โ live metrics, hardware panel, expert tiers">+</p>+<p align="center"><em>The web dashboard (<code>./coli web</code>): a 744B model at <strong>4 tok/s, TTFT 1.6 s, disk 0</strong> โ+full expert residency on 6ร RTX 5090, with live token metrics, the per-turn time breakdown,+the VRAM/RAM/disk tier bar and the live mini-brain in the corner.</em></p>++<p align="center">+ <img src="docs/media/colibri-brain.png" width="900" alt="the Brain page โ 19,456 experts as a live cortex">+</p>+<p align="center"><em>The <strong>Brain</strong> page: all 19,456 experts as a living cortex โ colour is the storage tier,+brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's+<a href="https://github.com/JustVugg/colibri/issues/175">measured topic affinity</a>.</em></p>++<p align="center">+ <img src="docs/media/colibri-atlas.png" width="900" alt="the Atlas page โ the measured expert atlas as a 3-D galaxy">+</p>+<p align="center"><em>The <strong>Atlas</strong> page: the <a href="https://github.com/JustVugg/colibri/issues/175">measured expert atlas</a>+as a 3-D galaxy โ 13,260 characterised experts, 1,041 replicated specialists clustering by topic+(poetry, law, Chinese, SQLโฆ). Position is measured routing affinity, not a learned embedding. Drag to spin.</em></p>++## The research mission++With Colibrรฌ, private frontier model access is not limited by availability of hyperscaler-class hardware.++With its multitiering features Colibrรฌ **removes proprietary hardware dependencies aggressively+optimizing functional inference engine pipelines**.++Our operational mission includes changing how weights are represented and moved, deciding what+lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing+launch and synchronization overhead, exploiting sparsity and reuse, and testing+new decoding algorithms. Nothing is protected merely because it is conventional;+nothing is adopted merely because a microbenchmark looks fast. The deciding+result is end-to-end inference on real machines, with correctness and quality+measured alongside throughput, latency, memory, and cost.++The practical consequence is **accessibility**: run a 744B-parameter model on+hardware you already own, watch every expert fire in real time, and change the+code that does it. Not renting intelligence behind an API โ *holding* it:+probing it, measuring it, improving it. The engine is deliberately small enough+that the next useful optimization can come from anyone willing to measure it.++## Core techniques and measured findings++- **One hierarchy, not limited by tier capacity.** VRAM, RAM, and NVMe are placement+ tiers for the same weights; limited fast memory changes speed, not model semantics.+- **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learned+ pinned hot-store, and one-layer-ahead prefetch instead of loading every expert.+ It wins on repeatable workloads; history can overfit, and lookahead can lose on+ some hosts, so both remain measurable policies rather than promises.+- **I/O is part of the engine.** Batched expert unions, overlapped reads and+ compute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path+ rather than pretending storage latency is free. `O_DIRECT` is drive-dependent,+ and dual-SSD still needs broader end-to-end community A/Bs.+- **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full+ expert residency share one runtime and can be combined according to the machine;+ the profitable combination depends on compute, bandwidth, residency, and workload.+- **Compressed state without a different model.** Token-exact forward validation,+ 57ร smaller MLA KV state, persistent warm conversations, and faithful DSA keep+ optimization tied to correctness. These are memory, latency, and correctness+ properties โ not a blanket throughput claim.+- **Speculation that must earn its keep.** Native MTP and grammar-forced drafts+ are measured end to end and can be disabled when acceptance does not repay verification.++## Open hypotheses, experiments, and how to help++Colibrรฌ treats an optimization as a hypothesis until a controlled end-to-end A/B+shows otherwise. These are the main questions now:++| hypothesis | evidence so far | experiment still needed |+|---|---|---|+| Routing history can place experts better than plain LRU | learned pins improve repeated workloads, but can overfit a prompt | held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads |+| Multiple SSDs can turn independent bandwidth into decode speed | two independent NVMe drives measured +37.5% decode; a slower third drive was neutral after weighted striping ([measurements](docs/multidisk.md#what-has-been-measured)) | reproduce across drive speeds, controller layouts, and cache states |+| A hardware-aware planner can approach each machine's best configuration automatically | RAM/VRAM budgets and several backends are detected today | compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems |+| Lossless or quality-bounded representations can reduce weight movement enough to matter | format and quantization ablations exist, with correctness/quality gates | reproduce quality, bytes moved, latency, and cost per useful token together โ not compression ratio alone |+| Routing-aware speculation can pay before near-full residency | MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit | map the break-even surface across acceptance, expert hit rate, batch union, and draft depth |+| CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck | CUDA and Metal wins exist, but fast CPUs and low residency can erase them | per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines |++Want to help? Pick one row and publish the negative results too. Record the+hardware, commit, model/container, exact command, prompt, cache state, throughput,+TTFT, expert hit rate, bytes read, and quality check; change one variable, repeat+the run, and attach raw logs. Start with+[CONTRIBUTING.md](CONTRIBUTING.md), compare against+[the benchmark protocol](docs/benchmarking.md), then+[open an experiment issue](https://github.com/JustVugg/colibri/issues/new).+A well-controlled failure is more valuable here than an unexplained fast number.++## The idea++A 744B Mixture-of-Experts model activates only ~40B parameters per token โ and+only ~11 GB of those change from token to token (the routed experts):++<p align="center">+ <img src="docs/media/sparse.png" width="880" alt="only ~5.4% of parameters are active per token">+</p>++So the model doesn't need to *fit* in fast memory โ it needs to be **placed**:++- the **dense part** (attention, shared experts, embeddings โ ~17B params) stays+ **resident in RAM at int4** (~9.9 GB);+- the **19,456 routed experts** (75 MoE layers ร 256 + the MTP head, ~19 MB each+ at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a+ per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.++Think of the core algorithm as **a JIT, but for weights**. A compiler JIT never+compiles the whole program โ it watches what actually runs and compiles the hot+paths, just in time. colibrรฌ makes the same bet about a 744B parameter space:+parameters are not resident state to be held, they are **data to be staged**+across a heterogeneous storage hierarchy (VRAM / RAM / NVMe), exactly when the+router proves they are needed. Measured routing heat decides which experts earn+which tier, the router runs a layer ahead so prefetch hides the staging latency,+and โ like a JIT โ the engine learns your workload: the more you run, the hotter+the right experts get. It works because routing has measurable structure (see+the [expert atlas](https://github.com/JustVugg/colibri/issues/175)) โ and+structure is cacheable.++The engine is a single C file (`c/colibri.c`) plus small headers. No BLAS, no+Python at runtime, no GPU required.++### Local cluster mode++The coordinator keeps token generation, routing, and KV state local while+disk-backed expert workers execute routed FFNs on other Macs. A layer's routed+batch-union is sent as one persistent TCP request, so a token does not incur one+round trip per expert.++Start the optional registration service:++```bash+./coli cluster coordinator --host 0.0.0.0 --port 8765+```++On each worker, with the same converted model available locally:++```bash+./coli cluster worker --model /nvme/glm52_i4 --port 9100 \+ --coordinator http://COORDINATOR:8765 --advertise-host WORKER_IP+```++Run the coordinator with discovery, or provide `--cluster-workers+HOST:PORT,...` for a static setup:++```bash+./coli serve --model /nvme/glm52_i4 \+ --cluster-coordinator http://127.0.0.1:8765+```++The transport is disabled unless workers are configured, so the existing+single-machine path remains unchanged. Dense-layer sharding and browser/WebGPU+workers are separate follow-up seams.++## How it works++### The per-token path++<p align="center">+ <img src="docs/media/token-path.png" width="880" alt="route โ union โ place โ overlap โ learn">+</p>++Every layer of every token walks the same five steps. The design goal is that+**placement only ever decides speed** โ the router's decisions and the weights'+precision are the same whether an expert answered from VRAM or from disk.++### One memory hierarchy instead of one memory requirement++<p align="center">+ <img src="docs/media/tiers.png" width="880" alt="VRAM / RAM / NVMe three-tier expert residency">+</p>++<a id="dual-ssd-two-copies-of-the-model-twice-the-read-bandwidth"></a>++### Multiple SSDs: stream model copies from more than one drive++When decode is disk-bound, a **second SSD** can help: put a copy of the model on+it and let the engine read from both drives. For GLM-5.2, from `c/` in a source+checkout (or from an unpacked release):++```bash+COLI_MODEL_MIRROR=/second/glm52_i4 python3 ./coli chat --model /fast/glm52_i4+```++The engine measures the drives at startup to weight the read split. Buffered+reads use deterministic expert routing; eligible direct reads can stripe one+expert across replicas. Independent drives provide bandwidth headroom, not a+guaranteed token-rate multiplier: shared controllers, cache hits, and compute+can limit the gain. See the [multi-disk guide](docs/multidisk.md) for Bash and+PowerShell examples, measured gains and limits, and a single-drive comparison.+Details worth knowing:++- the mirror is **validated at startup** (per-file size + safetensors header must be byte-identical to the primary); divergent or missing files stay on the primary, so a **partial mirror is fine** โ a smaller second SSD can serve the shards it holds;+- the mirror is **never written**: `.coli_usage`, `.coli_kv` and all sidecars stay on the primary;+- a read error on the mirror falls back to the primary (one warning, no crash), so unplugging the second drive mid-run degrades instead of killing the server;+- routing never changes tokens โ both copies are byte-identical; enable `PROF=1` for the `MIRROR:` profile counters showing GB served per drive.++The same engine spans the whole range: on a 25 GB laptop everything streams from+disk (slow but correct); on a large host the entire expert set becomes resident+(`CUDA_EXPERT_GB=auto PIN_GB=all`) and disk drops out of the decode path+entirely. Between the tiers sits a **learning cache**: the engine records which+experts *your* workload routes to (`.coli_usage`, updated every turn) and pins+the hottest ones automatically โ colibrรฌ literally gets faster the more you use+it. On multi-socket hosts, `COLI_NUMA=1` interleaves the resident weights across+memory controllers ([#82](https://github.com/JustVugg/colibri/issues/82)).++For a second drive that cannot hold the whole model, Colibri can rank a partial+mirror from the expert history it already learns. Run a few representative+prompts first so `.coli_usage` reflects the workload, then plan, stage, and+verify the mirror:++```bash+./c/coli mirror plan --model /fast/glm52_i4 --mirror /second/glm52_i4 \+ --budget-gib 200 --reserve-gib 20+./c/coli mirror stage --model /fast/glm52_i4 --mirror /second/glm52_i4 \+ --budget-gib 200 --reserve-gib 20+./c/coli mirror verify --model /fast/glm52_i4 --mirror /second/glm52_i4+```++The planner reads safetensors headers directly, follows split-model directories+from `COLI_MODEL_DIRS`, and prioritizes shards that can serve the hottest routed+experts. Staging never changes the primary model: it copies through temporary+files, preserves the requested free-space reserve, verifies every shard with+SHA-256, never deletes an existing mirror shard, and atomically publishes a+receipt only after the selected mirror is ready.++### Never wait for the disk twice++Misses are expensive, so the engine spends most of its cleverness avoiding and+overlapping them: each expert's three matrices are stored adjacent and read in+one `pread`; a bounded async I/O pool (`PIPE=1`, default) loads missing experts+while resident ones compute; batched positions read each unique expert once+(**batch-union**); and a router-lookahead thread (`PILOT=1`) prefetches the next+layer's experts โ routing is measurably **71.6% predictable one layer ahead**.+On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream+on-device across layers so the CPU expert loop runs uninterrupted; on Apple+Silicon an experimental [Metal backend](docs/metal.md) does the batched expert+math on the unified-memory GPU; and a [Vulkan backend](docs/vulkan.md) brings+the expert tier, dense projections, and the MLA attention core to any GPU with+a Vulkan 1.2 driver โ including AMD cards via Mesa/RADV (the only backend for+cards the vendor stacks no longer support, like the RX 580, and competitive+with ROCm on RDNA4 โ see [the benchmarking notes](docs/vulkan.md)).++> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is+> often a large win on drives with DRAM cache and bandwidth headroom (+34%+> decode measured with `PIPE=1` on a Blackwell/Windows box; 4.25โ9.69 GB/s in+> iobench on a GB10) โ but it is drive-dependent: QLC/DRAM-less or virtualised+> disks can be neutral to negative. Try it first; keep what your hardware+> rewards.++### Faithful model, compressed state++The forward pass is validated against a `transformers` oracle (teacher-forcing+typically 30-32/32; two tiny-oracle positions are floating-point near-ties and+toolchain-dependent). MLA attention stores a compressed KV state โ 576+floats/token instead of 32,768 (**57ร smaller**) โ and persists it across+restarts (`.coli_kv`): conversations reopen warm with zero re-prefill,+byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's+lightning indexer) is implemented faithfully and validated by forcing full-key+selection to reproduce dense attention exactly.++### Speculative decoding, honestly++GLM-5.2's native MTP head drafts tokens that the main model verifies in one+batched forward โ 2.2โ2.8 tokens/forward when it pays. Two hard-won rules ship+as defaults: the MTP head must be **int8** (int4 heads collapse to 0โ4%+acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)), and draft and+verify must compute **the same function** โ `SPEC_PIN=1` pins both to one+kernel family ([#163](https://github.com/JustVugg/colibri/issues/163) is the+full forensic story). Grammar-forced drafts+([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) add ~free acceptance on+constrained JSON output. Whether speculation is a net win depends on your+cache temperature โ measure, and use `DRAFT=0` when it doesn't pay.++## What it achieves++<p align="center">+ <img src="docs/media/ladder.png" width="880" alt="measured decode speed by hardware class">+</p>++Same engine, same int4 container โ the hardware only changes where the experts+live. Highlights from the [full benchmark tables](docs/benchmarks.md):++- **6ร RTX 5090, full residency:** 5.8โ6.8 tok/s decode, TTFT ~13 s+ ([experiment log](docs/experiments/glm52-6x5090-2026-07-12.md));+- **128 GB CPU-only desktop:** ~1.8 tok/s warm ([#200](https://github.com/JustVugg/colibri/issues/200));+- **single RTX 5070 Ti laptop-class box:** 1.07 tok/s via the GPU-resident+ pipeline ([#273](https://github.com/JustVugg/colibri/issues/273));+- **25 GB dev box:** 0.05โ0.1 tok/s cold โ the proven floor where this project+ started, and still the honest baseline.++Quality is measured, not assumed: the int4 container's quantization cost and the+scale-granularity/rotation ablations live in+[docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) and+[#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81).++## Get started++You need two things: **the program** (a few hundred KB) and **the model**+(372 GB). Step-by-step for every platform in the+[Quick Start guide](docs/quickstart.md).++### 1. Get colibri++**Download a prebuilt release** โ Linux, macOS and Windows, no compiler needed.+Take the archive for your platform from+[Releases](https://github.com/JustVugg/colibri/releases) and unpack it:++```bash+mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri+python3 coli info # engine ready โ+```++Inside you get the engine (`colibri`, `colibri.exe` on Windows), the `coli`+launcher and its Python helpers. Nothing to rename or configure โ `coli` finds+the engine next to itself. You only need+[Python 3](https://www.python.org/downloads/) installed: the launcher and the+API gateway are Python scripts, while the engine itself is pure C with zero+dependencies.++**Or build from source** โ needs `gcc` (or clang) with OpenMP:++```bash+git clone https://github.com/JustVugg/colibri && cd colibri/c+./setup.sh # checks gcc/OpenMP, builds, self-tests+```++Want `coli` on your PATH? From a checkout, `pip install -e .` registers it (the+engine still lives in `c/` โ an editable install from the clone, not a wheel).++### 2. Get the model++A pre-converted **GLM-5.2 int4** container is on Hugging Face โ use the+**group-scaled (gs64)** build with the **int8 MTP head**. It is about **372 GB**,+so put it on a disk with the room, ideally a fast one:++**https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp**++**GLM-5.3** is the same family and loads with the same engine. It has its own+container, also group-scaled (gs64), about **419 GB**. It ships **without** the+MTP head, so speculative decoding stays off:++**https://huggingface.co/Justvugg/GLM-5.3-colibri-int4-g64**++> โ ๏ธ Use the **gs64** container above, not the older per-row int4 mirrors+> (`mateogrgic/โฆ`, `jlnsrk/โฆ`): those measure ~9pp worse on quality and are the+> root cause of the original think-mode loops and never-terminating generations+> in [#455](https://github.com/JustVugg/colibri/issues/455). The gs64 container+> fixed those controlled per-row A/Bs, but it is not a general repetition or+> EOS-starvation guard. The MTP head must also be **int8, not int4**+> (int4 โ 0% draft acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)):+> `ls -l <model>/out-mtp-*` โ int8 (correct) is `3527131672 / 5366238584 / 1065950496`+> as three files, or a single `out-mtp-00000.safetensors` of `9959321520` bytes+> (the current upload of the recommended container ships it as one file: same+> int8 tensors, 777 of them at one byte per element).++Or convert from the FP8 source yourself โ one resumable command that never needs+the full 756 GB on disk at once:++```bash+./coli convert --model /nvme/glm52_i4 # download+convert shard by shard (python, one-time)+```++#### Other supported models++GLM-5.2 is the reference model, but the same streaming approach runs six more+families. Each is a **sibling engine** โ one C file, its own architecture, the same+`coli chat` / `coli serve` / `coli web` front end (the launcher picks the binary from+the model's `config.json`):++> **What each one needs.** These differ a lot, and reading two of them together+> has confused people into thinking the requirements contradict each other+> ([#191](https://github.com/JustVugg/colibri/issues/191)). They do not โ they+> are different models. **None of them needs a GPU.**+>+> | Model | Disk for the weights | RAM | GPU |+> |---|---|---|---|+> | **OLMoE** | ~7 GB (int8 container) | 8 GB | not needed |+> | **GLM-5.2/5.3** | ~372 GB (5.2) / ~419 GB (5.3) | 16 GB min, 24 GB comfortable | not needed |+> | **GLM-5.3-Flash** | ~195 GB converted | 25 GB (12 GB weights at int4 + expert cache) | not needed |+> | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed |+> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |+> | **DeepSeek V4 Flash** | ~167 GB (REAP 150B: ~85 GB) | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |+> | **Qwen3.8-Flash-Next** | ~185.5 GB (official FP8 checkpoint) | 16 GB min, 24 GB comfortable at the default context | not supported; CPU only |+> | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU |+>+> A GPU only ever makes it faster. Speed is set by your disk, because the experts+> are streamed from it โ expect a fraction of a token per second on a slow drive+> and a few per second on a fast one with the cache warm.++| Family | Total / active | Weights | Build | Docs |+|---|---|---|---|---|+| **GLM-5.2/5.3** | 744B / 40B | [`mastouri/โฆ-int4-g64-with-int8-mtp`](https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp) (372 GB) or [`Justvugg/GLM-5.3-colibri-int4-g64`](https://huggingface.co/Justvugg/GLM-5.3-colibri-int4-g64) (419 GB) | `make -C c glm` | this page |+| **Inkling** (Thinking Machines) | 975B / 41B | [`nbeerbower/Inkling-colibri-int4`](https://huggingface.co/nbeerbower/Inkling-colibri-int4) (469 GB) | `make -C c inkling` | [inkling.md](docs/inkling.md) |+| **GLM-5.3-Flash** (Z.ai) | 321B / 40B | [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) โ converted to **int4-gs64** routed experts, dense stays BF16 and the precision is a load-time choice; vision included | `make -C c glm53` | [glm53-flash.md](docs/glm53-flash.md) |+| **Kimi K3** (Moonshot) | 2.8T / 104B | [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) โ original checkpoint, routed experts stay **native MXFP4** | `make -C c kimi_k3` | [kimi_k3.md](docs/kimi_k3.md) |+| **DeepSeek V4 Flash** | 284B / 13B | official sharded checkpoint โ routed experts stay **native fp4**, dense stays fp8-e4m3; the **REAP-pruned 150B** ([`puwaer/DeepSeek-V4-Flash-0731-reap-150b`](https://huggingface.co/puwaer/DeepSeek-V4-Flash-0731-reap-150b), 85 GB, 132 of 256 experts) loads with the same engine and no conversion | `make -C c deepseek-v4` | [deepseek-v4.md](docs/deepseek-v4.md) |+| **DeepSeek V4.1 Flash** | 552B / 16B | official checkpoint, **no conversion**: experts are already fp4, dense is fp8-e4m3. 203 GB of it is an n-gram memory read from disk a few hundred bytes at a time, and the routed experts cost **4.5 GB per token** against GLM-5.2's 12.7. Vision, tool calling and the DSpark draft head are all on | `make -C c deepseek_v41` | [deepseek-v41.md](docs/deepseek-v41.md) |+| **Qwen3.8-Flash-Next** (Alibaba) | 125B + 51B n-gram / 6B | [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) โ original checkpoint; PLE stays pageable and experts stay **native block-FP8** | `make -C c qwen38` (CPU only) | [qwen38.md](docs/qwen38.md) |+| **Qwen3.6** (Alibaba) | 35B / 3B | [`Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64) (~20 GB, **recommended**) โ hybrid Gated Attention + Gated DeltaNet | `make -C c qwen36` (`CUDA=1` for the VRAM expert tier) | [qwen36.md](docs/qwen36.md) |+| **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` โ **int8** container, ~7 GB | `make -C c olmoe` | โ |++Qwen3.6 ships three pre-converted containers: **int4-gs64** (recommended โ measured+cosine to the int8 anchor 0.98777 โ 0.99313 and KL 0.109 โ 0.080 against per-row, i.e.+~44% less quantization error), [int4 per-row](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4)+as the A/B baseline, and [KAT-Coder v2.5](https://huggingface.co/Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64),+which the same engine runs unchanged โ any architecture-identical checkpoint works+without a code path of its own. With `CUDA=1` the VRAM expert tier measured+**1.44 โ 10.05 tok/s (7.0ร) on two 8 GB cards**, output bit-identical to the CPU path.++Kimi K3 needs no conversion: its QAT-trained MXFP4 experts are streamed straight from+the original Hugging Face shards, and the bf16 dense set is quantized at load time.+Long agent sessions can opt into recurrent-state checkpoints (`COLI_K3_CKPT=N`+slots in RAM, or parked on disk with `COLI_K3_CKPT_DIR`): an edited or follow-up+prompt restores the deepest surviving checkpoint and re-prefills only the tail,+instead of replaying the whole conversation through the SSM layers. On Vulkan+hosts `K3_VK_UP=auto` sizes the expert tier upload from measured bandwidth. The+engine's KDA and MLA paths are validated token-exact in CI against the vendor+implementation.++Inkling ships int4 experts but **bf16 dense weights** (49.4 GB resident); on a host+that cannot hold those, [inkling.md](docs/inkling.md) has a one-pass tool that brings+the dense set to 15.3 GB and lets the 975B run on a 25 GB box โ with the honest+trade-off written down.++### 3. Run it++```bash+COLI_MODEL=/nvme/glm52_i4 ./coli chat # RAM budget, cache and MTP auto-detected+COLI_MODEL=/nvme/glm52_i4 ./coli plan # inspect the planned VRAM/RAM/disk placement+COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only readiness check+COLI_MODEL=/nvme/glm52_i4 ./coli doctor --deep # strict tensors/shards/index/mirror preflight+COLI_MODEL=/nvme/glm52_i4 ./coli tune # measure and save this machine's fastest safe execution profile+./coli web --model /nvme/glm52_i4 # API + dashboard, and opens a browser+./coli serve --model /nvme/glm52_i4 # API + dashboard, no browser (headless)+```++On Windows a release archive ships `coli.cmd`: double-click it for the quick+start, or run `coli.cmd chat --model D:\glm52_i4` from cmd or PowerShell.+From a source checkout the same commands work with `python coli chat --model+D:\glm52_i4`. The `.exe` files are the engines, not the launcher: started on+their own they have no model to load and exit immediately.+The engine at runtime is pure C โ python is only used by the one-time converter+and the optional API gateway.++#### The same commands run any of the models++`coli` reads the model's `config.json`, picks the matching engine binary, and+renders that family's chat template โ so **nothing about the command line+changes between models**. Build the engine you want once, then just point+`COLI_MODEL` at the right directory:++```bash+make -C c glm # GLM-5.2+make -C c inkling # Inkling+make -C c kimi_k3 # Kimi K3++COLI_MODEL=/nvme/glm52_i4 ./coli chat # TUI+COLI_MODEL=/nvme/inkling_i4 ./coli chat+COLI_MODEL=/nvme/kimi_k3 ./coli chat++./coli web --model /nvme/inkling_i4 # API + dashboard, opens a browser+./coli web --model /nvme/kimi_k3+./coli serve --model /nvme/inkling_i4 # API + dashboard, no browser+```++For the non-GLM engines `coli chat` starts the gateway locally and attaches the+TUI to it, so the TUI, the API and the dashboard all go through the same+arch-aware chat template โ you never have to pass the template yourself.++Two things that differ per model, both documented in the per-model page:++- **Inkling on a RAM-tight host** needs the int4 dense container and a small+ expert cache: `./coli chat --model /nvme/inkling_i4 --cap 2`+ (see [inkling.md](docs/inkling.md) โ the default `--cap 8` wants ~14 GB of+ cache on top of the resident set).+- **Kimi K3** streams its MXFP4 experts from the original checkpoint, so there+ is nothing to convert โ but the snapshot is ~1.6 TB+ (see [kimi_k3.md](docs/kimi_k3.md)).++### 4. Go deeper++| topic | doc |+|---|---|+| Benchmarks, community datapoints, quality measurements | [docs/benchmarks.md](docs/benchmarks.md) |+| Reproducible benchmark protocol and minimum report | [docs/benchmarking.md](docs/benchmarking.md) |+| Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) |+| Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) |+| CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) |+| Vulkan backend (any GPU: AMD via RADV, incl. cards ROCm dropped) | [docs/vulkan.md](docs/vulkan.md) |+| Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) |+| OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) |+| Experimental layer-segment embedding ABI | [docs/segment-runtime.md](docs/segment-runtime.md) |+| Experimental tokenizer/embedding/head Edge ABI | [docs/edge-runtime.md](docs/edge-runtime.md) |+| Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) |+| Environment variable inventory | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) |++## DeepSeek V4++**DeepSeek V4 Flash** streams the official checkpoint with no conversion: routed+experts stay **native fp4**, the dense set stays **fp8-e4m3** with UE8M0 block+scales. MLA + DSA sparse attention, 43 layers, 256 routed experts plus one+shared, top-6. Supported on x86-64/aarch64 Linux and Windows/MSYS2 (CPU), with+an optional CUDA tier (Windows runtime DLL; Linux `CUDA=1` direct link,+verified under WSL2) that keeps every stage CPU-canonical and falls back per stage.++```bash+cd c+make deepseek-v4+python ./coli chat --model /path/to/DeepSeek-V4-Flash --ram 32+# also: coli run / coli serve / coli web+# Windows CUDA tier: make cuda-dsv4-dll CUDA_ARCH=portable (+ make cuda-dsv4-dg-dll on RTX 50)+```++Two opt-in GPU levers are new and looking for community numbers, both default+off and byte-identical when unset: `DSV4_HYBRID=1` splits VRAM-tier misses+between the GPU fill branch and the CPU branch using bandwidths measured at+runtime, and `COLI_CUDA_MOE_DOUBLE=1` (on top of `COLI_CUDA_MOE_BATCH=1`)+prefetches the next layer's full expert set into a second VRAM bank while the+current layer computes, falling back to the single bank when VRAM is short.+The CUDA tier also runs on Pascal and Turing cards now (GTX 10 / RTX 20+series): build with `CUDA_ARCH=portable-pre-ampere NO_TC=1`.++Greedy decode and one KV slot. Tool calling is wired through the HTTP gateway+with V4's native prompt and DSML call blocks; grammar is not supported. See the+[per-engine API matrix](docs/api.md#tool-calling-support). Prefix checkpoints+(in memory + on disk) make agent sessions and follow-up turns start in seconds+after the first prefill of a system prompt. Measured on an RTX 5080 + 2 NVMe:+3324-token prefill 90 s, 8.3k-token first turn ~4 min once, later+sessions/turns 6-9 s, decode ~1.6 tok/s at 3k context โ see+[docs/deepseek-v4.md](docs/deepseek-v4.md).++**Give it RAM.** 43 ร 256 routed experts are ~137 GiB on disk and a token+touches 301 of them, so the expert cache hit rate is what sets tok/s โ `--ram`+is the single most valuable knob, and it changes speed only, never output.++**Speculative drafting exists and is off.** DSpark's markov drafter and full MTP+are both implemented and verified: a draft can save forward passes but never+change a token, because every accepted token is still the target's own argmax.+Measured on real multi-turn chat, they accepted 1 in 15 and 10 in 24, and the+rejected-suffix replay of this engine's recurrent attention state cost more than+the drafts saved โ one 14-token answer took 495 seconds. So `V4_DRAFT` and+`V4_MTP` default to `0` and the code stays, with the numbers beside it, for+whoever retries this on faster storage.++See [docs/deepseek-v4.md](docs/deepseek-v4.md) for the CUDA tier (build, DLL+selection, GPU coverage), the environment reference, performance numbers,+checkpoint validation, and the generated tiny independent oracle.++## What's next++- **Inference-systems research is the product.** The current hierarchy is LRU ++ a learned pin set; active work spans model formats, compression, placement,+ scheduling, I/O, CPU/GPU kernels, heterogeneous overlap, KV state, and+ routing-aware speculation. The objective is lower hardware requirements and+ lower cost per useful token. Everything lands the way this project works:+ measured end to end, reviewed, and developed in the open.+- **More open models.** The tiering algorithm is model-agnostic: any MoE with+ routed experts can be staged the same way. Nine families run today (GLM-5.2,+ GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.8-Flash-Next,+ Qwen3.6, OLMoE); further open-weight families โ **MiniMax** among the+ candidates โ earn an engine the way the first eight did: when someone+ measures one end to end.++## Supporting the project++colibrรฌ started as a one-person project on a 12-core laptop with 25 GB of RAM;+today its numbers come from a community of real machines. If it's useful to you:++- โญ star the repo and share it;+- ๐ open issues with benchmark numbers from your hardware โ datapoints move+ this project more than anything else;+- ๐ฌ join the [Discord community](https://discord.gg/RXV83nSZdk) to discuss+ experiments, hardware results, and research directions;+- ๐ฌ reach out via GitHub issues to sponsor development or donate hardware.++## Repo layout++```+Makefile root build/check entry point+c/+โโโ colibri.c GLM-5.2 engine (make glm)+โโโ inkling.c Inkling engine (make inkling)+โโโ kimi_k3.c Kimi K3 engine (make kimi_k3)+โโโ deepseek_v4.c DeepSeek V4 Flash engine (make deepseek-v4)+โโโ qwen38.c Qwen3.8-Flash-Next text engine (make qwen38)+โโโ qwen36.c Qwen3.6 engine (make qwen36)+โโโ olmoe.c OLMoE engine (make olmoe)+โ+โโโ st.h safetensors index and range reads+โโโ quant.h canonical container decoders+โโโ expert_ffn.h routed-expert FFN kernel shared by the MoE engines (planar int4, layer runner)+โโโ tok.h, json.h tokenizer and JSON parser+โโโ compat.h Windows/macOS shims (POSIX names, one place)+โโโ expert_store.h streaming expert cache+โโโ route_trace.h routing telemetry and .coli_usage, engine-agnostic+โโโ kv_prefix.h KV prefix reuse across turns+โ+โโโ backend_cuda.* optional CUDA tier (CUDA=1)+โโโ backend_metal.* optional Metal tier (METAL=1)+โโโ backend_vulkan.* optional Vulkan tier (VULKAN=1)+โ+โโโ Makefile build and local checks+โโโ coli user-facing CLI+โโโ openai_server.py OpenAI-compatible HTTP gateway+โโโ resource_plan.py RAM/VRAM planner behind `coli plan` and `coli doctor`+โโโ tools/ offline conversion, fixtures and benchmarks+โโโ scripts/ long-running conversion helpers+โโโ tests/ dependency-free C and Python tests+web/ browser UI (pure OpenAI-API client)+desktop/ Tauri v2 desktop shell wrapping the web UI+docker/ container images+docs/ reference docs, experiments, media+```++**One `.c` per model family, over shared single headers.** An engine owns its+architecture and nothing else; anything two engines both need โ the safetensors+reader, the container decoders, the tokenizer, the expert cache โ lives in a+header they both include, so a fix reaches all of them at once. That rule is not+decorative: the defects that keep recurring here are the ones where a mechanism+landed in one engine and never reached its siblings.++From the repository root, `make`, `make check` and `make clean` delegate to the+engine Makefile.++## Why "colibrรฌ"++The hummingbird weighs a few grams, hovers in place, and visits a thousand+flowers a day. This engine keeps a 744-billion-parameter giant alive on+hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.++## Acknowledgements++colibrรฌ is an engine; the minds it runs are a gift. Thank you to the teams+releasing frontier-class weights in the open โ **Z.ai** (GLM), **Moonshot AI**+(Kimi), **Alibaba Qwen**, **MiniMax**, and **Allen AI** (OLMoE) โ and to every+contributor who benchmarked, bisected, replicated an atlas run, or sent a patch.+This project is proof of what open weights make possible.++The project's expert placement, compression, and routing experiments also build+on ideas and evidence from the following open research and systems work:++- [REAP](https://github.com/CerebrasResearch/reap) and+ [EASY-EP](https://github.com/RUCAIBox/EASYEP) for output-aware and+ domain-specific expert importance.+- [SERE](https://github.com/JL-Cheng/SERE) for similarity-based expert+ re-routing, and [ReMoE](https://github.com/BUAA-OSCAR/ReMoE) for+ cache-locality-aware router fine-tuning.+- [MC-SMoE](https://github.com/UNITES-Lab/MC-SMoE) for routing-guided expert+ merging and compression.+- [MoBE](https://github.com/inclusionAI/MoBE) and+ [Dยฒ-MoE](https://github.com/lliai/D2MoE) for shared expert bases and+ low-rank expert deltas.+- [HybriMoE](https://github.com/PKU-SEC-Lab/HybriMoE) for hybrid CPU/GPU expert+ scheduling, [ScMoE](https://arxiv.org/abs/2404.05019) for overlapping expert+ communication with computation, and+ [OD-MoE](https://arxiv.org/abs/2512.03927) for distributed on-demand expert+ loading.+- [vLLM](https://github.com/vllm-project/vllm),+ [llama.cpp](https://github.com/ggml-org/llama.cpp), and+ [kTransformers](https://github.com/kvcache-ai/ktransformers) for the open+ inference systems and expert-offload work that make comparisons reproducible.++The engine also stands on concrete engineering work, not only ideas. Each of+these is used or reimplemented in the tree today:++- [safetensors](https://github.com/huggingface/safetensors) โ the container+ every engine reads (`c/st.h`), including its fp8 and I64 dtypes.+- [tiktoken](https://github.com/openai/tiktoken) โ `c/tok.h` reimplements its+ `byte_pair_encode` exactly, merging the adjacent pair whose concatenation has+ the lowest vocab id, so a tiktoken-derived vocabulary needs no merges list.+- [llama.cpp](https://github.com/ggml-org/llama.cpp) โ the GBNF grammar subset+ in `c/grammar.h` follows its syntax and its set-of-stacks PDA, and the Metal+ path borrows its `newBufferWithBytesNoCopy` residency trick.+- [vLLM](https://github.com/vllm-project/vllm) โ the reference for output+ semantics the engine matches position by position (e.g. where the final norm+ lands relative to the LM head).+- [transformers](https://github.com/huggingface/transformers) โ the oracle:+ CI reproduces a random-init model token for token against it.+- [DietGPU](https://github.com/facebookresearch/dietgpu) โ the GPU ANS codec+ behind the experimental compressed expert tier (`COLI_ANS`).+- [rocWMMA](https://github.com/ROCm/rocWMMA) โ the HIP backend maps CUDA's+ `nvcuda::wmma` fragment/mma_sync API onto it (`c/backend_gpu_compat.h`), which+ is what lets one .cu source compile for both vendors.++## License++Apache 2.0, Copyright 2026 Vincenzo Fornaro. See [LICENSE](LICENSE) and [NOTICE](NOTICE). GLM-5.2 weights are released by Z.ai under MIT.