Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: define the inference research mission
ZacharyZcRb583e9c3acb7
diff --git a/README.md b/README.mdindex a17696c3..0d2f481e 100644--- a/README.md+++ b/README.md@@ -12,12 +12,22 @@English · <a href="README.zh-CN.md">简体中文</a> · <a href="README.zh-TW.md">繁體中文</a> · <a href="README.it.md">Italiano</a></p>-**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.--Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM,-and storage as one managed memory hierarchy. Insufficient fast memory may reduce-speed, but the default policy **never silently changes model precision or router-semantics**.+**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a+consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by+streaming experts from disk.++> **Colibrì is an experimental inference engine and research platform.** Its+> primary goal is to pursue inference-side performance across the entire+> software/hardware boundary — model formats, memory hierarchy, storage I/O,+> placement, scheduling, kernels, speculation, and CPU/GPU overlap — so large+> models depend less on scarce hardware and cost less to run.++Colibrì treats VRAM, RAM, and storage as one managed memory hierarchy. It is+deliberately a place to test aggressive systems ideas, not a production runtime+with an SLA. Experiments must earn their place through reproducible end-to-end+measurements, and the default policy **never silently changes model precision+or router semantics**. Insufficient fast memory may reduce speed; it must not+quietly redefine the model.```$ ./coli chat@@ -50,15 +60,26 @@ brightness is routing heat, and every expert routed in a turn flashes white. Hovas a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic(poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.</em></p>-## The vision+## The research mission++Frontier inference should not require datacenter-class hardware by default.+Colibrì's research target is simple: **reduce the hardware dependency and total+cost of inference by optimizing every part of the inference path that evidence+shows is limiting it**.++That includes changing how weights are represented and moved, deciding what+lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing+launch and synchronization overhead, exploiting sparsity and reuse, and testing+new decoding algorithms. Nothing is protected merely because it is conventional;+nothing is adopted merely because a microbenchmark looks fast. The deciding+result is end-to-end inference on real machines, with correctness and quality+measured alongside throughput, latency, memory, and cost.-Frontier models should not be sealed inside datacenters. colibrì exists so that-**anyone curious enough can open one up**: run a 744B-parameter mind on hardware-you already own, watch every expert fire in real time, and change the code that-does it. Not renting intelligence behind an API — *holding* it: probing it,-measuring it, improving it. Every optimisation in this project started with-someone measuring something on their own machine; the engine is deliberately-small enough that the next one can come from you.+The practical consequence is accessibility: run a 744B-parameter model on+hardware you already own, watch every expert fire in real time, and change the+code that does it. Not renting intelligence behind an API — *holding* it:+probing it, measuring it, improving it. The engine is deliberately small enough+that the next useful optimization can come from anyone willing to measure it.## The idea@@ -244,9 +265,10 @@ so put it on a disk with the room, ideally a fast one:> ⚠️ Use the **gs64** container above, not the older per-row int4 mirrors> (`mateogrgic/…`, `jlnsrk/…`): those measure ~9pp worse on quality and are the-> root cause of the think-mode loops and never-terminating generations in-> [#455](https://github.com/JustVugg/colibri/issues/455) — the gs64 container-> cured every failing case there. The MTP head must also be **int8, not int4**+> root cause of the original think-mode loops and never-terminating generations+> in [#455](https://github.com/JustVugg/colibri/issues/455). The gs64 container+> fixed those controlled per-row A/Bs, but it is not a general repetition or+> EOS-starvation guard. The MTP head must also be **int8, not int4**> (int4 → 0% draft acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)):> `ls -l <model>/out-mtp-*` — int8 (correct) is `3527131672 / 5366238584 / 1065950496`.@@ -287,11 +309,12 @@ and the optional API gateway.## What's next-- **Algorithmic research is active.** The current hierarchy is LRU + a learned- pin set; the next step is under way — smarter placement and scheduling,- overlap of CPU and GPU expert execution, and routing-aware speculation.- Everything lands the way this project always works: measured, reviewed, and- merged in the open.+- **Inference-systems research is the product.** The current hierarchy is LRU ++ a learned pin set; active work spans model formats, compression, placement,+ scheduling, I/O, CPU/GPU kernels, heterogeneous overlap, KV state, and+ routing-aware speculation. The objective is lower hardware requirements and+ lower cost per useful token. Everything lands the way this project works:+ measured end to end, reviewed, and developed in the open.- **More open models.** The tiering algorithm is model-agnostic: any MoE withrouted experts can be staged the same way. GLM-5.2 and OLMoE run today;support for more open-weight families — **Kimi K2** (Moonshot AI),