Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: bring the Quick Start guide (incl. Windows prebuilt steps) to main

JustVuggec087f6bcaab
diff --git a/README.md b/README.md
index 797f1ca1..085e4e92 100644
--- a/README.md
+++ b/README.md
@@ -151,6 +151,15 @@ scale-granularity/rotation ablations live in
## Get started
+> **New here, or on Windows?** The [Quick Start guide](docs/quickstart.md) walks
+> through install β†’ build β†’ model β†’ first chat step by step for Linux, Windows,
+> and macOS. On **Windows** you don't even need to build: download the
+> `colibri-<version>-windows-x86_64.zip` from
+> [Releases](https://github.com/JustVugg/colibri/releases), unzip it, rename
+> `colibri-*-windows-x86_64.exe` β†’ `glm.exe`, install
+> [Python 3](https://www.python.org/downloads/), and run `coli chat` β€” full
+> details in the [Windows section](docs/quickstart.md#windows).
+
### 1. Get the model
A pre-converted **GLM-5.2 int4** container is on Hugging Face β€” **use the
diff --git a/docs/quickstart.md b/docs/quickstart.md
new file mode 100644
index 00000000..60d48777
--- /dev/null
+++ b/docs/quickstart.md
@@ -0,0 +1,184 @@
+# Quick Start β€” from zero to a running model
+
+A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**.
+No prior experience with C, CUDA, or model conversion is assumed. If you get
+stuck, `./coli doctor` (below) tells you exactly what's missing.
+
+> **What you're setting up:** colibrì runs a very large Mixture-of-Experts model
+> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's
+> experts from disk instead of needing them all in RAM. The engine is a single
+> C program; Python is only used once, to prepare the model files.
+
+---
+
+## 0. What you need first (prerequisites)
+
+| | Minimum | Recommended |
+|---|---|---|
+| **RAM** | ~16 GB | 24 GB+ |
+| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) |
+| **OS** | Linux, Windows 10/11, or macOS | any |
+| **Tools** | a C compiler + `make` + `git` + `python3` | β€” |
+
+You do **not** need a GPU. A GPU only helps if you have one; the engine runs
+CPU-only by default.
+
+---
+
+## 1. Install the build tools
+
+### Linux (Ubuntu / Debian)
+
+```bash
+sudo apt update
+sudo apt install -y build-essential git python3
+```
+
+`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) β€” everything the
+engine needs.
+
+### Windows
+
+You have two options.
+
+**Option A β€” download a prebuilt binary (no compiler needed).**
+Grab `colibri-<version>-windows-x86_64.zip` from the
+[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it.
+Inside you'll find:
+
+| File | What it is |
+|---|---|
+| `colibri-<version>-windows-x86_64.exe` | **the engine** β€” the C program that actually runs the model |
+| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, …) |
+| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner |
+
+Two setup steps:
+
+1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a
+ binary named `glm`):
+ ```powershell
+ Rename-Item colibri-*-windows-x86_64.exe glm.exe
+ ```
+2. **Install Python 3** from [python.org](https://www.python.org/downloads/) β€” the
+ `coli` launcher and the API gateway are Python scripts (the engine itself is
+ pure C and needs nothing).
+
+Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can
+run the engine directly β€” `.\glm.exe` reads the model path from the `SNAP`
+environment variable (see [docs/windows.md](windows.md)) β€” but `coli chat` is the
+easy path.
+
+**Option B β€” build from source with MSYS2.**
+Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run:
+
+```bash
+pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python
+```
+
+### macOS
+
+```bash
+xcode-select --install # C compiler (clang)
+brew install libomp git python # OpenMP for multithreading
+```
+
+---
+
+## 2. Get the code and build the engine
+
+```bash
+git clone https://github.com/JustVugg/colibri.git
+cd colibri/c
+./setup.sh
+```
+
+`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny
+self-test. When it prints:
+
+```
+engine self-test: 32/32 (expected 32/32)
+```
+
+the engine is working correctly. (On Windows Option A you already have the
+binary β€” you can skip this step.)
+
+---
+
+## 3. Get the model
+
+You have two paths.
+
+### Easiest β€” download a ready-made int4 container
+
+A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version
+with the int8 MTP heads** (the plain int4 heads disable speculative decoding β€”
+see [#8](https://github.com/JustVugg/colibri/issues/8)):
+
+**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**
+
+Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or
+`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space.
+
+### Or convert it yourself from the FP8 source
+
+One resumable command downloads and converts the model shard by shard, so it
+never needs the full ~756 GB on disk at once:
+
+```bash
+./coli convert --model /nvme/glm52_i4
+```
+
+This step uses Python and runs only once. Safe to interrupt and re-run β€” it
+resumes where it left off.
+
+---
+
+## 4. Run it
+
+Point `COLI_MODEL` at the folder from step 3 and start chatting:
+
+```bash
+# Linux / macOS
+COLI_MODEL=/nvme/glm52_i4 ./coli chat
+
+# Windows (UCRT64 shell)
+COLI_MODEL=/d/glm52_i4 ./coli chat
+```
+
+Useful first commands:
+
+```bash
+COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready?
+COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU)
+COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality
+```
+
+> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine β€” it reads
+> fewer expert bytes per token with no quality loss, which directly means more
+> tokens per second.
+
+---
+
+## 5. What to expect
+
+- **First launch loads the resident weights** (~10 GB) β€” this takes a moment.
+- **Speed depends on your disk.** The experts stream from storage, so a fast
+ NVMe SSD is the single biggest factor in tokens/second. On a slow or shared
+ disk, generation can be well under 1 token/second β€” that's expected, and it's
+ the honest cost of running a 744B model on a small machine.
+- **It's still the full model.** Placement only changes speed, never the model's
+ answers or precision.
+
+If something doesn't work, run `./coli doctor` β€” it reports exactly what's
+missing (compiler, model files, permissions) and how to fix it.
+
+---
+
+## Where to go next
+
+| Topic | Doc |
+|---|---|
+| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) |
+| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) |
+| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) |
+| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) |