Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: beginner-friendly Quick Start guide for Linux/Windows/macOS (#414)
JustVugg845af6378d21
diff --git a/README.md b/README.mdindex e5d16517..f9cf3f00 100644--- a/README.md+++ b/README.md@@ -151,6 +151,10 @@ scale-granularity/rotation ablations live in## Get started+> **New here?** The [Quick Start guide](docs/quickstart.md) walks through+> install β build β model β first chat step by step for Linux, Windows, and+> macOS, with copy-paste commands and no assumed background.+### 1. Get the modelA pre-converted **GLM-5.2 int4** container is on Hugging Face β **use thediff --git a/docs/quickstart.md b/docs/quickstart.mdnew file mode 100644index 00000000..7d6a9bfd--- /dev/null+++ b/docs/quickstart.md@@ -0,0 +1,164 @@+# Quick Start β from zero to a running model++A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**.+No prior experience with C, CUDA, or model conversion is assumed. If you get+stuck, `./coli doctor` (below) tells you exactly what's missing.++> **What you're setting up:** colibrΓ¬ runs a very large Mixture-of-Experts model+> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's+> experts from disk instead of needing them all in RAM. The engine is a single+> C program; Python is only used once, to prepare the model files.++---++## 0. What you need first (prerequisites)++| | Minimum | Recommended |+|---|---|---|+| **RAM** | ~16 GB | 24 GB+ |+| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) |+| **OS** | Linux, Windows 10/11, or macOS | any |+| **Tools** | a C compiler + `make` + `git` + `python3` | β |++You do **not** need a GPU. A GPU only helps if you have one; the engine runs+CPU-only by default.++---++## 1. Install the build tools++### Linux (Ubuntu / Debian)++```bash+sudo apt update+sudo apt install -y build-essential git python3+```++`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) β everything the+engine needs.++### Windows++You have two options.++**Option A β download a prebuilt binary (no compiler needed).**+Grab the latest `colibri-<version>-windows-x86_64.zip` from the+[Releases page](https://github.com/JustVugg/colibri/releases), unzip it, and+skip to [step 3](#3-get-the-model). Python 3 (from+[python.org](https://www.python.org/downloads/)) is still needed if you want to+convert a model yourself.++**Option B β build from source with MSYS2.**+Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run:++```bash+pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python+```++### macOS++```bash+xcode-select --install # C compiler (clang)+brew install libomp git python # OpenMP for multithreading+```++---++## 2. Get the code and build the engine++```bash+git clone https://github.com/JustVugg/colibri.git+cd colibri/c+./setup.sh+```++`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny+self-test. When it prints:++```+engine self-test: 32/32 (expected 32/32)+```++the engine is working correctly. (On Windows Option A you already have the+binary β you can skip this step.)++---++## 3. Get the model++You have two paths.++### Easiest β download a ready-made int4 container++A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version+with the int8 MTP heads** (the plain int4 heads disable speculative decoding β+see [#8](https://github.com/JustVugg/colibri/issues/8)):++**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**++Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or+`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space.++### Or convert it yourself from the FP8 source++One resumable command downloads and converts the model shard by shard, so it+never needs the full ~756 GB on disk at once:++```bash+./coli convert --model /nvme/glm52_i4+```++This step uses Python and runs only once. Safe to interrupt and re-run β it+resumes where it left off.++---++## 4. Run it++Point `COLI_MODEL` at the folder from step 3 and start chatting:++```bash+# Linux / macOS+COLI_MODEL=/nvme/glm52_i4 ./coli chat++# Windows (UCRT64 shell)+COLI_MODEL=/d/glm52_i4 ./coli chat+```++Useful first commands:++```bash+COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready?+COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU)+COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality+```++> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine β it reads+> fewer expert bytes per token with no quality loss, which directly means more+> tokens per second.++---++## 5. What to expect++- **First launch loads the resident weights** (~10 GB) β this takes a moment.+- **Speed depends on your disk.** The experts stream from storage, so a fast+ NVMe SSD is the single biggest factor in tokens/second. On a slow or shared+ disk, generation can be well under 1 token/second β that's expected, and it's+ the honest cost of running a 744B model on a small machine.+- **It's still the full model.** Placement only changes speed, never the model's+ answers or precision.++If something doesn't work, run `./coli doctor` β it reports exactly what's+missing (compiler, model files, permissions) and how to fix it.++---++## Where to go next++| Topic | Doc |+|---|---|+| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) |+| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) |+| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) |+| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) |