Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: bring the Quick Start guide (incl. Windows prebuilt steps) to main
JustVuggec087f6bcaab
diff --git a/README.md b/README.mdindex 797f1ca1..085e4e92 100644--- a/README.md+++ b/README.md@@ -151,6 +151,15 @@ scale-granularity/rotation ablations live in## Get started+> **New here, or on Windows?** The [Quick Start guide](docs/quickstart.md) walks+> through install β build β model β first chat step by step for Linux, Windows,+> and macOS. On **Windows** you don't even need to build: download the+> `colibri-<version>-windows-x86_64.zip` from+> [Releases](https://github.com/JustVugg/colibri/releases), unzip it, rename+> `colibri-*-windows-x86_64.exe` β `glm.exe`, install+> [Python 3](https://www.python.org/downloads/), and run `coli chat` β full+> details in the [Windows section](docs/quickstart.md#windows).+### 1. Get the modelA pre-converted **GLM-5.2 int4** container is on Hugging Face β **use thediff --git a/docs/quickstart.md b/docs/quickstart.mdnew file mode 100644index 00000000..60d48777--- /dev/null+++ b/docs/quickstart.md@@ -0,0 +1,184 @@+# Quick Start β from zero to a running model++A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**.+No prior experience with C, CUDA, or model conversion is assumed. If you get+stuck, `./coli doctor` (below) tells you exactly what's missing.++> **What you're setting up:** colibrΓ¬ runs a very large Mixture-of-Experts model+> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's+> experts from disk instead of needing them all in RAM. The engine is a single+> C program; Python is only used once, to prepare the model files.++---++## 0. What you need first (prerequisites)++| | Minimum | Recommended |+|---|---|---|+| **RAM** | ~16 GB | 24 GB+ |+| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) |+| **OS** | Linux, Windows 10/11, or macOS | any |+| **Tools** | a C compiler + `make` + `git` + `python3` | β |++You do **not** need a GPU. A GPU only helps if you have one; the engine runs+CPU-only by default.++---++## 1. Install the build tools++### Linux (Ubuntu / Debian)++```bash+sudo apt update+sudo apt install -y build-essential git python3+```++`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) β everything the+engine needs.++### Windows++You have two options.++**Option A β download a prebuilt binary (no compiler needed).**+Grab `colibri-<version>-windows-x86_64.zip` from the+[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it.+Inside you'll find:++| File | What it is |+|---|---|+| `colibri-<version>-windows-x86_64.exe` | **the engine** β the C program that actually runs the model |+| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, β¦) |+| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner |++Two setup steps:++1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a+ binary named `glm`):+ ```powershell+ Rename-Item colibri-*-windows-x86_64.exe glm.exe+ ```+2. **Install Python 3** from [python.org](https://www.python.org/downloads/) β the+ `coli` launcher and the API gateway are Python scripts (the engine itself is+ pure C and needs nothing).++Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can+run the engine directly β `.\glm.exe` reads the model path from the `SNAP`+environment variable (see [docs/windows.md](windows.md)) β but `coli chat` is the+easy path.++**Option B β build from source with MSYS2.**+Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run:++```bash+pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python+```++### macOS++```bash+xcode-select --install # C compiler (clang)+brew install libomp git python # OpenMP for multithreading+```++---++## 2. Get the code and build the engine++```bash+git clone https://github.com/JustVugg/colibri.git+cd colibri/c+./setup.sh+```++`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny+self-test. When it prints:++```+engine self-test: 32/32 (expected 32/32)+```++the engine is working correctly. (On Windows Option A you already have the+binary β you can skip this step.)++---++## 3. Get the model++You have two paths.++### Easiest β download a ready-made int4 container++A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version+with the int8 MTP heads** (the plain int4 heads disable speculative decoding β+see [#8](https://github.com/JustVugg/colibri/issues/8)):++**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**++Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or+`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space.++### Or convert it yourself from the FP8 source++One resumable command downloads and converts the model shard by shard, so it+never needs the full ~756 GB on disk at once:++```bash+./coli convert --model /nvme/glm52_i4+```++This step uses Python and runs only once. Safe to interrupt and re-run β it+resumes where it left off.++---++## 4. Run it++Point `COLI_MODEL` at the folder from step 3 and start chatting:++```bash+# Linux / macOS+COLI_MODEL=/nvme/glm52_i4 ./coli chat++# Windows (UCRT64 shell)+COLI_MODEL=/d/glm52_i4 ./coli chat+```++Useful first commands:++```bash+COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready?+COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU)+COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality+```++> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine β it reads+> fewer expert bytes per token with no quality loss, which directly means more+> tokens per second.++---++## 5. What to expect++- **First launch loads the resident weights** (~10 GB) β this takes a moment.+- **Speed depends on your disk.** The experts stream from storage, so a fast+ NVMe SSD is the single biggest factor in tokens/second. On a slow or shared+ disk, generation can be well under 1 token/second β that's expected, and it's+ the honest cost of running a 744B model on a small machine.+- **It's still the full model.** Placement only changes speed, never the model's+ answers or precision.++If something doesn't work, run `./coli doctor` β it reports exactly what's+missing (compiler, model files, permissions) and how to fix it.++---++## Where to go next++| Topic | Doc |+|---|---|+| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) |+| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) |+| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) |+| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) |