Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docker: add slim multi-stage image (Dockerfile.slim) + compose β€” 115MB non-root portable build alongside the beginner Dockerfile

ZacharyZcRaa65e21df29b
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 00000000..2ad637b4
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,20 @@
+# The Docker build only needs c/ (engine sources + launcher). Everything else
+# is either huge, host-specific, or irrelevant to the image.
+.git
+web/node_modules
+web/dist
+site
+docs
+**/*.o
+**/*.exe
+c/colibri
+c/glm
+c/olmoe
+c/glm_tiny
+**/.coli_usage
+**/.coli_kv*
+**/__pycache__
+*.safetensors
+*.log
+CODEX-BRIEF.md
+wt-inc4
diff --git a/docker/Dockerfile.slim b/docker/Dockerfile.slim
new file mode 100644
index 00000000..e41bab5a
--- /dev/null
+++ b/docker/Dockerfile.slim
@@ -0,0 +1,54 @@
+# colibri β€” CPU inference image (multi-stage: build then a slim runtime)
+#
+# The engine is portable C with a Python launcher/gateway. This image ships
+# ONLY the inference path (chat/serve/run/info/plan/doctor) β€” the ~372 GB
+# model is mounted, never baked in, and the converter (which needs torch) is
+# deliberately out of scope; convert from a source checkout instead.
+#
+# docker build -t colibri -f docker/Dockerfile.slim .
+# docker run --rm -v /nvme/glm52_i4:/model colibri info
+# docker run --rm -it -v /nvme/glm52_i4:/model colibri chat --ram 24
+# docker run --rm -p 5000:5000 -v /nvme/glm52_i4:/model colibri \
+# serve --host 0.0.0.0 --model-id glm-5.2
+#
+# --------------------------------------------------------------------------
+# Stage 1: build the engine. ARCH is x86-64-v3 (portable AVX2) via `make
+# portable`'s target, NOT native β€” a native build bakes in the builder CPU's
+# ISA and SIGILLs on a different host. Override ARCH at build time to pin one.
+FROM debian:stable-slim AS build
+ARG ARCH=x86-64-v3
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends build-essential make && \
+ rm -rf /var/lib/apt/lists/*
+WORKDIR /src
+COPY c/ ./c/
+RUN make -C c colibri ARCH=${ARCH}
+
+# --------------------------------------------------------------------------
+# Stage 2: runtime. openai_server.py and the launcher use only the Python
+# standard library, so no pip install β€” just python3 + the built binary and
+# its support files. libgomp1 provides the OpenMP runtime the engine links.
+FROM debian:stable-slim
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends python3 libgomp1 && \
+ rm -rf /var/lib/apt/lists/* && \
+ useradd -m -u 1000 colibri
+WORKDIR /app
+
+# The engine binary plus the launcher and the files it resolves at runtime
+# (version.py, the gateway, resource planner, doctor, tokenizer headers, tools).
+COPY --from=build /src/c/colibri ./colibri
+COPY c/coli c/version.py c/openai_server.py c/resource_plan.py c/doctor.py ./
+COPY c/tools/ ./tools/
+
+ENV COLI_MODEL=/model \
+ COLI_ENGINE=/app/colibri \
+ PYTHONUNBUFFERED=1
+# A bind-mounted model lands here; declaring the volume documents the contract.
+VOLUME ["/model"]
+# `coli serve` default port.
+EXPOSE 5000
+
+USER colibri
+ENTRYPOINT ["python3", "/app/coli"]
+CMD ["info"]
diff --git a/docker/README.md b/docker/README.md
index 9f5800c0..aed1a574 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -485,4 +485,48 @@ PS C:\quack\llm\colibri\docker> docker run --rm -it --name colibri-c -v "C:\quac
```
+---
+
+## Slim image (developers / production)
+
+The guide above uses `Dockerfile`, which clones the repo inside the image β€” one
+file to download, nothing else on your machine. For CI, servers, or anyone who
+already has the repo checked out, `Dockerfile.slim` is a multi-stage build that
+produces a **~115 MB** image (vs. the full toolchain the clone-in-image keeps
+around), runs as a **non-root** user, and compiles the engine with a
+**portable** ISA (`x86-64-v3`) so the image runs on any modern x86-64 host
+instead of only the machine that built it.
+
+Build it from the repo root (it needs the `c/` sources as context):
+
+```bash
+git clone https://github.com/JustVugg/colibri.git && cd colibri
+docker build -t colibri -f docker/Dockerfile.slim .
+```
+
+Run β€” the model is bind-mounted at `/model`, never baked in:
+
+```bash
+docker run --rm -v /nvme/glm52_i4:/model colibri info
+docker run --rm -it -v /nvme/glm52_i4:/model colibri chat --ram 24
+docker run --rm -p 5000:5000 -v /nvme/glm52_i4:/model colibri \
+ serve --host 0.0.0.0 --model-id glm-5.2 --ram 24
+```
+
+The entrypoint is `coli`, so the first argument is the subcommand (`info`,
+`chat`, `serve`, `run`, `plan`, `doctor`) β€” no `./coli` prefix. The container
+runs as UID 1000, so the model directory must be readable by that user
+(a directory you own already is; a root-owned one needs `chmod -R a+rX` or a
+matching `--user`).
+
+### One-command server (docker compose)
+
+```bash
+MODEL_DIR=/nvme/glm52_i4 COLI_RAM=24 docker compose -f docker/docker-compose.yml up
+```
+
+Serves the OpenAI-compatible API on `localhost:5000`. The converter (which
+needs PyTorch) is intentionally not in this image β€” convert from a source
+checkout, then point `MODEL_DIR` at the result.
+
[source: 1]
\ No newline at end of file
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
new file mode 100644
index 00000000..bcf0e30a
--- /dev/null
+++ b/docker/docker-compose.yml
@@ -0,0 +1,33 @@
+# colibri β€” OpenAI-compatible server in one command.
+#
+# MODEL_DIR=/nvme/glm52_i4 docker compose -f docker/docker-compose.yml up
+#
+# Then: curl http://localhost:5000/v1/chat/completions -d '{"model":"glm-5.2",
+# "messages":[{"role":"user","content":"Hi"}]}'
+#
+# The model is bind-mounted read-only from MODEL_DIR on the host β€” keep it on
+# NVMe/ext4, never a network mount (expert streaming is latency-bound).
+services:
+ colibri:
+ build:
+ context: ..
+ dockerfile: docker/Dockerfile.slim
+ image: colibri
+ ports:
+ - "5000:5000"
+ volumes:
+ - "${MODEL_DIR:?set MODEL_DIR to the int4 model directory}:/model:ro"
+ environment:
+ # RAM budget in GB: more cached experts = faster. Size to the host.
+ COLI_RAM: "${COLI_RAM:-24}"
+ command:
+ - serve
+ - --host
+ - 0.0.0.0
+ - --model-id
+ - glm-5.2
+ - --ram
+ - "${COLI_RAM:-24}"
+ restart: unless-stopped
+ # Streaming a 744B model touches a lot of page cache; give it headroom.
+ shm_size: "2gb"