Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docker: add slim multi-stage image (Dockerfile.slim) + compose β 115MB non-root portable build alongside the beginner Dockerfile
ZacharyZcRaa65e21df29b
diff --git a/.dockerignore b/.dockerignorenew file mode 100644index 00000000..2ad637b4--- /dev/null+++ b/.dockerignore@@ -0,0 +1,20 @@+# The Docker build only needs c/ (engine sources + launcher). Everything else+# is either huge, host-specific, or irrelevant to the image.+.git+web/node_modules+web/dist+site+docs+**/*.o+**/*.exe+c/colibri+c/glm+c/olmoe+c/glm_tiny+**/.coli_usage+**/.coli_kv*+**/__pycache__+*.safetensors+*.log+CODEX-BRIEF.md+wt-inc4diff --git a/docker/Dockerfile.slim b/docker/Dockerfile.slimnew file mode 100644index 00000000..e41bab5a--- /dev/null+++ b/docker/Dockerfile.slim@@ -0,0 +1,54 @@+# colibri β CPU inference image (multi-stage: build then a slim runtime)+#+# The engine is portable C with a Python launcher/gateway. This image ships+# ONLY the inference path (chat/serve/run/info/plan/doctor) β the ~372 GB+# model is mounted, never baked in, and the converter (which needs torch) is+# deliberately out of scope; convert from a source checkout instead.+#+# docker build -t colibri -f docker/Dockerfile.slim .+# docker run --rm -v /nvme/glm52_i4:/model colibri info+# docker run --rm -it -v /nvme/glm52_i4:/model colibri chat --ram 24+# docker run --rm -p 5000:5000 -v /nvme/glm52_i4:/model colibri \+# serve --host 0.0.0.0 --model-id glm-5.2+#+# --------------------------------------------------------------------------+# Stage 1: build the engine. ARCH is x86-64-v3 (portable AVX2) via `make+# portable`'s target, NOT native β a native build bakes in the builder CPU's+# ISA and SIGILLs on a different host. Override ARCH at build time to pin one.+FROM debian:stable-slim AS build+ARG ARCH=x86-64-v3+RUN apt-get update && \+ apt-get install -y --no-install-recommends build-essential make && \+ rm -rf /var/lib/apt/lists/*+WORKDIR /src+COPY c/ ./c/+RUN make -C c colibri ARCH=${ARCH}++# --------------------------------------------------------------------------+# Stage 2: runtime. openai_server.py and the launcher use only the Python+# standard library, so no pip install β just python3 + the built binary and+# its support files. libgomp1 provides the OpenMP runtime the engine links.+FROM debian:stable-slim+RUN apt-get update && \+ apt-get install -y --no-install-recommends python3 libgomp1 && \+ rm -rf /var/lib/apt/lists/* && \+ useradd -m -u 1000 colibri+WORKDIR /app++# The engine binary plus the launcher and the files it resolves at runtime+# (version.py, the gateway, resource planner, doctor, tokenizer headers, tools).+COPY --from=build /src/c/colibri ./colibri+COPY c/coli c/version.py c/openai_server.py c/resource_plan.py c/doctor.py ./+COPY c/tools/ ./tools/++ENV COLI_MODEL=/model \+ COLI_ENGINE=/app/colibri \+ PYTHONUNBUFFERED=1+# A bind-mounted model lands here; declaring the volume documents the contract.+VOLUME ["/model"]+# `coli serve` default port.+EXPOSE 5000++USER colibri+ENTRYPOINT ["python3", "/app/coli"]+CMD ["info"]diff --git a/docker/README.md b/docker/README.mdindex 9f5800c0..aed1a574 100644--- a/docker/README.md+++ b/docker/README.md@@ -485,4 +485,48 @@ PS C:\quack\llm\colibri\docker> docker run --rm -it --name colibri-c -v "C:\quac```+---++## Slim image (developers / production)++The guide above uses `Dockerfile`, which clones the repo inside the image β one+file to download, nothing else on your machine. For CI, servers, or anyone who+already has the repo checked out, `Dockerfile.slim` is a multi-stage build that+produces a **~115 MB** image (vs. the full toolchain the clone-in-image keeps+around), runs as a **non-root** user, and compiles the engine with a+**portable** ISA (`x86-64-v3`) so the image runs on any modern x86-64 host+instead of only the machine that built it.++Build it from the repo root (it needs the `c/` sources as context):++```bash+git clone https://github.com/JustVugg/colibri.git && cd colibri+docker build -t colibri -f docker/Dockerfile.slim .+```++Run β the model is bind-mounted at `/model`, never baked in:++```bash+docker run --rm -v /nvme/glm52_i4:/model colibri info+docker run --rm -it -v /nvme/glm52_i4:/model colibri chat --ram 24+docker run --rm -p 5000:5000 -v /nvme/glm52_i4:/model colibri \+ serve --host 0.0.0.0 --model-id glm-5.2 --ram 24+```++The entrypoint is `coli`, so the first argument is the subcommand (`info`,+`chat`, `serve`, `run`, `plan`, `doctor`) β no `./coli` prefix. The container+runs as UID 1000, so the model directory must be readable by that user+(a directory you own already is; a root-owned one needs `chmod -R a+rX` or a+matching `--user`).++### One-command server (docker compose)++```bash+MODEL_DIR=/nvme/glm52_i4 COLI_RAM=24 docker compose -f docker/docker-compose.yml up+```++Serves the OpenAI-compatible API on `localhost:5000`. The converter (which+needs PyTorch) is intentionally not in this image β convert from a source+checkout, then point `MODEL_DIR` at the result.+[source: 1]\ No newline at end of filediff --git a/docker/docker-compose.yml b/docker/docker-compose.ymlnew file mode 100644index 00000000..bcf0e30a--- /dev/null+++ b/docker/docker-compose.yml@@ -0,0 +1,33 @@+# colibri β OpenAI-compatible server in one command.+#+# MODEL_DIR=/nvme/glm52_i4 docker compose -f docker/docker-compose.yml up+#+# Then: curl http://localhost:5000/v1/chat/completions -d '{"model":"glm-5.2",+# "messages":[{"role":"user","content":"Hi"}]}'+#+# The model is bind-mounted read-only from MODEL_DIR on the host β keep it on+# NVMe/ext4, never a network mount (expert streaming is latency-bound).+services:+ colibri:+ build:+ context: ..+ dockerfile: docker/Dockerfile.slim+ image: colibri+ ports:+ - "5000:5000"+ volumes:+ - "${MODEL_DIR:?set MODEL_DIR to the int4 model directory}:/model:ro"+ environment:+ # RAM budget in GB: more cached experts = faster. Size to the host.+ COLI_RAM: "${COLI_RAM:-24}"+ command:+ - serve+ - --host+ - 0.0.0.0+ - --model-id+ - glm-5.2+ - --ram+ - "${COLI_RAM:-24}"+ restart: unless-stopped+ # Streaming a 744B model touches a lot of page cache; give it headroom.+ shm_size: "2gb"