Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: serve protocol reference β mux wire format, telemetry lines, HTTP surface (#310)
ZacharyZcRdeafa72b95f3
diff --git a/docs/serve_protocol.md b/docs/serve_protocol.mdnew file mode 100644index 00000000..b0eb61ec--- /dev/null+++ b/docs/serve_protocol.md@@ -0,0 +1,107 @@+# The serve protocols β engine β server wire format++The engine speaks two line-oriented protocols over stdin/stdout. Both are plain text+plus byte-counted payload frames; every outbound line is written with a trailing+`fflush`, one line per write. On Windows both ends of the pipe are switched to binary+mode at startup β the CRT's CRLF translation otherwise corrupts the sentinels and+stalls byte-counted reads (#195).++| protocol | entry | selected by | used by |+|---|---|---|---|+| **mux** (continuous batching, up to 16 KV slots) | `run_serve_mux` | `SERVE_BATCH=1` | `openai_server.py`, `coli web` |+| **legacy** (single slot, interactive) | `run_serve` | `SERVE=1` (without `SERVE_BATCH`) | `coli chat` |++This document is the reference for the **mux** protocol; the legacy protocol is+summarized at the end. Line formats below are quoted from the emitting `printf`s in+`glm.c` β if this document and the code disagree, the code wins and this file needs a PR.++## Startup handshake (engine β server)++```+\x01\x01READY\x01\x01+STAT 0 0.00 0.0 <rss_gb>+HWINFO <cores> <ram_total_gb> <ram_avail_gb> <ngpu> <vram_total_gb> <cpu_name>|<gpu_name>+TIERS <vram_experts> <ram_experts> <disk_experts> <vram_gb> <ram_gb>+EMAP <rows> <cols> <hex>+```++The server must not send requests before `READY`. `HWINFO`/`TIERS`/`EMAP` are+telemetry (see below) and may grow β **servers must ignore line kinds they do not+recognize**; that is the protocol's forward-compatibility rule.++## Requests (server β engine)++```+SUBMIT <id> <slot> <bytes> <max_tokens> <temperature> <top_p>\n<payload>\n+CANCEL <id>\n+```++- `id` β non-zero u64, unique among in-flight requests.+- `slot` β KV slot index, `0 β¦ KV_SLOTS-1` (`KV_SLOTS` env, 1β16, default 1). A slot+ holds one conversation's KV; the engine matches the tokenized payload against the+ slot's history and reuses the common prefix (truncate-and-extend), so stateless+ HTTP turns keep their cache.+- `bytes` β exact byte length of `payload` (UTF-8, may contain newlines). The engine+ reads exactly that many bytes after the header line, then one trailing `\n`.+- `payload` β the fully rendered prompt (the server owns the chat template).+- EOF on stdin = graceful shutdown: in-flight requests finish first.++Prefill is serial; decode is continuously batched β every active slot contributes+one row per forward.++## Responses (engine β server)++Per request, in order:++```+DATA <id> <n>\n<n bytes of UTF-8>\n # a decoded token's text; repeated+TOPK <id> 5 <logprob> <hextext> ... Γ5 # candidates for the sampled token (SERVE_TOPK=1)+HITS <rows> <cols> <hex> # ~every 6 tokens: routed-expert bitmap since last HITS+REPIN <layer> <eid> <old_tier> <gpu> # live re-pin swap events, as they happen+...+DONE <id> STAT <emitted> <tok_s> <hit_pct> <rss_gb> <prompt_tokens> <length_limited>+```++Errors replace the stream: `ERROR <id> <CODE>` with codes `BAD_FRAME`, `BAD_REQUEST`,+`SLOT_BUSY`, `DUPLICATE_ID`, `EMPTY_PROMPT`, `NOT_FOUND` (CANCEL of unknown id),+`CANCELLED`. A `CANCEL` is acknowledged by `ERROR <id> CANCELLED` after the slot's KV+is persisted.++Immediately before each `DONE` the engine emits a telemetry block for the finished+turn: `HWINFO`, `PERF`, `ENTROPY`, `GPUS`, `TIERS`, `EMAP`, `HITS` (formats below).+`.coli_usage` is persisted at every turn end, not only at exit.++## Telemetry lines++| line | format | meaning |+|---|---|---|+| `TIERS` | `TIERS <vram> <ram> <disk> <vram_gb> <ram_gb>` | expert count per tier + resident bytes |+| `HWINFO` | `HWINFO <cores> <ram_total> <ram_avail> <ngpu> <vram_total> <cpu>\|<gpu>` | host snapshot (GBs are floats) |+| `EMAP` | `EMAP <rows> <cols> <hex>` | one byte per expert, row-major over `rowsΓcols` (sparse layers +MTP Γ experts): `byte = (tier<<6) \| heat` β 2-bit tier (0 disk / 1 RAM / 2 VRAM), 6-bit logβ-bucketed usage heat |+| `HITS` | `HITS <rows> <cols> <hex>` | 1 bit per expert, experts routed since the previous `HITS` |+| `PERF` | `PERF <id> <dt> <t_edisk> <t_ewait> <t_emm> <t_attn> <t_kvb> <t_head>` | this turn's PROFILO deltas, seconds |+| `ENTROPY` | `ENTROPY <h0> <h1> β¦` | per-sparse-layer routing entropy of the turn, bits |+| `GPUS` | `GPUS <n> (<used_gb> <total_gb> <experts>)Γn` | per-device VRAM + resident expert count (CUDA builds) |+| `TOPK` | `TOPK <id> 5 (<logprob> <hextext>)Γ5` | token text hex-encoded so the line stays line-shaped |+| `REPIN` | `REPIN <layer> <eid> <old_tier> <gpu>` | one line per hot-store swap (`REPIN=n` mode) |++All telemetry is advisory: servers render what they know and skip the rest.++## HTTP surface (`openai_server.py`)++- `POST /v1/chat/completions` β OpenAI-compatible; streaming responses emit one extra+ SSE frame `data: {"colibri": {stats, perf, topk, entropy, gpus, repin}}` immediately+ before `data: [DONE]`; non-streaming responses attach the same object as a+ `"colibri"` field.+- `GET /experts` β the latest `EMAP`/`HITS` state: `{rows, cols, map, hits, seq,+ gpus, entropy, repin}`.+- `GET /*` β static hosting of `web/dist` (SPA fallback, path-traversal-safe), plus+ `experts.json` if published there (the measured expert atlas, #175/#218).++## Legacy protocol (`run_serve`, `coli chat`)++Interactive lines are prompts; control frames: `\x02RESET` (clear history),+`\x02MORE` (continue an NGEN-truncated answer), and+`\x02PROMPT <bytes> <max_tokens> <temperature> <top_p> [kv_slot]\n<prompt>\n`+(the pre-mux API mode). Responses are raw text terminated by `\x01\x01END\x01\x01`+plus a `STAT` line. New integrations should use the mux protocol.