Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
gateway: up to 16 KV slots on every text engine; docs
JustVugg20aef0b725dd
diff --git a/c/family_registry.py b/c/family_registry.pyindex 7c2fc117..87ecd2ee 100644--- a/c/family_registry.py+++ b/c/family_registry.py@@ -1357,7 +1357,7 @@ FAMILIES = (# e' una ghigliottina che cade DENTRO al blocco di pensiero e chiude il# turno senza risposta (#1278). 16384 e' il valore che hanno gia' tutte# le famiglie con lo stesso contesto massimo di 1048576.- limits=FamilyLimits(8192, 1048576, 1024, 16384, 1, 8, "CTX_MAX"),+ limits=FamilyLimits(8192, 1048576, 1024, 16384, 16, 8, "CTX_MAX"),capabilities=FamilyCapabilities(False, False, True, True),has_gateway_adapter=True,tune_prompt_template="<|user|>{prompt}<|assistant|>",@@ -1391,7 +1391,7 @@ FAMILIES = (# e' una ghigliottina che cade DENTRO al blocco di pensiero e chiude il# turno senza risposta (#1278). 16384 e' il valore che hanno gia' tutte# le famiglie con lo stesso contesto massimo di 1048576.- limits=FamilyLimits(8192, 1048576, 1024, 16384, 1, 8, "K3_MAXT"),+ limits=FamilyLimits(8192, 1048576, 1024, 16384, 16, 8, "K3_MAXT"),# tools=True: this family DOES render and parse tool calls. It used to# share COMMON_CAP, which says otherwise -- the flag is descriptive# (it only feeds the capability dict) so nothing broke, but a client@@ -1436,7 +1436,7 @@ FAMILIES = (# slower than the cache the same machine could hold: measured on a# 1204-token prefill, cap 8 gives 22.8% expert hit rate and 0.045 tok/s,# cap 64 gives 99.4% and 0.215 tok/s.- limits=FamilyLimits(4096, 4096, 1024, 1024, 1, 0, "CTX"),+ limits=FamilyLimits(4096, 4096, 1024, 1024, 16, 0, "CTX"),capabilities=FamilyCapabilities(False, False, False, False),has_gateway_adapter=True,has_cli_adapter=True,@@ -1487,7 +1487,7 @@ FAMILIES = (planner_unsupported_reason="",expert_inventory=_individual_expert_inventory(_GLM_EXPERT),config_section="text_config",- limits=FamilyLimits(8192, 262144, 1024, 8192, 1, 8, "Q36_MAXT"),+ limits=FamilyLimits(8192, 262144, 1024, 8192, 16, 8, "Q36_MAXT"),capabilities=FamilyCapabilities(False, False, False, True, image=True),has_gateway_adapter=True,# coli run stays unwired on purpose: cmd_run dispatches per arch after@@ -1520,7 +1520,7 @@ FAMILIES = (resident_inventory=_qwen38_resident_inventory,fixed_resident_inventory=_qwen38_fixed_resident_inventory,config_section="text_config",- limits=FamilyLimits(8192, 262144, 1024, 8192, 1, 1, "Q38_MAXT"),+ limits=FamilyLimits(8192, 262144, 1024, 8192, 16, 1, "Q38_MAXT"),capabilities=FamilyCapabilities(True, False, False, True, image=True),has_gateway_adapter=True,# Like Qwen3.6, direct `coli run` is intentionally not exposed until@@ -1561,7 +1561,7 @@ FAMILIES = (planner_unsupported_reason="",expert_inventory=_individual_expert_inventory(_V4_EXPERT),config_section="root",- limits=FamilyLimits(4096, 1048576, 1024, 16384, 1, 8, "CTX"),+ limits=FamilyLimits(4096, 1048576, 1024, 16384, 16, 8, "CTX"),capabilities=FamilyCapabilities(True, False, False, True),has_gateway_adapter=True,has_cli_adapter=True,@@ -1598,7 +1598,7 @@ FAMILIES = (expert_inventory=_dsv41_expert_inventory,resident_inventory=_dsv41_resident_inventory,config_section="text_config",- limits=FamilyLimits(4096, 1048576, 1024, 16384, 1, 8, "CTX"),+ limits=FamilyLimits(4096, 1048576, 1024, 16384, 16, 8, "CTX"),# tools yes (DSML, see v41_dsml.py), grammars no: the engine reads the six-field# SUBMIT header and has no constrained decoder, so a grammar has to be refused# at the gateway rather than desync the wire.@@ -1648,7 +1648,7 @@ FAMILIES = (resident_inventory=_mimo_resident_inventory,config_section="root",# top-8 routing: the engine raises any smaller cache to one routing step- limits=FamilyLimits(8192, 1048576, 1024, 16384, 1, 16, "CTX"),+ limits=FamilyLimits(8192, 1048576, 1024, 16384, 16, 16, "CTX"),# tools (the XML call form, parse_mimo_tool_calls), thinking (on by default,# as the template has it); no grammars, no audio in or outcapabilities=FamilyCapabilities(True, False, False, True, image=True),diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex e5ec0114..7237e5f2 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -53,7 +53,7 @@ Format: `VAR` β default β effect.| `TOPP` | `0` (off) | Top-p filter (`0` = use `NUCLEUS`). || `SEED` | unset β seeded from clock + PID | RNG seed for sampling. **Unset = different every run.** Set a fixed value for reproducible sampling. || `KVSAVE` | `1` (on) | Persist the KV cache to `<model>/.coli_kv` so a conversation reopens warm. `KVSAVE=0` disables save+load (lossless round-trip; does not change output). |-| `KV_SLOTS` | `1` | Number of independent KV conversation slots (1β16), used in serve mode. |+| `KV_SLOTS` | `1` | Number of independent KV conversation slots (1β16), used in serve mode. Every text engine decodes the active slots' requests together, one row each a step ([api.md](api.md#isolated-kv-contexts)). || `KV8` | `0` (off) | Store the MLA latent KV cache in fp8 e4m3 with a per-row scale: ~3.9Γ less KV RAM, and `.coli_kv` shrinks ~4Γ (saved as the v2 format; f32 v1 files are quantized on resume and rewritten). Adds DeepSeek-V3-class KV quantization noise to attention. CPU attention path only for now: the CUDA/Metal fused-attention fast paths read f32 KV rows, so under KV8 they fall back to the CPU consumer (native fp8 decode; a one-time notice is printed under `COLI_CUDA_ATTN=1`). Forces `COLI_CUDA_PIPE=0`. Native CUDA/Metal fp8-KV kernels are follow-up PRs. || `KV_TQ` | `0` (off) | Sub-byte MLA latent KV quantization, mutually exclusive with `KV8` (`KV_TQ` wins). `KV_TQ=4` is the recommended tier: rotated-int4 codec (randomized-Hadamard rotation + Lloyd codebook, per-row radius as the scale), ~7.6Γ less KV RAM than f32. `KV_TQ=2|3|5|6` selects the PolarQuant codec at that bit width (`KV_TQ_POLAR=1` forces PolarQuant at 4 bits too). Requires power-of-two row widths (`kv_lora`/`qk_rope`; the GLM MLA shapes 512/64 qualify) β on a model whose shapes don't, the engine refuses to start rather than silently zeroing the cache. A value below the 2β6 grid (e.g. `KV_TQ=1`) is treated as the recommended `4` with a notice, not as the most aggressive tier. `.coli_kv` is saved as the v3 format; a file saved under a different KV mode, codec, or bit width is refused with an explicit message and the cache restarts. Same CPU-only status as `KV8`: GPU fast paths fall back to the CPU consumer; native kernels are follow-up PRs. Forces `COLI_CUDA_PIPE=0`. || `THINK` | `0` (off) | Emit a `<think>` reasoning block. `THINK=1` turns on visible reasoning. |diff --git a/docs/api.md b/docs/api.mdindex 0deb2bd4..3830fc8b 100644--- a/docs/api.md+++ b/docs/api.md@@ -559,12 +559,27 @@ OpenAI clients omit it and keep the original slot 0 behavior.}```-Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and-crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches-each request's tokenized prompt against the slot's history and reuses the common-KV prefix, so stateless HTTP turns keep their cache across requests and even-across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent.-Start small: at the default 4096-token context, every slot costs hundreds of MB.+Each slot owns its token history and its conversation's state: the KV cache, and+on the engines that have them the DeltaNet, KDA and convolution states and the+compressed DeepSeek attention. The engine matches each request's tokenized prompt+against the slot's history and reuses the common prefix, so stateless HTTP turns+keep their cache across requests. On GLM-5.2 each slot also has its MTP window and+a crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...), so the cache+survives an engine restart too. Use `COLI_KV_SLOTS=N` as the environment+equivalent. Start small: at the default 4096-token context, every slot costs+hundreds of MB, and `coli plan` counts them.++Every text engine serves the slots at the same time. Requests on different slots+are decoded together: each step takes the next token of every active+conversation as one batch, so the weights and the routed experts a step reads+serve all of them, while each row's attention and recurrent state stay its own+conversation's. A request gets the tokens it would get alone (greedy on the CPU,+the same bytes; `tests/serve_mux_check.py` checks it on every engine). A prompt is+prefilled when it arrives, between two steps. With more than one slot, the+engines other than GLM-5.2 and GLM-5.3-Flash draft nothing (MTP, DSpark and+prompt lookup follow one conversation), and their Vulkan dense chain, DeltaNet on+the GPU and Metal paths stay off; the Vulkan expert tier serves every+conversation's experts.## Web dashboarddiff --git a/docs/serve_protocol.md b/docs/serve_protocol.mdindex e89ead31..3fdae93a 100644--- a/docs/serve_protocol.md+++ b/docs/serve_protocol.md@@ -80,7 +80,12 @@ DECIDE <id> <slot> <bytes>\n<payload>\n- `slot` β KV slot index, `0 β¦ KV_SLOTS-1` (`KV_SLOTS` env, 1β16, default 1). A slotholds one conversation's KV; the engine matches the tokenized payload against theslot's history and reuses the common prefix (truncate-and-extend), so stateless- HTTP turns keep their cache.+ HTTP turns keep their cache. With `KV_SLOTS` above 1 every text engine takes a+ `SUBMIT` on a free slot while others decode: its prompt is prefilled at once, and+ from then on each decode step runs one row for every active request. The frames of+ the requests interleave, each tagged with its `id`. A `SUBMIT` on a slot whose+ request is still running gets `ERROR <id> SLOT_BUSY`; a slot out of range,+ `ERROR <id> invalid cache slot`.- `bytes` β exact byte length of `payload` (UTF-8, may contain newlines). The enginereads exactly that many bytes after the header line, then one trailing `\n`.- `payload` β the fully rendered prompt (the server owns the chat template).diff --git a/docs/vulkan.md b/docs/vulkan.mdindex 3a883e61..ed8a271a 100644--- a/docs/vulkan.md+++ b/docs/vulkan.md@@ -1509,6 +1509,14 @@ rebuilds the KDA state on the CPU from the input rows the chain records since thcopy was last current (embedding rows, or the vision tower's), a prefill's worth of CPUwork.+**Several conversations at once (the other engines).** qwen36, qwen38, OLMoE, Inkling,+Kimi K3, MiMo, DeepSeek V4 and V4.1 decode `KV_SLOTS`' conversations together too, but+their dense chain keeps one conversation's state on the device, so it stays off with+more than one slot (a `[VK] <engine>: KV_SLOTS=n: the dense chain is off` line says+so): the dense part runs on the CPU, and the routed-expert tier serves the experts of+every conversation's row. `tests/vulkan_engines.sh mux` checks each engine's frames+against the same requests served alone, on the CPU and with the tier.+**Several conversations at once (colibri).** `KV_SLOTS`' batched decode (one row fromeach active conversation, each at its own position) runs on the chain: the norms, theprojections and the MoE take the batch's rows as one; each row's new KV rows go to its