Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

gateway: up to 16 KV slots on every text engine; docs

JustVugg20aef0b725dd
diff --git a/c/family_registry.py b/c/family_registry.py
index 7c2fc117..87ecd2ee 100644
--- a/c/family_registry.py
+++ b/c/family_registry.py
@@ -1357,7 +1357,7 @@ FAMILIES = (
# e' una ghigliottina che cade DENTRO al blocco di pensiero e chiude il
# turno senza risposta (#1278). 16384 e' il valore che hanno gia' tutte
# le famiglie con lo stesso contesto massimo di 1048576.
- limits=FamilyLimits(8192, 1048576, 1024, 16384, 1, 8, "CTX_MAX"),
+ limits=FamilyLimits(8192, 1048576, 1024, 16384, 16, 8, "CTX_MAX"),
capabilities=FamilyCapabilities(False, False, True, True),
has_gateway_adapter=True,
tune_prompt_template="<|user|>{prompt}<|assistant|>",
@@ -1391,7 +1391,7 @@ FAMILIES = (
# e' una ghigliottina che cade DENTRO al blocco di pensiero e chiude il
# turno senza risposta (#1278). 16384 e' il valore che hanno gia' tutte
# le famiglie con lo stesso contesto massimo di 1048576.
- limits=FamilyLimits(8192, 1048576, 1024, 16384, 1, 8, "K3_MAXT"),
+ limits=FamilyLimits(8192, 1048576, 1024, 16384, 16, 8, "K3_MAXT"),
# tools=True: this family DOES render and parse tool calls. It used to
# share COMMON_CAP, which says otherwise -- the flag is descriptive
# (it only feeds the capability dict) so nothing broke, but a client
@@ -1436,7 +1436,7 @@ FAMILIES = (
# slower than the cache the same machine could hold: measured on a
# 1204-token prefill, cap 8 gives 22.8% expert hit rate and 0.045 tok/s,
# cap 64 gives 99.4% and 0.215 tok/s.
- limits=FamilyLimits(4096, 4096, 1024, 1024, 1, 0, "CTX"),
+ limits=FamilyLimits(4096, 4096, 1024, 1024, 16, 0, "CTX"),
capabilities=FamilyCapabilities(False, False, False, False),
has_gateway_adapter=True,
has_cli_adapter=True,
@@ -1487,7 +1487,7 @@ FAMILIES = (
planner_unsupported_reason="",
expert_inventory=_individual_expert_inventory(_GLM_EXPERT),
config_section="text_config",
- limits=FamilyLimits(8192, 262144, 1024, 8192, 1, 8, "Q36_MAXT"),
+ limits=FamilyLimits(8192, 262144, 1024, 8192, 16, 8, "Q36_MAXT"),
capabilities=FamilyCapabilities(False, False, False, True, image=True),
has_gateway_adapter=True,
# coli run stays unwired on purpose: cmd_run dispatches per arch after
@@ -1520,7 +1520,7 @@ FAMILIES = (
resident_inventory=_qwen38_resident_inventory,
fixed_resident_inventory=_qwen38_fixed_resident_inventory,
config_section="text_config",
- limits=FamilyLimits(8192, 262144, 1024, 8192, 1, 1, "Q38_MAXT"),
+ limits=FamilyLimits(8192, 262144, 1024, 8192, 16, 1, "Q38_MAXT"),
capabilities=FamilyCapabilities(True, False, False, True, image=True),
has_gateway_adapter=True,
# Like Qwen3.6, direct `coli run` is intentionally not exposed until
@@ -1561,7 +1561,7 @@ FAMILIES = (
planner_unsupported_reason="",
expert_inventory=_individual_expert_inventory(_V4_EXPERT),
config_section="root",
- limits=FamilyLimits(4096, 1048576, 1024, 16384, 1, 8, "CTX"),
+ limits=FamilyLimits(4096, 1048576, 1024, 16384, 16, 8, "CTX"),
capabilities=FamilyCapabilities(True, False, False, True),
has_gateway_adapter=True,
has_cli_adapter=True,
@@ -1598,7 +1598,7 @@ FAMILIES = (
expert_inventory=_dsv41_expert_inventory,
resident_inventory=_dsv41_resident_inventory,
config_section="text_config",
- limits=FamilyLimits(4096, 1048576, 1024, 16384, 1, 8, "CTX"),
+ limits=FamilyLimits(4096, 1048576, 1024, 16384, 16, 8, "CTX"),
# tools yes (DSML, see v41_dsml.py), grammars no: the engine reads the six-field
# SUBMIT header and has no constrained decoder, so a grammar has to be refused
# at the gateway rather than desync the wire.
@@ -1648,7 +1648,7 @@ FAMILIES = (
resident_inventory=_mimo_resident_inventory,
config_section="root",
# top-8 routing: the engine raises any smaller cache to one routing step
- limits=FamilyLimits(8192, 1048576, 1024, 16384, 1, 16, "CTX"),
+ limits=FamilyLimits(8192, 1048576, 1024, 16384, 16, 16, "CTX"),
# tools (the XML call form, parse_mimo_tool_calls), thinking (on by default,
# as the template has it); no grammars, no audio in or out
capabilities=FamilyCapabilities(True, False, False, True, image=True),
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index e5ec0114..7237e5f2 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -53,7 +53,7 @@ Format: `VAR` β€” default β€” effect.
| `TOPP` | `0` (off) | Top-p filter (`0` = use `NUCLEUS`). |
| `SEED` | unset β†’ seeded from clock + PID | RNG seed for sampling. **Unset = different every run.** Set a fixed value for reproducible sampling. |
| `KVSAVE` | `1` (on) | Persist the KV cache to `<model>/.coli_kv` so a conversation reopens warm. `KVSAVE=0` disables save+load (lossless round-trip; does not change output). |
-| `KV_SLOTS` | `1` | Number of independent KV conversation slots (1–16), used in serve mode. |
+| `KV_SLOTS` | `1` | Number of independent KV conversation slots (1–16), used in serve mode. Every text engine decodes the active slots' requests together, one row each a step ([api.md](api.md#isolated-kv-contexts)). |
| `KV8` | `0` (off) | Store the MLA latent KV cache in fp8 e4m3 with a per-row scale: ~3.9Γ— less KV RAM, and `.coli_kv` shrinks ~4Γ— (saved as the v2 format; f32 v1 files are quantized on resume and rewritten). Adds DeepSeek-V3-class KV quantization noise to attention. CPU attention path only for now: the CUDA/Metal fused-attention fast paths read f32 KV rows, so under KV8 they fall back to the CPU consumer (native fp8 decode; a one-time notice is printed under `COLI_CUDA_ATTN=1`). Forces `COLI_CUDA_PIPE=0`. Native CUDA/Metal fp8-KV kernels are follow-up PRs. |
| `KV_TQ` | `0` (off) | Sub-byte MLA latent KV quantization, mutually exclusive with `KV8` (`KV_TQ` wins). `KV_TQ=4` is the recommended tier: rotated-int4 codec (randomized-Hadamard rotation + Lloyd codebook, per-row radius as the scale), ~7.6Γ— less KV RAM than f32. `KV_TQ=2|3|5|6` selects the PolarQuant codec at that bit width (`KV_TQ_POLAR=1` forces PolarQuant at 4 bits too). Requires power-of-two row widths (`kv_lora`/`qk_rope`; the GLM MLA shapes 512/64 qualify) β€” on a model whose shapes don't, the engine refuses to start rather than silently zeroing the cache. A value below the 2–6 grid (e.g. `KV_TQ=1`) is treated as the recommended `4` with a notice, not as the most aggressive tier. `.coli_kv` is saved as the v3 format; a file saved under a different KV mode, codec, or bit width is refused with an explicit message and the cache restarts. Same CPU-only status as `KV8`: GPU fast paths fall back to the CPU consumer; native kernels are follow-up PRs. Forces `COLI_CUDA_PIPE=0`. |
| `THINK` | `0` (off) | Emit a `<think>` reasoning block. `THINK=1` turns on visible reasoning. |
diff --git a/docs/api.md b/docs/api.md
index 0deb2bd4..3830fc8b 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -559,12 +559,27 @@ OpenAI clients omit it and keep the original slot 0 behavior.
}
```
-Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
-crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches
-each request's tokenized prompt against the slot's history and reuses the common
-KV prefix, so stateless HTTP turns keep their cache across requests and even
-across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent.
-Start small: at the default 4096-token context, every slot costs hundreds of MB.
+Each slot owns its token history and its conversation's state: the KV cache, and
+on the engines that have them the DeltaNet, KDA and convolution states and the
+compressed DeepSeek attention. The engine matches each request's tokenized prompt
+against the slot's history and reuses the common prefix, so stateless HTTP turns
+keep their cache across requests. On GLM-5.2 each slot also has its MTP window and
+a crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...), so the cache
+survives an engine restart too. Use `COLI_KV_SLOTS=N` as the environment
+equivalent. Start small: at the default 4096-token context, every slot costs
+hundreds of MB, and `coli plan` counts them.
+
+Every text engine serves the slots at the same time. Requests on different slots
+are decoded together: each step takes the next token of every active
+conversation as one batch, so the weights and the routed experts a step reads
+serve all of them, while each row's attention and recurrent state stay its own
+conversation's. A request gets the tokens it would get alone (greedy on the CPU,
+the same bytes; `tests/serve_mux_check.py` checks it on every engine). A prompt is
+prefilled when it arrives, between two steps. With more than one slot, the
+engines other than GLM-5.2 and GLM-5.3-Flash draft nothing (MTP, DSpark and
+prompt lookup follow one conversation), and their Vulkan dense chain, DeltaNet on
+the GPU and Metal paths stay off; the Vulkan expert tier serves every
+conversation's experts.
## Web dashboard
diff --git a/docs/serve_protocol.md b/docs/serve_protocol.md
index e89ead31..3fdae93a 100644
--- a/docs/serve_protocol.md
+++ b/docs/serve_protocol.md
@@ -80,7 +80,12 @@ DECIDE <id> <slot> <bytes>\n<payload>\n
- `slot` β€” KV slot index, `0 … KV_SLOTS-1` (`KV_SLOTS` env, 1–16, default 1). A slot
holds one conversation's KV; the engine matches the tokenized payload against the
slot's history and reuses the common prefix (truncate-and-extend), so stateless
- HTTP turns keep their cache.
+ HTTP turns keep their cache. With `KV_SLOTS` above 1 every text engine takes a
+ `SUBMIT` on a free slot while others decode: its prompt is prefilled at once, and
+ from then on each decode step runs one row for every active request. The frames of
+ the requests interleave, each tagged with its `id`. A `SUBMIT` on a slot whose
+ request is still running gets `ERROR <id> SLOT_BUSY`; a slot out of range,
+ `ERROR <id> invalid cache slot`.
- `bytes` β€” exact byte length of `payload` (UTF-8, may contain newlines). The engine
reads exactly that many bytes after the header line, then one trailing `\n`.
- `payload` β€” the fully rendered prompt (the server owns the chat template).
diff --git a/docs/vulkan.md b/docs/vulkan.md
index 3a883e61..ed8a271a 100644
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@ -1509,6 +1509,14 @@ rebuilds the KDA state on the CPU from the input rows the chain records since th
copy was last current (embedding rows, or the vision tower's), a prefill's worth of CPU
work.
+**Several conversations at once (the other engines).** qwen36, qwen38, OLMoE, Inkling,
+Kimi K3, MiMo, DeepSeek V4 and V4.1 decode `KV_SLOTS`' conversations together too, but
+their dense chain keeps one conversation's state on the device, so it stays off with
+more than one slot (a `[VK] <engine>: KV_SLOTS=n: the dense chain is off` line says
+so): the dense part runs on the CPU, and the routed-expert tier serves the experts of
+every conversation's row. `tests/vulkan_engines.sh mux` checks each engine's frames
+against the same requests served alone, on the CPU and with the tier.
+
**Several conversations at once (colibri).** `KV_SLOTS`' batched decode (one row from
each active conversation, each at its own position) runs on the chain: the norms, the
projections and the MoE take the batch's rows as one; each row's new KV rows go to its