Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
Tuning & runtime knobs
Everything here is opt-in; the defaults are chosen so a plain ./coli chat
is safe on any machine. See also SETTINGS.md and
ENVIRONMENT.md for the full variable inventory.
The knobs that matter most
| knob | what it does |
|---|---|
--temp T |
token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy) |
--topp 0.7 |
adaptive expert top-p (30–40% less disk; lossy — prints a warning) |
--ngen N |
max tokens per answer (:more in chat continues a truncated one) |
--repin N |
adapt RAM/VRAM hot experts every N emitted tokens |
RAM_GB=<n> |
claim more RAM for the expert cache than the conservative auto-detect |
PIN=stats PIN_GB=g |
pin the hottest experts from a measured usage profile |
DRAFT=n |
MTP draft depth (0 disables speculation) |
GRAMMAR=g.gbnf |
grammar-forced drafts for constrained JSON/NDJSON output (docs) |
THINK=1 |
enable GLM-5.2's reasoning block |
PILOT=1 |
router-lookahead disk prefetch (see below) |
URING=1 |
Linux-only batched expert I/O (implies PIPE=1) |
PIPE=0 |
disable the async expert-load pool (default ON — overlaps pread with matmul, −18% disk service) |
DIRECT=1 |
O_DIRECT expert reads (measured +65% alone on a Strix Halo, #200) |
COLI_NUMA=1 |
interleave resident weights across NUMA nodes on multi-socket hosts (#82) |
CACHE_ROUTE=1 |
cache-aware max-rank routing (opt-in, #199) |
AUTOPIN=0 |
disable the learning cache's auto-pin |
CAP_RAISE=0 |
don't auto-grow the expert cache |
KVSAVE=0 |
disable KV-cache persistence |
TF=1 |
teacher-forcing validation |
Automatic history pinning and the adaptive LRU share the same expert RAM
budget. Colibri caps automatic pinning to preserve the no-pin LRU capacity;
explicit PIN and PIN_GB settings remain authoritative.
Resource policy
coli plan reports the planned hot (VRAM), warm (RAM), and cold backing (disk)
tiers, the reason for each placement, and the expected bottleneck. The default
--policy quality and --policy balanced modes preserve checkpoint quantization
and router decisions unless --topk or --topp is passed; those explicit lossy
overrides print a warning and proceed.
The plan also emits a machine-readable next_actions list and renders it in
the terminal. A missing storage measurement is marked required; subsequent
profiling and tuning work is recommended. These actions produce the evidence
needed to improve a plan on this machine—they do not silently enable an
unmeasured optimization.
Auto-tier plans size OpenMP from physical cores and bind workers across cores.
Memory-bound quantized kernels can regress sharply when SMT siblings compete for
limited memory channels. The GLM, Kimi K3, and OLMoE engines also apply that
physical-core cap when launched directly; explicit OMP_NUM_THREADS and the
COLI_NO_OMP_TUNE kill switch always take precedence.
Note (#471): exporting
OMP_PROC_BIND/OMP_PLACESused to interact badly with the engine's one-time OpenMP tuning re-exec on Linux — the re-exec'd image inherited the first image's place-0 thread binding and the whole team landed on one core (~20× slowdown). The engine now resets its affinity to all online CPUs right before the re-exec, so explicitOMP_*pinning works as documented.COLI_OMP_TUNED=1remains the escape hatch that skips the re-exec entirely.
Hybrid CUDA/CPU OpenMP override
The main GLM engine intentionally skips its active-wait OpenMP tuning when
COLI_CUDA is set. Active worker teams have measured severe regressions by
contending with CUDA dispatch and synchronization, so a CUDA model that still
routes some experts through RAM is not, by itself, a reason to change that
default.
On a specific hybrid host where profiling shows the CPU expert window on the critical path, the supported experiment is an explicit user-owned OpenMP policy:
OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 KMP_BLOCKTIME=200 \
OMP_PROC_BIND=close OMP_DYNAMIC=FALSE \
COLI_CUDA=1 ./coli run --model /models/glm52_i4 "Benchmark prompt"
The OpenMP runtime reads these variables before main(), so they must be set
on the engine invocation rather than exported after startup. The engine uses
overwrite=0; explicit values remain authoritative. GOMP_SPINCOUNT applies
to libgomp and KMP_BLOCKTIME to Intel/LLVM OpenMP runtimes, so carrying both
keeps the command portable across common builds.
Treat this as a measured per-host override, not a recommended CUDA default.
Compare it against the unchanged command with stable page-cache state and an
interleaved run order; report CPU expert time, GPU critical time, disk wait,
and end-to-end throughput. Reject it if GPU time grows or the CPU window was
already hidden behind the GPU. Do not use active waiting on Apple Silicon: CPU
spin has measured slower there by stealing the shared CPU/GPU power budget.
COLI_NO_OMP_TUNE=1 remains the explicit passive-policy kill switch.
coli plan --model /models/glm52_i4 --policy quality
coli run --auto-tier --policy quality "Explain MoE offloading"
# Explicit research-only router reduction:
coli run --policy experimental-fast --topk 4 "Benchmark prompt"
Measured machine profiles
coli plan chooses a safe starting point from capacity and topology.
coli tune measures the remaining scheduling choices on the actual model and
machine, then saves a hardware/model/engine-specific profile:
coli tune --model /models/glm52_i4
coli run --model /models/glm52_i4 --auto-tier "Explain MoE offloading"
GLM generates one calibration continuation and teacher-forces it for every candidate. The sibling engines use their common serving protocol instead: one engine stays alive for all requests of a candidate, and deterministic prompts rotate in a fixed order. That preserves the expert cache and measures a real mixed chat instead of repeatedly loading one process or teaching the cache one exact prompt. Greedy output is compared per prompt across every candidate; any byte drift disqualifies the scheduling change.
The bounded sweep includes execution knobs such as OpenMP thread count, NUMA
placement, I/O overlap, direct I/O, CUDA pipelining, and DeepSeek V4 expert
loader lanes. When cold experts remain on disk it also tests 75% and 50% of
the planner's expert-cache allowance. Dense weights, KV/workspace reservations,
and the OS reserve are never reduced, and no candidate may exceed the planner's
safe baseline. GLM/Inkling/OLMoE/Qwen receive the measured slots-per-layer cap;
V4 receives a whole-process RAM_GB ceiling; Kimi receives K3_EXPERT_GB under
its unchanged whole-process ceiling. The sweep never changes weights,
quantization, router decisions, TOPK, TOPP, or sampling. Add repeatable
--rotate-prompt options to replace the secondary built-in workload prompt.
A candidate is saved only when median throughput improves by at least 3% while
expert hit rate remains within 0.5 percentage points and reported TTFT/p99
latency stay within 20% of the baseline. The winner is then rerun before a final
baseline; this reverse-order gate gives the baseline any remaining warm-cache
advantage and rejects startup drift. Otherwise the baseline is recorded and no
override is applied. Saved profiles are loaded by --auto-tier; --ram,
--cap, RAM_GB, K3_EXPERT_GB, and other explicit settings always win. A
resource winner is rechecked against the current plan at every launch, so a
profile measured with more free RAM is ignored rather than overcommitted later.
Use --no-tune-profile to bypass a saved profile.
Profiles live under $XDG_CONFIG_HOME/colibri/tuning (normally
~/.config/colibri/tuning) or %LOCALAPPDATA%\colibri\tuning on Windows. A
change to the engine binary, model metadata, CPU topology, or GPU inventory
produces a new fingerprint instead of reusing stale measurements.
Disk is an immutable recovery source, not a normal decode target. If the plan leaves cold expert bytes on disk, speed depends on cache hit rate; output quality does not.
Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute
while missing experts are loaded in a bounded background I/O pool, then the cold
results join before the layer completes. The pool engages only under PIPE=1;
PIPE_WORKERS=n sets its worker count (default 8). Profiling reports both disk
service time and the smaller foreground-visible wait time so overlap is explicit.
--policy balanced enables lossless live placement (REPIN=64). At safe request
boundaries, a per-layer LFRU score combines decaying session frequency with recent
access and replaces at most four sufficiently colder pinned experts. --policy quality leaves live replacement off by default; REPIN=0 always disables it.
The learning cache
The engine records which experts your usage actually routes to (.coli_usage
next to the model, updated every turn) and at startup automatically pins the
hottest ones in spare RAM — colibrì literally gets faster the more you use it.
PIN=auto seeds the pin directly from the live usage history
(#301).
The expert cache auto-sizes to your RAM (since 2026-07-10): the engine
raises the LRU cap to fill your --ram budget instead of only lowering it.
If you benchmarked colibrì before that date, rerun — your numbers were capped.
Live tier adaptation (--repin N, opt-in): at safe turn boundaries, a
decaying session heat map replaces cold pinned experts with hotter streamed
experts. A 25% hysteresis and a four-swap limit prevent tier thrashing.
Persistent .coli_usage remains the long-term signal and is not decayed.
The history's on-disk format, what happens when one engine is handed another
engine's history, and how PIN=<file> differs from PIN=auto in how much it
trusts a file are documented in
routing-telemetry.md.
Router-lookahead prefetch (PILOT=1, experimental)
GLM-5.2's expert routing is measurably predictable ahead of time — applying
layer L+1's router to layer L's post-attention state recalls 71.6% of the
true top-8 (vs 41.3% for "same experts as last token"). PILOT=1 issues
next-layer expert readahead from a dedicated I/O thread while the current layer
computes. PILOT_REAL=1 moves the prefetched loads off the critical path
(measured +11pp hit rate on a big-cache host), and PILOT_TWO=1 folds the
computed shared-expert into the prediction (+3% recall,
#200). On disk-saturated
hosts hint-only PILOT can be net negative — measure on yours.
Speculation and reproducibility
Speculative decoding requires that the draft and verify paths compute the same
function — SPEC_PIN=1 (default since #294)
pins every forward issued while drafts are live to the platform's S=1 kernel
family. For byte-exact reproducibility across runs: DRAFT=0, plus IDOT=0 COLI_CUDA=0 if you also want kernel-family/GPU independence. Acceptance
percentages are not comparable across engine versions under --topp
(#163 has the full story).
GLM-5.2's native MTP head drafts tokens that the main model verifies in one
batched forward: 2.2–2.8 tokens/forward when it pays. The MTP head must be
int8: int4 heads collapse to 0–4% acceptance
(#8). Grammar-forced drafts
(GRAMMAR=file.gbnf) add nearly free acceptance on
constrained JSON output. Whether speculation is a net win depends on your cache
temperature: measure, and use DRAFT=0 when it does not pay.
Verify batches can also opt into an exact attention core with
COLI_EXACT_VERIFY=1 (#689):
the CPU MLA-absorb score and context dots accumulate integer products and round
once, so a near-tie in a verify row resolves the same way on every host, at
roughly 0.6x tok/s on a tiny oracle (the dot itself is ~5–7x the float loop).
Two limits to know: with a quantised KV cache (tq1, TQ or int8 KV) the
context dot keeps the float path, so exactness there is not provided; and a
real near-tie flip has only been argued, not yet caught on GLM-5.2 at n=64.
Approximate mode: DEGRADE_ZERO (opt-in, OLMoE-calibrated)
DEGRADE_ZERO=1 enables an opt-in degraded inference policy: when a prefetch
deadline is missed, experts whose per-position gate weight falls below
DEGRADE_TAU (default 0.03) are zero-filled instead of loaded from disk.
The slot contributes nothing to the layer output; the approximation is the
dropped mass, not a rescaled version of it (renorm is catastrophically worse —
see issue #865 for the measured A/B).
This reduces blocking disk reads on NVMe-bound workloads at the cost of a small quality hit. Measured on OLMoE-1B-7B:
DEGRADE_TAU |
slots zeroed | ppl delta |
|---|---|---|
| 0.03 | ~22% | +2.9% |
| 0.05 | ~60% | +41% |
These numbers are OLMoE-specific. GLM-5.2 (norm_topk=1) and Kimi K3 have
different router contracts and expert counts — their operating points have not
been measured. Until they are, treat tau=0.03 as a starting point and verify
quality on your model before relying on it.
These numbers assume a warm expert cache. Cold-start sessions — where the cache begins empty and all experts miss initially — will see higher drop rates until the LRU fills. The steady-state perplexity delta above is what was measured; cold-start transient behavior has not been separately characterized.
The feature is decode-only (S≤4 guard, same as EXPERT_BUDGET): dropping
experts during prefill corrupts the KV cache. A rescue rule ensures no token
position is left with zero routed experts. Resident (pinned or LRU-cached)
experts are never dropped regardless of weight.
The [PROF] footer reports the total zeroed slot count and the top-3 layers by
drop share when the flag is active, so a miscalibrated tau is visible rather
than silent.
DEGRADE_ZERO=1 DEGRADE_TAU=0.03 COLI_MODEL=/nvme/glm52_i4 ./coli chat
See ENVIRONMENT.md for the full variable reference and issue #865 for the measurement methodology.
Conversations reopen warm
coli chat persists the compressed MLA KV-cache to disk after every turn
(.coli_kv, ~182 KB/token, appended incrementally, crash-safe). Close the chat,
reopen it tomorrow — the model still remembers the whole conversation and zero
re-prefill happens: validated byte-identical to an uninterrupted session.
:reset clears it, KVSAVE=0 disables it.