Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: publish open inference experiments

ZacharyZcR4de7f4c61fa3
diff --git a/README.it.md b/README.it.md
index 41d68584..88401717 100644
--- a/README.it.md
+++ b/README.it.md
@@ -75,23 +75,51 @@ lo rende possibile. Non noleggiare intelligenza dietro un'API, ma possederla,
analizzarla, misurarla e migliorarla. Il motore resta volutamente abbastanza piccolo
perché la prossima ottimizzazione utile possa arrivare da chiunque sia disposto a misurarla.
-## Punti di forza tecnici
+## Tecniche fondamentali e risultati misurati
- **Una gerarchia, non una soglia di memoria.** VRAM, RAM e NVMe sono livelli di
piazzamento degli stessi pesi; poca memoria veloce cambia la velocità, non il modello.
- **Un JIT per i pesi.** Il calore di routing misurato alimenta una LRU per layer,
un hot-store appreso e il prefetch del layer successivo senza caricare tutti gli expert.
+ Aiuta sui carichi ripetibili, ma la cronologia può sovradattarsi e il prefetch può
+ perdere su alcuni host: sono policy da misurare, non promesse.
- **L'I/O fa parte del motore.** Unione degli expert per batch, letture sovrapposte
- al calcolo, `O_DIRECT` e striping pesato su due SSD ottimizzano direttamente lo streaming.
+ al calcolo, `O_DIRECT` e striping pesato su due SSD ottimizzano direttamente lo
+ streaming. `O_DIRECT` dipende dal disco e il doppio SSD richiede più A/B end-to-end.
- **Esecuzione eterogenea.** CPU, CUDA, Metal, memoria NUMA e residenza parziale o
- completa degli expert condividono un runtime e si combinano in base alla macchina.
+ completa degli expert condividono un runtime; la combinazione utile dipende da
+ calcolo, banda, residenza e carico.
- **Stato compresso senza cambiare modello.** Validazione token-exact, stato MLA KV
57× più piccolo, conversazioni persistenti e DSA fedele vincolano l'ottimizzazione
- alla correttezza.
+ alla correttezza. Sono proprietà di memoria, latenza e correttezza, non una
+ promessa generale di throughput.
- **La speculazione deve meritarsi il costo.** MTP nativo e draft vincolati da
grammatica sono misurati end-to-end e si disattivano quando l'accettazione non
ripaga la verifica.
+## Ipotesi aperte, esperimenti e partecipazione
+
+Colibrì considera ogni ottimizzazione un'ipotesi finché un A/B end-to-end
+controllato non dimostra il contrario. Le domande principali sono:
+
+| ipotesi | evidenza attuale | esperimento ancora necessario |
+|---|---|---|
+| La cronologia di routing può piazzare gli expert meglio di una semplice LRU | i pin appresi migliorano carichi ripetuti, ma possono sovradattarsi al prompt | A/B cross-session su set esclusi: codice, chat, multilingua e contesti lunghi |
+| Più SSD possono trasformare banda indipendente in velocità di decode | routing pesato mirror/split implementato e validato; il modello di banda è solido | GLM-5.2 a cache fredda, uno contro due dischi su controller indipendenti reali |
+| Un planner hardware-aware può avvicinarsi automaticamente alla configurazione migliore | oggi rileva budget RAM/VRAM e diversi backend | confrontare il piano generato con sweep controllati su laptop, workstation, NUMA e multi-GPU |
+| Rappresentazioni lossless o a qualità limitata possono ridurre abbastanza il movimento dei pesi | esistono ablation di formato e quantizzazione con gate di qualità | riprodurre insieme qualità, byte mossi, latenza e costo per token utile, non solo il rapporto di compressione |
+| La speculazione routing-aware può convenire prima della residenza quasi completa | MTP e draft grammaticali funzionano, ma MTP ha anche perso il 32% intorno all'85% di expert hit | mappare il pareggio tra accettazione, hit rate, batch union e profondità del draft |
+| La sovrapposizione CPU/GPU può nascondere trasferimenti e sincronizzazione | esistono risultati positivi CUDA e Metal, ma CPU veloci e bassa residenza possono annullarli | profili per fase e A/B a variabile singola su PCIe, memoria unificata e piena residenza |
+
+Per contribuire, scegli una riga e pubblica anche i risultati negativi. Registra
+hardware, commit, container del modello, comando esatto, prompt, stato cache,
+throughput, TTFT, expert hit, byte letti e controllo qualità; cambia una sola
+variabile, ripeti e allega i log grezzi. Parti da
+[CONTRIBUTING.md](CONTRIBUTING.md), confronta il
+[protocollo di benchmark](docs/benchmarks.md), quindi
+[apri una issue di esperimento](https://github.com/JustVugg/colibri/issues/new).
+Un fallimento controllato vale più di un numero veloce senza spiegazione.
+
## L'idea
Un modello Mixture-of-Experts da 744B attiva solo ~40B parametri per token — e
diff --git a/README.md b/README.md
index d8a66229..5d965062 100644
--- a/README.md
+++ b/README.md
@@ -81,23 +81,51 @@ code that does it. Not renting intelligence behind an API — *holding* it:
probing it, measuring it, improving it. The engine is deliberately small enough
that the next useful optimization can come from anyone willing to measure it.
-## Technical highlights
+## Core techniques and measured findings
- **One hierarchy, not one memory threshold.** VRAM, RAM, and NVMe are placement
tiers for the same weights; limited fast memory changes speed, not model semantics.
- **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learned
pinned hot-store, and one-layer-ahead prefetch instead of loading every expert.
+ It wins on repeatable workloads; history can overfit, and lookahead can lose on
+ some hosts, so both remain measurable policies rather than promises.
- **I/O is part of the engine.** Batched expert unions, overlapped reads and
compute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path
- rather than pretending storage latency is free.
+ rather than pretending storage latency is free. `O_DIRECT` is drive-dependent,
+ and dual-SSD still needs broader end-to-end community A/Bs.
- **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full
- expert residency share one runtime and can be combined according to the machine.
+ expert residency share one runtime and can be combined according to the machine;
+ the profitable combination depends on compute, bandwidth, residency, and workload.
- **Compressed state without a different model.** Token-exact forward validation,
57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep
- optimization tied to correctness.
+ optimization tied to correctness. These are memory, latency, and correctness
+ properties — not a blanket throughput claim.
- **Speculation that must earn its keep.** Native MTP and grammar-forced drafts
are measured end to end and can be disabled when acceptance does not repay verification.
+## Open hypotheses, experiments, and how to help
+
+Colibrì treats an optimization as a hypothesis until a controlled end-to-end A/B
+shows otherwise. These are the main questions now:
+
+| hypothesis | evidence so far | experiment still needed |
+|---|---|---|
+| Routing history can place experts better than plain LRU | learned pins improve repeated workloads, but can overfit a prompt | held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads |
+| Multiple SSDs can turn independent bandwidth into decode speed | weighted mirror/split routing is implemented and validated; the bandwidth model is sound | cold-cache one-drive vs two-drive GLM-5.2 runs on real, independent controllers |
+| A hardware-aware planner can approach each machine's best configuration automatically | RAM/VRAM budgets and several backends are detected today | compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems |
+| Lossless or quality-bounded representations can reduce weight movement enough to matter | format and quantization ablations exist, with correctness/quality gates | reproduce quality, bytes moved, latency, and cost per useful token together — not compression ratio alone |
+| Routing-aware speculation can pay before near-full residency | MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit | map the break-even surface across acceptance, expert hit rate, batch union, and draft depth |
+| CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck | CUDA and Metal wins exist, but fast CPUs and low residency can erase them | per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines |
+
+Want to help? Pick one row and publish the negative results too. Record the
+hardware, commit, model/container, exact command, prompt, cache state, throughput,
+TTFT, expert hit rate, bytes read, and quality check; change one variable, repeat
+the run, and attach raw logs. Start with
+[CONTRIBUTING.md](CONTRIBUTING.md), compare against
+[the benchmark protocol](docs/benchmarks.md), then
+[open an experiment issue](https://github.com/JustVugg/colibri/issues/new).
+A well-controlled failure is more valuable here than an unexplained fast number.
+
## The idea
A 744B Mixture-of-Experts model activates only ~40B parameters per token — and
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 0d96e903..3c72c3ab 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -62,21 +62,44 @@ VRAM/RAM/磁盘层级条,以及角落的实时迷你大脑。</em></p>
不是从 API 租用智能,而是持有、探测、测量和改进它。引擎刻意保持足够小,让任何愿意测量
的人都可能贡献下一项有效优化。
-## 技术亮点
+## 核心技术与实测结论
- **统一层级,而非单一内存门槛。**VRAM、RAM 与 NVMe 是同一份权重的不同放置层级;
高速内存不足只影响速度,不改变模型语义。
- **权重的 JIT。**实测路由热度驱动逐层 LRU、学习型热门专家固定区和提前一层的预取,
- 无需加载所有专家。
+ 无需加载所有专家。它在可重复负载上有收益,但历史可能过拟合,预取在部分主机上也可能
+ 负优化,因此它们是需要测量的策略,不是性能承诺。
- **I/O 本身就是引擎的一部分。**专家批次并集、读算重叠、`O_DIRECT` 与加权双 SSD
- 分流直接优化流式路径,而不是假装存储延迟不存在。
+ 分流直接优化流式路径,而不是假装存储延迟不存在。`O_DIRECT` 取决于磁盘,双 SSD
+ 仍需要更多社区端到端 A/B。
- **异构执行。**CPU、CUDA、Metal、NUMA 内存以及专家的部分或全部常驻共用一个运行时,
- 可按机器条件组合。
+ 可按机器条件组合;最佳组合取决于算力、带宽、驻留率与负载。
- **压缩状态,不篡改模型。**逐 token 精确的前向验证、缩小 57 倍的 MLA KV 状态、
- 持久化热会话与忠实 DSA,让优化始终受正确性约束。
+ 持久化热会话与忠实 DSA,让优化始终受正确性约束。这些是内存、延迟和正确性属性,
+ 不是笼统的吞吐承诺。
- **必须证明收益的推测解码。**原生 MTP 与语法强制草稿均接受端到端测量;
接受率无法覆盖验证成本时可以关闭。
+## 开放猜想、实验与参与方式
+
+在受控的端到端 A/B 证明之前,Colibrì 将每项优化都视为猜想。当前主要问题如下:
+
+| 猜想 | 当前证据 | 仍需完成的实验 |
+|---|---|---|
+| 路由历史能比普通 LRU 更好地放置专家 | 学习型固定区能改善重复负载,但也会对 prompt 过拟合 | 在代码、对话、多语言和长上下文负载上做留出集、跨会话 A/B |
+| 多块 SSD 能将独立带宽转化为解码速度 | 加权镜像/分片路由已实现并通过校验,带宽模型成立 | 在独立控制器的真实磁盘上做冷缓存、单盘与双盘 GLM-5.2 对照 |
+| 硬件感知规划器能自动接近每台机器的最优配置 | 当前已检测 RAM/VRAM 预算与多个后端 | 将自动方案与参数扫描对比,覆盖笔记本、工作站、NUMA 和多 GPU 主机 |
+| 无损或质量受控的表示能充分减少权重搬运 | 已有格式与量化消融,并设置正确性/质量门槛 | 同时复现质量、搬运字节、延迟和每个有效 token 成本,而非只看压缩率 |
+| 路由感知推测能在接近全驻留前盈利 | MTP 与语法草稿可用,但 MTP 在约 85% expert hit 时也实测过 -32% | 绘制接受率、命中率、批次并集与草稿深度的盈亏边界 |
+| CPU/GPU 重叠能隐藏传输与同步,而非仅转移瓶颈 | CUDA 与 Metal 有成功数据,但强 CPU 和低驻留率会抹平收益 | 在 PCIe、统一内存与全驻留机器上做逐阶段 profile 和单变量 A/B |
+
+想参与就任选一行,负结果也请公开。请记录硬件、commit、模型容器、完整命令、prompt、
+缓存状态、吞吐、TTFT、expert hit、读取字节数与质量检查;每次只改一个变量,重复运行并附上
+原始日志。先阅读 [CONTRIBUTING.md](CONTRIBUTING.md) 和
+[benchmark 协议](docs/benchmarks.md),然后
+[创建实验 issue](https://github.com/JustVugg/colibri/issues/new)。
+在这里,一个受控的失败比一个无法解释的高数字更有价值。
+
## 核心概念
744B 的专家混合(Mixture-of-Experts)模型,每个 token 只会激活约 40B 参数——
diff --git a/README.zh-TW.md b/README.zh-TW.md
index ac679d2b..34222e0d 100644
--- a/README.zh-TW.md
+++ b/README.zh-TW.md
@@ -62,21 +62,44 @@ VRAM/RAM/硬碟層級長條,以及角落的即時迷你大腦。</em></p>
不是從 API 租用智慧,而是持有、探測、測量和改進它。引擎刻意維持足夠小,讓任何願意測量
的人都可能貢獻下一項有效最佳化。
-## 技術亮點
+## 核心技術與實測結論
- **統一階層,而非單一記憶體門檻。**VRAM、RAM 與 NVMe 是同一份權重的不同配置階層;
高速記憶體不足只影響速度,不改變模型語意。
- **權重的 JIT。**實測路由熱度驅動逐層 LRU、學習型熱門專家固定區和提前一層的預先載入,
- 無需載入所有專家。
+ 無需載入所有專家。它在可重複負載上有收益,但歷史可能過度擬合,預先載入在部分主機上
+ 也可能負最佳化,因此它們是需要測量的策略,不是效能承諾。
- **I/O 本身就是引擎的一部分。**專家批次聯集、讀算重疊、`O_DIRECT` 與加權雙 SSD
- 分流直接最佳化串流路徑,而不是假裝儲存延遲不存在。
+ 分流直接最佳化串流路徑,而不是假裝儲存延遲不存在。`O_DIRECT` 取決於磁碟,雙 SSD
+ 仍需要更多社群端到端 A/B。
- **異質執行。**CPU、CUDA、Metal、NUMA 記憶體以及專家的部分或全部常駐共用一個執行環境,
- 可依機器條件組合。
+ 可依機器條件組合;最佳組合取決於算力、頻寬、常駐率與負載。
- **壓縮狀態,不竄改模型。**逐 token 精確的前向驗證、縮小 57 倍的 MLA KV 狀態、
- 持久化熱會話與忠實 DSA,讓最佳化始終受正確性約束。
+ 持久化熱會話與忠實 DSA,讓最佳化始終受正確性約束。這些是記憶體、延遲和正確性屬性,
+ 不是籠統的吞吐承諾。
- **必須證明收益的推測解碼。**原生 MTP 與文法強制草稿均接受端到端測量;
接受率無法覆蓋驗證成本時可以關閉。
+## 開放猜想、實驗與參與方式
+
+在受控的端到端 A/B 證明之前,Colibrì 將每項最佳化都視為猜想。目前主要問題如下:
+
+| 猜想 | 目前證據 | 仍需完成的實驗 |
+|---|---|---|
+| 路由歷史能比普通 LRU 更好地配置專家 | 學習型固定區能改善重複負載,但也會對 prompt 過度擬合 | 在程式碼、對話、多語言和長上下文負載上做留出集、跨會話 A/B |
+| 多顆 SSD 能將獨立頻寬轉化為解碼速度 | 加權鏡像/分片路由已實作並通過驗證,頻寬模型成立 | 在獨立控制器的真實磁碟上做冷快取、單碟與雙碟 GLM-5.2 對照 |
+| 硬體感知規劃器能自動接近每台機器的最佳配置 | 目前已偵測 RAM/VRAM 預算與多個後端 | 將自動方案與參數掃描對比,涵蓋筆電、工作站、NUMA 和多 GPU 主機 |
+| 無損或品質受控的表示能充分減少權重搬運 | 已有格式與量化消融,並設置正確性/品質門檻 | 同時重現品質、搬運位元組、延遲和每個有效 token 成本,而非只看壓縮率 |
+| 路由感知推測能在接近全常駐前獲利 | MTP 與文法草稿可用,但 MTP 在約 85% expert hit 時也實測過 -32% | 繪製接受率、命中率、批次聯集與草稿深度的盈虧邊界 |
+| CPU/GPU 重疊能隱藏傳輸與同步,而非僅轉移瓶頸 | CUDA 與 Metal 有成功數據,但強 CPU 和低常駐率會抹平收益 | 在 PCIe、統一記憶體與全常駐機器上做逐階段 profile 和單變數 A/B |
+
+想參與就任選一行,負結果也請公開。請記錄硬體、commit、模型容器、完整指令、prompt、
+快取狀態、吞吐、TTFT、expert hit、讀取位元組數與品質檢查;每次只改一個變數,重複執行並附上
+原始日誌。先閱讀 [CONTRIBUTING.md](CONTRIBUTING.md) 和
+[benchmark 協議](docs/benchmarks.md),然後
+[建立實驗 issue](https://github.com/JustVugg/colibri/issues/new)。
+在這裡,一個受控的失敗比一個無法解釋的高數字更有價值。
+
## 核心概念
744B 的專家混合(Mixture-of-Experts)模型,每個 token 只會啟用約 40B 參數——