Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: publish open inference experiments
ZacharyZcR4de7f4c61fa3
diff --git a/README.it.md b/README.it.mdindex 41d68584..88401717 100644--- a/README.it.md+++ b/README.it.md@@ -75,23 +75,51 @@ lo rende possibile. Non noleggiare intelligenza dietro un'API, ma possederla,analizzarla, misurarla e migliorarla. Il motore resta volutamente abbastanza piccoloperché la prossima ottimizzazione utile possa arrivare da chiunque sia disposto a misurarla.-## Punti di forza tecnici+## Tecniche fondamentali e risultati misurati- **Una gerarchia, non una soglia di memoria.** VRAM, RAM e NVMe sono livelli dipiazzamento degli stessi pesi; poca memoria veloce cambia la velocità, non il modello.- **Un JIT per i pesi.** Il calore di routing misurato alimenta una LRU per layer,un hot-store appreso e il prefetch del layer successivo senza caricare tutti gli expert.+ Aiuta sui carichi ripetibili, ma la cronologia può sovradattarsi e il prefetch può+ perdere su alcuni host: sono policy da misurare, non promesse.- **L'I/O fa parte del motore.** Unione degli expert per batch, letture sovrapposte- al calcolo, `O_DIRECT` e striping pesato su due SSD ottimizzano direttamente lo streaming.+ al calcolo, `O_DIRECT` e striping pesato su due SSD ottimizzano direttamente lo+ streaming. `O_DIRECT` dipende dal disco e il doppio SSD richiede più A/B end-to-end.- **Esecuzione eterogenea.** CPU, CUDA, Metal, memoria NUMA e residenza parziale o- completa degli expert condividono un runtime e si combinano in base alla macchina.+ completa degli expert condividono un runtime; la combinazione utile dipende da+ calcolo, banda, residenza e carico.- **Stato compresso senza cambiare modello.** Validazione token-exact, stato MLA KV57× più piccolo, conversazioni persistenti e DSA fedele vincolano l'ottimizzazione- alla correttezza.+ alla correttezza. Sono proprietà di memoria, latenza e correttezza, non una+ promessa generale di throughput.- **La speculazione deve meritarsi il costo.** MTP nativo e draft vincolati dagrammatica sono misurati end-to-end e si disattivano quando l'accettazione nonripaga la verifica.+## Ipotesi aperte, esperimenti e partecipazione++Colibrì considera ogni ottimizzazione un'ipotesi finché un A/B end-to-end+controllato non dimostra il contrario. Le domande principali sono:++| ipotesi | evidenza attuale | esperimento ancora necessario |+|---|---|---|+| La cronologia di routing può piazzare gli expert meglio di una semplice LRU | i pin appresi migliorano carichi ripetuti, ma possono sovradattarsi al prompt | A/B cross-session su set esclusi: codice, chat, multilingua e contesti lunghi |+| Più SSD possono trasformare banda indipendente in velocità di decode | routing pesato mirror/split implementato e validato; il modello di banda è solido | GLM-5.2 a cache fredda, uno contro due dischi su controller indipendenti reali |+| Un planner hardware-aware può avvicinarsi automaticamente alla configurazione migliore | oggi rileva budget RAM/VRAM e diversi backend | confrontare il piano generato con sweep controllati su laptop, workstation, NUMA e multi-GPU |+| Rappresentazioni lossless o a qualità limitata possono ridurre abbastanza il movimento dei pesi | esistono ablation di formato e quantizzazione con gate di qualità | riprodurre insieme qualità, byte mossi, latenza e costo per token utile, non solo il rapporto di compressione |+| La speculazione routing-aware può convenire prima della residenza quasi completa | MTP e draft grammaticali funzionano, ma MTP ha anche perso il 32% intorno all'85% di expert hit | mappare il pareggio tra accettazione, hit rate, batch union e profondità del draft |+| La sovrapposizione CPU/GPU può nascondere trasferimenti e sincronizzazione | esistono risultati positivi CUDA e Metal, ma CPU veloci e bassa residenza possono annullarli | profili per fase e A/B a variabile singola su PCIe, memoria unificata e piena residenza |++Per contribuire, scegli una riga e pubblica anche i risultati negativi. Registra+hardware, commit, container del modello, comando esatto, prompt, stato cache,+throughput, TTFT, expert hit, byte letti e controllo qualità; cambia una sola+variabile, ripeti e allega i log grezzi. Parti da+[CONTRIBUTING.md](CONTRIBUTING.md), confronta il+[protocollo di benchmark](docs/benchmarks.md), quindi+[apri una issue di esperimento](https://github.com/JustVugg/colibri/issues/new).+Un fallimento controllato vale più di un numero veloce senza spiegazione.+## L'ideaUn modello Mixture-of-Experts da 744B attiva solo ~40B parametri per token — ediff --git a/README.md b/README.mdindex d8a66229..5d965062 100644--- a/README.md+++ b/README.md@@ -81,23 +81,51 @@ code that does it. Not renting intelligence behind an API — *holding* it:probing it, measuring it, improving it. The engine is deliberately small enoughthat the next useful optimization can come from anyone willing to measure it.-## Technical highlights+## Core techniques and measured findings- **One hierarchy, not one memory threshold.** VRAM, RAM, and NVMe are placementtiers for the same weights; limited fast memory changes speed, not model semantics.- **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learnedpinned hot-store, and one-layer-ahead prefetch instead of loading every expert.+ It wins on repeatable workloads; history can overfit, and lookahead can lose on+ some hosts, so both remain measurable policies rather than promises.- **I/O is part of the engine.** Batched expert unions, overlapped reads andcompute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path- rather than pretending storage latency is free.+ rather than pretending storage latency is free. `O_DIRECT` is drive-dependent,+ and dual-SSD still needs broader end-to-end community A/Bs.- **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full- expert residency share one runtime and can be combined according to the machine.+ expert residency share one runtime and can be combined according to the machine;+ the profitable combination depends on compute, bandwidth, residency, and workload.- **Compressed state without a different model.** Token-exact forward validation,57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep- optimization tied to correctness.+ optimization tied to correctness. These are memory, latency, and correctness+ properties — not a blanket throughput claim.- **Speculation that must earn its keep.** Native MTP and grammar-forced draftsare measured end to end and can be disabled when acceptance does not repay verification.+## Open hypotheses, experiments, and how to help++Colibrì treats an optimization as a hypothesis until a controlled end-to-end A/B+shows otherwise. These are the main questions now:++| hypothesis | evidence so far | experiment still needed |+|---|---|---|+| Routing history can place experts better than plain LRU | learned pins improve repeated workloads, but can overfit a prompt | held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads |+| Multiple SSDs can turn independent bandwidth into decode speed | weighted mirror/split routing is implemented and validated; the bandwidth model is sound | cold-cache one-drive vs two-drive GLM-5.2 runs on real, independent controllers |+| A hardware-aware planner can approach each machine's best configuration automatically | RAM/VRAM budgets and several backends are detected today | compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems |+| Lossless or quality-bounded representations can reduce weight movement enough to matter | format and quantization ablations exist, with correctness/quality gates | reproduce quality, bytes moved, latency, and cost per useful token together — not compression ratio alone |+| Routing-aware speculation can pay before near-full residency | MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit | map the break-even surface across acceptance, expert hit rate, batch union, and draft depth |+| CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck | CUDA and Metal wins exist, but fast CPUs and low residency can erase them | per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines |++Want to help? Pick one row and publish the negative results too. Record the+hardware, commit, model/container, exact command, prompt, cache state, throughput,+TTFT, expert hit rate, bytes read, and quality check; change one variable, repeat+the run, and attach raw logs. Start with+[CONTRIBUTING.md](CONTRIBUTING.md), compare against+[the benchmark protocol](docs/benchmarks.md), then+[open an experiment issue](https://github.com/JustVugg/colibri/issues/new).+A well-controlled failure is more valuable here than an unexplained fast number.+## The ideaA 744B Mixture-of-Experts model activates only ~40B parameters per token — anddiff --git a/README.zh-CN.md b/README.zh-CN.mdindex 0d96e903..3c72c3ab 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -62,21 +62,44 @@ VRAM/RAM/磁盘层级条,以及角落的实时迷你大脑。</em></p>不是从 API 租用智能,而是持有、探测、测量和改进它。引擎刻意保持足够小,让任何愿意测量的人都可能贡献下一项有效优化。-## 技术亮点+## 核心技术与实测结论- **统一层级,而非单一内存门槛。**VRAM、RAM 与 NVMe 是同一份权重的不同放置层级;高速内存不足只影响速度,不改变模型语义。- **权重的 JIT。**实测路由热度驱动逐层 LRU、学习型热门专家固定区和提前一层的预取,- 无需加载所有专家。+ 无需加载所有专家。它在可重复负载上有收益,但历史可能过拟合,预取在部分主机上也可能+ 负优化,因此它们是需要测量的策略,不是性能承诺。- **I/O 本身就是引擎的一部分。**专家批次并集、读算重叠、`O_DIRECT` 与加权双 SSD- 分流直接优化流式路径,而不是假装存储延迟不存在。+ 分流直接优化流式路径,而不是假装存储延迟不存在。`O_DIRECT` 取决于磁盘,双 SSD+ 仍需要更多社区端到端 A/B。- **异构执行。**CPU、CUDA、Metal、NUMA 内存以及专家的部分或全部常驻共用一个运行时,- 可按机器条件组合。+ 可按机器条件组合;最佳组合取决于算力、带宽、驻留率与负载。- **压缩状态,不篡改模型。**逐 token 精确的前向验证、缩小 57 倍的 MLA KV 状态、- 持久化热会话与忠实 DSA,让优化始终受正确性约束。+ 持久化热会话与忠实 DSA,让优化始终受正确性约束。这些是内存、延迟和正确性属性,+ 不是笼统的吞吐承诺。- **必须证明收益的推测解码。**原生 MTP 与语法强制草稿均接受端到端测量;接受率无法覆盖验证成本时可以关闭。+## 开放猜想、实验与参与方式++在受控的端到端 A/B 证明之前,Colibrì 将每项优化都视为猜想。当前主要问题如下:++| 猜想 | 当前证据 | 仍需完成的实验 |+|---|---|---|+| 路由历史能比普通 LRU 更好地放置专家 | 学习型固定区能改善重复负载,但也会对 prompt 过拟合 | 在代码、对话、多语言和长上下文负载上做留出集、跨会话 A/B |+| 多块 SSD 能将独立带宽转化为解码速度 | 加权镜像/分片路由已实现并通过校验,带宽模型成立 | 在独立控制器的真实磁盘上做冷缓存、单盘与双盘 GLM-5.2 对照 |+| 硬件感知规划器能自动接近每台机器的最优配置 | 当前已检测 RAM/VRAM 预算与多个后端 | 将自动方案与参数扫描对比,覆盖笔记本、工作站、NUMA 和多 GPU 主机 |+| 无损或质量受控的表示能充分减少权重搬运 | 已有格式与量化消融,并设置正确性/质量门槛 | 同时复现质量、搬运字节、延迟和每个有效 token 成本,而非只看压缩率 |+| 路由感知推测能在接近全驻留前盈利 | MTP 与语法草稿可用,但 MTP 在约 85% expert hit 时也实测过 -32% | 绘制接受率、命中率、批次并集与草稿深度的盈亏边界 |+| CPU/GPU 重叠能隐藏传输与同步,而非仅转移瓶颈 | CUDA 与 Metal 有成功数据,但强 CPU 和低驻留率会抹平收益 | 在 PCIe、统一内存与全驻留机器上做逐阶段 profile 和单变量 A/B |++想参与就任选一行,负结果也请公开。请记录硬件、commit、模型容器、完整命令、prompt、+缓存状态、吞吐、TTFT、expert hit、读取字节数与质量检查;每次只改一个变量,重复运行并附上+原始日志。先阅读 [CONTRIBUTING.md](CONTRIBUTING.md) 和+[benchmark 协议](docs/benchmarks.md),然后+[创建实验 issue](https://github.com/JustVugg/colibri/issues/new)。+在这里,一个受控的失败比一个无法解释的高数字更有价值。+## 核心概念744B 的专家混合(Mixture-of-Experts)模型,每个 token 只会激活约 40B 参数——diff --git a/README.zh-TW.md b/README.zh-TW.mdindex ac679d2b..34222e0d 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -62,21 +62,44 @@ VRAM/RAM/硬碟層級長條,以及角落的即時迷你大腦。</em></p>不是從 API 租用智慧,而是持有、探測、測量和改進它。引擎刻意維持足夠小,讓任何願意測量的人都可能貢獻下一項有效最佳化。-## 技術亮點+## 核心技術與實測結論- **統一階層,而非單一記憶體門檻。**VRAM、RAM 與 NVMe 是同一份權重的不同配置階層;高速記憶體不足只影響速度,不改變模型語意。- **權重的 JIT。**實測路由熱度驅動逐層 LRU、學習型熱門專家固定區和提前一層的預先載入,- 無需載入所有專家。+ 無需載入所有專家。它在可重複負載上有收益,但歷史可能過度擬合,預先載入在部分主機上+ 也可能負最佳化,因此它們是需要測量的策略,不是效能承諾。- **I/O 本身就是引擎的一部分。**專家批次聯集、讀算重疊、`O_DIRECT` 與加權雙 SSD- 分流直接最佳化串流路徑,而不是假裝儲存延遲不存在。+ 分流直接最佳化串流路徑,而不是假裝儲存延遲不存在。`O_DIRECT` 取決於磁碟,雙 SSD+ 仍需要更多社群端到端 A/B。- **異質執行。**CPU、CUDA、Metal、NUMA 記憶體以及專家的部分或全部常駐共用一個執行環境,- 可依機器條件組合。+ 可依機器條件組合;最佳組合取決於算力、頻寬、常駐率與負載。- **壓縮狀態,不竄改模型。**逐 token 精確的前向驗證、縮小 57 倍的 MLA KV 狀態、- 持久化熱會話與忠實 DSA,讓最佳化始終受正確性約束。+ 持久化熱會話與忠實 DSA,讓最佳化始終受正確性約束。這些是記憶體、延遲和正確性屬性,+ 不是籠統的吞吐承諾。- **必須證明收益的推測解碼。**原生 MTP 與文法強制草稿均接受端到端測量;接受率無法覆蓋驗證成本時可以關閉。+## 開放猜想、實驗與參與方式++在受控的端到端 A/B 證明之前,Colibrì 將每項最佳化都視為猜想。目前主要問題如下:++| 猜想 | 目前證據 | 仍需完成的實驗 |+|---|---|---|+| 路由歷史能比普通 LRU 更好地配置專家 | 學習型固定區能改善重複負載,但也會對 prompt 過度擬合 | 在程式碼、對話、多語言和長上下文負載上做留出集、跨會話 A/B |+| 多顆 SSD 能將獨立頻寬轉化為解碼速度 | 加權鏡像/分片路由已實作並通過驗證,頻寬模型成立 | 在獨立控制器的真實磁碟上做冷快取、單碟與雙碟 GLM-5.2 對照 |+| 硬體感知規劃器能自動接近每台機器的最佳配置 | 目前已偵測 RAM/VRAM 預算與多個後端 | 將自動方案與參數掃描對比,涵蓋筆電、工作站、NUMA 和多 GPU 主機 |+| 無損或品質受控的表示能充分減少權重搬運 | 已有格式與量化消融,並設置正確性/品質門檻 | 同時重現品質、搬運位元組、延遲和每個有效 token 成本,而非只看壓縮率 |+| 路由感知推測能在接近全常駐前獲利 | MTP 與文法草稿可用,但 MTP 在約 85% expert hit 時也實測過 -32% | 繪製接受率、命中率、批次聯集與草稿深度的盈虧邊界 |+| CPU/GPU 重疊能隱藏傳輸與同步,而非僅轉移瓶頸 | CUDA 與 Metal 有成功數據,但強 CPU 和低常駐率會抹平收益 | 在 PCIe、統一記憶體與全常駐機器上做逐階段 profile 和單變數 A/B |++想參與就任選一行,負結果也請公開。請記錄硬體、commit、模型容器、完整指令、prompt、+快取狀態、吞吐、TTFT、expert hit、讀取位元組數與品質檢查;每次只改一個變數,重複執行並附上+原始日誌。先閱讀 [CONTRIBUTING.md](CONTRIBUTING.md) 和+[benchmark 協議](docs/benchmarks.md),然後+[建立實驗 issue](https://github.com/JustVugg/colibri/issues/new)。+在這裡,一個受控的失敗比一個無法解釋的高數字更有價值。+## 核心概念744B 的專家混合(Mixture-of-Experts)模型,每個 token 只會啟用約 40B 參數——