Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs(fmt2): M1 Ultra fmt=2 benchmark context β issue/PR summaries, plans, scripts
boblaublaw1743054e489f
diff --git a/c/convert_fmt4_to_fmt2.sh b/c/convert_fmt4_to_fmt2.shnew file mode 100755index 00000000..f6d402e3--- /dev/null+++ b/c/convert_fmt4_to_fmt2.sh@@ -0,0 +1,8 @@+#!/bin/bash++rm -rf ~/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp++python3 ./coli convert \+ --model /mnt/zfs1/noprot/mlx-lm/models/GLM-5.2-colibri-int4-g64-with-int8-mtp \+ --outdir ~/mlx-models/GLM-5.2-colibri-int4-perrow \+ --group-size 0diff --git a/c/venv.sh b/c/venv.shnew file mode 100755index 00000000..37393177--- /dev/null+++ b/c/venv.sh@@ -0,0 +1,19 @@+#!/usr/bin/env bash+# colibri virtual environment helper+# Usage: source venv.sh++if [ -z "$COLI_VENV_ACTIVE" ]; then+ if [ -d ".venv" ]; then+ source .venv/bin/activate+ export COLI_VENV_ACTIVE=1+ echo "β colibri venv activated (numpy, huggingface_hub, safetensors)"+ else+ echo "Creating venv..."+ python3 -m venv .venv+ source .venv/bin/activate+ python -m pip install --quiet --upgrade pip+ python -m pip install --quiet numpy huggingface_hub safetensors+ export COLI_VENV_ACTIVE=1+ echo "β venv created and activated"+ fi+fidiff --git a/c/x.sh b/c/x.shnew file mode 100755index 00000000..b245ab69--- /dev/null+++ b/c/x.sh@@ -0,0 +1,24 @@+#!/bin/bash -x++# ============================================================+# M1 Ultra Metal benchmark β "Compare the myths of Lucifer and Prometheus"+#+# The original run (below, Config B from METAL-M5MAX-PERF-REPORT.md)+# produces ~0.14 tok/s because PIPE defaults to 0 on macOS and OMP+# active-spin throttles the GPU. See docs/METAL-M5MAX-PERF-REPORT.md+# for the full analysis.+#+# The winning config (E) from the M5 Max report achieves 2.24 tok/s+# by adding PIPE=1 PIPE_WORKERS=8 (async expert loading) and+# COLI_NO_OMP_TUNE=1 (passive OMP waits, preserves GPU clocks).+# M1 Ultra (24 GPU cores vs M5 Max's 40) should target >=2.0 tok/s.+# ============================================================++# OLD β missing PIPE + NO_OMP_TUNE; produces ~1.25 tok/s or worse+# COLI_METAL=1 DIRECT=1 MTP=0 ./coli run --model /Users/joe/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp/ --ram 110 "Compare the myths of Lucifer and Prometheus"++# SHORT RUN β verify PIPE+NO_OMP_TUNE fix with 128 tokens (fast check, ~60-120s at 0.16 tok/s or ~60s at 2 tok/s)+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 ./coli run --model /Users/joe/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp/ --ram 110 --ngen 128 "Compare the myths of Lucifer and Prometheus"++# FULL RUN β 1024 tokens once short run confirms correct throughput+# COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 ./coli run --model /Users/joe/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp/ --ram 110 "Compare the myths of Lucifer and Prometheus"diff --git a/docs/issue_103.md b/docs/issue_103.mdnew file mode 100644index 00000000..3cbd96fe--- /dev/null+++ b/docs/issue_103.md@@ -0,0 +1,164 @@+[Performance]: tok/s of 2.06 tok/s !!! Β· Issue #103 Β· JustVugg/colibri Β· GitHub++# [Performance]: tok/s of 2.06 tok/s !!! #103++**RDouglasSharp** opened on Jul 12, 2026++## Commit+pr-72-metal++## Hardware and storage+Apple M5 Max 128Mb 14" 1+8 core (6 Super and 12 Performance)+40 Core Metal 4 GPU++## Software environment+sudo sysctl iogpu.wired_limit_mb=120832++## Build and benchmark commands+COLI_METAL=1 DIRECT=1 MTP=0 ./coli run --model /Users/doug/glm52_i4 "Compare the myths of Lucifer and Prometheus" --ram 110++## Results+```+COLI_METAL=1 DIRECT=1 MTP=0 ./coli run --model /Users/doug/glm52_i4 "Compare the myths of Lucifer and Prometheus" --ram 110++ βββββ β colibrΓ¬ v1.0+ βββββββββββ piccolo motore, modello immenso+ βββββββ GLM-5.2 Β· 744B MoE Β· int4 Β· streaming CPU+ ββββ run+ β+ ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+[METAL] mode: batched routed experts on GPU (unified-memory zero-copy)+== Motore C GLM (glm_moe_dsa), cache=8 expert/layer | expert@8-bit densa@8-bit | idot: neon ==+caricato in 0.81s | densa residente: 9647.10 MB | layers=78 experts=256 | MTP assente (draft=0)+[MTP] assente (draft=0)+[USAGE] storia expert: 2936400 selezioni (/Users/doug/glm52_i4/.coli_usage)+[PIN] hot-store: 2478 expert in RAM (46.9 GB) in 3s da /Users/doug/glm52_i4/.coli_usage+[PIN] mlock: 46.9 GB inchiodati in RAM fisica / wired in physical RAM (niente compressione/no compression) in 1s+[RAM_GB=110.0] cap ALZATO 8->33: il budget lo consente (proiezione picco 109.9 GB; CAP_RAISE=0 per disattivare)+[stop] 3 token di stop: 154820 154827 154829+prompt: 13 token | genero fino a 1024 (stop EOS=154820) | draft n-gram=0+[gMASK]<sop><|user|>Compare the myths of Lucifer and Prometheus<|assistant|></think>[prefill] layer 1/78 Β· 13 token+[prefill] layer 5/78 Β· 13 token+[prefill] layer 9/78 Β· 13 token+[prefill] layer 13/78 Β· 13 token+[prefill] layer 17/78 Β· 13 token+[prefill] layer 21/78 Β· 13 token+[prefill] layer 25/78 Β· 13 token+[prefill] layer 29/78 Β· 13 token+[prefill] layer 33/78 Β· 13 token+[prefill] layer 37/78 Β· 13 token+[prefill] layer 41/78 Β· 13 token+[prefill] layer 45/78 Β· 13 token+[prefill] layer 49/78 Β· 13 token+[prefill] layer 53/78 Β· 13 token+[prefill] layer 57/78 Β· 13 token+[prefill] layer 61/78 Β· 13 token+[prefill] layer 65/78 Β· 13 token+[prefill] layer 69/78 Β· 13 token+[prefill] layer 73/78 Β· 13 token+[prefill] layer 77/78 Β· 13 token+[prefill] layer 78/78 Β· 13 token+The myths of Lucifer (in Christian tradition) and Prometheus (in Greek tradition)+[t=16 RSS 97.83 GB hit 61% 1.23 tok/s 1.00 tok/fw] represent two of the most profound and complex archetypes in human mythology: the "+[t=32 RSS 97.89 GB hit 67% 1.60 tok/s 1.00 tok/fw] rebel" or the "trickster" who defies the heavens.+[t=48 RSS 97.89 GB hit 70% 1.81 tok/s 1.00 tok/fw] Though they originate from entirely different cultural and theological contexts, their narratives share striking structural+[t=64 RSS 97.89 GB hit 70% 1.85 tok/s 1.00 tok/fw] and thematic similarities. as well as distinct differences.++Here is a detailed comparison of+[t=80 RSS 97.89 GB hit 71% 1.90 tok/s 1.00 tok/fw] the two figures:++## **1. Core Identity and Sphere of Influence**++**+[t=96 RSS 97.89 GB hit 72% 1.96 tok/s 1.00 tok/fw] Lucifer**+* **Origin:**** The name "Lucifer" translates+[t=112 RSS 97.89 GB hit 73% 2.02 tok/s 1.00 tok/fw] from Latin *lux* (light) + *ferre* (to carry+[t=128 RSS 97.89 GB hit 73% 2.05 tok/s 1.00 tok/fw] or to bring), meaning "Light-Bringer" or "Morning Star."+[t=144 RSS 97.89 GB hit 74% 2.08 tok/s 1.00 tok/fw] The title originally referred to the planet Venus in its morning appearance, signifying the+[t=160 RSS 97.89 GB hit 74% 2.09 tok/s 1.00 tok/fw] dawn and bringing of light to dispel the darkness of night.+* **+[t=176 RSS 97.89 GB hit 74% 2.10 tok/s 1.00 tok/fw] Christian Tradition:**** In Christian theology, Lucifer is often identified with Satan, the+[t=192 RSS 97.89 GB hit 74% 2.11 tok/s 1.00 tok/fw] adversary. However, strictly speaking, Lucifer is a Latin translation of the Hebrew phrase+[t=208 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] *helel ben-shahar* (meaning "Morning Star, son+[t=224 RSS 97.89 GB hit 74% 2.13 tok/s 1.00 tok/fw] of the Dawn") found in Isaiah 14:12. This passage was a+[t=240 RSS 97.89 GB hit 74% 2.13 tok/s 1.00 tok/fw] prophecy against the King of Babylon, mocking his human arrogance and ambition.+*+[t=256 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] **Mythological Evolution:**** Over time, through the influence of literature like+[t=272 RSS 97.89 GB hit 74% 2.13 tok/s 1.00 tok/fw] John Milton's *Paradise Lost* (1667) and Dante's+[t=288 RSS 97.89 GB hit 74% 2.13 tok/s 1.00 tok/fw] *Inferno* (1320), Lucifer became firmly established in the popular+[t=304 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] consciousness as the name of the archangel who, driven by pride, rebelled+[t=320 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] against God. He was cast down to hell...+* **Role:****+[t=336 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] In Christian theology, Lucifer/Satan plays the role of the "adversary+[t=352 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] " or the "accuser." He is the great antagonist in the Christian cosmic+[t=368 RSS 97.89 GB hit 74% 2.13 tok/s 1.00 tok/fw] drama: a fallen angel who defies the divine order and brings evil and suffering+[t=384 RSS 97.89 GB hit 74% 2.13 tok/s 1.00 tok/fw] into the world through his deception of Adam and Eve in the Garden of Eden.+[t=400 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] His "crime" is primarily one of *hubris*βoverweening+[t=416 RSS 97.89 GB hit 74% 2.12 tok/s 1.00 tok/fw] ambition and prideβwhich leads to a rebellion against the divine hierarchy.+* **+[t=432 RSS 97.89 GB hit 73% 2.12 tok/s 1.00 tok/fw] Symbolic Meaning:**** Symbolically, Lucifer represents the sin of pride ("*+[t=448 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] invidia*" in Latin, meaning "envy"). He is the embodiment of+[t=464 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] the rebellious intellect who seeks to become equal to the gods..++**Prom+[t=480 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] ometheus**+* **Origin:**** In Greek myth, Prometheus ("Fores+[t=496 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] ightful") is a second-generation Titan, a race of powerful beings who ruled+[t=512 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] the cosmos during the Golden Age.* This is a key difference: while Lucifer+[t=528 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] is an angel (a servant of God, created solely to obey and worship),+[t=544 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] Prometheus is a Titan, a primordial cosmic entity, born of the Earth (+[t=560 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] Gaia) and Sky (Uranus).+* **Mythological+[t=576 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] Role:**** Prometheus is the quintessential "trickster" and culture-hero+[t=592 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] of Greek myth. He is the creator of humanity, fashioning the first humans+[t=608 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] from clay. He is the benefactor of humanity, whose profound love for his+[t=624 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] human creations drives him to steal fire from the gods to give to humans, enabling+[t=640 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] human progress and the dawn of civilization. Because of this theft, he is chained+[t=656 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] to a rock in the Caucasus Mountains, where an eagle (the symbol of+[t=672 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] Zeus) eats his liver daily in perpetuity.+* **Symbolic Meaning+[t=688 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] :**** Prometheus represents human aspiration and intellect. He embodies "Promethean"+[t=704 RSS 97.89 GB hit 73% 2.10 tok/s 1.00 tok/fw] ambitionβthe drive to achieve greatness through technological and intellectual mastery over nature. He is+[t=720 RSS 97.89 GB hit 73% 2.10 tok/s 1.00 tok/fw] the patron of human striving and civilization. His myth explores the relationship between *mind+[t=736 RSS 97.89 GB hit 73% 2.10 tok/s 1.00 tok/fw] * (human intellect and craft) and *matter* (the physical world).+++[t=752 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] ## **2. Motivations: Rebellion, Pride, and Human Agency**+++[t=768 RSS 97.89 GB hit 73% 2.11 tok/s 1.00 tok/fw] **Lucifer: Rebellion as Sin**+In the Christian tradition, Lucifer's rebellion+[t=784 RSS 97.89 GB hit 73% 2.10 tok/s 1.00 tok/fw] is a sin born of pride and ambition. In Christian theology, angels are strictly+[t=800 RSS 97.89 GB hit 73% 2.10 tok/s 1.00 tok/fw] created beings, whose purpose is to serve and praise God.* When Lucifer says+[t=816 RSS 97.89 GB hit 73% 2.09 tok/s 1.00 tok/fw] , "I will ascend into heaven... I will make myself like the Most High+[t=832 RSS 97.89 GB hit 73% 2.09 tok/s 1.00 tok/fw] " (Isaiah :13-14), he is committing the sin+[t=848 RSS 97.89 GB hit 73% 2.09 tok/s 1.00 tok/fw] of *hubris* (overwhelming ambition) and *invidia* (+[t=864 RSS 97.89 GB hit 73% 2.09 tok/s 1.00 tok/fw] envy). He refuses to accept his divinely ordained place in the cosmos.++[t=880 RSS 97.89 GB hit 73% 2.09 tok/s 1.00 tok/fw] **Prometheus: Rebellion as Human Agency**+In contrast, Prometheus's defiance of+[t=896 RSS 97.89 GB hit 73% 2.08 tok/s 1.00 tok/fw] Zeus is not a sin, but an act of profound compassion and human agency.+[t=912 RSS 97.89 GB hit 73% 2.08 tok/s 1.00 tok/fw] His "crime" was motivated by pity for human beings' physical weakness and ignorance+[t=928 RSS 97.89 GB hit 73% 2.08 tok/s 1.00 tok/fw] . Unlike Lucifer, who is jealous of God, Prometheus is driven by *ag+[t=944 RSS 97.89 GB hit 73% 2.07 tok/s 1.00 tok/fw] ape* (unconditional love) for his creations, humans. By stealing fire+[t=960 RSS 97.89 GB hit 73% 2.07 tok/s 1.00 tok/fw] , he acts as a culture-bringer, giving humans the tools of civilizationβ+[t=976 RSS 97.89 GB hit 73% 2.07 tok/s 1.00 tok/fw] language, science,, and the artsβenabling them to become masters of+[t=992 RSS 97.89 GB hit 73% 2.07 tok/s 1.00 tok/fw] their own fate and to progress.+**Lucifer and Prometheus as "Light-+[t=1008 RSS 97.89 GB hit 73% 2.06 tok/s 1.00 tok/fw] Bringers"**+Despite their different roles, both Lucifer and Prometheus are "light++---+1024 token in 496.34s (2.06 tok/s) | hit-rate expert 72.5% | RSS 97.89 GB+expert caricati/token: 607.0 (per-layer 8.09 su 75; baseline topk=8) | TOPK=0 TOPP=0.00+speculazione: 1.00 token/forward (1023 fw per 1024 tok) | MTP acceptance 0% (0/0)+PROFILO: expert-disk 266.259s | expert-matmul 109.471s | attention 100.133s (di cui kvb 0.038s) | lm_head 0.004s | altro 20.475s+METAL-ATTN: layer GPU 79794 | gpu-wall 96.35s (kernel 76.36s | cpu-sched 0.62s gpu-sched 11.63s)+METAL: blocchi GPU 142391 | fallback CPU 0 | expert su GPU 618242 | setup 15.35s gpu-wall 86.02s (kernel 35.24s) scatter 1.46s+```++## Baseline comparison++Note that this Metal PR is based on an older base, missing features like interleaved expert loading.+So once properly merged, the tok/sec may be slightly higher!diff --git a/docs/issue_107.md b/docs/issue_107.mdnew file mode 100644index 00000000..858b8d06--- /dev/null+++ b/docs/issue_107.md@@ -0,0 +1,22 @@+# issue_107: Benchmark: Mac Mini M4 Pro (48 GB) β 0.30 tok/s with Metal backend (PR #72), 0.18 CPU-only++- **URL:** https://github.com/JustVugg/colibri/issues/107+- **State:** CLOSED (opened 2026-07-13 by bloxi-bot, closed 2026-07-13)++## Summary++First field report of the Metal backend (PR #72 branch) on a small-RAM Mac. Mac Mini M4 Pro (10P+4E CPU, 16-core GPU, 48 GB unified, ~273 GB/s), internal 1 TB SSD. Model: pre-converted int4 from `jlnsrk/GLM-5.2-colibri-int4` with **MTP heads swapped for the int8 ones** from `mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` (the main repo then still shipped the broken int4 sizes from #8).++- Disk: buffered 8.38 GB/s, F_NOCACHE 6.59 GB/s+- **Metal (PR #72), defaults, 128 tok: 0.30 tok/s** vs CPU-only 0.18 tok/s (all runs `--ram 38`)+- Metal run detail: `METAL: blocchi GPU 9018 | fallback CPU 0 | expert su GPU 100933`; expert-matmul 78.7s vs ~204s CPU-equivalent (~2.6x); METAL-ATTN 4368 layers on GPU; RSS 27.4 GB; profile 59% disk+- Throughput warmed from 0.23 at t=16 to steady 0.30-0.31 by t=64++## Relevance++- **Goal a (fmt_2_m1u):** Shows the Metal dispatch counters (`blocchi GPU`, `fallback CPU`, `expert su GPU`, `METAL-ATTN`) of a healthy run on a constrained box β the same counters the M1 Ultra fmt=4 run showed as all-zero (see [metal_dispatch_gap.md](metal_dispatch_gap.md)).+- **Goal b (fmt_4_m1u):** Context β what healthy Metal dispatch output looks like.++## See also++- [pr_72](pr_72.md), [issue_103](issue_103.md)diff --git a/docs/issue_180.md b/docs/issue_180.mdnew file mode 100644index 00000000..aa221bb2--- /dev/null+++ b/docs/issue_180.md@@ -0,0 +1,28 @@+# issue_180: Datapoint: M3 Ultra (Mac Studio, 256 GB, 80-core GPU) β 1.45 tok/s @ 82% hit, Metal + tuned flags++- **URL:** https://github.com/JustVugg/colibri/issues/180+- **State:** CLOSED (opened 2026-07-14 by NullNaveen, closed 2026-07-14)++## Summary++Only Ultra-class Apple Silicon datapoint in the tracker. Mac Studio M3 Ultra, 256 GB unified, 80-core GPU, `main` built `make glm METAL=1`, model `mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`. **Shared box** (concurrent 24-user Ollama workload, ~46 GB resident on the same GPU) β numbers are a lower bound.++| Config | tok/s | expert hit | experts/token | RSS |+|---|---|---|---|---|+| CPU, `--ram 40`, defaults | 0.07 | 19% | 1021 | 37 GB |+| Metal, `--ram 40`, defaults | 0.39 | 21% | 1142 | 37 GB |+| **Metal + tuned, `--ram 160`** | **1.45** (peak 1.75) | **82%** | **381** | 131 GB |++- Tuned config: `COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 IO_THREADS=8 ./coli run --ram 160 --topp 0.7`+- Profile of the 1.45 run: expert-disk 75.3s service / 0.0s wait, expert-matmul 49.3s, attention 45.0s, other 82.1s+- Confirms `MTP=0` is a large win: 1142 -> 381 experts/token; speculation was working (53-67% acceptance) but still not worth it+- Note: 82% hit rate helped here (256 GB box, `--ram 160`) β contrast with #379's minimal-cache finding on 128 GB M5 Max++## Relevance++- **Goal a (fmt_2_m1u):** The closest hardware analog to the M1 Ultra target β Ultra-class, unified memory, same tuned flag set. Template for the M1 Ultra Config E and `--ram` exploration.+- **Goal b (fmt_4_m1u):** Context only.++## See also++- [issue_103](issue_103.md), [pr_116](pr_116.md), [issue_379](issue_379.md)diff --git a/docs/issue_379.md b/docs/issue_379.mdnew file mode 100644index 00000000..c2eaa9cb--- /dev/null+++ b/docs/issue_379.md@@ -0,0 +1,25 @@+# issue_379: Metal β growing the expert cache makes decode SLOWER++- **URL:** https://github.com/JustVugg/colibri/issues/379+- **State:** OPEN (opened 2026-07-18 by monotophic)+- **Labels:** bug, metal++## Summary++On Metal (M5 Max 128 GB, macOS 26.5.2, `main @ 72d3d37`), **every expert-cache mechanism makes decode slower** β raising the LRU cap, adding a pin, or enabling AUTOPIN all reduce tok/s even as hit rate rises. Fastest config found: *minimal* cache at **~2.0 tok/s** vs ~0.4 at auto defaults (5Γ from config alone).++- Champion config: `MTP=0 CAP_RAISE=0 AUTOPIN=0 --cap 1`+- Suspected mechanism: GPU-side buffer residency / page-mapping cost on large, sparsely-reused host buffers registered via `newBufferWithBytesNoCopy` β on Metal, cache hits cost more than the page-cache misses they replace+- Cross-platform contrast: DGX Spark GB10 gets 2.39 tok/s at 82% hit with a large LRU + AUTOPIN; the M5 Max gets its best at **2% hit, no cache**+- Four alternative mechanisms falsified via controlled A/Bs (data in #387)+- Led directly to PR #386 (platform-aware cache defaults via storage probe)++## Relevance++- **Goal a (fmt_2_m1u):** Critical tuning context β on a 128 GB Apple Silicon box, a *big* expert cache can be anti-productive; M1 Ultra benchmark configs must A/B cache size (`--cap`, CAP_RAISE, AUTOPIN), not assume more cache = faster.+- **Goal b (fmt_4_m1u):** Shared β same cache tuning applies to the fmt=4 benchmark runs after backend fixes land.++## See also++- [issue_387](issue_387.md) β the A/B data behind this diagnosis+- [pr_386](pr_386.md) β the merged defaults changediff --git a/docs/issue_387.md b/docs/issue_387.mdnew file mode 100644index 00000000..3067ecfb--- /dev/null+++ b/docs/issue_387.md@@ -0,0 +1,27 @@+# issue_387: Datapoint: M5 Max 128 GB β decode-tuning A/Bs (MTP x cache 2x2): 0.4 -> 2.0 tok/s at identical output++- **URL:** https://github.com/JustVugg/colibri/issues/387+- **State:** OPEN (opened 2026-07-18 by monotophic)+- **Labels:** metal, benchmark++## Summary++Controlled, identical-output A/B campaign (greedy temp-0, md5-checked token streams, pristine `.coli_usage` snapshot restored per run) on M5 Max 128 GB, measured 2026-07-15..17, `main @ 72d3d37`, model `mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` (144 shards / 384 GB).++- **Champion config:** `MTP=0 CAP_RAISE=0 AUTOPIN=0 coli run --ram 90 --cap 1` β takes the box from ~0.4 tok/s (defaults) to ~2.0 tok/s+- **`MTP=0`:** +79% at a cached config, +10-22% at cap1; -20-45% bytes moved+- **`--cap 1` (no LRU cache):** smaller cache = faster, always; expert-matmul 20.6s -> 35.4s -> 78.4s as the cache grows+- **`--ram`:** flat on speed across 20-40 GB; choose on heat/energy, not speed+- **`COLI_MMAP=1`:** do not use on Metal β 39x slower (GPU demand-faults a 358 GB mapping)+- First two measurement rounds were invalid and discarded; the methodology section documents what survived (temp 0, usage-snapshot restore, etc.)++This is the data behind the #379 diagnosis and the PR #386 defaults change.++## Relevance++- **Goal a (fmt_2_m1u):** Direct tuning playbook for the M1 Ultra benchmark β MTP off, cache minimal, ram sizing, and a rigorous identical-output methodology worth mirroring.+- **Goal b (fmt_4_m1u):** Shared β same tuning applies to fmt=4 benchmark runs.++## See also++- [issue_379](issue_379.md), [pr_386](pr_386.md)diff --git a/docs/issue_47.md b/docs/issue_47.mdnew file mode 100644index 00000000..9d3fec56--- /dev/null+++ b/docs/issue_47.md@@ -0,0 +1,30 @@+# issue_47: Benchmark: Apple M3 Max Β· macOS Β· 128 GB Β· internal SSD (12.5 GB/s) β 0.35β0.45 tok/s; PILOT neutral; confirms #8++- **URL:** https://github.com/JustVugg/colibri/issues/47+- **State:** CLOSED (opened 2026-07-11 by fabio-rovai, closed 2026-07-13)++## Summary++Pre-Metal CPU-era benchmark datapoint on Apple M3 Max (12P+4E), 128 GB unified, internal SSD measured at 12.49 GB/s O_DIRECT. Container: `jlnsrk/GLM-5.2-colibri-int4` (144 shards, 379 GB). Greedy `--temp 0`, `--ngen 400`, ~1k-token prompt.++| config | tok/s | expert hit | RSS |+|---|---|---|---|+| baseline, `MTP=0` | 0.35 | 68.5% | 82.2 GB |+| + `PIN PIN_GB=90` | 0.36 | 70.6% | 95.8 GB |+| + `PILOT=1` | 0.36 (neutral) | 70.6% | 95.7 GB |+| + `DRAFT=4` | 0.34 | 70.4% | 97.5 GB |+| + `--topp 0.7` | **0.45 (+29%)** | 70.0% | 96.1 GB |++- Baseline profile per 400 tok: expert-disk 400s, expert-matmul 311s, attention 351s, altro 89s β attention ~30% of token cost on CPU+- `--topp 0.7` cut expert loads 28.7 -> 17.9/layer+- Confirmed #8: int4 MTP head gives 0% acceptance (0/24) in all configs+- Also reported a setup.sh libomp false-positive++## Relevance++- **Goal a (fmt_2_m1u):** Historical baseline β the CPU-only world before Metal (#72). Documents the `--topp 0.7` lever and the MTP acceptance problem (#8) that motivated `MTP=0` in later configs.+- **Goal b (fmt_4_m1u):** Not directly relevant.++## See also++- [issue_87](issue_87.md), [issue_103](issue_103.md), [pr_72](pr_72.md)diff --git a/docs/issue_622.md b/docs/issue_622.mdnew file mode 100644index 00000000..e9d6ae4e--- /dev/null+++ b/docs/issue_622.md@@ -0,0 +1,23 @@+# issue_622: [Bug]: Metal prefill GEMM is not token-exact vs CPU on near-tie logits (S >= GEMM_MIN); decode unaffected++- **URL:** https://github.com/JustVugg/colibri/issues/622+- **State:** OPEN (opened 2026-07-25 by lBroth)+- **Labels:** bug, metal++## Summary++On tiny fixture containers, teacher-forced prefill disagrees CPU vs Metal at 2 of 32 positions, deterministically, on stock `dev @ 7b03ce0` (M5 Pro, macOS 26.5.2). Greedy **decode is token-identical** on the same containers. Surfaced during the PR #587 hardware runs; filed separately since it reproduces on stock dev where that PR's code does not exist.++- Reproduces on both a grouped int4 g64 container and an int3-experts/per-row-int4-dense container β **format-agnostic**+- `COLI_METAL_GEMM_MIN=1000` (keep every `matmul_qt` GEMM on CPU; default threshold 16 rows) makes both containers bit-identical to the CPU reference β localizes the culprit to the Metal prefill GEMM at S >= 16 rows+- Explains why decode stays exact: the harness prefills a 12-token prompt, below the 16-row threshold+- Near-tie logits can flip token choice under different FP reduction order β benign for throughput, relevant for bit-exact CPU-vs-GPU validation++## Relevance++- **Goal a (fmt_2_m1u):** When validating the fmt=2 reproduction against CPU output, prefill (S >= GEMM_MIN) may diverge on near-ties β judge on coherence, not bit-identity, or set `COLI_METAL_GEMM_MIN=1000` for an exact A/B.+- **Goal b (fmt_4_m1u):** Fix B modifies `coli_metal_gemm` / `bind_gemv` β the same path this bug lives in. PR #763 (GPU prefill attention) is gated off by default because of this family of divergence.++## See also++- [pr_763](pr_763.md), [pr_587](pr_587.md)diff --git a/docs/issue_707.md b/docs/issue_707.mdnew file mode 100644index 00000000..530d9d25--- /dev/null+++ b/docs/issue_707.md@@ -0,0 +1,23 @@+# issue_707: [Bug]: OpenMP spin-wait tuning is inert on macOS β and applying it costs 2.2x decode on a 32 GB host++- **URL:** https://github.com/JustVugg/colibri/issues/707+- **State:** OPEN (opened 2026-07-30 by fredchu)+- **Labels:** bug, help wanted, performance, metal++## Summary++The OpenMP hot-thread tuning block in `main()` never reaches the OpenMP runtime on macOS, because the self-re-exec that makes it effective is compiled only for Linux and FreeBSD. On the reporter's M1 Max 32 GB host that is **load-bearing in the good direction**: exporting the same four variables externally (`OMP_WAIT_POLICY=active KMP_BLOCKTIME=200 OMP_DYNAMIC=FALSE GOMP_SPINCOUNT=200000`) makes decode **2.2x slower**.++- Item 1: the tuning is inert on macOS β and that is currently a good thing+- Item 2: also about how the OpenMP team is configured on macOS (see issue body)+- The report is less "fix the missing branch" and more "don't fix it without gating β here is the measurement"+- Environment: macOS 26.5.2, M1 Max (8P+2E), 32 GB, `make colibri METAL=1`, model `mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp`++## Relevance++- **Goal a (fmt_2_m1u):** Directly justifies `COLI_NO_OMP_TUNE=1` in benchmark Configs D/E (from the M5 Max report): active-spin costs GPU clocks on Apple's shared SoC power budget. M1 Ultra configs must keep OMP waits passive.+- **Goal b (fmt_4_m1u):** Shared β same flag discipline for fmt=4 runs.++## See also++- [pr_116](pr_116.md) β the M5 Max report that identified the OMP-spin regressiondiff --git a/docs/issue_87.md b/docs/issue_87.mdnew file mode 100644index 00000000..c41b617d--- /dev/null+++ b/docs/issue_87.md@@ -0,0 +1,23 @@+# issue_87: [Performance]: M5 MAX 128GB with METAL - 1.83 tok/s++- **URL:** https://github.com/JustVugg/colibri/issues/87+- **State:** CLOSED (opened 2026-07-12 by baha2046, closed 2026-07-13)++## Summary++First M5 Max Metal datapoint, using the pre-merge `codearranger/colibri` `metal-backend` branch (what became PR #72) with early Fable5 tuning: `DIRECT=1 IO_THREADS=8`, MTP off.++- Hardware: MacBook M5 Max 128 GB, 2 TB+- iobench on an expert shard: buffered 17.35 GB/s; O_DIRECT 64.68 GB/s (8 threads, 19 MB blocks)+- Result: **1.83 tok/s** with `COLI_MODEL=... ./coli run "Compare the myths of Lucifer and Prometheus" --ram 96`+- Log shows the auto macOS banner: "MTP ... su macOS il decode liscio e' piu' veloce (A/B misurato 1.68 vs 0.85 tok/s): draft OFF"+- Same Lucifer/Prometheus prompt later used by #103 and the M5 Max report++## Relevance++- **Goal a (fmt_2_m1u):** Part of the result lineage: 1.83 (#87) -> 2.06 (#103) -> 2.24 (PR #116 report). Establishes the prompt and the `DIRECT=1 IO_THREADS=8 MTP=0` lever set.+- **Goal b (fmt_4_m1u):** Context only.++## See also++- [issue_103](issue_103.md), [pr_72](pr_72.md), [pr_116](pr_116.md)diff --git a/docs/metal_dispatch_gap.md b/docs/metal_dispatch_gap.mdnew file mode 100644index 00000000..096470da--- /dev/null+++ b/docs/metal_dispatch_gap.md@@ -0,0 +1,412 @@+# Metal Dispatch Gap: fmt=4 Grouped Int4 Not Supported Anywhere in the Metal Backend++> **Scope:** This document is a **problem statement** only. It identifies the fmt=4 Metal backend gap and provides evidence. Implementation plans are in:+> - [`docs/metal_fmt2.md`](metal_fmt2.md) β fmt=2 (per-row int4) conversion and benchmark plan+> - [`docs/metal_fmt4.md`](metal_fmt4.md) β fmt=4 (grouped int4) Metal backend fix plan++---++## Symptom++M1 Ultra (128 GB unified memory) benchmark of GLM-5.2 744B MoE achieves **0.15 tok/s**+despite `COLI_METAL=1` being set. From `c/3_with_pipe_no_omp_tune128ngen.log`:++```+METAL: blocchi GPU 0 | fallback CPU 0 | expert su GPU 0 | setup 0.00s gpu-wall 0.00s (kernel 0.00s) scatter 0.00s+```++Zero GPU blocks dispatched, zero fallback blocks, zero experts on GPU β **zero dispatch+attempts** of any kind.++**Second, equally important signal: there is no `METAL-ATTN:` line in the log at all.**+That line is printed at `colibri.c:4968β4970` only when `aok > 0` (i.e. only when the fused+attention/layer kernel actually ran at least once). Its absence proves the fused attention+path never engaged either. Attention ran entirely on the CPU.++Decode profile from the same log (128 tokens, 877.98s, 127 forwards):++| phase | total | per token | where it ran |+|---|---|---|---|+| expert-matmul | 466.662s | 3.67s | **CPU** (should be GPU) |+| attention | 186.485s | 1.47s | **CPU** (should be GPU) |+| expert-disk wait | 154.126s | 1.21s | CPU/disk (correct) |+| other | 70.708s | 0.56s | CPU |++For comparison, the M5 Max report (`docs/METAL-M5MAX-PERF-REPORT.md`) achieves **2.24 tok/s**+(Config E). **Critical context:** that benchmark was run on **fmt=2 (per-row int4)** weights,+NOT the fmt=4 grouped int4 container used here. The M5 Max report explicitly shows+`fallback CPU 0` and non-zero `attn GPU kernel` times (79s in Config E), proving Metal was+active. Issue #585 documents the exact symptom on fmt=4: **2.39 tok/s on fmt=2 vs 0.81 tok/s+on fmt=4** (3Γ slowdown) on the same M5 Max hardware.++**Bottom line:** The 2.24 tok/s M5 Max result is valid Metal performance, but only for fmt=2+models. The fmt=4 grouped int4 GLM-5.2 container (gs=64) has **no Metal support anywhere** in+the current backend β this is why M1 Ultra shows zero Metal dispatch despite `COLI_METAL=1`.++## Issue #103 Evidence β How the M5 Max 2.06β2.24 tok/s Benchmark Was Achieved++Direct inspection of the original benchmark report ([issue_103.md](issue_103.md), Issue #103) reveals the M5 Max+result was achieved on a **fmt=2 (per-row int4)** model, NOT the official GLM-5.2 fmt=4 container:++- **Model path**: `/Users/doug/glm52_i4` β a custom/local conversion, not the official+ `GLM-5.2-colibri-int4-g64` container+- **Banner confirms fmt=2**: The log shows `expert@8-bit densa@8-bit` without the+ "cosmetically wrong" mismatch that appears for fmt=4 models (see line 82β88 below)+- **Metal fully engaged**: `fallback CPU 0`, `METAL-ATTN: layer GPU 79794`, and+ `attn GPU kernel 76.36s` prove all three Metal paths were active++**Conclusion**: To reproduce the 2.06β2.24 tok/s results, you must either:+1. Convert the model with **fmt=2 (per-row int4)** settings instead of fmt=4, OR+2. Apply the Metal backend fixes (A+B+C) to support fmt=4 grouped int4++The official GLM-5.2 colibri container uses fmt=4 by design β this is not a conversion+error, but a format choice that currently lacks Metal support.++## Verified: the model container is CORRECT β conversion did NOT fail++This was checked directly against the safetensors headers in+`/Users/joe/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp/` (3551 tensors scanned).++For every quantized tensor, the `.qs` scale count is **exactly `O Γ ceil(I/64)` floats**,+which is the definition of grouped int4 at gs=64. Examples (layer 0 / layer 77, identical):++| tensor | weight bytes (U8) | `.qs` floats | implies |+|---|---|---|---|+| `self_attn.q_a_proj` | 6,291,456 | 196,608 | OΒ·I=12,582,912 β OΒ·I/64 β fmt=4 g64 |+| `self_attn.q_b_proj` | 16,777,216 | 524,288 | OΒ·I=33,554,432 β /64 β fmt=4 g64 |+| `self_attn.kv_a_proj_with_mqa` | 1,769,472 | 55,296 | OΒ·I=3,538,944 β /64 β fmt=4 g64 |+| `self_attn.kv_b_proj` | 7,340,032 | **229,376** | 28,672 rows Γ **8 groups** β fmt=4 g64 |+| `self_attn.o_proj` | 50,331,648 | 1,572,864 | OΒ·I=100,663,296 β /64 β fmt=4 g64 |+| `mlp.experts.N.{gate,up,down}_proj` | 6,291,456 | 196,608 | 2048Γ6144 β /64 β fmt=4 g64 |+| `mlp.shared_experts.*` | 6,291,456 | 196,608 | β fmt=4 g64 |+| `mlp.gate.weight` (router) | F32 [256,6144] | β | f32, not quantized |++Config confirms the shapes: `hidden_size=6144`, `moe_intermediate_size=2048`,+`n_routed_experts=256`, `num_experts_per_tok=8`, `n_shared_experts=1`,+`num_hidden_layers=78`, `first_k_dense_replace=3`, `q_lora_rank=2048`, `kv_lora_rank=512`,+`qk_nope_head_dim=192`, `qk_rope_head_dim=64`, `v_head_dim=256`, `num_attention_heads=64`.++MTP head: `out-mtp-00000.safetensors` is 9,959,321,520 bytes β matches the sum of the three+int8 sizes the top-level `README.md` documents as correct (3,527,131,672 + 5,366,238,584 ++1,065,950,496 = 9,959,320,752, the ~768-byte delta being one merged safetensors header+instead of three). This is the int8 MTP container, not the broken int4 one.++Plus the strongest evidence of all: **the run produces coherent, well-formed output.** The+CPU `matmul_i4_grouped` path is decoding these weights correctly.++**Conclusion: this is a backend gap, not a bad conversion. Do not re-download or re-convert+the model.**++**Note on the banner as a format diagnostic:** the log line+`== GLM C engine (glm_moe_dsa), cache=8 experts/layer | experts@8-bit densa@8-bit | idot: neon ==`+says "8-bit" for both experts and dense. For fmt=4 models, this is cosmetically wrong:+`qt_from_disk` (`colibri.c:1055`) sees a `name.qs` sidecar and uses the container's+pre-quantized fmt=4 data, ignoring the bits setting. The banner correctly reflects fmt=2+(per-row int4) models (as seen in Issue #103), but misrepresents fmt=4 grouped int4.+This banner can serve as a quick visual check: if it says "8-bit" but you're loading a+grouped-int4 container, the format mismatch is confirmed.++## Root Cause: fmt=4 is rejected at every Metal entry point++GLM-5.2 stores **everything** quantized as `fmt=4` grouped int4 with `gs=64`.+`qt_resolve_fmt` (`colibri.c:1027`) resolves the container: it first classifies the weight+bytes as int4 (`fmt=2`), then `detect_group_size` (`colibri.c:1005`) sees+`ns == O*ng*4` and promotes it to `fmt=4, gs=64` (`colibri.c:1043`). Every path below then+sees `fmt=4`.++### Gate 1 β MoE experts (`moe_submit`, `backend_metal.mm:963`)++```cpp+if (!g_dev || (fmt != 1 && fmt != 2)) return nil; // <-- rejects fmt=4+```++`mfmt` is set from the first collected expert at `colibri.c:3210` (`if(mfmt<0) mfmt=e->g.fmt;`)+= 4, then passed to `coli_metal_moe_block_begin` (`colibri.c:3242`) β+`moe_submit` (`backend_metal.mm:1079`) β immediate `nil`.++This returns **before** the `g_moe_fb++` fallback counter at `backend_metal.mm:973β978`,+which is exactly why all three counters read zero instead of showing fallbacks:++- `blocchi GPU 0` = `g_moe_ok == 0`+- `fallback CPU 0` = `g_moe_fb == 0` β would be non-zero if the gate were passed and `resolve()` failed+- `expert su GPU 0` = `g_moe_experts == 0`++With `mh == NULL`, `colibri.c:3245` leaves `cpu_res=1`, the missed-expert submit at+`colibri.c:3299β3301` hits the same gate, so `metal_done = 0` at `colibri.c:3308` and the+CPU expert loop runs everything.++The `moe_gemv` kernel itself (`backend_metal.mm:70β99`) has no fmt=4 case: it handles+`fmt==2` (per-row int4) and `else` (int8), and applies a single per-row scale *after* the+simd reduction (`yout[row]=acc*sc[o]`, line 98) β structurally incompatible with a scale+that varies along the input dimension.++### Gate 2 β prefill GEMM (`coli_metal_gemm`, `backend_metal.mm:860`)++```cpp+if (!g_dev || (fmt!=1 && fmt!=2)) return 0;+```++plus the caller-side gate in `matmul_qt_ex` (`colibri.c:575`):++```c+if(g_metal_enabled && S>=g_metal_gemm_min && !spec_pinned() && (w->fmt==1||w->fmt==2) && !omp_in_parallel()){+```++So every batched (prefill) projection β `q_a`, `q_b`, `kv_a`, `kv_b`, `o_proj`, and the+dense-layer MLPs of layers 0β2 β skips Metal and runs `matmul_i4_grouped` on the CPU+(`colibri.c:591`).++### Gate 3 β fused decode attention (`colibri.c:2392` and `colibri.c:4231`)++Both fused paths require `l->kv_b.fmt==2`:++```c+&& c->qk_rope==64 && vh==256 && l->kv_b.fmt==2){ // attention_rows, colibri.c:2392+&& c->qk_nope==192 && c->qk_rope==64 && c->v_head==256 && l->kv_b.fmt==2 // layer_forward, colibri.c:4231+```++`kv_b` is fmt=4 (229,376 scales = 28,672 rows Γ 8 groups), so **both gates fail on every+layer of every token** β `coli_metal_attn_decode` / `coli_metal_layer_decode` are never+called β no `METAL-ATTN:` line β 186s of decode attention on the CPU.++Note this gate is currently *load-bearing for correctness*, not just performance: the+absorb-core dequant helper is per-row only (`backend_metal.mm:134β135`):++```cpp+inline float a_deqrow(device const uchar* base, int row, int i, device const float* sc){+ device const uchar* w=base+(long)row*((A_KVL+1)/2); uchar b=w[i>>1];+ int val=(i&1)?(b>>4):(b&0xF); return float(val-8)*sc[row]; } // <-- sc[row], per-row only+```++If the gate were simply relaxed to admit fmt=4 without fixing `a_deqrow`, `a_qabs`+(`backend_metal.mm:136`) and `a_ctx` (`backend_metal.mm:168`) would read `sc[row]` out of a+grouped scale array and produce silently wrong attention. **The gate must be relaxed and+`a_deqrow` made group-aware in the same change.**++### Latent hazard β `bind_gemv` does not validate fmt (`backend_metal.mm:637`)++`bind_gemv` forwards whatever `fmt` it is handed straight into `mm_gemv` without checking it.+`mm_gemv` (`backend_metal.mm:16β64`) handles fmt 1/2/3 and treats **everything else as f32**+(`else` branch, line 56). Today nothing reaches it with fmt=4 because Gate 3 blocks the whole+attention path on `kv_b` alone β but the other five projections' fmts are passed through+*unchecked*. A model with fmt=2 `kv_b` and fmt=4 `o_proj` would pass the gate and then read+packed int4 bytes as f32 floats: garbage output, no error, no fallback. This should be+hardened (`return false` on unsupported fmt) as part of the fix regardless of the fmt=4 work.++## Why fmt=4 Is Special β Scale Layout++- **fmt=1 (int8):** one scale per output row β `scale[o]`.+- **fmt=2 (int4 per-row):** one scale per output row β `scale[o]`.+- **fmt=4 (grouped int4):** `ng = ceil(I / gs)` scales **per output row**, `gs=64` here.++The authoritative layout is the CPU reference `matmul_i4_grouped` (`quant.h:168β202`):++```c+int rb=(I+1)/2; int ng=(I+gs-1)/gs; // ng is over the INPUT dim I+const uint8_t *w = q4 + (int64_t)o*rb;+const float *scl= scale + (int64_t)o*ng; // per output row+for(int g=0; g*gs<I; g++){ float sc=scl[g]; ... a += (partial dot over group g) * sc; }+```++and confirmed by CUDA (`backend_cuda.cu:195β205`, `absorb_scale` at `:163β168`):++```cuda+const float *scl = scales + (size_t)o * ng;+int g = i / gs; if (g >= ng) g = ng - 1; // clamp tail of last partial group+sum += xs[i] * weight_at(weights, fmt, row, i) * scl[g];+```++So for a GEMV with input dim `K`: **`ng = ceil(K/gs)`**, and the scale for output row `o`,+input element `i` is **`sc[o*ng + i/gs]`**, where `sc` is already the per-expert/per-tensor+base pointer. Per-gemv `ng` values in this model: gate/up `K=6144 β ng=96`;+down `K=2048 β ng=32`; kv_b absorb `K=512 β ng=8`.++Nibble decode convention: Metal's existing fmt=2 code uses **offset binary**+(`float(int(b&0xF)-8)`, `backend_metal.mm:88`), same as the CPU (`quant.h:195`). fmt=4 uses+the identical nibble encoding β **only the scale lookup changes.** (CUDA looks different+because it pre-XORs weights with `0x88` at upload β `backend_cuda.cu:170`,`:778` β and then+sign-extends; net effect is the same values. Metal needs no such conversion.)++Alignment luck worth exploiting: `detect_group_size` (`colibri.c:1010`) only ever returns+`{16,32,48,64,96,128,192,256}` β **all multiples of 8**. The Metal gemv kernels load 8+weights per iteration (`uchar4` β 8 nibbles), so **an 8-element load never straddles a group+boundary**, and the group scale can be applied once per 8-load instead of per element:+`g = (c*8)/gs`. The strided lane loop (`c += 32`, i.e. 256 elements) also preserves this.++## GitHub Issue Evidence β This Is a Known, Active Gap++Direct inspection of the issue tracker reveals this is not a newly-discovered bug but a+**documented, actively-worked gap** in Metal fmt=4 support. The following issues provide+context, precedent, and partial implementations:++### Active Metal fmt=4 Work++**#585** β "[Feature]: METAL Attention support for fmt:4" (OPEN)+- Documents the exact symptom: `PROF=1 COLI_METAL=1` achieves **2.39 tok/s on fmt=2** vs+ **0.81 tok/s on fmt=4** (3Γ slowdown) on M5 Max+- Root cause identified: attention path gated to `l->kv_b.fmt==2` only+- Proposed solution matches Fix C in this document++**#587** β "Metal fmt=4 grouped-int4 decode: attention + routed experts" (OPEN, needs-rebase)+- PR implementing the fix for #585+- **Verified performance gain**: 0.81 β **2.13 tok/s** after fixing both attention and+ expert paths on a real GLM-5.2 744B checkpoint+- **Scope limitation**: prefill GEMM path (`coli_metal_gemm`) remains unaddressed β+ this is a decode-only fix+- Test plan includes kernel unit tests vs CPU oracle for both `moe_gemv` and attention paths++**#457** β "metal: grouped-int4 (fmt=4) GEMV support" (MERGED 2026-07-30)+- Fixed the `mm_gemv` kernel for fmt=4 (dense projections, attention `q_a/q_b/kv_a/o`)+- Added proper `gs` parameter threading through `AttnW`, `coli_metal_attn_decode`,+ `coli_metal_layer_decode`, and `bind_gemv`+- **Critical discovery**: traced that `qt_from_disk` allocated fmt=4 scale buffers with+ `falloc` (bare `malloc`, never registered), which **accidentally masked the shader bug**+ β the scale buffer never resolved to GPU, so the broken shader code never executed+- Added `fmt_scale_bytes()` to correctly size fmt=4 scale buffers (`O*ng` floats)+- **Test coverage**: 12 new fmt=4 cases in `metal-test`, including grouped `coli_metal_gemm`+ and fused-decode-path (`run_attn_grouped`) with `qraw` metric to detect uniform-scale bugs++### CUDA Precedent β Same Bugs Hit First++**#298** β "cuda+engine: full fmt=4 support" (MERGED)+- CUDA's complete fmt=4 implementation across all paths+- **1.08 tok/s** on RTX 5070 Ti after fix (vs broken 0.05 tok/s before)+- Established the scale layout convention: `scale[o*ng + g]` with tail clamp `g >= ng β ng-1`+- Added fused `matmul_i4_grouped_pair` for gate+up (expert path optimization)++**#334** β "fmt=4 CUDA attention crash" (CLOSED)+- **Hard crash** (GPU memory fault β system reboot) when attention absorb kernels used+ per-row scales on grouped data+- Root cause: `attention_absorb_kernel` indexed scales as `wscale[row]` instead of+ `wscale[row*ng + k/gs]` for fmt=4+- **Prevention note**: "When adding a new quant format or editing a GPU kernel: any kernel+ that multiplies a quantized weight by a scale must index the scale per-group for fmt=4"++**#464** β "cuda: reject fmt=4 at per-row-only entry points" (MERGED)+- Hardens CUDA to explicitly reject fmt=4 at unsafe entry points (`coli_cuda_matmul`,+ `coli_cuda_expert_mlp`)+- Ensures fallback to CPU rather than silent garbage output+- Precedent for the `bind_gemv` hardening recommended in this document++### Key Insight: Accidental Safety via Allocator Bug++Issue #457's investigation revealed a **critical interaction** that explains why the+system didn't produce garbage output despite the shader bugs:++1. `qt_from_disk` allocated fmt=4 scale buffers with `falloc` (not `qalloc`)+2. `falloc` does not page-align and never calls `coli_metal_register`+3. `resolve()` in `backend_metal.mm` only finds pointers inside registered slabs+4. Therefore `resolve(t->s, ...)` **always fails** for fmt=4 scale buffers+5. `bind_gemv`/`coli_metal_gemm` return `false`/`0` β **CPU fallback**++**Consequence**: The allocator bug kept the broken shader code from ever executing.+Fixing the allocator (as #457 does) without fixing the shader would have **armed the bug**+β scale buffers would resolve, but the shader would read them incorrectly.++This is why #457's commit order matters:+- **First**: `f0fa5a9` β shader fmt=4 branch + host plumbing (capability lands **inert**)+- **Second**: `2ba12d8` β allocator fix (activates the already-correct capability)+- **Third**: `fc0f5a5` β attn-test blind-spot fix (strengthens test coverage)++### Current Status Summary++| Path | Status | Notes |+|---|---|---|+| `mm_gemv` (dense projections) | β Fixed (#457) | Merged 2026-07-30, includes `gs` parameter and fmt=4 branch |+| `moe_gemv` (routed experts) | π‘ In Progress (#587) | PR open, verified 0.81β2.13 tok/s gain on decode |+| Fused attention (`a_qabs`/`a_ctx`) | π‘ In Progress (#587) | PR open, requires `a_deqrow` to be group-aware |+| Prefill GEMM (`coli_metal_gemm`) | β Not Addressed | Gate still rejects fmt=4, out of scope of #587 |+| `bind_gemv` return value wiring | β Not Fixed | `encode_attention` ignores return at `:683,684,686,699` |++**Net effect**: Even with #587 merged, the M1 Ultra run in this document would still+experience CPU fallback on prefill (layers 0β2 dense MLPs, batched attention projections).+The 0.15 tok/s β ~0.7β0.9 tok/s improvement requires **all three paths** (A+B+C) to be fixed.++## Impact β Recalibrated, Honest++Per-token decode budget today: 3.67s expert-matmul + 1.47s attention + 1.21s disk wait ++0.56s other β 6.9s β 0.15 tok/s.++- **Fix A only (MoE experts on GPU):** removes ~3.67s/token of CPU matmul, but attention+ (1.47s) stays on CPU β roughly **~0.3 tok/s**. This is why the original MoE-only scope was+ insufficient.+- **Fixes A+B+C (all three Metal paths):** CPU keeps only disk streaming, routing and+ scatter; GPU compute is small (expert gemv is bandwidth-bound: ~11.3 GB/token of expert+ weights at unified-memory speeds β tens of ms) β the wall clock should collapse toward the+ **~1.2s/token disk-wait floor β 0.7β0.9 tok/s**.+- **Beating 2.06β2.24 tok/s additionally requires cutting disk traffic**, not more GPU: the+ run shows 69.2% expert hit rate (pin 7.4% + LRU 61.9%) with RSS 90.96 GB against a 110 GB+ budget on a 128 GB machine. Levers, in `ultra_benchmark_plan.md` Phase 5 order: higher+ `--ram` (115/120) for more LRU/pin residency, more `PIPE_WORKERS` (disk service 3.8s/token+ vs 1.21s/token blocked suggests imperfect I/O parallelism), and β since this box has a ZFS+ pool alongside the internal SSD β a `COLI_MODEL_MIRROR` second copy for aggregate read+ bandwidth (see top-level `README.md`, "Dual-SSD").+- **Fix D (NEON)** does not change the Metal-on path materially; it raises the floor for+ fallbacks and for the dense/indexer matmuls that never go to GPU. Expected speedup:+ 3β5Γ over the scalar implementation, based on `matmul_i4`'s existing NEON body.++**Risk note:** all of the above assumes expert slabs resolve to registered Metal buffers+once the format gate passes. Slabs *are* registered (`coli_metal_register` at+`colibri.c:1364`, `:1581`, `:1609`, `:1795`, `:1809`), but this has never actually executed on+this machine, so `g_moe_fb` (the `resolve()` failure counter) may well be non-zero on the+first successful build. If it is, the next thing to investigate is slab registration+lifetime, not the kernels.++## Implementation Plans++**This document is a problem statement only.** Implementation plans are in:++- [`docs/metal_fmt2.md`](metal_fmt2.md) β fmt=2 (per-row int4) conversion and benchmark plan+- [`docs/metal_fmt4.md`](metal_fmt4.md) β fmt=4 (grouped int4) Metal backend fix plan++**Files involved** (for reference in implementation plans):++**Metal backend β `c/backend_metal.mm`**+- `:16β64` `mm_gemv` kernel β needs fmt=4 branch + `gs`; final `else` must become f32-only+- `:70β99` `moe_gemv` kernel β needs fmt=4 branch + `gs`; `yout=acc*sc[o]` (`:98`) must be skipped for fmt=4+- `:134β135` `a_deqrow` β per-row `sc[row]`, needs grouped lookup+- `:136`,`:168` `a_qabs`, `a_ctx` β callers of `a_deqrow`, need `fmt`/`gs` args+- `:590β593` `coli_metal_matmul` β gate `fmt>3`; also wraps scales as `O` floats (wrong size for fmt=4)+- `:637β651` `bind_gemv` β no fmt validation (latent garbage hazard), needs `gs`+- `:665β701` `encode_attention` β ignores `bind_gemv` return values; needs `gs` plumbing+- `:716`,`:753` `coli_metal_attn_decode`, `coli_metal_layer_decode` β need `gs` params+- `:858β860` `coli_metal_gemm` β gate rejects fmt=4+- `:963` `moe_submit` β the primary gate; `:973β978` fallback counters; `:1000β1007` gemv dispatch lambda+- `:1041`,`:1067` `coli_metal_moe_block`, `coli_metal_moe_block_begin` β need `gs` params++**Metal header β `c/backend_metal.h`**+- `:36` `coli_metal_matmul`, `:69` `coli_metal_layer_decode`, `:85` `coli_metal_gemm`,+ `:106` `coli_metal_attn_decode`, `:138`/`:153` `coli_metal_moe_block`/`_begin` β signatures++**Engine β `c/colibri.c`**+- `:575β577` `matmul_qt_ex` Metal GEMM gate β excludes fmt=4+- `:591` CPU fmt=4 dispatch to `matmul_i4_grouped` (the path everything currently takes)+- `:593` IDOT gate β fmt=4 excluded by design+- `:1005` `detect_group_size` (candidates all multiples of 16), `:1027` `qt_resolve_fmt`, `:1043` fmt=2β4 promotion, `:1067` `t->gs` assignment+- `:2390β2412` `attention_rows` fused-attention gate (`kv_b.fmt==2` at `:2392`)+- `:3203β3234` `MB_BUILD` macro β `mfmt` at `:3210`, needs `mgs` + uniformity validation+- `:3242`,`:3301` MoE submit call sites β need `gs` argument+- `:3308` `metal_done` resolution+- `:4228β4279` `layer_forward` fused-layer gate (`kv_b.fmt==2` at `:4231`)+- `:4966β4974` Metal stats printing (`METAL-ATTN` only when `aok>0` β the missing-line clue)++**CPU kernels β `c/quant.h`**+- `:168β202` `matmul_i4_grouped` β AVX2 only, **no NEON** (Fix D)+- `:145β157` `matmul_i4` NEON body β the template to copy+- `:228β239` `matmul_i4_pair` NEON body β second reference+- `:513` `matmul_i4_grouped_pair` β check whether the fused gate+up grouped variant also lacks NEON (it is used at `colibri.c:279β280` when `S==1`, i.e. on the decode path)++**CUDA reference β `c/backend_cuda.cu`**+- `:101β103` `row_bytes`, `:145β156` `weight_at`, `:163β168` `absorb_scale`,+ `:195β205`/`:219` `quant_matmul`, `:844`/`:868β870` upload-time gs validation++**Tests**+- `c/tests/test_backend_metal.mm` (`:44` matmul, `:90` moe_block, `:163` attn_decode)+- `c/tests/test_gemm_largebatch.mm` (`:62`)+- `c/tests/test_i4_grouped.c` (CPU grouped reference, incl. the fused-pair variant)+- `c/tests/test_grouped_g4_cuda.cu` (CUDA fmt=4 semantics test β useful cross-check)++**Logs / plans**+- `c/3_with_pipe_no_omp_tune128ngen.log` (the analyzed run), `c/baseline.log`, `c/2_with_pipe_no_omp_tune.log`+- `ultra_benchmark_plan.md` (benchmark execution plan; Phase 1 complete but **blocked** on+ fmt=4 Metal support β see this document for the root cause)diff --git a/docs/metal_fmt2.md b/docs/metal_fmt2.mdnew file mode 100644index 00000000..a97c1ba9--- /dev/null+++ b/docs/metal_fmt2.md@@ -0,0 +1,224 @@+# Metal fmt=2 Benchmark Plan β M1 Ultra Per-Row Int4++## Objective++Convert the official GLM-5.2 fmt=4 (grouped int4) container to fmt=2 (per-row int4) and benchmark on M1 Ultra to establish a speed baseline with full Metal support.++**Rationale:** The M5 Max 2.24 tok/s record was achieved on fmt=2 weights. This plan measures what M1 Ultra's 80 GPU cores can achieve with the same format.++---++## Prerequisites++- fmt=4 model already present: `~/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp`+- Colibri built with Metal: `make clean && make METAL=1`+- Metal backend verified working: `make metal-test` passes++---++## Phase 1: fmt=2 Conversion++### 1.1 Source Model Format++**Important:** The source model `GLM-5.2-colibri-int4-g64-with-int8-mtp` is **already int4** (fmt=4 grouped, group_size=64), NOT FP8. The pathname `int4-g64` confirms this.++**No existing converter:** There is NO tool to convert fmt=4 β fmt=2 directly. The `coli convert` command only converts FP8βint4.++### 1.2 Required: fmt=4 β fmt=2 Converter++A new converter script must be created to:+1. Read existing packed int4 weights (unchanged between formats)+2. Read grouped scales from `.qs` files+3. Dequantize to approximate FP32+4. Recompute per-row scales+5. Write new `.qs` files with per-row scaling++**Size analysis:**+- Source (fmt=4 grouped): 406 GB+- Target (fmt=2 per-row): ~372 GB+- Savings: ~34 GB (from reduced scale file sizes)++**Workflow (disk-safe):**+```bash+# Option A: Convert in-place using fmt4_to_fmt2.py (recommended)+# 1. Create converter script (see c/tools/convert_fmt4_to_fmt2.py)+# 2. Run conversion from NFS source to SSD destination+python3 c/tools/convert_fmt4_to_fmt2.py \+ --indir /mnt/zfs1/noprot/mlx-lm/models/GLM-5.2-colibri-int4-g64-with-int8-mtp \+ --outdir ~/mlx-models/GLM-5.2-colibri-int4-perrow++# Option B: Delete fmt=4 from SSD first, then convert+rm -rf ~/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp+# Then run converter above+```++**Output:** New directory `~/mlx-models/GLM-5.2-colibri-int4-perrow` (~372 GB)++### 1.2 Verify Conversion++Check the banner on a short run:+```bash+COLI_METAL=1 ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow \+ --ngen 16 'test'+```++**Expected banner:** `expert@8-bit densa@8-bit` (no "cosmetic mismatch" warning)++**Expected Metal lines:**+```+METAL: blocchi GPU N | fallback CPU 0 | expert su GPU E+METAL-ATTN: layer GPU N+```++---++## Phase 2: Baseline Measurement++### 2.1 Config A β Metal Baseline++```bash+COLI_METAL=1 DIRECT=1 MTP=0 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++**Record:**+- tok/s+- wall time (s)+- expert-disk (s)+- expert-matmul (s)+- attention (s)+- attn GPU kernel (s)+- expert GPU overhead (s)+- expert hit rate (%)+- RSS (GB)++### 2.2 Config B β Default (OMP Tuning)++```bash+COLI_METAL=1 DIRECT=1 MTP=0 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++**Check:** If OMP active-spin is present, expect ~39% regression (M5 Max experience)++---++## Phase 3: Tuning Path++### 3.1 Add PIPE (Config C)++```bash+COLI_METAL=1 DIRECT=1 MTP=0 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++### 3.2 Kill OMP Spin (Config D)++```bash+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++### 3.3 Combined Optimal (Config E)++```bash+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++**Target:** β₯2.24 tok/s (match or beat M5 Max by leveraging M1 Ultra's 80 GPU cores)++---++## Phase 4: M1 Ultra Specific Tests++### 4.1 Vary PIPE_WORKERS++```bash+# Test higher worker counts for M1 Ultra's 32 CPU cores+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=12 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"++COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=16 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++### 4.2 Test MTP++```bash+# M5 Max had MTP=0; test if MTP=1 helps on M1 Ultra+COLI_METAL=1 DIRECT=1 MTP=1 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 110 \+ "Compare the myths of Lucifer and Prometheus"+```++### 4.3 Memory Pressure Test++```bash+# Test --ram 115, --ram 120 to find M1 Ultra's compression knee+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 115 \+ "Compare the myths of Lucifer and Prometheus"++COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ~/mlx-models/GLM-5.2-colibri-int4-perrow --ram 120 \+ "Compare the myths of Lucifer and Prometheus"+```++---++## Phase 5: Documentation++### 5.1 Create METAL-M1ULTRA-FMT2-REPORT.md++Mirror the M5 Max report structure:+- Setup section (hardware, model, workload, flags)+- Results table (configs A-E + M1-specific variants)+- Phase-by-phase analysis+- Comparison with M5 Max fmt=2 results+- Recommendations+- Caveats/untested levers++### 5.2 Update benchmarks.md++Add row to community benchmarks table:+```markdown+| Apple M1 Ultra (32 cores) Β· macOS Β· 128 GB unified Β· internal SSD | ~5-6 GB/s cold | Metal on (fmt=2) Β· `--ram 110` Β· COLI_NO_OMP_TUNE=1 Β· PIPE=1 | **X.XX tok/s** Β· hit YY% Β· [PR #NNN] |+```++---++## Success Criteria++1. **Primary:** Achieve β₯2.06 tok/s (match M5 Max baseline)+2. **Stretch:** Achieve β₯2.24 tok/s (beat M5 Max)+3. **Documentation:** Complete METAL-M1ULTRA-FMT2-REPORT.md with full methodology+4. **PR:** Submit report + benchmarks.md update++---++## Notes++- **Workload:** Use identical prompt ("Compare the myths of Lucifer and Prometheus") and 1024 tokens+- **Consistency:** Run each config multiple times, report best/warm-cache numbers+- **RSS:** Track RSS throughout, note when compression kicks in+- **Expert hit rate:** Record hit rate for each config+- **Thermals:** Monitor if M1 Ultra throttles during long runs (1024 tokens @ ~2 tok/s = ~8-10 min)+- **Quality caveat:** fmt=2 (per-row) has ~9pp lower quality than fmt=4 (grouped scales) β this is expected and documented in quant_ablation.py+- **Source format:** The source model is fmt=4 (grouped int4, group_size=64), NOT FP8. Conversion requires a fmt=4βfmt=2 converter, not `coli convert`.+- **Disk space:** fmt=4 source is 406 GB; fmt=2 output is ~372 GB. Delete fmt=4 from SSD before conversion to avoid 778 GB peak requirement.++---++## References++- [M5 Max Performance Report](METAL-M5MAX-PERF-REPORT.md)+- [Metal Dispatch Gap](metal_dispatch_gap.md) β fmt=4 backend gap context+- [ultra_benchmark_plan.md](ultra_benchmark_plan.md) β overall M1 Ultra benchmark plandiff --git a/docs/metal_issues.txt b/docs/metal_issues.txtnew file mode 100644index 00000000..3ff3c466--- /dev/null+++ b/docs/metal_issues.txt@@ -0,0 +1,29 @@+# contributing:+https://github.com/JustVugg/colibri/blob/main/CONTRIBUTING.md++# open issues:+https://github.com/JustVugg/colibri/issues/379 # Metal β growing the expert cache makes decode SLOWER+https://github.com/JustVugg/colibri/issues/387 # Datapoint: M5 Max 128 GB β decode-tuning A/Bs (MTP Γ cache 2Γ2): 0.4 β 2.0 tok/s at identical output+https://github.com/JustVugg/colibri/issues/585 # [Feature]: METAL Attention support for fmt:4+https://github.com/JustVugg/colibri/issues/622 # [Bug]: Metal prefill GEMM is not token-exact vs CPU on near-tie logits (S >= GEMM_MIN); decode unaffected+https://github.com/JustVugg/colibri/issues/707 # [Bug]: OpenMP spin-wait tuning is inert on macOS β and applying it costs 2.2x decode on a 32 GB host++# Closed issues:+https://github.com/JustVugg/colibri/issues/47 # Benchmark: Apple M3 Max Β· macOS Β· 128 GB Β· internal SSD (12.5 GB/s) β 0.35β0.45 tok/s; PILOT neutral; confirms #8; setup.sh libomp false-positive+https://github.com/JustVugg/colibri/issues/87 # [Performance]: M5 MAX 128GB with METAL - 1.83 tok/s+https://github.com/JustVugg/colibri/issues/103 # [Performance]: tok/s of 2.06 tok/s !!!+https://github.com/JustVugg/colibri/issues/180 # Datapoint: M3 Ultra (Mac Studio, 256 GB, 80-core GPU) β 1.45 tok/s @ 82% hit, Metal + tuned flags+https://github.com/JustVugg/colibri/issues/107 # Benchmark: Mac Mini M4 Pro (48 GB) β 0.30 tok/s with Metal backend (PR #72), 0.18 CPU-only++# Merged PRs:+https://github.com/JustVugg/colibri/pull/72 # Metal backend (Apple Silicon): experts + fused attention + full-layer GPU, token-exact - #72+https://github.com/JustVugg/colibri/pull/116 # docs: Metal backend (#72) β M5 Max performance report, OMP-spin regression + fix+https://github.com/JustVugg/colibri/pull/386 # Metal: platform-aware expert-cache defaults via honest storage probe+https://github.com/JustVugg/colibri/pull/750 # inkling: audio input, Metal expert MoE, and tuned Apple defaults++# Open PRs:+https://github.com/JustVugg/colibri/pull/587 # Metal fmt=4 grouped-int4 decode: attention + routed experts (#585)+https://github.com/JustVugg/colibri/pull/757 # inkling: shared experts to the GPU, 1.75 to 2.46 tok/s on Apple Silicon+https://github.com/JustVugg/colibri/pull/763 # metal: GPU attention for S>4 prefill (single command buffer)++diff --git a/docs/pr_116.md b/docs/pr_116.mdnew file mode 100644index 00000000..bb0f73ed--- /dev/null+++ b/docs/pr_116.md@@ -0,0 +1,23 @@+# pr_116: docs: Metal backend (#72) β M5 Max performance report, OMP-spin regression + fix++- **URL:** https://github.com/JustVugg/colibri/pull/116+- **State:** MERGED 2026-07-13 (opened 2026-07-13 by RDouglasSharp)++## Summary++Adds `docs/METAL-M5MAX-PERF-REPORT.md` β the independent M5 Max reproduction of the rebased Metal backend (#72), and the source of the Config A-E methodology the M1 Ultra plan mirrors.++- Tuned, the rebased Metal path reaches **2.24 tok/s β ~8.5% faster than the pre-rebase branch (2.06, issue #103)**, thanks to `PIPE` (new since the old base)+- The **default** rebased config regresses to **1.25 tok/s (-39%)**: the OMP hot-team active-spin (#77) pegs the CPU while it waits on the GPU, and on Apple's shared SoC power budget that throttles the Metal kernels (attention GPU kernel 76s -> 223s for identical work)+- Fix is two complementary flags: `COLI_NO_OMP_TUNE=1` restores GPU clocks; `PIPE=1 PIPE_WORKERS=8` hides the CPU->GPU dispatch latency the passive wait introduces+- Documents the 128 GB memory ceiling: `--ram 110` safe / `120` compresses+- Token-exactness note: output is non-deterministic run-to-run under the parallel config β benign FP-reduction ordering, throughput unaffected (see also #622)++## Relevance++- **Goal a (fmt_2_m1u):** **Methodology source.** Configs A-E, the Lucifer/Prometheus workload, the metrics list, and the 2.24 tok/s target all come from this report. The M1 Ultra fmt=2 plan ([metal_fmt2.md](metal_fmt2.md)) mirrors it directly.+- **Goal b (fmt_4_m1u):** Shared β same methodology for the fmt=4 benchmark after fixes.++## See also++- [issue_103](issue_103.md), [pr_72](pr_72.md), [issue_707](issue_707.md), [ultra_benchmark_plan.md](ultra_benchmark_plan.md)diff --git a/docs/pr_386.md b/docs/pr_386.mdnew file mode 100644index 00000000..cc1329bb--- /dev/null+++ b/docs/pr_386.md@@ -0,0 +1,23 @@+# pr_386: Metal: platform-aware expert-cache defaults via honest storage probe (#379)++- **URL:** https://github.com/JustVugg/colibri/pull/386+- **State:** MERGED 2026-07-31 (opened 2026-07-18 by monotophic)+- **Labels:** metal, needs-rebase++## Summary++Turns the #379/#387 findings into platform-aware defaults. On Apple Silicon with fast NVMe, stock defaults leave a large win on the table: 0.4 tok/s stock vs ~2.0 tuned (`MTP=0 CAP_RAISE=0 --ram 90 --cap 1`).++- Changes the defaults of exactly **two** knobs: `cap` and `CAP_RAISE`. MTP and RAM sizing remain manual β stock Metal runs land somewhere between 0.4 and 2.0 tok/s, not at 2.0+- Adds an **honest storage probe**: whether a volume is "fast" is measured, not assumed β buffered reads lie (page-cache hits read 23-97 GB/s; a real `F_NOCACHE` read measured 14 GB/s on the same file)+- Mechanism (from #379): Metal's per-buffer residency cost makes the engine's expert LRU anti-productive on fast NVMe; the OS page cache streams cold experts back cheaper than Metal can re-pin them+- Corroborated by #180 (M3 Ultra) and #107 (M4 Pro); explicitly invited by the maintainer (no Metal hardware in-repo)++## Relevance++- **Goal a (fmt_2_m1u):** The M1 Ultra benchmark runs on a tree that includes this PR β stock cache defaults are now platform-aware. When mirroring Configs A-E, know which knobs are already defaulted and which (`MTP`, `--ram`) remain manual.+- **Goal b (fmt_4_m1u):** Shared β same defaults context for fmt=4 runs.++## See also++- [issue_379](issue_379.md), [issue_387](issue_387.md)diff --git a/docs/pr_72.md b/docs/pr_72.mdnew file mode 100644index 00000000..c98c6cda--- /dev/null+++ b/docs/pr_72.md@@ -0,0 +1,27 @@+# pr_72: Metal backend (Apple Silicon): experts + fused attention + full-layer GPU, token-exact++- **URL:** https://github.com/JustVugg/colibri/pull/72+- **State:** MERGED 2026-07-13 (opened 2026-07-11 by codearranger)++## Summary++The foundational PR: colibri's first GPU tier for Apple Silicon, opt-in via `make glm METAL=1` + `COLI_METAL=1`. Unified memory removes the PCIe copy tax; default build untouched (`#ifdef COLI_METAL`); every GPU path falls back to CPU per-block on any fault or unsupported shape.++What runs on the GPU:++- **Routed-expert SwiGLU**, batched per block, read zero-copy from registered RAM slabs (bindless `gpuAddress`); resident experts' block submitted before missed experts' preads so disk I/O and GPU compute overlap+- **Fused decode attention** (S<=4, covers MTP verify): full MLA layer β q/kv projections, RoPE, absorption core, o_proj β in one command buffer+- **Full-layer command buffer**: in_ln -> attention -> residual -> post_ln -> shared expert -> router (f32 matvec + sigmoid) -> exact phase-A top-K on GPU+- **Prefill**: large row-batch `matmul_qt` GEMMs route to GPU above `COLI_METAL_GEMM_MIN` (default 16 rows)++Measured (M4 Max 128 GB, warm cache ~58% hit, MTP on, greedy): 0.38-0.43 tok/s Metal vs 0.30 CPU. **Token-exact**: greedy outputs byte-identical to CPU in every configuration. `make metal-test` = 16 standalone correctness tests vs CPU references.++## Relevance++- **Goal a (fmt_2_m1u):** The backend being benchmarked. Its gates (fmt=1/2 only) are why the fmt=2 track reproduces #103 without code changes.+- **Goal b (fmt_4_m1u):** The codebase being fixed β all fmt=4 work (fixes A/B/C) extends these kernels and gates.++## See also++- [pr_116](pr_116.md) β M5 Max performance report on the rebased backend+- [metal_dispatch_gap.md](metal_dispatch_gap.md) β where this backend rejects fmt=4diff --git a/docs/pr_750.md b/docs/pr_750.mdnew file mode 100644index 00000000..b042ea17--- /dev/null+++ b/docs/pr_750.md@@ -0,0 +1,22 @@+# pr_750: inkling: audio input, Metal expert MoE, and tuned Apple defaults++- **URL:** https://github.com/JustVugg/colibri/pull/750+- **State:** MERGED 2026-08-01 (opened 2026-08-01 by rgbkrk)+- **Labels:** enhancement, metal, model-support++## Summary++Adds audio input to Inkling (the small multimodal model) and speeds it up on Apple Silicon. Everything is `__APPLE__`/`COLI_METAL` guarded or opt-in; text-only Inkling and non-Apple builds unchanged.++- **Audio input (DMel):** frames embed via a small `[mel_bins*mel_vocab, D]` table + RMSNorm and *replace* `<|audio|>` placeholder embeddings; loads from a `--keep-audio` container or an `audio.safetensors` sidecar (~10 MB via HTTP range on lazy mounts). Entry points: `--audio file.dmel` CLI, 7th serve-protocol field, `"dmel"` in the oracle harness+- **Metal expert MoE:** routed-expert matmul on the Metal GPU (`COLI_METAL=1`, opt-in) for inkling+- **Three `__APPLE__` default changes** from a measured knob sweep (not a guess)++## Relevance++- **Goal a (fmt_2_m1u):** Marginal β inkling-specific, but its tuned Apple defaults touch shared `__APPLE__` build defaults that the GLM benchmark binary also sees. Worth knowing it landed before the M1 Ultra runs.+- **Goal b (fmt_4_m1u):** Not directly relevant; its follow-up [pr_757](pr_757.md) is more interesting (adds an fmt=5 branch to `moe_gemv` β a template for the fmt=4 branch).++## See also++- [pr_757](pr_757.md) β follow-up: shared experts on GPUdiff --git a/docs/pr_757.md b/docs/pr_757.mdnew file mode 100644index 00000000..9a148c84--- /dev/null+++ b/docs/pr_757.md@@ -0,0 +1,22 @@+# pr_757: inkling: shared experts to the GPU, 1.75 to 2.46 tok/s on Apple Silicon++- **URL:** https://github.com/JustVugg/colibri/pull/757+- **State:** OPEN (opened 2026-08-01 by rgbkrk)+- **Labels:** performance, metal++## Summary++Follow-up to #750 (inkling), but with two techniques worth knowing for GLM Metal work. Same validation discipline: greedy output token-identical through eos, tiny multimodal oracle exact at every step. Inkling-Small on Apple Silicon: **1.75 -> 2.46 tok/s decode, 10.3 -> 6.4 s prefill**.++- **Commit 1 β overlap shared experts with the last GPU round:** routed and shared experts are independent given the layer input; the last routed round submits with `moe_block_begin`/`_end` and the CPU computes shared experts while the GPU is in flight. `shared_done` flag guards double-compute on GPU fault. No new math, no new kernel. 1.75 -> 1.96 tok/s.+- **Commit 2 β shared experts on GPU as a bf16 block:** `moe_gemv` gains **fmt=5** (raw bf16 rows, no scales, dequant is a 16-bit shift into the upper half of an f32). Shared-expert weights move into one page-aligned slab per sparse layer; `Wt` handles repoint into the slab so the CPU fallback reads the same bytes. One fmt=5 block submitted per MoE layer before the routed rounds. Shared experts drop 8.0s -> 0.1s on the probe. 1.96 -> 2.46 tok/s.+- The fmt=5 branch is additive; glm/kimi call sites never pass fmt=5 (dead code for them). `INK_METAL_SHARED=0` opts out.++## Relevance++- **Goal a (fmt_2_m1u):** Perf technique reference β CPU/GPU overlap of shared experts and GPU-resident shared experts are candidate levers for beating 2.24 tok/s on M1 Ultra (GLM has 1 shared expert/layer currently on CPU).+- **Goal b (fmt_4_m1u):** **Code template** β the fmt=5 branch shows exactly how to add a new format branch to `moe_gemv` without touching existing paths, which is what Fix A (fmt=4 branch) needs.++## See also++- [pr_750](pr_750.md), [metal_fmt4.md](metal_fmt4.md)diff --git a/docs/pr_763.md b/docs/pr_763.mdnew file mode 100644index 00000000..c55506db--- /dev/null+++ b/docs/pr_763.md@@ -0,0 +1,23 @@+# pr_763: metal: GPU attention for S>4 prefill (single command buffer)++- **URL:** https://github.com/JustVugg/colibri/pull/763+- **State:** OPEN (opened 2026-08-01 by RDouglasSharp)+- **Labels:** performance, metal, quality++## Summary++Lifts the `S<=4` cap on the Metal fused-attention path (`coli_metal_attn_decode`) so large-S **prefill** attention runs on the GPU in a single command buffer, cutting prefill attention time ~4x with output consistent with the CPU reference.++- **Off by default** β `COLI_METAL_PREFILL=1` opts in β because GPU prefill can pick a different token on near-tie logits (the #622 family), so a greedy stream is not guaranteed bit-identical to CPU+- `c/colibri.c`: fused-attention gate no longer hard-caps at S<=4; engageable for any S when Metal absorption is active; adds a cumulative timer to the `[prefill]` progress line+- `c/backend_metal.mm`: projections -> attention core -> output GEMV in one command buffer ordered by memory barriers (same structure as the token-exact S<=4 path); single `a_score` dispatch is `S*AHEADS*T` threads, guarded under ~2^30, CPU fallback for very large prompts+- `docs/metal.md` documents the flag. CPU and CUDA paths unchanged++## Relevance++- **Goal a (fmt_2_m1u):** Optional prefill speedup for the benchmark (~4x prefill attention) β with the documented near-tie caveat; keep it off for an exact-reproduction claim, or on for a max-speed claim.+- **Goal b (fmt_4_m1u):** Interacts with the same fused-attention gate that Fix C relaxes for fmt=4 β if both land, the fmt=4 gate change and the S>4 change must be reconciled in the same `if` conditions.++## See also++- [issue_622](issue_622.md), [pr_72](pr_72.md), [metal_fmt4.md](metal_fmt4.md)diff --git a/docs/ultra_benchmark_plan.md b/docs/ultra_benchmark_plan.mdnew file mode 100644index 00000000..a5e19006--- /dev/null+++ b/docs/ultra_benchmark_plan.md@@ -0,0 +1,235 @@+# M1 Ultra Metal Benchmark Plan++## Objective++Benchmark colibri on M1 Ultra (128GB unified memory) using the same methodology as the M5 Max report, with the goal of meeting or exceeding the M5 Max's 2.06 tok/s baseline.++**Critical Context:** Two separate benchmark tracks exist due to the fmt=4 Metal backend gap:+- **fmt=2 (per-row int4):** Full Metal support available β see [`metal_fmt2.md`](metal_fmt2.md)+- **fmt=4 (grouped int4):** Metal backend fixes required β see [`metal_fmt4.md`](metal_fmt4.md)++---++## Hardware Context++- **Machine:** Apple M1 Ultra (2Γ M1 Max dies, 20+10 CPU cores, 48+32 GPU cores)+- **Memory:** 128GB unified (fabric-connected, ~512GB/s bandwidth per die)+- **SSD:** Internal (typically 7GB/s sequential, ~3-4GB/s random 4K)+- **Target:** β₯2.06 tok/s (M5 Max baseline), ideally 2.2+ tok/s++---++## Track Selection++### fmt=2 Track (Per-Row Int4)++**Status:** Ready to benchmark immediately++**Model:** Converted from official fmt=4 container using `--group-size 0`++**Advantage:** Full Metal support (no code changes needed)++**Tradeoff:** ~9pp lower quality than fmt=4 (grouped scales)++**Plan:** [`docs/metal_fmt2.md`](metal_fmt2.md)++---++### fmt=4 Track (Grouped Int4)++**Status:** Requires Metal backend implementation++**Model:** Official GLM-5.2-colibri-int4-g64-with-int8-mtp++**Advantage:** Higher quality (official container format)++**Requirement:** Implement fixes A+B+C from [`metal_dispatch_gap.md`](metal_dispatch_gap.md)++**Plan:** [`docs/metal_fmt4.md`](metal_fmt4.md)++---++## Prerequisites (Both Tracks)++### 1. Model Availability++- **fmt=4:** `~/mlx-models/GLM-5.2-colibri-int4-g64-with-int8-mtp` (already present)+- **fmt=2:** Convert from fmt=4 using plan in [`metal_fmt2.md`](metal_fmt2.md#phase-1)++### 2. Build with Metal++```bash+make clean && make METAL=1+```++### 3. Verify Metal Backend++```bash+make metal-test+```++All 30+ kernel tests should pass (int8/int4/int2 matmul, MoE block, large-batch GEMM, fused attention).++**Note:** Current tests cover fmt=1/2/3 only. fmt=4 tests added as part of [`metal_fmt4.md`](metal_fmt4.md#phase-4).++---++## Benchmark Methodology (Both Tracks)++### Standard Configurations++Mirror M5 Max Configs AβE from [`METAL-M5MAX-PERF-REPORT.md`](METAL-M5MAX-PERF-REPORT.md):++| Config | Flags | Purpose |+|--------|-------|---------|+| **A** | `COLI_METAL=1 DIRECT=1 MTP=0` | Metal baseline |+| **B** | Default (no extra flags) | Check for OMP spin regression |+| **C** | `+ PIPE=1 PIPE_WORKERS=8` | Add I/O overlap |+| **D** | `+ COLI_NO_OMP_TUNE=1` | Kill OMP active-spin |+| **E** | `D + C` | Combined optimal |++### Workload++- **Prompt:** "Compare the myths of Lucifer and Prometheus"+- **Tokens:** 1024+- **Rationale:** Matches M5 Max methodology for comparability++### Metrics to Record++For each configuration:+- tok/s+- wall time (s)+- expert-disk (s)+- expert-matmul (s)+- attention (s)+- attn GPU kernel (s)+- expert GPU overhead (s)+- expert hit rate (%)+- RSS (GB)++---++## M1 Ultra Specific Tests++### PIPE_WORKERS Scaling++M1 Ultra has 32 CPU cores (vs M5 Max's 18). Test higher worker counts:++```bash+# Beyond PIPE_WORKERS=8+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=12 ./coli run ...+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=16 ./coli run ...+```++### MTP (Multi-Token Prediction)++M5 Max report used MTP=0. Test if MTP=1 helps on M1 Ultra:++```bash+COLI_METAL=1 DIRECT=1 MTP=1 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 ./coli run ...+```++### Memory Pressure++Find M1 Ultra's compression knee:++```bash+# Test --ram 115, --ram 120+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ... --ram 115 ...+COLI_METAL=1 DIRECT=1 MTP=0 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=8 \+ ./coli run --model ... --ram 120 ...+```++---++## Documentation++### 6.1 Create Performance Report++**fmt=2 track:** [`METAL-M1ULTRA-FMT2-REPORT.md`](metal_fmt2.md#phase-5)++**fmt=4 track:** [`METAL-M1ULTRA-FMT4-REPORT.md`](metal_fmt4.md#phase-6)++Mirror M5 Max report structure:+- Setup section (hardware, model, workload, flags)+- Results table (configs A-E + M1-specific variants)+- Phase-by-phase analysis+- Recommendations+- Caveats/untested levers++### 6.2 Update benchmarks.md++Add row to community benchmarks table:++```markdown+| Apple M1 Ultra (32 cores) Β· macOS Β· 128 GB unified Β· internal SSD | ~5-6 GB/s cold | Metal on Β· `--ram 110` Β· COLI_NO_OMP_TUNE=1 Β· PIPE=1 | **X.XX tok/s** Β· hit YY% Β· [PR #NNN] |+```++**Note:** Create separate rows for fmt=2 and fmt=4 results if both tracks complete.++---++## Success Criteria++### fmt=2 Track++1. **Primary:** Achieve β₯2.06 tok/s (match M5 Max baseline)+2. **Stretch:** Achieve β₯2.24 tok/s (beat M5 Max)+3. **Documentation:** Complete METAL-M1ULTRA-FMT2-REPORT.md+4. **PR:** Submit report + benchmarks.md update++### fmt=4 Track++1. **Primary:** Metal dispatch confirmed (METAL-ATTN and METAL lines present)+2. **Stretch:** Achieve β₯2.06 tok/s (match M5 Max with fmt=4)+3. **Documentation:** Complete METAL-M1ULTRA-FMT4-REPORT.md+4. **PR:** Submit report + benchmarks.md update++---++## Execution Order++### Recommended Sequence++1. **fmt=2 track first** β Immediate benchmark, establishes speed baseline+2. **fmt=4 track second** β After backend fixes, measures quality-correct performance++**Rationale:** fmt=2 requires no code changes and can be benchmarked immediately. fmt=4 requires implementing fixes A+B+C from [`metal_dispatch_gap.md`](metal_dispatch_gap.md).++### Parallel Execution++If resources allow, both tracks can run in parallel:+- fmt=2: Benchmarking while fmt=4: Implementation in progress++---++## References++### Core Documents++- [`metal_dispatch_gap.md`](metal_dispatch_gap.md) β fmt=4 backend gap problem statement+- [`metal_fmt2.md`](metal_fmt2.md) β fmt=2 conversion and benchmark plan+- [`metal_fmt4.md`](metal_fmt4.md) β fmt=4 backend fix implementation plan++### Methodology Reference++- [`METAL-M5MAX-PERF-REPORT.md`](METAL-M5MAX-PERF-REPORT.md) β M5 Max benchmark methodology+- [`README.md`](../README.md) β Dual-SSD, PIPE, COLI_NO_OMP_TUNE tuning guidance++### Related Issues++- **#457** β Metal fmt=4 GEMV support (MERGED)+- **#585** β METAL Attention support for fmt:4 (OPEN)+- **#587** β Metal fmt=4 decode: attention + routed experts (OPEN, needs-rebase)++---++## Next Steps++**Choose your track:**++1. **fmt=2 (immediate):** Follow [`metal_fmt2.md`](metal_fmt2.md) β convert model and benchmark+2. **fmt=4 (backend fix):** Follow [`metal_fmt4.md`](metal_fmt4.md) β implement fixes A+B+C, then benchmark++**Both tracks contribute to the same goal:** Understanding M1 Ultra's performance potential with colibri, with different tradeoffs between speed (fmt=2) and quality (fmt=4).