Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

tests: efficiency suite β€” tiny-model regression gates + full-model optimization dossier

woolcoxmd43b54534f4f
diff --git a/c/Makefile b/c/Makefile
index af173573..12899688 100644
--- a/c/Makefile
+++ b/c/Makefile
@@ -363,6 +363,29 @@ test-python:
test: test-c test-python
+# --- Efficiency / regression suite (issue: "test the program for inefficiencies") ---
+# The tiny-model assertions live in test_inefficiency.py and run as part of
+# test-python (they're discovered by the test_*.py glob). These targets are
+# convenience entry points; the opt-in full-model report is NEVER in `make test`.
+#
+# make efficiency tiny-model asserted regression tests (CPU; part of test-python)
+# make efficiency-cuda the CUDA-path tests (requires a CUDA build β€” see below)
+# make efficiency-report opt-in full-model 🟒/πŸ”΄ diagnostic, never fails CI
+#
+# CUDA build (Windows): the CUDA tests need a host built with -DCOLI_CUDA plus
+# the runtime DLL. Do this FIRST β€” note CUDA_DLL=1 on BOTH the host and the
+# rule below, or `make glm.exe` will rebuild a CPU-only host and overwrite it:
+# make clean && make glm.exe CUDA_DLL=1 && make cuda-dll
+# The tests auto-skip with a clear message if the host is CPU-only.
+efficiency: test-python
+ $(PYTHON) -m unittest tests.test_inefficiency -v
+
+efficiency-cuda:
+ $(PYTHON) -m unittest tests.test_inefficiency.TinyCudaEfficiencyTest -v
+
+efficiency-report:
+ $(PYTHON) tests/test_efficiency_report.py
+
# Local validation: one portable CPU build and dependency-free tests.
check:
$(MAKE) clean
diff --git a/c/tests/README_efficiency.md b/c/tests/README_efficiency.md
new file mode 100644
index 00000000..1f8d339a
--- /dev/null
+++ b/c/tests/README_efficiency.md
@@ -0,0 +1,98 @@
+# Efficiency suite β€” regression tests + optimization dossier
+
+Two layers:
+
+1. **`test_inefficiency.py`** β€” tiny-model *asserted* regression tests. Fast
+ (~0.15s/run), gate CI, catch breakage. Run as part of `make test`.
+2. **`test_efficiency_report.py`** β€” an *opt-in optimization dossier* for a real
+ model. Runs every instrumentation flag, prints a 9-section report answering
+ *what is doing what, when, with what, is it inefficient, how to improve*.
+ Never fails CI (it's a report, not a gate).
+
+## The dossier (what you run when optimizing)
+
+```bash
+# CPU-only (safe, fast to validate):
+COLI_EFFICIENCY_MODEL=../glm52_i4_g64 make efficiency-report
+
+# CUDA (dense + expert tiers β€” needs a CUDA build, see below):
+COLI_EFFICIENCY_MODEL=../glm52_i4_g64 COLI_EFFICIENCY_CUDA=1 make efficiency-report
+```
+
+It turns ON every observability flag the engine supports β€” `PROF=1`,
+`COLI_CUDA_PROFILE=1`, `CACHE_ROUTE=1` (auto-unlocks `route_agree`/`route_kl`),
+`DISK_SPLIT=1`, `LOOKA=1` β€” so nothing the engine can tell you is left dark.
+None of these change the computed output; they only add telemetry.
+
+The 9 sections, and the question each answers:
+
+| Β§ | section | answers |
+|---|---|---|
+| 1 | PROVENANCE | what is running, on what CPU/backend, with what effective config |
+| 2 | THROUGHPUT | tok/s + forward-latency p50/p90/p99/max (is the tail healthy?) |
+| 3 | WHERE TIME GOES | the 5 PROFILE phases as % of decode + absolute seconds + verdict |
+| 3a | ATTENTION BREAKDOWN | attention split into projection/RoPE, score-softmax-value, output |
+| 4 | EXPERT CACHE | hit %, experts-loaded/token vs baseline topk |
+| 5 | DISK I/O | GB fetched, MB/token, GB/s, read-service vs felt-wait, phase split |
+| 5a | DISK-LOAD SPLIT | loads by decode phase (draft/absorb/verify) + MTP-vs-main bytes |
+| 6 | ROUTING QUALITY | route_agree %, route_kl, cache swaps |
+| 6a | ROUTING PREDICTABILITY | LOOKAHEAD recall per predictor (which prefetch wins) |
+| 7 | SPECULATION | tokens/forward, MTP acceptance % |
+| 8 | GPU TIERS | resident tensors, expert tier (count/GB/calls), H2D/kernel/D2H ms |
+
+Every line that crosses an advisory threshold is marked `[FLAG]` with the
+concrete lever to pull (raise RAM_GB, add PIN_GB, try DIRECT=1, lower CTX, …),
+and all flags repeat in a summary at the end.
+
+## Tunable thresholds
+
+The `IS IT INEFFICIENT?` lines are advisory constants at the top of
+`test_efficiency_report.py`:
+
+| constant | default | meaning |
+|---|---|---|
+| `DISK_WAIT_DOMINANT` | 0.40 | >40% decode waiting on expert reads β†’ I/O-bound |
+| `LOW_HIT_RATE` | 0.30 | <30% cache hit β†’ thrashing |
+| `LOW_ROUTE_AGREE` | 0.80 | <80% routing overlap β†’ prefetch guessing wrong |
+| `HIGH_TAIL_RATIO` | 3.0 | p99 > 3Γ— p50 β†’ decode stalls |
+| `LOW_MTP_ACCEPT` | 0.20 | <20% MTP acceptance β†’ draft decoder is dead weight |
+
+The tiny-model asserted floors live in `tools/efficiency.py` (`TINY_TOK_S_FLOOR`,
+`MAX_DISK_WAIT_SHARE`, `MIN_CPU_CUDA_AGREEMENT`).
+
+## The regression tests (what gates CI)
+
+`test_inefficiency.py` runs on the bundled `glm_tiny` model and asserts:
+
+- telemetry parses (no format drift)
+- tiny tok/s β‰₯ floor (throughput regression)
+- PROFILE phases present and non-negative (accounting sanity)
+- disk-wait not dominant on a resident model (I/O-path regression)
+- CPU determinism (two greedy runs agree)
+- **CUDA** (skip unless CUDA built): init path, dense uploads VRAM, CPU-vs-CUDA
+ argmax agreement β‰₯ 70% (kernel-correctness guard)
+
+```bash
+make efficiency # tiny CPU tests
+make efficiency-cuda # tiny CUDA tests (needs CUDA build)
+```
+
+## CUDA build prerequisite
+
+The default `make glm.exe` builds **without** CUDA. The CUDA tests and the CUDA
+dossier need a host built with `-DCOLI_CUDA` plus the runtime DLL:
+
+```bash
+make clean && make glm.exe CUDA_DLL=1 && make cuda-dll
+```
+
+`make efficiency-cuda` auto-skips with a clear message if the host is CPU-only
+(it scans the binary for the "CPU-only" marker the engine embeds).
+
+## Files
+
+- `tools/efficiency.py` β€” shared harness: `parse_run()` (captures every
+ telemetry signal), `run_engine()`, thresholds. Reuses `PROFILE_RE`/`SPEED_RE`
+ from `tools/benchmark_cuda_fixture.py`.
+- `tests/test_inefficiency.py` β€” tiny-model asserted tests (CPU + CUDA).
+- `tests/test_efficiency_report.py` β€” the opt-in optimization dossier.
diff --git a/c/tests/test_efficiency_report.py b/c/tests/test_efficiency_report.py
new file mode 100644
index 00000000..ba01b131
--- /dev/null
+++ b/c/tests/test_efficiency_report.py
@@ -0,0 +1,315 @@
+#!/usr/bin/env python3
+"""Exhaustive optimization dossier for a colibri engine run.
+
+This is NOT a pass/fail test. It runs the engine with every instrumentation flag
+on (PROF, COLI_CUDA_PROFILE, CACHE_ROUTE, DISK_SPLIT, LOOKA) and prints a section-
+by-section report answering, for each subsystem:
+
+ WHAT is doing it β€” which phase/kernel/tier
+ WHEN it is doing it β€” how much of decode wall-time it owns
+ WITH WHAT β€” the config/weights/tier it used
+ IS IT INEFFICIENT? β€” a verdict, with the threshold
+ HOW TO IMPROVE β€” the concrete knob, named
+
+Activation (opt-in only β€” NOT in `make test`):
+ COLI_EFFICIENCY_MODEL=<model_dir> python tests/test_efficiency_report.py
+
+Optional env:
+ COLI_EFFICIENCY_CUDA=1 also exercise the CUDA dense/expert tiers
+ COLI_EFFICIENCY_NGEN=N decode tokens (default 24)
+ COLI_EFFICIENCY_RAM_GB=N RAM budget (default 28)
+ COLI_EFFICIENCY_VRAM_GB=N CUDA expert-tier budget GB (default 4)
+ COLI_EFFICIENCY_PROMPT=... prompt (default: a code-gen prompt)
+
+Exit code is always 0 (it's a dossier, not a gate). Lines marked FLAG point at
+the most likely lever to move tok/s for the observed bottleneck.
+"""
+import os
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from tools.efficiency import run_engine, disk_wait_share # noqa: E402
+
+
+# --- advisory thresholds (the "IS IT INEFFICIENT?" lines) ---
+DISK_WAIT_DOMINANT = 0.40 # >40% decode waiting on expert reads -> I/O-bound
+LOW_HIT_RATE = 0.30 # <30% cache hit -> thrashing (cap too small)
+LOW_ROUTE_AGREE = 0.80 # <80% routing overlap -> prefetch is guessing wrong
+HIGH_TAIL_RATIO = 3.0 # p99 > 3x p50 -> decode stalls (I/O hiccups / KV grow)
+LOW_MTP_ACCEPT = 0.20 # <20% MTP acceptance -> draft decoder is dead weight
+VRAM_WASTE_CALLS = 0 # experts pinned in VRAM but 0 calls served
+
+
+def _flag(ok): return "OK " if ok else "FLAG"
+
+
+def _bar(frac, width=24):
+ """A simple ASCII bar for share visualization."""
+ n = max(0, min(width, round(frac * width)))
+ return "#" * n + "." * (width - n)
+
+
+def _line(label, value, flag=None, note=""):
+ tag = f" [{flag}]" if flag else ""
+ print(f" {label:<22} {value}{tag} {note}" if note else f" {label:<22} {value}{tag}")
+
+
+def main() -> int:
+ model = os.environ.get("COLI_EFFICIENCY_MODEL")
+ if not model:
+ print(__doc__)
+ print("\nNot activated: set COLI_EFFICIENCY_MODEL=<model_dir> to run.")
+ return 0
+ model = str(Path(model).resolve())
+ if not Path(model).is_dir():
+ print(f"ERROR: {model} is not a directory", file=sys.stderr)
+ return 0
+
+ ngen = int(os.environ.get("COLI_EFFICIENCY_NGEN", "24"))
+ ram_gb = os.environ.get("COLI_EFFICIENCY_RAM_GB", "28")
+ vram_gb = os.environ.get("COLI_EFFICIENCY_VRAM_GB", "4")
+ prompt = os.environ.get(
+ "COLI_EFFICIENCY_PROMPT",
+ "Write a Python function that computes the factorial of a number. "
+ "Include error handling and a docstring.")
+ use_cuda = os.environ.get("COLI_EFFICIENCY_CUDA") == "1"
+
+ # Turn ON every instrumentation flag so the dossier has maximum detail.
+ # These are all observability toggles (PROF/COLI_CUDA_PROFILE/CACHE_ROUTE/
+ # DISK_SPLIT/LOOKA); none change the computed output.
+ overlay = dict(
+ NGEN=str(ngen), TEMP="0", RAM_GB=ram_gb, PROMPT=prompt,
+ PROF="1", CACHE_ROUTE="1", DISK_SPLIT="1", LOOKA="1", ROUTE_AGREE="1",
+ )
+ if use_cuda:
+ overlay.update(COLI_CUDA="1", COLI_GPU="0", CUDA_DENSE="1",
+ COLI_CUDA_PROFILE="1", CUDA_EXPERT_GB=vram_gb)
+
+ print("=" * 78)
+ print(f"OPTIMIZATION DOSSIER β€” {Path(model).name}")
+ print(f" mode : {'CUDA (dense+expert tiers)' if use_cuda else 'CPU-only'} "
+ f"ngen : {ngen} ram : {ram_gb} GB" +
+ (f" vram : {vram_gb} GB" if use_cuda else ""))
+ print("=" * 78)
+
+ t0 = time.time()
+ t, proc = run_engine(overlay, snap=model, timeout=3600.0)
+ wall = time.time() - t0
+ flags = [] # collected FLAG lines for the summary
+
+ print(f"\n[0] RUN")
+ _line("wall clock", f"{wall:.0f}s")
+ _line("exit code", proc.returncode,
+ None if proc.returncode == 0 else "FLAG",
+ "" if proc.returncode == 0 else "non-zero exit")
+ if proc.returncode != 0:
+ print(" stderr tail:")
+ for ln in proc.stderr.strip().splitlines()[-8:]:
+ print(f" {ln}")
+ return 0
+
+ # ---------------------------------------------------------------- [1] WHO ----
+ print(f"\n[1] PROVENANCE β€” what is running, on what, with what config")
+ if t.get("machine"):
+ m = t["machine"]
+ _line("CPU", m["cpu"])
+ _line("cores / omp", f"{m['cores']} cores")
+ _line("backend", m["backend"])
+ if t.get("load"):
+ ld = t["load"]
+ _line("model load time", f"{ld['load_s']:.2f}s")
+ _line("resident dense", f"{ld['resident_dense_mb']:.1f} MB")
+ _line("layers / experts", f"{ld['layers']} layers, {ld['experts']} experts")
+ _line("MTP", f"{ld['mtp_status']} (draft={ld['draft']})")
+ if t.get("config_str"):
+ _line("resolved config", t["config_str"])
+ print(" (this is the EFFECTIVE config after auto-budgeting β€” not your env verbatim)")
+
+ # ---------------------------------------------------------------- [2] SPEED --
+ print(f"\n[2] THROUGHPUT β€” is it fast, is the tail healthy")
+ if t.get("tok_s") is not None:
+ _line("tok/s", f"{t['tok_s']:.3f}")
+ else:
+ flags.append("throughput line missing β€” engine output format may have changed")
+ if t.get("latency"):
+ la = t["latency"]
+ _line("decode forwards", f"{int(la['forwards'])}")
+ _line("p50 / p90", f"{la['p50_ms']:.1f} / {la['p90_ms']:.1f} ms")
+ _line("p99 / max", f"{la['p99_ms']:.1f} / {la['max_ms']:.1f} ms")
+ tail_ok = la["p99_ms"] <= HIGH_TAIL_RATIO * la["p50_ms"]
+ _line("tail ratio (p99/p50)", f"{la['p99_ms']/max(la['p50_ms'],1e-9):.2f}x",
+ _flag(tail_ok),
+ "high tail = decode stalls (I/O hiccups, KV growth, re-pin)")
+ if not tail_ok:
+ flags.append(f"tail latency p99={la['p99_ms']:.1f}ms >> p50={la['p50_ms']:.1f}ms "
+ "(look for REPIN swaps or disk stalls)")
+
+ # ---------------------------------------------------------------- [3] TIME ---
+ print(f"\n[3] WHERE TIME GOES β€” what is doing it, when (share of decode)")
+ ts = t.get("time_shares")
+ prof = t.get("profile")
+ if ts:
+ order = [("io", "expert-disk I/O", DISK_WAIT_DOMINANT, "the cache is too small / disk is slow"),
+ ("matmul", "expert matmul", 0.40, "compute-bound; more cores or a GPU expert tier"),
+ ("attention", "attention", 0.35, "context length is the cost; lower CTX"),
+ ("head", "lm_head", 0.10, "vocab projection; unusual to dominate"),
+ ("other", "other", 0.30, "scheduling / KV bookkeeping overhead")]
+ for key, name, thresh, lever in order:
+ f = ts[key]
+ ok = f < thresh
+ _line(name, f"{f:5.0%} {_bar(f)}", _flag(ok),
+ "" if ok else f"->{lever}")
+ if not ok:
+ flags.append(f"{name} dominates ({f:.0%}) -> {lever}")
+ if t.get("verdict"):
+ print(f" engine verdict : {t['verdict']}")
+ elif prof:
+ print(" (no [PROF] time shares β€” set PROF=1 for phase percentages)")
+ if prof:
+ print(" absolute seconds :")
+ for k in ("disk", "expert_matmul", "attention", "lm_head", "other"):
+ _line(k, f"{prof[k]:.3f}s")
+
+ # attention sub-breakdown: how is attention being read
+ ab = t.get("attn_breakdown")
+ if ab:
+ print(f"\n[3a] ATTENTION BREAKDOWN β€” how the attention phase is spent")
+ atot = sum(ab.values()) or 1.0
+ for k, label in (("proj_rope", "projection + RoPE"),
+ ("score_sm_value", "score-softmax-value"),
+ ("out_proj", "output projection")):
+ _line(label, f"{ab[k]:.3f}s ({ab[k]/atot:.0%} of attn)")
+
+ # ---------------------------------------------------------------- [4] CACHE --
+ print(f"\n[4] EXPERT CACHE β€” is the cache efficient")
+ hit = t.get("hit_pct")
+ if hit is not None:
+ ok = hit >= LOW_HIT_RATE * 100
+ _line("hit rate", f"{hit:.1f}%", _flag(ok),
+ "" if ok else "<30% = thrashing; raise RAM_GB or cap")
+ if not ok:
+ flags.append(f"cache hit {hit:.1f}% is low -> raise RAM_GB (or cap), add PIN_GB")
+ el = t.get("experts_loaded")
+ if el:
+ per_tok = el["per_tok"]
+ _line("experts loaded/token", f"{per_tok:.1f}")
+ _line(" per-layer", f"{el['per_layer']:.2f} across {el['n_sparse_layers']} sparse layers")
+ _line(" baseline", f"topk={el['baseline_topk']} active experts/token")
+ base_topk = el["baseline_topk"]
+ if base_topk > 0 and per_tok > 2 * base_topk:
+ flags.append(f"loading {per_tok:.0f} experts/token vs topk={base_topk} "
+ "-> redundant I/O; cache is re-fetching evicted experts")
+
+ # ---------------------------------------------------------------- [5] DISK ---
+ print(f"\n[5] DISK I/O β€” is I/O the bottleneck, and where")
+ eio = t.get("expert_io")
+ if eio:
+ _line("total fetched", f"{eio['gb_fetched']:.3f} GB")
+ _line("per token", f"{eio['mb_per_tok']:.1f} MB/token")
+ _line("disk throughput", f"{eio['gb_per_s']:.2f} GB/s over the run")
+ _line("read service", f"{eio['read_service_s']:.2f}s (on I/O threads)")
+ _line("felt wait", f"{eio['felt_wait_s']:.2f}s (stall compute felt)")
+ if eio["felt_wait_s"] > eio["read_service_s"] * 0.5 and eio["read_service_s"] > 0:
+ flags.append("felt wait is a large fraction of read service -> PIPE=1 may not be "
+ "overlapping fully, or DIRECT=1 on NVMe")
+ ds = t.get("disk_split")
+ if ds:
+ print(f"\n[5a] DISK-LOAD SPLIT β€” which decode phase reads the bytes")
+ _line("draft phase", f"{ds['draft']} loads")
+ _line("absorb phase", f"{ds['absorb']} loads")
+ _line("verify/main", f"{ds['verify_main']} loads")
+ _line("MTP-layer bytes", f"{ds['mtp_loads']} loads, {ds['mtp_gb']:.2f} GB")
+ _line("main-layer bytes", f"{ds['main_loads']} loads, {ds['main_gb']:.2f} GB")
+ if ds.get("mtp_bytes_pct") is not None:
+ _line("MTP share of bytes", f"{ds['mtp_bytes_pct']:.1f}%")
+ share = disk_wait_share(t)
+ if share is not None:
+ ok = share < DISK_WAIT_DOMINANT
+ _line("disk-wait share", f"{share:.0%}", _flag(ok),
+ "" if ok else "I/O-bound (see levers in [3])")
+
+ # ---------------------------------------------------------------- [6] ROUTE --
+ print(f"\n[6] ROUTING QUALITY β€” is the router / prefetch accurate")
+ ra = t.get("route_agree")
+ if ra:
+ ok = ra["agree_pct"] >= LOW_ROUTE_AGREE * 100
+ _line("route_agree", f"{ra['agree_pct']:.1f}% overlap with true top-K",
+ _flag(ok),
+ "" if ok else "prefetch is guessing wrong; CACHE_ROUTE params may need tuning")
+ _line("route_kl", f"{ra['kl']:.4f} mean KL (lower = closer to true routing)")
+ if not ok:
+ flags.append(f"route_agree {ra['agree_pct']:.1f}% low -> tune ROUTE_J/M/P, "
+ "or prefetch is hurting more than helping")
+ sw = t.get("swap")
+ if sw:
+ _line("cache swaps", f"{sw['swaps']}/{sw['slots']} ({sw['pct']:.1f}%)",
+ None, "high swap = churn between turns")
+ la = t.get("lookahead")
+ if la:
+ print(f"\n[6a] ROUTING PREDICTABILITY β€” recall of true experts in predicted top-8")
+ print(" (which predictor should drive prefetch? highest recall wins)")
+ for row in la:
+ _line(row["predictor"][:34], f"{row['pct']:5.1f}% ({row['hit']}/{row['tot']})")
+
+ # ---------------------------------------------------------------- [7] SPEC ---
+ print(f"\n[7] SPECULATION β€” is the draft decoder pulling weight")
+ sp = t.get("speculation")
+ if sp:
+ _line("tokens/forward", f"{sp['tok_per_fw']:.2f} (>1.0 means speculation helps)")
+ _line("forwards/tokens", f"{sp['forwards']} forwards for {sp['tokens']} tokens")
+ # Speculation helps only if acceptance is high enough that tok/forward > 1.
+ # tok_per_fw already == 1.0 when nothing verifies, so judge by acceptance.
+ ok = sp["mtp_accept_pct"] >= LOW_MTP_ACCEPT * 100
+ _line("MTP acceptance", f"{sp['mtp_accept_pct']:.0f}%", _flag(ok),
+ "" if ok else "<20% -> drafts rarely verify; DRAFT=0 may be faster")
+ if not ok:
+ flags.append(f"MTP acceptance {sp['mtp_accept_pct']:.0f}% low -> "
+ "drafts cost more I/O than they save; try DRAFT=0")
+
+ # ---------------------------------------------------------------- [8] GPU ----
+ print(f"\n[8] GPU TIERS β€” is the GPU actually used")
+ cuda = t.get("cuda") or {}
+ if not cuda.get("enabled"):
+ print(" (CUDA not enabled β€” CPU-only run)")
+ else:
+ if cuda.get("resident_tensors") is not None:
+ _line("resident dense tensors", f"{cuda['resident_tensors']} tensors, "
+ f"{cuda['resident_gb']:.2f} GB")
+ if cuda.get("expert_count") is not None:
+ waste = cuda["calls_served"] <= VRAM_WASTE_CALLS
+ _line("expert tier", f"{cuda['expert_count']} experts pinned "
+ f"({cuda['expert_gb']:.2f} GB)", _flag(not waste))
+ _line(" calls served", f"{cuda['calls_served']} from VRAM",
+ _flag(not waste),
+ "" if not waste else "WASTE: pinned but never routed -> lower CUDA_EXPERT_GB")
+ if waste:
+ flags.append("VRAM expert tier has 0 calls served -> experts pinned but unused; "
+ "PIN stats may not match this workload")
+ if cuda.get("groups"):
+ g = cuda["groups"]
+ _line("expert groups", f"{g['calls']} calls, {g['experts']} experts, "
+ f"{g['rows']} rows ({g['experts_per_call']:.1f} experts/call)")
+ if cuda.get("groups_timing"):
+ gt = cuda["groups_timing"]
+ _line("GPU timing", f"H2D {gt['h2d_ms']:.1f} ms | kernel {gt['kernel_ms']:.1f} ms | "
+ f"D2H {gt['d2h_ms']:.1f} ms")
+ if gt["h2d_ms"] + gt["d2h_ms"] > gt["kernel_ms"]:
+ flags.append("CUDA H2D+D2H > kernel time -> transfer-bound; "
+ "consider larger expert tier to keep weights resident")
+
+ # ---------------------------------------------------------------- summary ----
+ print("\n" + "=" * 78)
+ if flags:
+ print(f" {len(flags)} FLAG(s) β€” the most likely levers to move tok/s:")
+ for f in flags:
+ print(f" - {f}")
+ else:
+ print(" no flags β€” every measured subsystem is within advisory thresholds.")
+ print("=" * 78)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/c/tests/test_inefficiency.py b/c/tests/test_inefficiency.py
new file mode 100644
index 00000000..0a5eb650
--- /dev/null
+++ b/c/tests/test_inefficiency.py
@@ -0,0 +1,239 @@
+"""Inefficiency / regression tests for the colibri engine (tiny model, asserted).
+
+These run against the bundled glm_tiny model (~0.6 MB resident, ~0.1s/run) and
+gate CI: a regression here means something broke. They run on a plain CPU-only
+`glm.exe` build; the CUDA_* tests auto-skip if the engine wasn't built with
+CUDA_DLL=1 (see tests/README_efficiency.md for the build command).
+
+The signals under test, and the inefficiency each catches:
+ - tok/s floor : a throughput regression (broken build / bad config)
+ - profile phases sum : telemetry accounting bug (other balloons)
+ - disk-wait not dominant: a tiny resident model should never be I/O-bound
+ - CPU determinism : greedy decode is reproducible (no stray RNG/threading)
+ - CUDA init path : COLI_CUDA=1 initializes and does not silently exit 2
+ - CUDA dense uses VRAM : CUDA_DENSE=1 actually uploads tensors (no silent CPU fallback)
+ - CPU vs CUDA TF-match : identical weights+inputs β†’ identical argmax (kernel bug guard)
+"""
+import os
+import shutil
+import unittest
+from pathlib import Path
+
+from tools.efficiency import (
+ parse_run, run_engine, disk_wait_share, tf_agreement,
+ TINY_TOK_S_FLOOR, MAX_DISK_WAIT_SHARE, MIN_CPU_CUDA_AGREEMENT,
+)
+
+HERE = Path(__file__).resolve().parent
+C_DIR = HERE.parent
+ENGINE = C_DIR / "glm.exe"
+TINY = C_DIR / "glm_tiny"
+
+
+def _engine_present() -> bool:
+ return ENGINE.exists()
+
+
+def _cuda_available() -> bool:
+ """True iff the engine binary has the CUDA loader compiled in AND the DLL is present.
+
+ The host is built with -DCOLI_CUDA only when CUDA_DLL=1 (Makefile). A binary
+ built without it embeds the string "this binary is CPU-only; rebuild" and
+ exits 2 on any CUDA env var β€” so we detect the CPU-only build by scanning
+ the binary for that marker (avoids a slow ldd/strings on every import; we
+ only read enough to find it). On Windows the DLL is also required
+ (backend_loader.c loads it at runtime); on Linux it's direct-linked.
+ """
+ if not _engine_present():
+ return False
+ try:
+ # Read once; the marker is near the read-only string table. 256 KB is
+ # plenty for this string and avoids loading a 1 MB binary into memory.
+ with open(ENGINE, "rb") as f:
+ blob = f.read(2 * 1024 * 1024)
+ if b"this binary is CPU-only" in blob:
+ return False # CPU-only build: COLI_CUDA=1 would exit 2.
+ except OSError:
+ pass
+ # Windows: DLL required at runtime. Linux: direct-linked (no DLL).
+ if os.name == "nt":
+ return (C_DIR / "coli_cuda.dll").exists()
+ import subprocess
+ try:
+ out = subprocess.run(["ldd", str(ENGINE)], capture_output=True, text=True)
+ return "libcudart" in out.stdout
+ except (FileNotFoundError, OSError):
+ return False
+
+
[email protected](_engine_present(), "glm.exe not built (run: make glm.exe)")
+class TinyEfficiencyTest(unittest.TestCase):
+ """Asserted regression tests on the resident tiny model. Gates CI."""
+
+ def _run(self, **overlay):
+ return run_engine(overlay, engine=str(ENGINE), snap=str(TINY))[0]
+
+ # -- telemetry contract ---------------------------------------------------
+
+ def test_telemetry_parses(self):
+ """A REPLAY run must emit the throughput + PROFILE lines the suite keys on.
+
+ If this fails, either the engine changed its output format (update the
+ parsers in tools/efficiency.py) or the run crashed early."""
+ t = self._run(REPLAY="1", TEMP="0", NGEN="4")
+ self.assertEqual(t["returncode"], 0, f"engine exited non-zero:\n{t['stderr']}")
+ self.assertIn("tok_s", t["parsed"], f"missing tok/s line:\n{t['stderr']}")
+ self.assertIn("profile", t["parsed"], f"missing PROFILE line:\n{t['stderr']}")
+ self.assertIn("hit_pct", t["parsed"], f"missing expert hit line:\n{t['stderr']}")
+ self.assertIsNotNone(t["tok_s"])
+ self.assertIsNotNone(t["profile"])
+
+ # -- throughput floor -----------------------------------------------------
+
+ def test_tiny_tok_s_floor(self):
+ """Tiny decode must beat TINY_TOK_S_FLOOR.
+
+ The tiny model is fully resident and runs ~200 tok/s; the 20 tok/s
+ default floor is a 10x margin that catches broken builds or a pathological
+ config cascade (the cap=1 trap from ISSUE_new_model_resource_regression.md)
+ without flapping on machine noise."""
+ t = self._run(REPLAY="1", TEMP="0", NGEN="8")
+ self.assertEqual(t["returncode"], 0, f"engine exited non-zero:\n{t['stderr']}")
+ self.assertGreaterEqual(
+ t["tok_s"], TINY_TOK_S_FLOOR,
+ f"tok/s {t['tok_s']:.1f} below floor {TINY_TOK_S_FLOOR} "
+ f"(regression, or a config cascade starving the cache)",
+ )
+
+ # -- accounting sanity ----------------------------------------------------
+
+ def test_profile_phases_present_and_nonneg(self):
+ """Every PROFILE phase must be present and non-negative.
+
+ `other` can go slightly negative from timer overhead (the engine allows
+ it), but a large negative means the timers are double-counting."""
+ t = self._run(REPLAY="1", TEMP="0", NGEN="4")
+ p = t["profile"]
+ for phase in ("disk", "expert_matmul", "attention", "lm_head"):
+ self.assertGreaterEqual(p[phase], -0.01, f"{phase} went negative: {p}")
+ # 'other' is a residual; allow a small negative from timer overlap.
+ self.assertGreaterEqual(p["other"], -0.05, f"other too negative (double-count): {p}")
+
+ # -- disk-wait not dominant on a resident model ---------------------------
+
+ def test_disk_wait_not_dominant(self):
+ """A fully-resident tiny model must NOT be I/O-bound.
+
+ Everything fits in RAM; the expert-disk wait share should be ~0. If it
+ exceeds MAX_DISK_WAIT_SHARE, the cache/I/O path regressed β€” on a real
+ model this same regression would make decode I/O-bound (the exact
+ failure mode the [PROF] verdict flags)."""
+ t = self._run(REPLAY="1", TEMP="0", NGEN="8", PROF="1")
+ self.assertIn("time_shares", t["parsed"], f"missing [PROF] time shares:\n{t['stderr']}")
+ share = disk_wait_share(t)
+ self.assertIsNotNone(share)
+ self.assertLess(
+ share, MAX_DISK_WAIT_SHARE,
+ f"expert-I/O share {share:.0%} on a resident model β€” I/O path regressed",
+ )
+
+ # -- determinism ----------------------------------------------------------
+
+ def test_cpu_vs_cpu_determinism(self):
+ """Two greedy REPLAY runs with the same seed produce identical telemetry.
+
+ TEMP=0 = greedy (no sampling), so tok/s and hit-rate must be reproducible.
+ A drift here means non-determinism crept into the decode path (stray
+ threading, uninitialized state) β€” which on a real model would make A/B
+ comparisons meaningless."""
+ a = self._run(REPLAY="1", TEMP="0", NGEN="8", SEED="1")
+ b = self._run(REPLAY="1", TEMP="0", NGEN="8", SEED="1")
+ self.assertEqual(a["returncode"], 0)
+ self.assertEqual(b["returncode"], 0)
+ self.assertEqual(a["hit_pct"], b["hit_pct"], "greedy hit-rate drifted between runs")
+ # tok/s within 25% β€” exact equality is too strict across scheduler noise.
+ self.assertLess(abs(a["tok_s"] - b["tok_s"]) / max(a["tok_s"], b["tok_s"]), 0.25)
+
+
[email protected](_cuda_available(),
+ "CUDA build not present (run: make clean && make glm.exe CUDA_DLL=1 && make cuda-dll)")
+class TinyCudaEfficiencyTest(unittest.TestCase):
+ """CUDA-path regression tests on the tiny model. Skip unless CUDA built.
+
+ glm_tiny is small and fully resident, so CUDA here is fast and exercises the
+ real GPU code path (init, dense upload, kernel correctness) without the
+ long load time or memory pressure of the full model. These guard the
+ silent-failure modes that are otherwise invisible:
+ - COLI_CUDA=1 silently falling back to CPU (loader/DLL missing)
+ - CUDA_DENSE=1 uploading nothing
+ - a CUDA kernel producing different argmax than CPU on identical inputs
+ """
+
+ def _run(self, **overlay):
+ return run_engine(overlay, engine=str(ENGINE), snap=str(TINY))[0]
+
+ def test_cuda_init_path(self):
+ """COLI_CUDA=1 must initialize the device and NOT exit 2.
+
+ Exit 2 is the engine's "requested backend is unavailable" path
+ (glm.c: g_cuda_enabled check). A clean init prints the [CUDA] device
+ banner to stderr. If this fails, the DLL is broken or the loader can't
+ resolve symbols (ABI drift between backend_cuda.h and the dll)."""
+ t = self._run(COLI_CUDA="1", COLI_GPU="0", REPLAY="1", TEMP="0", NGEN="4")
+ self.assertNotEqual(t["returncode"], 2,
+ f"engine refused CUDA backend:\n{t['stderr']}")
+ self.assertTrue(t["cuda"]["enabled"],
+ f"no [CUDA] device banner on stderr:\n{t['stderr']}")
+
+ def test_cuda_dense_uses_vram(self):
+ """CUDA_DENSE=1 must actually upload dense tensors to VRAM.
+
+ The minimal GPU-exercising config (per backend_loader.c analysis):
+ COLI_CUDA=1 + CUDA_DENSE=1. Without CUDA_DENSE the dense path stays on
+ CPU and [CUDA] resident set reports 0 tensors β€” a silent no-op. This
+ catches that regression: after the run, resident_tensors > 0."""
+ t = self._run(COLI_CUDA="1", COLI_GPU="0", CUDA_DENSE="1",
+ REPLAY="1", TEMP="0", NGEN="4")
+ self.assertEqual(t["returncode"], 0, f"engine exited non-zero:\n{t['stderr']}")
+ rt = t["cuda"]["resident_tensors"]
+ self.assertIsNotNone(rt, f"no [CUDA] resident set line:\n{t['stderr']}")
+ self.assertGreater(rt, 0,
+ f"CUDA_DENSE=1 but {rt} tensors resident β€” silent CPU fallback")
+
+ def test_cpu_vs_cuda_tf_match(self):
+ """CPU and CUDA teacher-forcing must agree on most positions (DIRECTLY).
+
+ Both paths prefill the SAME oracle sequence on the SAME weights, so their
+ argmaxes should match position-for-position β€” but not exactly: the two
+ backends accumulate dot-products in different orders (x86 SIMD vs CUDA
+ kernel), so a few near-tied logits flip. That divergence is expected
+ numeric behavior, not a kernel bug. A *catastrophic* kernel regression
+ (wrong GEMM, wrong scale, wrong fmt) would collapse agreement toward
+ random (~1/vocab = ~4%); the MIN_CPU_CUDA_AGREEMENT floor (default 70%)
+ catches that while tolerating harmless drift.
+
+ We compare CPU-vs-CUDA *directly* (not via the oracle match-counts),
+ because both backends differ from the oracle at different positions and
+ the summary line can't tell "CPU≠CUDA" from "CPU≠oracle"."""
+ import json
+ ref = json.loads((C_DIR / "ref_glm.json").read_text())
+ oracle = ref["tf_pred"]
+
+ cpu = self._run(TF="1", TEMP="0")
+ cuda = self._run(COLI_CUDA="1", COLI_GPU="0", CUDA_DENSE="1", TF="1", TEMP="0")
+ self.assertEqual(cpu["returncode"], 0, f"CPU TF run failed:\n{cpu['stderr']}")
+ self.assertEqual(cuda["returncode"], 0, f"CUDA TF run failed:\n{cuda['stderr']}")
+ self.assertIn("tf_mismatches", cpu["parsed"], "CPU run missing per-position mismatches")
+ self.assertIn("tf_mismatches", cuda["parsed"], "CUDA run missing per-position mismatches")
+
+ agree, diff = tf_agreement(cpu, cuda, oracle)
+ self.assertGreaterEqual(
+ agree, MIN_CPU_CUDA_AGREEMENT,
+ f"CPU-vs-CUDA argmax agreement {agree:.0%} below floor "
+ f"{MIN_CPU_CUDA_AGREEMENT:.0%} β€” CUDA kernel likely regressed. "
+ f"Differing positions: {diff[:10]}{'...' if len(diff)>10 else ''}",
+ )
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/c/tools/efficiency.py b/c/tools/efficiency.py
new file mode 100644
index 00000000..775e8cf0
--- /dev/null
+++ b/c/tools/efficiency.py
@@ -0,0 +1,458 @@
+"""Efficiency / regression harness for the colibri engine.
+
+The engine already emits rich telemetry (REPLAY tok/s, PROFILE phase timings,
+[PROF] time shares + verdict, CUDA expert-tier utilization). Until now every
+consumer of that telemetry β€” `benchmark_cuda_fixture.py`, `bench_full.sh`,
+`bench_ux.sh` β€” has only *printed* it for a human to eyeball. This module turns
+each signal into a parseable field so tests can assert on it.
+
+Design:
+ - Reuses SPEED_RE / PROFILE_RE from tools.benchmark_cuda_fixture (no drift).
+ - parse_run() is pure: stdout+stderr in, dict out. Easy to unit-test against
+ captured strings (like the existing test_benchmark_cuda_fixture does).
+ - run_engine() is the subprocess wrapper. Captures stdout and stderr
+ separately, because the engine splits them: PROFILE/REPLAY/CUDA-tier go to
+ stdout, the [CUDA]/[PROF]/[prefill] banners go to stderr.
+ - Floor defaults are module constants (tunable in one place, not scattered).
+
+No model file is required to import this module; only run_engine() invokes the
+binary. parse_run() works on any captured text, so most test surface is covered
+by string fixtures without spinning the engine at all.
+"""
+from __future__ import annotations
+
+import os
+import re
+import subprocess
+from pathlib import Path
+from typing import Optional
+
+# Reuse the validated regexes from the existing A/B benchmark harness so the
+# PROFILE field order (disk, expert_matmul, attention, lm_head, other) and the
+# tok/s capture stay identical. Drift here would silently break every consumer.
+from tools.benchmark_cuda_fixture import SPEED_RE as _SPEED_RE_REPLAY, PROFILE_RE, PROFILE_KEYS
+
+# SPEED_RE (from benchmark_cuda_fixture) matches the REPLAY-mode line only:
+# "REPLAY decode: ... | 12.34 tok/s | ..."
+# run_text / PROMPT mode uses a DIFFERENT format (glm.c:4682):
+# "decode N tokens in X.XXs (12.34 tok/s) | expert hit rate ..."
+# This alt regex catches the parenthesized form so the full-model report (which
+# uses PROMPT mode) gets a real tok/s instead of reporting it missing.
+SPEED_RE_TEXT = re.compile(r"decode \d+ tokens in [0-9.]+s \(([0-9.]+) tok/s\)")
+
+
+def _first_speed(stdout: str):
+ """Find tok/s in whichever run-mode format the engine used."""
+ for rx in (_SPEED_RE_REPLAY, SPEED_RE_TEXT):
+ m = rx.search(stdout)
+ if m:
+ return m
+ return None
+
+
+# Public alias so existing imports keep working (tests reference SPEED_RE).
+SPEED_RE = _SPEED_RE_REPLAY
+
+
+# --- additional parsers (formats verified against glm.c printf strings) ---
+
+# "expert hit rate 88.1%" (summary line) | "expert hit 95.0%" (REPLAY line)
+HIT_RE = re.compile(r"expert hit(?:\s+rate)?\s+([0-9.]+)%")
+
+# "[PROF] time shares: expert-I/O 3% | expert-matmul 12% | attention 56% | lm_head 2% | other 27%"
+SHARES_RE = re.compile(
+ r"\[PROF\] time shares: expert-I/O\s+([0-9.]+)%\s*\|\s*expert-matmul\s+([0-9.]+)%\s*"
+ r"\|\s*attention\s+([0-9.]+)%\s*\|\s*lm_head\s+([0-9.]+)%\s*\|\s*other\s+([0-9.]+)%"
+)
+
+# "[PROF] verdict: I/O-bound β€” 60% of the time ..." (also compute-bound / attention-bound / balanced)
+VERDICT_RE = re.compile(r"\[PROF\] verdict:\s*(I/O-bound|compute-bound|attention-bound|balanced)")
+
+# "[PROF] expert I/O: ... hit 95.0% (76 hit / 4 load) | 4.0 loads/token"
+LOADS_PER_TOK_RE = re.compile(r"\|\s*([0-9.]+)\s+loads/token")
+
+# "CUDA expert tier: 111 resident experts (2.36 GB) | 5400 calls served from VRAM" (stdout)
+CUDA_TIER_RE = re.compile(
+ r"CUDA expert tier:\s+(\d+)\s+resident experts\s+\(([0-9.]+)\s+GB\)\s*\|\s+(\d+)\s+calls served from VRAM"
+)
+
+# "[CUDA] device 0: NVIDIA ..., 14.4 GB VRAM, sm_120" (stderr, per device at init)
+CUDA_DEVICE_RE = re.compile(r"\[CUDA\] device\s+\d+:")
+
+# "[CUDA] resident set: 12 tensors, 0.45 GB VRAM" (stderr, cuda_stats_print)
+CUDA_RESIDENT_RE = re.compile(
+ r"\[CUDA\] resident set:\s+(\d+)\s+tensors,\s+([0-9.]+)\s+GB\s+VRAM"
+)
+
+# "PREFILL (teacher-forcing) C vs oracle: 11/32 positions | 1700.4 pos/s" (TF=1 mode, stdout)
+TF_MATCH_RE = re.compile(r"PREFILL \(teacher-forcing\).*:\s+(\d+)/(\d+)\s+positions")
+
+# --- the six deeper signals (added after "are you gathering everything?" audit) ---
+
+# "ATTENTION: projection/RoPE 0.050s | score-softmax-value 0.009s | output projection 0.011s"
+# Sub-breakdown of the attention phase β€” answers "how is attention being read".
+ATTN_BREAKDOWN_RE = re.compile(
+ r"ATTENTION: projection/RoPE\s+([0-9.]+)s\s*\|\s*score-softmax-value\s+([0-9.]+)s\s*"
+ r"\|\s*output projection\s+([0-9.]+)s"
+)
+
+# "[PROF] decode forwards: 20 | latency p50 7.3 ms | p90 8.0 ms | p99 8.1 ms | max 8.2 ms | 1.00 tok/forward"
+# Per-forward tail latency β€” a p99 >> p50 means decode stalls (I/O hiccups, KV grow).
+LATENCY_RE = re.compile(
+ r"\[PROF\] decode forwards:\s+(\d+)\s*\| latency p50\s+([0-9.]+)\s*ms\s*\| "
+ r"p90\s+([0-9.]+)\s*ms\s*\|\s*p99\s+([0-9.]+)\s*ms\s*\|\s*max\s+([0-9.]+)\s*ms"
+)
+
+# "[PROF] expert I/O: 0.004 GB fetched (0.2 MB/token, 0.03 GB/s over the run) | hit 95.0% ... |
+# 4.0 loads/token | 0.0s read service / 0.0s felt wait"
+# Absolute disk throughput + the felt-wait split (the [PROF] version, more detailed than PROFILE).
+EXPERT_IO_RE = re.compile(
+ r"\[PROF\] expert I/O:\s+([0-9.]+)\s+GB fetched\s+\(([0-9.]+)\s+MB/token,\s*([0-9.]+)\s+GB/s"
+ r".*?\|\s*([0-9.]+)\s+loads/token\s*\|\s*([0-9.]+)s\s+read service\s*/\s*([0-9.]+)s\s+felt wait"
+)
+
+# "speculation: 1.05 tokens/forward (19 forwards per 20 tokens) | MTP acceptance 44% (7/16)"
+# Draft efficiency β€” is the speculative decoder pulling weight or dead overhead?
+SPECULATION_RE = re.compile(
+ r"speculation:\s+([0-9.]+)\s+tokens/forward\s+\((\d+)\s+forwards per\s+(\d+)\s+tokens\)"
+ r"\s*\|\s*MTP acceptance\s+([0-9.]+)%"
+)
+
+# "experts loaded/token: 450.0 (per-layer 56.25 across 8; baseline topk=8) | TOPK=0 TOPP=0.00"
+# Fuller than loads_per_tok: includes the per-layer spread + the active topk/topp.
+EXPERTS_LOADED_RE = re.compile(
+ r"experts loaded/token:\s+([0-9.]+)\s+\(per-layer\s+([0-9.]+)\s+across\s+(\d+);\s*baseline topk=(\d+)\)"
+)
+
+# "[PROF] machine: Intel(...) | 22 cores (22 omp threads) | RAM 34.1 GB total, 27.1 GB available | backend CUDA"
+# Provenance β€” makes a report reproducible across machines/runs.
+MACHINE_RE = re.compile(r"\[PROF\] machine:\s*(.*)\|\s*(\d+)\s+cores.*backend\s+(\S+)")
+
+# "[PROF] config: RAM_GB=auto 23.9 CTX=4096 | expert cache cap 8/layer ... | DRAFT=0 PIPE=1 DIRECT=0 ..."
+# Effective resolved config (after auto-budgeting). Answers "what config actually ran".
+CONFIG_RE = re.compile(r"\[PROF\] config:\s*(.*)")
+
+# "[ORACLE] mismatch pos=7 expected=197 got=22" (TF=1 mode, stderr, per position)
+# Captures the engine's *actual* argmax at each position, so two backends can be
+# compared DIRECTLY (independent of how each relates to the oracle).
+TF_MISMATCH_RE = re.compile(r"\[ORACLE\] mismatch pos=(\d+) expected=\d+ got=(\d+)")
+
+# --- routing-quality + disk-split + cuda-groups (the deep signals) ---
+
+# summary suffix: " | swap 3.1% (12/384)" (CACHE_ROUTE inline)
+SWAP_RE = re.compile(r"swap\s+([0-9.]+)%\s+\((\d+)/(\d+)\)")
+
+# summary suffix: " | route_agree 85.0% | route_kl 0.0123" (CACHE_ROUTE/ROUTE_AGREE)
+ROUTE_AGREE_RE = re.compile(r"route_agree\s+([0-9.]+)%\s*\|\s*route_kl\s+([0-9.]+)")
+
+# "disk-load split: draft 8 + absorb 0 + verify/main 3 misses | MTP-layer 0 loads 0.00 GB |
+# main-layers 11 loads 0.01 GB (MTP 0.0% of bytes)" (DISK_SPLIT=1)
+DISK_SPLIT_RE = re.compile(
+ r"disk-load split: draft\s+(\d+)\s+\+\s+absorb\s+(\d+)\s+\+\s+verify/main\s+(\d+)\s+misses"
+ r"\s*\|\s*MTP-layer\s+(\d+)\s+loads\s+([0-9.]+)\s+GB\s*\|\s*main-layers\s+(\d+)\s+loads\s+([0-9.]+)\s+GB"
+ r"(?:\s+\(MTP\s+([0-9.]+)%\s+of bytes\))?"
+)
+
+# "[CUDA] expert groups: 120 call, 840 expert, 1200 righe (7.00 expert/call)"
+CUDA_GROUPS_RE = re.compile(
+ r"\[CUDA\] expert groups:\s+(\d+)\s+call,\s+(\d+)\s+expert,\s+(\d+)\s+righe\s+\(([0-9.]+)\s+expert/call\)"
+)
+
+# "[CUDA] expert groups timing: H2D 12.3 ms | kernel 45.6 ms | D2H 7.8 ms" (COLI_CUDA_PROFILE=1)
+CUDA_GROUPS_TIME_RE = re.compile(
+ r"\[CUDA\] expert groups timing: H2D\s+([0-9.]+)\s+ms\s*\|\s*kernel\s+([0-9.]+)\s+ms\s*\|\s*D2H\s+([0-9.]+)\s+ms"
+)
+
+# LOOKAHEAD recall block β€” 4 named rows. (name, pct, hit, tot)
+LOOKAHEAD_RE = re.compile(
+ r"^\s*(.+?)\s+([0-9.]+)%\s+\((\d+)/(\d+)\)\s*$", re.MULTILINE
+)
+
+# "loaded in 0.02s | resident dense: 0.21 MB | layers=5 experts=8 | MTP absent (draft=0)"
+LOAD_BANNER_RE = re.compile(
+ r"loaded in\s+([0-9.]+)s\s*\|\s*resident dense:\s+([0-9.]+)\s+MB\s*\|"
+ r"\s*layers=(\d+)\s+experts=(\d+)\s*\|\s*MTP\s+(\w+)\s+\(draft=(\d+)\)"
+)
+
+
+# --- tunable floors -----------------------------------------------------------
+# These are deliberately generous so they catch *regressions* (broken builds,
+# pathological configs, telemetry accounting bugs) without flapping on machine
+# noise. The tiny model is fully resident at ~200 tok/s, so a 20 tok/s floor is
+# a 10x margin. Tune per-host via env if needed (documented in README).
+TINY_TOK_S_FLOOR = float(os.environ.get("COLI_TINY_TOK_S_FLOOR", "20.0"))
+# On a fully-resident tiny model the expert-disk wait share should be tiny.
+# If it exceeds this, something regressed in the I/O accounting or cache path.
+MAX_DISK_WAIT_SHARE = float(os.environ.get("COLI_MAX_DISK_WAIT_SHARE", "0.20"))
+# decode wall-time should be roughly the sum of PROFILE phases (other = residual).
+PROFILE_SUM_TOLERANCE = float(os.environ.get("COLI_PROFILE_SUM_TOL", "0.05"))
+# Minimum direct CPU-vs-CUDA argmax agreement on the tiny TF fixture. The two
+# backends use different accumulation orders (SIMD dot vs CUDA kernel), so a
+# few near-tied positions flip argmax β€” that's expected numeric divergence, not
+# a kernel bug. Measured baseline ~84% (27/32) on this fixture; the 70% floor
+# leaves headroom for machine noise while still catching a catastrophic kernel
+# regression (e.g. a wrong GEMM would drop this to ~random = ~4%).
+MIN_CPU_CUDA_AGREEMENT = float(os.environ.get("COLI_MIN_CPU_CUDA_AGREE", "0.70"))
+
+
+def parse_run(stdout: str, stderr: str = "") -> dict:
+ """Parse one engine run's output into a telemetry dict.
+
+ Returns keys: tok_s, hit_pct, profile (dict, seconds), profile_sum,
+ time_shares (dict, fractions 0..1), verdict, loads_per_tok, cuda (dict),
+ tf_match (tuple or None), parsed (set of field names found).
+
+ Raises RuntimeError only if the core throughput line is missing β€” everything
+ else is optional and absent on some run modes (e.g. [PROF] needs PROF=1,
+ CUDA tier needs gpu_expert_count>0).
+ """
+ out = dict(
+ tok_s=None, hit_pct=None, profile=None, profile_sum=None,
+ time_shares=None, verdict=None, loads_per_tok=None,
+ cuda=None, tf_match=None, stderr=stderr,
+ )
+ parsed = set()
+ blob = stdout + "\n" + stderr # [PROF]/[CUDA] live on stderr; scan both.
+
+ m = _first_speed(stdout)
+ if m:
+ out["tok_s"] = float(m.group(1)); parsed.add("tok_s")
+
+ m = HIT_RE.search(blob)
+ if m:
+ out["hit_pct"] = float(m.group(1)); parsed.add("hit_pct")
+
+ m = PROFILE_RE.search(stdout)
+ if m:
+ service, wait, emm, attn, head, other = (float(x) for x in m.groups())
+ disk = service + (wait or 0.0)
+ out["profile"] = dict(zip(PROFILE_KEYS, (disk, emm, attn, head, other)))
+ out["profile_sum"] = disk + emm + attn + head + other
+ parsed.add("profile")
+
+ # ATTENTION sub-breakdown: projection/RoPE | score-softmax-value | output.
+ m = ATTN_BREAKDOWN_RE.search(stdout)
+ if m:
+ out["attn_breakdown"] = dict(zip(
+ ("proj_rope", "score_sm_value", "out_proj"),
+ (float(x) for x in m.groups())))
+ parsed.add("attn_breakdown")
+
+ m = SHARES_RE.search(blob)
+ if m:
+ io, emm, attn, head, other = (float(x) / 100.0 for x in m.groups())
+ out["time_shares"] = dict(io=io, matmul=emm, attention=attn, head=head, other=other)
+ parsed.add("time_shares")
+
+ m = VERDICT_RE.search(blob)
+ if m:
+ out["verdict"] = m.group(1); parsed.add("verdict")
+
+ # [PROF] decode forwards + latency p50/p90/p99/max.
+ m = LATENCY_RE.search(blob)
+ if m:
+ out["latency"] = dict(zip(
+ ("forwards", "p50_ms", "p90_ms", "p99_ms", "max_ms"),
+ (float(x) for x in m.groups())))
+ parsed.add("latency")
+
+ # [PROF] expert I/O throughput: GB fetched, MB/token, GB/s, service vs felt wait.
+ m = EXPERT_IO_RE.search(blob)
+ if m:
+ out["expert_io"] = dict(zip(
+ ("gb_fetched", "mb_per_tok", "gb_per_s", "loads_per_tok",
+ "read_service_s", "felt_wait_s"),
+ (float(x) for x in m.groups())))
+ parsed.add("expert_io")
+
+ m = LOADS_PER_TOK_RE.search(blob)
+ if m:
+ out["loads_per_tok"] = float(m.group(1)); parsed.add("loads_per_tok")
+
+ # experts loaded/token with per-layer spread + baseline topk (run_text summary).
+ m = EXPERTS_LOADED_RE.search(stdout)
+ if m:
+ out["experts_loaded"] = dict(
+ per_tok=float(m.group(1)), per_layer=float(m.group(2)),
+ n_sparse_layers=int(m.group(3)), baseline_topk=int(m.group(4)))
+ parsed.add("experts_loaded")
+
+ # speculation: tokens/forward, forwards, tokens, MTP acceptance%.
+ m = SPECULATION_RE.search(stdout)
+ if m:
+ out["speculation"] = dict(zip(
+ ("tok_per_fw", "forwards", "tokens", "mtp_accept_pct"),
+ (float(m.group(1)), int(m.group(2)), int(m.group(3)), float(m.group(4)))))
+ parsed.add("speculation")
+
+ # routing quality (CACHE_ROUTE inline suffixes on the summary line).
+ m = SWAP_RE.search(stdout)
+ if m:
+ out["swap"] = dict(pct=float(m.group(1)), swaps=int(m.group(2)), slots=int(m.group(3)))
+ parsed.add("swap")
+ m = ROUTE_AGREE_RE.search(stdout)
+ if m:
+ out["route_agree"] = dict(agree_pct=float(m.group(1)), kl=float(m.group(2)))
+ parsed.add("route_agree")
+
+ # disk-load split by decode phase (DISK_SPLIT=1).
+ m = DISK_SPLIT_RE.search(stdout)
+ if m:
+ out["disk_split"] = dict(zip(
+ ("draft", "absorb", "verify_main", "mtp_loads", "mtp_gb",
+ "main_loads", "main_gb", "mtp_bytes_pct"),
+ (int(m.group(1)), int(m.group(2)), int(m.group(3)), int(m.group(4)),
+ float(m.group(5)), int(m.group(6)), float(m.group(7)),
+ float(m.group(8)) if m.group(8) else None)))
+ parsed.add("disk_split")
+
+ # provenance: machine + resolved config.
+ m = MACHINE_RE.search(blob)
+ if m:
+ out["machine"] = dict(cpu=m.group(1).strip(), cores=int(m.group(2)), backend=m.group(3))
+ parsed.add("machine")
+ m = CONFIG_RE.search(blob)
+ if m:
+ out["config_str"] = m.group(1).strip(); parsed.add("config")
+
+ # load banner: load time, resident dense MB, layers, experts, MTP status.
+ m = LOAD_BANNER_RE.search(stdout)
+ if m:
+ out["load"] = dict(zip(
+ ("load_s", "resident_dense_mb", "layers", "experts", "mtp_status", "draft"),
+ (float(m.group(1)), float(m.group(2)), int(m.group(3)),
+ int(m.group(4)), m.group(5), int(m.group(6)))))
+ parsed.add("load")
+
+ # LOOKAHEAD routing-recall block (LOOKA=1): list of {predictor, pct, hit, tot}.
+ la_block = re.search(
+ r"LOOKAHEAD routing.*?recall.*?:\n((?:^\s+.+?\s+[0-9.]+%\s+\(\d+/\d+\)\s*$\n?)+)",
+ blob, re.MULTILINE)
+ if la_block:
+ out["lookahead"] = []
+ for row in LOOKAHEAD_RE.finditer(la_block.group(1)):
+ out["lookahead"].append(dict(
+ predictor=row.group(1).strip(), pct=float(row.group(2)),
+ hit=int(row.group(3)), tot=int(row.group(4))))
+ parsed.add("lookahead")
+
+ cuda = dict(enabled=False, expert_count=None, expert_gb=None,
+ calls_served=None, resident_tensors=None, resident_gb=None,
+ groups=None, groups_timing=None)
+ if CUDA_DEVICE_RE.search(stderr):
+ cuda["enabled"] = True
+ m = CUDA_TIER_RE.search(stdout)
+ if m:
+ cuda["expert_count"] = int(m.group(1))
+ cuda["expert_gb"] = float(m.group(2))
+ cuda["calls_served"] = int(m.group(3))
+ m = CUDA_RESIDENT_RE.search(stderr)
+ if m:
+ cuda["resident_tensors"] = int(m.group(1))
+ cuda["resident_gb"] = float(m.group(2))
+ m = CUDA_GROUPS_RE.search(stderr)
+ if m:
+ cuda["groups"] = dict(zip(
+ ("calls", "experts", "rows", "experts_per_call"),
+ (int(m.group(1)), int(m.group(2)), int(m.group(3)), float(m.group(4)))))
+ m = CUDA_GROUPS_TIME_RE.search(stderr)
+ if m:
+ cuda["groups_timing"] = dict(zip(
+ ("h2d_ms", "kernel_ms", "d2h_ms"),
+ (float(m.group(1)), float(m.group(2)), float(m.group(3)))))
+ out["cuda"] = cuda
+
+ m = TF_MATCH_RE.search(stdout)
+ if m:
+ out["tf_match"] = (int(m.group(1)), int(m.group(2))); parsed.add("tf_match")
+
+ # Capture per-position argmax divergences from the oracle, keyed by position.
+ mismatches = {int(mm.group(1)): int(mm.group(2))
+ for mm in TF_MISMATCH_RE.finditer(blob)}
+ if out["tf_match"] is not None:
+ out["tf_mismatches"] = mismatches
+ parsed.add("tf_mismatches")
+
+ out["parsed"] = parsed
+ return out
+
+
+def run_engine(
+ env_overlay: dict,
+ *,
+ engine: Optional[str] = None,
+ cap: int = 4,
+ ebits: int = 4,
+ dbits: int = 4,
+ timeout: float = 600.0,
+ snap: Optional[str] = None,
+) -> tuple[dict, subprocess.CompletedProcess]:
+ """Run the engine with an env overlay; return (parsed_telemetry, proc).
+
+ `engine` defaults to ./glm.exe (colibri's Windows host). `snap` defaults to
+ the bundled tiny model (glm_tiny) so callers can omit it for fast tests.
+ The positional argv is `cap ebits dbits`, matching the engine's main().
+ """
+ if engine is None:
+ engine = str(Path(__file__).resolve().parent.parent / "glm.exe")
+ env = os.environ.copy()
+ # Strip CUDA vars by default so a CPU run isn't accidentally GPU-accelerated
+ # by a leftover env; callers opt in by passing them in env_overlay.
+ for k in ("COLI_CUDA", "COLI_GPU", "COLI_GPUS", "CUDA_DENSE", "CUDA_EXPERT_GB"):
+ env.pop(k, None)
+ env.update(env_overlay)
+ if snap is not None:
+ env["SNAP"] = snap
+ elif "SNAP" not in env:
+ env["SNAP"] = str(Path(__file__).resolve().parent.parent / "glm_tiny")
+
+ proc = subprocess.run(
+ [engine, str(cap), str(ebits), str(dbits)],
+ env=env, capture_output=True, text=True, timeout=timeout,
+ )
+ telemetry = parse_run(proc.stdout, proc.stderr)
+ telemetry["returncode"] = proc.returncode
+ telemetry["env"] = {k: env_overlay[k] for k in env_overlay}
+ return telemetry, proc
+
+
+def disk_wait_share(t: dict) -> Optional[float]:
+ """Fraction of decode wall-time spent waiting on expert disk reads.
+
+ Preferred source: [PROF] time_shares (the engine's own accounting, which
+ separates felt-wait from read-service). Falls back to PROFILE disk / sum if
+ [PROF] wasn't emitted (PROF=0 runs). None if neither is available.
+ """
+ if t.get("time_shares"):
+ return t["time_shares"]["io"]
+ if t.get("profile") and t.get("profile_sum"):
+ return t["profile"]["disk"] / t["profile_sum"]
+ return None
+
+
+def tf_agreement(cpu: dict, cuda: dict, oracle: list[int]) -> tuple[float, list[int]]:
+ """Direct CPU-vs-CUDA argmax agreement on the TF fixture.
+
+ Both runs prefilled the SAME oracle sequence; tf_mismatches holds each
+ backend's actual argmax where it diverged from the oracle. Where a backend
+ is ABSENT from the mismatch map, its prediction equals the oracle token at
+ that position. So the reconstructed per-position prediction is:
+ oracle[i] if i not in mismatches else mismatches[i]
+ and agreement is the fraction of positions where CPU and CUDA predictions
+ are identical β€” independent of how each relates to the oracle.
+
+ `oracle` is ref_glm.json's tf_pred (the per-position oracle argmax). Pass
+ n_positions = len(oracle).
+
+ Returns (agreement_fraction, list_of_differing_positions).
+ """
+ cm = cpu.get("tf_mismatches") or {}
+ gm = cuda.get("tf_mismatches") or {}
+ diff = []
+ for i, orc in enumerate(oracle):
+ cpu_tok = cm.get(i, orc) # matched oracle => oracle token
+ cuda_tok = gm.get(i, orc)
+ if cpu_tok != cuda_tok:
+ diff.append(i)
+ agree = (len(oracle) - len(diff)) / len(oracle) if oracle else 0.0
+ return agree, diff