Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: collapse WINDOWS.md/windows.md case collision into one file (#371)
JustVuggeb6d4325dc90
diff --git a/docs/WINDOWS.md b/docs/WINDOWS.mddeleted file mode 100644index 52d83cd0..00000000--- a/docs/WINDOWS.md+++ /dev/null@@ -1,100 +0,0 @@-# Windows 11 native install β a complete walkthrough (no WSL)--A start-to-finish, reproducible path from a fresh Windows 11 machine to GLM-5.2 generating tokens, with the GPU tier. Every step and every failure mode below was hit and verified on real hardware: Core Ultra 9 285K (AVX-VNNI) / RTX 5080 (sm_120) / 128 GB RAM / Windows 11 24H2 (issue #306). Steps are ordered so the long downloads run while you build.--## 0. What you need--| Piece | Why | Get it |-|---|---|---|-| git, Python 3 | clone + `coli` launcher | winget / python.org |-| MinGW-w64 gcc + make | builds the engine (MSVC can't) | `scoop install mingw-winlibs`, MSYS2, or portable **w64devkit** (no admin, unzip and go) |-| CUDA Toolkit β₯ 12.8 | GPU tier; β₯12.8 required for Blackwell/sm_120 | `winget install Nvidia.CUDA` |-| MSVC Build Tools (C++ workload) | nvcc's host compiler for the CUDA DLL | `winget install Microsoft.VisualStudio.2022.BuildTools` + "Desktop development with C++" |-| ~400 GB free on a local NVMe | the int4 model (~370β384 GB) | NTFS is fine; **never** a network mount |--RAM: 16 GB minimum, more = bigger expert cache = faster. The build itself needs none of the CUDA/MSVC pieces β do the CPU build first, add the GPU tier later.--## 1. Start the model download first (it's the long pole)--```powershell-python -m pip install -U "huggingface_hub[hf_transfer]"-$env:HF_HUB_ENABLE_HF_TRANSFER = "1"-hf download <model-repo> --local-dir D:\glm52_i4-```--Use the container recommended in the README (with **int8 MTP heads** β int4 heads silently give 0% draft acceptance). The download is resumable: if it stops, rerun the same command. Expect hours; everything below fits inside them.--## 2. Build the engine (CPU)--From a normal PowerShell, in the repo's `c\` directory:--```powershell-make glm.exe ARCH=native # ARCH=native unlocks AVX-VNNI on Alder Lake+/Arrow Lake-make iobench.exe # disk benchmark, useful before committing to the download-```--Warnings about `#pragma comment` and unused variables are normal (MSVC-isms gcc ignores). The engine banner should print `idot: avx-vnni` on VNNI-capable CPUs β if it says avx2, you built without `ARCH=native`.--### β οΈ Smart App Control will block your fresh binary--On Windows 11 machines with **Smart App Control** enforced (`VerifiedAndReputablePolicyState = 1`), running your self-compiled `glm.exe` fails with:--```-Program 'glm.exe' failed to run: An Application Control policy has blocked this file-```--This is not Defender and not Mark-of-the-Web β SAC blocks *all* unsigned, unknown binaries, which includes anything you compile yourself. **Fix:** Windows Security β App & browser control β Smart App Control settings β **Off**, then **reboot** (the policy only reloads on restart). Note SAC is one-way: re-enabling later requires resetting Windows. If the settings page is missing, the registry equivalent is setting `HKLM:\SYSTEM\CurrentControlSet\Control\CI\Policy\VerifiedAndReputablePolicyState` to `0` (admin PowerShell), then rebooting. Check your current state before touching anything:--```powershell-(Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Control\CI\Policy").VerifiedAndReputablePolicyState-# 0 = off, 1 = enforced, 2 = evaluation-```--## 3. Build the CUDA DLL (GPU tier)--nvcc needs MSVC as host compiler, so this one step must run from a shell with the MSVC environment: open **"x64 Native Tools Command Prompt for VS 2022"** from the Start menu (plain PowerShell will fail the `cl` check). Then:--```cmd-make cuda-dll CUDA_ARCH=sm_120 # match your GPU: sm_120 Blackwell, sm_89 Ada, ...-make glm.exe CUDA_DLL=1 ARCH=native # relink host with the runtime loader-```--Two pitfalls, both fixed on current `dev` (#314) but worth knowing on older checkouts:--- **Spaces in `CUDA_HOME`** (`C:\Program Files\...`) used to break the recipe β fixed; nvcc now comes from PATH and `"$(NVCC)"` is quoted.-- **`make glm.exe CUDA_DLL=1` after a CPU-only build** used to report `up to date` and silently keep the CPU-only binary (GPU tier never engages, no error). Current `dev` has a build-config stamp that forces the relink. On older trees: delete `glm.exe` first.--Sanity check: first GPU run should print `[CUDA] device 0: <your GPU>, ... sm_XX` and `[CUDA] mode: routed experts + resident dense tensors`.--## 4. First run--```powershell-cd <repo>\c-$env:OMP_NUM_THREADS = "<physical cores>"-python coli run "Explain what a mixture-of-experts model is." --model D:\glm52_i4 --ngen 48-```--The first run is cold β expect the profile to be dominated by `expert-disk` while the cache warms; hit rate climbs run over run. GPU tier on top:--```powershell-$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_DENSE="1"; $env:CUDA_EXPERT_GB="4"-python coli run "..." --model D:\glm52_i4 --ngen 64-```--Size `CUDA_EXPERT_GB` so dense (~10 GB) + experts + working set stays under your VRAM. Note MTP speculation is off by default under CUDA (#293, float-accumulation divergence between draft and verify) β `COLI_CUDA_MTP=1` opts back in.--## 5. Reference numbers from this walkthrough's hardware--285K / RTX 5080 / 128 GB / NVMe at 5.85 GB/s random-read (19 MB blocks, `iobench`): 0.26 tok/s cold CPU β 0.30 warm CPU (MTP 2.2β2.3 tok/forward) β 0.42 tok/s GPU tier + auto-pin, expert hit 66%, ~65% of wall time in expert-disk. Disk-bound is the expected shape at ~25% expert residency β a faster disk and more RAM move the floor, the GPU moves the compute.--## Quick failure index--| Symptom | Cause | Fix |-|---|---|---|-| `An Application Control policy has blocked this file` | Smart App Control | Β§2 β turn SAC off + **reboot** |-| `cuda-dll ... Error 1` immediately | old tree: spaced CUDA_HOME / MSVC rejects `-Wextra` | update to current `dev` (#314) |-| `glm.exe is up to date` but GPU never engages | old tree: stale CPU-only binary | update to `dev`, or delete `glm.exe` and rebuild |-| `cl.exe (MSVC) not in PATH` | built from plain PowerShell | use the x64 Native Tools prompt |-| `nvcc fatal: unsupported gpu architecture 'sm_120'` | CUDA < 12.8 | install CUDA 12.8+ |-| MTP `0% (0/0)` on CPU path | int4 MTP heads in the container | use the int8-MTP container |-| MTP `draft=0` under CUDA | intended default since #293 | `COLI_CUDA_MTP=1` to opt in |diff --git a/docs/windows.md b/docs/windows.mdindex 94a4f694..ecb81e70 100644--- a/docs/windows.md+++ b/docs/windows.md@@ -1,25 +1,107 @@-# Windows 11 (native, no WSL)+# Windows 11 native install β a complete walkthrough (no WSL)-colibrΓ¬ builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port-adds a `_WIN32` compatibility layer in `c/compat.h` that maps POSIX I/O to the-Windows API (pread β ReadFile+OVERLAPPED, posix_fadvise no-op, aligned-allocation, MoveFileEx rename, GlobalMemoryStatusEx RAM detection). All platform-differences stay in `compat.h`; the engine source is unchanged.+A start-to-finish, reproducible path from a fresh Windows 11 machine to GLM-5.2 generating tokens, with the GPU tier. Every step and every failure mode below was hit and verified on real hardware: Core Ultra 9 285K (AVX-VNNI) / RTX 5080 (sm_120) / 128 GB RAM / Windows 11 24H2 (issue #306). Steps are ordered so the long downloads run while you build.-**Toolchain:** GCC via [winlibs](https://winlibs.com/) or MSYS2 MinGW-w64.-Tested with GCC 16.1.0 (x86_64-ucrt-posix-seh).+## 0. What you need++| Piece | Why | Get it |+|---|---|---|+| git, Python 3 | clone + `coli` launcher | winget / python.org |+| MinGW-w64 gcc + make | builds the engine (MSVC can't) | `scoop install mingw-winlibs`, MSYS2, or portable **w64devkit** (no admin, unzip and go) |+| CUDA Toolkit β₯ 12.8 | GPU tier; β₯12.8 required for Blackwell/sm_120 | `winget install Nvidia.CUDA` |+| MSVC Build Tools (C++ workload) | nvcc's host compiler for the CUDA DLL | `winget install Microsoft.VisualStudio.2022.BuildTools` + "Desktop development with C++" |+| ~400 GB free on a local NVMe | the int4 model (~370β384 GB) | NTFS is fine; **never** a network mount |++RAM: 16 GB minimum, more = bigger expert cache = faster. The build itself needs none of the CUDA/MSVC pieces β do the CPU build first, add the GPU tier later.++## 1. Start the model download first (it's the long pole)++```powershell+python -m pip install -U "huggingface_hub[hf_transfer]"+$env:HF_HUB_ENABLE_HF_TRANSFER = "1"+hf download <model-repo> --local-dir D:\glm52_i4+```++Use the container recommended in the README (with **int8 MTP heads** β int4 heads silently give 0% draft acceptance). The download is resumable: if it stops, rerun the same command. Expect hours; everything below fits inside them.++## 2. Build the engine (CPU)++From a normal PowerShell, in the repo's `c\` directory:++```powershell+make glm.exe ARCH=native # ARCH=native unlocks AVX-VNNI on Alder Lake+/Arrow Lake+make iobench.exe # disk benchmark, useful before committing to the download+```++Warnings about `#pragma comment` and unused variables are normal (MSVC-isms gcc ignores). The engine banner should print `idot: avx-vnni` on VNNI-capable CPUs β if it says avx2, you built without `ARCH=native`.++### β οΈ Smart App Control will block your fresh binary++On Windows 11 machines with **Smart App Control** enforced (`VerifiedAndReputablePolicyState = 1`), running your self-compiled `glm.exe` fails with:++```+Program 'glm.exe' failed to run: An Application Control policy has blocked this file+```++This is not Defender and not Mark-of-the-Web β SAC blocks *all* unsigned, unknown binaries, which includes anything you compile yourself. **Fix:** Windows Security β App & browser control β Smart App Control settings β **Off**, then **reboot** (the policy only reloads on restart). Note SAC is one-way: re-enabling later requires resetting Windows. If the settings page is missing, the registry equivalent is setting `HKLM:\SYSTEM\CurrentControlSet\Control\CI\Policy\VerifiedAndReputablePolicyState` to `0` (admin PowerShell), then rebooting. Check your current state before touching anything:++```powershell+(Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Control\CI\Policy").VerifiedAndReputablePolicyState+# 0 = off, 1 = enforced, 2 = evaluation+```++## 3. Build the CUDA DLL (GPU tier)++nvcc needs MSVC as host compiler, so this one step must run from a shell with the MSVC environment: open **"x64 Native Tools Command Prompt for VS 2022"** from the Start menu (plain PowerShell will fail the `cl` check). Then:++```cmd+make cuda-dll CUDA_ARCH=sm_120 # match your GPU: sm_120 Blackwell, sm_89 Ada, ...+make glm.exe CUDA_DLL=1 ARCH=native # relink host with the runtime loader+```++Two pitfalls, both fixed on current `dev` (#314) but worth knowing on older checkouts:++- **Spaces in `CUDA_HOME`** (`C:\Program Files\...`) used to break the recipe β fixed; nvcc now comes from PATH and `"$(NVCC)"` is quoted.+- **`make glm.exe CUDA_DLL=1` after a CPU-only build** used to report `up to date` and silently keep the CPU-only binary (GPU tier never engages, no error). Current `dev` has a build-config stamp that forces the relink. On older trees: delete `glm.exe` first.++Sanity check: first GPU run should print `[CUDA] device 0: <your GPU>, ... sm_XX` and `[CUDA] mode: routed experts + resident dense tensors`.++## 4. First run```powershell-# One-time toolchain install (pick one):-scoop install mingw-winlibs # portable, no shell needed-# or: pacman -S mingw-w64-x86_64-gcc make # via MSYS2--# Build (from c/ directory):-make glm.exe # GLM-5.2 engine (static, no DLL dependencies)-make olmoe.exe # OLMoE engine (same shims)-make iobench.exe # disk I/O benchmark-make test-c # run C tests-make test-python # run Python tests (requires python)+cd <repo>\c+$env:OMP_NUM_THREADS = "<physical cores>"+python coli run "Explain what a mixture-of-experts model is." --model D:\glm52_i4 --ngen 48+```++The first run is cold β expect the profile to be dominated by `expert-disk` while the cache warms; hit rate climbs run over run. GPU tier on top:++```powershell+$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_DENSE="1"; $env:CUDA_EXPERT_GB="4"+python coli run "..." --model D:\glm52_i4 --ngen 64+```++Size `CUDA_EXPERT_GB` so dense (~10 GB) + experts + working set stays under your VRAM. Note MTP speculation is off by default under CUDA (#293, float-accumulation divergence between draft and verify) β `COLI_CUDA_MTP=1` opts back in.++## 5. Reference numbers from this walkthrough's hardware++285K / RTX 5080 / 128 GB / NVMe at 5.85 GB/s random-read (19 MB blocks, `iobench`): 0.26 tok/s cold CPU β 0.30 warm CPU (MTP 2.2β2.3 tok/forward) β 0.42 tok/s GPU tier + auto-pin, expert hit 66%, ~65% of wall time in expert-disk. Disk-bound is the expected shape at ~25% expert residency β a faster disk and more RAM move the floor, the GPU moves the compute.++## Quick failure index++| Symptom | Cause | Fix |+|---|---|---|+| `An Application Control policy has blocked this file` | Smart App Control | Β§2 β turn SAC off + **reboot** |+| `cuda-dll ... Error 1` immediately | old tree: spaced CUDA_HOME / MSVC rejects `-Wextra` | update to current `dev` (#314) |+| `glm.exe is up to date` but GPU never engages | old tree: stale CPU-only binary | update to `dev`, or delete `glm.exe` and rebuild |+| `cl.exe (MSVC) not in PATH` | built from plain PowerShell | use the x64 Native Tools prompt |+| `nvcc fatal: unsupported gpu architecture 'sm_120'` | CUDA < 12.8 | install CUDA 12.8+ |+| MTP `0% (0/0)` on CPU path | int4 MTP heads in the container | use the int8-MTP container |+| MTP `draft=0` under CUDA | intended default since #293 | `COLI_CUDA_MTP=1` to opt in |++---++## Reference: build flags & warmup# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster