Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
build(v4/cuda): fetch the pinned DeepGEMM sm120 headers on first build β cuda-dsv4-dg-dll / DEEPGEMM=1 need no external checkout, and nothing is vendored
dcutugno467bec4ac1ba
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 321ab856..77ed7e32 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -535,7 +535,30 @@ jobs:path-type: inheritinstall: >-make+ gitmingw-w64-ucrt-x86_64-gcc+ # The DeepSeek V4 DeepGEMM flavour compiles against a pinned checkout+ # that tools/fetch_deepgemm.sh makes on first use (nothing vendored).+ # CI cannot compile it (sm_120a needs CUDA >= 12.8; this job pins 12.6)+ # but it does exercise the fetch β pin verification + MSVC patch β and+ # caches the tree keyed on the pin so a bump is the only cache miss.+ - name: DeepGEMM pin (for the cache key)+ id: deepgemm-pin+ shell: bash+ run: echo "pin=$(sed -n 's/^DEEPGEMM_PIN ?= //p' c/Makefile | head -1)" >> "$GITHUB_OUTPUT"+ - name: Cache the pinned DeepGEMM checkout+ uses: actions/cache@v4+ with:+ path: c/third_party/deepgemm+ key: deepgemm-${{ steps.deepgemm-pin.outputs.pin }}-${{ hashFiles('c/patches-deepgemm-sm120-msvc.patch') }}+ - name: make deepgemm-fetch (pinned checkout + MSVC patch, no compile)+ shell: msys2 {0}+ run: |+ cd c+ make deepgemm-fetch+ test -f third_party/deepgemm/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh || { echo "fetch_deepgemm produced no sm120 headers" >&2; exit 1; }+ # second run must be a no-op (idempotency)+ make deepgemm-fetch | grep -q "already at" || { echo "fetch_deepgemm is not idempotent" >&2; exit 1; }- name: make cuda-dll (nvcc + MSVC host)shell: msys2 {0}run: |diff --git a/.gitignore b/.gitignoreindex 982972c7..c80c1ed5 100644--- a/.gitignore+++ b/.gitignore@@ -121,6 +121,10 @@ c/tiny_inkling/# Claude Code working directory (agent worktrees, session scratch) β never a repo artifact.claude/+# DeepGEMM sm120 checkout made by c/tools/fetch_deepgemm.sh (pinned commit,+# not vendored): the DeepSeek V4 DeepGEMM flavour's headers.+c/third_party/deepgemm/+# DLL flavours + MSVC import/export artifacts of the V4 CUDA tier (regenerated# by cuda-dsv4-dll / cuda-dsv4-dg-dll); the loader picks the DLL at runtime.c/coli_cuda_dsv4*.dlldiff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdnew file mode 100644index 00000000..ca473c78--- /dev/null+++ b/THIRD_PARTY_NOTICES.md@@ -0,0 +1,50 @@+# Third-party notices++Portions of `backend_cuda_dsv4.cu`, including the DeepSeek-V4 GPU router+selection algorithm, are adapted from `ds4_cuda.cu` in the ds4 project:+https://github.com/antirez/ds4++MIT License++Copyright (c) 2026 The ds4.c authors+Copyright (c) 2023-2026 The ggml authors++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.++The pinned DeepSeek-V4 reference runner in `c/tools/dsv4_vllm_reference.py`+executes an unmodified vLLM checkout (commit+`ffd46bfab2128bb84146050e98b51a617c6575ab`) as a behavioural oracle for the+native port; no vLLM code is vendored.++## DeepGEMM sm120 headers (fetched, not vendored: `c/third_party/deepgemm/`)++The DeepSeek V4 CUDA tier's DeepGEMM flavour (`make cuda-dsv4-dg-dll`,+`Makefile.deepseek-v4 DEEPGEMM=1`) compiles against headers that+`c/tools/fetch_deepgemm.sh` checks out at a pinned commit into the gitignored+`c/third_party/deepgemm/`; nothing from them is committed to this repository.+What that checkout contains, and the licences that apply when you build with it:++- DeepGEMM (MIT, Copyright (c) 2025 DeepSeek), from the community sm120 port+ https://github.com/bvolpato/DeepGEMM branch `sm120-full` @+ 39fb4447a062b418fd08ce17cd308adb28559417, with+ `c/patches-deepgemm-sm120-msvc.patch` applied at fetch time. License text:+ `c/third_party/deepgemm/LICENSE` after the fetch.+- NVIDIA CUTLASS / CuTe (BSD-3-Clause), the `third-party/cutlass` submodule of+ that commit @ f3fde58372d33e9a5650ba7b80fc48b3b49d40c8. License text:+ `c/third_party/deepgemm/third-party/cutlass/LICENSE.txt` after the fetch.diff --git a/c/Makefile b/c/Makefileindex ee941025..2276b5d2 100644--- a/c/Makefile+++ b/c/Makefile@@ -201,6 +201,12 @@ COLI_ANS ?= 0DIETGPU_ROOT ?=DEEPGEMM ?= 0NCCL ?= 0+# DeepGEMM sm120 headers for the DeepSeek V4 DeepGEMM flavour: a pinned+# checkout made by tools/fetch_deepgemm.sh on first use (gitignored), see the+# DeepSeek V4 CUDA kernels block and the cuda-dsv4-dg-dll target below.+DEEPGEMM_HOME ?= third_party/deepgemm+DEEPGEMM_PIN ?= 39fb4447a062b418fd08ce17cd308adb28559417+DEEPGEMM_STAMP = $(DEEPGEMM_HOME)/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuhifneq ($(IS_WIN),)# the CUDA installer sets CUDA_PATH system-wide (e.g. C:\Program Files\NVIDIA# GPU Computing Toolkit\CUDA\v13.2); fall back to it before the POSIX default.@@ -488,14 +494,19 @@ endif# objects are self-contained: backend_cuda_dsv4.* include only their own header# and the CUDA runtime, so they build and their tests run without the engine.#-# DEEPGEMM is unsupported: CI cannot compile it without an external checkout,-# and its kernels require Blackwell to run. It gates nothing and blocks nothing;-# the default build is the supported configuration.+# DEEPGEMM is an opt-in: its kernels require Blackwell (sm_120a) to run and CI+# does not compile it. It gates nothing and blocks nothing; the default build+# (generic kernels, any sm_80+) is the supported configuration and the one to+# report bugs against. The sm120 headers it needs are NOT in this repository:+# tools/fetch_deepgemm.sh checks out the community sm120 port at a pinned+# commit into third_party/deepgemm (gitignored), verifies the pin, applies+# patches-deepgemm-sm120-msvc.patch, and DEEPGEMM=1 / cuda-dsv4-dg-dll invoke+# it on first build (deepgemm-fetch). DEEPGEMM_HOME points at an existing+# checkout instead; DEEPGEMM_PIN bumps the commit. Attribution:+# THIRD_PARTY_NOTICES.md.DSV4_CUDA_OBJ = backend_cuda_dsv4.oifeq ($(DEEPGEMM),1)-ifndef DEEPGEMM_HOME-$(error DEEPGEMM_HOME must point to a DeepGEMM include tree)-endif+DSV4_DEEPGEMM_DEP = $(DEEPGEMM_STAMP)CFLAGS += -DCOLI_DSV4_DEEPGEMM# Append, never replace. This was `:=` and discarded the whole inherited flag# set: $(CUDA_GENCODE), so any CUDA_ARCH the user chose was silently swapped for@@ -718,10 +729,17 @@ backend_cuda.o: backend_cuda.cu backend_cuda.h backend_gpu_compat.h .build-confi"$(GPUCC)" $(GPUFLAGS) -c backend_cuda.cu -o $@-backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h .build-config+backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h .build-config $(DSV4_DEEPGEMM_DEP)@command -v "$(NVCC)" >/dev/null 2>&1 || { echo "nvcc not found: set CUDA_HOME or NVCC" >&2; exit 1; }"$(NVCC)" $(NVCCFLAGS) -c backend_cuda_dsv4.cu -o $@+# The pinned DeepGEMM sm120 checkout (see the DEEPGEMM block above). The stamp+# is a header that only exists in the sm120 port, so an empty or wrong tree+# refetches; the script itself is a no-op when the tree is already at the pin.+deepgemm-fetch $(DEEPGEMM_STAMP):+ DEEPGEMM_HOME="$(DEEPGEMM_HOME)" tools/fetch_deepgemm.sh "$(DEEPGEMM_PIN)"+.PHONY: deepgemm-fetch+# Windows DeepSeek V4 CUDA DLLs: the sibling of cuda-dll for the DeepSeek V4# engine. deepseek-v4.exe is built with MinGW gcc and cannot link MSVC/nvcc# objects, so backend_cuda_dsv4.cu compiles into its own DLL (nvcc + MSVC@@ -742,9 +760,11 @@ cuda-dsv4-dll: backend_cuda_dsv4.cu backend_cuda_dsv4.h dsv4.def-L"$(CUDA_HOME)/lib/x64" -lcublasLt -lcudart -lcuda \backend_cuda_dsv4.cu -o coli_cuda_dsv4.dll+# The DeepGEMM sm120 headers come from tools/fetch_deepgemm.sh (pinned+# checkout into $(DEEPGEMM_HOME), see the DEEPGEMM block above); this target+# fetches them on first use.DEEPGEMM_INC = -I"$(DEEPGEMM_HOME)/deep_gemm/include" -I"$(DEEPGEMM_HOME)/cutlass/include" -I"$(DEEPGEMM_HOME)/third-party/cutlass/include"-cuda-dsv4-dg-dll: backend_cuda_dsv4.cu backend_cuda_dsv4.h dsv4.def- @test -n "$(DEEPGEMM_HOME)" || { echo "DEEPGEMM_HOME must point to the sm120 DeepGEMM checkout" >&2; exit 1; }+cuda-dsv4-dg-dll: backend_cuda_dsv4.cu backend_cuda_dsv4.h dsv4.def $(DEEPGEMM_STAMP)@command -v "$(NVCC)" >/dev/null 2>&1 || { echo "nvcc not found: set CUDA_HOME or NVCC" >&2; exit 1; }@command -v cl >/dev/null 2>&1 || { echo "cl.exe (MSVC) not in PATH β run vcvars64.bat first" >&2; exit 1; }"$(NVCC)" -O3 -std=c++20 -ftz=false --expt-relaxed-constexpr --expt-extended-lambda -Xcompiler=/Zc:preprocessor \diff --git a/c/Makefile.deepseek-v4 b/c/Makefile.deepseek-v4index 47083716..c6fcee2c 100644--- a/c/Makefile.deepseek-v4+++ b/c/Makefile.deepseek-v4@@ -122,9 +122,12 @@ ifeq ($(DEEPGEMM),1)# The DeepGEMM kernels use sm_120a-only PTX (block-scaled MMA); they cannot be# compiled for compute_120 or any other target, so DEEPGEMM=1 builds for# sm_120a only (CUDA_ARCH is ignored). Use the generic build for other cards.-ifndef DEEPGEMM_HOME-$(error DEEPGEMM_HOME must point to the sm120 DeepGEMM checkout (see patches-deepgemm-sm120-msvc.patch))-endif+# Headers: a pinned checkout made by tools/fetch_deepgemm.sh on first build+# (gitignored third_party/deepgemm); DEEPGEMM_HOME points at an existing tree.+DEEPGEMM_HOME ?= third_party/deepgemm+DEEPGEMM_PIN ?= 39fb4447a062b418fd08ce17cd308adb28559417+DEEPGEMM_STAMP = $(DEEPGEMM_HOME)/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh+V4_DEEPGEMM_DEP = $(DEEPGEMM_STAMP)V4_CUDA_GENCODE = -gencode arch=compute_120a,code=sm_120aV4_DEEPGEMM_FLAGS = -DCOLI_DSV4_DEEPGEMM \-I$(DEEPGEMM_HOME)/deep_gemm/include -I$(DEEPGEMM_HOME)/cutlass/include \@@ -201,9 +204,14 @@ backend_loader_dsv4.o: backend_loader_dsv4.c backend_cuda_dsv4.h$(CC) $(CFLAGS) -c backend_loader_dsv4.c -o $@# Linux/macOS CUDA=1: the tier compiled straight into the engine.-backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h+backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h $(V4_DEEPGEMM_DEP)"$(NVCC)" $(V4_NVCCFLAGS) -c backend_cuda_dsv4.cu -o $@+ifneq ($(DEEPGEMM_STAMP),)+$(DEEPGEMM_STAMP):+ DEEPGEMM_HOME="$(DEEPGEMM_HOME)" tools/fetch_deepgemm.sh "$(DEEPGEMM_PIN)"+endif+deepseek-v4-test-objs: $(TEST_UNIT_OBJS)# Registry unit test (no engine link β stubs the auto open fn).diff --git a/c/tools/fetch_deepgemm.sh b/c/tools/fetch_deepgemm.shnew file mode 100755index 00000000..707d82fc--- /dev/null+++ b/c/tools/fetch_deepgemm.sh@@ -0,0 +1,78 @@+#!/usr/bin/env bash+# Fetch the sm120 DeepGEMM headers the DeepSeek V4 CUDA tier's DeepGEMM flavour+# needs (cuda-dsv4-dg-dll on Windows, Makefile.deepseek-v4 DEEPGEMM=1 on Linux)+# at a pinned commit, verify the pin, and apply patches-deepgemm-sm120-msvc.patch.+#+# tools/fetch_deepgemm.sh [<pinned-sha>] [<dest>]+#+# Defaults: the pin below and c/third_party/deepgemm (gitignored). Nothing is+# vendored in this repository: the tree is a git checkout of+# https://github.com/bvolpato/DeepGEMM (the community sm120 port β upstream+# DeepGEMM has no sm120 kernels; MIT) plus its NVIDIA CUTLASS/CuTe submodule+# (BSD-3-Clause), both at the commits recorded here. Attribution lives in+# THIRD_PARTY_NOTICES.md regardless of whether the tree is present.+#+# Idempotent: a tree already at the pin with the patch applied is left alone.+# A tree at a different commit is refetched (bump the pin in one place). The+# checkout is shallow (one commit each), ~40 MB.+set -euo pipefail++DEEPGEMM_REPO=${DEEPGEMM_REPO:-https://github.com/bvolpato/DeepGEMM.git}+DEEPGEMM_PIN_DEFAULT=39fb4447a062b418fd08ce17cd308adb28559417 # branch sm120-full, "Add SM120 heuristic calibration harness"+CUTLASS_PIN_DEFAULT=f3fde58372d33e9a5650ba7b80fc48b3b49d40c8 # third-party/cutlass gitlink at that commit++pin=${1:-${DEEPGEMM_PIN:-$DEEPGEMM_PIN_DEFAULT}}+here=$(cd "$(dirname "$0")/.." && pwd) # c/+dest=${2:-${DEEPGEMM_HOME:-$here/third_party/deepgemm}}+case "$dest" in /*|[A-Za-z]:*) ;; *) dest="$here/$dest" ;; esac+patch="$here/patches-deepgemm-sm120-msvc.patch"+cutlass_pin=${CUTLASS_PIN:-$CUTLASS_PIN_DEFAULT}+marker="$dest/.coli-deepgemm-fetched"++die() { echo "fetch_deepgemm: $*" >&2; exit 1; }+command -v git >/dev/null 2>&1 || die "git is required (MSYS2: pacman -S git)"+[ -f "$patch" ] || die "missing $patch"+# The patch is LF in git; on a core.autocrlf checkout it arrives CRLF, so+# normalise before hashing/applying (git apply is strict about CR).+patch_sum=$(tr -d '\r' < "$patch" | git hash-object --stdin)++if [ -f "$marker" ] && [ "$(cat "$marker" 2>/dev/null)" = "$pin $cutlass_pin $patch_sum" ] \+ && [ -f "$dest/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh" ]; then+ echo "fetch_deepgemm: $dest already at $pin (patched)"+ exit 0+fi++if [ -e "$dest" ] && [ ! -d "$dest/.git" ]; then+ die "$dest exists but is not a git checkout; move it away or set DEEPGEMM_HOME"+fi+if [ -d "$dest/.git" ]; then+ echo "fetch_deepgemm: $dest is not at $pin β refetching"+ rm -rf "$dest"+fi++mkdir -p "$dest"+git -C "$dest" init -q+# The checkout must stay LF regardless of a global core.autocrlf (Windows/MSYS2+# users): the patch is LF and nvcc/MSVC do not care about the endings.+git -C "$dest" config core.autocrlf false+git -C "$dest" config core.eol lf+git -C "$dest" remote add origin "$DEEPGEMM_REPO"+echo "fetch_deepgemm: fetching $DEEPGEMM_REPO @ $pin"+git -C "$dest" fetch -q --depth 1 origin "$pin"+git -C "$dest" checkout -q FETCH_HEAD+git -C "$dest" -c core.autocrlf=false -c core.eol=lf submodule update -q --init --depth 1 third-party/cutlass++# Verify the pins: the commit ids are the checksums (content-addressed), for+# the port and for the CUTLASS gitlink it records.+got=$(git -C "$dest" rev-parse HEAD)+[ "$got" = "$pin" ] || die "DeepGEMM checkout is $got, expected $pin"+gotc=$(git -C "$dest/third-party/cutlass" rev-parse HEAD)+[ "$gotc" = "$cutlass_pin" ] || die "CUTLASS submodule is $gotc, expected $cutlass_pin"+[ -f "$dest/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh" ] || die "no sm120 kernels in $dest (wrong pin?)"++# MSVC ignores alignas(64) on by-value CUtensorMap kernel parameters; the+# patch pads them (and a few constexpr/inline-asm host fixes for cl.exe).+tr -d '\r' < "$patch" | git -C "$dest" apply --check || die "patch does not apply cleanly at $pin"+tr -d '\r' < "$patch" | git -C "$dest" apply+echo "$pin $cutlass_pin $patch_sum" > "$marker"+echo "fetch_deepgemm: ready at $dest (DeepGEMM $pin, CUTLASS $cutlass_pin, patch $patch_sum)"diff --git a/docs/deepseek-v4.md b/docs/deepseek-v4.mdindex b37ba559..611ee478 100644--- a/docs/deepseek-v4.md+++ b/docs/deepseek-v4.md@@ -77,15 +77,19 @@ picks at start-up by GPU:| DLL | build target | kernels | GPUs ||---|---|---|---|-| `coli_cuda_dsv4_dg.dll` | `make cuda-dsv4-dg-dll DEEPGEMM_HOME=<sm120 DeepGEMM checkout>` (from a vcvars64 shell) | DeepGEMM tensor-core prefill paths + generic kernels | compute capability 12.x (RTX 50-series) |+| `coli_cuda_dsv4_dg.dll` | `make cuda-dsv4-dg-dll` (from a vcvars64 shell; fetches the pinned DeepGEMM headers on first use) | DeepGEMM tensor-core prefill paths + generic kernels | compute capability 12.x (RTX 50-series) || `coli_cuda_dsv4.dll` | `make cuda-dsv4-dll CUDA_ARCH=portable` | generic CUDA kernels only (fp32 compute from fp8/fp4 decode) | any sm_80+ (RTX 30/40/50, A/H series) |-The DeepGEMM flavour is an opt-in build (as in #772): `DEEPGEMM_HOME` must-point at the community sm120 port of DeepGEMM (bvolpato/DeepGEMM branch-`sm120-full`; upstream DeepGEMM has no sm120 kernels) with-`c/patches-deepgemm-sm120-msvc.patch` applied β MSVC ignores `alignas(64)` on-by-value TMA descriptors and the patch pads them. The generic DLL needs no-external tree. Loader rules:+The DeepGEMM sm120 headers the DLL needs are not in the repository:+`c/tools/fetch_deepgemm.sh` checks out the community sm120 port of DeepGEMM+(MIT; upstream DeepGEMM has no sm120 kernels) plus its CUTLASS/CuTe submodule+(BSD-3) at a pinned commit into the gitignored `c/third_party/deepgemm/`,+verifies both commit ids, and applies `c/patches-deepgemm-sm120-msvc.patch`+(MSVC ignores `alignas(64)` on by-value TMA descriptors β the patch pads+them). `make cuda-dsv4-dg-dll` and `DEEPGEMM=1` run it on first build+(`make deepgemm-fetch` runs it alone; ~40 MB, git required);+`DEEPGEMM_HOME=<dir>` points at an existing tree instead and `DEEPGEMM_PIN`+bumps the commit. Attribution: `THIRD_PARTY_NOTICES.md`. Loader rules:try `_dg` first, ask it whether device 0 fits (`dsv4_cuda_backend_arch_ok`),otherwise load the generic DLL; `COLI_DSV4_DLL=<file name>` forces one. Thestart-up banner names the choice: `[DSV4 CUDA] backend=coli_cuda_dsv4_dg.dll@@ -96,7 +100,7 @@ start-up banner names the choice: `[DSV4 CUDA] backend=coli_cuda_dsv4_dg.dll```bashcd cmake -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=portable # generic kernels, any sm_80+-make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 DEEPGEMM=1 DEEPGEMM_HOME=<dir> # DeepGEMM flavour, sm_120a only+make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 DEEPGEMM=1 # DeepGEMM flavour, sm_120a only (fetches the pinned headers)make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=sm_86 NO_TC=1 # toolkit older than 12.8```