Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

build(v4/cuda): fetch the pinned DeepGEMM sm120 headers on first build β€” cuda-dsv4-dg-dll / DEEPGEMM=1 need no external checkout, and nothing is vendored

dcutugno467bec4ac1ba
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 321ab856..77ed7e32 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -535,7 +535,30 @@ jobs:
path-type: inherit
install: >-
make
+ git
mingw-w64-ucrt-x86_64-gcc
+ # The DeepSeek V4 DeepGEMM flavour compiles against a pinned checkout
+ # that tools/fetch_deepgemm.sh makes on first use (nothing vendored).
+ # CI cannot compile it (sm_120a needs CUDA >= 12.8; this job pins 12.6)
+ # but it does exercise the fetch β€” pin verification + MSVC patch β€” and
+ # caches the tree keyed on the pin so a bump is the only cache miss.
+ - name: DeepGEMM pin (for the cache key)
+ id: deepgemm-pin
+ shell: bash
+ run: echo "pin=$(sed -n 's/^DEEPGEMM_PIN ?= //p' c/Makefile | head -1)" >> "$GITHUB_OUTPUT"
+ - name: Cache the pinned DeepGEMM checkout
+ uses: actions/cache@v4
+ with:
+ path: c/third_party/deepgemm
+ key: deepgemm-${{ steps.deepgemm-pin.outputs.pin }}-${{ hashFiles('c/patches-deepgemm-sm120-msvc.patch') }}
+ - name: make deepgemm-fetch (pinned checkout + MSVC patch, no compile)
+ shell: msys2 {0}
+ run: |
+ cd c
+ make deepgemm-fetch
+ test -f third_party/deepgemm/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh || { echo "fetch_deepgemm produced no sm120 headers" >&2; exit 1; }
+ # second run must be a no-op (idempotency)
+ make deepgemm-fetch | grep -q "already at" || { echo "fetch_deepgemm is not idempotent" >&2; exit 1; }
- name: make cuda-dll (nvcc + MSVC host)
shell: msys2 {0}
run: |
diff --git a/.gitignore b/.gitignore
index 982972c7..c80c1ed5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -121,6 +121,10 @@ c/tiny_inkling/
# Claude Code working directory (agent worktrees, session scratch) β€” never a repo artifact
.claude/
+# DeepGEMM sm120 checkout made by c/tools/fetch_deepgemm.sh (pinned commit,
+# not vendored): the DeepSeek V4 DeepGEMM flavour's headers.
+c/third_party/deepgemm/
+
# DLL flavours + MSVC import/export artifacts of the V4 CUDA tier (regenerated
# by cuda-dsv4-dll / cuda-dsv4-dg-dll); the loader picks the DLL at runtime.
c/coli_cuda_dsv4*.dll
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
new file mode 100644
index 00000000..ca473c78
--- /dev/null
+++ b/THIRD_PARTY_NOTICES.md
@@ -0,0 +1,50 @@
+# Third-party notices
+
+Portions of `backend_cuda_dsv4.cu`, including the DeepSeek-V4 GPU router
+selection algorithm, are adapted from `ds4_cuda.cu` in the ds4 project:
+https://github.com/antirez/ds4
+
+MIT License
+
+Copyright (c) 2026 The ds4.c authors
+Copyright (c) 2023-2026 The ggml authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+The pinned DeepSeek-V4 reference runner in `c/tools/dsv4_vllm_reference.py`
+executes an unmodified vLLM checkout (commit
+`ffd46bfab2128bb84146050e98b51a617c6575ab`) as a behavioural oracle for the
+native port; no vLLM code is vendored.
+
+## DeepGEMM sm120 headers (fetched, not vendored: `c/third_party/deepgemm/`)
+
+The DeepSeek V4 CUDA tier's DeepGEMM flavour (`make cuda-dsv4-dg-dll`,
+`Makefile.deepseek-v4 DEEPGEMM=1`) compiles against headers that
+`c/tools/fetch_deepgemm.sh` checks out at a pinned commit into the gitignored
+`c/third_party/deepgemm/`; nothing from them is committed to this repository.
+What that checkout contains, and the licences that apply when you build with it:
+
+- DeepGEMM (MIT, Copyright (c) 2025 DeepSeek), from the community sm120 port
+ https://github.com/bvolpato/DeepGEMM branch `sm120-full` @
+ 39fb4447a062b418fd08ce17cd308adb28559417, with
+ `c/patches-deepgemm-sm120-msvc.patch` applied at fetch time. License text:
+ `c/third_party/deepgemm/LICENSE` after the fetch.
+- NVIDIA CUTLASS / CuTe (BSD-3-Clause), the `third-party/cutlass` submodule of
+ that commit @ f3fde58372d33e9a5650ba7b80fc48b3b49d40c8. License text:
+ `c/third_party/deepgemm/third-party/cutlass/LICENSE.txt` after the fetch.
diff --git a/c/Makefile b/c/Makefile
index ee941025..2276b5d2 100644
--- a/c/Makefile
+++ b/c/Makefile
@@ -201,6 +201,12 @@ COLI_ANS ?= 0
DIETGPU_ROOT ?=
DEEPGEMM ?= 0
NCCL ?= 0
+# DeepGEMM sm120 headers for the DeepSeek V4 DeepGEMM flavour: a pinned
+# checkout made by tools/fetch_deepgemm.sh on first use (gitignored), see the
+# DeepSeek V4 CUDA kernels block and the cuda-dsv4-dg-dll target below.
+DEEPGEMM_HOME ?= third_party/deepgemm
+DEEPGEMM_PIN ?= 39fb4447a062b418fd08ce17cd308adb28559417
+DEEPGEMM_STAMP = $(DEEPGEMM_HOME)/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh
ifneq ($(IS_WIN),)
# the CUDA installer sets CUDA_PATH system-wide (e.g. C:\Program Files\NVIDIA
# GPU Computing Toolkit\CUDA\v13.2); fall back to it before the POSIX default.
@@ -488,14 +494,19 @@ endif
# objects are self-contained: backend_cuda_dsv4.* include only their own header
# and the CUDA runtime, so they build and their tests run without the engine.
#
-# DEEPGEMM is unsupported: CI cannot compile it without an external checkout,
-# and its kernels require Blackwell to run. It gates nothing and blocks nothing;
-# the default build is the supported configuration.
+# DEEPGEMM is an opt-in: its kernels require Blackwell (sm_120a) to run and CI
+# does not compile it. It gates nothing and blocks nothing; the default build
+# (generic kernels, any sm_80+) is the supported configuration and the one to
+# report bugs against. The sm120 headers it needs are NOT in this repository:
+# tools/fetch_deepgemm.sh checks out the community sm120 port at a pinned
+# commit into third_party/deepgemm (gitignored), verifies the pin, applies
+# patches-deepgemm-sm120-msvc.patch, and DEEPGEMM=1 / cuda-dsv4-dg-dll invoke
+# it on first build (deepgemm-fetch). DEEPGEMM_HOME points at an existing
+# checkout instead; DEEPGEMM_PIN bumps the commit. Attribution:
+# THIRD_PARTY_NOTICES.md.
DSV4_CUDA_OBJ = backend_cuda_dsv4.o
ifeq ($(DEEPGEMM),1)
-ifndef DEEPGEMM_HOME
-$(error DEEPGEMM_HOME must point to a DeepGEMM include tree)
-endif
+DSV4_DEEPGEMM_DEP = $(DEEPGEMM_STAMP)
CFLAGS += -DCOLI_DSV4_DEEPGEMM
# Append, never replace. This was `:=` and discarded the whole inherited flag
# set: $(CUDA_GENCODE), so any CUDA_ARCH the user chose was silently swapped for
@@ -718,10 +729,17 @@ backend_cuda.o: backend_cuda.cu backend_cuda.h backend_gpu_compat.h .build-confi
"$(GPUCC)" $(GPUFLAGS) -c backend_cuda.cu -o $@
-backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h .build-config
+backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h .build-config $(DSV4_DEEPGEMM_DEP)
@command -v "$(NVCC)" >/dev/null 2>&1 || { echo "nvcc not found: set CUDA_HOME or NVCC" >&2; exit 1; }
"$(NVCC)" $(NVCCFLAGS) -c backend_cuda_dsv4.cu -o $@
+# The pinned DeepGEMM sm120 checkout (see the DEEPGEMM block above). The stamp
+# is a header that only exists in the sm120 port, so an empty or wrong tree
+# refetches; the script itself is a no-op when the tree is already at the pin.
+deepgemm-fetch $(DEEPGEMM_STAMP):
+ DEEPGEMM_HOME="$(DEEPGEMM_HOME)" tools/fetch_deepgemm.sh "$(DEEPGEMM_PIN)"
+.PHONY: deepgemm-fetch
+
# Windows DeepSeek V4 CUDA DLLs: the sibling of cuda-dll for the DeepSeek V4
# engine. deepseek-v4.exe is built with MinGW gcc and cannot link MSVC/nvcc
# objects, so backend_cuda_dsv4.cu compiles into its own DLL (nvcc + MSVC
@@ -742,9 +760,11 @@ cuda-dsv4-dll: backend_cuda_dsv4.cu backend_cuda_dsv4.h dsv4.def
-L"$(CUDA_HOME)/lib/x64" -lcublasLt -lcudart -lcuda \
backend_cuda_dsv4.cu -o coli_cuda_dsv4.dll
+# The DeepGEMM sm120 headers come from tools/fetch_deepgemm.sh (pinned
+# checkout into $(DEEPGEMM_HOME), see the DEEPGEMM block above); this target
+# fetches them on first use.
DEEPGEMM_INC = -I"$(DEEPGEMM_HOME)/deep_gemm/include" -I"$(DEEPGEMM_HOME)/cutlass/include" -I"$(DEEPGEMM_HOME)/third-party/cutlass/include"
-cuda-dsv4-dg-dll: backend_cuda_dsv4.cu backend_cuda_dsv4.h dsv4.def
- @test -n "$(DEEPGEMM_HOME)" || { echo "DEEPGEMM_HOME must point to the sm120 DeepGEMM checkout" >&2; exit 1; }
+cuda-dsv4-dg-dll: backend_cuda_dsv4.cu backend_cuda_dsv4.h dsv4.def $(DEEPGEMM_STAMP)
@command -v "$(NVCC)" >/dev/null 2>&1 || { echo "nvcc not found: set CUDA_HOME or NVCC" >&2; exit 1; }
@command -v cl >/dev/null 2>&1 || { echo "cl.exe (MSVC) not in PATH β€” run vcvars64.bat first" >&2; exit 1; }
"$(NVCC)" -O3 -std=c++20 -ftz=false --expt-relaxed-constexpr --expt-extended-lambda -Xcompiler=/Zc:preprocessor \
diff --git a/c/Makefile.deepseek-v4 b/c/Makefile.deepseek-v4
index 47083716..c6fcee2c 100644
--- a/c/Makefile.deepseek-v4
+++ b/c/Makefile.deepseek-v4
@@ -122,9 +122,12 @@ ifeq ($(DEEPGEMM),1)
# The DeepGEMM kernels use sm_120a-only PTX (block-scaled MMA); they cannot be
# compiled for compute_120 or any other target, so DEEPGEMM=1 builds for
# sm_120a only (CUDA_ARCH is ignored). Use the generic build for other cards.
-ifndef DEEPGEMM_HOME
-$(error DEEPGEMM_HOME must point to the sm120 DeepGEMM checkout (see patches-deepgemm-sm120-msvc.patch))
-endif
+# Headers: a pinned checkout made by tools/fetch_deepgemm.sh on first build
+# (gitignored third_party/deepgemm); DEEPGEMM_HOME points at an existing tree.
+DEEPGEMM_HOME ?= third_party/deepgemm
+DEEPGEMM_PIN ?= 39fb4447a062b418fd08ce17cd308adb28559417
+DEEPGEMM_STAMP = $(DEEPGEMM_HOME)/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh
+V4_DEEPGEMM_DEP = $(DEEPGEMM_STAMP)
V4_CUDA_GENCODE = -gencode arch=compute_120a,code=sm_120a
V4_DEEPGEMM_FLAGS = -DCOLI_DSV4_DEEPGEMM \
-I$(DEEPGEMM_HOME)/deep_gemm/include -I$(DEEPGEMM_HOME)/cutlass/include \
@@ -201,9 +204,14 @@ backend_loader_dsv4.o: backend_loader_dsv4.c backend_cuda_dsv4.h
$(CC) $(CFLAGS) -c backend_loader_dsv4.c -o $@
# Linux/macOS CUDA=1: the tier compiled straight into the engine.
-backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h
+backend_cuda_dsv4.o: backend_cuda_dsv4.cu backend_cuda_dsv4.h $(V4_DEEPGEMM_DEP)
"$(NVCC)" $(V4_NVCCFLAGS) -c backend_cuda_dsv4.cu -o $@
+ifneq ($(DEEPGEMM_STAMP),)
+$(DEEPGEMM_STAMP):
+ DEEPGEMM_HOME="$(DEEPGEMM_HOME)" tools/fetch_deepgemm.sh "$(DEEPGEMM_PIN)"
+endif
+
deepseek-v4-test-objs: $(TEST_UNIT_OBJS)
# Registry unit test (no engine link β€” stubs the auto open fn).
diff --git a/c/tools/fetch_deepgemm.sh b/c/tools/fetch_deepgemm.sh
new file mode 100755
index 00000000..707d82fc
--- /dev/null
+++ b/c/tools/fetch_deepgemm.sh
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Fetch the sm120 DeepGEMM headers the DeepSeek V4 CUDA tier's DeepGEMM flavour
+# needs (cuda-dsv4-dg-dll on Windows, Makefile.deepseek-v4 DEEPGEMM=1 on Linux)
+# at a pinned commit, verify the pin, and apply patches-deepgemm-sm120-msvc.patch.
+#
+# tools/fetch_deepgemm.sh [<pinned-sha>] [<dest>]
+#
+# Defaults: the pin below and c/third_party/deepgemm (gitignored). Nothing is
+# vendored in this repository: the tree is a git checkout of
+# https://github.com/bvolpato/DeepGEMM (the community sm120 port β€” upstream
+# DeepGEMM has no sm120 kernels; MIT) plus its NVIDIA CUTLASS/CuTe submodule
+# (BSD-3-Clause), both at the commits recorded here. Attribution lives in
+# THIRD_PARTY_NOTICES.md regardless of whether the tree is present.
+#
+# Idempotent: a tree already at the pin with the patch applied is left alone.
+# A tree at a different commit is refetched (bump the pin in one place). The
+# checkout is shallow (one commit each), ~40 MB.
+set -euo pipefail
+
+DEEPGEMM_REPO=${DEEPGEMM_REPO:-https://github.com/bvolpato/DeepGEMM.git}
+DEEPGEMM_PIN_DEFAULT=39fb4447a062b418fd08ce17cd308adb28559417 # branch sm120-full, "Add SM120 heuristic calibration harness"
+CUTLASS_PIN_DEFAULT=f3fde58372d33e9a5650ba7b80fc48b3b49d40c8 # third-party/cutlass gitlink at that commit
+
+pin=${1:-${DEEPGEMM_PIN:-$DEEPGEMM_PIN_DEFAULT}}
+here=$(cd "$(dirname "$0")/.." && pwd) # c/
+dest=${2:-${DEEPGEMM_HOME:-$here/third_party/deepgemm}}
+case "$dest" in /*|[A-Za-z]:*) ;; *) dest="$here/$dest" ;; esac
+patch="$here/patches-deepgemm-sm120-msvc.patch"
+cutlass_pin=${CUTLASS_PIN:-$CUTLASS_PIN_DEFAULT}
+marker="$dest/.coli-deepgemm-fetched"
+
+die() { echo "fetch_deepgemm: $*" >&2; exit 1; }
+command -v git >/dev/null 2>&1 || die "git is required (MSYS2: pacman -S git)"
+[ -f "$patch" ] || die "missing $patch"
+# The patch is LF in git; on a core.autocrlf checkout it arrives CRLF, so
+# normalise before hashing/applying (git apply is strict about CR).
+patch_sum=$(tr -d '\r' < "$patch" | git hash-object --stdin)
+
+if [ -f "$marker" ] && [ "$(cat "$marker" 2>/dev/null)" = "$pin $cutlass_pin $patch_sum" ] \
+ && [ -f "$dest/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh" ]; then
+ echo "fetch_deepgemm: $dest already at $pin (patched)"
+ exit 0
+fi
+
+if [ -e "$dest" ] && [ ! -d "$dest/.git" ]; then
+ die "$dest exists but is not a git checkout; move it away or set DEEPGEMM_HOME"
+fi
+if [ -d "$dest/.git" ]; then
+ echo "fetch_deepgemm: $dest is not at $pin β€” refetching"
+ rm -rf "$dest"
+fi
+
+mkdir -p "$dest"
+git -C "$dest" init -q
+# The checkout must stay LF regardless of a global core.autocrlf (Windows/MSYS2
+# users): the patch is LF and nvcc/MSVC do not care about the endings.
+git -C "$dest" config core.autocrlf false
+git -C "$dest" config core.eol lf
+git -C "$dest" remote add origin "$DEEPGEMM_REPO"
+echo "fetch_deepgemm: fetching $DEEPGEMM_REPO @ $pin"
+git -C "$dest" fetch -q --depth 1 origin "$pin"
+git -C "$dest" checkout -q FETCH_HEAD
+git -C "$dest" -c core.autocrlf=false -c core.eol=lf submodule update -q --init --depth 1 third-party/cutlass
+
+# Verify the pins: the commit ids are the checksums (content-addressed), for
+# the port and for the CUTLASS gitlink it records.
+got=$(git -C "$dest" rev-parse HEAD)
+[ "$got" = "$pin" ] || die "DeepGEMM checkout is $got, expected $pin"
+gotc=$(git -C "$dest/third-party/cutlass" rev-parse HEAD)
+[ "$gotc" = "$cutlass_pin" ] || die "CUTLASS submodule is $gotc, expected $cutlass_pin"
+[ -f "$dest/deep_gemm/include/deep_gemm/impls/sm120_bf16_gemm.cuh" ] || die "no sm120 kernels in $dest (wrong pin?)"
+
+# MSVC ignores alignas(64) on by-value CUtensorMap kernel parameters; the
+# patch pads them (and a few constexpr/inline-asm host fixes for cl.exe).
+tr -d '\r' < "$patch" | git -C "$dest" apply --check || die "patch does not apply cleanly at $pin"
+tr -d '\r' < "$patch" | git -C "$dest" apply
+echo "$pin $cutlass_pin $patch_sum" > "$marker"
+echo "fetch_deepgemm: ready at $dest (DeepGEMM $pin, CUTLASS $cutlass_pin, patch $patch_sum)"
diff --git a/docs/deepseek-v4.md b/docs/deepseek-v4.md
index b37ba559..611ee478 100644
--- a/docs/deepseek-v4.md
+++ b/docs/deepseek-v4.md
@@ -77,15 +77,19 @@ picks at start-up by GPU:
| DLL | build target | kernels | GPUs |
|---|---|---|---|
-| `coli_cuda_dsv4_dg.dll` | `make cuda-dsv4-dg-dll DEEPGEMM_HOME=<sm120 DeepGEMM checkout>` (from a vcvars64 shell) | DeepGEMM tensor-core prefill paths + generic kernels | compute capability 12.x (RTX 50-series) |
+| `coli_cuda_dsv4_dg.dll` | `make cuda-dsv4-dg-dll` (from a vcvars64 shell; fetches the pinned DeepGEMM headers on first use) | DeepGEMM tensor-core prefill paths + generic kernels | compute capability 12.x (RTX 50-series) |
| `coli_cuda_dsv4.dll` | `make cuda-dsv4-dll CUDA_ARCH=portable` | generic CUDA kernels only (fp32 compute from fp8/fp4 decode) | any sm_80+ (RTX 30/40/50, A/H series) |
-The DeepGEMM flavour is an opt-in build (as in #772): `DEEPGEMM_HOME` must
-point at the community sm120 port of DeepGEMM (bvolpato/DeepGEMM branch
-`sm120-full`; upstream DeepGEMM has no sm120 kernels) with
-`c/patches-deepgemm-sm120-msvc.patch` applied β€” MSVC ignores `alignas(64)` on
-by-value TMA descriptors and the patch pads them. The generic DLL needs no
-external tree. Loader rules:
+The DeepGEMM sm120 headers the DLL needs are not in the repository:
+`c/tools/fetch_deepgemm.sh` checks out the community sm120 port of DeepGEMM
+(MIT; upstream DeepGEMM has no sm120 kernels) plus its CUTLASS/CuTe submodule
+(BSD-3) at a pinned commit into the gitignored `c/third_party/deepgemm/`,
+verifies both commit ids, and applies `c/patches-deepgemm-sm120-msvc.patch`
+(MSVC ignores `alignas(64)` on by-value TMA descriptors β€” the patch pads
+them). `make cuda-dsv4-dg-dll` and `DEEPGEMM=1` run it on first build
+(`make deepgemm-fetch` runs it alone; ~40 MB, git required);
+`DEEPGEMM_HOME=<dir>` points at an existing tree instead and `DEEPGEMM_PIN`
+bumps the commit. Attribution: `THIRD_PARTY_NOTICES.md`. Loader rules:
try `_dg` first, ask it whether device 0 fits (`dsv4_cuda_backend_arch_ok`),
otherwise load the generic DLL; `COLI_DSV4_DLL=<file name>` forces one. The
start-up banner names the choice: `[DSV4 CUDA] backend=coli_cuda_dsv4_dg.dll
@@ -96,7 +100,7 @@ start-up banner names the choice: `[DSV4 CUDA] backend=coli_cuda_dsv4_dg.dll
```bash
cd c
make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=portable # generic kernels, any sm_80+
-make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 DEEPGEMM=1 DEEPGEMM_HOME=<dir> # DeepGEMM flavour, sm_120a only
+make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 DEEPGEMM=1 # DeepGEMM flavour, sm_120a only (fetches the pinned headers)
make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=sm_86 NO_TC=1 # toolkit older than 12.8
```