Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

build(v4/cuda): add CUDA_ARCH=portable-pre-ampere for Pascal and Turing

8PotatoChip8214643972e08
diff --git a/c/Makefile.deepseek-v4 b/c/Makefile.deepseek-v4
index e52e725a..3c2301be 100644
--- a/c/Makefile.deepseek-v4
+++ b/c/Makefile.deepseek-v4
@@ -111,7 +111,8 @@ endif
# backend_loader_dsv4.o. Elsewhere there is no loader: the same
# backend_cuda_dsv4.cu is compiled by nvcc and linked straight into the
# engine, so the dsv4_cuda_* symbols are the real ones. Defaults to the
-# GENERIC kernels (any sm_80+; CUDA_ARCH=native|portable|sm_XX);
+# GENERIC kernels (CUDA_ARCH=native|portable|portable-pre-ampere|sm_XX;
+# portable covers sm_80+, portable-pre-ampere adds sm_61/sm_75 and needs NO_TC=1);
# DEEPGEMM=1 adds the sm_120a DeepGEMM tensor-core paths and needs the
# patched sm120 DeepGEMM checkout at DEEPGEMM_HOME (see
# patches-deepgemm-sm120-msvc.patch and docs/deepseek-v4.md). Untested here
@@ -142,6 +143,34 @@ ifeq ($(CUDA_ARCH),portable)
V4_CUDA_GENCODE = -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 \
-gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 \
-gencode arch=compute_120,code=sm_120 -gencode arch=compute_120,code=compute_120
+else ifeq ($(CUDA_ARCH),portable-pre-ampere)
+# Adds Pascal (sm_61) and Turing (sm_75) to the portable set. The generic
+# kernels do not need Ampere: bf16 appears only as storage (every use is
+# __bfloat162float / __float2bfloat16, and the compute is fp32), and fp8 is
+# weights are decoded through the 256-entry constant lookup table the engine
+# already carries -- not fp8 tensor-core MMA. The two activation-quantise
+# kernels in the generic build (fp8_sim64, fp8_sim64_dynamic) do call
+# __nv_cvt_float_to_fp8, but cuda_fp8.h resolves that to native cvt PTX only
+# at __CUDA_ARCH__ >= 890 and to its own software path below, so it builds
+# and runs on older silicon. All of it compiles for sm_61/sm_75 unchanged.
+#
+# Requires NO_TC=1: the DSV4_CUDA_TC path calls cuBLASLt block-scaling APIs
+# (CUBLASLT_MATMUL_DESC_A_SCALE_MODE and friends) that need CUDA >= 12.8, and
+# those are a TOOLKIT dependency rather than an architecture one.
+#
+# DEEPGEMM is unavailable here and cannot be made available: it calls
+# cuTensorMapEncodeTiled (TMA), a hardware unit introduced in sm_90, so there
+# is nothing to fall back to on older silicon.
+#
+# Deliberately stops at sm_90 rather than mirroring `portable` all the way to
+# sm_120: compute_120 needs CUDA >= 12.8, which is the very toolkit floor the
+# NO_TC=1 companion exists to stay below, so including it would make this
+# preset unbuildable by the toolkits its users are most likely to have. The
+# trailing compute_90 PTX keeps newer cards working through driver JIT.
+V4_CUDA_GENCODE = -gencode arch=compute_61,code=sm_61 -gencode arch=compute_75,code=sm_75 \
+ -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 \
+ -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 \
+ -gencode arch=compute_90,code=compute_90
else
V4_CUDA_GENCODE = -arch=$(CUDA_ARCH)
endif
diff --git a/docs/deepseek-v4.md b/docs/deepseek-v4.md
index 6f823517..69f2c856 100644
--- a/docs/deepseek-v4.md
+++ b/docs/deepseek-v4.md
@@ -236,10 +236,36 @@ telemetry after each request.
## GPU coverage
+The generic kernels do not require Ampere. bf16 appears only as *storage* β€”
+every use is `__bfloat162float` / `__float2bfloat16`, and the arithmetic is
+fp32 β€” and fp8 weights are decoded through the 256-entry constant lookup
+table the engine already carries, not fp8 tensor-core MMA. The generic
+build's two activation-quantise kernels (`fp8_sim64`, `fp8_sim64_dynamic`)
+do call `__nv_cvt_float_to_fp8`, but `cuda_fp8.h` resolves that to native
+`cvt` PTX only at `__CUDA_ARCH__ >= 890` and to its own software path
+below, so it builds and runs on older cards. All of it compiles unchanged
+for sm_61 and sm_75, so `CUDA_ARCH=portable-pre-ampere NO_TC=1` extends the
+generic tier to Pascal and Turing.
+
+`NO_TC=1` is required because the opt-in `DSV4_CUDA_TC` path calls cuBLASLt
+block-scaling APIs (`CUBLASLT_MATMUL_DESC_A_SCALE_MODE` and friends) that need
+CUDA >= 12.8 β€” a *toolkit* dependency, not an architecture one.
+
+This preset stops at `sm_90` plus `compute_90` PTX rather than mirroring
+`portable` up to `sm_120`. `compute_120` requires CUDA >= 12.8 β€” the same
+toolkit floor `NO_TC=1` exists to stay below β€” so including it would make the
+preset unbuildable by exactly the older toolkits its users tend to have. Newer
+cards are still covered, through driver JIT of the `compute_90` PTX.
+
+DeepGEMM is not available below sm_90 and cannot be made available: it calls
+`cuTensorMapEncodeTiled` (TMA), a hardware unit introduced in Hopper, so there
+is no software path to fall back to.
+
| build | GPUs | prefill | decode |
|---|---|---|---|
| CPU only (no DLL / `DSV4_CUDA=0` / Linux default) | β€” | CPU reference | CPU reference |
| generic DLL / `CUDA=1` | any sm_80+ | GPU attention block, indexer, generic batched MoE on the VRAM bank | GPU attention, indexer, expert mirrors |
+| generic, pre-Ampere / `CUDA=1 CUDA_ARCH=portable-pre-ampere NO_TC=1` | sm_61 (Pascal), sm_75 (Turing) | as generic | as generic |
| DeepGEMM DLL / `CUDA=1 DEEPGEMM=1` | compute 12.x | as generic + tensor-core dense/MoE GEMMs | same as generic |
| multi-GPU | β€” | single device today (`DSV4_CUDA_DEVICE` selects); expert-parallel design drafted, not implemented | β€” |