Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
build(v4/cuda): add CUDA_ARCH=portable-pre-ampere for Pascal and Turing
8PotatoChip8214643972e08
diff --git a/c/Makefile.deepseek-v4 b/c/Makefile.deepseek-v4index e52e725a..3c2301be 100644--- a/c/Makefile.deepseek-v4+++ b/c/Makefile.deepseek-v4@@ -111,7 +111,8 @@ endif# backend_loader_dsv4.o. Elsewhere there is no loader: the same# backend_cuda_dsv4.cu is compiled by nvcc and linked straight into the# engine, so the dsv4_cuda_* symbols are the real ones. Defaults to the-# GENERIC kernels (any sm_80+; CUDA_ARCH=native|portable|sm_XX);+# GENERIC kernels (CUDA_ARCH=native|portable|portable-pre-ampere|sm_XX;+# portable covers sm_80+, portable-pre-ampere adds sm_61/sm_75 and needs NO_TC=1);# DEEPGEMM=1 adds the sm_120a DeepGEMM tensor-core paths and needs the# patched sm120 DeepGEMM checkout at DEEPGEMM_HOME (see# patches-deepgemm-sm120-msvc.patch and docs/deepseek-v4.md). Untested here@@ -142,6 +143,34 @@ ifeq ($(CUDA_ARCH),portable)V4_CUDA_GENCODE = -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 \-gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 \-gencode arch=compute_120,code=sm_120 -gencode arch=compute_120,code=compute_120+else ifeq ($(CUDA_ARCH),portable-pre-ampere)+# Adds Pascal (sm_61) and Turing (sm_75) to the portable set. The generic+# kernels do not need Ampere: bf16 appears only as storage (every use is+# __bfloat162float / __float2bfloat16, and the compute is fp32), and fp8 is+# weights are decoded through the 256-entry constant lookup table the engine+# already carries -- not fp8 tensor-core MMA. The two activation-quantise+# kernels in the generic build (fp8_sim64, fp8_sim64_dynamic) do call+# __nv_cvt_float_to_fp8, but cuda_fp8.h resolves that to native cvt PTX only+# at __CUDA_ARCH__ >= 890 and to its own software path below, so it builds+# and runs on older silicon. All of it compiles for sm_61/sm_75 unchanged.+#+# Requires NO_TC=1: the DSV4_CUDA_TC path calls cuBLASLt block-scaling APIs+# (CUBLASLT_MATMUL_DESC_A_SCALE_MODE and friends) that need CUDA >= 12.8, and+# those are a TOOLKIT dependency rather than an architecture one.+#+# DEEPGEMM is unavailable here and cannot be made available: it calls+# cuTensorMapEncodeTiled (TMA), a hardware unit introduced in sm_90, so there+# is nothing to fall back to on older silicon.+#+# Deliberately stops at sm_90 rather than mirroring `portable` all the way to+# sm_120: compute_120 needs CUDA >= 12.8, which is the very toolkit floor the+# NO_TC=1 companion exists to stay below, so including it would make this+# preset unbuildable by the toolkits its users are most likely to have. The+# trailing compute_90 PTX keeps newer cards working through driver JIT.+V4_CUDA_GENCODE = -gencode arch=compute_61,code=sm_61 -gencode arch=compute_75,code=sm_75 \+ -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 \+ -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 \+ -gencode arch=compute_90,code=compute_90elseV4_CUDA_GENCODE = -arch=$(CUDA_ARCH)endifdiff --git a/docs/deepseek-v4.md b/docs/deepseek-v4.mdindex 6f823517..69f2c856 100644--- a/docs/deepseek-v4.md+++ b/docs/deepseek-v4.md@@ -236,10 +236,36 @@ telemetry after each request.## GPU coverage+The generic kernels do not require Ampere. bf16 appears only as *storage* β+every use is `__bfloat162float` / `__float2bfloat16`, and the arithmetic is+fp32 β and fp8 weights are decoded through the 256-entry constant lookup+table the engine already carries, not fp8 tensor-core MMA. The generic+build's two activation-quantise kernels (`fp8_sim64`, `fp8_sim64_dynamic`)+do call `__nv_cvt_float_to_fp8`, but `cuda_fp8.h` resolves that to native+`cvt` PTX only at `__CUDA_ARCH__ >= 890` and to its own software path+below, so it builds and runs on older cards. All of it compiles unchanged+for sm_61 and sm_75, so `CUDA_ARCH=portable-pre-ampere NO_TC=1` extends the+generic tier to Pascal and Turing.++`NO_TC=1` is required because the opt-in `DSV4_CUDA_TC` path calls cuBLASLt+block-scaling APIs (`CUBLASLT_MATMUL_DESC_A_SCALE_MODE` and friends) that need+CUDA >= 12.8 β a *toolkit* dependency, not an architecture one.++This preset stops at `sm_90` plus `compute_90` PTX rather than mirroring+`portable` up to `sm_120`. `compute_120` requires CUDA >= 12.8 β the same+toolkit floor `NO_TC=1` exists to stay below β so including it would make the+preset unbuildable by exactly the older toolkits its users tend to have. Newer+cards are still covered, through driver JIT of the `compute_90` PTX.++DeepGEMM is not available below sm_90 and cannot be made available: it calls+`cuTensorMapEncodeTiled` (TMA), a hardware unit introduced in Hopper, so there+is no software path to fall back to.+| build | GPUs | prefill | decode ||---|---|---|---|| CPU only (no DLL / `DSV4_CUDA=0` / Linux default) | β | CPU reference | CPU reference || generic DLL / `CUDA=1` | any sm_80+ | GPU attention block, indexer, generic batched MoE on the VRAM bank | GPU attention, indexer, expert mirrors |+| generic, pre-Ampere / `CUDA=1 CUDA_ARCH=portable-pre-ampere NO_TC=1` | sm_61 (Pascal), sm_75 (Turing) | as generic | as generic || DeepGEMM DLL / `CUDA=1 DEEPGEMM=1` | compute 12.x | as generic + tensor-core dense/MoE GEMMs | same as generic || multi-GPU | β | single device today (`DSV4_CUDA_DEVICE` selects); expert-parallel design drafted, not implemented | β |