Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
DeepSeek engine packages: add sm_89 (RTX 40) to the CUDA builds (#3935)
Niko12214a759f82d485
diff --git a/CMakeLists.txt b/CMakeLists.txtindex 9c6f28790..1ff02992d 100644--- a/CMakeLists.txt+++ b/CMakeLists.txt@@ -179,7 +179,13 @@ elseif(STRATA_ENABLE_CUDA)# a build with no option was an sm_75 build. -DCMAKE_CUDA_ARCHITECTURES and the CUDAARCHS environment variable win.set(_strata_cuda_arch_defaulted OFF)if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES AND NOT DEFINED ENV{CUDAARCHS})- set(CMAKE_CUDA_ARCHITECTURES 120 CACHE STRING "CUDA architectures")+ if(STRATA_BUILD_DSV41)+ # #3935: the DeepSeek engine's own release shape (sm_86, sm_89, sm_120 + PTX 120, as packaging/build_dsv41_cuda.sh), so a hand+ # build of strata-dsv41 with no architecture runs on an RTX 30, 40 or 50 card; the Qwen engine's default below is unchanged+ set(CMAKE_CUDA_ARCHITECTURES "86-real;89-real;120" CACHE STRING "CUDA architectures")+ else()+ set(CMAKE_CUDA_ARCHITECTURES 120 CACHE STRING "CUDA architectures")+ endif()set(_strata_cuda_arch_defaulted ON)endif()enable_language(CUDA)@@ -189,7 +195,7 @@ elseif(STRATA_ENABLE_CUDA)option(STRATA_EXPERIMENTAL_SM60 "community build for GPUs below compute capability 7.5: Pascal sm_60, Volta sm_70 (#236)" OFF)if(_strata_cuda_arch_defaulted)# a recommendation, not a refusal: the release engine is built for 75;86;89;120, a hand build names its card- message(WARNING "Strata: CMAKE_CUDA_ARCHITECTURES was not set, so this build is for sm_120 (RTX 50) only. For another "+ message(WARNING "Strata: CMAKE_CUDA_ARCHITECTURES was not set, so this build is for ${CMAKE_CUDA_ARCHITECTURES} only (sm_120 = RTX 50). For another ""card pass it: -DCMAKE_CUDA_ARCHITECTURES=native (the GPUs in this PC), 75 (RTX 20), 86 (RTX 30), ""89 (RTX 40), or a list like \"86;120\".")endif()diff --git a/docs/DEEPSEEK41.md b/docs/DEEPSEEK41.mdindex 86a98e63f..418a57e4d 100644--- a/docs/DEEPSEEK41.md+++ b/docs/DEEPSEEK41.md@@ -55,6 +55,12 @@ Measured: an RTX 5070 (12 GB) with 64 GB of RAM and a 3.6 GB/s NVMe drive answertokens/s (4K) to 231 tokens/s (8K, layer-major path); an RTX 3060 (12 GB) with 46 GB of RAM about 8 to 10 tokens/s (code 9.9, agent work 7.8; 149 tokens/s at 8K; DSpark stays off there, `--dspark on` adds 2-13%); four R9700 about 48tokens/s on 2.00 and 80 on 2.25 (code, with the shipped hot-expert profile). Under about 40 GB of RAM it is untested and slow; setup says so and still installs.+**RTX 40 series (sm_89).** The engine packages carry machine code for sm_86, sm_89 and sm_120 since 0.2.7 (before, an RTX 40 card ran the sm_86 code through+CUDA's same-generation rule). We have no RTX 40 card: the sm_89 build was compiled here but not run. A user's own build for sm_89 ran the 2.00 set on an+RTX 4090 48 GB with 128 GB of RAM (#3935, their numbers): decode 40.2 tokens/s (median of 5 answers of 64 tokens, `--ssd-cache 108 --dspark on --dspark-conf 0.65`),+prompts 141 tokens/s at 1.2K and 615 at 4.6K. They saw 16 to 22 tokens/s with `--dspark auto`, which decided "off" on their computer. This is one report on one+computer; other cards and RAM sizes can differ, and `--dspark auto` stays the default.+Setup downloads with resume and SHA-256, packs the model with the plain recipe (`pack_gguf.py --model SET --shards ... --engramengram-IQ4_NL.gguf --engram-meta <UD shard 1> --dspark-gguf dspark-...-Q8_0.gguf`; do not add `--skip-tensors` or `--dense-override`:that packs without an error and gives garbage text), copies the shipped hot-expert profile (`data/dsv41/profile-2.00.bin` or `-2.25.bin`)@@ -144,6 +150,28 @@ Strata checks the package before it uses it: it must be the DeepSeek engine (notStrata trusts, and every file must match the hash in the manifest. A damaged or foreign file is refused with a plain line and nothing is installed.The unpacked folder works as well as the archive. The Strata (Qwen) engine is not touched either way.+### Building the engine for your card++The packages hold machine code for these NVIDIA cards (CUDA 13, Windows and Linux): sm_86 (RTX 30), sm_89 (RTX 40) and sm_120 (RTX 50), plus PTX for+sm_120. If your card is none of these, or you want your own build, compile `strata-dsv41` and put it where Strata looks for it. You need the CUDA toolkit+(the packages are built with 13.0), a C++ compiler, CMake, Ninja and the llama.cpp checkout Strata pins (`third_party/llama.cpp`).++```+cmake -S . -B build-dsv41 -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_DSV41=ON -DSTRATA_NATIVE_EXPERTS=ON "-DCMAKE_CUDA_ARCHITECTURES=89-real" -DSTRATA_GGML_DIR=third_party/llama.cpp+cmake --build build-dsv41 --target strata-dsv41 -j 8+```++Use your card's number in `CMAKE_CUDA_ARCHITECTURES`: 86 for RTX 30, 89 for RTX 40, 120 for RTX 50, or `native` for the cards in this computer. Without the+option a DeepSeek build is made for 86, 89 and 120. On Windows, `packaginguild_dsv41_windows.bat BUILD_DIR` does the same (`DSV41_ARCH` names the cards);+on Linux, `packaging/build_dsv41_cuda.sh` (`ARCH=...`). Then copy the file into the `engine` folder of your Strata install (next to `strata`); setup finds it there and does not download an engine:++```+python setup.py --family deepseek41 --model IST-DAS-2.00bpw+```++`strata-dsv41 --version` prints the commit it was built from. A build you made yourself is not signed by Strata, which is fine for your own computer.+Please tell us how it runs on your card (the speed, and the card and RAM): we only list a card as tested after someone has run it.+## Compressed experts (trellis), for testing`strata-dsv41` reads the routed experts in the trellis types as well as the source MXFP4: TQK6 and TQK7 (agentionai's files), and thediff --git a/docs/MODELS.md b/docs/MODELS.mdindex 4eba8f4cb..8f3fbedf8 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -78,6 +78,8 @@ Tested on an RTX 3060 (Linux), an RTX 5070 (Windows), four R9700 cards (Linux) aand Strix Halo on Linux do not run it yet; on Strix Halo Windows the graphics memory is the RAM, so most of the experts areread from the SSD and an NVMe drive matters.+RTX 40 cards are covered by the engine packages since 0.2.7 (sm_89 code); we have not run one ourselves, see [DEEPSEEK41.md](DEEPSEEK41.md#the-engine-package) for a user's measurement.+**What you need:** a graphics card with 12 GB or more, 46 GB of RAM at least (64 GB is the tested comfortable size),an NVMe SSD, and the disk space above. Setup warns when disk or RAM is short and does not stop you.diff --git a/packaging/build_dsv41_cuda.sh b/packaging/build_dsv41_cuda.shindex 4ebfd5584..9a82b28df 100644--- a/packaging/build_dsv41_cuda.sh+++ b/packaging/build_dsv41_cuda.sh@@ -3,18 +3,19 @@# beside `strata` (setup.py dsv41_exe() looks for strata-dsv41 in the engine folder). Same toolchain and RUNPATH rule as the# engine build: the result finds cudart/cublas beside itself ($ORIGIN), never the toolkit it was built with.#-# SRC=<checkout> B=<build dir> CUDA=/usr/local/cuda-13.0 GG=<llama.cpp checkout> [ARCH="86-real;120"] [JOBS=6] packaging/build_dsv41_cuda.sh+# SRC=<checkout> B=<build dir> CUDA=/usr/local/cuda-13.0 GG=<llama.cpp checkout> [ARCH="86-real;89-real;120"] [JOBS=6] packaging/build_dsv41_cuda.sh#-# ARCH defaults to the 0.2.4 release shape: SASS for sm_86 (RTX 3090 / 3060) and sm_120 (RTX Blackwell) + PTX sm_120 (the Windows twin-# built and ran on an RTX 5070; the Linux sm_86 build is the one tested on the RTX 3060 box). Needs CUDA 13 (the sm_120 code); for a-# CUDA 12.9 package pass ARCH="86-real;89-real;89-virtual" or leave strata-dsv41 out (the Pascal cuda12 variant has no DeepSeek path).+# ARCH defaults to the 0.2.7 release shape: SASS for sm_86 (RTX 3090 / 3060), sm_89 (RTX 40, #3935) and sm_120 (RTX Blackwell) + PTX sm_120, i.e.+# the package lists sm_86, sm_89, sm_120, compute_120 (the Windows twin built and ran on an RTX 5070; the Linux sm_86 build is the one tested on the+# RTX 3060 box; sm_89 is built here, not run: the one reporter's own build ran at 40.2 tok/s on an RTX 4090). Needs CUDA 13 (the sm_120 code);+# for a CUDA 12.9 package pass ARCH="86-real;89" (sm_86, sm_89, compute_89) or leave strata-dsv41 out (the Pascal cuda12 variant has no DeepSeek path).# Prints the binary path on the last line; collect its libraries with packaging/collect_libs.py --binary <it> --search $CUDA/lib64.set -euSRC=${SRC:-$(cd "$(dirname "$0")/.." && pwd)}B=${B:?B=<build dir>}CUDA=${CUDA:?CUDA=<toolkit root>}GG=${GG:?GG=<llama.cpp checkout at the pinned commit>}-ARCH=${ARCH:-86-real;120}+ARCH=${ARCH:-86-real;89-real;120}JOBS=${JOBS:-6}export PATH=$CUDA/bin:$PATH# the build id: the commit, 9 characters. A git checkout gives it to CMake itself; a tarball has no .git, so COMMIT=<hex> is required there and goes to BUILD_ID.diff --git a/packaging/build_dsv41_windows.bat b/packaging/build_dsv41_windows.batindex 8fe958e46..8faea6518 100644--- a/packaging/build_dsv41_windows.bat+++ b/packaging/build_dsv41_windows.bat@@ -3,7 +3,7 @@ rem strata-dsv41.exe (the DeepSeek-V4.1 engine) for NVIDIA on Windows, built nexrem beside strata.exe (setup.py dsv41_exe() looks for it in the engine folder). cudart is static; the only extra DLL is cublas64_13.dll,rem which package_engine.py copies from CUDA_PATH like it does for strata.exe.rem build_dsv41_windows.bat <build dir> [source dir] env: STRATA_GGML_DIR (llama.cpp), CUDA_ROOT (default CUDA 13.0), BUILD_JOBS (default 4),-rem DSV41_ARCH (default "86-real;120" = sm_86 + sm_120 SASS, PTX sm_120)+rem DSV41_ARCH (default "86-real;89-real;120" = sm_86 + sm_89 + sm_120 SASS, PTX sm_120)rem Result: <build dir>\strata-dsv41.exe (about 66 MB). Windows: only the cuda13 variant gets it (sm_120 needs CUDA 13).setlocalset "BD=%~1"@@ -11,7 +11,7 @@ set "SRCD=%~2"if "%BD%"=="" ( echo usage: build_dsv41_windows.bat ^<build dir^> [source dir] & exit /b 2 )if "%SRCD%"=="" set "SRCD=%~dp0.."if "%CUDA_ROOT%"=="" set "CUDA_ROOT=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v13.0"-if "%DSV41_ARCH%"=="" set "DSV41_ARCH=86-real;120"+if "%DSV41_ARCH%"=="" set "DSV41_ARCH=86-real;89-real;120"if "%BUILD_JOBS%"=="" set "BUILD_JOBS=4"if "%STRATA_GGML_DIR%"=="" ( echo set STRATA_GGML_DIR to the llama.cpp checkout & exit /b 2 )call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat" >nul 2>&1diff --git a/setup.py b/setup.pyindex 55f8941f2..7081a73cf 100644--- a/setup.py+++ b/setup.py@@ -8214,11 +8214,13 @@ def install_dsv41(a, family, gpu, hip, amd, cuda_tk, multi, port, ram, adopted,lib_dirs = [*lib_dirs, *dsv41_engine.lib_dirs(exe)] # its own package, and the rocm/bin an AMD package keeps the ROCm DLLs inif exe is None:if os.environ.get("STRATA_VERBOSE") == "1" or "--verbose" in sys.argv:- say(" Details for developers: build it with cmake -DSTRATA_BUILD_DSV41=ON and point STRATA_DSV41_EXE at it.")+ say(" Details for developers: build it with cmake -DSTRATA_BUILD_DSV41=ON -DCMAKE_CUDA_ARCHITECTURES=<your card: 86 RTX 30, 89 RTX 40, 120 RTX 50> "+ "and point STRATA_DSV41_EXE at it (docs/DEEPSEEK41.md, \"Building the engine for your card\").")fail("DeepSeek's engine is not available for this computer yet","Supported graphics cards: NVIDIA RTX 30, 40 and 50 series, the Radeon AI PRO R9700, and a Ryzen AI Max (Strix Halo) on ""Windows with the AMD DeepSeek engine package. " + dsv41_manual_steps(eng)- + " Or update Strata later (a newer version may add your card), or pick another model. Nothing has been downloaded.")+ + " Or update Strata later (a newer version may add your card), pick another model, or build the engine for your card yourself "+ "(docs/DEEPSEEK41.md, \"Building the engine for your card\"). Nothing has been downloaded.")ok(f"engine: {exe}")step(5, f"downloading {fam['title']} {model}")diff --git a/strata_control/engine_component.py b/strata_control/engine_component.pyindex c7f60e0b0..4fbb7e977 100644--- a/strata_control/engine_component.py+++ b/strata_control/engine_component.py@@ -808,6 +808,14 @@ def arch_covers(archs, present: list[str]) -> bool:return False+def arch_native(archs, present: list[str]) -> bool:+ """Whether a build with `archs` holds machine code made for one of the `present` cards itself (``sm_89`` on an ``sm_89`` card),+ not only code that runs there through the same-major rule of `arch_covers` (``sm_86`` on an RTX 40) or as PTX. Only a tie-breaker+ between packages that fit equally: a package with the card's own image is taken first (#3935)."""+ cards = {item[3:] for item in present if item.startswith("sm_") and item[3:].isdigit()}+ return any(arch.startswith("sm_") and arch[3:] in cards for arch in archs)++# ----------------------------------------------------------------------------------------------- resolvedef scan(root: Path | None = None) -> tuple[list[EngineComponent], list[dict]]:@@ -974,7 +982,7 @@ def resolve(root: Path | None = None, *, backend: str | None = None, target: strreturn item# newest version first; with equal versions the build for the newer driver (the CUDA 13 build beside the CUDA 12 one)return max(usable, key=lambda item: (_version_key(item.version), _version_key(item.min_driver.get("nvidia", "0")),- hip_preference(item.hip_archs)))+ hip_preference(item.hip_archs), arch_native(item.cuda_archs, archs_here)))#: A run config can name its engine as "component:strata-engine" instead of a path: the server then asks thediff --git a/strata_control/feed.py b/strata_control/feed.pyindex ddbdaafe2..eb0e78352 100644--- a/strata_control/feed.py+++ b/strata_control/feed.py@@ -412,7 +412,8 @@ def select_engine(packages, hardware: dict) -> tuple[EnginePackage | None, list[if not fitting:return None, reasonsbest = max(fitting, key=lambda p: (version_key(p.version), ec._version_key(p.min_driver.get("nvidia", "0")),- ec.hip_preference(p.archs) if p.backend == "hip" else 1))+ ec.hip_preference(p.archs) if p.backend == "hip" else 1,+ ec.arch_native(p.archs, hardware.get("cuda_archs") or []) if p.backend == "cuda" else False))kept = str(hardware.get("installed_variant") or "")if best.backend == "cuda" and kept and best.variant != kept:# An update never switches the CUDA build (#3835): a model on the CUDA 12 engine (its card is one CUDA 13 has no code for) must notdiff --git a/tests/terminal/test_engine_manifest.py b/tests/terminal/test_engine_manifest.pyindex 3cb5b00d4..46375ffd7 100644--- a/tests/terminal/test_engine_manifest.py+++ b/tests/terminal/test_engine_manifest.py@@ -218,6 +218,54 @@ class Arch(unittest.TestCase):self.assertFalse(ec.arch_covers(["sm_89"], ["sm_100"]), "sm_100 is major 10, not major 1 + minor 00")self.assertTrue(ec.arch_covers(["sm_86", "sm_120"], ["sm_75", "sm_89"]), "one covered card is enough")+ def test_a_native_image_is_the_card_itself_only(self):+ """#3935: arch_native tells a package with the card's own machine code from one that only runs there."""+ self.assertTrue(ec.arch_native(["sm_86", "sm_89", "sm_120", "compute_120"], ["sm_89"]))+ self.assertFalse(ec.arch_native(["sm_86", "sm_120", "compute_120"], ["sm_89"]), "sm_86 code runs on an RTX 40 but is not its image")+ self.assertFalse(ec.arch_native(["sm_86", "compute_89"], ["sm_89"]), "PTX is not machine code")+ self.assertFalse(ec.arch_native(["sm_89"], []))+ self.assertTrue(ec.arch_native(["sm_75", "sm_89"], ["sm_75", "sm_120"]))++ def test_the_deepseek_package_lists_89_and_an_rtx_40_card_is_accepted(self):+ """#3935: the build scripts and the CMake default name sm_89, the manifest list a build of them gets covers an sm_89 card+ with its own image, and the engine picker takes it for an RTX 4090 with no refusal and prefers it to a package without it."""+ import re+ from strata_control import feed++ def package_archs(spec):+ """CMAKE_CUDA_ARCHITECTURES to the package's archs list: N-real = sm_N, N = sm_N + compute_N."""+ out = []+ for item in spec.split(";"):+ number, _, kind = item.partition("-")+ out.append(f"sm_{number}")+ if not kind:+ out.append(f"compute_{number}")+ return out++ sh = (ROOT / "packaging" / "build_dsv41_cuda.sh").read_text(encoding="utf-8")+ bat = (ROOT / "packaging" / "build_dsv41_windows.bat").read_text(encoding="utf-8")+ cmake = (ROOT / "CMakeLists.txt").read_text(encoding="utf-8")+ linux = re.search(r"^ARCH=\$\{ARCH:-([^}]+)\}", sh, re.M).group(1)+ windows = re.search(r'set "DSV41_ARCH=([^"]+)"', bat).group(1)+ self.assertEqual(linux, windows, "Windows and Linux ship the same DeepSeek architectures")+ self.assertEqual(package_archs(linux), ["sm_86", "sm_89", "sm_120", "compute_120"])+ self.assertIn('set(CMAKE_CUDA_ARCHITECTURES "86-real;89-real;120" CACHE', cmake)++ def package(archs):+ return feed.EnginePackage(version="0.1.50", platform="linux-x86_64", backend="cuda", variant="cuda13", archs=tuple(archs),+ min_driver={}, min_runtime=None, build_id="b", url="u", format="tar.gz", size=1, sha256="a" * 64,+ manifest_sha256="b" * 64, installed_bytes=1)++ hw = {"platform": "linux-x86_64", "backend": "cuda", "cuda_archs": ["sm_89"], "nvidia_driver": None}+ old, new = package(["sm_86", "sm_120", "compute_120"]), package(package_archs(linux))+ self.assertFalse(feed.engine_fit_reasons(new, hw), "an RTX 4090 is accepted")+ self.assertFalse(feed.engine_fit_reasons(old, hw), "the old package still fits through sm_86 (no refusal)")+ for order in ((old, new), (new, old)):+ best, _ = feed.select_engine(list(order), hw)+ self.assertIs(best, new, "the package with sm_89 machine code wins an otherwise equal choice")+ best, _ = feed.select_engine([old, new], {**hw, "cuda_archs": ["sm_86"]})+ self.assertIsNotNone(best)+def test_the_packages_of_each_engine(self):"""The Qwen engine (sm_86 + PTX 89) and the DeepSeek engine (sm_86 + sm_120 + PTX 120) on the RTX 20 to 50 families."""from strata_control import feed