Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
dgx_concurrent: --parallel N and --distinct (a prompt and a hidden code for every client, the answers checked: no error, readable, its own code and nobody else's)
Niko1221773fc4e2280f
diff --git a/docs/DGX_SPARK_LINUX.md b/docs/DGX_SPARK_LINUX.mdindex b3211d47b..207654318 100644--- a/docs/DGX_SPARK_LINUX.md+++ b/docs/DGX_SPARK_LINUX.md@@ -1,90 +1,113 @@-# Strata on the NVIDIA DGX Spark (Linux, arm64)+# Strata on the NVIDIA DGX Spark (Linux, ARM64)-One desktop box with a GB10 GPU, 20 Arm CPU cores and 128 GB of memory shared by the GPU and the CPU. This page is the-arm64 Linux build of Strata from source: what works, how to build it, how to install a model and how the speed was-measured. It is experimental: one Spark, one driver, one toolkit.+The DGX Spark is a small desktop box with a GB10 chip: a Blackwell GPU (compute capability 12.1, 48 SMs) and 20 Arm CPU cores that share+128 GB of memory. Strata 0.2.7.1 runs Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on it. This page says what works, how to install it, what+the Spark does with its memory, and how the speeds were measured. It was tested on one Spark (Ubuntu 24.04, driver 580.173, CUDA 13.0).+Nothing here is known to work on another ARM machine.## What works, and what does not-- **Works:** the CUDA engine (`strata`), Qwen3.8-Flash-Next IQ3_S and the NVFP4 GPTQ pack, 64K context, pictures off,- the DeepSeek-V4.1 engine built from source (`strata-dsv41`).-- **Not checked here:** the picture reader (`strata-vision` builds, not exercised in these runs), 128K context,- DeepSeek serving (the engine builds; no model was downloaded in this session), the app, the Terminal and the- ready-made engine feed, which are x86_64 only.-- **CPU experts:** Strata's own CPU expert kernels are x86 AVX/AVX-512; the arm64 build leaves them out and a native- pack's CPU experts run on ggml-cpu's NEON rows. A canonical Q2_0 pack (no `native_experts.txt`) still needs AVX-512- and is refused at start.+- **Works:** Qwen3.8-Flash-Next IQ3_S and the NVFP4 GPTQ pack (64K context, every expert in the GPU's cache), DeepSeek-V4.1-Flash 2.00 bpw+ (a short chat), the Terminal (`strata launch`, `strata update`, `strata serve`) and the engine packages of the signed feed.+- **Not built for ARM:** the desktop app (no AppImage; use the Terminal), the HIP, SYCL and CPU-only engines, and the Q2_0 and Gyro packs,+ whose CPU kernels are x86-only (the engine refuses them at start). Picture reading is compiled in and was not tried on the Spark.+- **Not tested:** contexts above 128K, batches of more than four chats, a long soak. `strata doctor --live` lists the GB10 with no memory+ figure, because `nvidia-smi` reports none for it.-## Build the engine--Needs an arm64 Linux with the CUDA 13.0 toolkit (`/usr/local/cuda`; DGX OS has it) and Python 3.10+ with venv.+## Install with the Terminal```sh-python3 -m venv .venv-build-.venv-build/bin/pip install -r requirements.txt -r requirements-build.txt-PATH="$PWD/.venv-build/bin:/usr/local/cuda/bin:$PATH" PYTHON="$PWD/.venv-build/bin/python" \- JOBS=20 sh spark/build_engine.sh+curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh+strata launch```-About 3 minutes on the Spark with 20 jobs (the first run also fetches llama.cpp at the pinned commit). It leaves-`out-spark/engine/` with `strata`, `libstrata_kq.so`, the CUDA libraries it needs and a `BUILD.json` that says-`linux-aarch64`, arch `121`.+`install.sh` unpacks the Terminal into `~/.local/share/strata` (no administrator rights) and checks the release's signed feed. `strata launch`+fetches the `linux-aarch64` engine package, shows what each model needs, downloads the one you choose and starts it. Pick Qwen3.8-Flash-Next+IQ3_S (84 GB download) or the NVFP4 pack, or DeepSeek-V4.1-Flash 2.00 bpw (257 GB). The engine package needs glibc 2.39 or newer+(Ubuntu 24.04, DGX OS 7) and NVIDIA driver 580.65 or newer.-`BUILD_VISION=0` skips the picture reader; `BUILD_DSV41=0` skips the DeepSeek engine; `ARCHS=121` is the default and-is what a Spark needs.--Put it where setup looks for an engine:+## Install from a source checkout```sh-cp -r out-spark/engine ./engine+./setup.sh --setup --family qwen --model IQ3_S --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data+./setup.sh --setup --family nvfp4 --model NVFP4 --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data```-## Install a model+`--data-dir` must be a folder you can write (the default, next to the checkout, may be under `/home`, which a normal user cannot create).+Setup finds the GB10, calls it unified memory, takes the engine from the feed (or compiles it with `spark/build_engine.sh` when the release+has none) and writes the run config with the settings measured on this part:++| Setting | Why |+|---|---|+| `--mmap-experts` | the experts are read from the model files in place, so the GPU's cache holds all 24,576 of them instead of a second copy in RAM |+| `--prefill auto:32768` | a 32,768-token prompt chunk; the automatic 8,192 measured 2 to 6% slower here |+| `--mtp-q4 all` | the draft layer as 4-bit copies: +3.5% decode, the answers do not change because the target checks every draft |+| `--mtp-hnorm stream` | one norm per hyper-connection stream in the draft layer: draft acceptance 0.60 to 0.63, +4.3% decode |++Setup prints `--draft-vocab en` as a tip and does not set it: it measured +3.1% decode on English prompts, and it would cut the Chinese,+Japanese and Korean drafts.++Start a model with its run script (`./run-iq3_s.sh`) or `./setup.sh`.++## Build the engine yourself++You need the CUDA 13.0 toolkit (`/usr/local/cuda`; DGX OS has it), cmake, ninja and Python 3.10 or newer.```sh-./setup.sh --setup --family qwen --model IQ3_S --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data-./setup.sh --setup --family nvfp4 --model NVFP4 --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data+python3 -m venv .venv-build+.venv-build/bin/pip install -r requirements.txt -r requirements-build.txt+PATH="$PWD/.venv-build/bin:/usr/local/cuda/bin:$PATH" PYTHON="$PWD/.venv-build/bin/python" JOBS=16 sh spark/build_engine.sh+cp -r out-spark/engine ./engine```-`--data-dir` must be a folder you can write (on a DGX Spark the default `../Strata-data` next to the checkout is under-`/home`, which `dev` cannot create; `~/Strata-data` works). Setup finds the GB10, reports it as unified memory, and-passes `--mmap-experts`: the experts are read from the model files in place, so the GPU's cache holds all 24,576 of them-instead of a second copy in RAM.+About 3 minutes for the Qwen engine and 6 with the DeepSeek engine and the picture reader (the first run also fetches llama.cpp at the+pinned commit). `out-spark/engine/` holds `strata`, `strata-dsv41`, `strata-vision`, `libstrata_kq.so`, the CUDA libraries and a `BUILD.json`.+`BUILD_VISION=0` and `BUILD_DSV41=0` leave a part out. `packaging/build_cuda_linux_aarch64.sh` runs this build and makes the engine packages+for the feed ([packaging/README.md](../packaging/README.md)). A model on a Q2_0 pack cannot run: its CPU kernels need AVX-512.-Start a model with its run script (`./run-iq3_s.sh`) or `./setup.sh`. `./bench.sh` runs the quick speed check.+## What the Spark does with its memory-## What the Spark does with the memory+The GPU's memory and the CPU's are the same 121.6 GB, so a model's experts are held once. With `--mmap-experts` there is one copy: the model+file's pages, which the GPU cache uses.| | IQ3_S | NVFP4 GPTQ ||---|---|---|| Expert bytes | 50.3 GB | 68.0 GB || Expert cache | 24,576 of 24,576 slots, 46.84 GiB | 24,576 of 24,576 slots, 63.29 GiB |-| n-gram table | 28.8 GB, locked in RAM (`--ple-io ram`) | 51.2 GB, read from the SSD (`direct`) |+| n-gram table | 28.8 GB, locked in RAM (`--ple-io ram`) | 51.2 GB, read from the SSD || Context | 64K, 8-bit KV, 32,768 cells streamed | same |-| Free VRAM at start | ~1.0-1.2 GiB | ~1.2 GiB |+| Free memory at start | about 1 GiB | about 1.2 GiB |-The NVFP4 table does not fit in RAM beside the page cache the 68 GB expert file needs: locking it measured 137 tok/s-on a 4K prompt and 26 tok/s decode against 1,252 and 39 without. Leave it on the SSD.+The NVFP4 table does not fit in RAM beside the page cache the 68 GB expert file needs. Locking it measured 137 tok/s on a 4K prompt and 26 tok/s+decode, against 1,252 and 39 without. Leave it on the SSD (setup does).-## Speeds (medians, one Spark, 26 October 2026)+DeepSeek-V4.1-Flash 2.00 bpw places 12,163 of its 15,744 experts in the GPU's share and runs the rest on the CPU lane from the model's files.+The engine sizes that from the memory Linux can still give back (`MemAvailable` less 6 GiB, `DS4_UMA_HEADROOM_GIB`), because the model files' page+cache is most of the memory `cudaMemGetInfo` calls used.-Measured with `tools/dgx_bench.py`: fresh 4K / 16K / 32K prompts at temperature 0, 256-token replies, the engine's own-`timings`, three paired rounds against steddyman's 0.1.49.2 engine built on the same machine.+## Speeds-| Model | prompt tok/s (4K / 16K / 32K) | decode tok/s |+[[SPEEDS]]++## The four speed defaults of this part++On a GB10 the engine switches on four settings that were measured here, and prints them at every start. `STRATA_SM121_PERF=0` (or+`STRATA_DEFAULTS=026`, which gives back everything 0.2.7.1 changed) turns the whole set off; a switch you set yourself is kept.++| Switch | What it does | Measured on the Spark ||---|---|---|-| Qwen3.8-Flash-Next IQ3_S, this build | 1,416 / 1,597 / 1,581 | 45.5 / 45.0 / 44.0 |-| Qwen3.8-Flash-Next IQ3_S, the fork | 1,237 / 1,331 / 1,402 | 44.4 / 43.4 / 43.1 |-| Qwen3.8-Flash-Next NVFP4, this build | see HANDOVER.md | see HANDOVER.md |-| Qwen3.8-Flash-Next NVFP4, the fork | see HANDOVER.md | see HANDOVER.md |+| `STRATA_PF_FUSED=1` | the fused IQ prompt kernels | IQ3_S prompt +10 to 27% |+| `STRATA_PREFILL_NVFP4=w4a4` | NVFP4 prompt matrix product with FP4 activations | NVFP4 prompt +16% |+| `STRATA_HC_UPMIX=1` | the hyper-connection read's up projection and mix in one kernel | decode +5.4% |+| `STRATA_SPEC_COST=1` | the draft window follows what a draft costs | decode +4.7% |++[[QUALITY]]-On IQ3_S that is +16% prompts over the fork and +3% decode. The prompt gain is `--prefill auto:32768` plus the fused-IQ prompt path (`STRATA_PF_FUSED=1`, opt-in on this card because the fused kernels round differently from MMQ).-Decode is GPU-work-bound on the 48-SM GB10 (109 kernel launches per token, ~99% GPU busy); the remaining decode work is-kernel work, not configuration - see the handover's backlog.+With `"parallel": 4` (the default) the set adds up: 32 clients at 2,000-token prompts read 41.3 tok/s with all four on and 35.7 with all off. At+`"parallel": 8` the two decode switches cost about 3.5% each; set `STRATA_HC_UPMIX=0 STRATA_SPEC_COST=0` there.-## Rules for this machine+## What this does not cover-- Don't shut it down or restart it without agreement.-- Never publish the SSH key or put it in a repository.-- Ask the machine's owner for extra rights or installs.+Decode on the 48-SM GB10 is limited by the GPU's work, about 930 kernel launches per token. No configuration switch measured here moved it more+than a few percent. A DFlash2 drafter is slower here than the MTP drafter (37 against 45 tok/s) and is not used. Teacher-forced quality over+long text and task suites was run only for the prompt path (above).diff --git a/tools/dgx_concurrent.py b/tools/dgx_concurrent.pyindex 08e2a1e61..2281ea116 100644--- a/tools/dgx_concurrent.py+++ b/tools/dgx_concurrent.py@@ -3,6 +3,13 @@same N against a second configuration to compare.python tools/dgx_concurrent.py CONFIG OUT.json --clients 8 [--clients 16] [--prompt 2000] [--max-new 128]+ [--parallel 4] [--distinct]++--parallel N runs the server with "parallel": N (the engine's batch slots: up to N requests decode together; the run config of setup has+none, one request at a time). --distinct gives every client its own prompt with its own hidden code and checks the answer: every request+must finish without an error, and its answer must name ITS code and no other client's (a batch that mixed two requests' state would fail+that), so the aggregate figure comes with a correctness check. Without --distinct all clients send the same prompt, as the+2026-10-26 handover's numbers were measured."""from __future__ import annotations@@ -29,8 +36,19 @@ def make_prompt(tokens: int) -> str:return "Summarise the following log in two sentences.\n\n" + " ".join(body)-def serve(cfg_path: str, port: int):+def distinct_prompt(tokens: int, idx: int) -> tuple[str, str]:+ """(prompt, code): about `tokens` tokens of a log that is different for every client, with one code in the middle."""+ code = f"{(idx * 7919 + 1357) % 9000 + 1000}-{WORDS[(idx * 5) % len(WORDS)]}"+ n = int(tokens / 1.35)+ body = [WORDS[(i * (idx % 7 + 1) + idx) % len(WORDS)] for i in range(n)]+ body.insert(n // 2, f"(the access code of this log is {code})")+ return ("Read the following log. First give the access code of this log, then describe in about 80 words what the log looks like.\n\n" + " ".join(body), code)+++def serve(cfg_path: str, port: int, parallel: int | None = None):cfg = json.loads(Path(cfg_path).read_text())+ if parallel:+ cfg["parallel"] = parallelcfg["port"] = portcfg["log"] = f"/tmp/dgx-conc-{port}.log"tmp = Path(f"/tmp/dgx-conc-{port}.json")@@ -53,6 +71,14 @@ def serve(cfg_path: str, port: int):def one(port: int, prompt: str, max_new: int, out: list, idx: int):+ t0 = time.time()+ try:+ out[idx] = _one(port, prompt, max_new)+ except Exception as e: # a request that errors or times out is a finding, not a crash of the run+ out[idx] = {"error": f"{type(e).__name__}: {e}", "latency": time.time() - t0, "n": 0, "text": "", "finish": None}+++def _one(port: int, prompt: str, max_new: int) -> dict:body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,"chat_template_kwargs": {"enable_thinking": False}}).encode()req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,@@ -60,11 +86,20 @@ def one(port: int, prompt: str, max_new: int, out: list, idx: int):t0 = time.time()with urllib.request.urlopen(req, timeout=3600) as r:res = json.loads(r.read())- out[idx] = (time.time() - t0, res.get("timings", {}).get("predicted_n", 0))+ choice = (res.get("choices") or [{}])[0]+ return {"latency": time.time() - t0, "n": res.get("timings", {}).get("predicted_n", 0),+ "text": (choice.get("message") or {}).get("content") or "", "finish": choice.get("finish_reason"), "error": None}+++def coherent(text: str) -> bool:+ """A readable answer: not empty, not one repeated token or a wall of the same few words."""+ words = text.split()+ return len(words) >= 3 and len(set(words)) / len(words) > 0.35-def run(cfg_path: str, port: int, clients: int, prompt: str, max_new: int) -> dict:- proc = serve(cfg_path, port)+def run(cfg_path: str, port: int, clients: int, prompt: str, max_new: int, parallel: int | None = None,+ distinct: int | None = None) -> dict:+ proc = serve(cfg_path, port, parallel)try:with urllib.request.urlopen(urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions",@@ -72,18 +107,27 @@ def run(cfg_path: str, port: int, clients: int, prompt: str, max_new: int) -> diheaders={"Content-Type": "application/json"}), timeout=1800) as r:r.read()out: list = [None] * clients+ prompts = [distinct_prompt(distinct, i) for i in range(clients)] if distinct else [(prompt, None)] * clientst0 = time.time()- threads = [threading.Thread(target=one, args=(port, prompt, max_new, out, i)) for i in range(clients)]+ threads = [threading.Thread(target=one, args=(port, prompts[i][0], max_new, out, i)) for i in range(clients)]for t in threads:t.start()for t in threads:t.join()wall = time.time() - t0- tokens = sum(n for _, n in out if n)- times = [d for d, _ in out if d]- return {"clients": clients, "wall_s": round(wall, 2), "tokens": tokens,- "aggregate_tok_s": round(tokens / wall, 2) if wall > 0 else None,- "median_latency_s": round(statistics.median(times), 2)}+ tokens = sum(r["n"] for r in out if r["n"])+ times = [r["latency"] for r in out if not r["error"]]+ row = {"clients": clients, "parallel": parallel, "wall_s": round(wall, 2), "tokens": tokens,+ "aggregate_tok_s": round(tokens / wall, 2) if wall > 0 else None,+ "median_latency_s": round(statistics.median(times), 2) if times else None,+ "errors": [r["error"] for r in out if r["error"]], "finish": sorted({str(r["finish"]) for r in out}),+ "all_coherent": all(coherent(r["text"]) for r in out if not r["error"])}+ if distinct:+ right = [(prompts[i][1] in r["text"]) and not any(prompts[j][1] in r["text"] for j in range(clients) if j != i)+ for i, r in enumerate(out) if not r["error"]]+ row["own_code_named"] = f"{sum(right)} of {clients}"+ row["answers"] = [r["text"][:100] for r in out]+ return rowfinally:proc.terminate()try:@@ -100,9 +144,11 @@ def main() -> int:ap.add_argument("--prompt", type=int, default=2000)ap.add_argument("--max-new", type=int, default=128)ap.add_argument("--port", type=int, default=8091)+ ap.add_argument("--parallel", type=int, default=None, help='run the server with "parallel": N (batch slots)')+ ap.add_argument("--distinct", action="store_true", help="a prompt of its own with a hidden code for every client, and check the answers")a = ap.parse_args()prompt = make_prompt(a.prompt)- rows = [run(a.config, a.port, c, prompt, a.max_new) for c in a.clients]+ rows = [run(a.config, a.port, c, prompt, a.max_new, a.parallel, a.prompt if a.distinct else None) for c in a.clients]Path(a.out).write_text(json.dumps(rows, indent=1))for r in rows:print(json.dumps(r))