Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
The DGX Spark benchmark family and guide (from the owner's handover, 2026-10-26)
Niko1221694e71170828
diff --git a/docs/DGX_SPARK_LINUX.md b/docs/DGX_SPARK_LINUX.mdnew file mode 100644index 000000000..b3211d47b--- /dev/null+++ b/docs/DGX_SPARK_LINUX.md@@ -0,0 +1,90 @@+# Strata on the NVIDIA DGX Spark (Linux, arm64)++One desktop box with a GB10 GPU, 20 Arm CPU cores and 128 GB of memory shared by the GPU and the CPU. This page is the+arm64 Linux build of Strata from source: what works, how to build it, how to install a model and how the speed was+measured. It is experimental: one Spark, one driver, one toolkit.++## What works, and what does not++- **Works:** the CUDA engine (`strata`), Qwen3.8-Flash-Next IQ3_S and the NVFP4 GPTQ pack, 64K context, pictures off,+ the DeepSeek-V4.1 engine built from source (`strata-dsv41`).+- **Not checked here:** the picture reader (`strata-vision` builds, not exercised in these runs), 128K context,+ DeepSeek serving (the engine builds; no model was downloaded in this session), the app, the Terminal and the+ ready-made engine feed, which are x86_64 only.+- **CPU experts:** Strata's own CPU expert kernels are x86 AVX/AVX-512; the arm64 build leaves them out and a native+ pack's CPU experts run on ggml-cpu's NEON rows. A canonical Q2_0 pack (no `native_experts.txt`) still needs AVX-512+ and is refused at start.++## Build the engine++Needs an arm64 Linux with the CUDA 13.0 toolkit (`/usr/local/cuda`; DGX OS has it) and Python 3.10+ with venv.++```sh+python3 -m venv .venv-build+.venv-build/bin/pip install -r requirements.txt -r requirements-build.txt+PATH="$PWD/.venv-build/bin:/usr/local/cuda/bin:$PATH" PYTHON="$PWD/.venv-build/bin/python" \+ JOBS=20 sh spark/build_engine.sh+```++About 3 minutes on the Spark with 20 jobs (the first run also fetches llama.cpp at the pinned commit). It leaves+`out-spark/engine/` with `strata`, `libstrata_kq.so`, the CUDA libraries it needs and a `BUILD.json` that says+`linux-aarch64`, arch `121`.++`BUILD_VISION=0` skips the picture reader; `BUILD_DSV41=0` skips the DeepSeek engine; `ARCHS=121` is the default and+is what a Spark needs.++Put it where setup looks for an engine:++```sh+cp -r out-spark/engine ./engine+```++## Install a model++```sh+./setup.sh --setup --family qwen --model IQ3_S --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data+./setup.sh --setup --family nvfp4 --model NVFP4 --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data+```++`--data-dir` must be a folder you can write (on a DGX Spark the default `../Strata-data` next to the checkout is under+`/home`, which `dev` cannot create; `~/Strata-data` works). Setup finds the GB10, reports it as unified memory, and+passes `--mmap-experts`: the experts are read from the model files in place, so the GPU's cache holds all 24,576 of them+instead of a second copy in RAM.++Start a model with its run script (`./run-iq3_s.sh`) or `./setup.sh`. `./bench.sh` runs the quick speed check.++## What the Spark does with the memory++| | IQ3_S | NVFP4 GPTQ |+|---|---|---|+| Expert bytes | 50.3 GB | 68.0 GB |+| Expert cache | 24,576 of 24,576 slots, 46.84 GiB | 24,576 of 24,576 slots, 63.29 GiB |+| n-gram table | 28.8 GB, locked in RAM (`--ple-io ram`) | 51.2 GB, read from the SSD (`direct`) |+| Context | 64K, 8-bit KV, 32,768 cells streamed | same |+| Free VRAM at start | ~1.0-1.2 GiB | ~1.2 GiB |++The NVFP4 table does not fit in RAM beside the page cache the 68 GB expert file needs: locking it measured 137 tok/s+on a 4K prompt and 26 tok/s decode against 1,252 and 39 without. Leave it on the SSD.++## Speeds (medians, one Spark, 26 October 2026)++Measured with `tools/dgx_bench.py`: fresh 4K / 16K / 32K prompts at temperature 0, 256-token replies, the engine's own+`timings`, three paired rounds against steddyman's 0.1.49.2 engine built on the same machine.++| Model | prompt tok/s (4K / 16K / 32K) | decode tok/s |+|---|---|---|+| Qwen3.8-Flash-Next IQ3_S, this build | 1,416 / 1,597 / 1,581 | 45.5 / 45.0 / 44.0 |+| Qwen3.8-Flash-Next IQ3_S, the fork | 1,237 / 1,331 / 1,402 | 44.4 / 43.4 / 43.1 |+| Qwen3.8-Flash-Next NVFP4, this build | see HANDOVER.md | see HANDOVER.md |+| Qwen3.8-Flash-Next NVFP4, the fork | see HANDOVER.md | see HANDOVER.md |++On IQ3_S that is +16% prompts over the fork and +3% decode. The prompt gain is `--prefill auto:32768` plus the fused+IQ prompt path (`STRATA_PF_FUSED=1`, opt-in on this card because the fused kernels round differently from MMQ).+Decode is GPU-work-bound on the 48-SM GB10 (109 kernel launches per token, ~99% GPU busy); the remaining decode work is+kernel work, not configuration - see the handover's backlog.++## Rules for this machine++- Don't shut it down or restart it without agreement.+- Never publish the SSH key or put it in a repository.+- Ask the machine's owner for extra rights or installs.diff --git a/tools/dgx_128k.py b/tools/dgx_128k.pynew file mode 100644index 000000000..e0bca55a7--- /dev/null+++ b/tools/dgx_128k.py@@ -0,0 +1,88 @@+#!/usr/bin/env python3+"""Does the engine hold a context past the installed 64K? Copy the shipped config, raise the context to 128K, and send a+100K-token prompt with a hidden code. Success = an answer that recalls it; failure is recorded as a limit.++ python dgx_128k.py CONFIG OUT.json [--tokens 100000] [--ctx 131072]+"""+import json+import random+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()+++def build(tokens, code, rng):+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]+ at = rng.randrange(len(body) // 3, 2 * len(body) // 3)+ body[at:at] = [f"The access code for the harbor gate is {code}."]+ return ("Read the following log carefully, then answer the question at the end.\n\n" + " ".join(body) ++ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")+++def main():+ cfg_path, out_path = sys.argv[1], sys.argv[2]+ tokens = int(sys.argv[3]) if len(sys.argv) > 3 else 100000+ ctx = int(sys.argv[4]) if len(sys.argv) > 4 else 131072+ port = 8091+ cfg = json.loads(Path(cfg_path).read_text())+ a = cfg["args"]+ if "--max-context" in a:+ a[a.index("--max-context") + 1] = str(ctx)+ if "--kv-resident" in a:+ a[a.index("--kv-resident") + 1] = "32768"+ cfg["port"] = port+ cfg["log"] = "/tmp/long-128k.log"+ tmp = Path("/tmp/long-128k.json")+ tmp.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(tmp), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ t0 = time.time()+ while time.time() - t0 < 1800:+ if proc.poll() is not None:+ raise SystemExit(f"server exited {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(3)+ print(f"ready in {time.time() - t0:.0f} s", flush=True)+ rng = random.Random(11)+ code = "271828"+ prompt = build(tokens, code, rng)+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 32, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t1 = time.time()+ with urllib.request.urlopen(req, timeout=5400) as r:+ out = json.loads(r.read())+ wall = time.time() - t1+ t = out.get("timings") or {}+ answer = out["choices"][0]["message"].get("content", "")+ row = {"tokens": tokens, "ctx": ctx, "prompt_n": t.get("prompt_n"),+ "prompt_per_second": t.get("prompt_per_second"),+ "predicted_per_second": t.get("predicted_per_second"), "wall_s": round(wall, 1),+ "answer": answer[:80], "recalled": code in answer}+ Path(out_path).write_text(json.dumps(row, indent=1))+ print(json.dumps(row))+ finally:+ proc.terminate()+ try:+ proc.wait(180)+ except subprocess.TimeoutExpired:+ proc.kill()+++if __name__ == "__main__":+ main()diff --git a/tools/dgx_bench.py b/tools/dgx_bench.pynew file mode 100644index 000000000..768121a0f--- /dev/null+++ b/tools/dgx_bench.py@@ -0,0 +1,254 @@+#!/usr/bin/env python3+"""dgx_bench - the DGX Spark benchmark family for Strata (Qwen3.8-Flash-Next first).++ python tools/dgx_bench.py --config strata-iq3_s.json --tag base+ python tools/dgx_bench.py --config strata-iq3_s.json --tag gu-swiglu --env STRATA_GU_SWIGLU=1+ python tools/dgx_bench.py --config strata-iq3_s.json --tag base --cells 4096:0:256,4096:0.7:256 --rounds 3++It starts the server on the model's own config (serve/server.py, the path setup writes), waits for it, sends the cells+through the real HTTP API, restarts the server between arms, and writes every request plus a summary.++A cell is PROMPT_TOKENS:TEMPERATURE:MAX_NEW. The default family is the one the DGX Spark guide's contract names:+512 / 4096 / 16384 / 32768 fresh prompt tokens at temperature 0 and 0.7, with a 256-token generation cap, plus one+1024-token decode cell. Every request carries a unique nonce as its first token sequence, so the conversation cache+cannot serve the prompt (verified: timings.cache_n must be 0, or the row is marked reused and excluded from the summary).++Per request it records the client's own time to first content token (TTFT), the wall time, and the server's timings+(prompt_n, prompt_ms, prompt_per_second, predicted_n, predicted_ms, predicted_per_second, draft counts) read from+GET /v1/status right after the streamed answer. Medians over rounds; nothing is retried or filtered except requests+that failed (they are kept and reported as failures).++Standard library only, like tools/strata_bench.py and tools/ab_engine.py.+"""+from __future__ import annotations++import argparse+import json+import os+import random+import statistics+import subprocess+import sys+import time+import urllib.error+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+DEFAULT_CELLS = "512:0:256,4096:0:256,4096:0.7:256,16384:0:256,16384:0.7:256,32768:0:256,32768:0.7:256,4096:0:1024"+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr "+ "basalt cinder dune elm fjord granite heather inlet jetty kiln lagoon mesa niche opal prairie quill").split()+++def build_prompt(target_tokens: int, nonce: str) -> str:+ """A prompt of about target_tokens tokens: a unique nonce line, a word list, and a task that keeps the answer going."""+ rng = random.Random(hash((target_tokens, nonce)) & 0xFFFFFFFF)+ # ~1.35 tokens per word for this vocabulary in Qwen's tokenizer+ words = max(8, int(target_tokens / 1.35))+ body = " ".join(rng.choice(WORDS) for _ in range(words))+ return (f"Request nonce {nonce}. Read the word list below carefully.\n\n{body}\n\n"+ f"Write a long, detailed analysis of the word list above: its sounds, its themes and its structure. "+ f"Keep writing for at least 400 words; do not stop early and do not ask questions.")+++def post_stream(port: int, prompt: str, max_tokens: int, temperature: float, timeout: float = 3600) -> dict:+ """One streamed chat completion; returns the client's times and the token count."""+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens,+ "temperature": temperature, "stream": True,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t0 = time.time()+ ttft = None+ n = 0+ text_len = 0+ finish = None+ with urllib.request.urlopen(req, timeout=timeout) as r:+ for raw in r:+ line = raw.decode("utf-8", "replace").strip()+ if not line.startswith("data:"):+ continue+ data = line[5:].strip()+ if data == "[DONE]":+ break+ try:+ obj = json.loads(data)+ except ValueError:+ continue+ for ch in obj.get("choices") or []:+ d = ch.get("delta") or {}+ piece = d.get("content") or d.get("reasoning_content") or ""+ if piece:+ if ttft is None:+ ttft = time.time() - t0+ text_len += len(piece)+ if ch.get("finish_reason"):+ finish = ch["finish_reason"]+ if obj.get("usage"):+ n = int(obj["usage"].get("completion_tokens") or n)+ wall = time.time() - t0+ return {"ttft_s": ttft, "wall_s": round(wall, 3), "stream_tokens": n, "chars": text_len, "finish": finish}+++def last_timings(port: int) -> dict:+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/status", timeout=10) as r:+ st = json.loads(r.read())+ return st.get("last_timings") or {}+ except Exception:+ return {}+++def wait_ready(port: int, proc: subprocess.Popen, timeout: float = 3600) -> float:+ t0 = time.time()+ while time.time() - t0 < timeout:+ if proc.poll() is not None:+ raise RuntimeError(f"server exited with {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ return time.time() - t0+ except Exception:+ pass+ time.sleep(3)+ raise TimeoutError("server not ready")+++def calibrate(port: int, target: int, max_tokens: int, temperature: float, log) -> str:+ """Build a prompt whose prompt_n is within ~2% of target (two probes at most)."""+ nonce = f"{random.getrandbits(48):012x}"+ prompt = build_prompt(target, nonce)+ for attempt in range(3):+ r = post_stream(port, prompt, 8, 0.0)+ t = last_timings(port)+ got = int(t.get("prompt_n") or 0)+ log(f" calibrate {target}: prompt_n {got}")+ if got and abs(got - target) <= max(16, target // 50):+ return prompt+ if not got:+ break+ words = len(prompt.split()) * target / got+ rng = random.Random(target + attempt)+ body = " ".join(rng.choice(WORDS) for _ in range(max(8, int(words))))+ prompt = (f"Request nonce {nonce}. Read the word list below carefully.\n\n{body}\n\n"+ f"Write a long, detailed analysis of the word list above: its sounds, its themes and its structure. "+ f"Keep writing for at least 400 words; do not stop early and do not ask questions.")+ return prompt+++def run_arm(cfg_path: Path, tag: str, cells, rounds: int, port: int, env: dict, out: Path, warm: int,+ exe: str | None = None, xargs: list[str] | None = None) -> list[dict]:+ base = json.loads(cfg_path.read_text())+ cfg = dict(base)+ if exe:+ cfg["exe"] = exe+ if xargs:+ cfg["args"] = list(cfg.get("args") or []) + list(xargs)+ cfg["port"] = port+ cfg["log"] = str(out / f"{tag}.engine.log")+ cpath = out / f"{tag}.config.json"+ cpath.write_text(json.dumps(cfg, indent=1))+ rows: list[dict] = []+ meta = {"tag": tag, "config": str(cfg_path), "cells": cells, "rounds": rounds, "env": env,+ "engine": cfg.get("exe"), "args": cfg.get("args"), "started": time.strftime("%Y-%m-%dT%H:%M:%S")}+ (out / f"{tag}.meta.json").write_text(json.dumps(meta, indent=1))+ logf = open(out / f"{tag}.jsonl", "a", encoding="utf-8")+ penv = dict(os.environ, **env)+ for rnd in range(rounds):+ srvlog = open(out / f"{tag}.server.log", "a", encoding="utf-8")+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(cpath), "--port", str(port)], cwd=str(ROOT), env=penv,+ stdout=srvlog, stderr=srvlog)+ try:+ ready = wait_ready(port, proc)+ print(f"round {rnd} {tag}: ready in {ready:.0f} s", flush=True)+ nonce = f"{random.getrandbits(48):012x}"+ post_stream(port, f"Request nonce {nonce}. Say hello in one short sentence, then list five colors.", warm, 0.0)+ # a 4K prompt first: with --mmap-experts the first request pages the experts in (a one-time cold cost of+ # tens of seconds); every measured cell then runs on the same warm state the fork's baselines had+ nonce = f"{random.getrandbits(48):012x}"+ post_stream(port, build_prompt(4096, nonce), 8, 0.0)+ prompts = {}+ for (length, temp, maxnew) in cells:+ key = (length, maxnew)+ if key not in prompts:+ prompts[key] = calibrate(port, length, maxnew, 0.0, print)+ nonce = f"{random.getrandbits(48):012x}"+ prompt = prompts[key]+ prompt = prompt.replace("Request nonce ", "Request nonce ", 1)+ # a fresh nonce: rewrite the first line so no prefix can be reused+ prompt = prompt.split("\n", 1)+ prompt = f"Request nonce {nonce}. Read the word list below carefully.\n" + prompt[1]+ r = post_stream(port, prompt, maxnew, temp)+ t = last_timings(port)+ row = {"round": rnd, "arm": tag, "len": length, "temp": temp, "max_new": maxnew, **r, **t}+ rows.append(row)+ logf.write(json.dumps(row) + "\n")+ logf.flush()+ print(f" {length:6d} T={temp} n={maxnew:4d}: prompt_n {t.get('prompt_n')} "+ f"prompt {t.get('prompt_per_second')} tok/s decode {t.get('predicted_per_second')} tok/s "+ f"n {t.get('predicted_n')} ttft {r['ttft_s'] and round(r['ttft_s'], 2)} wall {r['wall_s']}",+ flush=True)+ finally:+ proc.terminate()+ try:+ proc.wait(60)+ except subprocess.TimeoutExpired:+ proc.kill()+ time.sleep(5)+ logf.close()+ return rows+++def summarize(rows: list[dict], cells) -> str:+ lines = ["| cell | n | prompt tok/s | decode tok/s | ttft s | wall s |", "|---|---|---|---|---|---|"]+ for (length, temp, maxnew) in cells:+ sel = [r for r in rows if r["len"] == length and r["temp"] == temp and r["max_new"] == maxnew+ and not r.get("cache_n")]+ if not sel:+ lines.append(f"| {length} T={temp} n={maxnew} | 0 | - | - | - | - |")+ continue+ med = lambda k: statistics.median([r[k] for r in sel if r.get(k)]) if any(r.get(k) for r in sel) else None+ p, d, ttft, wall = med("prompt_per_second"), med("predicted_per_second"), med("ttft_s"), med("wall_s")+ fmt = lambda v, f="{:.1f}": "-" if v is None else f.format(v)+ lines.append(f"| {length} T={temp} n={maxnew} | {len(sel)} | {fmt(p)} | {fmt(d)} | {fmt(ttft, '{:.2f}')} | "+ f"{fmt(wall, '{:.1f}')} |")+ return "\n".join(lines)+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("--config", required=True)+ ap.add_argument("--tag", default="base")+ ap.add_argument("--cells", default=DEFAULT_CELLS)+ ap.add_argument("--rounds", type=int, default=3)+ ap.add_argument("--port", type=int, default=8091)+ ap.add_argument("--env", action="append", default=[], help="NAME=VALUE for the server process (repeatable)")+ ap.add_argument("--exe", default="", help="engine binary to run instead of the config's")+ ap.add_argument("--xargs", default="", help="extra engine arguments, ';'-separated")+ ap.add_argument("--warm", type=int, default=64)+ ap.add_argument("--out", default="")+ a = ap.parse_args()+ cells = []+ for s in a.cells.split(","):+ l, t, n = s.split(":")+ cells.append((int(l), float(t), int(n)))+ out = Path(a.out) if a.out else ROOT / "bench" / "results" / time.strftime("dgx-%Y-%m-%d-%H%M")+ out.mkdir(parents=True, exist_ok=True)+ env = {}+ for e in a.env:+ k, v = e.split("=", 1)+ env[k] = v+ print(f"out {out}; cells {cells}; rounds {a.rounds}", flush=True)+ rows = run_arm(Path(a.config), a.tag, cells, a.rounds, a.port, env, out, a.warm,+ exe=(a.exe or None), xargs=(a.xargs.split(";") if a.xargs else None))+ summary = summarize(rows, cells)+ (out / f"{a.tag}.summary.md").write_text(f"# {a.tag}\n\n{summary}\n")+ print("\n" + summary)+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tools/dgx_compare.py b/tools/dgx_compare.pynew file mode 100644index 000000000..fae149784--- /dev/null+++ b/tools/dgx_compare.py@@ -0,0 +1,135 @@+#!/usr/bin/env python3+"""dgx_compare - side-by-side medians and ratios of two dgx_bench result folders (or .jsonl files).++ python tools/dgx_compare.py bench/results/dgx-.../base.jsonl bench/results/dgx-.../gu.jsonl++Prints, per cell, the median prompt and decode tokens/s of both arms, their ratio, and the geometric mean of the+ratios over the cells (the "aggregate" the DGX Spark acceptance target is stated on). A request whose prompt was+served from the conversation cache (cache_n > 0) is not counted.+"""+from __future__ import annotations++import json+import math+import random+import statistics+import sys+from pathlib import Path+++def load(path: Path) -> list[dict]:+ rows = []+ for line in path.read_text().splitlines():+ line = line.strip()+ if not line:+ continue+ try:+ rows.append(json.loads(line))+ except ValueError:+ continue+ return rows+++def cells(rows: list[dict]) -> list[tuple]:+ return sorted({(r["len"], r["temp"], r["max_new"]) for r in rows})+++def med(rows: list[dict], key: str):+ vals = [float(r[key]) for r in rows if r.get(key) and not r.get("cache_n")]+ return statistics.median(vals) if vals else None+++def arm_stats(rows: list[dict]) -> dict:+ out = {}+ for c in cells(rows):+ sel = [r for r in rows if (r["len"], r["temp"], r["max_new"]) == c]+ out[c] = (med(sel, "prompt_per_second"), med(sel, "predicted_per_second"), med(sel, "ttft_s"),+ len([r for r in sel if not r.get("cache_n")]))+ return out++++def _by_cell(rows, key):+ out = {}+ for r in rows:+ if r.get(key) and not r.get("cache_n"):+ out.setdefault((r["len"], r["temp"], r["max_new"]), []).append(float(r[key]))+ return out+++def bootstrap(rows_a, rows_b, iters, seed=7):+ """Per-cell and aggregate ratio confidence intervals, resampling each arm's rows within a cell."""+ rng = random.Random(seed)+ res = {}+ for key in ("prompt_per_second", "predicted_per_second"):+ a, b = _by_cell(rows_a, key), _by_cell(rows_b, key)+ cells = sorted(set(a) & set(b))+ if not cells:+ continue+ draws = {c: [] for c in cells}+ agg = []+ for _ in range(iters):+ logs = []+ for c in cells:+ va, vb = a[c], b[c]+ ra = statistics.median([rng.choice(va) for _ in va])+ rb = statistics.median([rng.choice(vb) for _ in vb])+ if ra <= 0 or rb <= 0:+ continue+ draws[c].append(rb / ra)+ logs.append(math.log(rb / ra))+ if logs:+ agg.append(math.exp(sum(logs) / len(logs)))+ pct = lambda xs, p: sorted(xs)[min(len(xs) - 1, max(0, int(p * len(xs))))]+ res[key] = {c: (statistics.median(draws[c]), pct(draws[c], 0.05), pct(draws[c], 0.95)) for c in cells}+ res[key]["aggregate"] = (statistics.median(agg), pct(agg, 0.05), pct(agg, 0.95)) if agg else None+ return res++def main() -> int:+ if len(sys.argv) < 3:+ print(__doc__)+ return 2+ ci = int(sys.argv[3]) if len(sys.argv) > 3 else 0+ a_path, b_path = Path(sys.argv[1]), Path(sys.argv[2])+ a, b = load(a_path), load(b_path)+ sa, sb = arm_stats(a), arm_stats(b)+ name_a, name_b = a_path.stem, b_path.stem+ print(f"A = {name_a} B = {name_b}\n")+ print(f"| cell | A prompt | B prompt | ratio | A decode | B decode | ratio | A ttft | B ttft |")+ print("|---|---|---|---|---|---|---|---|---|")+ pr, dr = [], []+ for c in sorted(set(sa) | set(sb)):+ pa, da, ta, na = sa.get(c, (None, None, None, 0))+ pb, db, tb, nb = sb.get(c, (None, None, None, 0))+ f = lambda v, w="{:.1f}": "-" if v is None else w.format(v)+ rp = (pb / pa) if (pa and pb) else None+ rd = (db / da) if (da and db) else None+ if rp:+ pr.append(rp)+ if rd:+ dr.append(rd)+ print(f"| {c[0]} T={c[1]} n={c[2]} | {f(pa)} | {f(pb)} | {f(rp, '{:.3f}')}x | {f(da)} | {f(db)} | "+ f"{f(rd, '{:.3f}')}x | {f(ta, '{:.2f}')} | {f(tb, '{:.2f}')} |")+ gm = lambda xs: math.exp(sum(math.log(x) for x in xs) / len(xs)) if xs else float("nan")+ print(f"\nprompt aggregate (geomean of ratios over {len(pr)} cells): {gm(pr):.3f}x")+ print(f"decode aggregate (geomean of ratios over {len(dr)} cells): {gm(dr):.3f}x")+ if pr:+ print(f"lowest prompt ratio: {min(pr):.3f}x lowest decode ratio: {min(dr) if dr else float('nan'):.3f}x")+ if ci > 0:+ bs = bootstrap(a, b, ci)+ print(f"\nbootstrap 90% intervals over {ci} resamples (5th .. 95th percentile of the ratio):")+ for key, label in (("prompt_per_second", "prompt"), ("predicted_per_second", "decode")):+ if key not in bs:+ continue+ for c, v in sorted(bs[key].items(), key=lambda kv: str(kv[0])):+ if c == "aggregate":+ continue+ print(f" {label:6s} {c[0]:6d}/{c[1]}/{c[2]:4d}: {v[0]:.3f}x [{v[1]:.3f}, {v[2]:.3f}]")+ agg = bs[key].get("aggregate")+ if agg:+ print(f" {label:6s} aggregate: {agg[0]:.3f}x [{agg[1]:.3f}, {agg[2]:.3f}]")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tools/dgx_concurrent.py b/tools/dgx_concurrent.pynew file mode 100644index 000000000..08e2a1e61--- /dev/null+++ b/tools/dgx_concurrent.py@@ -0,0 +1,113 @@+#!/usr/bin/env python3+"""Aggregate output: N concurrent clients against one server, measuring tokens per second summed over the clients. Run the+same N against a second configuration to compare.++ python tools/dgx_concurrent.py CONFIG OUT.json --clients 8 [--clients 16] [--prompt 2000] [--max-new 128]+"""+from __future__ import annotations++import argparse+import json+import statistics+import subprocess+import sys+import threading+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()+++def make_prompt(tokens: int) -> str:+ body = []+ n = int(tokens / 1.35)+ for i in range(n):+ body.append(WORDS[i % len(WORDS)])+ return "Summarise the following log in two sentences.\n\n" + " ".join(body)+++def serve(cfg_path: str, port: int):+ cfg = json.loads(Path(cfg_path).read_text())+ cfg["port"] = port+ cfg["log"] = f"/tmp/dgx-conc-{port}.log"+ tmp = Path(f"/tmp/dgx-conc-{port}.json")+ tmp.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(tmp), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ t0 = time.time()+ while time.time() - t0 < 1800:+ if proc.poll() is not None:+ raise SystemExit(f"server exited {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ return proc+ except Exception:+ pass+ time.sleep(3)+ raise SystemExit("server did not come up")+++def one(port: int, prompt: str, max_new: int, out: list, idx: int):+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t0 = time.time()+ with urllib.request.urlopen(req, timeout=3600) as r:+ res = json.loads(r.read())+ out[idx] = (time.time() - t0, res.get("timings", {}).get("predicted_n", 0))+++def run(cfg_path: str, port: int, clients: int, prompt: str, max_new: int) -> dict:+ proc = serve(cfg_path, port)+ try:+ with urllib.request.urlopen(urllib.request.Request(+ f"http://127.0.0.1:{port}/v1/chat/completions",+ data=json.dumps({"messages": [{"role": "user", "content": "hi"}], "max_tokens": 8}).encode(),+ headers={"Content-Type": "application/json"}), timeout=1800) as r:+ r.read()+ out: list = [None] * clients+ t0 = time.time()+ threads = [threading.Thread(target=one, args=(port, prompt, max_new, out, i)) for i in range(clients)]+ for t in threads:+ t.start()+ for t in threads:+ t.join()+ wall = time.time() - t0+ tokens = sum(n for _, n in out if n)+ times = [d for d, _ in out if d]+ return {"clients": clients, "wall_s": round(wall, 2), "tokens": tokens,+ "aggregate_tok_s": round(tokens / wall, 2) if wall > 0 else None,+ "median_latency_s": round(statistics.median(times), 2)}+ finally:+ proc.terminate()+ try:+ proc.wait(180)+ except subprocess.TimeoutExpired:+ proc.kill()+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("config")+ ap.add_argument("out")+ ap.add_argument("--clients", type=int, nargs="+", default=[8, 16])+ ap.add_argument("--prompt", type=int, default=2000)+ ap.add_argument("--max-new", type=int, default=128)+ ap.add_argument("--port", type=int, default=8091)+ a = ap.parse_args()+ prompt = make_prompt(a.prompt)+ rows = [run(a.config, a.port, c, prompt, a.max_new) for c in a.clients]+ Path(a.out).write_text(json.dumps(rows, indent=1))+ for r in rows:+ print(json.dumps(r))+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tools/dgx_kvresident.py b/tools/dgx_kvresident.pynew file mode 100644index 000000000..aa60a825a--- /dev/null+++ b/tools/dgx_kvresident.py@@ -0,0 +1,81 @@+#!/usr/bin/env python3+"""Is decode at 56K limited by KV streaming? Raise --kv-resident to the whole context if it fits, and re-measure the same+56K prompt. Writes one row; compare it with the 31.0 tok/s of the shipped 32,768-cell setting.++ python dgx_kvresident.py CONFIG OUT.json [--tokens 56000]+"""+import json+import random+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()+++def build(tokens, code, rng):+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]+ at = rng.randrange(len(body) // 3, 2 * len(body) // 3)+ body[at:at] = [f"The access code for the harbor gate is {code}."]+ return ("Read the following log carefully, then answer the question at the end.\n\n" + " ".join(body) ++ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")+++def main():+ cfg_path, out_path = sys.argv[1], sys.argv[2]+ tokens = int(sys.argv[3]) if len(sys.argv) > 3 else 56000+ port = 8091+ cfg = json.loads(Path(cfg_path).read_text())+ cfg["port"] = port+ cfg["log"] = "/tmp/kvresident.log"+ tmp = Path("/tmp/kvresident.json")+ tmp.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(tmp), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ t0 = time.time()+ while time.time() - t0 < 1800:+ if proc.poll() is not None:+ raise SystemExit(f"server exited {proc.returncode} (see /tmp/kvresident.log)")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(3)+ print(f"ready in {time.time() - t0:.0f} s", flush=True)+ rng = random.Random(7)+ code = "481516"+ prompt = build(tokens, code, rng)+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 64, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t1 = time.time()+ with urllib.request.urlopen(req, timeout=3600) as r:+ out = json.loads(r.read())+ wall = time.time() - t1+ t = out.get("timings") or {}+ answer = out["choices"][0]["message"].get("content", "")+ row = {"tokens": tokens, "kv_resident": cfg["args"][cfg["args"].index("--kv-resident") + 1],+ "prompt_n": t.get("prompt_n"), "prompt_per_second": t.get("prompt_per_second"),+ "predicted_n": t.get("predicted_n"), "predicted_per_second": t.get("predicted_per_second"),+ "wall_s": round(wall, 1), "recalled": code in answer}+ Path(out_path).write_text(json.dumps(row, indent=1))+ print(json.dumps(row))+ finally:+ proc.terminate()+ try:+ proc.wait(180)+ except subprocess.TimeoutExpired:+ proc.kill()+++if __name__ == "__main__":+ main()diff --git a/tools/dgx_kvresident2.py b/tools/dgx_kvresident2.pynew file mode 100644index 000000000..a2b98a1c6--- /dev/null+++ b/tools/dgx_kvresident2.py@@ -0,0 +1,70 @@+#!/usr/bin/env python3+"""Clean long-context decode comparison: same 47K-token prompt, but a request that generates ~200 tokens in both cases, so+the decode rate is not dominated by per-request overhead. Compares --kv-resident 32768 against 65536.++ python dgx_kvresident2.py OUT.json [--tokens 47000]+"""+import json+import random+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+root = Path.home() / "workspace" / "strata-spark"+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()+++def build(tokens, rng):+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]+ return ("Read the following log carefully.\n\n" + " ".join(body) ++ "\n\nWrite a detailed 250-word analysis of the vocabulary and tone of the log above. Do not stop early.")+++def run(cfg_name, tag, port=8091):+ cfg = json.loads((root / cfg_name).read_text())+ cfg["port"] = port+ cfg["log"] = f"/tmp/kvres2-{tag}.log"+ tmp = Path(f"/tmp/kvres2-{tag}.json")+ tmp.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(tmp), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ t0 = time.time()+ while time.time() - t0 < 1800:+ if proc.poll() is not None:+ raise SystemExit(f"server exited {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(3)+ prompt = build(int(sys.argv[2]) if len(sys.argv) > 2 else 47000, random.Random(7))+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 256, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ with urllib.request.urlopen(req, timeout=3600) as r:+ out = json.loads(r.read())+ t = out.get("timings") or {}+ return {"kv_resident": cfg["args"][cfg["args"].index("--kv-resident") + 1],+ "prompt_n": t.get("prompt_n"), "prompt_per_second": t.get("prompt_per_second"),+ "predicted_n": t.get("predicted_n"), "predicted_per_second": t.get("predicted_per_second")}+ finally:+ proc.terminate()+ try:+ proc.wait(180)+ except subprocess.TimeoutExpired:+ proc.kill()+++rows = [run("strata-iq3_s.json", "a"), run("strata-iq3s-kv64k.json", "b")]+Path(sys.argv[1]).write_text(json.dumps(rows, indent=1))+for r in rows:+ print(json.dumps(r))diff --git a/tools/dgx_longctx.py b/tools/dgx_longctx.pynew file mode 100644index 000000000..601d0b370--- /dev/null+++ b/tools/dgx_longctx.py@@ -0,0 +1,81 @@+#!/usr/bin/env python3+"""Coverage past 32K: one 60K-token prompt on the 64K-context configuration, with a hidden code, so the run measures prompt+throughput at that length and checks recall in the same request.++ python long_context.py CONFIG OUT.json [--tokens 60000]+"""+import json+import random+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()+++def build(tokens, code, rng):+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]+ at = rng.randrange(len(body) // 3, 2 * len(body) // 3)+ body[at:at] = [f"The access code for the harbor gate is {code}."]+ return ("Read the following log carefully, then answer the question at the end.\n\n" + " ".join(body) ++ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")+++def main():+ cfg_path, out_path = sys.argv[1], sys.argv[2]+ tokens = int(sys.argv[3]) if len(sys.argv) > 3 else 60000+ port = 8091+ cfg = json.loads(Path(cfg_path).read_text())+ cfg["port"] = port+ cfg["log"] = "/tmp/long-context.log"+ tmp = Path("/tmp/long-context.json")+ tmp.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(tmp), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ t0 = time.time()+ while time.time() - t0 < 1800:+ if proc.poll() is not None:+ raise SystemExit(f"server exited {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(3)+ print(f"ready in {time.time() - t0:.0f} s", flush=True)+ rng = random.Random(7)+ code = "481516"+ prompt = build(tokens, code, rng)+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 32, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t1 = time.time()+ with urllib.request.urlopen(req, timeout=3600) as r:+ out = json.loads(r.read())+ wall = time.time() - t1+ t = out.get("timings") or {}+ answer = out["choices"][0]["message"].get("content", "")+ row = {"tokens": tokens, "prompt_n": t.get("prompt_n"), "prompt_per_second": t.get("prompt_per_second"),+ "predicted_per_second": t.get("predicted_per_second"), "wall_s": round(wall, 1),+ "answer": answer[:80], "recalled": code in answer}+ Path(out_path).write_text(json.dumps(row, indent=1))+ print(json.dumps(row))+ finally:+ proc.terminate()+ try:+ proc.wait(180)+ except subprocess.TimeoutExpired:+ proc.kill()+++if __name__ == "__main__":+ main()diff --git a/tools/dgx_needle.py b/tools/dgx_needle.pynew file mode 100644index 000000000..024b48d75--- /dev/null+++ b/tools/dgx_needle.py@@ -0,0 +1,96 @@+#!/usr/bin/env python3+"""Long-context recall check: hide one fact in a long prompt and ask for it back (the guide's "long-context retrieval" cell).++ python tools/dgx_needle.py --config strata-iq3s-final.json --tokens 12000++Starts the model's server, sends one request, and reports whether the answer contains the fact. The fact is placed at a+random position in the prompt and never appears in the question, so a correct answer means the context was read.+"""+from __future__ import annotations++import argparse+import json+import random+import subprocess+import sys+import time+import urllib.error+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()+++def build(tokens: int, needle: str, rng: random.Random) -> str:+ words = max(8, int(tokens / 1.35))+ body = [rng.choice(WORDS) for _ in range(words)]+ at = rng.randrange(len(body) // 4, 3 * len(body) // 4)+ body[at:at] = [needle]+ return ("Read the following notes carefully, then answer the question at the end.\n\n" + " ".join(body) ++ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")+++def post(port: int, prompt: str, max_tokens: int = 32) -> dict:+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ try:+ with urllib.request.urlopen(req, timeout=3600) as r:+ return json.loads(r.read())+ except urllib.error.HTTPError as e:+ raise SystemExit(f"HTTP {e.code}: {e.read()[:400]!r}")+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("--config", required=True)+ ap.add_argument("--tokens", type=int, default=12000)+ ap.add_argument("--port", type=int, default=8091)+ ap.add_argument("--runs", type=int, default=3)+ a = ap.parse_args()+ cfg = json.loads(Path(a.config).read_text())+ cfg["port"] = a.port+ cfg["log"] = "/tmp/dgx-needle.log"+ cpath = Path("/tmp/dgx-needle.json")+ cpath.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(cpath), "--port", str(a.port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ ok = 0+ try:+ for _ in range(600):+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{a.port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(2)+ post(a.port, "Say hi.", 8)+ for i in range(a.runs):+ rng = random.Random(1000 + i)+ code = f"{rng.randrange(100000, 999999)}"+ needle = f"The access code for the harbor gate is {code}."+ r = post(a.port, build(a.tokens, needle, rng))+ answer = r["choices"][0]["message"].get("content", "")+ t = r.get("timings") or {}+ hit = code in answer+ ok += hit+ print(f"run {i}: prompt_n {t.get('prompt_n')} at {t.get('prompt_per_second')} tok/s, "+ f"answer {answer[:40]!r} -> {'HIT' if hit else 'MISS'} (code {code})")+ finally:+ proc.terminate()+ try:+ proc.wait(60)+ except subprocess.TimeoutExpired:+ proc.kill()+ print(f"{ok} of {a.runs} recalled")+ return 0 if ok == a.runs else 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tools/dgx_quality.py b/tools/dgx_quality.pynew file mode 100644index 000000000..74d6818d0--- /dev/null+++ b/tools/dgx_quality.py@@ -0,0 +1,83 @@+#!/usr/bin/env python3+"""Quality check for a numerically different path (the fused IQ prompt kernels): run the same greedy prompts through+two configs and compare the answers.++ python quality_fused.py strata-iq3s-nofused.json strata-iq3s-best.json out.json+"""+import json+import os+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+PROMPTS = [+ "Write a Python function that merges two sorted lists into one sorted list, with a docstring and two tests.",+ "Explain in two short paragraphs how a heat pump moves heat from outside to inside a house.",+ "A train leaves at 09:15 and travels 210 km at 84 km/h. At what time does it arrive? Show the arithmetic.",+ "Summarize the main trade-offs between 4-bit and 8-bit weight quantization for LLM inference.",+ "List five concrete steps to debug a CUDA kernel that returns wrong results, one sentence each.",+]+++def post(port, prompt, max_tokens=160):+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens,+ "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ with urllib.request.urlopen(req, timeout=1800) as r:+ out = json.loads(r.read())+ return out["choices"][0]["message"].get("content", ""), out.get("timings") or {}+++def run(cfg, port, tag):+ cpath = Path(f"/tmp/quality-{tag}.json")+ c = json.loads(Path(cfg).read_text())+ c["port"] = port+ c["log"] = f"/tmp/quality-{tag}.log"+ cpath.write_text(json.dumps(c, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(cpath), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ for _ in range(600):+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(2)+ post(port, "Say hi.", 8)+ out = []+ for p in PROMPTS:+ txt, t = post(port, p)+ out.append(txt)+ print(f" {tag}: {t.get('prompt_per_second')} tok/s prompt, {t.get('predicted_per_second')} tok/s decode")+ return out+ finally:+ proc.terminate()+ try:+ proc.wait(60)+ except subprocess.TimeoutExpired:+ proc.kill()+ time.sleep(5)+++a = run(sys.argv[1], 8097, "a")+b = run(sys.argv[2], 8097, "b")+res = []+same = 0+for i, (x, y) in enumerate(zip(a, b)):+ eq = x == y+ same += eq+ pre = 0+ while pre < min(len(x), len(y)) and x[pre] == y[pre]:+ pre += 1+ res.append({"prompt": PROMPTS[i], "identical": eq, "common_prefix_chars": pre,+ "len_a": len(x), "len_b": len(y), "a": x, "b": y})+ print(f"prompt {i}: {'IDENTICAL' if eq else 'differs'} (prefix {pre} chars, {len(x)} vs {len(y)} chars)")+Path(sys.argv[3]).write_text(json.dumps({"identical": same, "of": len(PROMPTS), "items": res}, indent=1))+print(f"{same} of {len(PROMPTS)} answers identical")diff --git a/tools/dgx_quality24.py b/tools/dgx_quality24.pynew file mode 100644index 000000000..e9df7ba5e--- /dev/null+++ b/tools/dgx_quality24.py@@ -0,0 +1,109 @@+#!/usr/bin/env python3+"""A wider identity gate than tools/dgx_quality.py's five: 24 prompts across code, prose, math, multilingual and tool-shaped+asks, greedy, comparing the fused prompt path against MMQ byte for byte.++ python quality24.py CONFIG_A CONFIG_B OUT.json [--port 8091]+"""+import json+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+PROMPTS = [+ "Write a Python function that returns the longest palindromic substring of a string, with a docstring.",+ "In C, what does `static inline` mean at file scope and when should it be used? Two sentences.",+ "Explain the difference between a mutex and a semaphore to a junior developer, in under 100 words.",+ "Translate 'The server is down and we are investigating' into German, French and Spanish, one line each.",+ "Summarise the plot of Pride and Prejudice in exactly three sentences.",+ "What is the time complexity of building a binary heap, and why?",+ "Write a SQL query that returns the five customers with the highest total order value.",+ "List four failure modes of a distributed lock and one mitigation each, as a table.",+ "给一个刚学编程的人解释什么是递归,100字以内。",+ "Erkläre den Unterschied zwischen TCP und UDP in drei Sätzen.",+ "Écris une fonction JavaScript qui inverse une chaîne sans utiliser reverse().",+ "What happens in a Linux system when you run `rm -rf /` as root? Be technical and brief.",+ "Give me a shell one-liner that finds the ten largest files under /var/log.",+ "Why does a transformer need positional encodings? Answer in two sentences.",+ "Write a haiku about a GPU fan spinning up.",+ "What is the difference between `git rebase` and `git merge` for a shared branch? Two sentences.",+ "Explain Bloom filters and give one production use case, under 80 words.",+ "Sort these numbers ascending and show the result: 42, 7, 19, 3, 88, 15.",+ "What does the error `CUDA error: an illegal memory access was encountered` usually mean?",+ "Describe how to measure the power draw of a single process on Linux.",+ "Write a regex that matches ISO-8601 dates and explain each part.",+ "What is the difference between a process and a thread in one sentence?",+ "Give me a JSON object with three fields describing a book, and no prose.",+ "Explain why 0.1 + 0.2 != 0.3 in floating point, briefly.",+]+++def wait(port, proc):+ t0 = time.time()+ while time.time() - t0 < 900:+ if proc.poll() is not None:+ raise SystemExit(f"server exited {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:+ if r.status == 200:+ return+ except Exception:+ pass+ time.sleep(3)+ raise SystemExit("server did not come up")+++def ask(port, prompt, max_new=96):+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ with urllib.request.urlopen(req, timeout=1800) as r:+ out = json.loads(r.read())+ return out["choices"][0]["message"].get("content", "")+++def run_config(cfg_path, port, tag):+ cfg = json.loads(Path(cfg_path).read_text())+ cfg["port"] = port+ cfg["log"] = f"/tmp/quality24-{tag}.log"+ tmp = Path(f"/tmp/quality24-{tag}.json")+ tmp.write_text(json.dumps(cfg, indent=1))+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(tmp), "--port", str(port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ answers = []+ try:+ wait(port, proc)+ ask(port, "Say hi.", 8)+ for p in PROMPTS:+ answers.append(ask(port, p))+ finally:+ proc.terminate()+ try:+ proc.wait(120)+ except subprocess.TimeoutExpired:+ proc.kill()+ return answers+++def main():+ a_cfg, b_cfg, out_json = sys.argv[1], sys.argv[2], sys.argv[3]+ port = int(sys.argv[4]) if len(sys.argv) > 4 else 8091+ a = run_config(a_cfg, port, "a")+ b = run_config(b_cfg, port, "b")+ same = sum(1 for x, y in zip(a, b) if x == y)+ rows = [{"prompt": p, "a": x, "b": y, "identical": x == y} for p, x, y in zip(PROMPTS, a, b)]+ Path(out_json).write_text(json.dumps({"identical": same, "total": len(PROMPTS), "rows": rows}, indent=1))+ print(f"{same} of {len(PROMPTS)} answers identical")+ for i, r in enumerate(rows):+ if not r["identical"]:+ print(f" differs at prompt {i}: {r['prompt'][:60]!r}")+ print(f" A: {r['a'][:90]!r}")+ print(f" B: {r['b'][:90]!r}")+++if __name__ == "__main__":+ main()diff --git a/tools/dgx_smoke.py b/tools/dgx_smoke.pynew file mode 100644index 000000000..4daa83a01--- /dev/null+++ b/tools/dgx_smoke.py@@ -0,0 +1,110 @@+#!/usr/bin/env python3+"""The drop's acceptance gate in one command: setup detection, one benchmark cell, the identity gate and a needle, each+with a band that the numbers on a DGX Spark should land in. Exit code 0 = all passed.++ python tools/dgx_smoke.py [--config strata-iq3_s.json] [--tokens 12000] [--data-dir ~/Strata-data]++Bands are deliberately loose (they must survive page-cache state and drift); a failure means something is wrong, not that+the machine is 5% slower than the number in the documents.+"""+from __future__ import annotations++import argparse+import json+import os+import re+import subprocess+import sys+import tempfile+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+PY = sys.executable+WORK = Path(tempfile.mkdtemp(prefix="dgx-smoke-")) # this run's own scratch folder (never a fixed /tmp name another run could share)+++def sh(cmd: str) -> tuple[int, str]:+ p = subprocess.run(cmd, shell=True, cwd=str(ROOT), capture_output=True, text=True, timeout=3600)+ return p.returncode, (p.stdout or "") + (p.stderr or "")+++def check_setup(data_dir: str) -> tuple[bool, str]:+ rc, out = sh(f"{PY} setup.py --check --yes --data-dir '{data_dir}' 2>&1 | head -8")+ ok = "unified memory" in out and "arm64" in out+ return ok, out.strip().splitlines()[0] if out.strip() else "no output"+++def bench(config: str, cell: str, out_dir: str) -> dict | None:+ rc, out = sh(f"{PY} tools/dgx_bench.py --config {config} --tag smoke --cells {cell} --rounds 1 --port 8093 "+ f"--out {out_dir} 2>&1")+ m = re.search(r"prompt_n\s+(\d+)\s+prompt\s+([0-9.]+)\s+tok/s\s+decode\s+([0-9.]+)", out)+ if not m:+ return None+ return {"prompt_n": int(m.group(1)), "prompt": float(m.group(2)), "decode": float(m.group(3))}+++def identity(config: str) -> tuple[bool, str]:+ base = json.loads((ROOT / config).read_text())+ off = dict(base)+ off["args"] = list(base["args"])+ env = dict(base.get("env") or {})+ env["STRATA_PF_FUSED"] = "0"+ off["env"] = env+ off_path = WORK / "nofused.json"+ off_path.write_text(json.dumps(off, indent=1))+ rc, out = sh(f"{PY} tools/dgx_quality.py {off_path} {config} {WORK / 'quality.json'} 2>&1")+ m = re.search(r"(\d+) of (\d+) answers identical", out)+ if not m:+ return False, out.strip()[-200:]+ return m.group(1) == m.group(2), out.strip().splitlines()[-1]+++def needle(config: str, tokens: int) -> tuple[bool, str]:+ rc, out = sh(f"{PY} tools/dgx_needle.py --config {config} --tokens {tokens} --runs 2 2>&1")+ m = re.search(r"(\d+) of (\d+) recalled", out)+ if not m:+ return False, out.strip()[-200:]+ return m.group(1) == m.group(2), out.strip().splitlines()[-1]+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("--config", default="strata-iq3_s.json")+ ap.add_argument("--cell", default="4096:0:256")+ ap.add_argument("--tokens", type=int, default=12000)+ ap.add_argument("--data-dir", default=os.environ.get("STRATA_DATA_DIR") or str(Path.home() / "Strata-data"),+ help="the data folder setup uses (default: $STRATA_DATA_DIR or ~/Strata-data)")+ a = ap.parse_args()+ results = []+ t0 = time.time()++ ok, detail = check_setup(a.data_dir)+ results.append(("setup detection", ok, detail))++ b = bench(a.config, a.cell, str(WORK / "bench"))+ if b is None:+ results.append(("bench cell", False, "no parsable cell"))+ else:+ ok = b["prompt"] > 800 and b["decode"] > 30+ results.append(("bench cell", ok, f"{b['prompt']:.0f} prompt / {b['decode']:.1f} decode tok/s"))++ ok, detail = identity(a.config)+ results.append(("identity, fused vs MMQ", ok, detail))++ ok, detail = needle(a.config, a.tokens)+ results.append(("needle, recall", ok, detail))++ print()+ width = max(len(n) for n, _, _ in results)+ for name, ok, detail in results:+ print(f"{'PASS' if ok else 'FAIL'} {name:<{width}} {detail}")+ bad = [n for n, ok, _ in results if not ok]+ print(f"\n{len(results) - len(bad)} of {len(results)} passed in {time.time() - t0:.0f} s"+ + (f"; failed: {', '.join(bad)}" if bad else ""))+ return 0 if not bad else 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tools/dgx_turns.py b/tools/dgx_turns.pynew file mode 100644index 000000000..351f0faaa--- /dev/null+++ b/tools/dgx_turns.py@@ -0,0 +1,97 @@+#!/usr/bin/env python3+"""Multi-turn (agent-style) comparison: one conversation, five turns of ~1,200 new tokens each, cache reuse measured.++ python tools/dgx_turns.py CONFIG TAG OUTDIR [--turns 5] [--tokens 1200] [--port 8091]++Per turn it records the engine's prompt_n / cache_n, prompt tok/s and TTFT, so the cache path is compared, not just the+fresh-prompt path the other tools measure.+"""+import argparse+import json+import random+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()+++def blob(n, seed):+ rng = random.Random(seed)+ return " ".join(rng.choice(WORDS) for _ in range(n))+++def chat(port, messages, max_tokens=64):+ body = json.dumps({"messages": messages, "max_tokens": max_tokens, "temperature": 0, "stream": False,+ "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t0 = time.time()+ with urllib.request.urlopen(req, timeout=3600) as r:+ out = json.loads(r.read())+ return out, time.time() - t0+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("config")+ ap.add_argument("tag")+ ap.add_argument("outdir")+ ap.add_argument("--turns", type=int, default=5)+ ap.add_argument("--tokens", type=int, default=1200)+ ap.add_argument("--port", type=int, default=8091)+ a = ap.parse_args()+ out = Path(a.outdir)+ out.mkdir(parents=True, exist_ok=True)+ cfg = json.loads(Path(a.config).read_text())+ cfg["port"] = a.port+ cfg["log"] = str(out / f"{a.tag}.engine.log")+ cpath = out / f"{a.tag}.config.json"+ cpath.write_text(json.dumps(cfg, indent=1))+ log = open(out / f"{a.tag}.jsonl", "a", encoding="utf-8")+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(cpath), "--port", str(a.port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ t0 = time.time()+ while time.time() - t0 < 3600:+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{a.port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(3)+ print(f"{a.tag}: ready in {time.time() - t0:.0f} s", flush=True)+ msgs = [{"role": "user", "content": "Read this note and answer with one short sentence.\n" + blob(a.tokens, 1)}]+ outj, wall = chat(a.port, msgs)+ msgs.append({"role": "assistant", "content": outj["choices"][0]["message"].get("content", "")})+ for turn in range(a.turns):+ msgs.append({"role": "user", "content": "More notes.\n" + blob(a.tokens, 100 + turn) ++ "\nSummarise the last note in one sentence."})+ outj, wall = chat(a.port, msgs)+ t = outj.get("timings") or {}+ msgs.append({"role": "assistant", "content": outj["choices"][0]["message"].get("content", "")})+ row = {"round": turn, "arm": a.tag, "len": 0, "temp": 0.0, "max_new": 64, "wall_s": round(wall, 2),+ "ttft_s": (t.get("prompt_ms") or 0) / 1000.0 or None, **t}+ log.write(json.dumps(row) + "\n")+ log.flush()+ print(f" turn {turn}: prompt_n {t.get('prompt_n')} cache_n {t.get('cache_n')} "+ f"prompt {t.get('prompt_per_second')} tok/s decode {t.get('predicted_per_second')} tok/s "+ f"ttft {row['ttft_s']}", flush=True)+ finally:+ proc.terminate()+ try:+ proc.wait(90)+ except subprocess.TimeoutExpired:+ proc.kill()+ log.close()+++if __name__ == "__main__":+ main()diff --git a/tools/dgx_warm.py b/tools/dgx_warm.pynew file mode 100644index 000000000..91ef57700--- /dev/null+++ b/tools/dgx_warm.py@@ -0,0 +1,107 @@+#!/usr/bin/env python3+"""One server, several repeats per cell: the warm-state runner (the per-start page cache and the tens-of-GB expert+prefetch are the main variance sources on a unified-memory box, so repeats inside one server are the comparable ones).++ python tools/dgx_warm.py CONFIG TAG OUTDIR [--cells 4096,16384,32768] [--repeats 3] [--port 8091] [--new 256]++Writes <OUTDIR>/<TAG>.jsonl rows in tools/dgx_bench.py's schema (round, arm, len, temp, max_new, prompt_n,+prompt_per_second, predicted_n, predicted_per_second, ttft_s, cache_n) so tools/dgx_compare.py reads them.+"""+import argparse+import json+import random+import subprocess+import sys+import time+import urllib.request+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()+++def prompt_for(tokens, nonce):+ rng = random.Random(tokens)+ body = " ".join(rng.choice(WORDS) for _ in range(max(8, int(tokens / 1.35))))+ return (f"Request nonce {nonce}. Read the word list below carefully.\n\n{body}\n\n"+ "Write a long, detailed analysis of the word list above. Keep writing for at least 400 words.")+++def post(port, prompt, max_new):+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,+ "stream": False, "chat_template_kwargs": {"enable_thinking": False}}).encode()+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,+ headers={"Content-Type": "application/json"})+ t0 = time.time()+ with urllib.request.urlopen(req, timeout=3600) as r:+ out = json.loads(r.read())+ return out, time.time() - t0+++def status(port):+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/status", timeout=10) as r:+ return json.loads(r.read()).get("last_timings") or {}+ except Exception:+ return {}+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("config")+ ap.add_argument("tag")+ ap.add_argument("outdir")+ ap.add_argument("--cells", default="512,2048,4096")+ ap.add_argument("--repeats", type=int, default=4)+ ap.add_argument("--port", type=int, default=8091)+ ap.add_argument("--new", type=int, default=256)+ a = ap.parse_args()+ out = Path(a.outdir)+ out.mkdir(parents=True, exist_ok=True)+ cfg = json.loads(Path(a.config).read_text())+ cfg["port"] = a.port+ cfg["log"] = str(out / f"{a.tag}.engine.log")+ cpath = out / f"{a.tag}.config.json"+ cpath.write_text(json.dumps(cfg, indent=1))+ log = open(out / f"{a.tag}.jsonl", "a", encoding="utf-8")+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",+ str(cpath), "--port", str(a.port)], cwd=str(ROOT),+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)+ try:+ t0 = time.time()+ while time.time() - t0 < 3600:+ if proc.poll() is not None:+ raise SystemExit(f"server exited with {proc.returncode}")+ try:+ with urllib.request.urlopen(f"http://127.0.0.1:{a.port}/v1/models", timeout=5) as r:+ if r.status == 200:+ break+ except Exception:+ pass+ time.sleep(3)+ print(f"{a.tag}: ready in {time.time() - t0:.0f} s", flush=True)+ post(a.port, prompt_for(256, "warm"), 8)+ for rep in range(a.repeats):+ for length in [int(x) for x in a.cells.split(",")]:+ nonce = f"{random.getrandbits(48):012x}"+ outj, wall = post(a.port, prompt_for(length, nonce), a.new)+ t = status(a.port)+ row = {"round": rep, "arm": a.tag, "len": length, "temp": 0.0, "max_new": a.new,+ "wall_s": round(wall, 2), "ttft_s": (t.get("prompt_ms") or 0) / 1000.0 or None,+ "cache_n": t.get("cache_n", 0), **t}+ log.write(json.dumps(row) + "\n")+ log.flush()+ print(f" {length:5d} prompt_n {t.get('prompt_n')} prompt {t.get('prompt_per_second')} tok/s "+ f"decode {t.get('predicted_per_second')} tok/s", flush=True)+ finally:+ proc.terminate()+ try:+ proc.wait(90)+ except subprocess.TimeoutExpired:+ proc.kill()+ log.close()+++if __name__ == "__main__":+ main()