Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

The DGX Spark benchmark family and guide (from the owner's handover, 2026-10-26)

Niko1221694e71170828
diff --git a/docs/DGX_SPARK_LINUX.md b/docs/DGX_SPARK_LINUX.md
new file mode 100644
index 000000000..b3211d47b
--- /dev/null
+++ b/docs/DGX_SPARK_LINUX.md
@@ -0,0 +1,90 @@
+# Strata on the NVIDIA DGX Spark (Linux, arm64)
+
+One desktop box with a GB10 GPU, 20 Arm CPU cores and 128 GB of memory shared by the GPU and the CPU. This page is the
+arm64 Linux build of Strata from source: what works, how to build it, how to install a model and how the speed was
+measured. It is experimental: one Spark, one driver, one toolkit.
+
+## What works, and what does not
+
+- **Works:** the CUDA engine (`strata`), Qwen3.8-Flash-Next IQ3_S and the NVFP4 GPTQ pack, 64K context, pictures off,
+ the DeepSeek-V4.1 engine built from source (`strata-dsv41`).
+- **Not checked here:** the picture reader (`strata-vision` builds, not exercised in these runs), 128K context,
+ DeepSeek serving (the engine builds; no model was downloaded in this session), the app, the Terminal and the
+ ready-made engine feed, which are x86_64 only.
+- **CPU experts:** Strata's own CPU expert kernels are x86 AVX/AVX-512; the arm64 build leaves them out and a native
+ pack's CPU experts run on ggml-cpu's NEON rows. A canonical Q2_0 pack (no `native_experts.txt`) still needs AVX-512
+ and is refused at start.
+
+## Build the engine
+
+Needs an arm64 Linux with the CUDA 13.0 toolkit (`/usr/local/cuda`; DGX OS has it) and Python 3.10+ with venv.
+
+```sh
+python3 -m venv .venv-build
+.venv-build/bin/pip install -r requirements.txt -r requirements-build.txt
+PATH="$PWD/.venv-build/bin:/usr/local/cuda/bin:$PATH" PYTHON="$PWD/.venv-build/bin/python" \
+ JOBS=20 sh spark/build_engine.sh
+```
+
+About 3 minutes on the Spark with 20 jobs (the first run also fetches llama.cpp at the pinned commit). It leaves
+`out-spark/engine/` with `strata`, `libstrata_kq.so`, the CUDA libraries it needs and a `BUILD.json` that says
+`linux-aarch64`, arch `121`.
+
+`BUILD_VISION=0` skips the picture reader; `BUILD_DSV41=0` skips the DeepSeek engine; `ARCHS=121` is the default and
+is what a Spark needs.
+
+Put it where setup looks for an engine:
+
+```sh
+cp -r out-spark/engine ./engine
+```
+
+## Install a model
+
+```sh
+./setup.sh --setup --family qwen --model IQ3_S --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data
+./setup.sh --setup --family nvfp4 --model NVFP4 --context 65536 --vision no --yes --no-start --data-dir ~/Strata-data
+```
+
+`--data-dir` must be a folder you can write (on a DGX Spark the default `../Strata-data` next to the checkout is under
+`/home`, which `dev` cannot create; `~/Strata-data` works). Setup finds the GB10, reports it as unified memory, and
+passes `--mmap-experts`: the experts are read from the model files in place, so the GPU's cache holds all 24,576 of them
+instead of a second copy in RAM.
+
+Start a model with its run script (`./run-iq3_s.sh`) or `./setup.sh`. `./bench.sh` runs the quick speed check.
+
+## What the Spark does with the memory
+
+| | IQ3_S | NVFP4 GPTQ |
+|---|---|---|
+| Expert bytes | 50.3 GB | 68.0 GB |
+| Expert cache | 24,576 of 24,576 slots, 46.84 GiB | 24,576 of 24,576 slots, 63.29 GiB |
+| n-gram table | 28.8 GB, locked in RAM (`--ple-io ram`) | 51.2 GB, read from the SSD (`direct`) |
+| Context | 64K, 8-bit KV, 32,768 cells streamed | same |
+| Free VRAM at start | ~1.0-1.2 GiB | ~1.2 GiB |
+
+The NVFP4 table does not fit in RAM beside the page cache the 68 GB expert file needs: locking it measured 137 tok/s
+on a 4K prompt and 26 tok/s decode against 1,252 and 39 without. Leave it on the SSD.
+
+## Speeds (medians, one Spark, 26 October 2026)
+
+Measured with `tools/dgx_bench.py`: fresh 4K / 16K / 32K prompts at temperature 0, 256-token replies, the engine's own
+`timings`, three paired rounds against steddyman's 0.1.49.2 engine built on the same machine.
+
+| Model | prompt tok/s (4K / 16K / 32K) | decode tok/s |
+|---|---|---|
+| Qwen3.8-Flash-Next IQ3_S, this build | 1,416 / 1,597 / 1,581 | 45.5 / 45.0 / 44.0 |
+| Qwen3.8-Flash-Next IQ3_S, the fork | 1,237 / 1,331 / 1,402 | 44.4 / 43.4 / 43.1 |
+| Qwen3.8-Flash-Next NVFP4, this build | see HANDOVER.md | see HANDOVER.md |
+| Qwen3.8-Flash-Next NVFP4, the fork | see HANDOVER.md | see HANDOVER.md |
+
+On IQ3_S that is +16% prompts over the fork and +3% decode. The prompt gain is `--prefill auto:32768` plus the fused
+IQ prompt path (`STRATA_PF_FUSED=1`, opt-in on this card because the fused kernels round differently from MMQ).
+Decode is GPU-work-bound on the 48-SM GB10 (109 kernel launches per token, ~99% GPU busy); the remaining decode work is
+kernel work, not configuration - see the handover's backlog.
+
+## Rules for this machine
+
+- Don't shut it down or restart it without agreement.
+- Never publish the SSH key or put it in a repository.
+- Ask the machine's owner for extra rights or installs.
diff --git a/tools/dgx_128k.py b/tools/dgx_128k.py
new file mode 100644
index 000000000..e0bca55a7
--- /dev/null
+++ b/tools/dgx_128k.py
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Does the engine hold a context past the installed 64K? Copy the shipped config, raise the context to 128K, and send a
+100K-token prompt with a hidden code. Success = an answer that recalls it; failure is recorded as a limit.
+
+ python dgx_128k.py CONFIG OUT.json [--tokens 100000] [--ctx 131072]
+"""
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "
+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()
+
+
+def build(tokens, code, rng):
+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]
+ at = rng.randrange(len(body) // 3, 2 * len(body) // 3)
+ body[at:at] = [f"The access code for the harbor gate is {code}."]
+ return ("Read the following log carefully, then answer the question at the end.\n\n" + " ".join(body) +
+ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")
+
+
+def main():
+ cfg_path, out_path = sys.argv[1], sys.argv[2]
+ tokens = int(sys.argv[3]) if len(sys.argv) > 3 else 100000
+ ctx = int(sys.argv[4]) if len(sys.argv) > 4 else 131072
+ port = 8091
+ cfg = json.loads(Path(cfg_path).read_text())
+ a = cfg["args"]
+ if "--max-context" in a:
+ a[a.index("--max-context") + 1] = str(ctx)
+ if "--kv-resident" in a:
+ a[a.index("--kv-resident") + 1] = "32768"
+ cfg["port"] = port
+ cfg["log"] = "/tmp/long-128k.log"
+ tmp = Path("/tmp/long-128k.json")
+ tmp.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(tmp), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ t0 = time.time()
+ while time.time() - t0 < 1800:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(3)
+ print(f"ready in {time.time() - t0:.0f} s", flush=True)
+ rng = random.Random(11)
+ code = "271828"
+ prompt = build(tokens, code, rng)
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 32, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t1 = time.time()
+ with urllib.request.urlopen(req, timeout=5400) as r:
+ out = json.loads(r.read())
+ wall = time.time() - t1
+ t = out.get("timings") or {}
+ answer = out["choices"][0]["message"].get("content", "")
+ row = {"tokens": tokens, "ctx": ctx, "prompt_n": t.get("prompt_n"),
+ "prompt_per_second": t.get("prompt_per_second"),
+ "predicted_per_second": t.get("predicted_per_second"), "wall_s": round(wall, 1),
+ "answer": answer[:80], "recalled": code in answer}
+ Path(out_path).write_text(json.dumps(row, indent=1))
+ print(json.dumps(row))
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(180)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/dgx_bench.py b/tools/dgx_bench.py
new file mode 100644
index 000000000..768121a0f
--- /dev/null
+++ b/tools/dgx_bench.py
@@ -0,0 +1,254 @@
+#!/usr/bin/env python3
+"""dgx_bench - the DGX Spark benchmark family for Strata (Qwen3.8-Flash-Next first).
+
+ python tools/dgx_bench.py --config strata-iq3_s.json --tag base
+ python tools/dgx_bench.py --config strata-iq3_s.json --tag gu-swiglu --env STRATA_GU_SWIGLU=1
+ python tools/dgx_bench.py --config strata-iq3_s.json --tag base --cells 4096:0:256,4096:0.7:256 --rounds 3
+
+It starts the server on the model's own config (serve/server.py, the path setup writes), waits for it, sends the cells
+through the real HTTP API, restarts the server between arms, and writes every request plus a summary.
+
+A cell is PROMPT_TOKENS:TEMPERATURE:MAX_NEW. The default family is the one the DGX Spark guide's contract names:
+512 / 4096 / 16384 / 32768 fresh prompt tokens at temperature 0 and 0.7, with a 256-token generation cap, plus one
+1024-token decode cell. Every request carries a unique nonce as its first token sequence, so the conversation cache
+cannot serve the prompt (verified: timings.cache_n must be 0, or the row is marked reused and excluded from the summary).
+
+Per request it records the client's own time to first content token (TTFT), the wall time, and the server's timings
+(prompt_n, prompt_ms, prompt_per_second, predicted_n, predicted_ms, predicted_per_second, draft counts) read from
+GET /v1/status right after the streamed answer. Medians over rounds; nothing is retried or filtered except requests
+that failed (they are kept and reported as failures).
+
+Standard library only, like tools/strata_bench.py and tools/ab_engine.py.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import random
+import statistics
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+DEFAULT_CELLS = "512:0:256,4096:0:256,4096:0.7:256,16384:0:256,16384:0.7:256,32768:0:256,32768:0.7:256,4096:0:1024"
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "
+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr "
+ "basalt cinder dune elm fjord granite heather inlet jetty kiln lagoon mesa niche opal prairie quill").split()
+
+
+def build_prompt(target_tokens: int, nonce: str) -> str:
+ """A prompt of about target_tokens tokens: a unique nonce line, a word list, and a task that keeps the answer going."""
+ rng = random.Random(hash((target_tokens, nonce)) & 0xFFFFFFFF)
+ # ~1.35 tokens per word for this vocabulary in Qwen's tokenizer
+ words = max(8, int(target_tokens / 1.35))
+ body = " ".join(rng.choice(WORDS) for _ in range(words))
+ return (f"Request nonce {nonce}. Read the word list below carefully.\n\n{body}\n\n"
+ f"Write a long, detailed analysis of the word list above: its sounds, its themes and its structure. "
+ f"Keep writing for at least 400 words; do not stop early and do not ask questions.")
+
+
+def post_stream(port: int, prompt: str, max_tokens: int, temperature: float, timeout: float = 3600) -> dict:
+ """One streamed chat completion; returns the client's times and the token count."""
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens,
+ "temperature": temperature, "stream": True,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t0 = time.time()
+ ttft = None
+ n = 0
+ text_len = 0
+ finish = None
+ with urllib.request.urlopen(req, timeout=timeout) as r:
+ for raw in r:
+ line = raw.decode("utf-8", "replace").strip()
+ if not line.startswith("data:"):
+ continue
+ data = line[5:].strip()
+ if data == "[DONE]":
+ break
+ try:
+ obj = json.loads(data)
+ except ValueError:
+ continue
+ for ch in obj.get("choices") or []:
+ d = ch.get("delta") or {}
+ piece = d.get("content") or d.get("reasoning_content") or ""
+ if piece:
+ if ttft is None:
+ ttft = time.time() - t0
+ text_len += len(piece)
+ if ch.get("finish_reason"):
+ finish = ch["finish_reason"]
+ if obj.get("usage"):
+ n = int(obj["usage"].get("completion_tokens") or n)
+ wall = time.time() - t0
+ return {"ttft_s": ttft, "wall_s": round(wall, 3), "stream_tokens": n, "chars": text_len, "finish": finish}
+
+
+def last_timings(port: int) -> dict:
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/status", timeout=10) as r:
+ st = json.loads(r.read())
+ return st.get("last_timings") or {}
+ except Exception:
+ return {}
+
+
+def wait_ready(port: int, proc: subprocess.Popen, timeout: float = 3600) -> float:
+ t0 = time.time()
+ while time.time() - t0 < timeout:
+ if proc.poll() is not None:
+ raise RuntimeError(f"server exited with {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ return time.time() - t0
+ except Exception:
+ pass
+ time.sleep(3)
+ raise TimeoutError("server not ready")
+
+
+def calibrate(port: int, target: int, max_tokens: int, temperature: float, log) -> str:
+ """Build a prompt whose prompt_n is within ~2% of target (two probes at most)."""
+ nonce = f"{random.getrandbits(48):012x}"
+ prompt = build_prompt(target, nonce)
+ for attempt in range(3):
+ r = post_stream(port, prompt, 8, 0.0)
+ t = last_timings(port)
+ got = int(t.get("prompt_n") or 0)
+ log(f" calibrate {target}: prompt_n {got}")
+ if got and abs(got - target) <= max(16, target // 50):
+ return prompt
+ if not got:
+ break
+ words = len(prompt.split()) * target / got
+ rng = random.Random(target + attempt)
+ body = " ".join(rng.choice(WORDS) for _ in range(max(8, int(words))))
+ prompt = (f"Request nonce {nonce}. Read the word list below carefully.\n\n{body}\n\n"
+ f"Write a long, detailed analysis of the word list above: its sounds, its themes and its structure. "
+ f"Keep writing for at least 400 words; do not stop early and do not ask questions.")
+ return prompt
+
+
+def run_arm(cfg_path: Path, tag: str, cells, rounds: int, port: int, env: dict, out: Path, warm: int,
+ exe: str | None = None, xargs: list[str] | None = None) -> list[dict]:
+ base = json.loads(cfg_path.read_text())
+ cfg = dict(base)
+ if exe:
+ cfg["exe"] = exe
+ if xargs:
+ cfg["args"] = list(cfg.get("args") or []) + list(xargs)
+ cfg["port"] = port
+ cfg["log"] = str(out / f"{tag}.engine.log")
+ cpath = out / f"{tag}.config.json"
+ cpath.write_text(json.dumps(cfg, indent=1))
+ rows: list[dict] = []
+ meta = {"tag": tag, "config": str(cfg_path), "cells": cells, "rounds": rounds, "env": env,
+ "engine": cfg.get("exe"), "args": cfg.get("args"), "started": time.strftime("%Y-%m-%dT%H:%M:%S")}
+ (out / f"{tag}.meta.json").write_text(json.dumps(meta, indent=1))
+ logf = open(out / f"{tag}.jsonl", "a", encoding="utf-8")
+ penv = dict(os.environ, **env)
+ for rnd in range(rounds):
+ srvlog = open(out / f"{tag}.server.log", "a", encoding="utf-8")
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(cpath), "--port", str(port)], cwd=str(ROOT), env=penv,
+ stdout=srvlog, stderr=srvlog)
+ try:
+ ready = wait_ready(port, proc)
+ print(f"round {rnd} {tag}: ready in {ready:.0f} s", flush=True)
+ nonce = f"{random.getrandbits(48):012x}"
+ post_stream(port, f"Request nonce {nonce}. Say hello in one short sentence, then list five colors.", warm, 0.0)
+ # a 4K prompt first: with --mmap-experts the first request pages the experts in (a one-time cold cost of
+ # tens of seconds); every measured cell then runs on the same warm state the fork's baselines had
+ nonce = f"{random.getrandbits(48):012x}"
+ post_stream(port, build_prompt(4096, nonce), 8, 0.0)
+ prompts = {}
+ for (length, temp, maxnew) in cells:
+ key = (length, maxnew)
+ if key not in prompts:
+ prompts[key] = calibrate(port, length, maxnew, 0.0, print)
+ nonce = f"{random.getrandbits(48):012x}"
+ prompt = prompts[key]
+ prompt = prompt.replace("Request nonce ", "Request nonce ", 1)
+ # a fresh nonce: rewrite the first line so no prefix can be reused
+ prompt = prompt.split("\n", 1)
+ prompt = f"Request nonce {nonce}. Read the word list below carefully.\n" + prompt[1]
+ r = post_stream(port, prompt, maxnew, temp)
+ t = last_timings(port)
+ row = {"round": rnd, "arm": tag, "len": length, "temp": temp, "max_new": maxnew, **r, **t}
+ rows.append(row)
+ logf.write(json.dumps(row) + "\n")
+ logf.flush()
+ print(f" {length:6d} T={temp} n={maxnew:4d}: prompt_n {t.get('prompt_n')} "
+ f"prompt {t.get('prompt_per_second')} tok/s decode {t.get('predicted_per_second')} tok/s "
+ f"n {t.get('predicted_n')} ttft {r['ttft_s'] and round(r['ttft_s'], 2)} wall {r['wall_s']}",
+ flush=True)
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ time.sleep(5)
+ logf.close()
+ return rows
+
+
+def summarize(rows: list[dict], cells) -> str:
+ lines = ["| cell | n | prompt tok/s | decode tok/s | ttft s | wall s |", "|---|---|---|---|---|---|"]
+ for (length, temp, maxnew) in cells:
+ sel = [r for r in rows if r["len"] == length and r["temp"] == temp and r["max_new"] == maxnew
+ and not r.get("cache_n")]
+ if not sel:
+ lines.append(f"| {length} T={temp} n={maxnew} | 0 | - | - | - | - |")
+ continue
+ med = lambda k: statistics.median([r[k] for r in sel if r.get(k)]) if any(r.get(k) for r in sel) else None
+ p, d, ttft, wall = med("prompt_per_second"), med("predicted_per_second"), med("ttft_s"), med("wall_s")
+ fmt = lambda v, f="{:.1f}": "-" if v is None else f.format(v)
+ lines.append(f"| {length} T={temp} n={maxnew} | {len(sel)} | {fmt(p)} | {fmt(d)} | {fmt(ttft, '{:.2f}')} | "
+ f"{fmt(wall, '{:.1f}')} |")
+ return "\n".join(lines)
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--config", required=True)
+ ap.add_argument("--tag", default="base")
+ ap.add_argument("--cells", default=DEFAULT_CELLS)
+ ap.add_argument("--rounds", type=int, default=3)
+ ap.add_argument("--port", type=int, default=8091)
+ ap.add_argument("--env", action="append", default=[], help="NAME=VALUE for the server process (repeatable)")
+ ap.add_argument("--exe", default="", help="engine binary to run instead of the config's")
+ ap.add_argument("--xargs", default="", help="extra engine arguments, ';'-separated")
+ ap.add_argument("--warm", type=int, default=64)
+ ap.add_argument("--out", default="")
+ a = ap.parse_args()
+ cells = []
+ for s in a.cells.split(","):
+ l, t, n = s.split(":")
+ cells.append((int(l), float(t), int(n)))
+ out = Path(a.out) if a.out else ROOT / "bench" / "results" / time.strftime("dgx-%Y-%m-%d-%H%M")
+ out.mkdir(parents=True, exist_ok=True)
+ env = {}
+ for e in a.env:
+ k, v = e.split("=", 1)
+ env[k] = v
+ print(f"out {out}; cells {cells}; rounds {a.rounds}", flush=True)
+ rows = run_arm(Path(a.config), a.tag, cells, a.rounds, a.port, env, out, a.warm,
+ exe=(a.exe or None), xargs=(a.xargs.split(";") if a.xargs else None))
+ summary = summarize(rows, cells)
+ (out / f"{a.tag}.summary.md").write_text(f"# {a.tag}\n\n{summary}\n")
+ print("\n" + summary)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/dgx_compare.py b/tools/dgx_compare.py
new file mode 100644
index 000000000..fae149784
--- /dev/null
+++ b/tools/dgx_compare.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""dgx_compare - side-by-side medians and ratios of two dgx_bench result folders (or .jsonl files).
+
+ python tools/dgx_compare.py bench/results/dgx-.../base.jsonl bench/results/dgx-.../gu.jsonl
+
+Prints, per cell, the median prompt and decode tokens/s of both arms, their ratio, and the geometric mean of the
+ratios over the cells (the "aggregate" the DGX Spark acceptance target is stated on). A request whose prompt was
+served from the conversation cache (cache_n > 0) is not counted.
+"""
+from __future__ import annotations
+
+import json
+import math
+import random
+import statistics
+import sys
+from pathlib import Path
+
+
+def load(path: Path) -> list[dict]:
+ rows = []
+ for line in path.read_text().splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ rows.append(json.loads(line))
+ except ValueError:
+ continue
+ return rows
+
+
+def cells(rows: list[dict]) -> list[tuple]:
+ return sorted({(r["len"], r["temp"], r["max_new"]) for r in rows})
+
+
+def med(rows: list[dict], key: str):
+ vals = [float(r[key]) for r in rows if r.get(key) and not r.get("cache_n")]
+ return statistics.median(vals) if vals else None
+
+
+def arm_stats(rows: list[dict]) -> dict:
+ out = {}
+ for c in cells(rows):
+ sel = [r for r in rows if (r["len"], r["temp"], r["max_new"]) == c]
+ out[c] = (med(sel, "prompt_per_second"), med(sel, "predicted_per_second"), med(sel, "ttft_s"),
+ len([r for r in sel if not r.get("cache_n")]))
+ return out
+
+
+
+def _by_cell(rows, key):
+ out = {}
+ for r in rows:
+ if r.get(key) and not r.get("cache_n"):
+ out.setdefault((r["len"], r["temp"], r["max_new"]), []).append(float(r[key]))
+ return out
+
+
+def bootstrap(rows_a, rows_b, iters, seed=7):
+ """Per-cell and aggregate ratio confidence intervals, resampling each arm's rows within a cell."""
+ rng = random.Random(seed)
+ res = {}
+ for key in ("prompt_per_second", "predicted_per_second"):
+ a, b = _by_cell(rows_a, key), _by_cell(rows_b, key)
+ cells = sorted(set(a) & set(b))
+ if not cells:
+ continue
+ draws = {c: [] for c in cells}
+ agg = []
+ for _ in range(iters):
+ logs = []
+ for c in cells:
+ va, vb = a[c], b[c]
+ ra = statistics.median([rng.choice(va) for _ in va])
+ rb = statistics.median([rng.choice(vb) for _ in vb])
+ if ra <= 0 or rb <= 0:
+ continue
+ draws[c].append(rb / ra)
+ logs.append(math.log(rb / ra))
+ if logs:
+ agg.append(math.exp(sum(logs) / len(logs)))
+ pct = lambda xs, p: sorted(xs)[min(len(xs) - 1, max(0, int(p * len(xs))))]
+ res[key] = {c: (statistics.median(draws[c]), pct(draws[c], 0.05), pct(draws[c], 0.95)) for c in cells}
+ res[key]["aggregate"] = (statistics.median(agg), pct(agg, 0.05), pct(agg, 0.95)) if agg else None
+ return res
+
+def main() -> int:
+ if len(sys.argv) < 3:
+ print(__doc__)
+ return 2
+ ci = int(sys.argv[3]) if len(sys.argv) > 3 else 0
+ a_path, b_path = Path(sys.argv[1]), Path(sys.argv[2])
+ a, b = load(a_path), load(b_path)
+ sa, sb = arm_stats(a), arm_stats(b)
+ name_a, name_b = a_path.stem, b_path.stem
+ print(f"A = {name_a} B = {name_b}\n")
+ print(f"| cell | A prompt | B prompt | ratio | A decode | B decode | ratio | A ttft | B ttft |")
+ print("|---|---|---|---|---|---|---|---|---|")
+ pr, dr = [], []
+ for c in sorted(set(sa) | set(sb)):
+ pa, da, ta, na = sa.get(c, (None, None, None, 0))
+ pb, db, tb, nb = sb.get(c, (None, None, None, 0))
+ f = lambda v, w="{:.1f}": "-" if v is None else w.format(v)
+ rp = (pb / pa) if (pa and pb) else None
+ rd = (db / da) if (da and db) else None
+ if rp:
+ pr.append(rp)
+ if rd:
+ dr.append(rd)
+ print(f"| {c[0]} T={c[1]} n={c[2]} | {f(pa)} | {f(pb)} | {f(rp, '{:.3f}')}x | {f(da)} | {f(db)} | "
+ f"{f(rd, '{:.3f}')}x | {f(ta, '{:.2f}')} | {f(tb, '{:.2f}')} |")
+ gm = lambda xs: math.exp(sum(math.log(x) for x in xs) / len(xs)) if xs else float("nan")
+ print(f"\nprompt aggregate (geomean of ratios over {len(pr)} cells): {gm(pr):.3f}x")
+ print(f"decode aggregate (geomean of ratios over {len(dr)} cells): {gm(dr):.3f}x")
+ if pr:
+ print(f"lowest prompt ratio: {min(pr):.3f}x lowest decode ratio: {min(dr) if dr else float('nan'):.3f}x")
+ if ci > 0:
+ bs = bootstrap(a, b, ci)
+ print(f"\nbootstrap 90% intervals over {ci} resamples (5th .. 95th percentile of the ratio):")
+ for key, label in (("prompt_per_second", "prompt"), ("predicted_per_second", "decode")):
+ if key not in bs:
+ continue
+ for c, v in sorted(bs[key].items(), key=lambda kv: str(kv[0])):
+ if c == "aggregate":
+ continue
+ print(f" {label:6s} {c[0]:6d}/{c[1]}/{c[2]:4d}: {v[0]:.3f}x [{v[1]:.3f}, {v[2]:.3f}]")
+ agg = bs[key].get("aggregate")
+ if agg:
+ print(f" {label:6s} aggregate: {agg[0]:.3f}x [{agg[1]:.3f}, {agg[2]:.3f}]")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/dgx_concurrent.py b/tools/dgx_concurrent.py
new file mode 100644
index 000000000..08e2a1e61
--- /dev/null
+++ b/tools/dgx_concurrent.py
@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+"""Aggregate output: N concurrent clients against one server, measuring tokens per second summed over the clients. Run the
+same N against a second configuration to compare.
+
+ python tools/dgx_concurrent.py CONFIG OUT.json --clients 8 [--clients 16] [--prompt 2000] [--max-new 128]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import statistics
+import subprocess
+import sys
+import threading
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()
+
+
+def make_prompt(tokens: int) -> str:
+ body = []
+ n = int(tokens / 1.35)
+ for i in range(n):
+ body.append(WORDS[i % len(WORDS)])
+ return "Summarise the following log in two sentences.\n\n" + " ".join(body)
+
+
+def serve(cfg_path: str, port: int):
+ cfg = json.loads(Path(cfg_path).read_text())
+ cfg["port"] = port
+ cfg["log"] = f"/tmp/dgx-conc-{port}.log"
+ tmp = Path(f"/tmp/dgx-conc-{port}.json")
+ tmp.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(tmp), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ t0 = time.time()
+ while time.time() - t0 < 1800:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ return proc
+ except Exception:
+ pass
+ time.sleep(3)
+ raise SystemExit("server did not come up")
+
+
+def one(port: int, prompt: str, max_new: int, out: list, idx: int):
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t0 = time.time()
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ res = json.loads(r.read())
+ out[idx] = (time.time() - t0, res.get("timings", {}).get("predicted_n", 0))
+
+
+def run(cfg_path: str, port: int, clients: int, prompt: str, max_new: int) -> dict:
+ proc = serve(cfg_path, port)
+ try:
+ with urllib.request.urlopen(urllib.request.Request(
+ f"http://127.0.0.1:{port}/v1/chat/completions",
+ data=json.dumps({"messages": [{"role": "user", "content": "hi"}], "max_tokens": 8}).encode(),
+ headers={"Content-Type": "application/json"}), timeout=1800) as r:
+ r.read()
+ out: list = [None] * clients
+ t0 = time.time()
+ threads = [threading.Thread(target=one, args=(port, prompt, max_new, out, i)) for i in range(clients)]
+ for t in threads:
+ t.start()
+ for t in threads:
+ t.join()
+ wall = time.time() - t0
+ tokens = sum(n for _, n in out if n)
+ times = [d for d, _ in out if d]
+ return {"clients": clients, "wall_s": round(wall, 2), "tokens": tokens,
+ "aggregate_tok_s": round(tokens / wall, 2) if wall > 0 else None,
+ "median_latency_s": round(statistics.median(times), 2)}
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(180)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("config")
+ ap.add_argument("out")
+ ap.add_argument("--clients", type=int, nargs="+", default=[8, 16])
+ ap.add_argument("--prompt", type=int, default=2000)
+ ap.add_argument("--max-new", type=int, default=128)
+ ap.add_argument("--port", type=int, default=8091)
+ a = ap.parse_args()
+ prompt = make_prompt(a.prompt)
+ rows = [run(a.config, a.port, c, prompt, a.max_new) for c in a.clients]
+ Path(a.out).write_text(json.dumps(rows, indent=1))
+ for r in rows:
+ print(json.dumps(r))
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/dgx_kvresident.py b/tools/dgx_kvresident.py
new file mode 100644
index 000000000..aa60a825a
--- /dev/null
+++ b/tools/dgx_kvresident.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Is decode at 56K limited by KV streaming? Raise --kv-resident to the whole context if it fits, and re-measure the same
+56K prompt. Writes one row; compare it with the 31.0 tok/s of the shipped 32,768-cell setting.
+
+ python dgx_kvresident.py CONFIG OUT.json [--tokens 56000]
+"""
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()
+
+
+def build(tokens, code, rng):
+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]
+ at = rng.randrange(len(body) // 3, 2 * len(body) // 3)
+ body[at:at] = [f"The access code for the harbor gate is {code}."]
+ return ("Read the following log carefully, then answer the question at the end.\n\n" + " ".join(body) +
+ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")
+
+
+def main():
+ cfg_path, out_path = sys.argv[1], sys.argv[2]
+ tokens = int(sys.argv[3]) if len(sys.argv) > 3 else 56000
+ port = 8091
+ cfg = json.loads(Path(cfg_path).read_text())
+ cfg["port"] = port
+ cfg["log"] = "/tmp/kvresident.log"
+ tmp = Path("/tmp/kvresident.json")
+ tmp.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(tmp), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ t0 = time.time()
+ while time.time() - t0 < 1800:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited {proc.returncode} (see /tmp/kvresident.log)")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(3)
+ print(f"ready in {time.time() - t0:.0f} s", flush=True)
+ rng = random.Random(7)
+ code = "481516"
+ prompt = build(tokens, code, rng)
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 64, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t1 = time.time()
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ out = json.loads(r.read())
+ wall = time.time() - t1
+ t = out.get("timings") or {}
+ answer = out["choices"][0]["message"].get("content", "")
+ row = {"tokens": tokens, "kv_resident": cfg["args"][cfg["args"].index("--kv-resident") + 1],
+ "prompt_n": t.get("prompt_n"), "prompt_per_second": t.get("prompt_per_second"),
+ "predicted_n": t.get("predicted_n"), "predicted_per_second": t.get("predicted_per_second"),
+ "wall_s": round(wall, 1), "recalled": code in answer}
+ Path(out_path).write_text(json.dumps(row, indent=1))
+ print(json.dumps(row))
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(180)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/dgx_kvresident2.py b/tools/dgx_kvresident2.py
new file mode 100644
index 000000000..a2b98a1c6
--- /dev/null
+++ b/tools/dgx_kvresident2.py
@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""Clean long-context decode comparison: same 47K-token prompt, but a request that generates ~200 tokens in both cases, so
+the decode rate is not dominated by per-request overhead. Compares --kv-resident 32768 against 65536.
+
+ python dgx_kvresident2.py OUT.json [--tokens 47000]
+"""
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+root = Path.home() / "workspace" / "strata-spark"
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()
+
+
+def build(tokens, rng):
+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]
+ return ("Read the following log carefully.\n\n" + " ".join(body) +
+ "\n\nWrite a detailed 250-word analysis of the vocabulary and tone of the log above. Do not stop early.")
+
+
+def run(cfg_name, tag, port=8091):
+ cfg = json.loads((root / cfg_name).read_text())
+ cfg["port"] = port
+ cfg["log"] = f"/tmp/kvres2-{tag}.log"
+ tmp = Path(f"/tmp/kvres2-{tag}.json")
+ tmp.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(tmp), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ t0 = time.time()
+ while time.time() - t0 < 1800:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(3)
+ prompt = build(int(sys.argv[2]) if len(sys.argv) > 2 else 47000, random.Random(7))
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 256, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ out = json.loads(r.read())
+ t = out.get("timings") or {}
+ return {"kv_resident": cfg["args"][cfg["args"].index("--kv-resident") + 1],
+ "prompt_n": t.get("prompt_n"), "prompt_per_second": t.get("prompt_per_second"),
+ "predicted_n": t.get("predicted_n"), "predicted_per_second": t.get("predicted_per_second")}
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(180)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+
+
+rows = [run("strata-iq3_s.json", "a"), run("strata-iq3s-kv64k.json", "b")]
+Path(sys.argv[1]).write_text(json.dumps(rows, indent=1))
+for r in rows:
+ print(json.dumps(r))
diff --git a/tools/dgx_longctx.py b/tools/dgx_longctx.py
new file mode 100644
index 000000000..601d0b370
--- /dev/null
+++ b/tools/dgx_longctx.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Coverage past 32K: one 60K-token prompt on the 64K-context configuration, with a hidden code, so the run measures prompt
+throughput at that length and checks recall in the same request.
+
+ python long_context.py CONFIG OUT.json [--tokens 60000]
+"""
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "
+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()
+
+
+def build(tokens, code, rng):
+ body = [rng.choice(WORDS) for _ in range(int(tokens / 1.35))]
+ at = rng.randrange(len(body) // 3, 2 * len(body) // 3)
+ body[at:at] = [f"The access code for the harbor gate is {code}."]
+ return ("Read the following log carefully, then answer the question at the end.\n\n" + " ".join(body) +
+ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")
+
+
+def main():
+ cfg_path, out_path = sys.argv[1], sys.argv[2]
+ tokens = int(sys.argv[3]) if len(sys.argv) > 3 else 60000
+ port = 8091
+ cfg = json.loads(Path(cfg_path).read_text())
+ cfg["port"] = port
+ cfg["log"] = "/tmp/long-context.log"
+ tmp = Path("/tmp/long-context.json")
+ tmp.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(tmp), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ t0 = time.time()
+ while time.time() - t0 < 1800:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(3)
+ print(f"ready in {time.time() - t0:.0f} s", flush=True)
+ rng = random.Random(7)
+ code = "481516"
+ prompt = build(tokens, code, rng)
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": 32, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t1 = time.time()
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ out = json.loads(r.read())
+ wall = time.time() - t1
+ t = out.get("timings") or {}
+ answer = out["choices"][0]["message"].get("content", "")
+ row = {"tokens": tokens, "prompt_n": t.get("prompt_n"), "prompt_per_second": t.get("prompt_per_second"),
+ "predicted_per_second": t.get("predicted_per_second"), "wall_s": round(wall, 1),
+ "answer": answer[:80], "recalled": code in answer}
+ Path(out_path).write_text(json.dumps(row, indent=1))
+ print(json.dumps(row))
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(180)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/dgx_needle.py b/tools/dgx_needle.py
new file mode 100644
index 000000000..024b48d75
--- /dev/null
+++ b/tools/dgx_needle.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Long-context recall check: hide one fact in a long prompt and ask for it back (the guide's "long-context retrieval" cell).
+
+ python tools/dgx_needle.py --config strata-iq3s-final.json --tokens 12000
+
+Starts the model's server, sends one request, and reports whether the answer contains the fact. The fact is placed at a
+random position in the prompt and never appears in the question, so a correct answer means the context was read.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "
+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()
+
+
+def build(tokens: int, needle: str, rng: random.Random) -> str:
+ words = max(8, int(tokens / 1.35))
+ body = [rng.choice(WORDS) for _ in range(words)]
+ at = rng.randrange(len(body) // 4, 3 * len(body) // 4)
+ body[at:at] = [needle]
+ return ("Read the following notes carefully, then answer the question at the end.\n\n" + " ".join(body) +
+ "\n\nQuestion: what is the access code for the harbor gate? Answer with the code only.")
+
+
+def post(port: int, prompt: str, max_tokens: int = 32) -> dict:
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ try:
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ return json.loads(r.read())
+ except urllib.error.HTTPError as e:
+ raise SystemExit(f"HTTP {e.code}: {e.read()[:400]!r}")
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--config", required=True)
+ ap.add_argument("--tokens", type=int, default=12000)
+ ap.add_argument("--port", type=int, default=8091)
+ ap.add_argument("--runs", type=int, default=3)
+ a = ap.parse_args()
+ cfg = json.loads(Path(a.config).read_text())
+ cfg["port"] = a.port
+ cfg["log"] = "/tmp/dgx-needle.log"
+ cpath = Path("/tmp/dgx-needle.json")
+ cpath.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(cpath), "--port", str(a.port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ ok = 0
+ try:
+ for _ in range(600):
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{a.port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(2)
+ post(a.port, "Say hi.", 8)
+ for i in range(a.runs):
+ rng = random.Random(1000 + i)
+ code = f"{rng.randrange(100000, 999999)}"
+ needle = f"The access code for the harbor gate is {code}."
+ r = post(a.port, build(a.tokens, needle, rng))
+ answer = r["choices"][0]["message"].get("content", "")
+ t = r.get("timings") or {}
+ hit = code in answer
+ ok += hit
+ print(f"run {i}: prompt_n {t.get('prompt_n')} at {t.get('prompt_per_second')} tok/s, "
+ f"answer {answer[:40]!r} -> {'HIT' if hit else 'MISS'} (code {code})")
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ print(f"{ok} of {a.runs} recalled")
+ return 0 if ok == a.runs else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/dgx_quality.py b/tools/dgx_quality.py
new file mode 100644
index 000000000..74d6818d0
--- /dev/null
+++ b/tools/dgx_quality.py
@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+"""Quality check for a numerically different path (the fused IQ prompt kernels): run the same greedy prompts through
+two configs and compare the answers.
+
+ python quality_fused.py strata-iq3s-nofused.json strata-iq3s-best.json out.json
+"""
+import json
+import os
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+PROMPTS = [
+ "Write a Python function that merges two sorted lists into one sorted list, with a docstring and two tests.",
+ "Explain in two short paragraphs how a heat pump moves heat from outside to inside a house.",
+ "A train leaves at 09:15 and travels 210 km at 84 km/h. At what time does it arrive? Show the arithmetic.",
+ "Summarize the main trade-offs between 4-bit and 8-bit weight quantization for LLM inference.",
+ "List five concrete steps to debug a CUDA kernel that returns wrong results, one sentence each.",
+]
+
+
+def post(port, prompt, max_tokens=160):
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens,
+ "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req, timeout=1800) as r:
+ out = json.loads(r.read())
+ return out["choices"][0]["message"].get("content", ""), out.get("timings") or {}
+
+
+def run(cfg, port, tag):
+ cpath = Path(f"/tmp/quality-{tag}.json")
+ c = json.loads(Path(cfg).read_text())
+ c["port"] = port
+ c["log"] = f"/tmp/quality-{tag}.log"
+ cpath.write_text(json.dumps(c, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(cpath), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ for _ in range(600):
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(2)
+ post(port, "Say hi.", 8)
+ out = []
+ for p in PROMPTS:
+ txt, t = post(port, p)
+ out.append(txt)
+ print(f" {tag}: {t.get('prompt_per_second')} tok/s prompt, {t.get('predicted_per_second')} tok/s decode")
+ return out
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ time.sleep(5)
+
+
+a = run(sys.argv[1], 8097, "a")
+b = run(sys.argv[2], 8097, "b")
+res = []
+same = 0
+for i, (x, y) in enumerate(zip(a, b)):
+ eq = x == y
+ same += eq
+ pre = 0
+ while pre < min(len(x), len(y)) and x[pre] == y[pre]:
+ pre += 1
+ res.append({"prompt": PROMPTS[i], "identical": eq, "common_prefix_chars": pre,
+ "len_a": len(x), "len_b": len(y), "a": x, "b": y})
+ print(f"prompt {i}: {'IDENTICAL' if eq else 'differs'} (prefix {pre} chars, {len(x)} vs {len(y)} chars)")
+Path(sys.argv[3]).write_text(json.dumps({"identical": same, "of": len(PROMPTS), "items": res}, indent=1))
+print(f"{same} of {len(PROMPTS)} answers identical")
diff --git a/tools/dgx_quality24.py b/tools/dgx_quality24.py
new file mode 100644
index 000000000..e9df7ba5e
--- /dev/null
+++ b/tools/dgx_quality24.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""A wider identity gate than tools/dgx_quality.py's five: 24 prompts across code, prose, math, multilingual and tool-shaped
+asks, greedy, comparing the fused prompt path against MMQ byte for byte.
+
+ python quality24.py CONFIG_A CONFIG_B OUT.json [--port 8091]
+"""
+import json
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+PROMPTS = [
+ "Write a Python function that returns the longest palindromic substring of a string, with a docstring.",
+ "In C, what does `static inline` mean at file scope and when should it be used? Two sentences.",
+ "Explain the difference between a mutex and a semaphore to a junior developer, in under 100 words.",
+ "Translate 'The server is down and we are investigating' into German, French and Spanish, one line each.",
+ "Summarise the plot of Pride and Prejudice in exactly three sentences.",
+ "What is the time complexity of building a binary heap, and why?",
+ "Write a SQL query that returns the five customers with the highest total order value.",
+ "List four failure modes of a distributed lock and one mitigation each, as a table.",
+ "给一个刚学编程的人解释什么是递归,100字以内。",
+ "Erkläre den Unterschied zwischen TCP und UDP in drei Sätzen.",
+ "Écris une fonction JavaScript qui inverse une chaîne sans utiliser reverse().",
+ "What happens in a Linux system when you run `rm -rf /` as root? Be technical and brief.",
+ "Give me a shell one-liner that finds the ten largest files under /var/log.",
+ "Why does a transformer need positional encodings? Answer in two sentences.",
+ "Write a haiku about a GPU fan spinning up.",
+ "What is the difference between `git rebase` and `git merge` for a shared branch? Two sentences.",
+ "Explain Bloom filters and give one production use case, under 80 words.",
+ "Sort these numbers ascending and show the result: 42, 7, 19, 3, 88, 15.",
+ "What does the error `CUDA error: an illegal memory access was encountered` usually mean?",
+ "Describe how to measure the power draw of a single process on Linux.",
+ "Write a regex that matches ISO-8601 dates and explain each part.",
+ "What is the difference between a process and a thread in one sentence?",
+ "Give me a JSON object with three fields describing a book, and no prose.",
+ "Explain why 0.1 + 0.2 != 0.3 in floating point, briefly.",
+]
+
+
+def wait(port, proc):
+ t0 = time.time()
+ while time.time() - t0 < 900:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ return
+ except Exception:
+ pass
+ time.sleep(3)
+ raise SystemExit("server did not come up")
+
+
+def ask(port, prompt, max_new=96):
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req, timeout=1800) as r:
+ out = json.loads(r.read())
+ return out["choices"][0]["message"].get("content", "")
+
+
+def run_config(cfg_path, port, tag):
+ cfg = json.loads(Path(cfg_path).read_text())
+ cfg["port"] = port
+ cfg["log"] = f"/tmp/quality24-{tag}.log"
+ tmp = Path(f"/tmp/quality24-{tag}.json")
+ tmp.write_text(json.dumps(cfg, indent=1))
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(tmp), "--port", str(port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ answers = []
+ try:
+ wait(port, proc)
+ ask(port, "Say hi.", 8)
+ for p in PROMPTS:
+ answers.append(ask(port, p))
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(120)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ return answers
+
+
+def main():
+ a_cfg, b_cfg, out_json = sys.argv[1], sys.argv[2], sys.argv[3]
+ port = int(sys.argv[4]) if len(sys.argv) > 4 else 8091
+ a = run_config(a_cfg, port, "a")
+ b = run_config(b_cfg, port, "b")
+ same = sum(1 for x, y in zip(a, b) if x == y)
+ rows = [{"prompt": p, "a": x, "b": y, "identical": x == y} for p, x, y in zip(PROMPTS, a, b)]
+ Path(out_json).write_text(json.dumps({"identical": same, "total": len(PROMPTS), "rows": rows}, indent=1))
+ print(f"{same} of {len(PROMPTS)} answers identical")
+ for i, r in enumerate(rows):
+ if not r["identical"]:
+ print(f" differs at prompt {i}: {r['prompt'][:60]!r}")
+ print(f" A: {r['a'][:90]!r}")
+ print(f" B: {r['b'][:90]!r}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/dgx_smoke.py b/tools/dgx_smoke.py
new file mode 100644
index 000000000..4daa83a01
--- /dev/null
+++ b/tools/dgx_smoke.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""The drop's acceptance gate in one command: setup detection, one benchmark cell, the identity gate and a needle, each
+with a band that the numbers on a DGX Spark should land in. Exit code 0 = all passed.
+
+ python tools/dgx_smoke.py [--config strata-iq3_s.json] [--tokens 12000] [--data-dir ~/Strata-data]
+
+Bands are deliberately loose (they must survive page-cache state and drift); a failure means something is wrong, not that
+the machine is 5% slower than the number in the documents.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+PY = sys.executable
+WORK = Path(tempfile.mkdtemp(prefix="dgx-smoke-")) # this run's own scratch folder (never a fixed /tmp name another run could share)
+
+
+def sh(cmd: str) -> tuple[int, str]:
+ p = subprocess.run(cmd, shell=True, cwd=str(ROOT), capture_output=True, text=True, timeout=3600)
+ return p.returncode, (p.stdout or "") + (p.stderr or "")
+
+
+def check_setup(data_dir: str) -> tuple[bool, str]:
+ rc, out = sh(f"{PY} setup.py --check --yes --data-dir '{data_dir}' 2>&1 | head -8")
+ ok = "unified memory" in out and "arm64" in out
+ return ok, out.strip().splitlines()[0] if out.strip() else "no output"
+
+
+def bench(config: str, cell: str, out_dir: str) -> dict | None:
+ rc, out = sh(f"{PY} tools/dgx_bench.py --config {config} --tag smoke --cells {cell} --rounds 1 --port 8093 "
+ f"--out {out_dir} 2>&1")
+ m = re.search(r"prompt_n\s+(\d+)\s+prompt\s+([0-9.]+)\s+tok/s\s+decode\s+([0-9.]+)", out)
+ if not m:
+ return None
+ return {"prompt_n": int(m.group(1)), "prompt": float(m.group(2)), "decode": float(m.group(3))}
+
+
+def identity(config: str) -> tuple[bool, str]:
+ base = json.loads((ROOT / config).read_text())
+ off = dict(base)
+ off["args"] = list(base["args"])
+ env = dict(base.get("env") or {})
+ env["STRATA_PF_FUSED"] = "0"
+ off["env"] = env
+ off_path = WORK / "nofused.json"
+ off_path.write_text(json.dumps(off, indent=1))
+ rc, out = sh(f"{PY} tools/dgx_quality.py {off_path} {config} {WORK / 'quality.json'} 2>&1")
+ m = re.search(r"(\d+) of (\d+) answers identical", out)
+ if not m:
+ return False, out.strip()[-200:]
+ return m.group(1) == m.group(2), out.strip().splitlines()[-1]
+
+
+def needle(config: str, tokens: int) -> tuple[bool, str]:
+ rc, out = sh(f"{PY} tools/dgx_needle.py --config {config} --tokens {tokens} --runs 2 2>&1")
+ m = re.search(r"(\d+) of (\d+) recalled", out)
+ if not m:
+ return False, out.strip()[-200:]
+ return m.group(1) == m.group(2), out.strip().splitlines()[-1]
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--config", default="strata-iq3_s.json")
+ ap.add_argument("--cell", default="4096:0:256")
+ ap.add_argument("--tokens", type=int, default=12000)
+ ap.add_argument("--data-dir", default=os.environ.get("STRATA_DATA_DIR") or str(Path.home() / "Strata-data"),
+ help="the data folder setup uses (default: $STRATA_DATA_DIR or ~/Strata-data)")
+ a = ap.parse_args()
+ results = []
+ t0 = time.time()
+
+ ok, detail = check_setup(a.data_dir)
+ results.append(("setup detection", ok, detail))
+
+ b = bench(a.config, a.cell, str(WORK / "bench"))
+ if b is None:
+ results.append(("bench cell", False, "no parsable cell"))
+ else:
+ ok = b["prompt"] > 800 and b["decode"] > 30
+ results.append(("bench cell", ok, f"{b['prompt']:.0f} prompt / {b['decode']:.1f} decode tok/s"))
+
+ ok, detail = identity(a.config)
+ results.append(("identity, fused vs MMQ", ok, detail))
+
+ ok, detail = needle(a.config, a.tokens)
+ results.append(("needle, recall", ok, detail))
+
+ print()
+ width = max(len(n) for n, _, _ in results)
+ for name, ok, detail in results:
+ print(f"{'PASS' if ok else 'FAIL'} {name:<{width}} {detail}")
+ bad = [n for n, ok, _ in results if not ok]
+ print(f"\n{len(results) - len(bad)} of {len(results)} passed in {time.time() - t0:.0f} s"
+ + (f"; failed: {', '.join(bad)}" if bad else ""))
+ return 0 if not bad else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/dgx_turns.py b/tools/dgx_turns.py
new file mode 100644
index 000000000..351f0faaa
--- /dev/null
+++ b/tools/dgx_turns.py
@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""Multi-turn (agent-style) comparison: one conversation, five turns of ~1,200 new tokens each, cache reuse measured.
+
+ python tools/dgx_turns.py CONFIG TAG OUTDIR [--turns 5] [--tokens 1200] [--port 8091]
+
+Per turn it records the engine's prompt_n / cache_n, prompt tok/s and TTFT, so the cache path is compared, not just the
+fresh-prompt path the other tools measure.
+"""
+import argparse
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel "
+ "lattice mirror nectar oasis pepper quarry raven saddle tundra umber vessel willow xenon yellow zephyr").split()
+
+
+def blob(n, seed):
+ rng = random.Random(seed)
+ return " ".join(rng.choice(WORDS) for _ in range(n))
+
+
+def chat(port, messages, max_tokens=64):
+ body = json.dumps({"messages": messages, "max_tokens": max_tokens, "temperature": 0, "stream": False,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t0 = time.time()
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ out = json.loads(r.read())
+ return out, time.time() - t0
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("config")
+ ap.add_argument("tag")
+ ap.add_argument("outdir")
+ ap.add_argument("--turns", type=int, default=5)
+ ap.add_argument("--tokens", type=int, default=1200)
+ ap.add_argument("--port", type=int, default=8091)
+ a = ap.parse_args()
+ out = Path(a.outdir)
+ out.mkdir(parents=True, exist_ok=True)
+ cfg = json.loads(Path(a.config).read_text())
+ cfg["port"] = a.port
+ cfg["log"] = str(out / f"{a.tag}.engine.log")
+ cpath = out / f"{a.tag}.config.json"
+ cpath.write_text(json.dumps(cfg, indent=1))
+ log = open(out / f"{a.tag}.jsonl", "a", encoding="utf-8")
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(cpath), "--port", str(a.port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ t0 = time.time()
+ while time.time() - t0 < 3600:
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{a.port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(3)
+ print(f"{a.tag}: ready in {time.time() - t0:.0f} s", flush=True)
+ msgs = [{"role": "user", "content": "Read this note and answer with one short sentence.\n" + blob(a.tokens, 1)}]
+ outj, wall = chat(a.port, msgs)
+ msgs.append({"role": "assistant", "content": outj["choices"][0]["message"].get("content", "")})
+ for turn in range(a.turns):
+ msgs.append({"role": "user", "content": "More notes.\n" + blob(a.tokens, 100 + turn) +
+ "\nSummarise the last note in one sentence."})
+ outj, wall = chat(a.port, msgs)
+ t = outj.get("timings") or {}
+ msgs.append({"role": "assistant", "content": outj["choices"][0]["message"].get("content", "")})
+ row = {"round": turn, "arm": a.tag, "len": 0, "temp": 0.0, "max_new": 64, "wall_s": round(wall, 2),
+ "ttft_s": (t.get("prompt_ms") or 0) / 1000.0 or None, **t}
+ log.write(json.dumps(row) + "\n")
+ log.flush()
+ print(f" turn {turn}: prompt_n {t.get('prompt_n')} cache_n {t.get('cache_n')} "
+ f"prompt {t.get('prompt_per_second')} tok/s decode {t.get('predicted_per_second')} tok/s "
+ f"ttft {row['ttft_s']}", flush=True)
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(90)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ log.close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/dgx_warm.py b/tools/dgx_warm.py
new file mode 100644
index 000000000..91ef57700
--- /dev/null
+++ b/tools/dgx_warm.py
@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+"""One server, several repeats per cell: the warm-state runner (the per-start page cache and the tens-of-GB expert
+prefetch are the main variance sources on a unified-memory box, so repeats inside one server are the comparable ones).
+
+ python tools/dgx_warm.py CONFIG TAG OUTDIR [--cells 4096,16384,32768] [--repeats 3] [--port 8091] [--new 256]
+
+Writes <OUTDIR>/<TAG>.jsonl rows in tools/dgx_bench.py's schema (round, arm, len, temp, max_new, prompt_n,
+prompt_per_second, predicted_n, predicted_per_second, ttft_s, cache_n) so tools/dgx_compare.py reads them.
+"""
+import argparse
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+WORDS = ("river stone market lantern copper winter garden signal harbor needle orbit velvet canyon ember ledger quartz "
+ "meadow anchor violet thunder cedar marble piston ribbon cobalt falcon glacier hollow ivory juniper kernel").split()
+
+
+def prompt_for(tokens, nonce):
+ rng = random.Random(tokens)
+ body = " ".join(rng.choice(WORDS) for _ in range(max(8, int(tokens / 1.35))))
+ return (f"Request nonce {nonce}. Read the word list below carefully.\n\n{body}\n\n"
+ "Write a long, detailed analysis of the word list above. Keep writing for at least 400 words.")
+
+
+def post(port, prompt, max_new):
+ body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_new, "temperature": 0,
+ "stream": False, "chat_template_kwargs": {"enable_thinking": False}}).encode()
+ req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=body,
+ headers={"Content-Type": "application/json"})
+ t0 = time.time()
+ with urllib.request.urlopen(req, timeout=3600) as r:
+ out = json.loads(r.read())
+ return out, time.time() - t0
+
+
+def status(port):
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/status", timeout=10) as r:
+ return json.loads(r.read()).get("last_timings") or {}
+ except Exception:
+ return {}
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("config")
+ ap.add_argument("tag")
+ ap.add_argument("outdir")
+ ap.add_argument("--cells", default="512,2048,4096")
+ ap.add_argument("--repeats", type=int, default=4)
+ ap.add_argument("--port", type=int, default=8091)
+ ap.add_argument("--new", type=int, default=256)
+ a = ap.parse_args()
+ out = Path(a.outdir)
+ out.mkdir(parents=True, exist_ok=True)
+ cfg = json.loads(Path(a.config).read_text())
+ cfg["port"] = a.port
+ cfg["log"] = str(out / f"{a.tag}.engine.log")
+ cpath = out / f"{a.tag}.config.json"
+ cpath.write_text(json.dumps(cfg, indent=1))
+ log = open(out / f"{a.tag}.jsonl", "a", encoding="utf-8")
+ proc = subprocess.Popen([sys.executable, str(ROOT / "serve" / "server.py"), "--engine", "strata", "--config",
+ str(cpath), "--port", str(a.port)], cwd=str(ROOT),
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ t0 = time.time()
+ while time.time() - t0 < 3600:
+ if proc.poll() is not None:
+ raise SystemExit(f"server exited with {proc.returncode}")
+ try:
+ with urllib.request.urlopen(f"http://127.0.0.1:{a.port}/v1/models", timeout=5) as r:
+ if r.status == 200:
+ break
+ except Exception:
+ pass
+ time.sleep(3)
+ print(f"{a.tag}: ready in {time.time() - t0:.0f} s", flush=True)
+ post(a.port, prompt_for(256, "warm"), 8)
+ for rep in range(a.repeats):
+ for length in [int(x) for x in a.cells.split(",")]:
+ nonce = f"{random.getrandbits(48):012x}"
+ outj, wall = post(a.port, prompt_for(length, nonce), a.new)
+ t = status(a.port)
+ row = {"round": rep, "arm": a.tag, "len": length, "temp": 0.0, "max_new": a.new,
+ "wall_s": round(wall, 2), "ttft_s": (t.get("prompt_ms") or 0) / 1000.0 or None,
+ "cache_n": t.get("cache_n", 0), **t}
+ log.write(json.dumps(row) + "\n")
+ log.flush()
+ print(f" {length:5d} prompt_n {t.get('prompt_n')} prompt {t.get('prompt_per_second')} tok/s "
+ f"decode {t.get('predicted_per_second')} tok/s", flush=True)
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(90)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ log.close()
+
+
+if __name__ == "__main__":
+ main()