Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
bench: say what the short prompt and the output cap mean; measured duration replaces the 20-30 minute guess
Niko1221f5410a6feba8
diff --git a/BENCH.bat b/BENCH.batindex a5ba90058..f54bc90dc 100644--- a/BENCH.bat+++ b/BENCH.bat@@ -11,7 +11,7 @@ if not exist ".venv\Scripts\python.exe" (pauseexit /b 1)-echo Quick speed check. For results other people can compare with (the standard benchmark, 20 to 30 minutes), run: .venv\Scripts\python.exe strata.py bench run+echo Quick speed check. For results other people can compare with (the standard benchmark, a few minutes), run: .venv\Scripts\python.exe strata.py bench runecho.".venv\Scripts\python.exe" tools\strata_bench.py %*set "RC=%ERRORLEVEL%"diff --git a/bench.sh b/bench.shindex 3cb68d1d2..1d76b4f40 100755--- a/bench.sh+++ b/bench.sh@@ -5,7 +5,7 @@# This is the quick speed check. The standard benchmark (docs/BENCHMARK_PROTOCOL.md), whose results other people can compare with, is `strata bench run`.cd "$(dirname "$0")" || exit 1if [ -x .venv/bin/python ]; then- echo "Quick speed check. For results other people can compare with (the standard benchmark, 20 to 30 minutes), run: .venv/bin/python strata.py bench run"+ echo "Quick speed check. For results other people can compare with (the standard benchmark, a few minutes), run: .venv/bin/python strata.py bench run"echoexec .venv/bin/python tools/strata_bench.py "$@"fidiff --git a/docs/BENCHMARK_PROTOCOL.md b/docs/BENCHMARK_PROTOCOL.mdindex 03f2c8ca9..595699689 100644--- a/docs/BENCHMARK_PROTOCOL.md+++ b/docs/BENCHMARK_PROTOCOL.md@@ -4,9 +4,11 @@Result format: `strata-standard-benchmark`, schema version `1`.This document specifies a repeatable community benchmark and the data a future website can consume. The-required suite targets **20–30 minutes after model loading**. This duration has not been measured for this-protocol. Faster machines may finish sooner and slower machines may take longer; everyone runs the same work.-Downloads, installation, compilation, and model loading are excluded. Report their duration separately if measured.+required suite took **2 to 4 minutes after model loading** on the two machines it has been run on (RTX 3060 and+2x RTX A4000, IQ3_XXS, Strata 0.2.1: 3 min 22 s and 2 min 22 s; the optional concurrency stage added 1.5 to 2 minutes).+Slower machines take longer; everyone runs the same work. Downloads, installation, compilation, and model loading are+excluded; so is the hashing of the model files that the runner does first (about 1 to 5 minutes for a 44 to 71 GB model).+Report their duration separately if measured.`strata bench run` implements this protocol from Strata 0.2.1. It sends the frozen fixtures in`bench/prompts/standard-v1` (corpus SHA-256 `e75eff8b9feea04f745520b392c4c17d97aefc1ba185e222aa6740efa8f68407`)@@ -210,6 +212,12 @@ reading needs a new protocol ID.A different tokenizer or template breaks that and makes the run `nonconforming`.- **Truncation.** `finish_reason: "length"` is `truncated`. It stays in the speed summaries and fails a sanity case.Any other finish reason except `stop` and `tool_calls` is a failure.+- **Reading it out.** On the first real runs (RTX 3060, 2x RTX A4000) every 4K and 16K performance request stopped at the+ 512-token cap while still reasoning, and so did one of three `short` requests. Their total time is the time to the cap, and+ they have no first-answer time. The plain summary and the README therefore carry a `Capped` column (how many of a case's+ requests reached the cap). The `short` prompt is about 80 tokens, so its prompt rate (130 to 190 tok/s there) measures the+ fixed cost of a request and is left out of those two readouts; `results.json` keeps it. Changing the cap or the prompts+ would be a new protocol ID.- **Numbers.** JSON equality compares types: `3` is not `3.0`, and `true` is not `1`. A tool case needs exactly onestructured call; text next to it is ignored.- **After a failure.** The runner asks `/health` once. If the server does not answer, the remaining requests arediff --git a/docs/COMMUNITY_BENCHMARKS.md b/docs/COMMUNITY_BENCHMARKS.mdindex b08cffe2e..1c518d6b5 100644--- a/docs/COMMUNITY_BENCHMARKS.md+++ b/docs/COMMUNITY_BENCHMARKS.md@@ -5,7 +5,7 @@ might run on a similar machine. This guide suggests a report format and a pullrequest workflow; a single model, GPU, or context length is enough to contribute.There are two ways to measure. For results other people can compare with, run the **standard benchmark**: `strata bench run`follows the draft protocol in [BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md) (fixed prompts, thinking on, the model card's-sampling, twelve sanity checks, about 20 to 30 minutes after the model has loaded) and writes `results.json` with+sampling, twelve sanity checks, about 2 to 4 minutes after the model has loaded on a fast card, more on a slow one) and writes `results.json` withthe hardware, versions and every request. For a quick look at your own machine, [`BENCH.bat` / `./bench.sh`](#run-the-benchmark)takes 3 to 10 minutes and writes a report in the format below.diff --git a/strata_cli/bench_standard.py b/strata_cli/bench_standard.pyindex 7c6bbdc59..9ee5fb351 100644--- a/strata_cli/bench_standard.py+++ b/strata_cli/bench_standard.py@@ -1076,14 +1076,32 @@ def _minutes(seconds: float | None) -> str:return f"{int(seconds // 60)} min {int(seconds % 60)} s"+SHORT_PROMPT_NOTE = ("short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not "+ "shown; read prompt speed from prompt-4k and prompt-16k")+CAP_NOTE = ("Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the "+ "time to the cap, not to an answer, and it has no first-answer time when it was still reasoning")+++def capped(doc: dict, phase: str, case: str) -> str:+ """'k/n': how many of the case's measured requests stopped at the output cap."""+ rows = [r for r in doc["requests"] if r["phase"] == phase and r["case_id"] == case]+ return f"{sum(1 for r in rows if r['status'] == 'truncated')}/{len(rows)}"+++def prompt_cell(doc: dict, phase: str, case: str, digits: int = 1) -> str:+ return "n/a" if case == "short" else _span(_find(doc, phase, case, "prompt_tps"), digits)++def table_lines(doc: dict, phase: str, cases: tuple[str, ...]) -> list[str]:- head = ["Case", "Prompt tok/s", "Decode tok/s", "TTFT s", "Total s"]+ head = ["Case", "Prompt tok/s", "Decode tok/s", "TTFT s", "Total s", "Capped"]rows = [head]for case in cases:- rows.append([case, _span(_find(doc, phase, case, "prompt_tps")), _span(_find(doc, phase, case, "decode_tps")),- _span(_find(doc, phase, case, "ttft_s"), 2), _span(_find(doc, phase, case, "total_s"), 1)])+ rows.append([case, prompt_cell(doc, phase, case), _span(_find(doc, phase, case, "decode_tps")),+ _span(_find(doc, phase, case, "ttft_s"), 2), _span(_find(doc, phase, case, "total_s"), 1),+ capped(doc, phase, case)])widths = [max(len(r[i]) for r in rows) for i in range(len(head))]- return [" " + " ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)).rstrip() for row in rows]+ return ([" " + " ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)).rstrip() for row in rows]+ + [" " + SHORT_PROMPT_NOTE + ".", " " + CAP_NOTE + "."])def format_plain(doc: dict) -> str:@@ -1167,12 +1185,13 @@ def report_markdown(doc: dict, redact=lambda text: text) -> str:"monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. "f"Time after the model loaded: {_minutes(run_block['required_duration_s'])}.", "","## Results", ""]- lines += ["| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s |", "| --- | --- | --- | --- | --- | --- |"]+ lines += ["| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |", "| --- | --- | --- | --- | --- | --- | --- |"]for case in PERFORMANCE_CASES:- lines.append(f"| {case} | " + " | ".join(- _span(_find(doc, "performance", case, m), d) for m, d in (("prompt_tps", 1), ("decode_tps", 1), ("ttft_s", 2),- ("first_answer_s", 2), ("total_s", 1))) + " |")- lines += ["", "Cells are median (min-max) n=samples.", "",+ lines.append(f"| {case} | " + prompt_cell(doc, "performance", case) + " | " + " | ".join(+ _span(_find(doc, "performance", case, m), d) for m, d in (("decode_tps", 1), ("ttft_s", 2),+ ("first_answer_s", 2), ("total_s", 1)))+ + f" | {capped(doc, 'performance', case)} |")+ lines += ["", "Cells are median (min-max) n=samples. " + SHORT_PROMPT_NOTE[0].upper() + SHORT_PROMPT_NOTE[1:] + ". " + CAP_NOTE + ".", "","| Phase | Planned | Completed | Truncated | Failed | Skipped |", "| --- | ---: | ---: | ---: | ---: | ---: |"]for phase in PHASES:c = counts[phase]diff --git a/tests/terminal/test_bench_standard.py b/tests/terminal/test_bench_standard.pyindex 3a5350c6f..bce01fa08 100644--- a/tests/terminal/test_bench_standard.py+++ b/tests/terminal/test_bench_standard.py@@ -705,6 +705,24 @@ class RunTests(StandardBase):self.assertIn("Prompt tok/s", text)self.assertIn("1000.0 (1000.0-1000.0) n=3", text)+ def test_the_summary_says_what_the_short_prompt_and_the_cap_mean(self):+ doc, _ = self.run_standard()+ for text in (bs.format_plain(doc), bs.report_markdown(doc)):+ self.assertIn("Capped", text)+ self.assertIn("not shown; read prompt speed from prompt-4k and prompt-16k", text)+ self.assertIn("time to the cap, not to an answer", text)+ short_row = next(line for line in bs.format_plain(doc).splitlines() if line.strip().startswith("short "))+ self.assertIn("n/a", short_row)+ self.assertNotIn("1000.0", short_row)+ self.assertIn("0/3", short_row)+ # the data keeps the short case's prompt rate: only the readable summaries leave it out+ self.assertIsNotNone(self.summary(doc, "short", "prompt_tps"))+ # a request that stopped at the cap is counted+ record = next(r for r in doc["requests"] if r["id"] == "performance-prompt-4k-r1")+ record["status"] = "truncated"+ row = next(line for line in bs.format_plain(doc).splitlines() if line.strip().startswith("prompt-4k "))+ self.assertTrue(row.rstrip().endswith("1/3"))+def test_the_server_is_not_asked_for_anything_outside_the_protocol(self):self.run_standard()self.assertTrue(all(body["messages"] == corpus_specs()[rid]["messages"] for body, rid in zip(self.server.bodies, self.server.order)))