Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

bench: say what the short prompt and the output cap mean; measured duration replaces the 20-30 minute guess

Niko1221f5410a6feba8
diff --git a/BENCH.bat b/BENCH.bat
index a5ba90058..f54bc90dc 100644
--- a/BENCH.bat
+++ b/BENCH.bat
@@ -11,7 +11,7 @@ if not exist ".venv\Scripts\python.exe" (
pause
exit /b 1
)
-echo Quick speed check. For results other people can compare with (the standard benchmark, 20 to 30 minutes), run: .venv\Scripts\python.exe strata.py bench run
+echo Quick speed check. For results other people can compare with (the standard benchmark, a few minutes), run: .venv\Scripts\python.exe strata.py bench run
echo.
".venv\Scripts\python.exe" tools\strata_bench.py %*
set "RC=%ERRORLEVEL%"
diff --git a/bench.sh b/bench.sh
index 3cb68d1d2..1d76b4f40 100755
--- a/bench.sh
+++ b/bench.sh
@@ -5,7 +5,7 @@
# This is the quick speed check. The standard benchmark (docs/BENCHMARK_PROTOCOL.md), whose results other people can compare with, is `strata bench run`.
cd "$(dirname "$0")" || exit 1
if [ -x .venv/bin/python ]; then
- echo "Quick speed check. For results other people can compare with (the standard benchmark, 20 to 30 minutes), run: .venv/bin/python strata.py bench run"
+ echo "Quick speed check. For results other people can compare with (the standard benchmark, a few minutes), run: .venv/bin/python strata.py bench run"
echo
exec .venv/bin/python tools/strata_bench.py "$@"
fi
diff --git a/docs/BENCHMARK_PROTOCOL.md b/docs/BENCHMARK_PROTOCOL.md
index 03f2c8ca9..595699689 100644
--- a/docs/BENCHMARK_PROTOCOL.md
+++ b/docs/BENCHMARK_PROTOCOL.md
@@ -4,9 +4,11 @@
Result format: `strata-standard-benchmark`, schema version `1`.
This document specifies a repeatable community benchmark and the data a future website can consume. The
-required suite targets **20–30 minutes after model loading**. This duration has not been measured for this
-protocol. Faster machines may finish sooner and slower machines may take longer; everyone runs the same work.
-Downloads, installation, compilation, and model loading are excluded. Report their duration separately if measured.
+required suite took **2 to 4 minutes after model loading** on the two machines it has been run on (RTX 3060 and
+2x RTX A4000, IQ3_XXS, Strata 0.2.1: 3 min 22 s and 2 min 22 s; the optional concurrency stage added 1.5 to 2 minutes).
+Slower machines take longer; everyone runs the same work. Downloads, installation, compilation, and model loading are
+excluded; so is the hashing of the model files that the runner does first (about 1 to 5 minutes for a 44 to 71 GB model).
+Report their duration separately if measured.
`strata bench run` implements this protocol from Strata 0.2.1. It sends the frozen fixtures in
`bench/prompts/standard-v1` (corpus SHA-256 `e75eff8b9feea04f745520b392c4c17d97aefc1ba185e222aa6740efa8f68407`)
@@ -210,6 +212,12 @@ reading needs a new protocol ID.
A different tokenizer or template breaks that and makes the run `nonconforming`.
- **Truncation.** `finish_reason: "length"` is `truncated`. It stays in the speed summaries and fails a sanity case.
Any other finish reason except `stop` and `tool_calls` is a failure.
+- **Reading it out.** On the first real runs (RTX 3060, 2x RTX A4000) every 4K and 16K performance request stopped at the
+ 512-token cap while still reasoning, and so did one of three `short` requests. Their total time is the time to the cap, and
+ they have no first-answer time. The plain summary and the README therefore carry a `Capped` column (how many of a case's
+ requests reached the cap). The `short` prompt is about 80 tokens, so its prompt rate (130 to 190 tok/s there) measures the
+ fixed cost of a request and is left out of those two readouts; `results.json` keeps it. Changing the cap or the prompts
+ would be a new protocol ID.
- **Numbers.** JSON equality compares types: `3` is not `3.0`, and `true` is not `1`. A tool case needs exactly one
structured call; text next to it is ignored.
- **After a failure.** The runner asks `/health` once. If the server does not answer, the remaining requests are
diff --git a/docs/COMMUNITY_BENCHMARKS.md b/docs/COMMUNITY_BENCHMARKS.md
index b08cffe2e..1c518d6b5 100644
--- a/docs/COMMUNITY_BENCHMARKS.md
+++ b/docs/COMMUNITY_BENCHMARKS.md
@@ -5,7 +5,7 @@ might run on a similar machine. This guide suggests a report format and a pull
request workflow; a single model, GPU, or context length is enough to contribute.
There are two ways to measure. For results other people can compare with, run the **standard benchmark**: `strata bench run`
follows the draft protocol in [BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md) (fixed prompts, thinking on, the model card's
-sampling, twelve sanity checks, about 20 to 30 minutes after the model has loaded) and writes `results.json` with
+sampling, twelve sanity checks, about 2 to 4 minutes after the model has loaded on a fast card, more on a slow one) and writes `results.json` with
the hardware, versions and every request. For a quick look at your own machine, [`BENCH.bat` / `./bench.sh`](#run-the-benchmark)
takes 3 to 10 minutes and writes a report in the format below.
diff --git a/strata_cli/bench_standard.py b/strata_cli/bench_standard.py
index 7c6bbdc59..9ee5fb351 100644
--- a/strata_cli/bench_standard.py
+++ b/strata_cli/bench_standard.py
@@ -1076,14 +1076,32 @@ def _minutes(seconds: float | None) -> str:
return f"{int(seconds // 60)} min {int(seconds % 60)} s"
+SHORT_PROMPT_NOTE = ("short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not "
+ "shown; read prompt speed from prompt-4k and prompt-16k")
+CAP_NOTE = ("Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the "
+ "time to the cap, not to an answer, and it has no first-answer time when it was still reasoning")
+
+
+def capped(doc: dict, phase: str, case: str) -> str:
+ """'k/n': how many of the case's measured requests stopped at the output cap."""
+ rows = [r for r in doc["requests"] if r["phase"] == phase and r["case_id"] == case]
+ return f"{sum(1 for r in rows if r['status'] == 'truncated')}/{len(rows)}"
+
+
+def prompt_cell(doc: dict, phase: str, case: str, digits: int = 1) -> str:
+ return "n/a" if case == "short" else _span(_find(doc, phase, case, "prompt_tps"), digits)
+
+
def table_lines(doc: dict, phase: str, cases: tuple[str, ...]) -> list[str]:
- head = ["Case", "Prompt tok/s", "Decode tok/s", "TTFT s", "Total s"]
+ head = ["Case", "Prompt tok/s", "Decode tok/s", "TTFT s", "Total s", "Capped"]
rows = [head]
for case in cases:
- rows.append([case, _span(_find(doc, phase, case, "prompt_tps")), _span(_find(doc, phase, case, "decode_tps")),
- _span(_find(doc, phase, case, "ttft_s"), 2), _span(_find(doc, phase, case, "total_s"), 1)])
+ rows.append([case, prompt_cell(doc, phase, case), _span(_find(doc, phase, case, "decode_tps")),
+ _span(_find(doc, phase, case, "ttft_s"), 2), _span(_find(doc, phase, case, "total_s"), 1),
+ capped(doc, phase, case)])
widths = [max(len(r[i]) for r in rows) for i in range(len(head))]
- return [" " + " ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)).rstrip() for row in rows]
+ return ([" " + " ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)).rstrip() for row in rows]
+ + [" " + SHORT_PROMPT_NOTE + ".", " " + CAP_NOTE + "."])
def format_plain(doc: dict) -> str:
@@ -1167,12 +1185,13 @@ def report_markdown(doc: dict, redact=lambda text: text) -> str:
"monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. "
f"Time after the model loaded: {_minutes(run_block['required_duration_s'])}.", "",
"## Results", ""]
- lines += ["| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s |", "| --- | --- | --- | --- | --- | --- |"]
+ lines += ["| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |", "| --- | --- | --- | --- | --- | --- | --- |"]
for case in PERFORMANCE_CASES:
- lines.append(f"| {case} | " + " | ".join(
- _span(_find(doc, "performance", case, m), d) for m, d in (("prompt_tps", 1), ("decode_tps", 1), ("ttft_s", 2),
- ("first_answer_s", 2), ("total_s", 1))) + " |")
- lines += ["", "Cells are median (min-max) n=samples.", "",
+ lines.append(f"| {case} | " + prompt_cell(doc, "performance", case) + " | " + " | ".join(
+ _span(_find(doc, "performance", case, m), d) for m, d in (("decode_tps", 1), ("ttft_s", 2),
+ ("first_answer_s", 2), ("total_s", 1)))
+ + f" | {capped(doc, 'performance', case)} |")
+ lines += ["", "Cells are median (min-max) n=samples. " + SHORT_PROMPT_NOTE[0].upper() + SHORT_PROMPT_NOTE[1:] + ". " + CAP_NOTE + ".", "",
"| Phase | Planned | Completed | Truncated | Failed | Skipped |", "| --- | ---: | ---: | ---: | ---: | ---: |"]
for phase in PHASES:
c = counts[phase]
diff --git a/tests/terminal/test_bench_standard.py b/tests/terminal/test_bench_standard.py
index 3a5350c6f..bce01fa08 100644
--- a/tests/terminal/test_bench_standard.py
+++ b/tests/terminal/test_bench_standard.py
@@ -705,6 +705,24 @@ class RunTests(StandardBase):
self.assertIn("Prompt tok/s", text)
self.assertIn("1000.0 (1000.0-1000.0) n=3", text)
+ def test_the_summary_says_what_the_short_prompt_and_the_cap_mean(self):
+ doc, _ = self.run_standard()
+ for text in (bs.format_plain(doc), bs.report_markdown(doc)):
+ self.assertIn("Capped", text)
+ self.assertIn("not shown; read prompt speed from prompt-4k and prompt-16k", text)
+ self.assertIn("time to the cap, not to an answer", text)
+ short_row = next(line for line in bs.format_plain(doc).splitlines() if line.strip().startswith("short "))
+ self.assertIn("n/a", short_row)
+ self.assertNotIn("1000.0", short_row)
+ self.assertIn("0/3", short_row)
+ # the data keeps the short case's prompt rate: only the readable summaries leave it out
+ self.assertIsNotNone(self.summary(doc, "short", "prompt_tps"))
+ # a request that stopped at the cap is counted
+ record = next(r for r in doc["requests"] if r["id"] == "performance-prompt-4k-r1")
+ record["status"] = "truncated"
+ row = next(line for line in bs.format_plain(doc).splitlines() if line.strip().startswith("prompt-4k "))
+ self.assertTrue(row.rstrip().endswith("1/3"))
+
def test_the_server_is_not_asked_for_anything_outside_the_protocol(self):
self.run_standard()
self.assertTrue(all(body["messages"] == corpus_specs()[rid]["messages"] for body, rid in zip(self.server.bodies, self.server.order)))