Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Docs: BENCHMARK_PROTOCOL (implemented, still draft: field mapping, Metal, versions, runner reading, SECURITY link), COMMUNITY_BENCHMARKS, TERMINAL; BENCH.bat, bench.sh and strata_bench.py point to strata bench run
Niko1221c8e5d2f3ce33
diff --git a/BENCH.bat b/BENCH.batindex 8d1ed0e81..a5ba90058 100644--- a/BENCH.bat+++ b/BENCH.bat@@ -2,6 +2,7 @@rem Benchmark this Strata install the way it is used and write a report you can share (docs/COMMUNITY_BENCHMARKS.md).rem It starts the model with your settings (or uses the one already running), takes about 10 minutes and changes nothing.rem Options: --clients 4 (several requests at once), --long (a 32K prompt), --compare <older results folder>.+rem This is the quick speed check. The standard benchmark (docs/BENCHMARK_PROTOCOL.md), whose results other people can compare with, is `strata bench run`.setlocaltitle Strata - benchmarkcd /d "%~dp0"@@ -10,6 +11,8 @@ if not exist ".venv\Scripts\python.exe" (pauseexit /b 1)+echo Quick speed check. For results other people can compare with (the standard benchmark, 20 to 30 minutes), run: .venv\Scripts\python.exe strata.py bench run+echo.".venv\Scripts\python.exe" tools\strata_bench.py %*set "RC=%ERRORLEVEL%"pausediff --git a/bench.sh b/bench.shindex de0e451af..3cb68d1d2 100755--- a/bench.sh+++ b/bench.sh@@ -2,8 +2,11 @@# Benchmark this Strata install the way it is used and write a report you can share (docs/COMMUNITY_BENCHMARKS.md).# It starts the model with your settings (or uses the one already running), takes about 10 minutes and changes nothing.# Options: --clients 4 (several requests at once), --long (a 32K prompt), --compare <older results folder>.+# This is the quick speed check. The standard benchmark (docs/BENCHMARK_PROTOCOL.md), whose results other people can compare with, is `strata bench run`.cd "$(dirname "$0")" || exit 1if [ -x .venv/bin/python ]; then+ echo "Quick speed check. For results other people can compare with (the standard benchmark, 20 to 30 minutes), run: .venv/bin/python strata.py bench run"+ echoexec .venv/bin/python tools/strata_bench.py "$@"fiecho "Strata is not set up here yet: run ./setup.sh first."diff --git a/docs/BENCHMARK_PROTOCOL.md b/docs/BENCHMARK_PROTOCOL.mdindex 350bdad2f..03f2c8ca9 100644--- a/docs/BENCHMARK_PROTOCOL.md+++ b/docs/BENCHMARK_PROTOCOL.md@@ -1,6 +1,6 @@# Strata standard benchmark protocol-**Status: draft for implementation and pilot validation.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.+**Status: draft, implemented in `strata bench run`; pilot validation pending.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.Result format: `strata-standard-benchmark`, schema version `1`.This document specifies a repeatable community benchmark and the data a future website can consume. The@@ -8,11 +8,14 @@ required suite targets **20–30 minutes after model loading**. This duration haprotocol. Faster machines may finish sooner and slower machines may take longer; everyone runs the same work.Downloads, installation, compilation, and model loading are excluded. Report their duration separately if measured.-**Today's `BENCH.bat`, `./bench.sh`, and `tools/strata_bench.py` do not implement this protocol.** Continue using-the [community guide](COMMUNITY_BENCHMARKS.md) for current reports. Do not rename their output or add a protocol-ID to make it appear standard. Runner implementation, frozen fixtures, validation, and a measured pilot must-precede activation. The approved manifest described here does not yet exist.+`strata bench run` implements this protocol from Strata 0.2.1. It sends the frozen fixtures in+`bench/prompts/standard-v1` (corpus SHA-256 `e75eff8b9feea04f745520b392c4c17d97aefc1ba185e222aa6740efa8f68407`)+and writes the result format below. `BENCH.bat`, `./bench.sh`, and `tools/strata_bench.py` do not implement it.+They stay the quick speed check described in the [community guide](COMMUNITY_BENCHMARKS.md). Do not rename their+output or add a protocol ID to make it appear standard. The draft becomes final only after the measured pilot in the+[activation checklist](#activation-checklist). The approved manifest described here does not exist yet.+- [Run it](#run-it)- [Profile and settings](#profile-and-settings)- [Workload and scoring](#workload-and-scoring)- [Measurements](#measurements)@@ -20,6 +23,31 @@ precede activation. The approved manifest described here does not yet exist.- [Submit and review](#submit-and-review)- [Activation checklist](#activation-checklist)+## Run it++Install Strata, set up the Flash-Next model you want to measure, and close other programs that use the GPU. Then:++```sh+strata bench run --profile NAME --track default --track-notes "0.2.1 release, setup defaults"+strata bench run --url http://127.0.0.1:8080 # a server that is already running+```++With `--profile` the runner starts that profile's engine, waits for the model to load, runs the protocol, and stops+the engine it started. If the profile is already serving, the runner uses that server and leaves it running; a+server that has already answered requests is not freshly loaded, and the report says how many it had answered.+`--no-start` forbids starting an engine. `--optional-concurrency` adds the optional stage.++The runner checks every fixture against the bundle manifest before it sends anything, refuses to run if one byte+differs, and rejects flags that would change the workload (`--concurrency`, `--repeat`, `--request-timeout`).+`strata bench report DIR` prints a saved result again and re-validates it. Output goes to+`bench/results/<UTC-date>-standard-<gpu>-<run-id>/` when the current folder has `bench/results`, otherwise to+`./bench-results/`. Read the folder once before you share it: home folders, user and host names, private addresses,+and keys are replaced, but the runner cannot know everything you consider private.++Options that describe what the engine cannot tell the runner: `--track`, `--track-notes`, `--model-revision`,+`--source-revision`, `--tokenizer-revision`, `--quantization`, `--custom-pack`, `--background` (repeatable), and+`--build-options`. The runner never guesses these; an unset value is `null` with a reason.+## Profile and settingsThe first profile is `qwen3.8-flash-next-thinking`: the original **Qwen/Qwen3.8-Flash-Next** weights and their@@ -89,10 +117,20 @@ engine dies or the user cancels. Do not retry failed requests inside a run or re### Frozen fixtures-The runner implementation must commit a fixture bundle before the pilot. It must contain exact messages, tool-schemas, expected outputs, case IDs, and validation rules. Hash the bundle's files with a SHA-256 manifest and-store that manifest's hash as `protocol.corpus_sha256`. Record the protocol Git commit. Changes to prompts,-order, scoring, caps, or defaults require a new protocol ID; schema changes use a separate schema version.+The bundle is committed in `bench/prompts/standard-v1`: one JSON file per scheduled request (the 22 required and+the 12 optional ones) with its exact messages, tool schemas, seed, cap, expected output and frozen token count,+plus `build.json` (how it was made) and `MANIFEST.sha256`. The SHA-256 of that manifest is+`protocol.corpus_sha256`. `.gitattributes` marks `bench/prompts/**` as `-text`: a checkout must never convert+their line endings, and the runner refuses a bundle whose bytes differ from the manifest. The runner records the+protocol Git commit. Changes to prompts, order, scoring, caps, or defaults require a new protocol ID; schema+changes use a separate schema version.++`python tools/build_standard_corpus.py --tokenizer PACK/tokenizer` rebuilds the bundle byte for byte. It needs the+tokenizer files (`vocab.json`, `merges.txt`, `token_type.json`) of a Flash-Next pack and the chat template Strata+ships (`serve/chat_template.jinja`). The frozen counts are 211 records for `prompt-4k` (4,093 tokens) and 858+records for `prompt-16k` (16,386 tokens). Besides each request's token count, the bundle stores the longest token+prefix the request shares with any other request (`max_shared_prefix_tokens`); that is the cache reuse the+protocol allows.Use synthetic data only. Do not read the checkout's current README or other changing project files into prompts.Build the candidate bundle as follows; these are construction instructions, not runtime adaptation:@@ -151,6 +189,39 @@ No text answer is required in a successful tool-call case. Record malformed outperrors, and skipped cases individually. Only a completed, correct response passes. The denominator remains`12`, with category denominators `4`; skipped cases also make the suite partial.+### How the runner reads the protocol++Where the rules leave room, `strata bench run` takes the most literal reading. These are its choices; a different+reading needs a new protocol ID.++- **Settings.** Every request carries all sampling values, `reasoning_effort: "xhigh"`, both thinking flags, the+ seed and the cap, plus `reasoning_budget_tokens: 0` (no budget) and `stream_options.include_usage`. The server's+ chat-completions handler passes `enable_thinking` and `reasoning_effort` to the template; it ignores+ `preserve_thinking`, which changes nothing for a single-turn conversation. A server that answers a request with+ HTTP 400 or 422 makes the run `nonconforming`; the runner never retries with other settings. The runner cannot see+ the server's repeat-stop and loop-recovery switches, so it does not claim to have disabled them.+- **Track.** `--track` defaults to `custom`. The runner cannot tell whether a setup is untouched, so `default` is+ only what the operator declares.+- **Unexpected reuse.** A request's `cached_tokens` may be at most its `max_shared_prefix_tokens` (see the fixtures)+ plus 8 tokens for merges at the boundary. Above that the user payload was read from the cache: the request is+ left out of the speed summaries, and a performance request makes the run `nonconforming`. Running the protocol twice+ against the same server therefore flags the second run; restart the engine between runs.+- **Prompt length.** The measured `input_tokens` of the 4K and 16K cases must stay within 2% of 4,096 and 16,384.+ A different tokenizer or template breaks that and makes the run `nonconforming`.+- **Truncation.** `finish_reason: "length"` is `truncated`. It stays in the speed summaries and fails a sanity case.+ Any other finish reason except `stop` and `tool_calls` is a failure.+- **Numbers.** JSON equality compares types: `3` is not `3.0`, and `true` is not `1`. A tool case needs exactly one+ structured call; text next to it is ignored.+- **After a failure.** The runner asks `/health` once. If the server does not answer, the remaining requests are+ `skipped`. A failed warm-up skips everything after it. Ctrl-C ends the run the same way and still writes the result.+- **Timeout.** The 1,800 seconds are counted from dispatch with a timer that closes the connection, so a stalled+ stream and a long queue both count.+- **Duration.** `required_duration_s` runs from the warm-up's dispatch to the end of the last sanity request. Hashing+ the model files and probing the machine come before it.+- **Model files.** The runner hashes the GGUF files it can find (the server's `model_path`, every shard of a split file, and+ files named in the profile's engine arguments). Without a file there is no hash and the run is `nonconforming`.+- **Remote servers.** A server that is not on loopback gives a `nonconforming` result, because standard latency is a loopback measurement.+## MeasurementsKeep the inference host's hardware separate from any client description. Standard latency uses loopback;@@ -158,6 +229,13 @@ remote/network measurements belong in a separate nonstandard report. Record allindices, dedicated or shared memory, CPU, RAM, storage, OS, driver/runtime versions, power limits when available,and other workloads. Use bytes for memory, watts for power, seconds for durations, and tokens/second for rates.+On Apple silicon (Metal) the GPU and the CPU share one memory pool: record one GPU entry named after the chip,+`memory_kind: "shared"`, `memory_bytes` equal to the installed memory, and `power_limit_w: null` with a reason.+There is no separate GPU driver (`drivers` stays `null` with a reason); put the macOS version in `runtime` as+`{"name": "metal", "version": "<macOS version>"}`. AMD APUs such as Strix Halo also use shared memory; the runner+reports `shared` only for Apple silicon and the operator must correct any other unified-memory machine in the+README.+| Metric | Definition and evidence || --- | --- || Prompt tokens/sec | Engine `read_tokens / prompt_s`, using only freshly processed tokens and their engine duration. Cached tokens never enter the numerator. |@@ -167,6 +245,24 @@ and other workloads. Use bytes for memory, watts for power, seconds for duration| Total duration | Client dispatch through stream completion, error, or cancellation, including queueing and prompt processing. || Concurrent aggregate tokens/sec | For one round, total accepted output tokens from completed or capped requests divided by wall time from first dispatch to last termination in that round. Keep failed-request time in that wall time; report failures alongside the rate. |+### Where the numbers come from++Strata's streaming chat answer ends with a chunk that carries `usage` and `timings` (llama.cpp's names). The runner+maps them as follows. Nothing else feeds a throughput rate.++| Protocol field | Source in the answer |+| --- | --- |+| `input_tokens` | `usage.prompt_tokens` |+| `cached_tokens` | `usage.prompt_tokens_details.cached_tokens`, else `timings.cache_n` |+| `read_tokens` | `timings.prompt_n`; if absent, `input_tokens - cached_tokens`, and `sources` says it was derived |+| `prompt_s` | `timings.prompt_ms / 1000` |+| `output_tokens` | `usage.completion_tokens` |+| `decode_s` | `timings.predicted_ms / 1000` |+| `decode_tokens` | `timings.predicted_n` when it agrees with `timings.predicted_per_second x predicted_s` (the engine rounds its rate to 0.1 tokens/s, so the check allows `0.05 x decode_s + 1` tokens); otherwise the count the rate implies, with a `token_accounting_note` |+| `prompt_tps` | `read_tokens / prompt_s`, null when `read_tokens` is 0 (the whole prompt came from the cache) or a duration is 0 |+| `decode_tps` | `decode_tokens / decode_s`, null when either is 0 |+| `ttft_s`, `first_answer_s`, `total_s` | the client's monotonic clock from dispatch; never an engine field |+Retain API `output_tokens` separately from engine `decode_tokens`. If the counts differ, explain the engineaccounting and preserve its raw timing lines; do not assume `predicted_n` is always the count used by`predicted_per_second`. Reasoning/answer token splits are optional if the engine cannot provide them reliably.@@ -219,6 +315,14 @@ explaining why, including not-applicable values. Empty arrays mean genuinely no| `missing` | Map of JSON Pointers to specific reasons for null values. || `extensions` | Optional object for data beyond this contract; never a substitute for required fields. |+Where the software fields come from. `strata_commit` and `runner_commit` are the full commit of the install: `git rev-parse+HEAD` in a source checkout, or the `COMMIT` file a release archive carries (written by `packaging/package_terminal.py`).+An install with neither has no commit, and the run is `nonconforming`. `engine_version` is the engine's own version+from `/props` (`build_info`), or `/v1/status` when `/props` has none. `build_kind` is `source` when the+folder has a `.git`, otherwise `release`. The runner cannot read build options from a server; pass `--build-options`+for a source build. The drivers and the CUDA version come from `nvidia-smi` (or `rocm-smi`, or the Windows display+driver key), read on the machine that runs the engine.+Except for typed objects, arrays, numbers and booleans identified above, identity and descriptive fields arestrings (or null with a reason where unavailable). Empty configuration objects mean no relevant options;unknown settings are null. `configuration.track_notes` states setup choices for a default run or modifications@@ -323,7 +427,8 @@ with the JSON; derived summaries can always be rebuilt from request records.## Submit and review-Follow [CONTRIBUTING.md](../CONTRIBUTING.md). During the draft phase, submit pilot data clearly labelled with+Follow [CONTRIBUTING.md](../CONTRIBUTING.md). If a log or output shows a vulnerability or a secret, report it privately as+[SECURITY.md](../SECURITY.md) describes instead of putting it in a pull request. During the draft phase, submit pilot data clearly labelled withthe draft protocol ID. Existing reports remain welcome under the [community guide](COMMUNITY_BENCHMARKS.md).1. Run one hardware/model/quant/configuration combination and retain every scheduled outcome. Add a separate@@ -386,24 +491,30 @@ actual lengths/cache counts, and concurrency; there is no universal combined sco## Activation checklist-The current runner hardcodes temperature `0`, disables thinking by default, has no twelve-case sanity suite,-uses changing repository files in its agent workload, and can fall back from token counts to stream-chunk counts.-It also lacks the finish/first-answer accounting and versioned result contract above. `--think` alone does not-fix those differences. These are implementation tasks for a later change, not capabilities of the current CLI.+`tools/strata_bench.py` (behind `BENCH.bat` and `./bench.sh`) hardcodes temperature `0`, turns thinking off by default,+has no twelve-case sanity suite, reads changing repository files into its agent workload, and can fall back from token+counts to stream-chunk counts. `--think` alone does not fix those differences. `strata bench run` does not share+them: it is the protocol's runner.++Done in 0.2.1, with tests against a scripted mock server (`tests/terminal/test_bench_standard.py`) and the real+API server's mock engine:++- The fixtures, the profile settings, the runner, a strict result validator (`strata bench report` runs it), and the+ evidence files. Complete runs, early EOS, cap truncation, wrong answers, malformed JSON and tool calls, duplicate JSON+ keys, missing usage or timings, empty deltas and keep-alives, timeouts, cancellation, a stream cut before its end,+ server death, count identities, summary reconstruction, unexpected payload reuse, a refused setting, a remote server,+ optional concurrency, null reasons, and unknown format versions.Before removing draft status:-- Implement and freeze the fixtures, profile, runner, strict result validator, and evidence generation. Verify- default sampling/template behavior and complete token/timing accounting against the engine.-- Test complete runs, early EOS, cap truncation, wrong answers, malformed JSON/tool calls, duplicate JSON keys,- missing usage/timings, empty deltas, timeouts, cancellation, errors after a first delta, and server death.-- Test count identities, summary reconstruction, shared-prefix versus unexpected payload reuse, insufficient- context, unsupported defaults, optional concurrency with queueing, and zero/invalid timing denominators.-- Check hashes/provenance, null reasons, redaction, unknown format versions, draft/nonconforming exclusion, and- changed artifacts after review. Independently recompute sanity scores from saved final outputs.+- Verify against a real engine that the shipped chat template matches the pinned upstream behavior (tool+ formatting included), and pin the model-card and tokenizer revisions (`protocol.model_card_revision` is `null`+ with a reason until then). The shipped `serve/chat_template.jinja` differs from the template inside the+ NVFP4 pack the fixtures were counted with; the frozen token counts use the shipped one.+- Test the cases the mock server cannot show: insufficient context, an error after a first delta on a real engine, and+ optional concurrency with queueing on a server that has fewer than four slots.+- Recompute the sanity scores independently from the saved final outputs, and check draft/nonconforming exclusion+ and changed artifacts after review in the website that reads the results.- Run a measured pilot on at least two supported hardware configurations, including a slower system. Publishactual duration, caps reached, failures, and sanity outcomes. Adjust the candidate workload if needed, thenfreeze a new protocol ID before collecting comparable standard results. Do not promise a universal runtime.--For this documentation change, validate links and anchors, issue-form YAML, JSON example syntax, request-count-arithmetic, and agreement between these rules and contribution templates. No runtime behavior is changed.diff --git a/docs/COMMUNITY_BENCHMARKS.md b/docs/COMMUNITY_BENCHMARKS.mdindex 60f35190a..b08cffe2e 100644--- a/docs/COMMUNITY_BENCHMARKS.md+++ b/docs/COMMUNITY_BENCHMARKS.md@@ -3,9 +3,14 @@Share measurements from your own hardware so other users can judge how Stratamight run on a similar machine. This guide suggests a report format and a pullrequest workflow; a single model, GPU, or context length is enough to contribute.-The quickest way is the built-in benchmark: [`BENCH.bat` / `./bench.sh`](#run-the-benchmark) writes a report in this format.+There are two ways to measure. For results other people can compare with, run the **standard benchmark**: `strata bench run`+follows the draft protocol in [BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md) (fixed prompts, thinking on, the model card's+sampling, twelve sanity checks, about 20 to 30 minutes after the model has loaded) and writes `results.json` with+the hardware, versions and every request. For a quick look at your own machine, [`BENCH.bat` / `./bench.sh`](#run-the-benchmark)+takes 3 to 10 minutes and writes a report in the format below.-A standard benchmark protocol for future submissions exists only as a **draft**: [BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md). The tools described on this page do not implement it, and the reports below are not standard results; keep using this guide for now.+The protocol is still a draft until a pilot has run on two kinds of hardware. The reports listed below were made before+it existed and are not standard results; they stay valid under this guide.Existing [benchmark reports](../bench/results/) provide examples. The[engine 0.1.26 speed report](../bench/results/2026-09-29-speed-0126/README.md)@@ -69,12 +74,31 @@ and their limits. Report what you actually measured and label estimates separate- [2026-10-12: Radeon 780M (gfx1103), Ryzen 7 8745HS, 60 GB RAM, GTT 50 GiB](../bench/results/2026-10-12-community-gfx1103/README.md):the gfx1103 port (native dot4, the gfx11 WMMA kernels, the exact speed switches and a calibrated hipBLASLt table), hand-built on ROCm 10.1, Q2_0 + MTP at 32K: prompts 90 tok/s plain, 186 with the table, 277 with the WMMA opt-ins (4K), decode 13.1 -> 14.3 tok/s; zero GPU resets.+## Run the standard benchmark++```sh+strata bench run --profile NAME --track default --track-notes "release 0.2.1, setup defaults"+strata bench run --url http://127.0.0.1:8080 # a server that is already running+```++With `--profile` the command starts that profile's engine, waits for the model, runs the protocol and stops the engine. With `--url`+it uses the server that is running. Start from a freshly loaded engine and close other programs that use the GPU: the+protocol counts a second run against the same engine as reuse of the cache and marks it `nonconforming`. The command prints a+summary when it ends and writes a folder `<date>-standard-<gpu>-<run id>` with `README.md`, `results.json`, `outputs.jsonl`,+the settings, the chat template and, when it started the engine, the engine log. `strata bench report FOLDER` shows it again.++A run is `complete`, `partial` (something was skipped or cancelled) or `nonconforming` (for example a remote server, a missing+commit or model hash, or reused prompt text). A complete run can still fail sanity checks; low scores and slow hardware are valid+results. The runner does not know the quantization repository's revision, the original weights' revision or what else ran on the+machine: pass `--model-revision`, `--source-revision` and `--background`, or the result says `null` with a reason. To share+a result, follow [Submit and review](BENCHMARK_PROTOCOL.md#submit-and-review) in the protocol, not the pull request steps below.+## Run the benchmarkThe simplest report is one command after setup. Windows: double-click `BENCH.bat`. Linux: `./bench.sh`. Or`python tools/strata_bench.py` from the Strata folder. Setup also offers it at the end of an interactive install-(default no). The Terminal's `strata bench run` ([TERMINAL.md](TERMINAL.md)) is a different tool: it measures serving suites and-writes JSON evidence for your own comparisons, but not the `README.md` report in this format and not the draft standard protocol.+(default no). The Terminal's `strata bench run` runs the standard benchmark (next section); its `--suite serving-smoke` and+other serving suites ([TERMINAL.md](TERMINAL.md)) write JSON evidence for your own comparisons, not a report in this format.It starts the model with the settings you already have (the newest `strata-*.json`; `--config FILE` picks another one),or uses a Strata server that is already running on that port (`--url http://127.0.0.1:8080` for another address). It thendiff --git a/docs/TERMINAL.md b/docs/TERMINAL.mdindex abdc75be3..e67a489ee 100644--- a/docs/TERMINAL.md+++ b/docs/TERMINAL.md@@ -134,6 +134,7 @@ strata engine calibrate --profile four-gpu --rollback-result calibration.json --To measure the **running** server after selecting the intended model and batching profile, run an explicit serving workload and keep its local evidence:```sh+strata bench run --profile four-gpu # the standard benchmark (docs/BENCHMARK_PROTOCOL.md)strata bench run --profile four-gpu --suite serving-smoke --concurrency 8 --out ./bench-smoke-01strata bench report ./bench-smoke-01 --output jsonstrata bench run --url http://127.0.0.1:8095 --suite parallel-mixed-v1 --concurrency 8 --repeat 3 --out ./bench-mixed-01@@ -148,7 +149,7 @@ strata bench run --profile four-gpu --require-managed-match --suite parallel-mixstrata bench compare ./bench-baseline ./bench-monitored --output json```-`bench run` is a serving benchmark for your own tuning and comparisons; it is not the draft community benchmark protocol ([BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md)), and its output is not a standard submission. It requires a loaded instance and versioned telemetry. It sends one warmup request, a two-request serial phase and a mixed short/long streaming phase; `parallel-mixed-v1` repeats the mixed phase twice per trial. `parallel-overload-v1` instead sends one warmup, a synchronized short/long burst and a recovery request. It requires configured `max_queue` and `queue_timeout`, and `--concurrency` must exceed the reported admission limit; use up to 64 workers for this explicit suite, or up to 16 for the others. The example of 48 workers only fits a server whose reported admission limit is below 48. Expected `admission_full` and `queue_wait_timeout` responses are counted separately from unexpected errors; the suite passes only if some calls run, some are refused or time out, and recovery works. `parallel-cancel-v1` closes one active stream after its first content, then fills the effective slots and closes a queued stream, followed by a recovery request. It requires at least one queue place, a queue timeout of at least two seconds, disconnect telemetry and `--concurrency` at least the effective slot count. It checks that the target was visible, the server observed a client disconnect, companions completed and admission and slots settled. `parallel-prefill-cancel-v1` separately closes a target while the engine reports partial prompt reading in a batch slot, with one companion decoding; it requires at least two effective slots, concurrency at least two and disconnect telemetry. Both cancellation suites verify companion completion, settled resources and recovery. Run them without unrelated client traffic because the disconnect counter is server-wide. `--concurrency` is a client-worker count, not the engine's effective slot count. Supply an inference key through `STRATA_API_KEY`; if monitoring has a separate key, add `--telemetry-credential-env STRATA_TELEMETRY_KEY`. The new output directory contains `results.json`, with request latency, first-token time, output token counts when the API provides them, and sampled active/queued/occupied slots, groups and per-card readings. Prompts, answer text and credentials are excluded. The report records null when a backend signal is unavailable and labels the limits of client-observed timing. `bench run --profile NAME --require-managed-match` is an opt-in gate for a server launched by the POSIX owner host. It rejects a changed profile, changed source config, unverified owner launch, wrong telemetry instance or endpoint before creating benchmark output. It checks again after the workload and retains a failed report if the match was lost. Reports label the result `managed_launch_checked`; default, foreground and remote runs label it `unverified`. This is a beginning/end check of the owner's startup attestation, not continuous verification of running files. These source-run suites do not verify model artifact hashes or driver versions or establish four-GPU throughput on the target host.+`bench run` without `--suite` runs the standard benchmark ([BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md), `--suite standard-v1`): the frozen requests, thinking on, the model card's sampling, twelve sanity checks, and a result other people can compare with. It works against a running server (`--url`, or a `--profile` that is already serving) or starts the profile's engine and stops it afterwards (`--no-start` forbids that). It rejects `--concurrency`, `--repeat`, `--request-timeout`, `--condition` and `--require-managed-match`, which belong to the serving suites, and `--out` is optional for it. The serving suites below are for your own tuning and comparisons, their output is not a standard submission, and they need `--out`. A serving suite requires a loaded instance and versioned telemetry. It sends one warmup request, a two-request serial phase and a mixed short/long streaming phase; `parallel-mixed-v1` repeats the mixed phase twice per trial. `parallel-overload-v1` instead sends one warmup, a synchronized short/long burst and a recovery request. It requires configured `max_queue` and `queue_timeout`, and `--concurrency` must exceed the reported admission limit; use up to 64 workers for this explicit suite, or up to 16 for the others. The example of 48 workers only fits a server whose reported admission limit is below 48. Expected `admission_full` and `queue_wait_timeout` responses are counted separately from unexpected errors; the suite passes only if some calls run, some are refused or time out, and recovery works. `parallel-cancel-v1` closes one active stream after its first content, then fills the effective slots and closes a queued stream, followed by a recovery request. It requires at least one queue place, a queue timeout of at least two seconds, disconnect telemetry and `--concurrency` at least the effective slot count. It checks that the target was visible, the server observed a client disconnect, companions completed and admission and slots settled. `parallel-prefill-cancel-v1` separately closes a target while the engine reports partial prompt reading in a batch slot, with one companion decoding; it requires at least two effective slots, concurrency at least two and disconnect telemetry. Both cancellation suites verify companion completion, settled resources and recovery. Run them without unrelated client traffic because the disconnect counter is server-wide. `--concurrency` is a client-worker count, not the engine's effective slot count. Supply an inference key through `STRATA_API_KEY`; if monitoring has a separate key, add `--telemetry-credential-env STRATA_TELEMETRY_KEY`. The new output directory contains `results.json`, with request latency, first-token time, output token counts when the API provides them, and sampled active/queued/occupied slots, groups and per-card readings. Prompts, answer text and credentials are excluded. The report records null when a backend signal is unavailable and labels the limits of client-observed timing. `bench run --profile NAME --require-managed-match` is an opt-in gate for a server launched by the POSIX owner host. It rejects a changed profile, changed source config, unverified owner launch, wrong telemetry instance or endpoint before creating benchmark output. It checks again after the workload and retains a failed report if the match was lost. Reports label the result `managed_launch_checked`; default, foreground and remote runs label it `unverified`. This is a beginning/end check of the owner's startup attestation, not continuous verification of running files. These source-run suites do not verify model artifact hashes or driver versions or establish four-GPU throughput on the target host.`bench run --condition NAME` records a short operator label, such as `baseline`, `monitor-default` or `monitor-fast`; it does not verify that a monitor was attached. `bench compare BASELINE CANDIDATE` reads two saved reports without contacting the server or writing files. It rejects failed runs, different suites/models, changed request fingerprints or concurrency, changed source-config digests and conflicting reported device/settings identity. It pairs phases and reports client-observed p95 TTFT, p95 latency and output throughput, plus admission/queue deltas. Missing values remain unavailable. The comparison is **provisional** even when both runs have `managed_launch_checked`: the owner match is checked only at the beginning and end, and exact driver/engine/model hashes, monitor attachment, host CPU/RSS and telemetry traffic are not attested. For the four-card overhead gate, keep at least three repeated measured phases, save `doctor --live` card/driver evidence, record PCIe and layer topology separately, measure host and monitor CPU/RSS and HTTP calls/bytes separately, and investigate changes against trial variability. A pair of reports alone is not release qualification.diff --git a/tools/strata_bench.py b/tools/strata_bench.pyindex 89dc30e5b..3a9877fc9 100644--- a/tools/strata_bench.py+++ b/tools/strata_bench.py@@ -24,6 +24,8 @@ The result goes to bench/results/<date>-community-<gpu>-<model>/ : README.md indocs/COMMUNITY_BENCHMARKS.md, results.json (every request) and an engine log excerpt. Home folders, user and host names,IP addresses and API keys are replaced by placeholders before anything is written; read it once before you share it.`--compare <older results folder>` prints the two side by side. Standard library only.+This is the quick speed check, not the standard benchmark: `strata bench run` follows docs/BENCHMARK_PROTOCOL.md (fixed prompts, thinking on,+the model card sampling, twelve sanity checks) and writes results other people can compare with."""from __future__ import annotations@@ -1027,6 +1029,7 @@ def main(argv=None) -> int:say(f"\nReport: {shown / 'README.md'}\nShare it: open a pull request that adds the folder {shown} (see docs/COMMUNITY_BENCHMARKS.md), "f"or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; "f"read it once before you share it.")+ say("For results other people can compare with, run the standard benchmark: strata bench run (docs/BENCHMARK_PROTOCOL.md).")return 1 if out["failures"] and not any(out[k] for k in ("chat", "prompts", "agent")) else 0finally:if started: