Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Add CONTRIBUTING.md and the draft standard benchmark protocol (owner text, unchanged)

Niko1221e9e08adb3731
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 000000000..5afaee4ac
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,116 @@
+# Contributing to Strata
+
+**AI assistance is welcome. The contributor is responsible for understanding the change and supplying real evidence.**
+
+These rules apply to issues, pull requests, benchmark reports, and the agents helping prepare them.
+You should be able to explain every changed line and answer review questions about it.
+
+## Before starting
+
+1. Search [issues](https://github.com/Niko1221/Strata/issues) and
+ [pull requests](https://github.com/Niko1221/Strata/pulls), including closed discussions. Add new evidence to an
+ existing thread when it covers your problem.
+2. Read [AGENTS.md](AGENTS.md) and the documentation for the component you want to change.
+3. Get maintainer agreement on scope before substantial features, refactors, architecture changes, or changes to
+ defaults. Describe the user problem, proposed behavior, and tradeoffs in an issue first. Agreement on scope
+ does not guarantee a merge. Focused fixes and benchmark reports can go straight to a PR.
+4. Keep one problem per PR. Update that PR after feedback; do not open competing attempts for the same change.
+
+Do not resubmit a declined proposal under a different title or through another agent. If new evidence changes
+the situation, bring it to the original discussion and let a maintainer decide whether to reopen it.
+
+## Report a problem people can investigate
+
+Use the matching [issue form](https://github.com/Niko1221/Strata/issues/new/choose). A bug report should include:
+
+- The smallest steps or command that reproduce the problem, and the expected and actual behavior.
+- Strata version or commit, OS, GPU and VRAM, and relevant model, quantization, and configuration.
+- The exact error and relevant logs, with credentials and private content removed.
+- Whether you reproduced it yourself. Label a finding based only on reading code or a scanner as
+ **unconfirmed static analysis**, and explain the suspected path without presenting it as an observed failure.
+
+Do not invent reproduction steps, errors, affected hardware, or severity. If you cannot reproduce something,
+say what you inspected and what would verify the hypothesis. Feature requests should explain a concrete user
+need and the current workaround. Questions should say what you tried and where you got stuck.
+
+Example structure, with placeholders to replace with actual observations:
+
+```text
+Problem: <one observed behavior>
+Environment: <commit, OS, GPU/VRAM, model/quant, relevant settings>
+Reproduce: <exact command or short sequence>
+Expected: <documented or intended behavior>
+Actual: <observed behavior and exact error>
+Evidence: <sanitized log attachment>
+Reproduction status: <reproduced locally OR unconfirmed static analysis>
+```
+
+## Prepare a focused pull request
+
+Lead with the problem and resulting behavior. Explain why the approach is needed, the important limits, and
+any compatibility effects. Link the issue and scope agreement when applicable. A file-by-file narration or
+an AI-generated summary of the diff is not a substitute for this explanation.
+
+- Keep unrelated formatting, cleanup, dependency changes, and speculative abstractions out of the PR.
+- Add documentation where it changes what a user or developer needs to know. Avoid repeated summaries,
+ speculative roadmaps, and generated documentation with no actionable information.
+- Preserve existing behavior outside the intended change. New features are opt-in; follow the default-path
+ and performance requirements in [AGENTS.md](AGENTS.md).
+- Read the complete diff, including generated files, before submitting. Remove debugging output, secrets,
+ private paths, and accidental files.
+- Prefer a small relevant regression check over tests that only restate the implementation. Documentation-only
+ changes need link, example, and consistency checks; they do not require GPU builds.
+
+## Show the validation you actually performed
+
+List exact commands, the revision and environment tested, and observed outcomes. Keep **passed**, **failed**,
+and **not run** separate. Include relevant output or a log attachment. A command you suggest running is not
+evidence that it passed; a CI run on a different revision is not validation of the submitted code.
+
+Build every backend a code change touches, including CUDA, HIP, and SYCL where applicable, before requesting
+review. For changes that need hardware execution, report that execution separately from compilation. A mock,
+static check, or successful build does not establish hardware correctness. If you lack a required environment,
+keep the PR as a draft, identify the missing checks, and ask for targeted help in the same thread. Do not claim
+compatibility you have not tested.
+
+Performance claims need before/after runs on the same hardware and workload, with every changed setting
+identified. Preserve the individual runs, failures, and variability. Follow the
+[community measurement guide](docs/COMMUNITY_BENCHMARKS.md); standard submissions will use the
+[benchmark protocol](docs/BENCHMARK_PROTOCOL.md) once it is activated. One fast run does not justify changing a
+default or claiming a speedup for other machines.
+
+Example PR body; placeholders are not test results:
+
+```text
+Problem and result: <trigger; behavior before and after the change>
+Related discussion: <issue URL and scope agreement, when required>
+Approach: <why this fix; relevant compatibility effects>
+Validation:
+ Passed: <exact command, revision/environment, observed result, log>
+ Failed: <exact command and failure, or none observed>
+ Not run: <check, reason, and what remains unverified>
+Limits: <known limitations or remaining work>
+```
+
+## Rules for AI-assisted work
+
+Use agents to help investigate, implement, and explain work that you can review. Before submitting, verify
+their claims against the source, actual command output, or a cited primary reference.
+
+Never submit fabricated measurements, invented test results, fake logs, or copied claims you have not checked.
+Do not turn a repository scan into a batch of speculative issues or PRs. Consolidate related findings, establish
+their effect, and submit the useful evidence. Do not present an agent's confidence or another model's approval
+as validation. Keep review replies concise and address the requested change or evidence directly.
+
+Maintainers may request a smaller change, return a PR to draft, or close a submission that lacks evidence,
+duplicates existing work, or ignores agreed scope. A smaller, understandable contribution with honest limits
+is easier to review and maintain.
+
+## Share a benchmark
+
+A results-only PR should contain the human report, raw measurements, sanitized configuration, and supporting
+logs. Keep it separate from engine changes and include poor results and failed requests.
+
+The [current community benchmark guide](docs/COMMUNITY_BENCHMARKS.md) remains available for measurements made
+with today's tools. The [new protocol](docs/BENCHMARK_PROTOCOL.md) defines the draft standard and future approval
+workflow. Its draft status must remain visible: existing reports are not automatically standard or approved.
diff --git a/docs/BENCHMARK_PROTOCOL.md b/docs/BENCHMARK_PROTOCOL.md
new file mode 100644
index 000000000..350bdad2f
--- /dev/null
+++ b/docs/BENCHMARK_PROTOCOL.md
@@ -0,0 +1,409 @@
+# Strata standard benchmark protocol
+
+**Status: draft for implementation and pilot validation.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.
+Result format: `strata-standard-benchmark`, schema version `1`.
+
+This document specifies a repeatable community benchmark and the data a future website can consume. The
+required suite targets **20–30 minutes after model loading**. This duration has not been measured for this
+protocol. Faster machines may finish sooner and slower machines may take longer; everyone runs the same work.
+Downloads, installation, compilation, and model loading are excluded. Report their duration separately if measured.
+
+**Today's `BENCH.bat`, `./bench.sh`, and `tools/strata_bench.py` do not implement this protocol.** Continue using
+the [community guide](COMMUNITY_BENCHMARKS.md) for current reports. Do not rename their output or add a protocol
+ID to make it appear standard. Runner implementation, frozen fixtures, validation, and a measured pilot must
+precede activation. The approved manifest described here does not yet exist.
+
+- [Profile and settings](#profile-and-settings)
+- [Workload and scoring](#workload-and-scoring)
+- [Measurements](#measurements)
+- [Result contract](#result-contract)
+- [Submit and review](#submit-and-review)
+- [Activation checklist](#activation-checklist)
+
+## Profile and settings
+
+The first profile is `qwen3.8-flash-next-thinking`: the original **Qwen/Qwen3.8-Flash-Next** weights and their
+quantizations. Record the quantization repository and immutable revision, source model revision, every model
+file's name and SHA-256, and any pack preparation or custom expert profile. Fine-tuned, Coder, Swift, or other
+derivative models require a separate profile. Results for different models, quants, builds, or configurations
+must remain distinguishable.
+
+Use the published [Qwen model-card defaults](https://huggingface.co/Qwen/Qwen3.8-Flash-Next):
+
+| Setting | Required value |
+| --- | --- |
+| `chat_template_kwargs.enable_thinking` | `true` |
+| `chat_template_kwargs.preserve_thinking` | `true` |
+| `reasoning_effort` | `"xhigh"` |
+| `temperature` | `1.0` |
+| `top_p` / `top_k` | `0.95` / `20` |
+| `min_p` | `0.0` |
+| `presence_penalty` / `repetition_penalty` | `0.0` / `1.0` |
+
+Pin the tokenizer revision, effective chat-template file and SHA-256, and model-card revision used to define
+the profile. Attach the effective template as a provenance artifact. The pilot must verify that the shipped
+template matches the pinned upstream behavior, including
+tool formatting. There is no extra benchmark system prompt. Tools are supplied only for the tool-call cases.
+Use streaming chat completions and request usage data.
+
+The future runner must explicitly request these settings and verify the resolved server configuration. UI,
+saved chat, and configuration defaults must not override them. Disable custom reasoning budgets, automatic
+effort changes, and additional stop strings. Unsupported settings make a run nonconforming; do not silently
+substitute greedy sampling or disable thinking. Output caps and fixed seeds below are benchmark constraints,
+not claims to use Qwen's full recommended output allowance. A fixed seed does not guarantee identical outputs
+across hardware or engine versions.
+
+Use one installed configuration per run. Record effective context, KV and expert cache settings, offloading,
+prefill, MTP, calibration, experimental options, and server capacity. Use `configuration.track = "default"` for
+an unmodified release's setup-generated settings; identify the release and setup choices. Use `"custom"` for
+tuning, experimental options, or modified builds and explain the changes. Never merge these tracks into one
+headline result. A claimed improvement requires a separate, same-machine baseline.
+
+## Workload and scoring
+
+Run the following stages in order through a dedicated server on the inference machine, using loopback HTTP.
+Start with a newly loaded engine, close other inference clients, and record unavoidable background work.
+Use the same configuration throughout the required stages. Include warm-up time in `required_duration_s`,
+but exclude warm-up requests from performance and quality summaries.
+
+| Stage | Cases and order | Repetitions | Output cap | Requests |
+| --- | --- | ---: | ---: | ---: |
+| Warm-up | `warmup`, short-prompt fixture | 1 | 128 | 1, excluded |
+| Performance | `short`, `prompt-4k`, `prompt-16k`; all three in each repetition | 3 | 512 | 9 |
+| Sanity | `I1`–`I4`, `R1`–`R4`, `T1`–`T4` | 1 each | 2,048 | 12 |
+| Optional concurrency | Four short-prompt requests released together, then wait for the entire round | 3 rounds | 512 each | 12, separate |
+
+The required suite schedules **21 measured requests plus one warm-up** at concurrency `1`. The optional
+extension schedules another `12` at concurrency `4` and has its own duration and summaries. Record the server's
+configured capacity: four clients may queue on a server with fewer than four slots. Do not describe that as
+four-way parallel decoding or silently change the server configuration. Testing another capacity is another run.
+
+Use seed `1` for warm-up; performance repetitions use seeds `1`, `2`, and `3`. Sanity cases use seed `1`.
+Optional round `r` uses seeds `4 * (r - 1) + j` for client `j` in `1..4`. Preserve early EOS; do not force more
+output to improve a rate. The caps include reasoning and final output together. Do not add a forced-answer
+instruction or a continuation request when a case reaches its cap.
+
+Apply a fixed 1,800-second wall-clock timeout per request, including queueing; cancel timed-out work and record
+it as failed. This is a safeguard, not a 30-minute suite deadline. Stop and mark remaining cases skipped if the
+engine dies or the user cancels. Do not retry failed requests inside a run or reduce the workload to fit hardware.
+
+### Frozen fixtures
+
+The runner implementation must commit a fixture bundle before the pilot. It must contain exact messages, tool
+schemas, expected outputs, case IDs, and validation rules. Hash the bundle's files with a SHA-256 manifest and
+store that manifest's hash as `protocol.corpus_sha256`. Record the protocol Git commit. Changes to prompts,
+order, scoring, caps, or defaults require a new protocol ID; schema changes use a separate schema version.
+
+Use synthetic data only. Do not read the checkout's current README or other changing project files into prompts.
+Build the candidate bundle as follows; these are construction instructions, not runtime adaptation:
+
+- The short task is: `Explain how a hash table finds a value, including collisions and one small example.`
+- Long performance payloads are records of the form
+ `record000001: alpha beta gamma delta epsilon zeta eta theta.` with sequential six-digit IDs and LF newlines.
+ Append: `Summarize the structure of these records and explain how to look up a record by its ID.`
+- During bundle preparation, use the pinned tokenizer and template to select the number of complete records
+ whose rendered input is closest to 4,096 and 16,384 tokens. Freeze those messages, including all repetition
+ variants; require lengths within 2% of the target. Record actual token counts for every request.
+- Prefix every user message with `[case <request-id>]` and a newline. IDs are fixed and unique within a run,
+ including warm-up and concurrency requests. Start each request as a fresh conversation. Different markers
+ prevent intentional full-prompt reuse; they do not prove that the shared template prefix was uncached.
+- For retrieval fixtures, use the same record construction at 4K or 16K, with the key/value facts below inserted
+ at the stated fractions of the record list. Freeze the complete resulting messages and record actual lengths.
+ Distractor records must not contain the queried keys or answers.
+
+The warmed expert cache stays enabled and evolves normally. Record reused and freshly processed prompt tokens
+on every request. Shared template-prefix reuse is allowed and must be reported. Unexpected reuse of the user
+payload invalidates that request's performance comparison; preserve the evidence. Do not call these runs
+zero-cache or cold-model measurements.
+
+### Twelve sanity checks
+
+These are small functional checks, not an intelligence ranking, a full coding evaluation, or proof of lossless
+quantization. Score the final answer or parsed API tool call automatically. Do not score reasoning text or use
+an LLM judge. Each case is worth one point, with no partial credit or best-of reruns.
+
+| ID | Fixed task or retrieval construction | Required result |
+| --- | --- | --- |
+| I1 | `Return only a JSON object with ok set to true and count set to the integer 3. No other keys.` | `{"ok":true,"count":3}` |
+| I2 | `Sort these integers in ascending order and remove duplicates: 9, 2, -4, 2, 1. Return only a JSON array.` | `[-4,1,2,9]` |
+| I3 | `Project: Orion. Build: 17. Return only a JSON object with keys project (string) and build (integer).` | `{"project":"Orion","build":17}` |
+| I4 | `Reply with exactly the word READY in uppercase and nothing else.` | `READY` |
+| R1 | 4K records; at 10%, insert `access_code = ORBIT-731`. Ask `What is access_code? Reply only with its value.` | `ORBIT-731` |
+| R2 | 4K records; at 50%, insert `storage_label = CEDAR-284`. Ask `What is storage_label? Reply only with its value.` | `CEDAR-284` |
+| R3 | 16K records; at 90%, insert `archive_code = DELTA-906`. Ask `What is archive_code? Reply only with its value.` | `DELTA-906` |
+| R4 | 16K records; at 10% insert `north = N-124`, at 90% insert `south = S-875`. Ask `Return only a JSON object containing north and south and their values.` | `{"north":"N-124","south":"S-875"}` |
+| T1 | `Use lookup_record to look up record A-17.` | `lookup_record({"id":"A-17"})` |
+| T2 | `Use get_weather for Zagreb in celsius.` | `get_weather({"city":"Zagreb","unit":"celsius"})` |
+| T3 | `Use convert_units to convert 3 km to m.` | `convert_units({"value":3,"from_unit":"km","to_unit":"m"})` |
+| T4 | `Use get_build_status for project Orion, build 17.` | `get_build_status({"project":"Orion","build":17})` |
+
+For retrieval, insert before record index `floor(fraction * record_count)` using zero-based indices. Append the
+question after the records. Each tool case exposes only its named mock function, with an object parameter schema,
+all listed properties required, and `additionalProperties: false`. Properties are strings except T3's `value`
+(`number`) and T4's `build` (`integer`). Use `tool_choice: "auto"`, description `Synthetic benchmark lookup; no
+external action is performed.`, and exactly the names and arguments above. Validate one structured tool call;
+prose that merely describes a call fails. Do not execute tools, contact services, or add tool-result turns.
+
+Exact-text checks strip only surrounding ASCII whitespace. JSON checks parse one complete JSON value, reject
+duplicate keys, extra prose and Markdown fences, and compare values and types; object key order is irrelevant.
+Tool arguments use the same strict parsing; function name, required values, and absence of extra keys must match.
+No text answer is required in a successful tool-call case. Record malformed output, wrong answers, truncation,
+errors, and skipped cases individually. Only a completed, correct response passes. The denominator remains
+`12`, with category denominators `4`; skipped cases also make the suite partial.
+
+## Measurements
+
+Keep the inference host's hardware separate from any client description. Standard latency uses loopback;
+remote/network measurements belong in a separate nonstandard report. Record all inference GPUs, selected device
+indices, dedicated or shared memory, CPU, RAM, storage, OS, driver/runtime versions, power limits when available,
+and other workloads. Use bytes for memory, watts for power, seconds for durations, and tokens/second for rates.
+
+| Metric | Definition and evidence |
+| --- | --- |
+| Prompt tokens/sec | Engine `read_tokens / prompt_s`, using only freshly processed tokens and their engine duration. Cached tokens never enter the numerator. |
+| Decode tokens/sec | Engine `decode_tokens / decode_s`. Include accepted reasoning and answer tokens; exclude rejected speculative drafts. Preserve the exact engine count used for this timer. |
+| TTFT | Client monotonic time from request dispatch to first nonempty reasoning, answer, or generated tool-call delta. Ignore headers, role-only chunks, empty deltas, and keep-alives. Record the delta kind. |
+| First-answer latency | Time from the same dispatch to first nonempty final-answer text or generated tool-call delta. Reasoning alone does not count. Null if no answer or tool call arrives. |
+| Total duration | Client dispatch through stream completion, error, or cancellation, including queueing and prompt processing. |
+| Concurrent aggregate tokens/sec | For one round, total accepted output tokens from completed or capped requests divided by wall time from first dispatch to last termination in that round. Keep failed-request time in that wall time; report failures alongside the rate. |
+
+Retain API `output_tokens` separately from engine `decode_tokens`. If the counts differ, explain the engine
+accounting and preserve its raw timing lines; do not assume `predicted_n` is always the count used by
+`predicted_per_second`. Reasoning/answer token splits are optional if the engine cannot provide them reliably.
+Streaming chunks are never tokens. Do not use characters, words, or chunks to fill missing token counts, or
+divide output tokens by total request time and label the result decode speed.
+
+Use engine counts and durations for the two primary throughput rates. Client estimates, if included as extra
+data, must have a distinct name and source and cannot enter those summaries. When a required count or timing is
+unavailable, publish `null` with a reason. Do not divide by zero or publish NaN/infinity. For a cancelled prefill,
+retain actual processed tokens if known; do not invent complete-prompt timing.
+
+Compute medians, minima, maxima, and sample counts for each performance case separately. Use completed requests
+and requests that normally reach the specified output cap; preserve their different finish reasons. Exclude
+errors, cancellations, unexpected cache reuse, and missing/invalid measurements, and list the exclusions. A cap
+is an expected performance boundary but a failed sanity answer. Never hide early EOS, failed requests, or
+slow samples. A partial sample set keeps its actual `n`; do not imply that three repeats succeeded.
+
+Do not pool prompt sizes, quantizations, configurations, or concurrency levels. Optional concurrency has per-request
+records plus per-round aggregates and separate summaries. Three rounds are too few to advertise a stable p95.
+Quality scores remain separate from throughput; an incorrect answer must not disappear from the report.
+
+## Result contract
+
+The following is the **normative documentation for the future JSON format**, not an implementation or a schema
+validator. All named keys are required unless explicitly optional. Unknown extra fields must be under
+`extensions`; a consumer must reject unknown schema/protocol versions instead of silently combining results.
+
+Use UTF-8 JSON, finite numbers, nonnegative integer token/request counts, UTC ISO 8601 timestamps, and lowercase
+64-character SHA-256 strings. Paths are relative to the report directory, without absolute paths or `..`.
+Unavailable values are `null`, not zero, empty strings, or `"N/A"`. Every null has a JSON Pointer entry in `missing`
+explaining why, including not-applicable values. Empty arrays mean genuinely no items; unknown arrays are null.
+
+### Top-level fields
+
+| Field | Required content |
+| --- | --- |
+| `format`, `schema_version` | `"strata-standard-benchmark"`, `1`; distinct from the current runner's legacy `schema: 1`. |
+| `protocol` | `id`, `git_commit`, `profile`, `corpus_sha256`, `model_card_revision`. Git commits are full immutable IDs. |
+| `run` | Unique string `id`, `started_at`, `status`, `required_duration_s`, `optional_duration_s`, boolean `optional_concurrency`, and sanitized command string `command`. Status is `complete`, `partial`, or `nonconforming`; optional duration is null with a reason when not selected. |
+| `hardware` | `gpus` array of `{device_index, name, memory_bytes, memory_kind, power_limit_w}`; string `cpu`, integer `ram_bytes`, `storage` array of `{kind, model, bus}`, and `background_workloads` array of strings. GPU index is an integer, names/storage attributes are strings. Memory kind is `dedicated` or `shared`; do not count shared VRAM again as extra system RAM. |
+| `software` | `os` as `{name, version}`, `drivers` and `runtime` arrays of `{name, version}`, strings `strata_commit`, `engine_version`, `build_kind`, `runner_commit`, and string-array `build_options`. Build kind is `release` or `source`; names/versions are strings. |
+| `model` | `id`, `revision`, `source_revision`, `quantization`, `files` array of `{path, sha256}`, `tokenizer_revision`, and `custom_pack` (description/preparation reference, or null when absent). Model file paths identify pack-relative filenames, not private machine paths. |
+| `configuration` | Strings `track`, `track_notes`; integers `context_tokens`, `server_capacity`; objects `kv_cache`, `expert_cache`, `offloading`, `prefill`, `mtp`, `calibration`, `experimental_options`; `template` as `{path, sha256}` pointing to the attached template; `sampling` and `chat_template_kwargs` objects. Configuration objects preserve effective engine option names and JSON values. `sampling` contains temperature, top_p, top_k, min_p, presence_penalty, repetition_penalty, and reasoning_effort. The two thinking flags belong in `chat_template_kwargs`; seeds/caps are per request. |
+| `request_counts` | Objects for `warmup`, `performance`, `sanity`, `concurrency`, each with `planned`, `attempted`, `completed`, `truncated`, `failed`, `skipped`. |
+| `requests` | One record per scheduled request, including warm-up, failed and skipped requests; shape below. |
+| `summaries` | Array of `{phase, case_id, concurrency, metric, unit, median, min, max, n, included_request_ids, excluded}`. Each exclusion has `request_id` and `reason`. Zero valid samples means `n: 0`, null statistics, and reasons. |
+| `concurrency_rounds` | Array of `{round, request_ids, wall_s, output_tokens, aggregate_tps, failed}`; empty when the optional stage was not selected. `output_tokens` includes only completed/capped requests; null if any included request lacks a count. |
+| `quality` | `passed`, `total` (12), `categories` with `instruction`, `retrieval`, `tool_call`, each `{passed, total}` (total 4). Per-case verdicts are in `requests`. |
+| `artifacts` | Array of `{kind, path, sha256}` for sanitized configuration, engine logs, and request/output evidence; optional additional provenance. Kind is `config`, `engine_log`, `outputs`, or `provenance`. Hash final sanitized bytes. |
+| `missing` | Map of JSON Pointers to specific reasons for null values. |
+| `extensions` | Optional object for data beyond this contract; never a substitute for required fields. |
+
+Except for typed objects, arrays, numbers and booleans identified above, identity and descriptive fields are
+strings (or null with a reason where unavailable). Empty configuration objects mean no relevant options;
+unknown settings are null. `configuration.track_notes` states setup choices for a default run or modifications
+for a custom run. A warm-up failure aborts the measured stages with explicit skipped records.
+
+`complete` means all required requests were attempted and their outcomes were recorded, including failures;
+it does not mean all checks passed. `partial` means required work was skipped or interrupted. `nonconforming`
+takes precedence for changed profile/workload settings, unsupported defaults, claimed counts without traceable sources, unexpected
+user-payload cache reuse, or missing required identity such as the commit, model files, template, or corpus.
+Unknown ancillary hardware facts are allowed with reasons. A run lacking engine timings can retain other
+measurements with those rates null; it cannot supply complete speed comparisons.
+
+For each phase, `attempted = completed + truncated + failed` and `planned = attempted + skipped`. Planned counts
+are `1`, `9`, `12`, and either `12` or `0` for the optional extension. Omitted optional work is not skipped required
+work. The full `requests` array therefore has 22 or 34 records. All twelve sanity IDs appear even after aborting.
+There are no invisible retries. A later run gets a new ID and preserves the earlier run when discussing failures.
+
+Each summary uses `phase = "performance"` or `"concurrency"`, the exact case ID (or `"short-burst"` for the
+optional stage), and metric `prompt_tps`, `decode_tps`, `ttft_s`, `first_answer_s`, or `total_s`. Units are
+`tokens/s` for rates and `s` for latency. Publish all five summaries per selected performance/concurrency case,
+including null summaries when there are no valid samples. Warm-up and sanity do not enter speed
+summaries. Keep concurrent round aggregates in `concurrency_rounds`; do not average per-client rates to obtain
+aggregate throughput. The website can derive a median/range across those three rounds.
+
+### Request records
+
+The following is a **shape example only** for a request. It contains no measured result and is not a valid
+submission. The enclosing `missing` map must explain every null when adapting it to real evidence.
+
+```json
+{
+ "id": "performance-short-r1",
+ "phase": "performance",
+ "case_id": "short",
+ "repetition": 1,
+ "seed": 1,
+ "max_output_tokens": 512,
+ "concurrency": 1,
+ "round": null,
+ "status": null,
+ "finish_reason": null,
+ "error": null,
+ "tokens": {
+ "input_tokens": null,
+ "cached_tokens": null,
+ "read_tokens": null,
+ "output_tokens": null,
+ "decode_tokens": null
+ },
+ "timings": {
+ "prompt_s": null,
+ "decode_s": null,
+ "ttft_s": null,
+ "first_answer_s": null,
+ "total_s": null
+ },
+ "metrics": {
+ "prompt_tps": null,
+ "decode_tps": null
+ },
+ "sources": {
+ "input_tokens": null,
+ "cached_tokens": null,
+ "read_tokens": null,
+ "output_tokens": null,
+ "decode_tokens": null,
+ "engine_timings": null,
+ "client_timings": null
+ },
+ "first_delta_kind": null,
+ "token_accounting_note": null,
+ "evaluation": null
+}
+```
+
+`phase` is `warmup`, `performance`, `sanity`, or `concurrency`. Repetition is one-based; `round` is `1..3` for
+concurrency, otherwise null. Request IDs are `warmup`, `performance-<case>-r<1..3>`, `sanity-<case>`, or
+`concurrency-r<1..3>-c<1..4>`. Concurrency records use case `short-burst` and repetition equal to the round.
+
+Request `status` is `completed`, `truncated`, `failed`, or `skipped`. Preserve the server's original
+`finish_reason`; reaching the cap maps to `truncated`, HTTP/stream errors, malformed streams and timeout to
+`failed`, and work never dispatched to `skipped`. `error` is null for completed/truncated requests and otherwise
+an object with `kind` and `message`, including a skip reason. No response must be treated as successful merely
+because one delta arrived; require a normal stream end and finish reason.
+
+`sources` identifies the raw API field or engine-log field for each count, the engine timer fields, and the
+client monotonic-clock trace. Link the relevant evidence by request ID in the artifacts. `first_delta_kind` is
+`reasoning`, `answer`, or `tool_call`, or null if none arrived. `token_accounting_note` explains count differences,
+or is null when counts agree. Do not retokenize displayed final text to estimate hidden reasoning tokens.
+
+For sanity requests, `evaluation` is `{ "category": "instruction|retrieval|tool_call", "passed": false,
+"reason": "..." }`, using one actual category, a boolean verdict, and a specific reason. It is null for other
+phases. A false verdict can accompany a normally completed request. Failed, truncated and skipped sanity requests
+also carry a false verdict. Report passed/12 and the outcome counts together so a partial run is obvious.
+
+The `outputs` artifact is JSON Lines keyed by `request_id`. Preserve the exact sanitized messages and tools,
+final answer, structured tool calls, finish reason, raw usage/timings, error, and client event offsets sufficient
+to check TTFT, first-answer latency and total duration. Keep request dispatch at offset zero and record the
+first qualifying delta's type. Reasoning text is optional; its authoritative token count must still contribute
+to throughput. For skipped work record the request ID and skip reason. Logs and output evidence must agree
+with the JSON; derived summaries can always be rebuilt from request records.
+
+## Submit and review
+
+Follow [CONTRIBUTING.md](../CONTRIBUTING.md). During the draft phase, submit pilot data clearly labelled with
+the draft protocol ID. Existing reports remain welcome under the [community guide](COMMUNITY_BENCHMARKS.md).
+
+1. Run one hardware/model/quant/configuration combination and retain every scheduled outcome. Add a separate
+ run for another configuration or baseline; do not select only the best repetitions.
+2. Add `bench/results/<UTC-date>-standard-<hardware>-<run-id>/` containing `README.md`, `results.json`, sanitized
+ configuration, engine logs, and `outputs.jsonl`. Small extra provenance files are allowed. Never add model
+ weights or generated packs. Inspect redaction before publishing and compute artifact hashes afterwards.
+3. The report states the protocol/status, hardware, model/quant, commit/configuration, actual duration, counts,
+ per-case performance, passed/12 sanity score, failures, and limitations. Link raw evidence. Use `not measured`
+ in prose and null with a reason in JSON. Do not paste hypothetical performance numbers into examples.
+4. Open a results-only PR against `Niko1221/Strata:main`. Suggested title:
+ `benchmark: <GPU> / <model> / <quant> / <protocol-id>`. The body links the report, names required versus optional
+ stages, and states any failed, skipped, missing, or nonconforming measurements. Keep engine changes separate.
+5. Maintainers check the exact revision, profile and corpus, artifact hashes, counts, formulas, scores, logs,
+ redaction, and limitations. Changes to evidence after review require another review. Low scores and slow
+ hardware are valid results; invented evidence or changed settings represented as defaults are not.
+
+### Future approved manifest
+
+After activation, use **submission → maintainer review → approved manifest entry**. The website will read
+`bench/approved.json` on the default branch, not scrape PR prose, labels, or an `approved` field supplied by a
+contributor. A maintainer adds an entry only after reviewing the exact artifacts. A merged report alone does not
+mean it is in the approved list. Do not create a populated manifest for this draft.
+
+The proposed manifest has `format: "strata-approved-benchmarks"`, `schema_version: 1`, and an `entries` array.
+Each entry has the following fields; the example has placeholders, no approved result:
+
+```json
+{
+ "run_id": "<reviewed-run-id>",
+ "protocol_id": "<activated-protocol-id>",
+ "results_path": "bench/results/<report-directory>/results.json",
+ "results_sha256": "<sha256-of-reviewed-results-json>",
+ "evidence_commit": "<full-git-commit-containing-reviewed-artifacts>",
+ "review_pr": "<benchmark-pull-request-url>",
+ "reviewed_by": "<maintainer-github-handle>",
+ "reviewed_at": "<utc-iso8601-time>",
+ "verification": "evidence-reviewed",
+ "reproduction_run_ids": []
+}
+```
+
+Resolve the repository-relative result path at `evidence_commit`; verify its hash and the hashes of its
+referenced artifacts before displaying data. The evidence commit is the commit that introduced the reviewed
+report, not the later manifest commit. Only maintainer-reviewed manifest changes count as approval. Changes
+to results require a newly reviewed entry/hash; preserve old records or mark their replacement explicitly.
+Remove a withdrawn entry from the current manifest with the reason in its review PR; Git history retains it.
+
+Approval means **evidence reviewed**, not independent replication or a guarantee of quality. Use
+`verification: "independently-reproduced"` only when another person has run the protocol and supplied the
+linked approved reproduction run IDs; document hardware/configuration differences. Quality failures and partial
+conforming runs can be reviewed and listed, with conspicuous outcome/status labels. Partial runs never appear
+as complete results or gain a full-suite rank. Nonconforming and draft runs stay outside the activated standard
+manifest, with their reports still accessible. A website must preserve these distinctions and missing values.
+
+Existing community reports retain their original formats and are **legacy/nonstandard** for this protocol.
+Do not infer absent fields, relabel their sampling, or retroactively approve them. A new conforming run is needed
+for the standard list. Comparisons must expose protocol, model/quant, build/configuration track, prompt case,
+actual lengths/cache counts, and concurrency; there is no universal combined score.
+
+## Activation checklist
+
+The current runner hardcodes temperature `0`, disables thinking by default, has no twelve-case sanity suite,
+uses changing repository files in its agent workload, and can fall back from token counts to stream-chunk counts.
+It also lacks the finish/first-answer accounting and versioned result contract above. `--think` alone does not
+fix those differences. These are implementation tasks for a later change, not capabilities of the current CLI.
+
+Before removing draft status:
+
+- Implement and freeze the fixtures, profile, runner, strict result validator, and evidence generation. Verify
+ default sampling/template behavior and complete token/timing accounting against the engine.
+- Test complete runs, early EOS, cap truncation, wrong answers, malformed JSON/tool calls, duplicate JSON keys,
+ missing usage/timings, empty deltas, timeouts, cancellation, errors after a first delta, and server death.
+- Test count identities, summary reconstruction, shared-prefix versus unexpected payload reuse, insufficient
+ context, unsupported defaults, optional concurrency with queueing, and zero/invalid timing denominators.
+- Check hashes/provenance, null reasons, redaction, unknown format versions, draft/nonconforming exclusion, and
+ changed artifacts after review. Independently recompute sanity scores from saved final outputs.
+- Run a measured pilot on at least two supported hardware configurations, including a slower system. Publish
+ actual duration, caps reached, failures, and sanity outcomes. Adjust the candidate workload if needed, then
+ freeze a new protocol ID before collecting comparable standard results. Do not promise a universal runtime.
+
+For this documentation change, validate links and anchors, issue-form YAML, JSON example syntax, request-count
+arithmetic, and agreement between these rules and contribution templates. No runtime behavior is changed.