Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Add CONTRIBUTING.md and the draft standard benchmark protocol (owner text, unchanged)
Niko1221e9e08adb3731
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdnew file mode 100644index 000000000..5afaee4ac--- /dev/null+++ b/CONTRIBUTING.md@@ -0,0 +1,116 @@+# Contributing to Strata++**AI assistance is welcome. The contributor is responsible for understanding the change and supplying real evidence.**++These rules apply to issues, pull requests, benchmark reports, and the agents helping prepare them.+You should be able to explain every changed line and answer review questions about it.++## Before starting++1. Search [issues](https://github.com/Niko1221/Strata/issues) and+ [pull requests](https://github.com/Niko1221/Strata/pulls), including closed discussions. Add new evidence to an+ existing thread when it covers your problem.+2. Read [AGENTS.md](AGENTS.md) and the documentation for the component you want to change.+3. Get maintainer agreement on scope before substantial features, refactors, architecture changes, or changes to+ defaults. Describe the user problem, proposed behavior, and tradeoffs in an issue first. Agreement on scope+ does not guarantee a merge. Focused fixes and benchmark reports can go straight to a PR.+4. Keep one problem per PR. Update that PR after feedback; do not open competing attempts for the same change.++Do not resubmit a declined proposal under a different title or through another agent. If new evidence changes+the situation, bring it to the original discussion and let a maintainer decide whether to reopen it.++## Report a problem people can investigate++Use the matching [issue form](https://github.com/Niko1221/Strata/issues/new/choose). A bug report should include:++- The smallest steps or command that reproduce the problem, and the expected and actual behavior.+- Strata version or commit, OS, GPU and VRAM, and relevant model, quantization, and configuration.+- The exact error and relevant logs, with credentials and private content removed.+- Whether you reproduced it yourself. Label a finding based only on reading code or a scanner as+ **unconfirmed static analysis**, and explain the suspected path without presenting it as an observed failure.++Do not invent reproduction steps, errors, affected hardware, or severity. If you cannot reproduce something,+say what you inspected and what would verify the hypothesis. Feature requests should explain a concrete user+need and the current workaround. Questions should say what you tried and where you got stuck.++Example structure, with placeholders to replace with actual observations:++```text+Problem: <one observed behavior>+Environment: <commit, OS, GPU/VRAM, model/quant, relevant settings>+Reproduce: <exact command or short sequence>+Expected: <documented or intended behavior>+Actual: <observed behavior and exact error>+Evidence: <sanitized log attachment>+Reproduction status: <reproduced locally OR unconfirmed static analysis>+```++## Prepare a focused pull request++Lead with the problem and resulting behavior. Explain why the approach is needed, the important limits, and+any compatibility effects. Link the issue and scope agreement when applicable. A file-by-file narration or+an AI-generated summary of the diff is not a substitute for this explanation.++- Keep unrelated formatting, cleanup, dependency changes, and speculative abstractions out of the PR.+- Add documentation where it changes what a user or developer needs to know. Avoid repeated summaries,+ speculative roadmaps, and generated documentation with no actionable information.+- Preserve existing behavior outside the intended change. New features are opt-in; follow the default-path+ and performance requirements in [AGENTS.md](AGENTS.md).+- Read the complete diff, including generated files, before submitting. Remove debugging output, secrets,+ private paths, and accidental files.+- Prefer a small relevant regression check over tests that only restate the implementation. Documentation-only+ changes need link, example, and consistency checks; they do not require GPU builds.++## Show the validation you actually performed++List exact commands, the revision and environment tested, and observed outcomes. Keep **passed**, **failed**,+and **not run** separate. Include relevant output or a log attachment. A command you suggest running is not+evidence that it passed; a CI run on a different revision is not validation of the submitted code.++Build every backend a code change touches, including CUDA, HIP, and SYCL where applicable, before requesting+review. For changes that need hardware execution, report that execution separately from compilation. A mock,+static check, or successful build does not establish hardware correctness. If you lack a required environment,+keep the PR as a draft, identify the missing checks, and ask for targeted help in the same thread. Do not claim+compatibility you have not tested.++Performance claims need before/after runs on the same hardware and workload, with every changed setting+identified. Preserve the individual runs, failures, and variability. Follow the+[community measurement guide](docs/COMMUNITY_BENCHMARKS.md); standard submissions will use the+[benchmark protocol](docs/BENCHMARK_PROTOCOL.md) once it is activated. One fast run does not justify changing a+default or claiming a speedup for other machines.++Example PR body; placeholders are not test results:++```text+Problem and result: <trigger; behavior before and after the change>+Related discussion: <issue URL and scope agreement, when required>+Approach: <why this fix; relevant compatibility effects>+Validation:+ Passed: <exact command, revision/environment, observed result, log>+ Failed: <exact command and failure, or none observed>+ Not run: <check, reason, and what remains unverified>+Limits: <known limitations or remaining work>+```++## Rules for AI-assisted work++Use agents to help investigate, implement, and explain work that you can review. Before submitting, verify+their claims against the source, actual command output, or a cited primary reference.++Never submit fabricated measurements, invented test results, fake logs, or copied claims you have not checked.+Do not turn a repository scan into a batch of speculative issues or PRs. Consolidate related findings, establish+their effect, and submit the useful evidence. Do not present an agent's confidence or another model's approval+as validation. Keep review replies concise and address the requested change or evidence directly.++Maintainers may request a smaller change, return a PR to draft, or close a submission that lacks evidence,+duplicates existing work, or ignores agreed scope. A smaller, understandable contribution with honest limits+is easier to review and maintain.++## Share a benchmark++A results-only PR should contain the human report, raw measurements, sanitized configuration, and supporting+logs. Keep it separate from engine changes and include poor results and failed requests.++The [current community benchmark guide](docs/COMMUNITY_BENCHMARKS.md) remains available for measurements made+with today's tools. The [new protocol](docs/BENCHMARK_PROTOCOL.md) defines the draft standard and future approval+workflow. Its draft status must remain visible: existing reports are not automatically standard or approved.diff --git a/docs/BENCHMARK_PROTOCOL.md b/docs/BENCHMARK_PROTOCOL.mdnew file mode 100644index 000000000..350bdad2f--- /dev/null+++ b/docs/BENCHMARK_PROTOCOL.md@@ -0,0 +1,409 @@+# Strata standard benchmark protocol++**Status: draft for implementation and pilot validation.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.+Result format: `strata-standard-benchmark`, schema version `1`.++This document specifies a repeatable community benchmark and the data a future website can consume. The+required suite targets **20–30 minutes after model loading**. This duration has not been measured for this+protocol. Faster machines may finish sooner and slower machines may take longer; everyone runs the same work.+Downloads, installation, compilation, and model loading are excluded. Report their duration separately if measured.++**Today's `BENCH.bat`, `./bench.sh`, and `tools/strata_bench.py` do not implement this protocol.** Continue using+the [community guide](COMMUNITY_BENCHMARKS.md) for current reports. Do not rename their output or add a protocol+ID to make it appear standard. Runner implementation, frozen fixtures, validation, and a measured pilot must+precede activation. The approved manifest described here does not yet exist.++- [Profile and settings](#profile-and-settings)+- [Workload and scoring](#workload-and-scoring)+- [Measurements](#measurements)+- [Result contract](#result-contract)+- [Submit and review](#submit-and-review)+- [Activation checklist](#activation-checklist)++## Profile and settings++The first profile is `qwen3.8-flash-next-thinking`: the original **Qwen/Qwen3.8-Flash-Next** weights and their+quantizations. Record the quantization repository and immutable revision, source model revision, every model+file's name and SHA-256, and any pack preparation or custom expert profile. Fine-tuned, Coder, Swift, or other+derivative models require a separate profile. Results for different models, quants, builds, or configurations+must remain distinguishable.++Use the published [Qwen model-card defaults](https://huggingface.co/Qwen/Qwen3.8-Flash-Next):++| Setting | Required value |+| --- | --- |+| `chat_template_kwargs.enable_thinking` | `true` |+| `chat_template_kwargs.preserve_thinking` | `true` |+| `reasoning_effort` | `"xhigh"` |+| `temperature` | `1.0` |+| `top_p` / `top_k` | `0.95` / `20` |+| `min_p` | `0.0` |+| `presence_penalty` / `repetition_penalty` | `0.0` / `1.0` |++Pin the tokenizer revision, effective chat-template file and SHA-256, and model-card revision used to define+the profile. Attach the effective template as a provenance artifact. The pilot must verify that the shipped+template matches the pinned upstream behavior, including+tool formatting. There is no extra benchmark system prompt. Tools are supplied only for the tool-call cases.+Use streaming chat completions and request usage data.++The future runner must explicitly request these settings and verify the resolved server configuration. UI,+saved chat, and configuration defaults must not override them. Disable custom reasoning budgets, automatic+effort changes, and additional stop strings. Unsupported settings make a run nonconforming; do not silently+substitute greedy sampling or disable thinking. Output caps and fixed seeds below are benchmark constraints,+not claims to use Qwen's full recommended output allowance. A fixed seed does not guarantee identical outputs+across hardware or engine versions.++Use one installed configuration per run. Record effective context, KV and expert cache settings, offloading,+prefill, MTP, calibration, experimental options, and server capacity. Use `configuration.track = "default"` for+an unmodified release's setup-generated settings; identify the release and setup choices. Use `"custom"` for+tuning, experimental options, or modified builds and explain the changes. Never merge these tracks into one+headline result. A claimed improvement requires a separate, same-machine baseline.++## Workload and scoring++Run the following stages in order through a dedicated server on the inference machine, using loopback HTTP.+Start with a newly loaded engine, close other inference clients, and record unavoidable background work.+Use the same configuration throughout the required stages. Include warm-up time in `required_duration_s`,+but exclude warm-up requests from performance and quality summaries.++| Stage | Cases and order | Repetitions | Output cap | Requests |+| --- | --- | ---: | ---: | ---: |+| Warm-up | `warmup`, short-prompt fixture | 1 | 128 | 1, excluded |+| Performance | `short`, `prompt-4k`, `prompt-16k`; all three in each repetition | 3 | 512 | 9 |+| Sanity | `I1`–`I4`, `R1`–`R4`, `T1`–`T4` | 1 each | 2,048 | 12 |+| Optional concurrency | Four short-prompt requests released together, then wait for the entire round | 3 rounds | 512 each | 12, separate |++The required suite schedules **21 measured requests plus one warm-up** at concurrency `1`. The optional+extension schedules another `12` at concurrency `4` and has its own duration and summaries. Record the server's+configured capacity: four clients may queue on a server with fewer than four slots. Do not describe that as+four-way parallel decoding or silently change the server configuration. Testing another capacity is another run.++Use seed `1` for warm-up; performance repetitions use seeds `1`, `2`, and `3`. Sanity cases use seed `1`.+Optional round `r` uses seeds `4 * (r - 1) + j` for client `j` in `1..4`. Preserve early EOS; do not force more+output to improve a rate. The caps include reasoning and final output together. Do not add a forced-answer+instruction or a continuation request when a case reaches its cap.++Apply a fixed 1,800-second wall-clock timeout per request, including queueing; cancel timed-out work and record+it as failed. This is a safeguard, not a 30-minute suite deadline. Stop and mark remaining cases skipped if the+engine dies or the user cancels. Do not retry failed requests inside a run or reduce the workload to fit hardware.++### Frozen fixtures++The runner implementation must commit a fixture bundle before the pilot. It must contain exact messages, tool+schemas, expected outputs, case IDs, and validation rules. Hash the bundle's files with a SHA-256 manifest and+store that manifest's hash as `protocol.corpus_sha256`. Record the protocol Git commit. Changes to prompts,+order, scoring, caps, or defaults require a new protocol ID; schema changes use a separate schema version.++Use synthetic data only. Do not read the checkout's current README or other changing project files into prompts.+Build the candidate bundle as follows; these are construction instructions, not runtime adaptation:++- The short task is: `Explain how a hash table finds a value, including collisions and one small example.`+- Long performance payloads are records of the form+ `record000001: alpha beta gamma delta epsilon zeta eta theta.` with sequential six-digit IDs and LF newlines.+ Append: `Summarize the structure of these records and explain how to look up a record by its ID.`+- During bundle preparation, use the pinned tokenizer and template to select the number of complete records+ whose rendered input is closest to 4,096 and 16,384 tokens. Freeze those messages, including all repetition+ variants; require lengths within 2% of the target. Record actual token counts for every request.+- Prefix every user message with `[case <request-id>]` and a newline. IDs are fixed and unique within a run,+ including warm-up and concurrency requests. Start each request as a fresh conversation. Different markers+ prevent intentional full-prompt reuse; they do not prove that the shared template prefix was uncached.+- For retrieval fixtures, use the same record construction at 4K or 16K, with the key/value facts below inserted+ at the stated fractions of the record list. Freeze the complete resulting messages and record actual lengths.+ Distractor records must not contain the queried keys or answers.++The warmed expert cache stays enabled and evolves normally. Record reused and freshly processed prompt tokens+on every request. Shared template-prefix reuse is allowed and must be reported. Unexpected reuse of the user+payload invalidates that request's performance comparison; preserve the evidence. Do not call these runs+zero-cache or cold-model measurements.++### Twelve sanity checks++These are small functional checks, not an intelligence ranking, a full coding evaluation, or proof of lossless+quantization. Score the final answer or parsed API tool call automatically. Do not score reasoning text or use+an LLM judge. Each case is worth one point, with no partial credit or best-of reruns.++| ID | Fixed task or retrieval construction | Required result |+| --- | --- | --- |+| I1 | `Return only a JSON object with ok set to true and count set to the integer 3. No other keys.` | `{"ok":true,"count":3}` |+| I2 | `Sort these integers in ascending order and remove duplicates: 9, 2, -4, 2, 1. Return only a JSON array.` | `[-4,1,2,9]` |+| I3 | `Project: Orion. Build: 17. Return only a JSON object with keys project (string) and build (integer).` | `{"project":"Orion","build":17}` |+| I4 | `Reply with exactly the word READY in uppercase and nothing else.` | `READY` |+| R1 | 4K records; at 10%, insert `access_code = ORBIT-731`. Ask `What is access_code? Reply only with its value.` | `ORBIT-731` |+| R2 | 4K records; at 50%, insert `storage_label = CEDAR-284`. Ask `What is storage_label? Reply only with its value.` | `CEDAR-284` |+| R3 | 16K records; at 90%, insert `archive_code = DELTA-906`. Ask `What is archive_code? Reply only with its value.` | `DELTA-906` |+| R4 | 16K records; at 10% insert `north = N-124`, at 90% insert `south = S-875`. Ask `Return only a JSON object containing north and south and their values.` | `{"north":"N-124","south":"S-875"}` |+| T1 | `Use lookup_record to look up record A-17.` | `lookup_record({"id":"A-17"})` |+| T2 | `Use get_weather for Zagreb in celsius.` | `get_weather({"city":"Zagreb","unit":"celsius"})` |+| T3 | `Use convert_units to convert 3 km to m.` | `convert_units({"value":3,"from_unit":"km","to_unit":"m"})` |+| T4 | `Use get_build_status for project Orion, build 17.` | `get_build_status({"project":"Orion","build":17})` |++For retrieval, insert before record index `floor(fraction * record_count)` using zero-based indices. Append the+question after the records. Each tool case exposes only its named mock function, with an object parameter schema,+all listed properties required, and `additionalProperties: false`. Properties are strings except T3's `value`+(`number`) and T4's `build` (`integer`). Use `tool_choice: "auto"`, description `Synthetic benchmark lookup; no+external action is performed.`, and exactly the names and arguments above. Validate one structured tool call;+prose that merely describes a call fails. Do not execute tools, contact services, or add tool-result turns.++Exact-text checks strip only surrounding ASCII whitespace. JSON checks parse one complete JSON value, reject+duplicate keys, extra prose and Markdown fences, and compare values and types; object key order is irrelevant.+Tool arguments use the same strict parsing; function name, required values, and absence of extra keys must match.+No text answer is required in a successful tool-call case. Record malformed output, wrong answers, truncation,+errors, and skipped cases individually. Only a completed, correct response passes. The denominator remains+`12`, with category denominators `4`; skipped cases also make the suite partial.++## Measurements++Keep the inference host's hardware separate from any client description. Standard latency uses loopback;+remote/network measurements belong in a separate nonstandard report. Record all inference GPUs, selected device+indices, dedicated or shared memory, CPU, RAM, storage, OS, driver/runtime versions, power limits when available,+and other workloads. Use bytes for memory, watts for power, seconds for durations, and tokens/second for rates.++| Metric | Definition and evidence |+| --- | --- |+| Prompt tokens/sec | Engine `read_tokens / prompt_s`, using only freshly processed tokens and their engine duration. Cached tokens never enter the numerator. |+| Decode tokens/sec | Engine `decode_tokens / decode_s`. Include accepted reasoning and answer tokens; exclude rejected speculative drafts. Preserve the exact engine count used for this timer. |+| TTFT | Client monotonic time from request dispatch to first nonempty reasoning, answer, or generated tool-call delta. Ignore headers, role-only chunks, empty deltas, and keep-alives. Record the delta kind. |+| First-answer latency | Time from the same dispatch to first nonempty final-answer text or generated tool-call delta. Reasoning alone does not count. Null if no answer or tool call arrives. |+| Total duration | Client dispatch through stream completion, error, or cancellation, including queueing and prompt processing. |+| Concurrent aggregate tokens/sec | For one round, total accepted output tokens from completed or capped requests divided by wall time from first dispatch to last termination in that round. Keep failed-request time in that wall time; report failures alongside the rate. |++Retain API `output_tokens` separately from engine `decode_tokens`. If the counts differ, explain the engine+accounting and preserve its raw timing lines; do not assume `predicted_n` is always the count used by+`predicted_per_second`. Reasoning/answer token splits are optional if the engine cannot provide them reliably.+Streaming chunks are never tokens. Do not use characters, words, or chunks to fill missing token counts, or+divide output tokens by total request time and label the result decode speed.++Use engine counts and durations for the two primary throughput rates. Client estimates, if included as extra+data, must have a distinct name and source and cannot enter those summaries. When a required count or timing is+unavailable, publish `null` with a reason. Do not divide by zero or publish NaN/infinity. For a cancelled prefill,+retain actual processed tokens if known; do not invent complete-prompt timing.++Compute medians, minima, maxima, and sample counts for each performance case separately. Use completed requests+and requests that normally reach the specified output cap; preserve their different finish reasons. Exclude+errors, cancellations, unexpected cache reuse, and missing/invalid measurements, and list the exclusions. A cap+is an expected performance boundary but a failed sanity answer. Never hide early EOS, failed requests, or+slow samples. A partial sample set keeps its actual `n`; do not imply that three repeats succeeded.++Do not pool prompt sizes, quantizations, configurations, or concurrency levels. Optional concurrency has per-request+records plus per-round aggregates and separate summaries. Three rounds are too few to advertise a stable p95.+Quality scores remain separate from throughput; an incorrect answer must not disappear from the report.++## Result contract++The following is the **normative documentation for the future JSON format**, not an implementation or a schema+validator. All named keys are required unless explicitly optional. Unknown extra fields must be under+`extensions`; a consumer must reject unknown schema/protocol versions instead of silently combining results.++Use UTF-8 JSON, finite numbers, nonnegative integer token/request counts, UTC ISO 8601 timestamps, and lowercase+64-character SHA-256 strings. Paths are relative to the report directory, without absolute paths or `..`.+Unavailable values are `null`, not zero, empty strings, or `"N/A"`. Every null has a JSON Pointer entry in `missing`+explaining why, including not-applicable values. Empty arrays mean genuinely no items; unknown arrays are null.++### Top-level fields++| Field | Required content |+| --- | --- |+| `format`, `schema_version` | `"strata-standard-benchmark"`, `1`; distinct from the current runner's legacy `schema: 1`. |+| `protocol` | `id`, `git_commit`, `profile`, `corpus_sha256`, `model_card_revision`. Git commits are full immutable IDs. |+| `run` | Unique string `id`, `started_at`, `status`, `required_duration_s`, `optional_duration_s`, boolean `optional_concurrency`, and sanitized command string `command`. Status is `complete`, `partial`, or `nonconforming`; optional duration is null with a reason when not selected. |+| `hardware` | `gpus` array of `{device_index, name, memory_bytes, memory_kind, power_limit_w}`; string `cpu`, integer `ram_bytes`, `storage` array of `{kind, model, bus}`, and `background_workloads` array of strings. GPU index is an integer, names/storage attributes are strings. Memory kind is `dedicated` or `shared`; do not count shared VRAM again as extra system RAM. |+| `software` | `os` as `{name, version}`, `drivers` and `runtime` arrays of `{name, version}`, strings `strata_commit`, `engine_version`, `build_kind`, `runner_commit`, and string-array `build_options`. Build kind is `release` or `source`; names/versions are strings. |+| `model` | `id`, `revision`, `source_revision`, `quantization`, `files` array of `{path, sha256}`, `tokenizer_revision`, and `custom_pack` (description/preparation reference, or null when absent). Model file paths identify pack-relative filenames, not private machine paths. |+| `configuration` | Strings `track`, `track_notes`; integers `context_tokens`, `server_capacity`; objects `kv_cache`, `expert_cache`, `offloading`, `prefill`, `mtp`, `calibration`, `experimental_options`; `template` as `{path, sha256}` pointing to the attached template; `sampling` and `chat_template_kwargs` objects. Configuration objects preserve effective engine option names and JSON values. `sampling` contains temperature, top_p, top_k, min_p, presence_penalty, repetition_penalty, and reasoning_effort. The two thinking flags belong in `chat_template_kwargs`; seeds/caps are per request. |+| `request_counts` | Objects for `warmup`, `performance`, `sanity`, `concurrency`, each with `planned`, `attempted`, `completed`, `truncated`, `failed`, `skipped`. |+| `requests` | One record per scheduled request, including warm-up, failed and skipped requests; shape below. |+| `summaries` | Array of `{phase, case_id, concurrency, metric, unit, median, min, max, n, included_request_ids, excluded}`. Each exclusion has `request_id` and `reason`. Zero valid samples means `n: 0`, null statistics, and reasons. |+| `concurrency_rounds` | Array of `{round, request_ids, wall_s, output_tokens, aggregate_tps, failed}`; empty when the optional stage was not selected. `output_tokens` includes only completed/capped requests; null if any included request lacks a count. |+| `quality` | `passed`, `total` (12), `categories` with `instruction`, `retrieval`, `tool_call`, each `{passed, total}` (total 4). Per-case verdicts are in `requests`. |+| `artifacts` | Array of `{kind, path, sha256}` for sanitized configuration, engine logs, and request/output evidence; optional additional provenance. Kind is `config`, `engine_log`, `outputs`, or `provenance`. Hash final sanitized bytes. |+| `missing` | Map of JSON Pointers to specific reasons for null values. |+| `extensions` | Optional object for data beyond this contract; never a substitute for required fields. |++Except for typed objects, arrays, numbers and booleans identified above, identity and descriptive fields are+strings (or null with a reason where unavailable). Empty configuration objects mean no relevant options;+unknown settings are null. `configuration.track_notes` states setup choices for a default run or modifications+for a custom run. A warm-up failure aborts the measured stages with explicit skipped records.++`complete` means all required requests were attempted and their outcomes were recorded, including failures;+it does not mean all checks passed. `partial` means required work was skipped or interrupted. `nonconforming`+takes precedence for changed profile/workload settings, unsupported defaults, claimed counts without traceable sources, unexpected+user-payload cache reuse, or missing required identity such as the commit, model files, template, or corpus.+Unknown ancillary hardware facts are allowed with reasons. A run lacking engine timings can retain other+measurements with those rates null; it cannot supply complete speed comparisons.++For each phase, `attempted = completed + truncated + failed` and `planned = attempted + skipped`. Planned counts+are `1`, `9`, `12`, and either `12` or `0` for the optional extension. Omitted optional work is not skipped required+work. The full `requests` array therefore has 22 or 34 records. All twelve sanity IDs appear even after aborting.+There are no invisible retries. A later run gets a new ID and preserves the earlier run when discussing failures.++Each summary uses `phase = "performance"` or `"concurrency"`, the exact case ID (or `"short-burst"` for the+optional stage), and metric `prompt_tps`, `decode_tps`, `ttft_s`, `first_answer_s`, or `total_s`. Units are+`tokens/s` for rates and `s` for latency. Publish all five summaries per selected performance/concurrency case,+including null summaries when there are no valid samples. Warm-up and sanity do not enter speed+summaries. Keep concurrent round aggregates in `concurrency_rounds`; do not average per-client rates to obtain+aggregate throughput. The website can derive a median/range across those three rounds.++### Request records++The following is a **shape example only** for a request. It contains no measured result and is not a valid+submission. The enclosing `missing` map must explain every null when adapting it to real evidence.++```json+{+ "id": "performance-short-r1",+ "phase": "performance",+ "case_id": "short",+ "repetition": 1,+ "seed": 1,+ "max_output_tokens": 512,+ "concurrency": 1,+ "round": null,+ "status": null,+ "finish_reason": null,+ "error": null,+ "tokens": {+ "input_tokens": null,+ "cached_tokens": null,+ "read_tokens": null,+ "output_tokens": null,+ "decode_tokens": null+ },+ "timings": {+ "prompt_s": null,+ "decode_s": null,+ "ttft_s": null,+ "first_answer_s": null,+ "total_s": null+ },+ "metrics": {+ "prompt_tps": null,+ "decode_tps": null+ },+ "sources": {+ "input_tokens": null,+ "cached_tokens": null,+ "read_tokens": null,+ "output_tokens": null,+ "decode_tokens": null,+ "engine_timings": null,+ "client_timings": null+ },+ "first_delta_kind": null,+ "token_accounting_note": null,+ "evaluation": null+}+```++`phase` is `warmup`, `performance`, `sanity`, or `concurrency`. Repetition is one-based; `round` is `1..3` for+concurrency, otherwise null. Request IDs are `warmup`, `performance-<case>-r<1..3>`, `sanity-<case>`, or+`concurrency-r<1..3>-c<1..4>`. Concurrency records use case `short-burst` and repetition equal to the round.++Request `status` is `completed`, `truncated`, `failed`, or `skipped`. Preserve the server's original+`finish_reason`; reaching the cap maps to `truncated`, HTTP/stream errors, malformed streams and timeout to+`failed`, and work never dispatched to `skipped`. `error` is null for completed/truncated requests and otherwise+an object with `kind` and `message`, including a skip reason. No response must be treated as successful merely+because one delta arrived; require a normal stream end and finish reason.++`sources` identifies the raw API field or engine-log field for each count, the engine timer fields, and the+client monotonic-clock trace. Link the relevant evidence by request ID in the artifacts. `first_delta_kind` is+`reasoning`, `answer`, or `tool_call`, or null if none arrived. `token_accounting_note` explains count differences,+or is null when counts agree. Do not retokenize displayed final text to estimate hidden reasoning tokens.++For sanity requests, `evaluation` is `{ "category": "instruction|retrieval|tool_call", "passed": false,+"reason": "..." }`, using one actual category, a boolean verdict, and a specific reason. It is null for other+phases. A false verdict can accompany a normally completed request. Failed, truncated and skipped sanity requests+also carry a false verdict. Report passed/12 and the outcome counts together so a partial run is obvious.++The `outputs` artifact is JSON Lines keyed by `request_id`. Preserve the exact sanitized messages and tools,+final answer, structured tool calls, finish reason, raw usage/timings, error, and client event offsets sufficient+to check TTFT, first-answer latency and total duration. Keep request dispatch at offset zero and record the+first qualifying delta's type. Reasoning text is optional; its authoritative token count must still contribute+to throughput. For skipped work record the request ID and skip reason. Logs and output evidence must agree+with the JSON; derived summaries can always be rebuilt from request records.++## Submit and review++Follow [CONTRIBUTING.md](../CONTRIBUTING.md). During the draft phase, submit pilot data clearly labelled with+the draft protocol ID. Existing reports remain welcome under the [community guide](COMMUNITY_BENCHMARKS.md).++1. Run one hardware/model/quant/configuration combination and retain every scheduled outcome. Add a separate+ run for another configuration or baseline; do not select only the best repetitions.+2. Add `bench/results/<UTC-date>-standard-<hardware>-<run-id>/` containing `README.md`, `results.json`, sanitized+ configuration, engine logs, and `outputs.jsonl`. Small extra provenance files are allowed. Never add model+ weights or generated packs. Inspect redaction before publishing and compute artifact hashes afterwards.+3. The report states the protocol/status, hardware, model/quant, commit/configuration, actual duration, counts,+ per-case performance, passed/12 sanity score, failures, and limitations. Link raw evidence. Use `not measured`+ in prose and null with a reason in JSON. Do not paste hypothetical performance numbers into examples.+4. Open a results-only PR against `Niko1221/Strata:main`. Suggested title:+ `benchmark: <GPU> / <model> / <quant> / <protocol-id>`. The body links the report, names required versus optional+ stages, and states any failed, skipped, missing, or nonconforming measurements. Keep engine changes separate.+5. Maintainers check the exact revision, profile and corpus, artifact hashes, counts, formulas, scores, logs,+ redaction, and limitations. Changes to evidence after review require another review. Low scores and slow+ hardware are valid results; invented evidence or changed settings represented as defaults are not.++### Future approved manifest++After activation, use **submission → maintainer review → approved manifest entry**. The website will read+`bench/approved.json` on the default branch, not scrape PR prose, labels, or an `approved` field supplied by a+contributor. A maintainer adds an entry only after reviewing the exact artifacts. A merged report alone does not+mean it is in the approved list. Do not create a populated manifest for this draft.++The proposed manifest has `format: "strata-approved-benchmarks"`, `schema_version: 1`, and an `entries` array.+Each entry has the following fields; the example has placeholders, no approved result:++```json+{+ "run_id": "<reviewed-run-id>",+ "protocol_id": "<activated-protocol-id>",+ "results_path": "bench/results/<report-directory>/results.json",+ "results_sha256": "<sha256-of-reviewed-results-json>",+ "evidence_commit": "<full-git-commit-containing-reviewed-artifacts>",+ "review_pr": "<benchmark-pull-request-url>",+ "reviewed_by": "<maintainer-github-handle>",+ "reviewed_at": "<utc-iso8601-time>",+ "verification": "evidence-reviewed",+ "reproduction_run_ids": []+}+```++Resolve the repository-relative result path at `evidence_commit`; verify its hash and the hashes of its+referenced artifacts before displaying data. The evidence commit is the commit that introduced the reviewed+report, not the later manifest commit. Only maintainer-reviewed manifest changes count as approval. Changes+to results require a newly reviewed entry/hash; preserve old records or mark their replacement explicitly.+Remove a withdrawn entry from the current manifest with the reason in its review PR; Git history retains it.++Approval means **evidence reviewed**, not independent replication or a guarantee of quality. Use+`verification: "independently-reproduced"` only when another person has run the protocol and supplied the+linked approved reproduction run IDs; document hardware/configuration differences. Quality failures and partial+conforming runs can be reviewed and listed, with conspicuous outcome/status labels. Partial runs never appear+as complete results or gain a full-suite rank. Nonconforming and draft runs stay outside the activated standard+manifest, with their reports still accessible. A website must preserve these distinctions and missing values.++Existing community reports retain their original formats and are **legacy/nonstandard** for this protocol.+Do not infer absent fields, relabel their sampling, or retroactively approve them. A new conforming run is needed+for the standard list. Comparisons must expose protocol, model/quant, build/configuration track, prompt case,+actual lengths/cache counts, and concurrency; there is no universal combined score.++## Activation checklist++The current runner hardcodes temperature `0`, disables thinking by default, has no twelve-case sanity suite,+uses changing repository files in its agent workload, and can fall back from token counts to stream-chunk counts.+It also lacks the finish/first-answer accounting and versioned result contract above. `--think` alone does not+fix those differences. These are implementation tasks for a later change, not capabilities of the current CLI.++Before removing draft status:++- Implement and freeze the fixtures, profile, runner, strict result validator, and evidence generation. Verify+ default sampling/template behavior and complete token/timing accounting against the engine.+- Test complete runs, early EOS, cap truncation, wrong answers, malformed JSON/tool calls, duplicate JSON keys,+ missing usage/timings, empty deltas, timeouts, cancellation, errors after a first delta, and server death.+- Test count identities, summary reconstruction, shared-prefix versus unexpected payload reuse, insufficient+ context, unsupported defaults, optional concurrency with queueing, and zero/invalid timing denominators.+- Check hashes/provenance, null reasons, redaction, unknown format versions, draft/nonconforming exclusion, and+ changed artifacts after review. Independently recompute sanity scores from saved final outputs.+- Run a measured pilot on at least two supported hardware configurations, including a slower system. Publish+ actual duration, caps reached, failures, and sanity outcomes. Adjust the candidate workload if needed, then+ freeze a new protocol ID before collecting comparable standard results. Do not promise a universal runtime.++For this documentation change, validate links and anchors, issue-form YAML, JSON example syntax, request-count+arithmetic, and agreement between these rules and contribution templates. No runtime behavior is changed.