Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge f-027-gh-eng-m (#3416 gated MTP: opt-in proposal law, window-invariant CPU dispatch, draft policy)
Niko12211e57a924d0ba
diff --cc docs/DETAILS.mdindex 4773ed404,ef2e2ec7c..5a7623810--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@@ -534,21 -507,23 +534,38 @@@ applies at the next start. Network, key`STRATA_SPEC_GUMBEL=1` raise the accepted-draft rate on some machines; `STRATA_SPEC_PROB=1` is speculative rejectionsampling. `STRATA_SPEC_DEPTH=1` prints acceptance by draft depth and `STRATA_SPEC_MIN_TEMP=0.9` limits sampled draftingto hotter requests. Try them on your own card before relying on them.+- **logprobs: how likely each token was (CUDA/HIP).** `"logprobs": true` with `"top_logprobs": 0..20` on+ `/v1/chat/completions`, streamed or not, adds `choices[0].logprobs.content[]` to the reply (`token`, `logprob`,+ `bytes`, `top_logprobs`); with thinking on, the thinking's entries come under `reasoning_content` instead. The+ tokens, joined, spell the reply exactly (a character split across two tokens lands in the one that finishes it).+ A reply is bounded by `max_tokens` -- or by the rest of the context when it is not sent -- and by no cap of our+ own, as on OpenAI and vLLM, neither of which ties a length limit to logprobs. A reply that is not streamed, or+ that answers a `json_schema` (buffered to be validated), does hold every entry until it ends, so a long one holds+ a lot of them; a streamed reply hands each entry out with its chunk and keeps none. `"allowed_token_ids": [ids]`+ keeps only those ids in `top_logprobs`, most likely first, renormalised to sum to 1; the chosen token is one of+ them when it was named (vLLM's field, but vLLM masks generation with it, while here it filters what is reported,+ so a token outside the named ids keeps its own `logprob` and an empty list). An id past the model's vocabulary is+ a 400. A token whose probability could not be read gets no entry, and the reply goes on. On a server started with+ `--parallel`, only a `max_tokens` 1 request gets them: it is the one a batch slot decodes through the single-token+ path, which writes an LP line. A longer reply there is a 400, since the slot's `BT` lines carry none -- entries for+ a prefix only, which a client cannot tell from the model's own answer.+ - `STRATA_SPEC_PROB=1` with the default `--spec-min-p` gate: the gate tests the token the draft head drew from its distribution+ q. A draft that fails the gate still counts as a proposal. The verifier keeps q's mass on the tokens that pass the gate+ (not renormalised; call it mu), accepts an offered token D with probability min(1, p(D)/mu(D)), and draws the replacement+ for a rejection or a failed gate from the positive part of p - mu. The older rule checked against the unfiltered q and+ treated a failed gate as a plain sample, which shifts the output: with p = (0.5, 0.5), q = (0.875, 0.125) and a threshold+ of 0.5 it gave (0.5625, 0.4375). `STRATA_SPEC_PROB_GATE=top` gates on q's largest probability before the draw instead and+ keeps the normalised-q rule. The gated rule needs serial decode on one device with `--lookup-chain 0 --adapt-swaps 0+ --pcie-frac 0` and without `STRATA_SPEC_COST`; otherwise the engine says so once and keeps the older rule. This is a+ correctness change, not a speed change. `spec_admission_test` enumerates the rule exactly for a two-token vocabulary,+ `gated_proposal_test` checks the proposal packet and `spec_proposal_parity` checks the GPU verifier against a CPU+ reference. A model-level distribution run is not published yet.+ - With `STRATA_SPEC_PROB=1` the CPU experts of a native (IQ) pack compute a token's rows the same way whatever the window+ width: the grouped kernels that round differently from the one-token result are replaced by the one-token arithmetic,+ and the grouped kernels that already match it stay. Prefill and every other decode keep their arithmetic.+ `native_arithmetic_parity` checks every group width from 1 to 8 for the supported formats, bit for bit. Comparing other+ decode modes against each other still needs one target evaluator; `STRATA_IQ_MT_MIN=1` selects the grouped IQ arithmetic+ at every width and also changes prefill, so keep it the same in both arms.- **Non-finite logits end the request and drop the cached state (on by default).** If a verify window's logits are NaN, thewindow emits no tokens, the request ends, and the next request drops the live session, checkpoints and parkedconversations and reads its prompt from token 0. The server retries a request that had sent nothing once, transparently.