Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Fix #3465: after non-finite logits in a prompt window the retry reads the whole prompt batched
Niko1221894f133b7d49
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex dbcf4e43f..2b8e96591 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -943,7 +943,10 @@ print(r.choices[0].message.content)conversation and every idle slot (a slot still running finishes its reply and is not kept after it) and reads itsprompt from token 0. The server retries a request that had sent nothing yet once, transparently; a reply that hadstarted ends as `"length"` with a line in the server window; a second fire on the retry ends the request with an- error (503 before an answer, an error event in a stream). With replicas, the server forgets what that replica held.+ error (503 before an answer, an error event in a stream). When the flagged window was one that reads the prompt's+ last few tokens (the part after the last turn boundary, read through the verify windows, `--short-read`), the retry+ reads its whole prompt through the batched path instead (#3465: a reporter's prompts failed in that tail every time and+ passed with `--short-read 0`); the engine prints a line saying so. With replicas, the server forgets what that replica held.`repeat_stop_tokens` stays as the backstop. The kernel writes only its flags and the host only reads them, so ananswer without a NaN is the one the engine gives with `STRATA_NAN_GUARD=0`, which records no kernel: on an RTX3080 with the RVN IQ3_S config at fixed expert sets (a chat replay of 8 turns on a 16K-token system prompt), thediff --git a/src/program/generate.cpp b/src/program/generate.cppindex ab1e4480a..8ab33cef2 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -9199,6 +9199,10 @@ int main(int argc, char** argv) {bool live_ok = false;// #879: a window met non-finite logits: the next request drops every cached state first and reads from token 0bool nan_flush = false;+ // #3465: the non-finite logits came in a window that reads the PROMPT's tail (the last few tokens, read through+ // the verify windows after the batched part). The retry reads that tail batched (--short-read 0 for one request):+ // a reporter's prompts failed there every time and passed with --short-read 0.+ bool nf_prompt_windows = false, short_read_off_once = false;std::vector<ConvCheckpoint> checks;// STRATA_CKPT_POOL: the dropped checkpoints' storage, reused. STRATA_SLOT_CKPT_MOVE turns it on too: a moved// checkpoint frees nothing at the next prune, so every save would fault in fresh pages (17 -> 40-60 ms)@@ -9485,6 +9489,7 @@ int main(int argc, char** argv) {std::fflush(stderr);nf_hit = true;nan_flush = true;+ if (std::strcmp(what, "prompt") == 0) nf_prompt_windows = true;return true;};auto imgs_below = [&](const std::vector<ImgKey>& all, int64_t L) {@@ -12363,6 +12368,10 @@ int main(int argc, char** argv) {// conversation and every idle batch slot go, and this prompt is read from token 0 (resume == 0 below)if (nan_flush) {nan_flush = false;+ if (nf_prompt_windows) {+ nf_prompt_windows = false;+ short_read_off_once = o.short_read > 0;+ }const size_t n_checks = checks.size();live_ok = false;checks.clear();@@ -12871,6 +12880,12 @@ int main(int argc, char** argv) {// STRATA_CKPT_REREAD compares a restored checkpoint with a batched re-read, so it keeps every read batched.static const bool no_short = std::getenv("STRATA_CKPT_REREAD") != nullptr;int64_t req_short_read = o.short_read; // STRATA_PIPELINE_SWITCH may set it per request (short_read=)+ if (short_read_off_once) { // #3465: the request after non-finite logits in a prompt window+ short_read_off_once = false;+ req_short_read = 0;+ std::fprintf(stderr, "strata serve: after non-finite logits in a prompt window (#3465): this request reads its "+ "whole prompt through the batched path, none of it through the verify windows\n");+ }auto windows_ok = [&](int64_t a, int64_t b) -> bool {if (no_short || b - a > req_short_read) return false;if (sp.embd_rows != nullptr)