Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Fix #3465: after non-finite logits in a prompt window the retry reads the whole prompt batched

Niko1221894f133b7d49
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index dbcf4e43f..2b8e96591 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -943,7 +943,10 @@ print(r.choices[0].message.content)
conversation and every idle slot (a slot still running finishes its reply and is not kept after it) and reads its
prompt from token 0. The server retries a request that had sent nothing yet once, transparently; a reply that had
started ends as `"length"` with a line in the server window; a second fire on the retry ends the request with an
- error (503 before an answer, an error event in a stream). With replicas, the server forgets what that replica held.
+ error (503 before an answer, an error event in a stream). When the flagged window was one that reads the prompt's
+ last few tokens (the part after the last turn boundary, read through the verify windows, `--short-read`), the retry
+ reads its whole prompt through the batched path instead (#3465: a reporter's prompts failed in that tail every time and
+ passed with `--short-read 0`); the engine prints a line saying so. With replicas, the server forgets what that replica held.
`repeat_stop_tokens` stays as the backstop. The kernel writes only its flags and the host only reads them, so an
answer without a NaN is the one the engine gives with `STRATA_NAN_GUARD=0`, which records no kernel: on an RTX
3080 with the RVN IQ3_S config at fixed expert sets (a chat replay of 8 turns on a 16K-token system prompt), the
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index ab1e4480a..8ab33cef2 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -9199,6 +9199,10 @@ int main(int argc, char** argv) {
bool live_ok = false;
// #879: a window met non-finite logits: the next request drops every cached state first and reads from token 0
bool nan_flush = false;
+ // #3465: the non-finite logits came in a window that reads the PROMPT's tail (the last few tokens, read through
+ // the verify windows after the batched part). The retry reads that tail batched (--short-read 0 for one request):
+ // a reporter's prompts failed there every time and passed with --short-read 0.
+ bool nf_prompt_windows = false, short_read_off_once = false;
std::vector<ConvCheckpoint> checks;
// STRATA_CKPT_POOL: the dropped checkpoints' storage, reused. STRATA_SLOT_CKPT_MOVE turns it on too: a moved
// checkpoint frees nothing at the next prune, so every save would fault in fresh pages (17 -> 40-60 ms)
@@ -9485,6 +9489,7 @@ int main(int argc, char** argv) {
std::fflush(stderr);
nf_hit = true;
nan_flush = true;
+ if (std::strcmp(what, "prompt") == 0) nf_prompt_windows = true;
return true;
};
auto imgs_below = [&](const std::vector<ImgKey>& all, int64_t L) {
@@ -12363,6 +12368,10 @@ int main(int argc, char** argv) {
// conversation and every idle batch slot go, and this prompt is read from token 0 (resume == 0 below)
if (nan_flush) {
nan_flush = false;
+ if (nf_prompt_windows) {
+ nf_prompt_windows = false;
+ short_read_off_once = o.short_read > 0;
+ }
const size_t n_checks = checks.size();
live_ok = false;
checks.clear();
@@ -12871,6 +12880,12 @@ int main(int argc, char** argv) {
// STRATA_CKPT_REREAD compares a restored checkpoint with a batched re-read, so it keeps every read batched.
static const bool no_short = std::getenv("STRATA_CKPT_REREAD") != nullptr;
int64_t req_short_read = o.short_read; // STRATA_PIPELINE_SWITCH may set it per request (short_read=)
+ if (short_read_off_once) { // #3465: the request after non-finite logits in a prompt window
+ short_read_off_once = false;
+ req_short_read = 0;
+ std::fprintf(stderr, "strata serve: after non-finite logits in a prompt window (#3465): this request reads its "
+ "whole prompt through the batched path, none of it through the verify windows\n");
+ }
auto windows_ok = [&](int64_t a, int64_t b) -> bool {
if (no_short || b - a > req_short_read) return false;
if (sp.embd_rows != nullptr)