Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++
4.0 KB
Raw Blame

One long document, many questions (0.1.40.2, opt-in)

Read a long document once, then ask many different questions without reading it again. All numbers: Tesla P100 16 GB, IQ2_XS, --kv int8 --kv-resident 32768, MTP on, greedy, the 0.1.40 engine unless a row says "pin".

What the engine already did (0.1.40)

The conversation cache keeps small checkpoints (the GDN and indexer state, about 118 MiB each); the K/V of the document stays in the session, so a question that starts from a checkpoint reads only what lies past it.

262K document (258,655 tokens) time to first token
first question (cold read) 904 s (286 tok/s)
questions 2-20, engine with a turn boundary at the document's end 0.53 s median (0.44-0.72)
the same through the server's chat API (no turn boundary at the document's end) 55.5 s each (15 measured): the engine re-reads up to --prompt-cache-every (16,384) tokens from the last periodic checkpoint
parallel 2 through the server the first two questions 918 s and 979 s; the next pair read the document again from token 0

Checkpoint RAM: +804 MiB resident after the first read (6 checkpoints); each later question adds at most 10 MiB. The K/V of 262K takes 3.06 GiB of pinned RAM, once per session (--batch slots each take their own). Identity (greedy, 64 tokens): questions 1 and 12 on a fresh engine give the same tokens as in the 20-question run. Caveat: IQ models are not byte-reproducible at temperature 0 between runs unless STRATA_IQ_MT_MIN=1 (DETAILS.md); some answers differ between two runs of the same prompt for that reason, with or without the field below.

strata_prefix (and the engine's pin=N)

"strata_prefix": {"messages": 1}, {"message": 0, "chars": N} or {"tokens": N} marks the start of the prompt as the shared document. The checkpoint at its end is pinned: never evicted, kept in the parked-conversation budget, kept by SAVE, and a question that resumes from it does not park the branch it leaves.

64K document (P100) no field with the field
engine, raw prompts, questions 2-5 48.6 s 0.35-0.53 s
a 100K-token branch between two questions, then a question 149 s 0.55 s
another 20K prompt between (--conversation-cache-mib 8192) 49 s 0.97 s (restored), then 0.44 s
server, 18K-token document, split messages / one message 8.8 s 0.6-0.8 s
server parallel 2, six questions 730 s (every pair re-reads) 220 s (questions 3-6: 2.5-3.8 s)

Copy-on-write K/V pages across --batch slots are NOT built: each slot keeps its own K/V (3.06 GiB at 262K, about 12 GiB at 1M), and admitting a branch copies the prefix K/V (3 s at 262K). The pinned prefix removes the re-read, not the copy.

tools/research_run.py runs a document and a question list against a server with and without the field.

R3: expert slots against K/V blocks (P100, IQ2_XS)

64K document, 5 questions. Prefill, then median first token of a short question, then decode:

expert slots (GiB) prefill first token decode
2,083 (2.81) 230 s 0.69 s 27.5 tok/s
3,500 200.7 s 0.61 s 30.9
5,000 200.7 s 0.55 s 34.2
7,000 200.7 s 0.51 s 36.7

Marginal value of VRAM: decode 0.31-0.41 s per GiB per 200-token answer between 2,000 and 5,000 slots, 0.15 above; prefill gains only below the prompt loan's floor. K/V cells in VRAM (--kv-resident): 4096 / 16384 / 32768 / 65536 give 34.05 / 34.06 / 34.21 / 34.92 tok/s at 64K; at 262K (5,000 slots) 8192 / 32768 / 131072 give 29.54 / 29.46 / 29.49 tok/s (block hit rate 92.6 / 94.5 / 95.7 %). So the K/V block cache is worth nothing per GiB, experts only help decode, and the read is already served by the loan. A split planner would gain almost nothing, so none was built. The one real lever in the read is the chunk: --prefill 16384 with 7,000 slots reads 64K in 186.4 s against 200.7 s (-7%), and 4096 takes 228 s; it needs the larger loan and changes the chunking, hence the rounding, so it is not a placement-only change.