Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++
3.2 KB
Raw Blame

Reuse long chat history when the last user message is edited

Why this helps

An agent can keep the same long history but replace the final user message. The final cached state no longer matches, and the nearest periodic checkpoint may be far behind. The engine then rereads history it already processed.

This opt-in change saves one checkpoint at the previous turn boundary, before the final short message. It uses the existing bounded cache and leaves normal token and image-prefix validation in charge of reuse.

Use and limits

Set STRATA_CACHE_MESSAGE_BOUNDARY=1 before starting the engine. It is off by default, requires prompt caching and a valid turn token, and only selects prefixes of at least 8,192 tokens with a tail of at most 1,024. Layer-split execution is excluded (multi_gpu); helper-expert GPUs still work. It can split prefill chunks and add snapshot cost to a fresh request.

Validation environment

  • Two modified RTX 3080 20 GB cards; CUDA SM86, CUDA 13.1, Linux.
  • Intel Xeon E5-2686 v4 (18 cores / 36 threads), about 94 GiB usable RAM.
  • PCIe 3.0, no GPU peer-to-peer access; 17 expert-pool workers.
  • Qwen3.8-Flash-Next IQ3_S, 131,072-token context, INT8 KV, 32,768 resident KV cells.
  • Medium thinking: 2,048-token reasoning cap, 8,192-token output cap.
  • Physical GPU power limits stayed at 250 W and 280 W. No power increase.
  • Historical performance tests used an isolated engine based on official v0.1.38 plus reviewed architectds/Strata changes (best, 05c0f36). This PR contains only our change, ported to official main (99f3dbd). The historical timings are NOT a measured speedup of this pure-upstream port.

Historical benefit

Same-binary ABCCBA comparison, two repetitions per variant:

Metric On-demand boundary + 16K chunks Anticipatory boundary + 32K chunks
First complete 100-record edit 17.236 s 11.527 s
Three complete edits 41.266 s 35.802 s
Cold setup plus three edits 58.688 s 53.146 s
Eleven-case wall-time sum 133.844 s 127.983 s

First-edit time fell by 33.1%; reused prefix grew from 22,016 to 32,831 tokens. This is a COMBINED boundary-policy and chunk-size comparison, not the isolated benefit of adding this checkpoint to pristine upstream. Hot generation was unchanged (97.55 vs 97.60 tokens/s). The mixed-suite reduction was 4.4%; ordinary cold 8K requests cost about 0.2 s more.

The historical comparison passed 77 task/cache checks and cancellation/recovery. A separate safety arm for the adopted 32K choice passed 22 task/cache checks, including 120K, image-history isolation, and cancellation/recovery.

Validation and related work

  • Standalone selector test passes all 21 cases, without assertions that disappear in Release builds.
  • Linux compile/link against pure official main (99f3dbd) passed. No engine was started and no production settings were changed.
  • PR #614 checkpoints an already-reached chunk near the tail without splitting it. This proposal chooses a message boundary, which can split chunks. They solve related cases with different cost and numerical tradeoffs.
  • Chunk geometry can change floating-point results. No pure-upstream bitwise parity or isolated upstream speedup is claimed. Keep draft until validated.