Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Merge branch 'f-024-rel' into f-024-slots

Niko12219797056e4d39
diff --cc docs/DEEPSEEK41.md
index 1be933872,0bd609bec..0cc8bc618
--- a/docs/DEEPSEEK41.md
+++ b/docs/DEEPSEEK41.md
@@@ -70,14 -79,40 +79,45 @@@ The server speaks to the engine as it d
server turns them into OpenAI `tool_calls` and Anthropic `tool_use`, including `tool_choice`, streamed arguments and calls written inside the thinking.
A value marked `string="true"` stays a string.
- Defaults when a request names none: temperature 1.0, top_p 1.0.
+- Agents and sub-agents that take turns can skip reading their prompts again (off in this version, a switch for testing). With
+ `STRATA_DS4_SLOT_KEEP=1` the engine sets a conversation aside when another request takes the model (about 17 MB per 6.5K tokens) and takes it
+ back when the next prompt continues it, also after a restart. Slots live in `ds4-slots` next to the run log, are capped at 2 GB
+ (`STRATA_DS4_SLOT_GB`) and are removed after a week. `STRATA_DS4_AUTO_PIN=1` also keeps the system prompt and tools shared by an agent
+ and its sub-agents as a checkpoint.
- Pictures and the DSpark drafter are in the engine (strata-dsv41 reads them from the unsloth mmproj and dspark files beside the model; the server side is not wired yet). Not in this version: several requests at once (`--parallel` is ignored with a note), tuning
- (`--tune` is skipped with a note).
+ Not in this version: several requests at once (`--parallel` is ignored with a warning) and pictures (`--vision` is ignored with a warning).
+
+ ## Tuning
+
+ `strata launch --tune quick` (or `thorough`, or the Tuning screen) works for DeepSeek with its own short tuner, `tools/dsv41/tune.py`. It starts
+ the engine a few times on a fixed short question and measures the answer speed: the standard settings first, then the DSpark drafter forced on,
+ then a shorter draft block (3), then one CPU thread for each physical core (thorough also tries every logical thread). A setting is kept only for a
+ gain of more than 5%, and the result is written into the launch's settings. Expect about 5 to 10 minutes (each run loads the model, which takes
+ 1 to 3 minutes); close other heavy programs first. If a run fails or the computer is too busy to tell, the standard settings stay and one plain line
+ says so. The RAM store (`--ssd-cache`) sizes itself when the model starts, so the tuner does not touch it, and the context stays what you chose.
+
+ ## The engine package
+
+ DeepSeek's engine, `strata-dsv41`, is a separate download from the Strata engine: when you choose DeepSeek, setup (or `strata launch`) fetches the
+ package for your computer from the signed update feed (resumable, signature and every file checked) and installs it in `engine-dsv41/` beside
+ `engine/`. The Qwen engine's files are never changed. If the feed has no package for your graphics card, setup says so in plain words and stops before
+ anything big is downloaded.
+
+ ### Getting the engine by hand
+
+ If the automatic download does not work for you (no access to the update feed, or an offline computer), download the engine package yourself from
+ the release page: the file is named `strata-dsv41-<version>-<system>-<backend>-<variant>.tar.gz` (Windows and Linux; pick the one for your system and
+ graphics card). Check it against `SHA256SUMS.txt` on the release page if you like. Then give it to Strata in either of two ways:
+
+ ```
+ strata launch --model ist-das-2.00bpw --dsv41-package PATH-TO-THE-FILE
+ python setup.py --family deepseek41 --dsv41-package PATH-TO-THE-FILE
+ ```
+
+ or put the file in the `engine-dsv41/` folder (setup tells you its exact place when it cannot find the engine) and run the same command again.
+ Strata checks the package before it uses it: it must be the DeepSeek engine (not the Strata engine) for your system, its manifest must be signed by a key
+ Strata trusts, and every file must match the hash in the manifest. A damaged or foreign file is refused with a plain line and nothing is installed.
+ The unpacked folder works as well as the archive. The Strata (Qwen) engine is not touched either way.
## Compressed experts (trellis), for testing
diff --cc serve/test_dsv41.py
index f623f5ba1,04b4d2bb8..28b1d02e5
--- a/serve/test_dsv41.py
+++ b/serve/test_dsv41.py
@@@ -20,10 -20,10 +20,10 @@@ ROOT = Path(__file__).resolve().parents
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "tools"))
import strata_tokenizer as ST # noqa: E402
-from serve import frontend # noqa: E402
+from serve import frontend, server # noqa: E402
from serve.dsml import DsmlOutputParser, DsmlTranslator, to_dsml_force # noqa: E402
from serve.frontend import ChatTemplate, OutputParser, openai_to_messages # noqa: E402
- from serve.server import (FAMILY_MODEL_NAMES, MockEngine, Service, default_model_name, model_card_sampling, # noqa: E402
+ from serve.server import (FAMILY_MODEL_NAMES, MockEngine, REASONING_WRAP_UP, Service, default_model_name, model_card_sampling, # noqa: E402
serve)
FIX = ROOT / "serve" / "fixtures" / "deepseek41"
@@@ -527,90 -529,130 +529,214 @@@ class Api(ApiBase)
self.assertEqual(last["decode_tok_s"], 100.0)
+ # -- the shared start of an agent's and its sub-agents' prompts (the engine keeps its state there) ------------
+ def test_the_system_prompt_and_tools_are_the_shared_start(self):
+ self.start("ok")
+ self.auto_pin("1")
+ sysmsg = {"role": "system", "content": "You are an agent. " * 20}
+ body = {"messages": [sysmsg, {"role": "user", "content": "task one"}], "tools": [WEATHER], "max_tokens": 40}
+ self.assertEqual(self.post("/v1/chat/completions", body)[0], 200)
+ n = self.engine.last_sampling["strata_prefix"]["tokens"]
+ ids = self.engine.last_prompt
+ self.assertTrue(0 < n < len(ids) - 1)
+ self.assertIn("get_weather", self.tok.decode(ids[:n])) # the tools are in the shared start
+ self.assertNotIn("task one", self.tok.decode(ids[:n])) # the first user message is not
+ # another agent behind the same system prompt and tools, another task: the same boundary, the same start
+ body["messages"] = [sysmsg, {"role": "user", "content": "a different task for the sub-agent"}]
+ self.assertEqual(self.post("/v1/chat/completions", body)[0], 200)
+ self.assertEqual(self.engine.last_sampling["strata_prefix"]["tokens"], n)
+ self.assertEqual(self.engine.last_prompt[:n], ids[:n])
+
+ def auto_pin(self, value):
+ old = os.environ.get("STRATA_DS4_AUTO_PIN")
+ if value is None:
+ os.environ.pop("STRATA_DS4_AUTO_PIN", None)
+ else:
+ os.environ["STRATA_DS4_AUTO_PIN"] = value
+ self.addCleanup(lambda: os.environ.pop("STRATA_DS4_AUTO_PIN", None) if old is None
+ else os.environ.__setitem__("STRATA_DS4_AUTO_PIN", old))
+
+ def test_no_shared_start_without_a_system_prompt_or_when_switched_off(self):
+ self.start("ok")
+ self.auto_pin("1")
+ self.chat()
+ self.assertNotIn("strata_prefix", self.engine.last_sampling) # no system message
+ body = {"messages": [{"role": "system", "content": "s"}, {"role": "user", "content": "u"}], "max_tokens": 40}
+ for off in (None, "0"): # off by default, and when switched off
+ self.auto_pin(off)
+ self.post("/v1/chat/completions", body)
+ self.assertNotIn("strata_prefix", self.engine.last_sampling)
+
+ def test_a_clients_own_prefix_wins(self):
+ self.start("ok")
+ self.auto_pin("1")
+ body = {"messages": [{"role": "system", "content": "s " * 30}, {"role": "user", "content": "u"}],
+ "max_tokens": 40, "strata_prefix": {"tokens": 7}}
+ self.post("/v1/chat/completions", body)
+ self.assertEqual(self.engine.last_sampling["strata_prefix"], {"tokens": 7})
+
+ def test_the_qwen_path_asks_for_no_prefix(self):
+ # the same request on a server that is not DeepSeek's: nothing is added (Qwen keeps its own cache logic)
+ svc = Service(MockEngine(self.tok, "ok"), self.tok, ChatTemplate(ROOT / "serve" / "chat_template.jinja"))
+ req = {}
+ svc.prepare([{"role": "system", "content": "s"}, {"role": "user", "content": "u"}], None, {}, 16, req=req)
+ self.assertEqual(req, {})
+
+
+class SlotEnv(unittest.TestCase):
+ """The engine's kept conversations: the folder and the switch reach the DeepSeek engine only."""
+
+ KEYS = ("STRATA_DS4_SLOT_KEEP", "STRATA_DS4_SLOT_DIR")
+
+ def env(self, cfg):
+ saved = {k: os.environ.pop(k, None) for k in self.KEYS}
+ try:
+ return server.child_env(cfg)
+ finally:
+ for k, v in saved.items():
+ if v is not None:
+ os.environ[k] = v
+
+ def test_deepseek_gets_a_folder_in_the_data_folder_and_the_default_switch(self):
+ env = self.env({"arch": "deepseek41", "log": str(Path("data") / "strata-deepseek.log"), "args": []})
+ self.assertEqual(Path(env["STRATA_DS4_SLOT_DIR"]), Path("data") / "ds4-slots")
+ self.assertEqual(env["STRATA_DS4_SLOT_KEEP"], "0")
+ self.assertEqual(env["STRATA_DS4_SLOT_GB"], "2")
+
+ def test_the_configs_own_settings_win(self):
+ env = self.env({"arch": "deepseek41", "log": "x/y.log",
+ "env": {"STRATA_DS4_SLOT_KEEP": "1", "STRATA_DS4_SLOT_DIR": "elsewhere"}})
+ self.assertEqual((env["STRATA_DS4_SLOT_KEEP"], env["STRATA_DS4_SLOT_DIR"]), ("1", "elsewhere"))
+
+ def test_other_families_get_nothing(self):
+ for arch in ("qwen4exp", None, "gyro"):
+ env = self.env({"arch": arch, "log": "x/y.log"} if arch else {"log": "x/y.log"})
+ self.assertFalse([k for k in env if k.startswith("STRATA_DS4_")], arch)
+
+ class ThinkForever(DsEngine):
+ """Thinks without end (random letters) until the thinking is closed in the prompt, then answers."""
+
+ ANSWER = "Four."
+
+ def __init__(self, tok):
+ super().__init__(tok, "")
+ self.prompts = []
+
+ def generate(self, ids, max_new, sampling, cancel, embeddings=None):
+ self.prompts.append(list(ids))
+ self.last_prompt = list(ids)
+ got = 0
+ try:
+ if self.tok.decode(list(ids[-14:])).endswith("</think>\n\n"):
+ out = self.tok.encode(self.ANSWER) + [EOS]
+ else:
+ import random
+ rng = random.Random(len(ids)) # (no repeated token: the server's loop guard would end it)
+ out = self.tok.encode("".join(chr(97 + rng.randrange(26)) for _ in range(max_new)))
+ for t in out[:max_new]:
+ if cancel.is_set():
+ return
+ got += 1
+ yield t
+ finally:
+ self.last = {"generated": got, "prompt_tokens": len(ids), "prompt_ms": 50.0, "decode_ms": got * 10.0,
+ "finish": "stop"}
+
+
+ class SmallCap(ApiBase):
+ """A short max_tokens (200-600) with the thinking on: it was all spent on the thinking and no text block came.
+ With no budget of the client's own the thinking is capped at half of max_tokens (this family only)."""
+
+ def start_forever(self):
+ self.start("")
+ engine = ThinkForever(self.tok)
+ self.svc.engine = engine
+ self.engine = engine
+ return engine
+
+ def text_of(self, r):
+ return "".join(c.get("text", "") for c in r["content"] if c["type"] == "text")
+
+ def test_every_short_cap_gets_an_answer_on_both_apis(self):
+ for cap in (200, 400, 600):
+ for api in ("anthropic", "openai"):
+ with self.subTest(cap=cap, api=api):
+ self.start_forever()
+ if api == "anthropic":
+ status, r = self.post("/v1/messages", {"model": "x", "max_tokens": cap, "thinking": {"type": "enabled"},
+ "messages": [{"role": "user", "content": "2+2?"}]})
+ self.assertEqual(status, 200, r)
+ self.assertEqual(self.text_of(r), ThinkForever.ANSWER)
+ self.assertLessEqual(r["usage"]["output_tokens"], cap)
+ else:
+ status, r = self.chat(max_tokens=cap)
+ self.assertEqual(status, 200, r)
+ self.assertEqual(r["choices"][0]["message"]["content"], ThinkForever.ANSWER)
+ self.assertLessEqual(r["usage"]["completion_tokens"], cap)
+ first = self.engine.prompts[0]
+ self.assertEqual(len(self.engine.prompts), 2) # the thinking was wrapped up, then answered
+ self.assertEqual(len(self.engine.prompts[1]) - len(first), cap // 2 - 0 + len(self.tok.encode(
+ REASONING_WRAP_UP, parse_special=True)))
+
+ def test_thinking_off_is_untouched(self):
+ for cap in (200, 400, 600):
+ with self.subTest(cap=cap):
+ self.start("Plain answer.")
+ status, r = self.chat(max_tokens=cap, reasoning_effort="none")
+ self.assertEqual(status, 200, r)
+ self.assertEqual(r["choices"][0]["message"]["content"], "Plain answer.")
+ status, r = self.post("/v1/messages", {"model": "x", "max_tokens": cap, "thinking": {"type": "disabled"},
+ "messages": [{"role": "user", "content": "hi"}]})
+ self.assertEqual(self.text_of(r), "Plain answer.")
+
+ def test_a_client_budget_wins(self):
+ self.start_forever()
+ status, r = self.chat(max_tokens=600, reasoning_budget_tokens=20)
+ self.assertEqual(status, 200, r)
+ self.assertEqual(len(self.engine.prompts[1]) - len(self.engine.prompts[0]), 20 + len(self.tok.encode(
+ REASONING_WRAP_UP, parse_special=True)))
+
+ def test_a_tool_call_under_a_short_cap_still_parses(self):
+ for cap in (200, 400, 600):
+ with self.subTest(cap=cap):
+ self.start("Need it.</think>" + call(("f", [("a", "1", True)])))
+ status, r = self.chat(max_tokens=cap, tools=[{"type": "function", "function": {
+ "name": "f", "description": "f", "parameters": {"type": "object", "properties": {"a": {"type": "string"}}}}}])
+ self.assertEqual(status, 200, r)
+ calls = r["choices"][0]["message"]["tool_calls"]
+ self.assertEqual(calls[0]["function"]["name"], "f")
+ self.assertEqual(json.loads(calls[0]["function"]["arguments"]), {"a": "1"})
+ self.assertEqual(len(self.engine.last_prompt) > 0, True)
+
+ def test_the_family_default_thinking_budget(self):
+ """DeepSeek-V4.1 only, as Maya ships it: 32768 thinking tokens when neither the request nor the config names a budget."""
+ self.start("ok")
+ self.assertEqual(self.svc.reasoning_budget({}), 32768)
+ self.assertIsNone(self.svc.reasoning_budget({"reasoning_budget_tokens": 0})) # a request turns it off
+ self.assertEqual(self.svc.reasoning_budget({"reasoning_budget_tokens": 50}), 50) # and its own wins
+ self.assertEqual(self.svc.reasoning_budget({"thinking_budget_tokens": 70}), 70)
+ self.svc.reasoning_budget_tokens, self.svc.reasoning_budget_configured = 90, True
+ self.assertEqual(self.svc.reasoning_budget({}), 90) # the config's wins
+ self.svc.reasoning_budget_tokens = 0 # the config says none
+ self.assertIsNone(self.svc.reasoning_budget({}))
+ self.svc.reasoning_budget_configured = False
+ self.svc.set_family(None)
+ self.assertIsNone(self.svc.reasoning_budget({})) # other families: none
+
+ def test_the_default_budget_closes_a_long_thinking(self):
+ self.start_forever()
+ status, r = self.chat(max_tokens=5000)
+ self.assertEqual(status, 200, r)
+ self.assertEqual(r["choices"][0]["message"]["content"], ThinkForever.ANSWER)
+ status, r = self.chat(max_tokens=300, reasoning_effort="none") # thinking off: no budget, no wrap-up
+ self.assertEqual(len(self.engine.prompts), 3)
+
+ def test_other_families_keep_the_old_behaviour(self):
+ self.start_forever()
+ self.svc.set_family(None)
+ status, r = self.chat(max_tokens=300)
+ self.assertEqual(len(self.engine.prompts), 1) # no wrap-up: Qwen's serving is as it was
+
class PickTemplateTests(unittest.TestCase):
"""A pack made from another GGUF can ship the V4-Flash template (`tool_calls`, no space): the V4.1 model then