Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Merge branch 'f-024-rel' into f-024-slots
Niko12219797056e4d39
diff --cc docs/DEEPSEEK41.mdindex 1be933872,0bd609bec..0cc8bc618--- a/docs/DEEPSEEK41.md+++ b/docs/DEEPSEEK41.md@@@ -70,14 -79,40 +79,45 @@@ The server speaks to the engine as it dserver turns them into OpenAI `tool_calls` and Anthropic `tool_use`, including `tool_choice`, streamed arguments and calls written inside the thinking.A value marked `string="true"` stays a string.- Defaults when a request names none: temperature 1.0, top_p 1.0.+- Agents and sub-agents that take turns can skip reading their prompts again (off in this version, a switch for testing). With+ `STRATA_DS4_SLOT_KEEP=1` the engine sets a conversation aside when another request takes the model (about 17 MB per 6.5K tokens) and takes it+ back when the next prompt continues it, also after a restart. Slots live in `ds4-slots` next to the run log, are capped at 2 GB+ (`STRATA_DS4_SLOT_GB`) and are removed after a week. `STRATA_DS4_AUTO_PIN=1` also keeps the system prompt and tools shared by an agent+ and its sub-agents as a checkpoint.- Pictures and the DSpark drafter are in the engine (strata-dsv41 reads them from the unsloth mmproj and dspark files beside the model; the server side is not wired yet). Not in this version: several requests at once (`--parallel` is ignored with a note), tuning- (`--tune` is skipped with a note).+ Not in this version: several requests at once (`--parallel` is ignored with a warning) and pictures (`--vision` is ignored with a warning).++ ## Tuning++ `strata launch --tune quick` (or `thorough`, or the Tuning screen) works for DeepSeek with its own short tuner, `tools/dsv41/tune.py`. It starts+ the engine a few times on a fixed short question and measures the answer speed: the standard settings first, then the DSpark drafter forced on,+ then a shorter draft block (3), then one CPU thread for each physical core (thorough also tries every logical thread). A setting is kept only for a+ gain of more than 5%, and the result is written into the launch's settings. Expect about 5 to 10 minutes (each run loads the model, which takes+ 1 to 3 minutes); close other heavy programs first. If a run fails or the computer is too busy to tell, the standard settings stay and one plain line+ says so. The RAM store (`--ssd-cache`) sizes itself when the model starts, so the tuner does not touch it, and the context stays what you chose.++ ## The engine package++ DeepSeek's engine, `strata-dsv41`, is a separate download from the Strata engine: when you choose DeepSeek, setup (or `strata launch`) fetches the+ package for your computer from the signed update feed (resumable, signature and every file checked) and installs it in `engine-dsv41/` beside+ `engine/`. The Qwen engine's files are never changed. If the feed has no package for your graphics card, setup says so in plain words and stops before+ anything big is downloaded.++ ### Getting the engine by hand++ If the automatic download does not work for you (no access to the update feed, or an offline computer), download the engine package yourself from+ the release page: the file is named `strata-dsv41-<version>-<system>-<backend>-<variant>.tar.gz` (Windows and Linux; pick the one for your system and+ graphics card). Check it against `SHA256SUMS.txt` on the release page if you like. Then give it to Strata in either of two ways:++ ```+ strata launch --model ist-das-2.00bpw --dsv41-package PATH-TO-THE-FILE+ python setup.py --family deepseek41 --dsv41-package PATH-TO-THE-FILE+ ```++ or put the file in the `engine-dsv41/` folder (setup tells you its exact place when it cannot find the engine) and run the same command again.+ Strata checks the package before it uses it: it must be the DeepSeek engine (not the Strata engine) for your system, its manifest must be signed by a key+ Strata trusts, and every file must match the hash in the manifest. A damaged or foreign file is refused with a plain line and nothing is installed.+ The unpacked folder works as well as the archive. The Strata (Qwen) engine is not touched either way.## Compressed experts (trellis), for testingdiff --cc serve/test_dsv41.pyindex f623f5ba1,04b4d2bb8..28b1d02e5--- a/serve/test_dsv41.py+++ b/serve/test_dsv41.py@@@ -20,10 -20,10 +20,10 @@@ ROOT = Path(__file__).resolve().parentssys.path.insert(0, str(ROOT))sys.path.insert(0, str(ROOT / "tools"))import strata_tokenizer as ST # noqa: E402-from serve import frontend # noqa: E402+from serve import frontend, server # noqa: E402from serve.dsml import DsmlOutputParser, DsmlTranslator, to_dsml_force # noqa: E402from serve.frontend import ChatTemplate, OutputParser, openai_to_messages # noqa: E402- from serve.server import (FAMILY_MODEL_NAMES, MockEngine, Service, default_model_name, model_card_sampling, # noqa: E402+ from serve.server import (FAMILY_MODEL_NAMES, MockEngine, REASONING_WRAP_UP, Service, default_model_name, model_card_sampling, # noqa: E402serve)FIX = ROOT / "serve" / "fixtures" / "deepseek41"@@@ -527,90 -529,130 +529,214 @@@ class Api(ApiBase)self.assertEqual(last["decode_tok_s"], 100.0)+ # -- the shared start of an agent's and its sub-agents' prompts (the engine keeps its state there) ------------+ def test_the_system_prompt_and_tools_are_the_shared_start(self):+ self.start("ok")+ self.auto_pin("1")+ sysmsg = {"role": "system", "content": "You are an agent. " * 20}+ body = {"messages": [sysmsg, {"role": "user", "content": "task one"}], "tools": [WEATHER], "max_tokens": 40}+ self.assertEqual(self.post("/v1/chat/completions", body)[0], 200)+ n = self.engine.last_sampling["strata_prefix"]["tokens"]+ ids = self.engine.last_prompt+ self.assertTrue(0 < n < len(ids) - 1)+ self.assertIn("get_weather", self.tok.decode(ids[:n])) # the tools are in the shared start+ self.assertNotIn("task one", self.tok.decode(ids[:n])) # the first user message is not+ # another agent behind the same system prompt and tools, another task: the same boundary, the same start+ body["messages"] = [sysmsg, {"role": "user", "content": "a different task for the sub-agent"}]+ self.assertEqual(self.post("/v1/chat/completions", body)[0], 200)+ self.assertEqual(self.engine.last_sampling["strata_prefix"]["tokens"], n)+ self.assertEqual(self.engine.last_prompt[:n], ids[:n])++ def auto_pin(self, value):+ old = os.environ.get("STRATA_DS4_AUTO_PIN")+ if value is None:+ os.environ.pop("STRATA_DS4_AUTO_PIN", None)+ else:+ os.environ["STRATA_DS4_AUTO_PIN"] = value+ self.addCleanup(lambda: os.environ.pop("STRATA_DS4_AUTO_PIN", None) if old is None+ else os.environ.__setitem__("STRATA_DS4_AUTO_PIN", old))++ def test_no_shared_start_without_a_system_prompt_or_when_switched_off(self):+ self.start("ok")+ self.auto_pin("1")+ self.chat()+ self.assertNotIn("strata_prefix", self.engine.last_sampling) # no system message+ body = {"messages": [{"role": "system", "content": "s"}, {"role": "user", "content": "u"}], "max_tokens": 40}+ for off in (None, "0"): # off by default, and when switched off+ self.auto_pin(off)+ self.post("/v1/chat/completions", body)+ self.assertNotIn("strata_prefix", self.engine.last_sampling)++ def test_a_clients_own_prefix_wins(self):+ self.start("ok")+ self.auto_pin("1")+ body = {"messages": [{"role": "system", "content": "s " * 30}, {"role": "user", "content": "u"}],+ "max_tokens": 40, "strata_prefix": {"tokens": 7}}+ self.post("/v1/chat/completions", body)+ self.assertEqual(self.engine.last_sampling["strata_prefix"], {"tokens": 7})++ def test_the_qwen_path_asks_for_no_prefix(self):+ # the same request on a server that is not DeepSeek's: nothing is added (Qwen keeps its own cache logic)+ svc = Service(MockEngine(self.tok, "ok"), self.tok, ChatTemplate(ROOT / "serve" / "chat_template.jinja"))+ req = {}+ svc.prepare([{"role": "system", "content": "s"}, {"role": "user", "content": "u"}], None, {}, 16, req=req)+ self.assertEqual(req, {})+++class SlotEnv(unittest.TestCase):+ """The engine's kept conversations: the folder and the switch reach the DeepSeek engine only."""++ KEYS = ("STRATA_DS4_SLOT_KEEP", "STRATA_DS4_SLOT_DIR")++ def env(self, cfg):+ saved = {k: os.environ.pop(k, None) for k in self.KEYS}+ try:+ return server.child_env(cfg)+ finally:+ for k, v in saved.items():+ if v is not None:+ os.environ[k] = v++ def test_deepseek_gets_a_folder_in_the_data_folder_and_the_default_switch(self):+ env = self.env({"arch": "deepseek41", "log": str(Path("data") / "strata-deepseek.log"), "args": []})+ self.assertEqual(Path(env["STRATA_DS4_SLOT_DIR"]), Path("data") / "ds4-slots")+ self.assertEqual(env["STRATA_DS4_SLOT_KEEP"], "0")+ self.assertEqual(env["STRATA_DS4_SLOT_GB"], "2")++ def test_the_configs_own_settings_win(self):+ env = self.env({"arch": "deepseek41", "log": "x/y.log",+ "env": {"STRATA_DS4_SLOT_KEEP": "1", "STRATA_DS4_SLOT_DIR": "elsewhere"}})+ self.assertEqual((env["STRATA_DS4_SLOT_KEEP"], env["STRATA_DS4_SLOT_DIR"]), ("1", "elsewhere"))++ def test_other_families_get_nothing(self):+ for arch in ("qwen4exp", None, "gyro"):+ env = self.env({"arch": arch, "log": "x/y.log"} if arch else {"log": "x/y.log"})+ self.assertFalse([k for k in env if k.startswith("STRATA_DS4_")], arch)++ class ThinkForever(DsEngine):+ """Thinks without end (random letters) until the thinking is closed in the prompt, then answers."""++ ANSWER = "Four."++ def __init__(self, tok):+ super().__init__(tok, "")+ self.prompts = []++ def generate(self, ids, max_new, sampling, cancel, embeddings=None):+ self.prompts.append(list(ids))+ self.last_prompt = list(ids)+ got = 0+ try:+ if self.tok.decode(list(ids[-14:])).endswith("</think>\n\n"):+ out = self.tok.encode(self.ANSWER) + [EOS]+ else:+ import random+ rng = random.Random(len(ids)) # (no repeated token: the server's loop guard would end it)+ out = self.tok.encode("".join(chr(97 + rng.randrange(26)) for _ in range(max_new)))+ for t in out[:max_new]:+ if cancel.is_set():+ return+ got += 1+ yield t+ finally:+ self.last = {"generated": got, "prompt_tokens": len(ids), "prompt_ms": 50.0, "decode_ms": got * 10.0,+ "finish": "stop"}+++ class SmallCap(ApiBase):+ """A short max_tokens (200-600) with the thinking on: it was all spent on the thinking and no text block came.+ With no budget of the client's own the thinking is capped at half of max_tokens (this family only)."""++ def start_forever(self):+ self.start("")+ engine = ThinkForever(self.tok)+ self.svc.engine = engine+ self.engine = engine+ return engine++ def text_of(self, r):+ return "".join(c.get("text", "") for c in r["content"] if c["type"] == "text")++ def test_every_short_cap_gets_an_answer_on_both_apis(self):+ for cap in (200, 400, 600):+ for api in ("anthropic", "openai"):+ with self.subTest(cap=cap, api=api):+ self.start_forever()+ if api == "anthropic":+ status, r = self.post("/v1/messages", {"model": "x", "max_tokens": cap, "thinking": {"type": "enabled"},+ "messages": [{"role": "user", "content": "2+2?"}]})+ self.assertEqual(status, 200, r)+ self.assertEqual(self.text_of(r), ThinkForever.ANSWER)+ self.assertLessEqual(r["usage"]["output_tokens"], cap)+ else:+ status, r = self.chat(max_tokens=cap)+ self.assertEqual(status, 200, r)+ self.assertEqual(r["choices"][0]["message"]["content"], ThinkForever.ANSWER)+ self.assertLessEqual(r["usage"]["completion_tokens"], cap)+ first = self.engine.prompts[0]+ self.assertEqual(len(self.engine.prompts), 2) # the thinking was wrapped up, then answered+ self.assertEqual(len(self.engine.prompts[1]) - len(first), cap // 2 - 0 + len(self.tok.encode(+ REASONING_WRAP_UP, parse_special=True)))++ def test_thinking_off_is_untouched(self):+ for cap in (200, 400, 600):+ with self.subTest(cap=cap):+ self.start("Plain answer.")+ status, r = self.chat(max_tokens=cap, reasoning_effort="none")+ self.assertEqual(status, 200, r)+ self.assertEqual(r["choices"][0]["message"]["content"], "Plain answer.")+ status, r = self.post("/v1/messages", {"model": "x", "max_tokens": cap, "thinking": {"type": "disabled"},+ "messages": [{"role": "user", "content": "hi"}]})+ self.assertEqual(self.text_of(r), "Plain answer.")++ def test_a_client_budget_wins(self):+ self.start_forever()+ status, r = self.chat(max_tokens=600, reasoning_budget_tokens=20)+ self.assertEqual(status, 200, r)+ self.assertEqual(len(self.engine.prompts[1]) - len(self.engine.prompts[0]), 20 + len(self.tok.encode(+ REASONING_WRAP_UP, parse_special=True)))++ def test_a_tool_call_under_a_short_cap_still_parses(self):+ for cap in (200, 400, 600):+ with self.subTest(cap=cap):+ self.start("Need it.</think>" + call(("f", [("a", "1", True)])))+ status, r = self.chat(max_tokens=cap, tools=[{"type": "function", "function": {+ "name": "f", "description": "f", "parameters": {"type": "object", "properties": {"a": {"type": "string"}}}}}])+ self.assertEqual(status, 200, r)+ calls = r["choices"][0]["message"]["tool_calls"]+ self.assertEqual(calls[0]["function"]["name"], "f")+ self.assertEqual(json.loads(calls[0]["function"]["arguments"]), {"a": "1"})+ self.assertEqual(len(self.engine.last_prompt) > 0, True)++ def test_the_family_default_thinking_budget(self):+ """DeepSeek-V4.1 only, as Maya ships it: 32768 thinking tokens when neither the request nor the config names a budget."""+ self.start("ok")+ self.assertEqual(self.svc.reasoning_budget({}), 32768)+ self.assertIsNone(self.svc.reasoning_budget({"reasoning_budget_tokens": 0})) # a request turns it off+ self.assertEqual(self.svc.reasoning_budget({"reasoning_budget_tokens": 50}), 50) # and its own wins+ self.assertEqual(self.svc.reasoning_budget({"thinking_budget_tokens": 70}), 70)+ self.svc.reasoning_budget_tokens, self.svc.reasoning_budget_configured = 90, True+ self.assertEqual(self.svc.reasoning_budget({}), 90) # the config's wins+ self.svc.reasoning_budget_tokens = 0 # the config says none+ self.assertIsNone(self.svc.reasoning_budget({}))+ self.svc.reasoning_budget_configured = False+ self.svc.set_family(None)+ self.assertIsNone(self.svc.reasoning_budget({})) # other families: none++ def test_the_default_budget_closes_a_long_thinking(self):+ self.start_forever()+ status, r = self.chat(max_tokens=5000)+ self.assertEqual(status, 200, r)+ self.assertEqual(r["choices"][0]["message"]["content"], ThinkForever.ANSWER)+ status, r = self.chat(max_tokens=300, reasoning_effort="none") # thinking off: no budget, no wrap-up+ self.assertEqual(len(self.engine.prompts), 3)++ def test_other_families_keep_the_old_behaviour(self):+ self.start_forever()+ self.svc.set_family(None)+ status, r = self.chat(max_tokens=300)+ self.assertEqual(len(self.engine.prompts), 1) # no wrap-up: Qwen's serving is as it was+class PickTemplateTests(unittest.TestCase):"""A pack made from another GGUF can ship the V4-Flash template (`tool_calls`, no space): the V4.1 model then