Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
serve: max_tokens -1 and 0 mean no cap again, as in 0.2.2 (clients send -1 for that); documented
Niko12215c8dd322528c
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex c66ea6d44..f037c65cb 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -1069,9 +1069,8 @@ print(r.choices[0].message.content)Claude Code ask for 32,000 on every request); the thinking budget follows the shortened cap. For the strict llama.cppbehaviour (400) set `"fit_max_tokens": false` in `strata-<model>.json` or pass `--no-fit-max-tokens` to`serve/server.py`. A prompt that leaves no room at all is always refused.-- **`max_tokens` and the `model` field (0.2.3).** `max_tokens`, `max_completion_tokens` and `max_output_tokens` must be at least 1 when- they are sent: `-1` and `0` used to mean "no cap" and are now a 400 that names the field ("max_tokens must be at least 1 ... Leave it- out to let the answer use the room that is left"). Leave the field out for the rest of the context. The `model` field is not matched+- **`max_tokens` and the `model` field.** `max_tokens`, `max_completion_tokens` and `max_output_tokens` of `-1` or `0`, like a+ field that is left out, mean "no cap": the answer may use the room left in the context (clients send `-1` for that). The `model` field is not matchedagainst the loaded model on purpose: one server serves one model, and OpenAI clients often send a name of their own (`gpt-4o`, `local`),so any name is answered by the loaded model (a name from `aliases` and the model's own name work the same). Only `GET /v1/models` says whatis loaded.diff --git a/docs/release-notes-0.2.3.md b/docs/release-notes-0.2.3.mdindex 82b45a975..0c8d883ea 100644--- a/docs/release-notes-0.2.3.md+++ b/docs/release-notes-0.2.3.md@@ -8,7 +8,7 @@- **The agent chat copes with a full context.** The chat now reads the prompt size the model measured, checks a message against the model's window before sending it, summarises earlier messages in pieces that fit, and recovers from the "context too large" error with two plain buttons (Summarize and continue, or Start a new chat; for a big message, attach it as a file or send a shortened preview). `/compact` works. On a scripted 64K window with five 34,000-character messages, the peak prompt fell from 83% to 61% of the window. A shell command with an `&` or a bracket inside quotes no longer asks for approval when it only looks. The approval gate refuses when it cannot read the permission mode. Custom command tools run in the right folder and shell. Tool-returned pictures show in the chat, and a picture-reader start failure now says why.- **HTTPS uses the operating system's certificates (#3722).** Setup, downloads, the server and the fetch tools trust the certificates your system trusts, so a PC behind a company or school proxy no longer fails on a certificate. A failed check gives a plain message.- **Better defaults for CPU workers, VRAM and memory.** On an RTX 5070 12 GB PC (Gyro-S, first-reply speed, 5 rounds, median): the app's Work profile now uses 5 CPU workers and keeps 1536 MiB of video memory free, and the answer speed rose from 31 to 45 tokens per second (JSON +37%, code +20%, agent chat +45%, tool calls +38% against the 0.2.1 app). On an RTX A4000 16 GB with a Q2_0 model the same rule gave +21% (JSON) and +19% (agent). The conversation cache is now sized from free memory when you do not set it (#3493): 40% of what is left above a safety floor, at most 8 GiB, and a line in the log says so. In a test with two agents sharing one model, 91% of the prompt was reused instead of 0%, and a follow-up turn took 3 seconds instead of 11.-- **Terminal stability, from a full stress test.** A second `strata launch` with the same choices says "Already serving" and changes nothing; with other choices it asks, or needs `--restart`. A context change on the same port works with `--restart` (37 s on a 64K, 32K, 128K round). `strata status` and `strata logs` find the running launch without an address. The API key a launch made is kept until you ask for `--new-key`. Retune stops its own model first. The Folder step checks the path before going on, shows the storage path on the Save screen, and tells a denied folder in plain words; models Strata already keeps stay where they are. Download sizes now count the draft layer and the picture reader. A missing shard is named by its file name, in the Terminal and in the app. A busy port gives "Port 8190 is already in use by another program" and leaves nothing behind. A failed engine test start gives a plain cause and one next step. `max_tokens` below 1 is refused with a 400. Ctrl+C quits the launch screens. An older update feed shows its real reason, and a folder with accented letters unpacks.+- **Terminal stability, from a full stress test.** A second `strata launch` with the same choices says "Already serving" and changes nothing; with other choices it asks, or needs `--restart`. A context change on the same port works with `--restart` (37 s on a 64K, 32K, 128K round). `strata status` and `strata logs` find the running launch without an address. The API key a launch made is kept until you ask for `--new-key`. Retune stops its own model first. The Folder step checks the path before going on, shows the storage path on the Save screen, and tells a denied folder in plain words; models Strata already keeps stay where they are. Download sizes now count the draft layer and the picture reader. A missing shard is named by its file name, in the Terminal and in the app. A busy port gives "Port 8190 is already in use by another program" and leaves nothing behind. A failed engine test start gives a plain cause and one next step. Ctrl+C quits the launch screens. An older update feed shows its real reason, and a folder with accented letters unpacks.- **App fixes.** Updates never cross GPU makes, so an AMD PC is not given an NVIDIA engine (#3688). Two GTX 1080 cards are a working PC on the CUDA 12 engine (#3655). Missing model files are named with one next step (#3478). The chat list saves at most once a second and stays small (#3604, #3603). The agent watchdog counts the engine's prompt reading as activity and local chats get a 60-minute idle limit (#3623). Switching pages no longer waits on the browser's animation frame (#3673). A retained older model library is reused without moving weights (#3362). A marker left by a force-killed app no longer blocks the next start. An interrupted update that a later one finished says so in the same alert (#3578). The web page keeps model settings inside narrow windows (#3601, #3594), and uses an edited API key at once (#3435). Tool calls with aliases and wrapper marks are read correctly (port of PR #3556).- **Mac.** A prompt of 4K tokens or more no longer aborts the Metal engine (the expert cache was borrowed in the prompt path; now it is not), in the server and in plain `strata run`. Gyro on a Mac needs engine 0.1.47 or newer.- **The fixes planned as a 0.2.2.1 hotfix are in this release.** `tools/calibrate.py` ships in the Terminal bundle and a setup crash says what it was (#3613, #3467). The test start no longer waits 10 minutes when a Windows launcher sits above the server (#3683). The bundled certificate list is certifi 2026.10.19 (#3670).diff --git a/serve/server.py b/serve/server.pyindex 011fd8a40..87f594592 100644--- a/serve/server.py+++ b/serve/server.py@@ -7914,18 +7914,6 @@ def start_replicas(cfg: dict, groups: list[list[int]], exe: str, effort_end: lisreturn rp.ReplicaEngine(engines, load_tokens=load)-def check_max_tokens(req) -> None:- """K05: an answer length below 1 (``"max_tokens": -1``, 0) is a 400 that names the field and says what to do. It used to be read as "no cap", so a- client bug could hold the only slot for the whole context. A field that is absent or null is not checked (the answer then uses the room left)."""- if not isinstance(req, dict):- return- for key in ("max_tokens", "max_completion_tokens", "max_output_tokens"):- value = req.get(key)- if isinstance(value, (int, float)) and not isinstance(value, bool) and value < 1:- raise ValueError(f"{key} must be at least 1 (the request said {value:g}). Leave it out to let the answer use the room that is left "- "in the context.")--def check_request_sampling(req) -> None:"""A request's sampling field of the wrong JSON type ("temperature": "0.7", "top_k": 40.0, "seed": true) is a 400that names the field, as the same value is through POST /settings or the run config. Unchecked, the value@@ -7933,7 +7921,6 @@ def check_request_sampling(req) -> None:types are checked: a number (int or float, never bool) for temperature, top_p, min_p and the three penalties; aninteger (never bool or float) for top_k, penalty_last_n and seed. Ranges behave as before (a temperature <= 0 isgreedy, top_k 0 or above 64 is clamped, a seed <= 0 is unset), and an absent or null field is not checked."""- check_max_tokens(req)if isinstance(req, dict): # #1819: llama.cpp's names; the OpenAI-style field wins when both are sentif req.get("repeat_penalty") is not None and req.get("repetition_penalty") is None:req["repetition_penalty"] = req["repeat_penalty"]diff --git a/serve/test_server.py b/serve/test_server.pyindex 62a17d10b..3e60dee4e 100644--- a/serve/test_server.py+++ b/serve/test_server.py@@ -115,8 +115,9 @@ class MaxTokens(unittest.TestCase):("98179", "0", "17565", "699.6", "0"))def test_unset_budget_is_the_rest_of_the_context(self):- cases = {"openai": [{}, {"max_tokens": None}, {"max_completion_tokens": None, "max_tokens": None}],- "anthropic": [{}, {"max_tokens": None}]}+ cases = {"openai": [{"max_tokens": -1}, {"max_tokens": 0}, {}, {"max_tokens": None},+ {"max_completion_tokens": -1}, {"max_completion_tokens": None, "max_tokens": None}],+ "anthropic": [{"max_tokens": -1}, {"max_tokens": 0}, {}, {"max_tokens": None}]}for api, budgets in cases.items():for budget in budgets:with self.subTest(api=api, budget=budget):@@ -125,16 +126,6 @@ class MaxTokens(unittest.TestCase):self.assertEqual(self.engine.last_max_new, CTX - CTX_SLACK - pt)self.assertGreater(ct, 1024) # the whole answer, not cut at the old 1024 fallback- def test_a_budget_below_one_is_a_400_that_names_the_field(self):- """K05 (0.2.3): -1 and 0 used to mean "no cap"; a client bug could then hold the only slot for the whole context."""- for api, field in (("openai", "max_tokens"), ("openai", "max_completion_tokens"), ("anthropic", "max_tokens")):- for value in (-1, 0):- with self.subTest(api=api, field=field, value=value):- s, b, _, _ = self.call(api, **{field: value})- self.assertEqual(s, 400, b)- self.assertIn(f"{field} must be at least 1", b["error"]["message"])- self.assertIn("Leave it out", b["error"]["message"])-def test_explicit_budget_is_honoured(self):for api, budget in [("openai", {"max_tokens": 50}), ("openai", {"max_completion_tokens": 50}),("openai", {"max_completion_tokens": 50, "max_tokens": 9}),@@ -234,7 +225,7 @@ class MaxTokens(unittest.TestCase):over = pa - pt0 # the Anthropic template may differ slightlywith self.subTest(api=api, room=5): # a few tokens left: the budget is exactly thosetext = "y" * (CTX - CTX_SLACK - overhead - over - 5)- s, b, pt, ct = self.call(api, text=text, max_tokens=None)+ s, b, pt, ct = self.call(api, text=text, max_tokens=-1)self.assertEqual(s, 200, b)self.assertEqual(self.engine.last_max_new, 5)self.assertEqual(ct, 5)@@ -253,7 +244,7 @@ class MaxTokens(unittest.TestCase):with self.subTest(api=api):out = io.StringIO()with contextlib.redirect_stdout(out):- _, _, pt, _ = self.call(api, max_tokens=None)+ _, _, pt, _ = self.call(api, max_tokens=-1)self.assertIn(f"max_new={CTX - CTX_SLACK - pt} ", out.getvalue())finally:del os.environ["STRATA_DEBUG"]