Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

serve: Anthropic thinking.budget_tokens is a thinking budget too; docs for the answer room and the budget; test

Niko1221194c26881a74
diff --git a/docs/DETAILS.md b/docs/DETAILS.md
index 54eeba27e..bdd6acf06 100644
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -778,7 +778,16 @@ print(r.choices[0].message.content)
answers from there (the engine continues from what it already holds, so nothing is read again). The wrap-up is
part of the thinking the client sees and counts as output tokens. `"reasoning_budget_tokens": N` in
`strata-<model>.json` sets it for every request; a request's own value wins, and `0` means no budget. Off by default;
- Anthropic's `"thinking": {"budget_tokens": N}` still only chooses the level, as above.
+ Anthropic's `"thinking": {"budget_tokens": N}` chooses the level, as above, and since 0.1.45 is also the budget (a
+ `reasoning_budget_tokens` in the same request wins).
+- **max_tokens counts the thinking, so the answer keeps its room (0.1.45).** A client that caps the whole reply (the
+ desktop agent's harness sends `max_completion_tokens: 8192` on every request) and a task the model thinks long about
+ used to be cut off in the middle of the thinking, with no answer. Now a request with thinking on, no budget of its
+ own and `max_tokens` of 1,024 or more gets one: the thinking is wrapped up when only a quarter of the cap (at least
+ 512 tokens) is left, and the model answers in that room (8,192: thinking up to 6,144, answer 2,048). Requests without
+ `max_tokens` (the rest of the context) get a budget of three quarters of it. `"answer_room": false` in
+ `strata-<model>.json` turns it off. Qwen recommends up to 262,144 tokens of reasoning and 131,072 of answer: a client
+ should not cap the reply far below what the context allows.
- **A reply that ends inside its thinking (#1053, opt-in).** Some turns write a sentence of reasoning and then the
end-of-turn token with no `</think>`: the content is empty and an agent stops. `"reasoning_close_retry": true` in
`strata-<model>.json` closes the thinking once (as the thinking budget does) and continues, once per request, only for
diff --git a/serve/frontend.py b/serve/frontend.py
index c36a44da1..893530862 100644
--- a/serve/frontend.py
+++ b/serve/frontend.py
@@ -637,6 +637,11 @@ def anthropic_to_messages(req: dict, think_unasked: bool = True) -> tuple[list[d
# "output_config": {"effort": "low" | "medium" | "high"}
thinking = req.get("thinking")
effort = (req.get("output_config") or {}).get("effort") if isinstance(req.get("output_config"), dict) else None
+ # 0.1.45: Anthropic's budget_tokens is a real budget, not only a level: the thinking is closed (wrap-up, </think>,
+ # the answer) after that many tokens. A reasoning_budget_tokens of the request's own wins.
+ bt = thinking.get("budget_tokens") if isinstance(thinking, dict) and thinking.get("type") != "disabled" else None
+ if isinstance(bt, int) and not isinstance(bt, bool) and bt > 0 and req.get("reasoning_budget_tokens") is None:
+ req["reasoning_budget_tokens"] = bt
if isinstance(thinking, dict) and thinking.get("type") == "disabled":
kwargs["enable_thinking"] = False
elif effort:
diff --git a/serve/test_server.py b/serve/test_server.py
index ba1361d8d..febd25e97 100644
--- a/serve/test_server.py
+++ b/serve/test_server.py
@@ -3837,6 +3837,17 @@ class ThinkingBudget(unittest.TestCase):
"messages": [{"role": "user", "content": "2+2?"}]})
self.assertEqual(len(self.engine.prompts), 3) # thinking off: no budget, one prompt each
+ def test_anthropic_thinking_budget_tokens_is_a_budget(self):
+ """0.1.45: thinking {type: enabled, budget_tokens: N} closes the thinking after N tokens (it was only a level)."""
+ from serve.server import REASONING_WRAP_UP
+ code, b = self.post("/v1/messages", {"model": "m", "max_tokens": 400, "thinking": {"type": "enabled", "budget_tokens": 30},
+ "messages": [{"role": "user", "content": "2+2?"}]})
+ self.assertEqual(code, 200, b)
+ thinking = "".join(c.get("thinking", "") for c in b["content"] if c["type"] == "thinking")
+ self.assertEqual(thinking, ThinkingEngine.THOUGHT[:30] + REASONING_WRAP_UP.split("</think>")[0])
+ self.assertEqual("".join(c.get("text", "") for c in b["content"] if c["type"] == "text"), ThinkingEngine.ANSWER)
+ self.assertEqual(len(self.engine.prompts), 2)
+
def test_a_budget_the_thinking_stays_under(self):
code, b = self.openai(reasoning_budget_tokens=10_000)
self.assertEqual(b["choices"][0]["message"]["reasoning_content"], ThinkingEngine.THOUGHT)