Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Fix #2141: honour llama-server's thinking_budget_tokens (from PR #2140)
xripa0b6d1cd1131
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex cc888218e..0a55c8eca 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -907,7 +907,8 @@ print(r.choices[0].message.content)from going to sleep while at least one request is running or waiting (Windows' `SetThreadExecutionState` with`ES_SYSTEM_REQUIRED`; the screen may still turn off), and gives the normal sleep rules back when the server is idleand when it exits. Default off. On Linux and macOS it does nothing (one line at startup says so).-- **A hard thinking budget (opt-in).** `"reasoning_budget_tokens": N` in a request (OpenAI or Anthropic) caps the+- **A hard thinking budget (opt-in).** `"reasoning_budget_tokens": N` (or llama.cpp's `"thinking_budget_tokens": N`, which+ clients written for llama-server send) in a request (OpenAI or Anthropic) caps thethinking at N tokens: when it gets there the server ends it with a short wrap-up line and `</think>`, and the modelanswers from there (the engine continues from what it already holds, so nothing is read again). The wrap-up ispart of the thinking the client sees and counts as output tokens. `"reasoning_budget_tokens": N` indiff --git a/serve/server.py b/serve/server.pyindex 7a5b1315c..300238b71 100644--- a/serve/server.py+++ b/serve/server.py@@ -3785,10 +3785,13 @@ class Service:return int(getattr(self.engine, "max_context", 0) or getattr(self.engine, "known_ctx", 0) or 0)def reasoning_budget(self, req) -> int | None:- """#123: the most tokens this request may think, or None: the request's `reasoning_budget_tokens`, else the- config's. 0 (or less) means no budget, so a request can turn a configured one off. ValueError (a 400) for- anything that is not a whole number."""- value = (req or {}).get("reasoning_budget_tokens") if isinstance(req, dict) else None+ """#123: the most tokens this request may think, or None: the request's `reasoning_budget_tokens` (or, as+ llama.cpp reads it, `thinking_budget_tokens`), else the config's. 0 (or less) means no budget, so a request can+ turn a configured one off. ValueError (a 400) for anything that is not a whole number."""+ req = req if isinstance(req, dict) else {}+ value = req.get("reasoning_budget_tokens")+ if value is None: # a null does not hide the other name+ value = req.get("thinking_budget_tokens")if value is None:value = self.reasoning_budget_tokensif isinstance(value, float) and value.is_integer():diff --git a/serve/test_server.py b/serve/test_server.pyindex 961236d3b..2d4bd85a0 100644--- a/serve/test_server.py+++ b/serve/test_server.py@@ -4128,6 +4128,17 @@ class ThinkingBudget(unittest.TestCase):self.assertEqual(b["usage"]["completion_tokens"], 20 + len(extra) + len(ThinkingEngine.ANSWER) + 1)self.assertEqual(b["usage"]["prompt_tokens"], len(first))+ def test_llama_cpp_thinking_budget_tokens_is_the_same_budget(self):+ """llama.cpp's name (pi-llama-cpp sends it): the same wrap-up; reasoning_budget_tokens wins when both are set."""+ from serve.server import REASONING_WRAP_UP+ wrap = REASONING_WRAP_UP.split("</think>")[0]+ code, b = self.openai(thinking_budget_tokens=20)+ self.assertEqual(code, 200, b)+ self.assertEqual(b["choices"][0]["message"]["reasoning_content"], ThinkingEngine.THOUGHT[:20] + wrap)+ code, b = self.openai(reasoning_budget_tokens=10, thinking_budget_tokens=20)+ self.assertEqual(code, 200, b)+ self.assertEqual(b["choices"][0]["message"]["reasoning_content"], ThinkingEngine.THOUGHT[:10] + wrap)+def test_a_stop_string_is_found_in_the_answer_after_the_wrap_up(self):"""#454: the continuation after the thinking budget is cut at a stop string too."""code, b = self.openai(reasoning_budget_tokens=20, stop=["answer is"])