Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Fix #2141: honour llama-server's thinking_budget_tokens (from PR #2140)
xripb611ec5f1e5d
diff --git a/docs/DETAILS.md b/docs/DETAILS.mdindex 80d6d7041..8090cea4e 100644--- a/docs/DETAILS.md+++ b/docs/DETAILS.md@@ -907,7 +907,8 @@ print(r.choices[0].message.content)from going to sleep while at least one request is running or waiting (Windows' `SetThreadExecutionState` with`ES_SYSTEM_REQUIRED`; the screen may still turn off), and gives the normal sleep rules back when the server is idleand when it exits. Default off. On Linux and macOS it does nothing (one line at startup says so).-- **A hard thinking budget (opt-in).** `"reasoning_budget_tokens": N` in a request (OpenAI or Anthropic) caps the+- **A hard thinking budget (opt-in).** `"reasoning_budget_tokens": N` (or llama.cpp's `"thinking_budget_tokens": N`, which+ clients written for llama-server send) in a request (OpenAI or Anthropic) caps thethinking at N tokens: when it gets there the server ends it with a short wrap-up line and `</think>`, and the modelanswers from there (the engine continues from what it already holds, so nothing is read again). The wrap-up ispart of the thinking the client sees and counts as output tokens. `"reasoning_budget_tokens": N` indiff --git a/serve/server.py b/serve/server.pyindex 9a04d39f1..0ad37df2c 100644--- a/serve/server.py+++ b/serve/server.py@@ -3785,10 +3785,13 @@ class Service:return int(getattr(self.engine, "max_context", 0) or getattr(self.engine, "known_ctx", 0) or 0)def reasoning_budget(self, req) -> int | None:- """#123: the most tokens this request may think, or None: the request's `reasoning_budget_tokens`, else the- config's. 0 (or less) means no budget, so a request can turn a configured one off. ValueError (a 400) for- anything that is not a whole number."""- value = (req or {}).get("reasoning_budget_tokens") if isinstance(req, dict) else None+ """#123: the most tokens this request may think, or None: the request's `reasoning_budget_tokens` (or, as+ llama.cpp reads it, `thinking_budget_tokens`), else the config's. 0 (or less) means no budget, so a request can+ turn a configured one off. ValueError (a 400) for anything that is not a whole number."""+ req = req if isinstance(req, dict) else {}+ value = req.get("reasoning_budget_tokens")+ if value is None: # a null does not hide the other name+ value = req.get("thinking_budget_tokens")if value is None:value = self.reasoning_budget_tokensif isinstance(value, float) and value.is_integer():diff --git a/serve/test_server.py b/serve/test_server.pyindex 235d69109..b1a5fe287 100644--- a/serve/test_server.py+++ b/serve/test_server.py@@ -4129,6 +4129,17 @@ class ThinkingBudget(unittest.TestCase):self.assertEqual(b["usage"]["completion_tokens"], 20 + len(extra) + len(ThinkingEngine.ANSWER) + 1)self.assertEqual(b["usage"]["prompt_tokens"], len(first))+ def test_llama_cpp_thinking_budget_tokens_is_the_same_budget(self):+ """llama.cpp's name (pi-llama-cpp sends it): the same wrap-up; reasoning_budget_tokens wins when both are set."""+ from serve.server import REASONING_WRAP_UP+ wrap = REASONING_WRAP_UP.split("</think>")[0]+ code, b = self.openai(thinking_budget_tokens=20)+ self.assertEqual(code, 200, b)+ self.assertEqual(b["choices"][0]["message"]["reasoning_content"], ThinkingEngine.THOUGHT[:20] + wrap)+ code, b = self.openai(reasoning_budget_tokens=10, thinking_budget_tokens=20)+ self.assertEqual(code, 200, b)+ self.assertEqual(b["choices"][0]["message"]["reasoning_content"], ThinkingEngine.THOUGHT[:10] + wrap)+def test_a_stop_string_is_found_in_the_answer_after_the_wrap_up(self):"""#454: the continuation after the thinking budget is cut at a stop string too."""code, b = self.openai(reasoning_budget_tokens=20, stop=["answer is"])