Skip to content

LLM inference in C/C++

C++ ggml

server: support "reasoning_effort": "none" in OAI API (#26045)

Nigel Bosch27209a598da5
diff --git a/tools/server/README.md b/tools/server/README.md
index e6b27a5576..d345654554 100644
--- a/tools/server/README.md
+++ b/tools/server/README.md
@@ -1247,6 +1247,8 @@ The `response_format` parameter supports both plain JSON output (e.g. `{"type":
`chat_template_kwargs`: Allows sending additional parameters to the json templating system. For example: `{"enable_thinking": false}`
+`reasoning_effort`: If set to `none`, reasoning will be disabled for this request. Other values (e.g., `low`, `max`) have no effect on reasoning.
+
`reasoning_format`: The reasoning format to be parsed. If set to `none`, it will output the raw generated text.
`reasoning_control`: Arms realtime reasoning control for this completion so it can be ended early via `/v1/chat/completions/control`. Defaults to `false`.
diff --git a/tools/server/server-chat.cpp b/tools/server/server-chat.cpp
index 31f94e0233..0322e54cce 100644
--- a/tools/server/server-chat.cpp
+++ b/tools/server/server-chat.cpp
@@ -283,6 +283,15 @@ json server_chat_convert_responses_to_chatcmpl(const json & response_body) {
chatcmpl_body["max_tokens"] = response_body["max_output_tokens"];
}
+ if (response_body.contains("reasoning")) {
+ // Only "effort" is handled so far
+ const json & reasoning = response_body.at("reasoning");
+ if (reasoning.contains("effort")) {
+ chatcmpl_body["reasoning_effort"] = reasoning.at("effort");
+ }
+ chatcmpl_body.erase("reasoning");
+ }
+
return chatcmpl_body;
}
diff --git a/tools/server/server-common.cpp b/tools/server/server-common.cpp
index 78b5c6819b..da58049308 100644
--- a/tools/server/server-common.cpp
+++ b/tools/server/server-common.cpp
@@ -1086,6 +1086,14 @@ json oaicompat_chat_params_parse(
throw std::invalid_argument("invalid type for \"enable_thinking\" (expected boolean, got string)");
}
+ // Parse also the OAI "reasoning_effort": "none" specific value
+ if (body.contains("reasoning_effort")) {
+ auto reasoning_effort = json_value(body, "reasoning_effort", std::string(""));
+ if (reasoning_effort == "none") {
+ inputs.enable_thinking = false;
+ } // other reasoning_effort values are model-specific and not yet handled
+ }
+
inputs.force_pure_content = opt.force_pure_content;
// Apply chat template to the list of messages