Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(vlm): decouple reasoning effort from OpenAI model defaults (#4690)
now-inge41b2467f0ff
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.mdindex 7c4e345be..04231deea 100644--- a/docs/en/guides/01-configuration.md+++ b/docs/en/guides/01-configuration.md@@ -646,7 +646,7 @@ Vision Language Model for semantic extraction (L0/L1 generation).| `timeout` | float | Per-request HTTP timeout in seconds passed to the underlying OpenAI/LiteLLM client. Increase for slow endpoints (e.g., DashScope, local inference). Must be `> 0` (default: `600.0`) || `extra_headers` | object | Custom HTTP headers for compatible HTTP providers. `kimi` also accepts header overrides, but already injects the required subscription headers by default || `extra_request_body` | object | Extra JSON body fields for OpenAI-compatible completion requests, useful for provider-specific options such as Ollama `{"think": false}` |-| `reasoning_effort` | str | Reasoning effort for OpenAI Codex Responses requests. Leave unset to use the model default |+| `reasoning_effort` | str | Reasoning effort for `openai`, `azure`, `kimi`, `glm`, and `openai-codex`; explicit values are forwarded, and accepted values depend on the model. When unset, GPT-5/o-series names retain `low`; other models omit the field. For Chat Completions, `extra_request_body.reasoning_effort` takes precedence || `media` | object | Audio/video runtime controls. Media understanding reuses this VLM's provider, model, credentials, client, timeout, retry, headers, output-token limit, failover, and token accounting || `media.enabled` | bool | Enable audio/video understanding (default: `false`) || `media.max_concurrent` | int | Maximum concurrent audio/video calls (default: `2`) |diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.mdindex 5f69d380d..385c28db0 100644--- a/docs/zh/guides/01-configuration.md+++ b/docs/zh/guides/01-configuration.md@@ -615,7 +615,7 @@ provider,并设置 `storage.vectordb.sparse_weight > 0`。自托管模型的| `timeout` | float | 单次 VLM API 请求的 HTTP 超时时间(秒),传递给底层 OpenAI/LiteLLM 客户端。慢端点(如 DashScope、本地推理)可调大。必须 `> 0`(默认:`600.0`) || `extra_headers` | object | 兼容 HTTP provider 的自定义请求头。`kimi` 默认已注入所需订阅请求头,也支持在这里覆盖或扩展 || `extra_request_body` | object | 传给 OpenAI 兼容 completion 请求的额外 JSON body 字段,可用于 Ollama `{"think": false}` 等 provider 专有参数 |-| `reasoning_effort` | str | OpenAI Codex Responses 请求的推理强度。不设置时使用模型默认值 |+| `reasoning_effort` | str | `openai`、`azure`、`kimi`、`glm` 和 `openai-codex` 的推理强度,显式配置时发送;可用值由模型决定。不设置时,GPT-5/o 系列名称保留 `low`,其他模型不发送。Chat Completions 请求中,`extra_request_body.reasoning_effort` 优先 || `media` | object | 音视频运行参数;音视频理解复用该 VLM 的 provider、模型、凭据、client、超时、重试、请求头、输出 token 限制、故障切换和 token 统计 || `media.enabled` | bool | 启用音视频理解(默认:`false`) || `media.max_concurrent` | int | 音视频调用最大并发数(默认:`2`) |diff --git a/openviking/models/vlm/backends/openai_vlm.py b/openviking/models/vlm/backends/openai_vlm.pyindex 2cdcbf643..847d20900 100644--- a/openviking/models/vlm/backends/openai_vlm.py+++ b/openviking/models/vlm/backends/openai_vlm.py@@ -34,18 +34,8 @@ _DASHSCOPE_HOSTS = {}-_REASONING_MODEL_PREFIXES = ("gpt-5", "o1", "o3", "o4")---def _is_reasoning_model(model: Optional[str]) -> bool:- """OpenAI reasoning-model families reject `max_tokens` and non-default `temperature`.-- They require `max_completion_tokens` and only accept `temperature=1` (server default).- """- if not model:- return False- name = model.lower()- return any(name.startswith(p) for p in _REASONING_MODEL_PREFIXES)+# These names select existing OpenAI request defaults, not general reasoning capability.+_OPENAI_REASONING_MODEL_PREFIXES = ("gpt-5", "o1", "o3", "o4")def _build_openai_client_kwargs(@@ -84,7 +74,7 @@ class OpenAIVLM(VLMBase):self._sync_client = Noneself._async_client_cache = LoopScopedAsyncClientCache()self.api_version = config.get("api_version")- self.reasoning_effort = config.get("reasoning_effort", "low")+ self.reasoning_effort = config.get("reasoning_effort")def get_client(self):"""Get sync client"""@@ -143,8 +133,23 @@ class OpenAIVLM(VLMBase):return host.lower() in _DASHSCOPE_HOSTS- def _apply_provider_specific_extra_body(self, kwargs: Dict[str, Any], thinking: bool) -> None:- """Attach provider-specific raw body parameters understood by compatible APIs."""+ def _apply_completion_params(+ self, kwargs: Dict[str, Any], thinking: bool, max_tokens: Optional[int] = None+ ) -> None:+ """Apply model defaults, explicit settings, and provider-specific body fields."""+ if kwargs["model"].lower().startswith(_OPENAI_REASONING_MODEL_PREFIXES):+ max_tokens_param = "max_completion_tokens"+ kwargs["reasoning_effort"] = "low"+ else:+ max_tokens_param = "max_tokens"+ kwargs["temperature"] = self.temperature++ effective_max_tokens = max_tokens if max_tokens is not None else self.max_tokens+ if effective_max_tokens is not None:+ kwargs[max_tokens_param] = effective_max_tokens+ if self.reasoning_effort is not None:+ kwargs["reasoning_effort"] = self.reasoning_effort+extra_body = dict(self.extra_request_body)if self._supports_enable_thinking():extra_body["enable_thinking"] = bool(thinking)@@ -227,23 +232,15 @@ class OpenAIVLM(VLMBase):max_tokens: Optional[int] = None,) -> Dict[str, Any]:effective_thinking = self.thinking if thinking is None else thinking- effective_max_tokens = max_tokens if max_tokens is not None else self.max_tokenskwargs_messages = sanitize_openai_messages(messages or [{"role": "user", "content": prompt}])model = self.model or "gpt-4o-mini"- is_reasoning = _is_reasoning_model(model)kwargs: Dict[str, Any] = {"model": model,"messages": kwargs_messages,}- if is_reasoning:- kwargs["reasoning_effort"] = self.reasoning_effort- else:- kwargs["temperature"] = self.temperature- self._apply_provider_specific_extra_body(kwargs, effective_thinking)- if effective_max_tokens is not None:- kwargs["max_completion_tokens" if is_reasoning else "max_tokens"] = effective_max_tokens+ self._apply_completion_params(kwargs, effective_thinking, max_tokens=max_tokens)if tools:kwargs["tools"] = toolskwargs["tool_choice"] = tool_choice or "auto"@@ -270,18 +267,11 @@ class OpenAIVLM(VLMBase):kwargs_messages = sanitize_openai_messages([{"role": "user", "content": content}])model = self.model or "gpt-4o-mini"- is_reasoning = _is_reasoning_model(model)kwargs: Dict[str, Any] = {"model": model,"messages": kwargs_messages,}- if is_reasoning:- kwargs["reasoning_effort"] = self.reasoning_effort- else:- kwargs["temperature"] = self.temperature- self._apply_provider_specific_extra_body(kwargs, effective_thinking)- if self.max_tokens is not None:- kwargs["max_completion_tokens" if is_reasoning else "max_tokens"] = self.max_tokens+ self._apply_completion_params(kwargs, effective_thinking)if tools:kwargs["tools"] = toolskwargs["tool_choice"] = tool_choice or "auto"diff --git a/tests/unit/test_extra_headers_vlm.py b/tests/unit/test_extra_headers_vlm.pyindex 9bb0a8ca0..46e35c144 100644--- a/tests/unit/test_extra_headers_vlm.py+++ b/tests/unit/test_extra_headers_vlm.py@@ -3,15 +3,20 @@"""Tests for VLM extra_headers support."""import asyncio+import jsonimport threadingfrom unittest.mock import AsyncMock, MagicMock, patch+import httpx+from openai import OpenAI+from openviking.models.vlm.backends.litellm_vlm import (LiteLLMVLMProvider,detect_provider_by_model,)from openviking.models.vlm.backends.openai_vlm import OpenAIVLMfrom openviking.models.vlm.backends.volcengine_vlm import VolcEngineVLM+from openviking_cli.utils.config.vlm_config import VLMConfigclass TestVLMExtraHeaders:@@ -459,27 +464,34 @@ class TestVLMExtraRequestBody:"""Test provider-specific VLM request body passthrough."""@patch("openviking.models.vlm.backends.openai_vlm.openai.OpenAI")- def test_openai_text_completion_passes_extra_request_body(self, mock_openai_class):- mock_client = MagicMock()- mock_openai_class.return_value = mock_client- mock_response = MagicMock()- mock_response.choices = [MagicMock(message=MagicMock(content="ok"), finish_reason="stop")]- mock_response.usage = None- mock_client.chat.completions.create.return_value = mock_response-- vlm = OpenAIVLM(- {- "api_key": "sk-test",- "api_base": "https://api.openai.com/v1",- "model": "gpt-4o-mini",- "extra_request_body": {"think": False, "keep_alive": "5m"},- }+ def test_completion_body_overrides_configured_reasoning_effort(self, mock_openai_class):+ send = MagicMock(+ return_value=httpx.Response(+ 200,+ json={+ "choices": [{"message": {"content": "ok"}, "finish_reason": "stop"}],+ },+ ))-- vlm.get_completion("hello")-- call_kwargs = mock_client.chat.completions.create.call_args.kwargs- assert call_kwargs["extra_body"] == {"think": False, "keep_alive": "5m"}+ with OpenAI(+ api_key="sk-test",+ http_client=httpx.Client(transport=httpx.MockTransport(send)),+ ) as client:+ mock_openai_class.return_value = client+ vlm = VLMConfig(+ provider="glm",+ model="glm-5.3-flash",+ api_key="sk-test",+ reasoning_effort="low",+ extra_request_body={"reasoning_effort": "high", "seed": 7},+ ).get_vlm_instance()++ assert vlm.get_completion("hello") == "ok"+ assert vlm.get_vision_completion("describe", images=[b"\x89PNG\r\n\x1a\n"]) == "ok"++ bodies = [json.loads(call.args[0].content) for call in send.call_args_list]+ assert [body["reasoning_effort"] for body in bodies] == ["high", "high"]+ assert [body["seed"] for body in bodies] == [7, 7]@patch("openviking.models.vlm.backends.openai_vlm.openai.OpenAI")def test_dashscope_thinking_merges_with_extra_request_body(self, mock_openai_class):@@ -494,7 +506,8 @@ class TestVLMExtraRequestBody:{"api_key": "sk-test","api_base": "https://dashscope.aliyuncs.com/compatible-mode/v1",- "model": "qwen3.5-plus",+ "model": "ZHIPU/GLM-5.3",+ "reasoning_effort": "low","extra_request_body": {"seed": 7},})@@ -503,6 +516,7 @@ class TestVLMExtraRequestBody:call_kwargs = mock_client.chat.completions.create.call_args.kwargsassert call_kwargs["extra_body"] == {"seed": 7, "enable_thinking": True}+ assert call_kwargs["reasoning_effort"] == "low"def test_litellm_build_kwargs_passes_extra_request_body(self):vlm = LiteLLMVLMProvider(diff --git a/tests/unit/test_vlm_reasoning_models.py b/tests/unit/test_vlm_reasoning_models.pyindex a5bf63821..976a6201c 100644--- a/tests/unit/test_vlm_reasoning_models.py+++ b/tests/unit/test_vlm_reasoning_models.py@@ -1,60 +1,13 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0-"""Tests for reasoning-model parameter translation in the OpenAI VLM backend.+"""Tests for OpenAI request defaults and explicitly configured reasoning effort."""-OpenAI reasoning-model families (gpt-5, o1, o3, o4) reject the `max_tokens` and-non-default `temperature` parameters. They require `max_completion_tokens` and-only accept `temperature=1` (the server default). This module verifies that the-OpenAI VLM backend translates parameters for reasoning models while leaving-non-reasoning models (e.g. gpt-4o-mini) unchanged.-"""+from openviking.models.vlm.backends.openai_vlm import OpenAIVLM+from openviking_cli.utils.config.vlm_config import VLMConfig-import pytest-from openviking.models.vlm.backends.openai_vlm import OpenAIVLM, _is_reasoning_model---class TestIsReasoningModel:- """Pure function: reasoning-model prefix detection."""-- @pytest.mark.parametrize(- "model",- [- "gpt-5",- "gpt-5-mini",- "gpt-5-nano-2025-08-07",- "GPT-5-Mini",- "o1",- "o1-preview",- "o1-mini",- "o3",- "o3-mini",- "o4-mini",- "o4-mini-2025-04-16",- ],- )- def test_reasoning_model_detected(self, model):- assert _is_reasoning_model(model) is True-- @pytest.mark.parametrize(- "model",- [- "gpt-4o",- "gpt-4o-mini",- "gpt-4.1",- "gpt-4.1-mini",- "doubao-seed-2-0-lite-260428",- "gpt-3.5-turbo",- "",- None,- ],- )- def test_non_reasoning_model_not_detected(self, model):- assert _is_reasoning_model(model) is False---class TestReasoningModelTextKwargs:- """`_build_text_kwargs` should translate params for reasoning models."""+class TestOpenAITextCompletionParams:+ """OpenAI defaults must not gate compatible-provider configuration."""def test_gpt5_mini_uses_max_completion_tokens(self):vlm = OpenAIVLM(@@ -65,8 +18,8 @@ class TestReasoningModelTextKwargs:"max_tokens": 512,})- kwargs = vlm._build_text_kwargs(prompt="hi")- assert kwargs["max_completion_tokens"] == 512+ kwargs = vlm._build_text_kwargs(prompt="hi", max_tokens=128)+ assert kwargs["max_completion_tokens"] == 128assert "max_tokens" not in kwargsassert "temperature" not in kwargsassert kwargs["reasoning_effort"] == "low"@@ -87,17 +40,19 @@ class TestReasoningModelTextKwargs:assert kwargs["reasoning_effort"] == "low"def test_reasoning_effort_overridable_via_config(self):- vlm = OpenAIVLM(- {- "api_key": "sk-test",- "model": "gpt-5-mini",- "api_base": "https://api.openai.com/v1",- "max_tokens": 512,- "reasoning_effort": "high",- }- )- kwargs = vlm._build_text_kwargs(prompt="hi")+ vlm = VLMConfig(+ provider="glm",+ model="glm-5.3-flash",+ api_key="sk-test",+ max_tokens=512,+ temperature=0.3,+ reasoning_effort="high",+ ).get_vlm_instance()+ kwargs = vlm._build_text_kwargs(prompt="hi", max_tokens=128)assert kwargs["reasoning_effort"] == "high"+ assert kwargs["max_tokens"] == 128+ assert "max_completion_tokens" not in kwargs+ assert kwargs["temperature"] == 0.3def test_gpt4o_mini_keeps_max_tokens_and_temperature(self):vlm = OpenAIVLM(@@ -129,8 +84,8 @@ class TestReasoningModelTextKwargs:assert "temperature" not in kwargs-class TestReasoningModelVisionKwargs:- """`_build_vision_kwargs` should apply the same translation."""+class TestOpenAIVisionCompletionParams:+ """Vision requests preserve the same OpenAI defaults as text requests."""def test_gpt5_mini_vision_uses_max_completion_tokens(self):vlm = OpenAIVLM(