Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(vlm): honor LiteLLM native routes (#2111)
Qin Haojie9846bbca5d5c
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.mdindex 9f5570998..c99404e6c 100644--- a/docs/en/guides/01-configuration.md+++ b/docs/en/guides/01-configuration.md@@ -580,7 +580,8 @@ Vision Language Model for semantic extraction (L0/L1 generation).| Parameter | Type | Description ||-----------|------|-------------|-| `api_key` | str | API key. Optional for `openai-codex` when Codex OAuth is available |+| `api_key` | str | API key. Optional for `openai-codex` when Codex OAuth is available, and optional for `litellm` routes that use provider-native credentials |+| `forward_api_key` | bool | LiteLLM only. Overrides whether `api_key` is forwarded to LiteLLM. By default, OpenViking does not forward placeholder keys for native AWS/GCP routes such as `bedrock/`, `sagemaker/`, and `vertex_ai/`; set to `true` when intentionally using a LiteLLM API-key route such as Bedrock bearer-token auth || `model` | str | Model name || `api_base` | str | API endpoint (optional) || `thinking` | bool | Enable thinking mode for VolcEngine models (default: `false`) |@@ -614,9 +615,16 @@ If VLM is not configured, L0/L1 will be generated from content directly (less se- `openai-codex`: Codex VLM via ChatGPT/Codex OAuth- `kimi`: Kimi Coding subscription endpoint with built-in provider defaults- `glm`: Z.AI GLM Coding Plan endpoint with OpenAI-compatible requests+- `litellm`: LiteLLM VLM API, including explicit LiteLLM routes such as `bedrock/`, `sagemaker/`, `vertex_ai/`, and `azure/`For `openai-codex`, authenticate through `openviking-server init`, then verify with `openviking-server doctor`.+For `litellm`, `api_key` can be omitted when the underlying route authenticates through+environment or provider-native credentials, such as AWS IAM/IRSA for Bedrock and+SageMaker or ADC/service-account credentials for Vertex AI. Azure routes still use+`api_key` normally. If you intentionally use LiteLLM's Bedrock bearer-token API-key+auth, set `forward_api_key` to `true`.+**Custom HTTP Headers**For OpenAI-compatible providers (e.g., OpenRouter), you can add custom HTTP headers via `extra_headers`:diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.mdindex dadd5c15f..bec043117 100644--- a/docs/zh/guides/01-configuration.md+++ b/docs/zh/guides/01-configuration.md@@ -551,7 +551,8 @@ openviking-server doctor| 参数 | 类型 | 说明 ||------|------|------|-| `api_key` | str | API Key |+| `api_key` | str | API Key。`openai-codex` 在 Codex OAuth 可用时可省略;使用 provider 原生凭据的 `litellm` 路由也可省略 |+| `forward_api_key` | bool | 仅 LiteLLM 使用。覆盖是否把 `api_key` 透传给 LiteLLM。默认情况下,OpenViking 不会把占位 key 透传给 `bedrock/`、`sagemaker/`、`vertex_ai/` 等 AWS/GCP 原生鉴权路由;如果明确使用 LiteLLM 的 Bedrock bearer-token API-key 鉴权,可设为 `true` || `model` | str | 模型名称 || `api_base` | str | API 端点(可选) || `thinking` | bool | 启用思考模式(仅对部分火山模型生效,默认:`false`) |@@ -585,9 +586,15 @@ openviking-server doctor- `openai-codex`:通过 ChatGPT/Codex OAuth 使用 Codex VLM- `kimi`:Kimi Coding 订阅端点,内置 provider 默认配置- `glm`:Z.AI GLM Coding Plan 端点,使用 OpenAI 兼容请求格式+- `litellm`:LiteLLM VLM API,支持 `bedrock/`、`sagemaker/`、`vertex_ai/`、`azure/` 等显式 LiteLLM 路由对于 `openai-codex`,请通过 `openviking-server init` 完成鉴权,再使用 `openviking-server doctor` 做校验。+对于 `litellm`,当底层路由使用环境变量或 provider 原生凭据时可以省略+`api_key`,例如 Bedrock/SageMaker 的 AWS IAM/IRSA,或 Vertex AI 的+ADC/service-account 凭据。Azure 路由仍会正常使用 `api_key`。如果明确要使用+LiteLLM 的 Bedrock bearer-token API-key 鉴权,请设置 `forward_api_key=true`。+**自定义 HTTP Headers**对于 OpenAI 兼容的 provider(如 OpenRouter),可以通过 `extra_headers` 添加自定义 HTTP 请求头:diff --git a/openviking/models/vlm/backends/litellm_vlm.py b/openviking/models/vlm/backends/litellm_vlm.pyindex cf58c76be..cdb81a681 100644--- a/openviking/models/vlm/backends/litellm_vlm.py+++ b/openviking/models/vlm/backends/litellm_vlm.py@@ -93,8 +93,39 @@ PROVIDER_CONFIGS: Dict[str, Dict[str, Any]] = {}+# Prefixes that are already complete LiteLLM routes. Keep them authoritative+# so keyword-based auto-detection does not rewrite cross-provider model names.+EXPLICIT_LITELLM_PREFIXES: tuple[str, ...] = (+ "azure/",+ "azure_ai/",+ "azure_text/",+ "bedrock/",+ "sagemaker/",+ "sagemaker_chat/",+ "sagemaker_nova/",+ "vertex_ai/",+)++# These explicit routes normally authenticate through cloud-native credential+# chains, so a placeholder api_key should not be forwarded by default.+NATIVE_AUTH_LITELLM_PREFIXES: tuple[str, ...] = (+ "bedrock/",+ "sagemaker/",+ "sagemaker_chat/",+ "sagemaker_nova/",+ "vertex_ai/",+)+++def _has_litellm_prefix(model: str, prefixes: tuple[str, ...]) -> bool:+ return model.lower().startswith(prefixes)++def detect_provider_by_model(model: str) -> str | None:"""Detect provider by model name."""+ if _has_litellm_prefix(model, EXPLICIT_LITELLM_PREFIXES):+ return None+model_lower = model.lower()for provider, config in PROVIDER_CONFIGS.items():if any(kw in model_lower for kw in config["keywords"]):@@ -115,6 +146,7 @@ class LiteLLMVLMProvider(VLMBase):self._provider_name = config.get("provider")self._extra_headers = config.get("extra_headers") or {}self._thinking = config.get("thinking", False)+ self._forward_api_key = config.get("forward_api_key")self._detected_provider: str | None = Noneif self.api_key:@@ -130,6 +162,8 @@ class LiteLLMVLMProvider(VLMBase):detected = detect_provider_by_model(model)if detected:provider = detected+ elif _has_litellm_prefix(model, EXPLICIT_LITELLM_PREFIXES):+ returnif provider and provider in PROVIDER_CONFIGS:env_key = PROVIDER_CONFIGS[provider]["env_key"]@@ -140,6 +174,9 @@ class LiteLLMVLMProvider(VLMBase):def _resolve_model(self, model: str) -> str:"""Resolve model name by applying provider prefixes."""+ if _has_litellm_prefix(model, EXPLICIT_LITELLM_PREFIXES):+ return model+provider = self._detected_provider or detect_provider_by_model(model)if provider and provider in PROVIDER_CONFIGS:@@ -154,6 +191,13 @@ class LiteLLMVLMProvider(VLMBase):return model+ def _should_forward_api_key(self, model: str) -> bool:+ if not self.api_key:+ return False+ if self._forward_api_key is not None:+ return self._forward_api_key is True+ return not _has_litellm_prefix(model, NATIVE_AUTH_LITELLM_PREFIXES)+def _detect_image_format(self, data: bytes) -> str:"""Detect image format from magic bytes.@@ -223,7 +267,7 @@ class LiteLLMVLMProvider(VLMBase):max_tokens = self.max_tokens or 32768kwargs["max_tokens"] = max_tokens- if self.api_key:+ if self._should_forward_api_key(model):kwargs["api_key"] = self.api_keyif self.api_base:is_google_endpoint = _is_google_generate_language_endpoint(self.api_base)diff --git a/openviking_cli/utils/config/vlm_config.py b/openviking_cli/utils/config/vlm_config.pyindex c2e52cd37..d8b7dc812 100644--- a/openviking_cli/utils/config/vlm_config.py+++ b/openviking_cli/utils/config/vlm_config.py@@ -21,9 +21,17 @@ def _normalize_provider_name(name: Optional[str]) -> Optional[str]:class VLMConfig(BaseModel):"""VLM configuration, supports multiple provider backends."""- backup: Optional["VLMConfig"] = Field(default=None, description="Backup VLM configuration for failover")+ backup: Optional["VLMConfig"] = Field(+ default=None, description="Backup VLM configuration for failover"+ )model: Optional[str] = Field(default=None, description="Model name")api_key: Optional[str] = Field(default=None, description="API key")+ forward_api_key: Optional[bool] = Field(+ default=None,+ description=(+ "Whether to pass api_key through to LiteLLM. None uses provider-aware defaults."+ ),+ )api_base: Optional[str] = Field(default=None, description="API base URL")temperature: float = Field(default=0.0, description="Generation temperature")max_retries: int = Field(default=3, description="Maximum retry attempts")@@ -130,7 +138,7 @@ class VLMConfig(BaseModel):raise ValueError("VLM configuration requires Codex OAuth credentials in ~/.openviking/codex_auth.json or an importable Codex CLI auth file")- elif not self._get_effective_api_key():+ elif provider_name != "litellm" and not self._get_effective_api_key():raise ValueError("VLM configuration requires 'api_key' to be set")return self@@ -139,16 +147,30 @@ class VLMConfig(BaseModel):"""Prevent recursive backup configurations"""if self.backup is not None:if self.backup.backup is not None:- raise ValueError("Backup VLM configuration cannot have its own backup (recursive backups are not allowed)")+ raise ValueError(+ "Backup VLM configuration cannot have its own backup (recursive backups are not allowed)"+ )return selfdef _migrate_legacy_config(self):"""Migrate legacy config to providers structure."""- if self.api_key and self.provider:+ if self.provider and (+ self.api_key+ or self.api_base+ or self.extra_headers+ or self.extra_request_body+ or self.stream+ or self.forward_api_key is not None+ ):if self.provider not in self.providers:self.providers[self.provider] = {}- if "api_key" not in self.providers[self.provider]:+ if self.api_key and "api_key" not in self.providers[self.provider]:self.providers[self.provider]["api_key"] = self.api_key+ if (+ self.forward_api_key is not None+ and "forward_api_key" not in self.providers[self.provider]+ ):+ self.providers[self.provider]["forward_api_key"] = self.forward_api_keyif self.api_base and "api_base" not in self.providers[self.provider]:self.providers[self.provider]["api_base"] = self.api_baseif self.extra_headers and "extra_headers" not in self.providers[self.provider]:@@ -182,6 +204,8 @@ class VLMConfig(BaseModel):config = dict(self.providers.get(provider_name) or {})if self.api_key and "api_key" not in config:config["api_key"] = self.api_key+ if self.forward_api_key is not None and "forward_api_key" not in config:+ config["forward_api_key"] = self.forward_api_keyif self.api_base and "api_base" not in config:config["api_base"] = self.api_baseif self.extra_headers and "extra_headers" not in config:@@ -195,6 +219,8 @@ class VLMConfig(BaseModel):def _provider_has_usable_credentials(self, provider_name: str, config: Dict[str, Any]) -> bool:if config.get("api_key"):return True+ if provider_name == "litellm":+ return Trueif provider_name == "openai-codex":has_codex_auth_available = _load_codex_auth_module().has_codex_auth_available@@ -246,7 +272,7 @@ class VLMConfig(BaseModel):"""Get VLM instance."""if self._vlm_instance is None:config_dict = self._build_vlm_config_dict()- from openviking.models.vlm import VLMFactory, FailoverVLM+ from openviking.models.vlm import FailoverVLM, VLMFactoryprimary = VLMFactory.create(config_dict)@@ -283,6 +309,8 @@ class VLMConfig(BaseModel):if config:if config.get("api_key"):result["api_key"] = config.get("api_key")+ if config.get("forward_api_key") is not None:+ result["forward_api_key"] = config.get("forward_api_key")if config.get("api_base"):result["api_base"] = config.get("api_base")if config.get("extra_headers"):@@ -330,6 +358,8 @@ class VLMConfig(BaseModel):has_codex_auth_available = _load_codex_auth_module().has_codex_auth_availablereturn bool(self._get_effective_api_key() or has_codex_auth_available())+ if self._resolve_provider_name() == "litellm":+ return bool(self.model)return self._get_effective_api_key() is not Nonedef get_vision_completion(diff --git a/tests/unit/test_extra_headers_vlm.py b/tests/unit/test_extra_headers_vlm.pyindex db676bb4e..ab43b6184 100644--- a/tests/unit/test_extra_headers_vlm.py+++ b/tests/unit/test_extra_headers_vlm.py@@ -4,7 +4,10 @@from unittest.mock import AsyncMock, MagicMock, patch-from openviking.models.vlm.backends.litellm_vlm import LiteLLMVLMProvider+from openviking.models.vlm.backends.litellm_vlm import (+ LiteLLMVLMProvider,+ detect_provider_by_model,+)from openviking.models.vlm.backends.openai_vlm import OpenAIVLM@@ -516,9 +519,119 @@ class TestVLMConfigExtraRequestBody:assert result["extra_request_body"] == {"think": False}+class TestVLMConfigLiteLLMAuth:+ """Test LiteLLM VLM config can rely on provider-native credentials."""++ def test_litellm_config_allows_no_api_key(self):+ from openviking_cli.utils.config.vlm_config import VLMConfig++ config = VLMConfig(+ model="bedrock/us.amazon.nova-micro-v1:0",+ provider="litellm",+ )++ result = config._build_vlm_config_dict()+ assert config.is_available()+ assert result["provider"] == "litellm"+ assert "api_key" not in result++ def test_litellm_forward_api_key_is_forwarded_from_provider_config(self):+ from openviking_cli.utils.config.vlm_config import VLMConfig++ config = VLMConfig(+ model="bedrock/us.amazon.nova-micro-v1:0",+ provider="litellm",+ providers={+ "litellm": {+ "api_key": "bedrock-bearer-token",+ "forward_api_key": True,+ }+ },+ )++ result = config._build_vlm_config_dict()+ assert result["api_key"] == "bedrock-bearer-token"+ assert result["forward_api_key"] is True++class TestLiteLLMVLMModelResolution:"""Regression tests for LiteLLM model prefix resolution."""+ def test_explicit_litellm_routes_are_authoritative(self):+ models = [+ "azure/gpt-4o",+ "bedrock/qwen.qwen3-coder-30b-a3b-v1:0",+ "bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0",+ "sagemaker/anthropic.claude-endpoint",+ "sagemaker_chat/qwen-endpoint",+ "sagemaker_nova/nova-endpoint",+ "vertex_ai/gemini-1.5-pro",+ ]++ for model in models:+ vlm = LiteLLMVLMProvider(+ {+ "model": model,+ "provider": "litellm",+ "api_key": "placeholder",+ }+ )++ assert detect_provider_by_model(model) is None+ assert vlm._detected_provider is None+ assert vlm._resolve_model(model) == model++ def test_native_auth_litellm_routes_skip_api_key_by_default(self):+ models = [+ "bedrock/us.amazon.nova-micro-v1:0",+ "bedrock/converse/anthropic.claude-3-haiku-20240307-v1:0",+ "sagemaker/anthropic.claude-endpoint",+ "sagemaker_chat/qwen-endpoint",+ "sagemaker_nova/nova-endpoint",+ "vertex_ai/gemini-1.5-pro",+ ]++ for model in models:+ vlm = LiteLLMVLMProvider(+ {+ "model": model,+ "provider": "litellm",+ "api_key": "placeholder",+ }+ )+ kwargs = vlm._build_text_kwargs(prompt="hello")++ assert kwargs["model"] == model+ assert "api_key" not in kwargs++ def test_azure_route_keeps_api_key(self):+ vlm = LiteLLMVLMProvider(+ {+ "model": "azure/gpt-4o",+ "provider": "litellm",+ "api_key": "azure-key",+ }+ )++ kwargs = vlm._build_text_kwargs(prompt="hello")++ assert kwargs["model"] == "azure/gpt-4o"+ assert kwargs["api_key"] == "azure-key"++ def test_forward_api_key_overrides_native_auth_default(self):+ vlm = LiteLLMVLMProvider(+ {+ "model": "bedrock/us.amazon.nova-micro-v1:0",+ "provider": "litellm",+ "api_key": "bedrock-bearer-token",+ "forward_api_key": True,+ }+ )++ kwargs = vlm._build_text_kwargs(prompt="hello")++ assert kwargs["api_key"] == "bedrock-bearer-token"+def test_zhipu_zai_model_keeps_existing_zai_prefix(self):"""Zhipu GLM models already using LiteLLM's zai/ prefix must not be double-prefixed."""vlm = LiteLLMVLMProvider(