Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
Fix image_summary bug (#277)
JJFc9b696dc7cad
diff --git a/README.md b/README.mdindex 9ef267d0b..fa5b5b65f 100644--- a/README.md+++ b/README.md@@ -91,23 +91,17 @@ OpenViking requires the following model capabilities:#### Supported VLM Providers-OpenViking supports multiple VLM providers:--| Provider | Model | Get API Key |-|----------|-------|-------------|-| `volcengine` | doubao | [Volcengine Console](https://console.volcengine.com/ark) |-| `openai` | gpt | [OpenAI Platform](https://platform.openai.com) |-| `anthropic` | claude | [Anthropic Console](https://console.anthropic.com) |-| `deepseek` | deepseek | [DeepSeek Platform](https://platform.deepseek.com) |-| `gemini` | gemini | [Google AI Studio](https://aistudio.google.com) |-| `moonshot` | kimi | [Moonshot Platform](https://platform.moonshot.cn) |-| `zhipu` | glm | [Zhipu Open Platform](https://open.bigmodel.cn) |-| `dashscope` | qwen | [DashScope Console](https://dashscope.console.aliyun.com) |-| `minimax` | minimax | [MiniMax Platform](https://platform.minimax.io) |-| `openrouter` | (any model) | [OpenRouter](https://openrouter.ai) |-| `vllm` | (local model) | — |--> 💡 **Tip**: OpenViking uses a **Provider Registry** for unified model access. The system automatically detects the provider based on model name keywords, so you can switch between providers seamlessly.+OpenViking supports three VLM providers:++| Provider | Description | Get API Key |+|----------|-------------|-------------|+| `volcengine` | 火山引擎豆包模型 | [Volcengine Console](https://console.volcengine.com/ark) |+| `openai` | OpenAI 官方 API | [OpenAI Platform](https://platform.openai.com) |+| `litellm` | 统一调用多种第三方模型 (Anthropic, DeepSeek, Gemini, vLLM, Ollama, etc.) | See [LiteLLM Providers](https://docs.litellm.ai/docs/providers) |++> 💡 **Tip**:+> - `litellm` 支持通过统一接口调用多种模型,model 字段需遵循 [LiteLLM 格式规范](https://docs.litellm.ai/docs/providers)+> - 系统自动检测常见模型(如 `claude-*`, `deepseek-*`, `gemini-*`, `hosted_vllm/*`, `ollama/*` 等),其他模型需按 LiteLLM 格式填写完整前缀#### Provider-Specific Notes@@ -122,7 +116,7 @@ Volcengine supports both model names and endpoint IDs. Using model names is reco"provider": "volcengine","model": "doubao-seed-1-6-240615","api_key": "your-api-key",- "api_base" : "https://ark.cn-beijing.volces.com/api/v3",+ "api_base": "https://ark.cn-beijing.volces.com/api/v3"}}```@@ -135,7 +129,7 @@ You can also use endpoint IDs (found in [Volcengine ARK Console](https://console"provider": "volcengine","model": "ep-20241220174930-xxxxx","api_key": "your-api-key",- "api_base" : "https://ark.cn-beijing.volces.com/api/v3",+ "api_base": "https://ark.cn-beijing.volces.com/api/v3"}}```@@ -143,17 +137,30 @@ You can also use endpoint IDs (found in [Volcengine ARK Console](https://console</details><details>-<summary><b>Zhipu AI (智谱)</b></summary>+<summary><b>OpenAI</b></summary>++Use OpenAI's official API:++```json+{+ "vlm": {+ "provider": "openai",+ "model": "gpt-4o",+ "api_key": "your-api-key",+ "api_base": "https://api.openai.com/v1"+ }+}+```-If you're on Zhipu's coding plan, use the coding API endpoint:+You can also use a custom OpenAI-compatible endpoint:```json{"vlm": {- "provider": "zhipu",- "model": "glm-4-plus",+ "provider": "openai",+ "model": "gpt-4o","api_key": "your-api-key",- "api_base": "https://open.bigmodel.cn/api/coding/paas/v4"+ "api_base": "https://your-custom-endpoint.com/v1"}}```@@ -161,44 +168,52 @@ If you're on Zhipu's coding plan, use the coding API endpoint:</details><details>-<summary><b>MiniMax (中国大陆)</b></summary>+<summary><b>LiteLLM (Anthropic, DeepSeek, Gemini, vLLM, Ollama, etc.)</b></summary>-For MiniMax's mainland China platform (minimaxi.com), specify the API base:+LiteLLM provides unified access to various models. The `model` field should follow LiteLLM's naming convention:```json{"vlm": {- "provider": "minimax",- "model": "abab6.5s-chat",- "api_key": "your-api-key",- "api_base": "https://api.minimaxi.com/v1"+ "provider": "litellm",+ "model": "claude-3-5-sonnet-20240620",+ "api_key": "your-anthropic-api-key"}}```-</details>+**Common model formats:**-<details>-<summary><b>Local Models (vLLM)</b></summary>+| Provider | Model Example | Notes |+|----------|---------------|-------|+| Anthropic | `claude-3-5-sonnet-20240620` | Auto-detected, uses `ANTHROPIC_API_KEY` |+| DeepSeek | `deepseek-chat` | Auto-detected, uses `DEEPSEEK_API_KEY` |+| Gemini | `gemini-pro` | Auto-detected, uses `GEMINI_API_KEY` |+| OpenRouter | `openrouter/openai/gpt-4o` | Full prefix required |+| vLLM | `hosted_vllm/llama-3.1-8b` | Set `api_base` to vLLM server |+| Ollama | `ollama/llama3.1` | Set `api_base` to Ollama server |-Run OpenViking with your own local models using vLLM:+**Local Models (vLLM / Ollama):**```bash-# Start vLLM server-vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000++# Start Ollama+ollama serve``````json+// Ollama{"vlm": {- "provider": "vllm",- "model": "meta-llama/Llama-3.1-8B-Instruct",- "api_key": "dummy",- "api_base": "http://localhost:8000/v1"+ "provider": "litellm",+ "model": "ollama/llama3.1",+ "api_base": "http://localhost:11434"}}```+For complete model support, see [LiteLLM Providers Documentation](https://docs.litellm.ai/docs/providers).+</details>### 3. Environment Configuration@@ -234,7 +249,7 @@ Create a configuration file `~/.openviking/ov.conf`, remove the comments before}```-> **Note**: For embedding models, currently `volcengine` (Doubao), `openai`, and `jina` providers are supported. For VLM models, we support multiple providers including volcengine, openai, deepseek, anthropic, gemini, moonshot, zhipu, dashscope, minimax, and more.+> **Note**: For embedding models, currently `volcengine` (Doubao), `openai`, and `jina` providers are supported. For VLM models, we support three providers: `volcengine`, `openai`, and `litellm`. The `litellm` provider supports various models including Anthropic (Claude), DeepSeek, Gemini, Moonshot, Zhipu, DashScope, MiniMax, vLLM, Ollama, and more.#### Configuration Examplesdiff --git a/openviking/models/vlm/__init__.py b/openviking/models/vlm/__init__.pyindex e9d01f4f1..e58b51966 100644--- a/openviking/models/vlm/__init__.py+++ b/openviking/models/vlm/__init__.py@@ -6,14 +6,7 @@ from .backends.litellm_vlm import LiteLLMVLMProviderfrom .backends.openai_vlm import OpenAIVLMfrom .backends.volcengine_vlm import VolcEngineVLMfrom .base import VLMBase, VLMFactory-from .registry import (- PROVIDERS,- ProviderSpec,- find_by_model,- find_by_name,- find_gateway,- get_all_provider_names,-)+from .registry import get_all_provider_names, is_valid_provider__all__ = ["VLMBase",@@ -21,10 +14,6 @@ __all__ = ["OpenAIVLM","VolcEngineVLM","LiteLLMVLMProvider",- "ProviderSpec",- "PROVIDERS",- "find_by_model",- "find_by_name",- "find_gateway","get_all_provider_names",+ "is_valid_provider",]diff --git a/openviking/models/vlm/backends/litellm_vlm.py b/openviking/models/vlm/backends/litellm_vlm.pyindex b31579c50..a537f106f 100644--- a/openviking/models/vlm/backends/litellm_vlm.py+++ b/openviking/models/vlm/backends/litellm_vlm.py@@ -2,6 +2,7 @@# SPDX-License-Identifier: Apache-2.0"""LiteLLM VLM Provider implementation with multi-provider support."""+import loggingimport osos.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"@@ -15,16 +16,82 @@ import litellmfrom litellm import acompletion, completionfrom ..base import VLMBase-from ..registry import find_by_model, find_gateway++logger = logging.getLogger(__name__)++PROVIDER_CONFIGS: Dict[str, Dict[str, Any]] = {+ "openrouter": {+ "keywords": ("openrouter",),+ "env_key": "OPENROUTER_API_KEY",+ "litellm_prefix": "openrouter",+ },+ "hosted_vllm": {+ "keywords": ("hosted_vllm",),+ "env_key": "HOSTED_VLLM_API_KEY",+ "litellm_prefix": "hosted_vllm",+ },+ "ollama": {+ "keywords": ("ollama",),+ "env_key": "OLLAMA_API_KEY",+ "litellm_prefix": "ollama",+ },+ "anthropic": {+ "keywords": ("claude", "anthropic"),+ "env_key": "ANTHROPIC_API_KEY",+ "litellm_prefix": "anthropic",+ },+ "deepseek": {+ "keywords": ("deepseek",),+ "env_key": "DEEPSEEK_API_KEY",+ "litellm_prefix": "deepseek",+ },+ "gemini": {+ "keywords": ("gemini",),+ "env_key": "GEMINI_API_KEY",+ "litellm_prefix": "gemini",+ },+ "openai": {+ "keywords": ("gpt", "o1", "o3", "o4"),+ "env_key": "OPENAI_API_KEY",+ "litellm_prefix": "",+ },+ "moonshot": {+ "keywords": ("moonshot", "kimi"),+ "env_key": "MOONSHOT_API_KEY",+ "litellm_prefix": "moonshot",+ },+ "zhipu": {+ "keywords": ("glm", "zhipu"),+ "env_key": "ZHIPUAI_API_KEY",+ "litellm_prefix": "zhipu",+ },+ "dashscope": {+ "keywords": ("qwen", "dashscope"),+ "env_key": "DASHSCOPE_API_KEY",+ "litellm_prefix": "dashscope",+ },+ "minimax": {+ "keywords": ("minimax",),+ "env_key": "MINIMAX_API_KEY",+ "litellm_prefix": "minimax",+ },+}+++def detect_provider_by_model(model: str) -> str | None:+ """Detect provider by model name."""+ model_lower = model.lower()+ for provider, config in PROVIDER_CONFIGS.items():+ if any(kw in model_lower for kw in config["keywords"]):+ return provider+ return Noneclass LiteLLMVLMProvider(VLMBase):"""Multi-provider VLM implementation based on LiteLLM.- Supports OpenRouter, Anthropic, OpenAI, Gemini, DeepSeek, VolcEngine and many other providers- through a unified interface. Provider-specific logic is driven by the registry- (see providers/registry.py) — no if-elif chains needed here.+ Supports various providers through LiteLLM's unified interface."""def __init__(self, config: Dict[str, Any]):@@ -33,11 +100,10 @@ class LiteLLMVLMProvider(VLMBase):self._provider_name = config.get("provider")self._extra_headers = config.get("extra_headers") or {}self._thinking = config.get("thinking", False)-- self._gateway = find_gateway(self._provider_name, self.api_key, self.api_base)+ self._detected_provider: str | None = Noneif self.api_key:- self._setup_env(self.api_key, self.api_base, self.model)+ self._setup_env(self.api_key, self.model)if self.api_base:litellm.api_base = self.api_base@@ -45,66 +111,63 @@ class LiteLLMVLMProvider(VLMBase):litellm.suppress_debug_info = Truelitellm.drop_params = True- def _setup_env(self, api_key: str, api_base: str | None, model: str | None) -> None:+ def _setup_env(self, api_key: str, model: str | None) -> None:"""Set environment variables based on detected provider."""- spec = self._gateway or find_by_model(model or "")- if not spec:- return-- if self._gateway:- os.environ[spec.env_key] = api_key+ provider = self._provider_name+ if not provider and model:+ provider = detect_provider_by_model(model)++ if provider and provider in PROVIDER_CONFIGS:+ env_key = PROVIDER_CONFIGS[provider]["env_key"]+ os.environ[env_key] = api_key+ self._detected_provider = providerelse:- os.environ.setdefault(spec.env_key, api_key)-- effective_base = api_base or spec.default_api_base- for env_name, env_val in spec.env_extras:- resolved = env_val.replace("{api_key}", api_key)- resolved = resolved.replace("{api_base}", effective_base or "")- os.environ.setdefault(env_name, resolved)+ os.environ["OPENAI_API_KEY"] = api_keydef _resolve_model(self, model: str) -> str:- """Resolve model name by applying provider/gateway prefixes."""- if self._gateway:- if self._gateway.strip_model_prefix:- model = model.split("/")[-1]- prefix = self._gateway.litellm_prefix+ """Resolve model name by applying provider prefixes."""+ provider = self._detected_provider or detect_provider_by_model(model)++ if provider and provider in PROVIDER_CONFIGS:+ prefix = PROVIDER_CONFIGS[provider]["litellm_prefix"]if prefix and not model.startswith(f"{prefix}/"):- model = f"{prefix}/{model}"+ return f"{prefix}/{model}"return model- if self._provider_name == "openai" and self.api_base:- from openviking.models.vlm.registry import find_by_name- openai_spec = find_by_name("openai")- is_openai_official = "api.openai.com" in self.api_base- if openai_spec and not is_openai_official and not model.startswith("openai/"):- return f"openai/{model}"-- spec = find_by_model(model)- if spec and spec.litellm_prefix:- if not any(model.startswith(s) for s in spec.skip_prefixes):- model = f"{spec.litellm_prefix}/{model}"+ if self.api_base and not model.startswith(("openai/", "hosted_vllm/", "ollama/")):+ return f"openai/{model}"+return model- def _apply_model_overrides(self, model: str, kwargs: dict[str, Any]) -> None:- """Apply model-specific parameter overrides from the registry."""- model_lower = model.lower()- spec = find_by_model(model)- if spec:- for pattern, overrides in spec.model_overrides:- if pattern in model_lower:- kwargs.update(overrides)- return+ def _detect_image_format(self, data: bytes) -> str:+ """Detect image format from magic bytes.++ Supported formats: PNG, JPEG, GIF, WebP+ """+ if len(data) < 8:+ logger.warning(f"[LiteLLMVLM] Image data too small: {len(data)} bytes")+ return "image/png"++ if data[:8] == b"\x89PNG\r\n\x1a\n":+ return "image/png"+ elif data[:2] == b"\xff\xd8":+ return "image/jpeg"+ elif data[:6] in (b"GIF87a", b"GIF89a"):+ return "image/gif"+ elif data[:4] == b"RIFF" and len(data) >= 12 and data[8:12] == b"WEBP":+ return "image/webp"- if self._provider_name == "volcengine":- kwargs["thinking"] = {"type": "enabled" if self._thinking else "disabled"}+ logger.warning(f"[LiteLLMVLM] Unknown image format, magic bytes: {data[:8].hex()}")+ return "image/png"def _prepare_image(self, image: Union[str, Path, bytes]) -> Dict[str, Any]:"""Prepare image data for vision completion."""if isinstance(image, bytes):b64 = base64.b64encode(image).decode("utf-8")+ mime_type = self._detect_image_format(image)return {"type": "image_url",- "image_url": {"url": f"data:image/png;base64,{b64}"},+ "image_url": {"url": f"data:{mime_type};base64,{b64}"},}elif isinstance(image, Path) or (isinstance(image, str) and not image.startswith(("http://", "https://"))@@ -119,7 +182,8 @@ class LiteLLMVLMProvider(VLMBase):".webp": "image/webp",}.get(suffix, "image/png")with open(path, "rb") as f:- b64 = base64.b64encode(f.read()).decode("utf-8")+ data = f.read()+ b64 = base64.b64encode(data).decode("utf-8")return {"type": "image_url","image_url": {"url": f"data:{mime_type};base64,{b64}"},@@ -135,8 +199,6 @@ class LiteLLMVLMProvider(VLMBase):"temperature": self.temperature,}- self._apply_model_overrides(model, kwargs)-if self.api_key:kwargs["api_key"] = self.api_keyif self.api_base:@@ -150,11 +212,7 @@ class LiteLLMVLMProvider(VLMBase):"""Get text completion synchronously."""model = self._resolve_model(self.model or "gpt-4o-mini")messages = [{"role": "user", "content": prompt}]- original_thinking = self._thinking- if thinking:- self._thinking = thinkingkwargs = self._build_kwargs(model, messages)- self._thinking = original_thinkingresponse = completion(**kwargs)self._update_token_usage_from_response(response)@@ -166,11 +224,7 @@ class LiteLLMVLMProvider(VLMBase):"""Get text completion asynchronously."""model = self._resolve_model(self.model or "gpt-4o-mini")messages = [{"role": "user", "content": prompt}]- original_thinking = self._thinking- if thinking:- self._thinking = thinkingkwargs = self._build_kwargs(model, messages)- self._thinking = original_thinkinglast_error = Nonefor attempt in range(max_retries + 1):@@ -202,11 +256,7 @@ class LiteLLMVLMProvider(VLMBase):content.append({"type": "text", "text": prompt})messages = [{"role": "user", "content": content}]- original_thinking = self._thinking- if thinking:- self._thinking = thinkingkwargs = self._build_kwargs(model, messages)- self._thinking = original_thinkingresponse = completion(**kwargs)self._update_token_usage_from_response(response)@@ -227,11 +277,7 @@ class LiteLLMVLMProvider(VLMBase):content.append({"type": "text", "text": prompt})messages = [{"role": "user", "content": content}]- original_thinking = self._thinking- if thinking:- self._thinking = thinkingkwargs = self._build_kwargs(model, messages)- self._thinking = original_thinkingresponse = await acompletion(**kwargs)self._update_token_usage_from_response(response)diff --git a/openviking/models/vlm/backends/openai_vlm.py b/openviking/models/vlm/backends/openai_vlm.pyindex d6f6effa9..de59b8dc7 100644--- a/openviking/models/vlm/backends/openai_vlm.py+++ b/openviking/models/vlm/backends/openai_vlm.py@@ -4,11 +4,14 @@import asyncioimport base64+import loggingfrom pathlib import Pathfrom typing import Any, Dict, List, Unionfrom ..base import VLMBase+logger = logging.getLogger(__name__)+class OpenAIVLM(VLMBase):"""OpenAI VLM backend"""@@ -17,7 +20,6 @@ class OpenAIVLM(VLMBase):super().__init__(config)self._sync_client = Noneself._async_client = None- # Ensure provider type is correctself.provider = "openai"def get_client(self):@@ -92,13 +94,35 @@ class OpenAIVLM(VLMBase):else:raise RuntimeError("Unknown error in async completion")+ def _detect_image_format(self, data: bytes) -> str:+ """Detect image format from magic bytes.++ Supported formats: PNG, JPEG, GIF, WebP+ """+ if len(data) < 8:+ logger.warning(f"[OpenAIVLM] Image data too small: {len(data)} bytes")+ return "image/png"++ if data[:8] == b"\x89PNG\r\n\x1a\n":+ return "image/png"+ elif data[:2] == b"\xff\xd8":+ return "image/jpeg"+ elif data[:6] in (b"GIF87a", b"GIF89a"):+ return "image/gif"+ elif data[:4] == b"RIFF" and len(data) >= 12 and data[8:12] == b"WEBP":+ return "image/webp"++ logger.warning(f"[OpenAIVLM] Unknown image format, magic bytes: {data[:8].hex()}")+ return "image/png"+def _prepare_image(self, image: Union[str, Path, bytes]) -> Dict[str, Any]:- """Prepare image data"""+ """Prepare image data for vision completion."""if isinstance(image, bytes):b64 = base64.b64encode(image).decode("utf-8")+ mime_type = self._detect_image_format(image)return {"type": "image_url",- "image_url": {"url": f"data:image/png;base64,{b64}"},+ "image_url": {"url": f"data:{mime_type};base64,{b64}"},}elif isinstance(image, Path) or (isinstance(image, str) and not image.startswith(("http://", "https://"))@@ -113,7 +137,8 @@ class OpenAIVLM(VLMBase):".webp": "image/webp",}.get(suffix, "image/png")with open(path, "rb") as f:- b64 = base64.b64encode(f.read()).decode("utf-8")+ data = f.read()+ b64 = base64.b64encode(data).decode("utf-8")return {"type": "image_url","image_url": {"url": f"data:{mime_type};base64,{b64}"},diff --git a/openviking/models/vlm/backends/volcengine_vlm.py b/openviking/models/vlm/backends/volcengine_vlm.pyindex e4c4d2909..3b80bdc67 100644--- a/openviking/models/vlm/backends/volcengine_vlm.py+++ b/openviking/models/vlm/backends/volcengine_vlm.py@@ -4,11 +4,14 @@import asyncioimport base64+import loggingfrom pathlib import Pathfrom typing import Any, Dict, List, Unionfrom .openai_vlm import OpenAIVLM+logger = logging.getLogger(__name__)+class VolcEngineVLM(OpenAIVLM):"""VolcEngine VLM backend"""@@ -98,13 +101,83 @@ class VolcEngineVLM(OpenAIVLM):else:raise RuntimeError("Unknown error in async completion")+ def _detect_image_format(self, data: bytes) -> str:+ """Detect image format from magic bytes.++ Returns the MIME type, or raises ValueError for unsupported formats like SVG.++ Supported formats per VolcEngine docs:+ https://www.volcengine.com/docs/82379/1362931+ - JPEG, PNG, GIF, WEBP, BMP, TIFF, ICO, DIB, ICNS, SGI, JPEG2000, HEIC, HEIF+ """+ if len(data) < 12:+ logger.warning(f"[VolcEngineVLM] Image data too small: {len(data)} bytes")+ return "image/png"++ # PNG: 89 50 4E 47 0D 0A 1A 0A+ if data[:8] == b"\x89PNG\r\n\x1a\n":+ return "image/png"+ # JPEG: FF D8+ elif data[:2] == b"\xff\xd8":+ return "image/jpeg"+ # GIF: GIF87a or GIF89a+ elif data[:6] in (b"GIF87a", b"GIF89a"):+ return "image/gif"+ # WEBP: RIFF....WEBP+ elif data[:4] == b"RIFF" and len(data) >= 12 and data[8:12] == b"WEBP":+ return "image/webp"+ # BMP: BM+ elif data[:2] == b"BM":+ return "image/bmp"+ # TIFF (little-endian): 49 49 2A 00+ # TIFF (big-endian): 4D 4D 00 2A+ elif data[:4] == b"II*\x00" or data[:4] == b"MM\x00*":+ return "image/tiff"+ # ICO: 00 00 01 00+ elif data[:4] == b"\x00\x00\x01\x00":+ return "image/ico"+ # ICNS: 69 63 6E 73 ("icns")+ elif data[:4] == b"icns":+ return "image/icns"+ # SGI: 01 DA+ elif data[:2] == b"\x01\xda":+ return "image/sgi"+ # JPEG2000: 00 00 00 0C 6A 50 20 20 (JP2 signature)+ elif data[:8] == b"\x00\x00\x00\x0cjP " or data[:4] == b"\xff\x4f\xff\x51":+ return "image/jp2"+ # HEIC/HEIF: ftyp box with heic/heif brand+ # 00 00 00 XX 66 74 79 70 68 65 69 63 (heic)+ # 00 00 00 XX 66 74 79 70 68 65 69 66 (heif)+ elif len(data) >= 12 and data[4:8] == b"ftyp":+ brand = data[8:12]+ if brand == b"heic":+ return "image/heic"+ elif brand == b"heif":+ return "image/heif"+ elif brand[:3] == b"mif":+ return "image/heif"+ # SVG (not supported)+ elif data[:4] == b"<svg" or (data[:5] == b"<?xml" and b"<svg" in data[:100]):+ raise ValueError(+ "SVG format is not supported by VolcEngine VLM API. "+ "Supported formats: JPEG, PNG, GIF, WEBP, BMP, TIFF, ICO, ICNS, SGI, JPEG2000, HEIC, HEIF"+ )++ # Unknown format - log and default to PNG+ logger.warning(f"[VolcEngineVLM] Unknown image format, magic bytes: {data[:16].hex()}")+ return "image/png"+def _prepare_image(self, image: Union[str, Path, bytes]) -> Dict[str, Any]:"""Prepare image data"""if isinstance(image, bytes):b64 = base64.b64encode(image).decode("utf-8")+ mime_type = self._detect_image_format(image)+ logger.info(+ f"[VolcEngineVLM] Preparing image from bytes, size={len(image)}, detected mime={mime_type}"+ )return {"type": "image_url",- "image_url": {"url": f"data:image/png;base64,{b64}"},+ "image_url": {"url": f"data:{mime_type};base64,{b64}"},}elif isinstance(image, Path) or (isinstance(image, str) and not image.startswith(("http://", "https://"))@@ -117,6 +190,21 @@ class VolcEngineVLM(OpenAIVLM):".jpeg": "image/jpeg",".gif": "image/gif",".webp": "image/webp",+ ".bmp": "image/bmp",+ ".dib": "image/bmp",+ ".tiff": "image/tiff",+ ".tif": "image/tiff",+ ".ico": "image/ico",+ ".icns": "image/icns",+ ".sgi": "image/sgi",+ ".j2c": "image/jp2",+ ".j2k": "image/jp2",+ ".jp2": "image/jp2",+ ".jpc": "image/jp2",+ ".jpf": "image/jp2",+ ".jpx": "image/jp2",+ ".heic": "image/heic",+ ".heif": "image/heif",}.get(suffix, "image/png")with open(path, "rb") as f:b64 = base64.b64encode(f.read()).decode("utf-8")diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.pyindex cd563f9c3..4b602f780 100644--- a/openviking/models/vlm/base.py+++ b/openviking/models/vlm/base.py@@ -125,23 +125,22 @@ class VLMFactory:ValueError: If provider is not supportedImportError: If related dependencies are not installed"""- provider = config.get("provider") or config.get("backend") or "openai"+ provider = (config.get("provider") or config.get("backend") or "openai").lower()- use_litellm = config.get("use_litellm", True)+ if provider == "volcengine":+ from .backends.volcengine_vlm import VolcEngineVLM- if not use_litellm:- if provider == "openai":- from .backends.openai_vlm import OpenAIVLM+ return VolcEngineVLM(config)- return OpenAIVLM(config)- elif provider == "volcengine":- from .backends.volcengine_vlm import VolcEngineVLM+ elif provider == "openai":+ from .backends.openai_vlm import OpenAIVLM- return VolcEngineVLM(config)+ return OpenAIVLM(config)- from .backends.litellm_vlm import LiteLLMVLMProvider+ else:+ from .backends.litellm_vlm import LiteLLMVLMProvider- return LiteLLMVLMProvider(config)+ return LiteLLMVLMProvider(config)@staticmethoddef get_available_providers() -> List[str]:diff --git a/openviking/models/vlm/registry.py b/openviking/models/vlm/registry.pyindex 815b637a5..ff3ddb3bf 100644--- a/openviking/models/vlm/registry.py+++ b/openviking/models/vlm/registry.py@@ -3,229 +3,19 @@"""Provider Registry — single source of truth for LLM provider metadata.-Adding a new provider:- 1. Add a ProviderSpec to PROVIDERS below.- 2. Use it in config with providers["newprovider"] = {"api_key": "xxx"}- Done. Env vars, prefixing, config matching, status display all derive from here.--Order matters — it controls match priority and fallback. Gateways first.-Every entry writes out all fields so you can copy-paste as a template.+Supported providers: volcengine, openai, litellm"""from __future__ import annotations-from dataclasses import dataclass-from typing import Any, Optional---@dataclass(frozen=True)-class ProviderSpec:- """VLM Provider metadata definition.-- Placeholders in env_extras values:- {api_key} - the user's API key- {api_base} - api_base from config, or this spec's default_api_base- """-- name: str- keywords: tuple[str, ...]- env_key: str- display_name: str = ""-- litellm_prefix: str = ""- skip_prefixes: tuple[str, ...] = ()-- env_extras: tuple[tuple[str, str], ...] = ()-- is_gateway: bool = False- is_local: bool = False- detect_by_key_prefix: str = ""- detect_by_base_keyword: str = ""- default_api_base: str = ""-- strip_model_prefix: bool = False-- model_overrides: tuple[tuple[str, dict[str, Any]], ...] = ()-- @property- def label(self) -> str:- return self.display_name or self.name.title()---PROVIDERS: tuple[ProviderSpec, ...] = (- ProviderSpec(- name="custom",- keywords=(),- env_key="OPENAI_API_KEY",- display_name="Custom",- litellm_prefix="openai",- skip_prefixes=("openai/",),- is_gateway=True,- strip_model_prefix=True,- ),-- ProviderSpec(- name="openrouter",- keywords=("openrouter",),- env_key="OPENROUTER_API_KEY",- display_name="OpenRouter",- litellm_prefix="openrouter",- is_gateway=True,- detect_by_key_prefix="sk-or-",- detect_by_base_keyword="openrouter",- default_api_base="https://openrouter.ai/api/v1",- ),-- ProviderSpec(- name="volcengine",- keywords=("doubao", "volcengine", "ep-"),- env_key="VOLCENGINE_API_KEY",- display_name="VolcEngine",- litellm_prefix="volcengine",- skip_prefixes=("volcengine/",),- default_api_base="https://ark.cn-beijing.volces.com/api/v3",- ),-- ProviderSpec(- name="openai",- keywords=("openai", "gpt"),- env_key="OPENAI_API_KEY",- display_name="OpenAI",- litellm_prefix="",- ),-- ProviderSpec(- name="anthropic",- keywords=("anthropic", "claude"),- env_key="ANTHROPIC_API_KEY",- display_name="Anthropic",- litellm_prefix="",- ),-- ProviderSpec(- name="deepseek",- keywords=("deepseek",),- env_key="DEEPSEEK_API_KEY",- display_name="DeepSeek",- litellm_prefix="deepseek",- skip_prefixes=("deepseek/",),- ),-- ProviderSpec(- name="gemini",- keywords=("gemini",),- env_key="GEMINI_API_KEY",- display_name="Gemini",- litellm_prefix="gemini",- skip_prefixes=("gemini/",),- ),-- ProviderSpec(- name="moonshot",- keywords=("moonshot", "kimi"),- env_key="MOONSHOT_API_KEY",- display_name="Moonshot",- litellm_prefix="moonshot",- skip_prefixes=("moonshot/",),- env_extras=(- ("MOONSHOT_API_BASE", "{api_base}"),- ),- default_api_base="https://api.moonshot.ai/v1",- model_overrides=(- ("kimi-k2.5", {"temperature": 1.0}),- ),- ),-- ProviderSpec(- name="zhipu",- keywords=("zhipu", "glm", "zai"),- env_key="ZAI_API_KEY",- display_name="Zhipu AI",- litellm_prefix="zai",- skip_prefixes=("zhipu/", "zai/"),- env_extras=(- ("ZHIPUAI_API_KEY", "{api_key}"),- ),- ),-- ProviderSpec(- name="dashscope",- keywords=("qwen", "dashscope"),- env_key="DASHSCOPE_API_KEY",- display_name="DashScope",- litellm_prefix="dashscope",- skip_prefixes=("dashscope/",),- ),-- ProviderSpec(- name="minimax",- keywords=("minimax",),- env_key="MINIMAX_API_KEY",- display_name="MiniMax",- litellm_prefix="minimax",- skip_prefixes=("minimax/",),- default_api_base="https://api.minimax.io/v1",- ),-- ProviderSpec(- name="vllm",- keywords=("vllm",),- env_key="HOSTED_VLLM_API_KEY",- display_name="vLLM/Local",- litellm_prefix="hosted_vllm",- is_local=True,- ),-)---def find_by_model(model: str) -> ProviderSpec | None:- """Match a standard provider by model-name keyword (case-insensitive).- Skips gateways/local — those are matched by api_key/api_base instead."""- model_lower = model.lower()- for spec in PROVIDERS:- if spec.is_gateway or spec.is_local:- continue- if any(kw in model_lower for kw in spec.keywords):- return spec- return None---def find_gateway(- provider_name: str | None = None,- api_key: str | None = None,- api_base: str | None = None,-) -> ProviderSpec | None:- """Detect gateway/local provider.-- Priority:- 1. provider_name — if it maps to a gateway/local spec, use it directly.- 2. api_key prefix — e.g. "sk-or-" → OpenRouter.- 3. api_base keyword — e.g. "aihubmix" in URL → AiHubMix.- """- if provider_name:- spec = find_by_name(provider_name)- if spec and (spec.is_gateway or spec.is_local):- return spec-- for spec in PROVIDERS:- if spec.detect_by_key_prefix and api_key and api_key.startswith(spec.detect_by_key_prefix):- return spec-- for spec in PROVIDERS:- if spec.detect_by_base_keyword and api_base and spec.detect_by_base_keyword in api_base:- return spec-- return None---def find_by_name(name: str) -> ProviderSpec | None:- """Find a provider spec by config field name, e.g. "dashscope"."""- for spec in PROVIDERS:- if spec.name == name:- return spec- return None+VALID_PROVIDERS: tuple[str, ...] = ("volcengine", "openai", "litellm")def get_all_provider_names() -> list[str]:"""Get all provider names list."""- return [spec.name for spec in PROVIDERS]+ return list(VALID_PROVIDERS)+++def is_valid_provider(name: str) -> bool:+ """Check if provider name is valid."""+ return name.lower() in VALID_PROVIDERSdiff --git a/openviking/parse/parsers/media/utils.py b/openviking/parse/parsers/media/utils.pyindex 1e8ad30d5..4615c285e 100644--- a/openviking/parse/parsers/media/utils.py+++ b/openviking/parse/parsers/media/utils.py@@ -17,6 +17,38 @@ from .constants import AUDIO_EXTENSIONS, IMAGE_EXTENSIONS, VIDEO_EXTENSIONSlogger = get_logger(__name__)+def _is_svg(data: bytes) -> bool:+ """Check if the data is an SVG file."""+ return data[:4] == b"<svg" or (data[:5] == b"<?xml" and b"<svg" in data[:100])+++# SVG to PNG conversion (disabled by default)+# Uncomment and install dependencies if you need SVG support:+# Ubuntu/Debian: sudo apt-get install libcairo2 && pip install cairosvg+# macOS: brew install cairo && pip install cairosvg+# Or use ImageMagick: sudo apt-get install libmagickwand-dev && pip install Wand+#+# def _convert_svg_to_png(svg_data: bytes) -> Optional[bytes]:+# """Convert SVG to PNG using cairosvg or wand."""+# try:+# import cairosvg+# return cairosvg.svg2png(bytestring=svg_data)+# except ImportError:+# pass+# except OSError:+# pass # libcairo not installed+#+# try:+# from wand.image import Image as WandImage+# with WandImage(blob=svg_data, format='svg') as img:+# img.format = 'png'+# return img.make_blob()+# except ImportError:+# pass+#+# return None++def get_media_type(source_path: Optional[str], source_format: Optional[str]) -> Optional[str]:"""Determine media type from source path or format.@@ -85,6 +117,14 @@ async def generate_image_summary(if not isinstance(image_bytes, bytes):raise ValueError(f"Expected bytes for image file, got {type(image_bytes)}")+ # Check for unsupported formats (SVG, etc.) by detecting magic bytes+ # SVG format is not supported by VolcEngine VLM API, skip VLM analysis+ if _is_svg(image_bytes):+ logger.info(+ f"[MediaUtils.generate_image_summary] SVG format detected, skipping VLM analysis: {image_uri}"+ )+ return {"name": file_name, "summary": "SVG image (format not supported by VLM)"}+logger.info(f"[MediaUtils.generate_image_summary] Generating summary for image: {image_uri}")@@ -107,6 +147,13 @@ async def generate_image_summary()return {"name": file_name, "summary": response.strip()}+ except ValueError as e:+ if "SVG format" in str(e) or "not supported" in str(e):+ logger.warning(+ f"[MediaUtils.generate_image_summary] Unsupported image format for {image_uri}: {e}"+ )+ return {"name": file_name, "summary": f"Unsupported image format: {str(e)}"}+ raiseexcept Exception as e:logger.error(f"[MediaUtils.generate_image_summary] Failed to generate image summary: {e}",diff --git a/openviking_cli/utils/config/vlm_config.py b/openviking_cli/utils/config/vlm_config.pyindex 411c7d76c..e3ce672ed 100644--- a/openviking_cli/utils/config/vlm_config.py+++ b/openviking_cli/utils/config/vlm_config.py@@ -21,7 +21,7 @@ class VLMConfig(BaseModel):providers: Dict[str, Dict[str, Any]] = Field(default_factory=dict,- description="Multi-provider configuration, e.g. {'deepseek': {'api_key': 'xxx', 'api_base': 'xxx'}}",+ description="Multi-provider configuration, e.g. {'openai': {'api_key': 'xxx', 'api_base': 'xxx'}}",)default_provider: Optional[str] = Field(default=None, description="Default provider name")@@ -85,55 +85,31 @@ class VLMConfig(BaseModel):return Nonedef _match_provider(self, model: str | None = None) -> tuple[Dict[str, Any] | None, str | None]:- """Match provider config by model name.+ """Match provider config.Returns:(provider_config_dict, provider_name)"""- from openviking.models.vlm.registry import PROVIDERS-- model_lower = (model or self.model or "").lower()-if self.provider:p = self.providers.get(self.provider)if p and p.get("api_key"):return p, self.provider- for spec in PROVIDERS:- p = self.providers.get(spec.name)- if p and any(kw in model_lower for kw in spec.keywords) and p.get("api_key"):- return p, spec.name-- for spec in PROVIDERS:- if spec.is_gateway:- p = self.providers.get(spec.name)- if p and p.get("api_key"):- return p, spec.name-- for spec in PROVIDERS:- if not spec.is_gateway:- p = self.providers.get(spec.name)- if p and p.get("api_key"):- return p, spec.name+ for name, config in self.providers.items():+ if config.get("api_key"):+ return config, namereturn None, Nonedef get_provider_config(self, model: str | None = None- ) -> tuple[Dict[str, Any] | None, str | None, "Any | None"]:- """Get provider config and spec.+ ) -> tuple[Dict[str, Any] | None, str | None]:+ """Get provider config.Returns:- (provider_config_dict, provider_name, ProviderSpec)+ (provider_config_dict, provider_name)"""- from openviking.models.vlm.registry import find_by_name, find_gateway-- config, name = self._match_provider(model)- if config and name:- spec = find_by_name(name)- gateway = find_gateway(name, config.get("api_key"), config.get("api_base"))- return config, name, gateway or spec- return None, None, None+ return self._match_provider(model)def get_vlm_instance(self) -> Any:"""Get VLM instance."""@@ -146,7 +122,7 @@ class VLMConfig(BaseModel):def _build_vlm_config_dict(self) -> Dict[str, Any]:"""Build VLM instance config dict."""- config, name, spec = self.get_provider_config()+ config, name = self.get_provider_config()result = {"model": self.model,@@ -161,9 +137,6 @@ class VLMConfig(BaseModel):result["api_base"] = config.get("api_base")result["extra_headers"] = config.get("extra_headers")- if spec and not result.get("api_base") and spec.default_api_base:- result["api_base"] = spec.default_api_base-return resultdef get_completion(self, prompt: str, thinking: bool = False) -> str: