Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat: 支持基于火山方舟的音视频多模态理解 (#3563)
Haoyu Zhang3f3554256bb6
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.mdindex 80b0e9dd9..19208067b 100644--- a/docs/en/guides/01-configuration.md+++ b/docs/en/guides/01-configuration.md@@ -610,7 +610,14 @@ Vision Language Model for semantic extraction (L0/L1 generation)."api_key": "your-api-key","model": "doubao-seed-2-0-lite-260428","api_base": "https://ark.cn-beijing.volces.com/api/v3",- "max_retries": 3+ "max_retries": 3,+ "media": {+ "enabled": true,+ "max_concurrent": 2,+ "file_processing_timeout": 1800,+ "file_poll_interval": 3,+ "video_fps": 1.0+ }}}```@@ -634,6 +641,12 @@ Vision Language Model for semantic extraction (L0/L1 generation).| `extra_headers` | object | Custom HTTP headers for compatible HTTP providers. `kimi` also accepts header overrides, but already injects the required subscription headers by default || `extra_request_body` | object | Extra JSON body fields for OpenAI-compatible completion requests, useful for provider-specific options such as Ollama `{"think": false}` || `stream` | bool | Enable streaming mode (for OpenAI-compatible providers, default: `false`) |+| `media` | object | Audio/video runtime controls. Media understanding reuses this VLM's provider, model, credentials, client, timeout, retry, headers, output-token limit, failover, and token accounting |+| `media.enabled` | bool | Enable audio/video understanding (default: `false`) |+| `media.max_concurrent` | int | Maximum concurrent audio/video calls (default: `2`) |+| `media.file_processing_timeout` | float | Maximum provider-side preprocessing wait in seconds (default: `1800`) |+| `media.file_poll_interval` | float | Provider-side preprocessing poll interval in seconds (default: `3`) |+| `media.video_fps` | float | Video frame sampling rate when supported by the provider, from `0.2` through `5.0` (default: `1.0`) |`vlm.max_retries` only applies to transient errors such as `429`, `5xx`, timeouts, and connection failures. Permanent authentication, authorization, and billing errors are not retried automatically. The backoff strategy is exponential backoff with jitter, starting at `0.5s` and capped at `8s`.@@ -728,6 +741,52 @@ For OpenAI-compatible providers that return SSE (Server-Sent Events) format resp> **Note**: The OpenAI SDK requires `stream=true` to properly parse SSE responses. When using providers that force SSE format, you must set this option to `true`.+**Audio/video understanding**++Audio and video understanding is an optional capability of the configured VLM. It uses the same provider, model, credentials, client, request timeout, retries, headers, maximum output tokens, failover chain, and token accounting as other VLM calls. Enable it with the nested `vlm.media` controls; there is no separate media model configuration.++```json+{+ "vlm": {+ "provider": "volcengine",+ "api_key": "${VOLCENGINE_API_KEY}",+ "model": "${VOLCENGINE_MODEL}",+ "api_base": "https://ark.cn-beijing.volces.com/api/v3",+ "timeout": 1200,+ "max_retries": 3,+ "max_tokens": 4096,+ "media": {+ "enabled": true,+ "file_processing_timeout": 1800,+ "file_poll_interval": 3,+ "max_concurrent": 2,+ "video_fps": 1.0+ }+ }+}+```++The VLM `model` value is the corresponding Ark model endpoint ID. `video_fps` applies only to video and controls the frame sampling rate sent to Ark.++The recommended starting models for audio and video understanding are `doubao-seed-2-0-lite-260428` and `doubao-seed-2-0-mini-260428`. These are recommended examples, not an exhaustive compatibility list; Ark continues to update its models and input capabilities. See Ark's official [video input capability list](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#ff5ef604) and [audio input capability list](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#9619c0ba) for other supported models. If `model` is an `ep-*` inference endpoint ID, verify that its underlying foundation model supports the corresponding media input. OpenViking does not validate audio or video model capabilities while loading configuration.++**Ingestible and understandable formats**++| Type | Stored by the existing parser | Understood by Ark in this release |+|------|-------------------------------|-----------------------------------|+| Audio | MP3, WAV, OGG, FLAC, AAC, M4A, OPUS, AC3 | MP3, WAV, AAC, M4A |+| Video | MP4, AVI, MOV, MKV, WEBM, FLV, WMV, TS | MP4, AVI, MOV |++Formats outside the understanding column continue to follow the existing parser and storage behavior; OpenViking does not transcode them or send them to the understanding model. When such a file is recognized as an audio or video leaf, an empty media summary is indexed using its filename.++For a supported file, OpenViking uploads the media to the Ark Files API without explicitly setting `expire_at`, so file retention follows Ark's default policy. After processing completes, OpenViking references the file's `file_id` from the Responses API with response storage disabled, then attempts to delete the Ark file under a short cleanup deadline. Remote deletion is best-effort and does not replace an otherwise successful result if cleanup fails; a file whose deletion fails or times out continues to follow Ark's default retention policy. Local temporary files are removed independently even when remote cleanup fails or is cancelled.++- A successful summary for a directory containing exactly one audio or video file becomes that directory's L1 directly, with L0 derived through the existing semantic path. No second generic VLM summarization is performed.+- Media in a mixed directory contributes its summary to the existing generic VLM aggregation.+- Disabled media understanding, an unsupported understanding format, or a final model failure yields an empty media summary. Generic directory L0/L1 generation keeps its existing behavior, while a recognized audio or video leaf uses its filename for the DETAIL vector and BM25 content. Provider errors and media-understanding status text are not written to the media summary or leaf index.++Media processing sends file content to the configured external provider. Disabled response storage and best-effort deletion reduce unintended retention but do not replace the provider's own privacy and retention controls; uploaded files do not receive an explicit expiration time, so their retention period is determined by Ark's default policy. Ark Files storage/processing and Responses model tokens can incur provider charges, so review your provider's privacy, retention, and billing terms before enabling this feature. See the official Volcengine Ark documentation for [audio understanding](https://docs.volcengine.com/docs/82379/2377589?lang=zh) and [video understanding](https://docs.volcengine.com/docs/82379/1895586?lang=zh).+### query_plannerOptional lightweight model for retrieval intent analysis and query planning. It uses the same configuration shape as `vlm`, but only affects `search()` intent analysis and query expansion. If `query_planner` is omitted or empty, OpenViking falls back to `vlm` for backward compatibility.diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.mdindex b16ed5fb4..b23cef2ce 100644--- a/docs/zh/guides/01-configuration.md+++ b/docs/zh/guides/01-configuration.md@@ -579,7 +579,14 @@ provider,并设置 `storage.vectordb.sparse_weight > 0`。自托管模型的"api_key": "your-api-key","model": "doubao-seed-2-0-lite-260428","api_base": "https://ark.cn-beijing.volces.com/api/v3",- "max_retries": 3+ "max_retries": 3,+ "media": {+ "enabled": true,+ "max_concurrent": 2,+ "file_processing_timeout": 1800,+ "file_poll_interval": 3,+ "video_fps": 1.0+ }}}```@@ -603,6 +610,12 @@ provider,并设置 `storage.vectordb.sparse_weight > 0`。自托管模型的| `extra_headers` | object | 兼容 HTTP provider 的自定义请求头。`kimi` 默认已注入所需订阅请求头,也支持在这里覆盖或扩展 || `extra_request_body` | object | 传给 OpenAI 兼容 completion 请求的额外 JSON body 字段,可用于 Ollama `{"think": false}` 等 provider 专有参数 || `stream` | bool | 启用流式模式(OpenAI 兼容 provider 可用,默认:`false`) |+| `media` | object | 音视频运行参数;音视频理解复用该 VLM 的 provider、模型、凭据、client、超时、重试、请求头、输出 token 限制、故障切换和 token 统计 |+| `media.enabled` | bool | 启用音视频理解(默认:`false`) |+| `media.max_concurrent` | int | 音视频调用最大并发数(默认:`2`) |+| `media.file_processing_timeout` | float | Provider 侧媒体预处理最长等待秒数(默认:`1800`) |+| `media.file_poll_interval` | float | Provider 侧媒体预处理轮询间隔秒数(默认:`3`) |+| `media.video_fps` | float | Provider 支持时使用的视频采样帧率,范围 `0.2` 到 `5.0`(默认:`1.0`) |`vlm.max_retries` 仅对瞬时错误生效,例如 `429`、`5xx`、超时和连接错误;认证、鉴权、欠费等永久错误不会自动重试。退避策略为指数退避,初始延迟 `0.5s`,上限 `8s`,并带随机抖动。@@ -696,6 +709,52 @@ LiteLLM 的 Bedrock bearer-token API-key 鉴权,请设置 `forward_api_key=tru> **注意**: OpenAI SDK 需要 `stream=true` 才能正确解析 SSE 响应。使用强制返回 SSE 格式的 provider 时,必须将此选项设置为 `true`。+**音视频理解**++音频和视频理解是当前 VLM 的可选能力,复用相同的 provider、模型、凭据、client、请求超时、重试、请求头、最大输出 token、故障切换链路和 token 统计。通过嵌套的 `vlm.media` 参数启用,不再单独配置媒体模型。++```json+{+ "vlm": {+ "provider": "volcengine",+ "api_key": "${VOLCENGINE_API_KEY}",+ "model": "${VOLCENGINE_MODEL}",+ "api_base": "https://ark.cn-beijing.volces.com/api/v3",+ "timeout": 1200,+ "max_retries": 3,+ "max_tokens": 4096,+ "media": {+ "enabled": true,+ "file_processing_timeout": 1800,+ "file_poll_interval": 3,+ "max_concurrent": 2,+ "video_fps": 1.0+ }+ }+}+```++VLM 的 `model` 填写对应的方舟模型 endpoint ID。`video_fps` 仅用于视频,控制发送给方舟的视频采样帧率。++推荐使用 `doubao-seed-2-0-lite-260428` 或 `doubao-seed-2-0-mini-260428` 作为音视频理解模型。它们是可直接采用的推荐示例,并非完整的支持模型列表;方舟会持续更新模型及其输入能力。视频理解的可选模型请参考方舟官方[视频输入能力列表](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#ff5ef604),音频理解的可选模型请参考方舟官方[音频输入能力列表](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#9619c0ba)。如果 `model` 填写的是 `ep-*` 推理接入点 ID,请确认该接入点背后的基础模型支持对应的媒体输入。OpenViking 不会在配置加载时校验模型的音频或视频能力。++**可接入格式与可理解格式**++| 类型 | 现有 Parser 可接入并保存 | 本版本可由方舟理解 |+|------|--------------------------|--------------------|+| 音频 | MP3、WAV、OGG、FLAC、AAC、M4A、OPUS、AC3 | MP3、WAV、AAC、M4A |+| 视频 | MP4、AVI、MOV、MKV、WEBM、FLV、WMV、TS | MP4、AVI、MOV |++不在“可理解”列中的格式继续沿用现有 Parser 和存储行为;OpenViking 不会对这些文件转码,也不会把它们发送给理解模型。当文件被识别为音频或视频叶子节点时,空媒体摘要会使用文件名入库。++对于支持的文件,OpenViking 将媒体上传到方舟 Files API,且不显式指定 `expire_at`,因此文件保留时间遵循方舟的默认策略。文件处理完成后,OpenViking 通过禁用响应存储的 Responses API 请求引用其 `file_id`,最后在较短的清理超时内尝试删除方舟文件。远端删除属于 best-effort;如果删除失败或超时,不会覆盖已经成功的理解结果,文件将继续遵循方舟的默认保留策略。本地临时文件独立清理,即使远端清理失败或请求被取消也会删除。++- 目录中只有一个音频或视频文件且理解成功时,该摘要直接成为目录 L1,并通过现有语义链路派生 L0,不再调用通用 VLM 做第二次总结。+- 媒体位于混合目录时,其摘要仍参与现有通用 VLM 聚合。+- 音视频理解未启用、理解格式不支持或模型最终失败时,媒体摘要为空;目录 L0/L1 生成保持原有通用行为,被识别为音频或视频的叶子节点则使用文件名作为 DETAIL 向量和 BM25 内容。Provider 错误和媒体理解状态文字不会写入媒体摘要或叶子索引。++媒体处理会把文件内容发送给所配置的外部 provider。禁用响应存储和 best-effort 删除可以降低非预期留存风险,但不能替代 provider 自身的隐私与留存控制;上传文件未显式指定过期时间,其保留周期由方舟的默认策略决定。方舟 Files 的存储/处理以及 Responses 的模型 token 可能产生费用;启用前请确认 provider 的隐私、留存和计费条款。详见火山方舟官方[音频理解文档](https://docs.volcengine.com/docs/82379/2377589?lang=zh)和[视频理解文档](https://docs.volcengine.com/docs/82379/1895586?lang=zh)。+### query_planner可选的轻量模型配置,用于检索前的意图分析和 query 规划/改写。配置结构与 `vlm` 相同,但只影响 `search()` 的意图分析和 query expansion。未配置或配置为空时,OpenViking 会回退到 `vlm`,保持向后兼容。diff --git a/examples/ov.conf.example b/examples/ov.conf.exampleindex efe65caae..301a0f48f 100644--- a/examples/ov.conf.example+++ b/examples/ov.conf.example@@ -113,6 +113,13 @@"max_retries": 2,"provider": "volcengine","thinking": false,+ "media": {+ "enabled": false,+ "file_processing_timeout": 1800,+ "file_poll_interval": 3,+ "max_concurrent": 2,+ "video_fps": 1.0,+ },},"vlm_codex_example": {"_comment": "For Codex OAuth via subscription. Complete sign-in from `openviking-server init`. OV stores tokens in ~/.openviking/codex_auth.json and can bootstrap from an existing Codex CLI auth file if needed.",diff --git a/openviking/models/vlm/backends/volcengine_media.py b/openviking/models/vlm/backends/volcengine_media.pynew file mode 100644index 000000000..40c1e1700--- /dev/null+++ b/openviking/models/vlm/backends/volcengine_media.py@@ -0,0 +1,376 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Ark Files/Responses helpers used by the VolcEngine VLM backend."""++from __future__ import annotations++import asyncio+import logging+import time+from collections.abc import Mapping+from pathlib import Path+from typing import TYPE_CHECKING, Any++from openviking.utils.media_limits import MAX_MEDIA_FILE_BYTES+from openviking.utils.model_retry import retry_async++if TYPE_CHECKING:+ from .volcengine_vlm import VolcEngineVLM++logger = logging.getLogger(__name__)++MEDIA_EXTENSIONS = {+ "audio": frozenset({".mp3", ".wav", ".aac", ".m4a"}),+ "video": frozenset({".mp4", ".avi", ".mov"}),+}+_CLEANUP_TIMEOUT_SECONDS = 5.0+_RETRYABLE_HTTP_STATUSES = frozenset({429, 500, 502, 503, 504})+_PERMANENT_429_MARKERS = (+ "accountoverdue",+ "overdue",+ "quota",+ "insufficient",+ "balance",+ "billing",+ "payment",+ "authentication",+ "unauthorized",+ "forbidden",+ "permission",+ "accessdenied",+ "contentpolicy",+ "content_filter",+ "moderation",+ "safety",+ "unsupported",+ "invalid",+ "badrequest",+ "modelnot",+ "notfound",+)+++class ArkFileProcessingFailedError(RuntimeError):+ """Terminal failure reported explicitly by Ark Files processing."""+++def _load_native_transient_error_types() -> tuple[type[BaseException], ...]:+ error_types: list[type[BaseException]] = []+ try:+ from volcenginesdkarkruntime._exceptions import (+ ArkAPIConnectionError,+ ArkAPITimeoutError,+ )++ error_types.extend((ArkAPIConnectionError, ArkAPITimeoutError))+ except (ImportError, AttributeError):+ pass++ try:+ import httpx++ error_types.extend((httpx.TimeoutException, httpx.ConnectError))+ except (ImportError, AttributeError):+ pass+ return tuple(error_types)+++_NATIVE_TRANSIENT_ERROR_TYPES = _load_native_transient_error_types()+++def _exception_chain(error: BaseException):+ pending = [error]+ seen = set()+ while pending:+ current = pending.pop()+ if id(current) in seen:+ continue+ seen.add(id(current))+ yield current+ cause = getattr(current, "__cause__", None)+ context = getattr(current, "__context__", None)+ if cause is not None:+ pending.append(cause)+ if context is not None:+ pending.append(context)+++def _safe_attribute(error: BaseException, name: str) -> Any:+ try:+ return getattr(error, name, None)+ except BaseException:+ return None+++def _structured_status_code(error: BaseException) -> int | None:+ value = _safe_attribute(error, "status_code")+ if value is None or isinstance(value, bool):+ return None+ try:+ return int(value)+ except (TypeError, ValueError, OverflowError):+ return None+++def _structured_error_text(error: BaseException) -> str:+ values: list[str] = []+ for name in ("code", "type", "message"):+ value = _safe_attribute(error, name)+ if isinstance(value, str):+ values.append(value)+ body = _safe_attribute(error, "body")+ if isinstance(body, Mapping):+ nested = body.get("error")+ mappings = (body, nested) if isinstance(nested, Mapping) else (body,)+ for item in mappings:+ for name in ("code", "type", "message"):+ value = item.get(name)+ if isinstance(value, str):+ values.append(value)+ return " ".join(values).lower().replace(" ", "")+++def _is_retryable_media_error(error: Exception) -> bool:+ try:+ chain = tuple(_exception_chain(error))+ if any(isinstance(item, ArkFileProcessingFailedError) for item in chain):+ return False++ structured_statuses = []+ has_structured_429 = False+ for item in chain:+ status_code = _structured_status_code(item)+ if status_code is None:+ continue+ structured_statuses.append(status_code)+ if status_code not in _RETRYABLE_HTTP_STATUSES:+ return False+ has_structured_429 = has_structured_429 or status_code == 429++ if has_structured_429 and any(+ any(marker in _structured_error_text(item) for marker in _PERMANENT_429_MARKERS)+ for item in chain+ ):+ return False+ if structured_statuses:+ return True++ return any(+ isinstance(+ item,+ (TimeoutError, ConnectionError, *_NATIVE_TRANSIENT_ERROR_TYPES),+ )+ for item in chain+ )+ except BaseException:+ return False+++def _first_structured_value(error: BaseException, name: str) -> str:+ for item in _exception_chain(error):+ value = _safe_attribute(item, name)+ if value is not None and not isinstance(value, (dict, list, tuple, set)):+ try:+ return str(value)[:128]+ except BaseException:+ continue+ return "-"+++def _log_sanitized_exception(stage: str, error: BaseException, level: int) -> None:+ status = next(+ (+ status+ for item in _exception_chain(error)+ if (status := _structured_status_code(item)) is not None+ ),+ None,+ )+ logger.log(+ level,+ "Ark media operation exception: stage=%s error_type=%s status=%s code=%s request_id=%s",+ stage,+ type(error).__name__,+ status if status is not None else "-",+ _first_structured_value(error, "code"),+ _first_structured_value(error, "request_id"),+ )+++def supports_media(*, media_type: str, filename: str, size_bytes: int) -> bool:+ extensions = MEDIA_EXTENSIONS.get(media_type)+ if extensions is None:+ return False+ return Path(filename).suffix.lower() in extensions and 0 <= size_bytes <= MAX_MEDIA_FILE_BYTES+++async def understand_media(+ vlm: VolcEngineVLM,+ *,+ prompt: str,+ media_path: Path,+ filename: str,+ media_type: str,+) -> str:+ size_bytes = media_path.stat().st_size+ if not supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ ):+ raise ValueError(f"Unsupported or oversized {media_type} input: {filename}")++ async def attempt_with_sanitized_logging() -> str:+ try:+ return await _attempt(+ vlm,+ prompt=prompt,+ media_path=media_path,+ media_type=media_type,+ )+ except Exception as error:+ _log_sanitized_exception("request", error, logging.WARNING)+ raise++ return await retry_async(+ attempt_with_sanitized_logging,+ max_retries=vlm.max_retries,+ is_retryable=_is_retryable_media_error,+ logger=None,+ )+++async def _attempt(+ vlm: VolcEngineVLM,+ *,+ prompt: str,+ media_path: Path,+ media_type: str,+) -> str:+ file_id = None+ client = vlm.get_async_client()+ media_options = dict(vlm.config.get("media") or {})+ poll_interval = float(media_options.get("file_poll_interval", 3.0))+ processing_timeout = float(media_options.get("file_processing_timeout", 1800.0))+ video_fps = float(media_options.get("video_fps", 1.0))+ preprocess = {"video": {"fps": video_fps}} if media_type == "video" else None++ try:+ with media_path.open("rb") as upload_file:+ uploaded = await client.files.create(+ file=upload_file,+ purpose="user_data",+ preprocess_configs=preprocess,+ extra_headers=vlm.extra_headers,+ )+ file_id = uploaded.id+ processed = await client.files.wait_for_processing(+ file_id,+ poll_interval=poll_interval,+ max_wait_seconds=processing_timeout,+ )+ if processed.status != "active":+ if processed.status == "failed":+ processing_error = getattr(processed, "error", None)+ message = getattr(processing_error, "message", None) or "Ark file processing failed"+ raise ArkFileProcessingFailedError(message)+ raise RuntimeError("Ark file processing did not become active")++ request = {+ "model": vlm.model,+ "input": [+ {+ "role": "user",+ "content": [+ {+ "type": ("input_audio" if media_type == "audio" else "input_video"),+ "file_id": file_id,+ },+ {"type": "input_text", "text": prompt},+ ],+ }+ ],+ "store": False,+ "extra_headers": vlm.extra_headers,+ }+ if vlm.max_tokens is not None:+ request["max_output_tokens"] = vlm.max_tokens++ started = time.monotonic()+ response = await client.responses.create(**request)+ if getattr(response, "status", None) != "completed":+ raise RuntimeError("Ark media response did not complete")+ text = _extract_response_text(response)+ if not text:+ raise RuntimeError("Ark media response contained no output text")+ _record_usage(vlm, response, time.monotonic() - started)+ return text+ finally:+ if file_id:+ await _delete_remote_file(vlm, file_id)+++async def _delete_remote_file(vlm: VolcEngineVLM, file_id: str) -> None:+ client = vlm.get_async_client()++ async def bounded_delete() -> None:+ await asyncio.wait_for(+ client.files.delete(file_id, extra_headers=vlm.extra_headers),+ timeout=_CLEANUP_TIMEOUT_SECONDS,+ )++ deletion = asyncio.create_task(bounded_delete())+ try:+ try:+ await asyncio.shield(deletion)+ except asyncio.CancelledError:+ try:+ await deletion+ except asyncio.CancelledError:+ pass+ except Exception as cleanup_error:+ _log_sanitized_exception(+ "remote_cleanup",+ cleanup_error,+ logging.WARNING,+ )+ raise+ except asyncio.CancelledError:+ raise+ except Exception as cleanup_error:+ _log_sanitized_exception("remote_cleanup", cleanup_error, logging.WARNING)+++def _extract_response_text(response: Any) -> str:+ parts = []+ for item in getattr(response, "output", []) or []:+ if getattr(item, "type", None) != "message":+ continue+ for content in getattr(item, "content", []) or []:+ if getattr(content, "type", None) == "output_text":+ text = getattr(content, "text", "")+ if text:+ parts.append(text)+ return "\n".join(parts).strip()+++def _record_usage(+ vlm: VolcEngineVLM,+ response: Any,+ duration_seconds: float,+) -> None:+ try:+ usage = getattr(response, "usage", None)+ input_details = getattr(usage, "input_tokens_details", None)+ output_details = getattr(usage, "output_tokens_details", None)+ vlm.update_token_usage(+ model_name=vlm.model,+ provider=vlm.provider,+ prompt_tokens=int(getattr(usage, "input_tokens", 0) or 0),+ completion_tokens=int(getattr(usage, "output_tokens", 0) or 0),+ duration_seconds=duration_seconds,+ prompt_cached_tokens=int(getattr(input_details, "cached_tokens", 0) or 0),+ completion_reasoning_tokens=int(getattr(output_details, "reasoning_tokens", 0) or 0),+ )+ except Exception as error:+ _log_sanitized_exception("usage", error, logging.DEBUG)diff --git a/openviking/models/vlm/backends/volcengine_vlm.py b/openviking/models/vlm/backends/volcengine_vlm.pyindex 0795b02ef..08b4ae3db 100644--- a/openviking/models/vlm/backends/volcengine_vlm.py+++ b/openviking/models/vlm/backends/volcengine_vlm.py@@ -116,6 +116,39 @@ class VolcEngineVLM(OpenAIVLM):max_retries=0,)+ def supports_media(+ self,+ *,+ media_type: str,+ filename: str,+ size_bytes: int,+ ) -> bool:+ from .volcengine_media import supports_media++ return supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ )++ async def get_media_completion_async(+ self,+ *,+ prompt: str,+ media_path: Path,+ filename: str,+ media_type: str,+ ) -> str:+ from .volcengine_media import understand_media++ return await understand_media(+ self,+ prompt=prompt,+ media_path=media_path,+ filename=filename,+ media_type=media_type,+ )+def get_completion(self,prompt: str = "",diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.pyindex 742ac34ad..7d092632b 100644--- a/openviking/models/vlm/base.py+++ b/openviking/models/vlm/base.py@@ -23,6 +23,10 @@ _THINK_TAG_RE = re.compile(r"<think>[\s\S]*?</think>")logger = get_logger(__name__)+class UnsupportedMediaInputError(RuntimeError):+ """Raised when a VLM backend cannot process an audio or video input."""++@dataclassclass ToolCall:"""Single tool call from LLM."""@@ -173,6 +177,31 @@ class VLMBase(ABC):"""pass+ def supports_media(+ self,+ *,+ media_type: str,+ filename: str,+ size_bytes: int,+ ) -> bool:+ """Return whether this backend can process the specified media input."""+ del media_type, filename, size_bytes+ return False++ async def get_media_completion_async(+ self,+ *,+ prompt: str,+ media_path: Path,+ filename: str,+ media_type: str,+ ) -> str:+ """Understand one audio or video file asynchronously."""+ del prompt, media_path, filename, media_type+ raise UnsupportedMediaInputError(+ f"VLM provider {self.provider!r} does not support audio/video inputs"+ )+def _clean_response(self, content: str) -> str:"""Strip reasoning tags (e.g. ``<think>...</think>``) from model output."""return _THINK_TAG_RE.sub("", content).strip()@@ -594,6 +623,74 @@ class FailoverVLM(VLMBase):messages=messages,)+ def supports_media(+ self,+ *,+ media_type: str,+ filename: str,+ size_bytes: int,+ ) -> bool:+ return any(+ vlm.supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ )+ for vlm in (self.primary, self.backup)+ )++ async def get_media_completion_async(+ self,+ *,+ prompt: str,+ media_path: Path,+ filename: str,+ media_type: str,+ ) -> str:+ size_bytes = media_path.stat().st_size if media_path.exists() else 0+ kwargs = {+ "prompt": prompt,+ "media_path": media_path,+ "filename": filename,+ "media_type": media_type,+ }+ primary_supported = self.primary.supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ )+ backup_supported = self.backup.supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ )+ last_error = None++ if primary_supported and (not backup_supported or self._switcher.should_try_primary()):+ try:+ result = await self.primary.get_media_completion_async(**kwargs)+ self._switcher.record_primary_success()+ return result+ except Exception as error:+ _annotate_vlm_error(error, self.primary)+ last_error = error+ if not backup_supported or not self._switcher.record_primary_failure(error):+ raise++ if backup_supported:+ try:+ self._switcher.record_backup_request()+ return await self.backup.get_media_completion_async(**kwargs)+ except Exception as error:+ _annotate_vlm_error(error, self.backup)+ raise++ if last_error is not None:+ raise last_error+ raise UnsupportedMediaInputError(+ f"No configured VLM supports {media_type} input {filename!r}"+ )+@propertydef is_using_backup(self) -> bool:"""Check if currently using the backup VLM instance."""@@ -900,6 +997,80 @@ class MultiCredentialVLM(VLMBase):messages=messages,)+ def supports_media(+ self,+ *,+ media_type: str,+ filename: str,+ size_bytes: int,+ ) -> bool:+ return any(+ vlm.supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ )+ for vlm in self._vlm_instances+ )++ async def get_media_completion_async(+ self,+ *,+ prompt: str,+ media_path: Path,+ filename: str,+ media_type: str,+ ) -> str:+ size_bytes = media_path.stat().st_size if media_path.exists() else 0+ start = self._switcher.maybe_failback()+ errors = []+ attempted = False+ active_credential_failed = False++ for offset in range(self._switcher.n):+ idx = (start + offset) % self._switcher.n+ vlm = self._vlm_instances[idx]+ if not vlm.supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ ):+ continue++ attempted = True+ credential_id = self._credential_ids[idx]+ try:+ result = await vlm.get_media_completion_async(+ prompt=prompt,+ media_path=media_path,+ filename=filename,+ media_type=media_type,+ )+ # Capability routing alone must not change the credential used+ # by text and vision calls.+ if idx == start or active_credential_failed:+ self._switcher.commit_success(idx)+ return result+ except Exception as error:+ _annotate_vlm_error(error, vlm)+ error_class = classify_api_error(error)+ errors.append((credential_id, error_class, error, idx))+ if idx == start:+ active_credential_failed = True+ if self._switcher.is_fail_fast(error_class):+ raise+ self._logger.warning(+ "Credential %s failed media understanding with %s, trying next credential",+ credential_id,+ error_class,+ )++ if attempted:+ raise AllCredentialsFailedError(errors)+ raise UnsupportedMediaInputError(+ f"No configured VLM supports {media_type} input {filename!r}"+ )+@propertydef active_credential_id(self) -> str:"""Get the ID of the currently active credential."""diff --git a/openviking/parse/parsers/media/utils.py b/openviking/parse/parsers/media/utils.pyindex 367b92873..daeb471eb 100644--- a/openviking/parse/parsers/media/utils.py+++ b/openviking/parse/parsers/media/utils.py@@ -3,6 +3,8 @@"""Media-related utilities for OpenViking."""import asyncio+import re+import tempfilefrom pathlib import Pathfrom typing import TYPE_CHECKING, Any, Dict, Optional@@ -10,13 +12,18 @@ from openviking.core.path_variables import CalendarVariableProviderfrom openviking.parse.parsers.constants import TYPESCRIPT_MPEG_TS_EXTENSIONfrom openviking.prompts import render_promptfrom openviking.storage.viking_fs import get_viking_fs+from openviking.utils.media_limits import MAX_MEDIA_FILE_BYTESfrom openviking_cli.utils.config import get_openviking_configfrom openviking_cli.utils.logger import get_loggerif TYPE_CHECKING:from openviking.server.identity import RequestContext-from .constants import AUDIO_EXTENSIONS, IMAGE_EXTENSIONS, VIDEO_EXTENSIONS+from .constants import (+ AUDIO_EXTENSIONS,+ IMAGE_EXTENSIONS,+ VIDEO_EXTENSIONS,+)logger = get_logger(__name__)@@ -24,6 +31,109 @@ MPEG_TS_PACKET_SIZE = 188MPEG_TS_PROBE_PACKETS = 10MPEG_TS_PROBE_BYTES = MPEG_TS_PACKET_SIZE * MPEG_TS_PROBE_PACKETSMPEG_TS_SYNC_BYTE = 0x47+_MEDIA_READ_CHUNK_BYTES = 4 * 1024 * 1024+++class _InvalidMediaContentError(ValueError):+ pass+++class _MediaTooLargeError(ValueError):+ pass+++def _exception_metadata(error: BaseException) -> tuple[str, str, str, str]:+ status = code = request_id = "-"+ pending = [error]+ seen = set()+ while pending:+ current = pending.pop()+ if id(current) in seen:+ continue+ seen.add(id(current))+ for name, current_value in (+ ("status", status),+ ("code", code),+ ("request_id", request_id),+ ):+ if current_value != "-":+ continue+ attribute = "status_code" if name == "status" else name+ try:+ value = getattr(current, attribute, None)+ except BaseException:+ value = None+ if value is not None and not isinstance(value, (dict, list, tuple, set)):+ try:+ rendered = str(value)[:128]+ except BaseException:+ rendered = "-"+ if name == "status":+ status = rendered+ elif name == "code":+ code = rendered+ else:+ request_id = rendered+ cause = getattr(current, "__cause__", None)+ context = getattr(current, "__context__", None)+ if cause is not None:+ pending.append(cause)+ if context is not None:+ pending.append(context)+ return type(error).__name__, status, code, request_id+++_ENGLISH_MEDIA_REFUSAL_RE = re.compile(+ r"^(?:(?:i(?:'m| am)\s+)?sorry[,!.]?(?:\s+but)?\s+)?"+ r"(?:(?:i\s+)?(?:am\s+unable\s+to|unable\s+to|cannot|can't|could\s+not|couldn't|"+ r"failed\s+to)|i'm\s+unable\s+to)\s+"+ r"(?:analy[sz]e|understand|process|transcribe)\s+"+ r"(?:(?:the|this)\s+)?(?:(?:provided|supplied)\s+)?"+ r"(?:audio|video|media|file|content)(?:\s+(?:file|content))?[.!]?$",+ flags=re.IGNORECASE,+)+_ENGLISH_MEDIA_PASSIVE_REFUSAL_RE = re.compile(+ r"^(?:(?:the|this)\s+)?(?:(?:provided|supplied)\s+)?"+ r"(?:audio|video|media|file|content)(?:\s+(?:file|content))?\s+"+ r"(?:cannot|can't|could\s+not|couldn't)\s+be\s+"+ r"(?:analy[sz]ed|understood|processed|transcribed)[.!]?$",+ flags=re.IGNORECASE,+)+_ENGLISH_MEDIA_NOT_UNDERSTOOD_RE = re.compile(+ r"^(?:no|not\s+enough)\s+"+ r"(?:(?:recognizable|understandable|identifiable|usable)\s+)?"+ r"(?:audio|video|media|speech|content|information)"+ r"(?:\s+(?:content|information))?\s+"+ r"(?:(?:could\s+be|was|were)\s+)?"+ r"(?:found|detected|recognized|understood|identified)[.!]?$",+ flags=re.IGNORECASE,+)+_ENGLISH_MEDIA_NO_ACCESS_RE = re.compile(+ r"^(?:(?:i(?:'m| am)\s+)?sorry[,!.]?(?:\s+but)?\s+)?(?:"+ r"i\s+(?:(?:cannot|can't|could\s+not|couldn't)\s+access|"+ r"(?:do\s+not|don't|did\s+not|didn't)\s+have\s+access\s+to)\s+"+ r"(?:(?:the|this)\s+)?(?:(?:provided|supplied)\s+)?"+ r"(?:audio|video|media|file|content)(?:\s+(?:file|content))?"+ r"|(?:(?:the|this)\s+)?(?:(?:provided|supplied)\s+)?"+ r"(?:audio|video|media|file|content)(?:\s+(?:file|content))?\s+"+ r"is\s+not\s+accessible\s+to\s+me)[.!]?$",+ flags=re.IGNORECASE,+)+_CHINESE_MEDIA_REFUSAL_RE = re.compile(+ r"^(?:抱歉[,,。.!!]?\s*)?(?:我\s*)?(?:无法|不能|未能)\s*"+ r"(?:分析|理解|识别|处理|转录)\s*(?:此|该|这个|所提供的|提供的)?\s*"+ r"(?:音频|视频|媒体|文件|内容)(?:文件|内容)?[。.!!]?$"+)+_CHINESE_MEDIA_PASSIVE_REFUSAL_RE = re.compile(+ r"^(?:抱歉[,,。.!!]?\s*)?(?:此|该|这个|所提供的|提供的)?\s*"+ r"(?:音频|视频|媒体|文件|内容)(?:文件|内容)?\s*(?:无法|不能|未能)\s*"+ r"(?:被)?(?:分析|理解|识别|处理|转录)[。.!!]?$"+)+_CHINESE_MEDIA_NOT_UNDERSTOOD_RE = re.compile(+ r"^(?:未|没有)(?:检测|识别|发现)到?"+ r"(?:可识别的|有效的|可理解的)?(?:音频|视频|语音|媒体|内容|信息)"+ r"(?:内容|信息)?[。.!!]?$"+)def _is_svg(data: bytes) -> bool:@@ -40,6 +150,7 @@ def _convert_svg_to_png(svg_data: bytes) -> Optional[bytes]:"""try:import cairosvg+return cairosvg.svg2png(bytestring=svg_data)except ImportError:pass@@ -48,8 +159,9 @@ def _convert_svg_to_png(svg_data: bytes) -> Optional[bytes]:try:from wand.image import Image as WandImage- with WandImage(blob=svg_data, format='svg') as img:- img.format = 'png'++ with WandImage(blob=svg_data, format="svg") as img:+ img.format = "png"return img.make_blob()except ImportError:pass@@ -73,7 +185,12 @@ def is_mpeg_ts(content: bytes) -> bool:)-def get_media_type(source_path: Optional[str], source_format: Optional[str]) -> Optional[str]:+def get_media_type(+ source_path: Optional[str],+ source_format: Optional[str],+ *,+ content: object = None,+) -> Optional[str]:"""Determine media type from source path or format.@@ -84,21 +201,26 @@ def get_media_type(source_path: Optional[str], source_format: Optional[str]) ->Returns:Media type ("image", "audio", "video") or None if not a media file"""+ ext = Path(source_path).suffix.lower() if source_path else ""+ if ext == TYPESCRIPT_MPEG_TS_EXTENSION:+ if content is not None:+ return "video" if isinstance(content, bytes) and is_mpeg_ts(content) else None+ if source_format == "video":+ return "video"+ path = Path(source_path)+ if not path.is_file():+ return None+ return "video" if is_mpeg_ts(read_mpeg_ts_probe(path)) else None+if source_format:if source_format in ["image", "audio", "video"]:return source_formatif source_path:- ext = Path(source_path).suffix.lower()if ext in IMAGE_EXTENSIONS:return "image"elif ext in AUDIO_EXTENSIONS:return "audio"- elif ext == TYPESCRIPT_MPEG_TS_EXTENSION:- path = Path(source_path)- if not path.is_file():- return None- return "video" if is_mpeg_ts(read_mpeg_ts_probe(path)) else Noneelif ext in VIDEO_EXTENSIONS:return "video"@@ -202,8 +324,7 @@ async def generate_audio_summary(llm_sem: Optional[asyncio.Semaphore] = None,ctx: Optional["RequestContext"] = None,) -> Dict[str, Any]:- """- Generate summary for an audio file (placeholder).+ """Generate a normalized semantic summary for an audio file.Args:audio_uri: URI to the audio file in VikingFS@@ -214,10 +335,13 @@ async def generate_audio_summary(Returns:Dictionary with "name" and "summary" keys"""- logger.info(- f"[MediaUtils.generate_audio_summary] Audio summary generation not yet implemented for: {audio_uri}"+ return await _generate_media_summary(+ audio_uri,+ original_filename,+ "audio",+ llm_sem=llm_sem,+ ctx=ctx,)- return {"name": original_filename, "summary": "Audio summary generation not yet implemented"}async def generate_video_summary(@@ -226,8 +350,7 @@ async def generate_video_summary(llm_sem: Optional[asyncio.Semaphore] = None,ctx: Optional["RequestContext"] = None,) -> Dict[str, Any]:- """- Generate summary for a video file (placeholder).+ """Generate a normalized semantic summary for a video file.Args:video_uri: URI to the video file in VikingFS@@ -238,7 +361,259 @@ async def generate_video_summary(Returns:Dictionary with "name" and "summary" keys"""- logger.info(- f"[MediaUtils.generate_video_summary] Video summary generation not yet implemented for: {video_uri}"+ return await _generate_media_summary(+ video_uri,+ original_filename,+ "video",+ llm_sem=llm_sem,+ ctx=ctx,+ )+++async def _write_media_to_path(+ path: Path,+ media_uri: str,+ viking_fs: Any,+ ctx: Optional["RequestContext"],+ expected_size: int,+) -> None:+ offset = 0+ with path.open("wb") as output:+ while True:+ chunk = await viking_fs.read(+ media_uri,+ offset=offset,+ size=_MEDIA_READ_CHUNK_BYTES,+ ctx=ctx,+ )+ if not isinstance(chunk, bytes):+ raise _InvalidMediaContentError("Media content is non-binary")+ if not chunk:+ break+ offset += len(chunk)+ if offset > MAX_MEDIA_FILE_BYTES:+ raise _MediaTooLargeError("Media content exceeds the hard size limit")+ if expected_size > 0 and offset > expected_size:+ raise _MediaTooLargeError("Media content exceeds the size reported by storage")+ output.write(chunk)+ if offset == 0:+ raise _InvalidMediaContentError("Media content is empty")+++async def _generate_media_summary(+ media_uri: str,+ original_filename: str,+ media_type: str,+ llm_sem: Optional[asyncio.Semaphore] = None,+ ctx: Optional["RequestContext"] = None,+) -> Dict[str, Any]:+ config = get_openviking_config()+ vlm = config.vlm+ result = {"name": original_filename, "summary": ""}++ viking_fs = get_viking_fs()+ stat = await viking_fs.stat(media_uri, ctx=ctx)+ size_bytes = int((stat or {}).get("size", 0))+ if not vlm.supports_media(+ media_type=media_type,+ filename=original_filename,+ size_bytes=size_bytes,+ ):+ logger.info(+ "Skipping unsupported or disabled %s understanding input: %s",+ media_type,+ original_filename,+ )+ return result++ from openviking.session.memory.utils.language import resolve_output_language++ fallback_language = resolve_output_language("", config=config)+ prompt = render_prompt(+ f"parsing.{media_type}_summary",+ {"filename": original_filename, "fallback_language": fallback_language},+ )++ temporary_path: Optional[Path] = None+ try:+ with tempfile.NamedTemporaryFile(+ suffix=Path(original_filename).suffix,+ delete=False,+ ) as temporary:+ temporary_path = Path(temporary.name)++ async def prepare_media() -> None:+ await _write_media_to_path(+ temporary_path,+ media_uri,+ viking_fs,+ ctx,+ expected_size=size_bytes,+ )++ async with llm_sem or asyncio.Semaphore(1):+ raw = await vlm.get_media_completion_async(+ prompt=prompt,+ media_path=temporary_path,+ filename=original_filename,+ media_type=media_type,+ prepare_media=prepare_media,+ )+ except (_InvalidMediaContentError, _MediaTooLargeError):+ return result+ except Exception as error:+ error_type, status, code, request_id = _exception_metadata(error)+ model = str(vlm.model).replace("\r", " ").replace("\n", " ")[:128]+ logger.warning(+ "Media understanding failed: stage=provider_call model=%s "+ "media_type=%s error_type=%s status=%s code=%s request_id=%s. "+ "Verify that the model supports this media type and that provider "+ "access/configuration is valid.",+ model,+ media_type,+ error_type,+ status,+ code,+ request_id,+ )+ return result+ finally:+ if temporary_path is not None:+ try:+ temporary_path.unlink(missing_ok=True)+ except OSError as cleanup_error:+ error_type, status, code, request_id = _exception_metadata(cleanup_error)+ logger.warning(+ "Media understanding cleanup failed: stage=local_cleanup "+ "media_type=%s error_type=%s status=%s code=%s request_id=%s",+ media_type,+ error_type,+ status,+ code,+ request_id,+ )++ result["summary"] = _normalize_media_markdown(+ raw,+ filename=original_filename,+ overview_max_chars=config.semantic.overview_max_chars,+ abstract_max_chars=config.semantic.abstract_max_chars,+ )+ return result+++def _normalize_media_markdown(+ raw: str,+ *,+ filename: str,+ overview_max_chars: int,+ abstract_max_chars: int,+) -> str:+ text = re.sub(r"<think>[\s\S]*?</think>", "", str(raw or ""), flags=re.IGNORECASE).strip()+ fenced = re.fullmatch(+ r"```(?:markdown)?\s*\n?([\s\S]*?)\n?```",+ text,+ flags=re.IGNORECASE,+ )+ if fenced:+ text = fenced.group(1).strip()+ if re.search(r"```|~~~", text):+ return ""+ if not text or overview_max_chars <= 0 or abstract_max_chars <= 0:+ return ""+ if _is_short_media_refusal(text):+ return ""++ title_match = re.search(r"^#\s+(.+?)\s*$", text, flags=re.MULTILINE)+ raw_title = title_match.group(1) if title_match else (Path(filename).stem or filename)+ raw_title = re.sub(r"\s+", " ", raw_title).strip()+ if not raw_title:+ return ""++ without_title = text+ if title_match:+ without_title = text[: title_match.start()] + text[title_match.end() :]+ without_heading = re.sub(+ rf"^###\s+{re.escape(filename)}\s*$",+ "",+ without_title,+ flags=re.MULTILINE,+ )+ blocks = [block.strip() for block in re.split(r"\n\s*\n", without_heading) if block.strip()]+ brief_index = next(+ (index for index, block in enumerate(blocks) if _is_prose_block(block)),+ None,+ )+ if brief_index is None:+ return ""++ raw_brief = re.sub(r"\s+", " ", blocks[brief_index]).strip()+ if not raw_brief:+ return ""+ if _is_short_media_refusal(raw_brief):+ return ""+ body = "\n\n".join(block for index, block in enumerate(blocks) if index != brief_index).strip()++ fixed_without_title = f"# \n\nx\n\n### {filename}"+ title_budget = overview_max_chars - len(fixed_without_title)+ if title_budget <= 0:+ return ""+ title = raw_title[: min(200, title_budget)].rstrip()+ fixed_without_briefs = f"# {title}\n\n\n\n### {filename}\n\n"+ brief_budget = (overview_max_chars - len(fixed_without_briefs)) // 2+ if brief_budget <= 0:+ return ""+ brief = raw_brief[: min(abstract_max_chars, brief_budget)].rstrip()+ if not brief:+ return ""++ prefix = f"# {title}\n\n{brief}\n\n### {filename}"+ recoverable_body = brief+ if _has_non_heading_content(body):+ recoverable_body = f"{brief}\n\n{body}"+ body_budget = overview_max_chars - len(prefix) - 2+ if body_budget <= 0:+ return ""+ recoverable_body = recoverable_body[:body_budget].rstrip()+ if not recoverable_body:+ return ""+ return f"{prefix}\n\n{recoverable_body}"+++def _is_short_media_refusal(text: str) -> bool:+ candidate = text.strip().replace("’", "'")+ candidate = re.sub(r"^#{1,6}\s+[^\n]+\n+", "", candidate, count=1).strip()+ candidate = re.sub(r"\s+", " ", candidate)+ if not candidate or len(candidate) > 240:+ return False+ return any(+ pattern.fullmatch(candidate)+ for pattern in (+ _ENGLISH_MEDIA_REFUSAL_RE,+ _ENGLISH_MEDIA_PASSIVE_REFUSAL_RE,+ _ENGLISH_MEDIA_NOT_UNDERSTOOD_RE,+ _ENGLISH_MEDIA_NO_ACCESS_RE,+ _CHINESE_MEDIA_REFUSAL_RE,+ _CHINESE_MEDIA_PASSIVE_REFUSAL_RE,+ _CHINESE_MEDIA_NOT_UNDERSTOOD_RE,+ ))- return {"name": original_filename, "summary": "Video summary generation not yet implemented"}+++def _has_non_heading_content(text: str) -> bool:+ return any(line.strip() and not line.lstrip().startswith("#") for line in text.splitlines())+++def _is_prose_block(block: str) -> bool:+ candidate = block.lstrip()+ if not candidate or not any(character.isalpha() for character in candidate):+ return False+ if re.match(r"^(?:#{1,6}\s|[-*+]\s|>\s?|\d+[.)]\s|`{3,}|~{3,}|\|)", candidate):+ return False++ lines = candidate.splitlines()+ if re.fullmatch(r"\s*(?:-{3,}|\*{3,}|_{3,})\s*", lines[0]):+ return False+ if len(lines) > 1 and "|" in lines[0] and re.match(r"^\s*\|?\s*:?-{3,}:?\s*\|", lines[1]):+ return False+ return Truediff --git a/openviking/prompts/templates/parsing/audio_summary.yaml b/openviking/prompts/templates/parsing/audio_summary.yamlnew file mode 100644index 000000000..519f62dac--- /dev/null+++ b/openviking/prompts/templates/parsing/audio_summary.yaml@@ -0,0 +1,34 @@+metadata:+ id: "parsing.audio_summary"+ name: "Audio Summary"+ description: "Generate a high-information audio overview for semantic indexing"+ version: "1.0.0"+ language: "en"+ category: "parsing"+variables:+ - name: "filename"+ type: "string"+ description: "Original audio filename"+ required: true+ - name: "fallback_language"+ type: "string"+ description: "Output language when the media has no identifiable language"+ required: true+template: |+ Analyze the supplied audio faithfully for semantic retrieval.+ Write in the audio's primary language; if it has no identifiable language, use {{ fallback_language }}.+ Return Markdown only, with no code fence.++ Required structure:+ # A specific content title++ A dense overview paragraph immediately after H1.++ ### {{ filename }}++ Then retain as much useful detail as fits: core topics, conclusions, named entities,+ important facts and numbers, speaker/language characteristics, short recognizable quotes,+ music, sound events, and environment. Use coarse time points only when recognizable.+ Do not produce a full transcript. Do not invent identity, wording, facts, or timestamps.+llm_config:+ temperature: 0.0diff --git a/openviking/prompts/templates/parsing/video_summary.yaml b/openviking/prompts/templates/parsing/video_summary.yamlnew file mode 100644index 000000000..5ef9494dd--- /dev/null+++ b/openviking/prompts/templates/parsing/video_summary.yaml@@ -0,0 +1,34 @@+metadata:+ id: "parsing.video_summary"+ name: "Video Summary"+ description: "Generate a high-information video overview for semantic indexing"+ version: "1.0.0"+ language: "en"+ category: "parsing"+variables:+ - name: "filename"+ type: "string"+ description: "Original video filename"+ required: true+ - name: "fallback_language"+ type: "string"+ description: "Output language when the media has no identifiable language"+ required: true+template: |+ Analyze the supplied video faithfully for semantic retrieval, using both visuals and audio.+ Write in the video's primary language; if it has no identifiable language, use {{ fallback_language }}.+ Return Markdown only, with no code fence.++ Required structure:+ # A specific content title++ A dense overview paragraph immediately after H1.++ ### {{ filename }}++ Then retain as much useful detail as fits: people, objects, scenes, actions, key events,+ conclusions, facts and numbers, short recognizable dialogue, on-screen text, music and sound.+ Include coarse time ranges only when recognizable. Do not create a full transcript or timeline.+ Do not invent identity, wording, facts, text, or timestamps.+llm_config:+ temperature: 0.0diff --git a/openviking/storage/queuefs/semantic_dag.py b/openviking/storage/queuefs/semantic_dag.pyindex d2590e87d..9b30b3e7b 100644--- a/openviking/storage/queuefs/semantic_dag.py+++ b/openviking/storage/queuefs/semantic_dag.py@@ -3,12 +3,14 @@"""Semantic DAG executor with event-driven lazy dispatch."""import asyncio+import reimport threadingfrom dataclasses import dataclass, fieldfrom typing import Any, ClassVar, Dict, List, Optional, Setfrom uuid import uuid4from weakref import WeakKeyDictionary+from openviking.parse.parsers.media import get_media_typefrom openviking.server.identity import RequestContextfrom openviking.service.task_work_index import bind_task_context, get_task_contextfrom openviking.storage.queuefs.semantic_sidecar import write_semantic_sidecars@@ -818,6 +820,39 @@ class SemanticDagExecutor:summaries.append(item)return summaries+ def _select_direct_media_overview(+ self,+ node: DirNode,+ file_summaries: List[Dict[str, str]],+ ) -> Optional[str]:+ if len(node.file_paths) != 1 or node.children_dirs or len(file_summaries) != 1:+ return None++ file_path = node.file_paths[0]+ filename = file_path.rsplit("/", 1)[-1]+ if get_media_type(file_path, None) not in {"audio", "video"}:+ return None++ summary = str(file_summaries[0].get("summary") or "").strip()+ if not summary.startswith("# "):+ return None++ lines = summary.splitlines()+ brief_start = next((idx for idx in range(1, len(lines)) if lines[idx].strip()), None)+ if brief_start is None or lines[brief_start].lstrip().startswith("#"):+ return None++ brief_end = brief_start+ while brief_end < len(lines) and lines[brief_end].strip():+ if lines[brief_end].lstrip().startswith("#"):+ return None+ brief_end += 1++ filename_heading = re.compile(rf"^###\s+{re.escape(filename)}\s*$", re.MULTILINE)+ if not any(filename_heading.fullmatch(line) for line in lines[brief_end:]):+ return None+ return summary+@propertydef stale(self) -> bool:return self._stale@@ -883,10 +918,12 @@ class SemanticDagExecutor:async with node.lock:file_summaries = self._finalize_file_summaries(node)children_abstracts = await self._finalize_children_abstracts(node)- async with self._llm_sem:- overview = await self._processor._generate_overview(- dir_uri, file_summaries, children_abstracts- )+ overview = self._select_direct_media_overview(node, file_summaries)+ if overview is None:+ async with self._llm_sem:+ overview = await self._processor._generate_overview(+ dir_uri, file_summaries, children_abstracts+ )overview, abstract = self._processor._normalize_overview_generation(overview)if self._closed:diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.pyindex 6578984e0..b9b93257b 100644--- a/openviking/storage/queuefs/semantic_processor.py+++ b/openviking/storage/queuefs/semantic_processor.py@@ -25,13 +25,13 @@ from openviking.parse.parsers.constants import (FILE_TYPE_OTHER,)from openviking.parse.parsers.media.utils import (+ MPEG_TS_PROBE_BYTES,generate_audio_summary,generate_image_summary,generate_video_summary,get_media_type,)from openviking.prompts import render_prompt-from openviking.utils.ingest_options import IngestOptionsfrom openviking.server.identity import RequestContext, Rolefrom openviking.storage.errors import LockAcquisitionErrorfrom openviking.storage.queuefs.named_queue import DequeueHandlerBase@@ -49,6 +49,7 @@ from openviking.utils.circuit_breaker import (CircuitBreakerOpen,classify_api_error,)+from openviking.utils.ingest_options import IngestOptionsfrom openviking.utils.model_retry import ERROR_CLASS_INPUT_TOO_LARGE, ERROR_CLASS_PERMANENTfrom openviking_cli.session.user_id import UserIdentifierfrom openviking_cli.utils import VikingURI@@ -1158,6 +1159,17 @@ class SemanticProcessor(DequeueHandlerBase):file_name = file_path.split("/")[-1]llm_sem = llm_sem or asyncio.Semaphore(self.max_concurrent_llm)media_type = get_media_type(file_name, None)+ if file_name.lower().endswith(".ts"):+ try:+ prefix = await get_viking_fs().read(+ file_path,+ offset=0,+ size=MPEG_TS_PROBE_BYTES,+ ctx=ctx,+ )+ except Exception:+ prefix = None+ media_type = get_media_type(file_name, None, content=prefix)if media_type == "image":return await generate_image_summary(file_path, file_name, llm_sem, ctx=ctx)elif media_type == "audio":diff --git a/openviking/utils/embedding_utils.py b/openviking/utils/embedding_utils.pyindex 978cd75f3..923b79a26 100644--- a/openviking/utils/embedding_utils.py+++ b/openviking/utils/embedding_utils.py@@ -8,6 +8,7 @@ Common logic for creating Context objects and enqueuing them to EmbeddingQueue.import osfrom datetime import datetime, timezone+from pathlib import Pathfrom typing import Any, Dict, Optionalfrom charset_normalizer import from_bytes@@ -18,6 +19,10 @@ from openviking.core.namespace import (is_session_uri,owner_space_for_uri,)+from openviking.parse.parsers.media.utils import (+ MPEG_TS_PROBE_BYTES,+ is_mpeg_ts,+)from openviking.server.identity import RequestContextfrom openviking.service.task_work_index import TaskWorkRejectedfrom openviking.storage.queuefs import get_queue_manager@@ -257,6 +262,29 @@ async def _build_image_data_uri(return None+async def _resolve_resource_content_type(+ file_path: str,+ file_name: str,+ viking_fs: Any,+ ctx: Optional[RequestContext],+) -> Optional[ResourceContentType]:+ content_type = get_resource_content_type(file_name)+ if Path(file_name).suffix.lower() != ".ts":+ return content_type+ try:+ prefix = await viking_fs.read(+ file_path,+ offset=0,+ size=MPEG_TS_PROBE_BYTES,+ ctx=ctx,+ )+ except Exception:+ return content_type+ if is_mpeg_ts(prefix):+ return ResourceContentType.VIDEO+ return content_type++def _coerce_text_file_content(raw: Any) -> str:"""Coerce known text-file content returned by VikingFS into str."""if isinstance(raw, bytes):@@ -507,12 +535,18 @@ async def vectorize_file(owner_space=owner_space_for_uri(file_path, ctx),)- content_type = get_resource_content_type(file_name)+ content_type = await _resolve_resource_content_type(+ file_path, file_name, viking_fs, ctx+ )embedding_cfg = get_openviking_config().embeddingconfigured_text_source = getattr(embedding_cfg, "text_source", "content_only")effective_text_source = "summary_only" if use_summary else configured_text_source- if content_type is None:+ if content_type in (ResourceContentType.AUDIO, ResourceContentType.VIDEO):+ effective_text = summary or file_name+ context.abstract = effective_text+ context.set_vectorize(Vectorize(text=effective_text, full_text=effective_text))+ elif content_type is None:# Unsupported file type: fall back to summary if availableif summary:logger.warning(diff --git a/openviking/utils/media_limits.py b/openviking/utils/media_limits.pynew file mode 100644index 000000000..d96038730--- /dev/null+++ b/openviking/utils/media_limits.py@@ -0,0 +1,5 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""Shared resource limits for audio and video processing."""++MAX_MEDIA_FILE_BYTES = 512 * 1024 * 1024diff --git a/openviking_cli/utils/config/__init__.py b/openviking_cli/utils/config/__init__.pyindex 6d879438e..b84173610 100644--- a/openviking_cli/utils/config/__init__.py+++ b/openviking_cli/utils/config/__init__.py@@ -51,8 +51,8 @@ from .consts import (SYSTEM_CONFIG_DIR,)from .embedding_config import EmbeddingConfig-from .grep_config import GrepConfig, GrepEnginefrom .git_config import GitConfig, GitLocalConfig, GitS3Config+from .grep_config import GrepConfig, GrepEnginefrom .log_config import LogConfigfrom .open_viking_config import (OpenVikingConfig,diff --git a/openviking_cli/utils/config/vlm_config.py b/openviking_cli/utils/config/vlm_config.pyindex b1b0d6aa4..75914888c 100644--- a/openviking_cli/utils/config/vlm_config.py+++ b/openviking_cli/utils/config/vlm_config.py@@ -1,9 +1,13 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+import asyncioimport importlib-from typing import Any, Dict, List, Optional, Union+import threading+import weakref+from pathlib import Path+from typing import Any, Awaitable, Callable, Dict, List, Optional, Union-from pydantic import BaseModel, Field, model_validator+from pydantic import BaseModel, Field, PrivateAttr, model_validatordef _load_codex_auth_module():@@ -54,6 +58,35 @@ class VLMCredential(BaseModel):model_config = {"extra": "forbid"}+class VLMMediaConfig(BaseModel):+ """Runtime controls for audio and video inputs handled by the configured VLM."""++ enabled: bool = Field(default=False, description="Enable VLM audio/video understanding")+ max_concurrent: int = Field(+ default=2,+ ge=1,+ description="Maximum concurrent audio/video VLM calls",+ )+ file_processing_timeout: float = Field(+ default=1800.0,+ gt=0.0,+ description="Maximum provider-side media preprocessing wait in seconds",+ )+ file_poll_interval: float = Field(+ default=3.0,+ gt=0.0,+ description="Provider-side media preprocessing poll interval in seconds",+ )+ video_fps: float = Field(+ default=1.0,+ ge=0.2,+ le=5.0,+ description="Video frame sampling rate for providers that support preprocessing",+ )++ model_config = {"extra": "forbid"}++class VLMConfig(BaseModel):"""VLM configuration, supports multiple provider backends and multi-credential failover."""@@ -104,6 +137,11 @@ class VLMConfig(BaseModel):default=64, description="Maximum number of concurrent LLM calls for semantic processing")+ media: VLMMediaConfig = Field(+ default_factory=VLMMediaConfig,+ description="Audio/video runtime controls for the configured VLM",+ )+api_version: Optional[str] = Field(default=None,description="API version for Azure OpenAI (e.g., '2025-01-01-preview').",@@ -139,6 +177,11 @@ class VLMConfig(BaseModel):)_vlm_instance: Optional[Any] = None+ _media_semaphores: weakref.WeakKeyDictionary[+ asyncio.AbstractEventLoop,+ weakref.ReferenceType[asyncio.Semaphore],+ ] = PrivateAttr(default_factory=weakref.WeakKeyDictionary)+ _media_semaphore_lock: threading.Lock = PrivateAttr(default_factory=threading.Lock)model_config = {"arbitrary_types_allowed": True, "extra": "forbid"}@@ -564,6 +607,7 @@ class VLMConfig(BaseModel):),"stream": credential.stream if credential.stream is not None else self.stream,"api_version": credential.api_version,+ "media": self.media.model_dump(),}if credential.api_key:@@ -598,6 +642,7 @@ class VLMConfig(BaseModel):"max_tokens": self.max_tokens,"stream": stream,"api_version": self.api_version,+ "media": self.media.model_dump(),}if config:@@ -676,6 +721,59 @@ class VLMConfig(BaseModel):messages=messages,)+ def _get_media_semaphore(self) -> asyncio.Semaphore:+ loop = asyncio.get_running_loop()+ with self._media_semaphore_lock:+ semaphore_ref = self._media_semaphores.get(loop)+ semaphore = semaphore_ref() if semaphore_ref is not None else None+ if semaphore is None:+ semaphore = asyncio.Semaphore(self.media.max_concurrent)+ self._media_semaphores[loop] = weakref.ref(semaphore)+ return semaphore++ def supports_media(+ self,+ *,+ media_type: str,+ filename: str,+ size_bytes: int,+ ) -> bool:+ """Return whether enabled VLM media processing supports this input."""+ if not self.media.enabled or not self.is_available():+ return False+ return bool(+ self.get_vlm_instance().supports_media(+ media_type=media_type,+ filename=filename,+ size_bytes=size_bytes,+ )+ )++ async def get_media_completion_async(+ self,+ *,+ prompt: str,+ media_path: Path,+ filename: str,+ media_type: str,+ prepare_media: Optional[Callable[[], Awaitable[None]]] = None,+ ) -> str:+ """Stage and understand audio/video under the configured media limit."""+ from openviking.models.vlm.base import UnsupportedMediaInputError++ if not self.media.enabled:+ raise UnsupportedMediaInputError("VLM media understanding is disabled")++ async with self._get_media_semaphore():+ if prepare_media is not None:+ await prepare_media()+ return await self.get_vlm_instance().get_media_completion_async(+ prompt=prompt,+ media_path=media_path,+ filename=filename,+ media_type=media_type,+ )+async def get_vision_completion_async(self,prompt: str = "",diff --git a/tests/models/vlm/test_volcengine_cache.py b/tests/models/vlm/test_volcengine_cache.pydeleted file mode 100644index 2b811c545..000000000--- a/tests/models/vlm/test_volcengine_cache.py+++ /dev/null@@ -1,262 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0-"""Tests for VolcEngineVLM cache logic."""--from unittest.mock import AsyncMock, MagicMock--import pytest--from openviking.models.vlm.backends.volcengine_vlm import VolcEngineVLM as VLMClass---def make_message(role: str, content: str, cache_control: bool = False) -> dict:- """Helper to create a message dict."""- msg = {"role": role, "content": content}- if cache_control:- msg["cache_control"] = {"type": "ephemeral"}- return msg---class TestGetOrCreateFromSegments:- """Tests for _get_or_create_from_segments method."""-- def _create_vlm_with_mock_cache(self):- """Create a VLM instance with mocked dependencies."""- vlm = VLMClass(- model="test-model",- api_key="test-key",- api_base="https://ark.cn-beijing.volces.com/api/v3",- )- # Mock the cache- vlm._response_cache = MagicMock()- vlm.get_async_client = MagicMock()- return vlm-- def test_single_segment_with_cache_hit(self):- """Test: Single segment, cache exists."""- vlm = self._create_vlm_with_mock_cache()-- # 只有一个 segment [msg0, msg1(cache_control)]- segments = [- [- make_message("system", "You are a helpful assistant"),- make_message("user", "Hello", cache_control=True),- ]- ]-- # Mock cache hit- vlm._response_cache.get.return_value = "resp_123"-- result = VLMClass._get_or_create_from_segments(vlm, segments, 1)-- assert result == "resp_123"- vlm._response_cache.get.assert_called_once()- vlm.get_async_client.assert_not_called()-- @pytest.mark.asyncio- async def test_single_segment_cache_miss_create_new(self):- """Test: Single segment, cache miss, create new cache."""- vlm = self._create_vlm_with_mock_cache()-- segments = [- [- make_message("system", "You are a helpful assistant"),- make_message("user", "Hello", cache_control=True),- ]- ]-- # Mock cache miss- vlm._response_cache.get.return_value = None-- # Mock API response- mock_response = MagicMock()- mock_response.id = "resp_new_123"- mock_client = AsyncMock()- mock_client.responses.create = AsyncMock(return_value=mock_response)- vlm.get_async_client.return_value = mock_client-- result = await vlm._get_or_create_from_segments(segments, 1)-- assert result == "resp_new_123"- vlm._response_cache.get.assert_called_once()- vlm._response_cache.set.assert_called_once()-- @pytest.mark.asyncio- async def test_two_segments_both_cached(self):- """Test: Two segments, both have cache."""- vlm = self._create_vlm_with_mock_cache()-- # segments = [[msg0, msg1(cc)], [msg2, msg3(cc)]]- segments = [- [- make_message("system", "You are a helpful assistant"),- make_message("user", "Hello", cache_control=True),- ],- [make_message("user", "How are you?", cache_control=True)],- ]-- # Mock cache hits for both segments- def cache_get(key):- if "seg0" in key:- return "resp_seg0"- if "seg1" in key:- return "resp_seg1"- return None-- vlm._response_cache.get.side_effect = cache_get-- # Should use seg0 cache with previous_response_id to create seg1- mock_response = MagicMock()- mock_response.id = "resp_combined"- mock_client = AsyncMock()- mock_client.responses.create = AsyncMock(return_value=mock_response)- vlm.get_async_client.return_value = mock_client-- result = await vlm._get_or_create_from_segments(segments, 2)-- # Should return combined response id- assert result == "resp_combined"-- @pytest.mark.asyncio- async def test_two_segments_first_not_cached(self):- """Test: Two segments, first not cached, second cached."""- vlm = self._create_vlm_with_mock_cache()-- segments = [- [make_message("system", "System"), make_message("user", "Hello", cache_control=True)],- [make_message("user", "How are you?", cache_control=True)],- ]-- # First segment not cached, second is cached- cache_returns = {- "prefix:seg0_system_hello": None, # First segment - not cached- "prefix:seg1_how_are_you": "resp_seg1", # Second segment - cached- }-- def cache_get(key):- return cache_returns.get(key)-- vlm._response_cache.get.side_effect = cache_get-- # Mock API: first call creates first segment cache, second call extends with previous_response_id- call_count = 0- mock_responses = ["resp_seg0", "resp_combined"]-- async def mock_create(**kwargs):- nonlocal call_count- resp = MagicMock()- resp.id = mock_responses[call_count]- call_count += 1- return resp-- mock_client = AsyncMock()- mock_client.responses.create = mock_create- vlm.get_async_client.return_value = mock_client-- result = await vlm._get_or_create_from_segments(segments, 2)-- # Should create first segment, then extend with second- assert result == "resp_combined"-- @pytest.mark.asyncio- async def test_two_segments_neither_cached(self):- """Test: Two segments, neither cached."""- vlm = self._create_vlm_with_mock_cache()-- segments = [- [make_message("system", "System"), make_message("user", "Hello", cache_control=True)],- [make_message("user", "How are you?", cache_control=True)],- ]-- # Neither segment cached- vlm._response_cache.get.return_value = None-- # Mock API responses- call_count = 0- mock_responses = ["resp_seg0", "resp_combined"]-- async def mock_create(**kwargs):- nonlocal call_count- resp = MagicMock()- resp.id = mock_responses[call_count]- call_count += 1- return resp-- mock_client = AsyncMock()- mock_client.responses.create = mock_create- vlm.get_async_client.return_value = mock_client-- result = await vlm._get_or_create_from_segments(segments, 2)-- # Should create both segments- assert result == "resp_combined"-- @pytest.mark.asyncio- async def test_three_segments_with_middle_cached(self):- """Test: Three segments, middle one cached, others not."""- vlm = self._create_vlm_with_mock_cache()-- segments = [- [make_message("system", "System"), make_message("user", "Hello", cache_control=True)],- [make_message("user", "How are you?", cache_control=True)],- [make_message("user", "Tell me a story", cache_control=True)],- ]-- # Only middle segment cached- cache_returns = {- "prefix:seg0_system_hello": None,- "prefix:seg1_how_are_you": "resp_seg1", # Cached- "prefix:seg2_tell_story": None,- }-- def cache_get(key):- return cache_returns.get(key)-- vlm._response_cache.get.side_effect = cache_get-- # Mock API: create seg0, extend to seg1, extend to seg2- call_count = 0- mock_responses = ["resp_seg0", "resp_01", "resp_012"]-- async def mock_create(**kwargs):- nonlocal call_count- resp = MagicMock()- resp.id = mock_responses[call_count]- call_count += 1- return resp-- mock_client = AsyncMock()- mock_client.responses.create = mock_create- vlm.get_async_client.return_value = mock_client-- result = await vlm._get_or_create_from_segments(segments, 3)-- # Should chain: seg0 -> seg1 (cached) -> seg2- assert result == "resp_012"-- def test_zero_segments(self):- """Test: end_idx = 0 returns None."""- vlm = self._create_vlm_with_mock_cache()-- segments = [[make_message("system", "System")]]-- result = VLMClass._get_or_create_from_segments(vlm, segments, 0)-- assert result is None---class TestCacheKeyGeneration:- """Tests for cache key generation logic."""-- def test_cache_key_includes_prefix(self):- """Test that cache keys include 'prefix:' prefix."""- vlm = VLMClass(- model="test-model",- api_key="test-key",- )-- messages = [make_message("system", "Hello")]- key = vlm._get_response_id_cache_key(messages)-- # Should include prefix in the key- assert "prefix:" in key or key.startswith("prefix:")diff --git a/tests/models/vlm/test_volcengine_media.py b/tests/models/vlm/test_volcengine_media.pynew file mode 100644index 000000000..a75613872--- /dev/null+++ b/tests/models/vlm/test_volcengine_media.py@@ -0,0 +1,155 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from openviking.models.vlm.backends.volcengine_vlm import VolcEngineVLM+++def _response(text: str):+ return SimpleNamespace(+ id="response-1",+ status="completed",+ output=[+ SimpleNamespace(+ type="message",+ content=[SimpleNamespace(type="output_text", text=text)],+ )+ ],+ usage=SimpleNamespace(+ input_tokens=120,+ output_tokens=40,+ input_tokens_details=SimpleNamespace(cached_tokens=2),+ output_tokens_details=SimpleNamespace(reasoning_tokens=3),+ ),+ )+++def _ark(response_text: str = "media result"):+ return SimpleNamespace(+ files=SimpleNamespace(+ create=AsyncMock(return_value=SimpleNamespace(id="file-1")),+ wait_for_processing=AsyncMock(return_value=SimpleNamespace(status="active")),+ delete=AsyncMock(),+ ),+ responses=SimpleNamespace(create=AsyncMock(return_value=_response(response_text))),+ )+++def _vlm(**overrides) -> VolcEngineVLM:+ config = {+ "provider": "volcengine",+ "api_key": "key",+ "api_base": "https://ark.example/v3",+ "model": "media-model",+ "max_tokens": 1024,+ "media": {+ "enabled": True,+ "max_concurrent": 2,+ "file_processing_timeout": 900.0,+ "file_poll_interval": 1.5,+ "video_fps": 0.5,+ },+ }+ config.update(overrides)+ return VolcEngineVLM(config)+++def test_volcengine_advertises_only_supported_media_inputs():+ vlm = _vlm()++ assert vlm.supports_media(media_type="audio", filename="meeting.mp3", size_bytes=1024)+ assert vlm.supports_media(media_type="video", filename="clip.MOV", size_bytes=1024)+ assert not vlm.supports_media(media_type="audio", filename="meeting.flac", size_bytes=1024)+ assert not vlm.supports_media(+ media_type="video",+ filename="clip.mp4",+ size_bytes=512 * 1024 * 1024 + 1,+ )+++async def test_audio_media_request_and_remote_cleanup(tmp_path, monkeypatch):+ path = tmp_path / "meeting.mp3"+ path.write_bytes(b"ID3-audio")+ ark = _ark("audio result")+ vlm = _vlm(extra_headers={"x-request-id": "request-1"})+ monkeypatch.setattr(vlm, "get_async_client", lambda: ark)++ result = await vlm.get_media_completion_async(+ prompt="analyze audio",+ media_path=path,+ filename=path.name,+ media_type="audio",+ )++ assert result == "audio result"+ upload = ark.files.create.await_args.kwargs+ assert upload["purpose"] == "user_data"+ assert upload["preprocess_configs"] is None+ assert upload["extra_headers"]["x-request-id"] == "request-1"+ ark.files.wait_for_processing.assert_awaited_once_with(+ "file-1",+ poll_interval=1.5,+ max_wait_seconds=900.0,+ )+ request = ark.responses.create.await_args.kwargs+ assert request["model"] == "media-model"+ assert request["input"][0]["content"] == [+ {"type": "input_audio", "file_id": "file-1"},+ {"type": "input_text", "text": "analyze audio"},+ ]+ assert request["max_output_tokens"] == 1024+ assert request["store"] is False+ ark.files.delete.assert_awaited_once_with(+ "file-1",+ extra_headers=upload["extra_headers"],+ )+++async def test_video_media_request_uses_configured_fps(tmp_path, monkeypatch):+ path = tmp_path / "clip.mov"+ path.write_bytes(b"video")+ ark = _ark("video result")+ vlm = _vlm()+ monkeypatch.setattr(vlm, "get_async_client", lambda: ark)++ result = await vlm.get_media_completion_async(+ prompt="analyze video",+ media_path=path,+ filename=path.name,+ media_type="video",+ )++ assert result == "video result"+ assert ark.files.create.await_args.kwargs["preprocess_configs"] == {"video": {"fps": 0.5}}+ assert ark.responses.create.await_args.kwargs["input"][0]["content"][0] == {+ "type": "input_video",+ "file_id": "file-1",+ }+ ark.files.delete.assert_awaited_once()+++async def test_failed_file_processing_is_terminal_and_cleans_remote_file(tmp_path, monkeypatch):+ path = tmp_path / "clip.mp4"+ path.write_bytes(b"video")+ ark = _ark()+ ark.files.wait_for_processing.return_value = SimpleNamespace(+ status="failed",+ error=SimpleNamespace(message="unsupported codec"),+ )+ vlm = _vlm(max_retries=3)+ monkeypatch.setattr(vlm, "get_async_client", lambda: ark)++ with pytest.raises(RuntimeError, match="unsupported codec"):+ await vlm.get_media_completion_async(+ prompt="analyze video",+ media_path=path,+ filename=path.name,+ media_type="video",+ )++ assert ark.files.create.await_count == 1+ ark.files.delete.assert_awaited_once()diff --git a/tests/parse/test_media_understanding_summary.py b/tests/parse/test_media_understanding_summary.pynew file mode 100644index 000000000..1ebe34cec--- /dev/null+++ b/tests/parse/test_media_understanding_summary.py@@ -0,0 +1,270 @@+import asyncio+from pathlib import Path+from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from openviking.parse.parsers.media import utils as media_utils+from openviking_cli.utils.config.vlm_config import VLMConfig++[email protected](autouse=True)+def _stub_media_prompt(monkeypatch):+ monkeypatch.setattr(media_utils, "render_prompt", lambda *_args, **_kwargs: "prompt")+++class _FS:+ def __init__(self, content=b"media"):+ self.content = content+ self.stat = AsyncMock(return_value={"size": len(content)})++ async def read(self, _uri, offset=0, size=-1, ctx=None):+ if offset >= len(self.content):+ return b""+ return self.content[offset : offset + size]+++class _BlockingReadFS:+ def __init__(self):+ self.read_calls = 0+ self.active_reads = 0+ self.peak_reads = 0+ self.two_reads_entered = asyncio.Event()+ self.release_reads = asyncio.Event()++ async def stat(self, *_args, **_kwargs):+ return {"size": 5}++ async def read(self, _uri, offset=0, size=-1, ctx=None):+ if offset:+ return b""+ self.read_calls += 1+ self.active_reads += 1+ self.peak_reads = max(self.peak_reads, self.active_reads)+ if self.active_reads == 2:+ self.two_reads_entered.set()+ try:+ await self.release_reads.wait()+ return b"video"+ finally:+ self.active_reads -= 1+++class _MediaVLM:+ model = "media-vlm"++ def supports_media(self, *, media_type, filename, size_bytes):+ extensions = {+ "audio": {".mp3", ".wav", ".aac", ".m4a"},+ "video": {".mp4", ".avi", ".mov"},+ }+ return (+ Path(filename).suffix.lower() in extensions.get(media_type, set())+ and 0 <= size_bytes <= 512 * 1024 * 1024+ )+++class _BlockingMediaClient(_MediaVLM):+ def __init__(self):+ self.active_inference = 0+ self.peak_inference = 0+ self.two_inferences_entered = asyncio.Event()+ self.release_inference = asyncio.Event()++ async def get_media_completion_async(self, **kwargs):+ self.active_inference += 1+ self.peak_inference = max(self.peak_inference, self.active_inference)+ if self.active_inference == 2:+ self.two_inferences_entered.set()+ try:+ await self.release_inference.wait()+ return f"# Clip\n\nUseful video summary.\n\n### {kwargs['filename']}\n\nDetailed scene."+ finally:+ self.active_inference -= 1+++class _CapturingPathClient(_MediaVLM):+ def __init__(self):+ self.path_calls = 0++ async def get_media_completion_async(+ self,+ *,+ media_path,+ prepare_media=None,+ **_kwargs,+ ):+ if prepare_media is not None:+ await prepare_media()+ self.path_calls += 1+ media_path.read_bytes()+ return "# Clip\n\nUseful summary."+++def _config(model_config, *, max_chars=4000):+ if hasattr(model_config, "get_client_instance"):+ vlm = model_config.get_client_instance()+ vlm.model = getattr(model_config, "model", getattr(vlm, "model", "media-vlm"))+ else:+ vlm = model_config+ return SimpleNamespace(+ vlm=vlm,+ semantic=SimpleNamespace(+ overview_max_chars=max_chars,+ abstract_max_chars=256,+ ),+ output_language_override="en",+ )+++def _lazy_client(*, return_value=None, side_effect=None):+ async def invoke(prepare_media=None, **_kwargs):+ if prepare_media is not None:+ await prepare_media()+ if side_effect is not None:+ raise side_effect+ return return_value++ client = _MediaVLM()+ client.get_media_completion_async = AsyncMock(side_effect=invoke)+ return client+++async def test_media_concurrency_bounds_staging_and_inference(monkeypatch):+ fs = _BlockingReadFS()+ client = _BlockingMediaClient()+ config_vlm = VLMConfig(+ provider="volcengine",+ api_key="test-key",+ model="media-model",+ media={"enabled": True, "max_concurrent": 2},+ )+ config_vlm._vlm_instance = client+ config = SimpleNamespace(+ vlm=config_vlm,+ semantic=SimpleNamespace(+ overview_max_chars=4000,+ abstract_max_chars=256,+ ),+ output_language_override="en",+ )+ monkeypatch.setattr(media_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(media_utils, "get_openviking_config", lambda: config)++ tasks = [+ asyncio.create_task(+ media_utils.generate_video_summary(+ f"viking://resources/video/clip-{index}.mp4",+ f"clip-{index}.mp4",+ llm_sem=asyncio.Semaphore(64),+ )+ )+ for index in range(4)+ ]++ await fs.two_reads_entered.wait()+ assert fs.read_calls == 2+ assert fs.peak_reads == 2+ fs.release_reads.set()+ await client.two_inferences_entered.wait()+ assert client.peak_inference == 2+ client.release_inference.set()+ results = await asyncio.gather(*tasks)++ assert all(result["summary"] for result in results)+ assert fs.peak_reads == 2+++async def test_unknown_size_media_stops_at_hard_staging_limit(+ monkeypatch,+):+ fs = _FS()+ fs.stat.return_value = {"size": 0}+ fs.read = AsyncMock(side_effect=[b"a" * 4, b"b" * 4, b"c", b""])+ client = _CapturingPathClient()+ model_config = SimpleNamespace(get_client_instance=lambda: client)+ monkeypatch.setattr(media_utils, "_MEDIA_READ_CHUNK_BYTES", 4)+ monkeypatch.setattr(media_utils, "MAX_MEDIA_FILE_BYTES", 8)+ monkeypatch.setattr(media_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ media_utils,+ "get_openviking_config",+ lambda: _config(model_config),+ )++ result = await media_utils.generate_video_summary(+ "viking://resources/video/unknown-size.mp4",+ "unknown-size.mp4",+ )++ assert result == {+ "name": "unknown-size.mp4",+ "summary": "",+ }+ assert fs.read.await_count == 3+ assert client.path_calls == 0+++async def test_success_normalizes_markdown_and_filename_heading(+ monkeypatch,+):+ raw = "```markdown\nA useful overview paragraph.\n\n## Facts\n\n- Revenue grew 20%.\n```"+ client = _lazy_client(return_value=raw)+ model_config = SimpleNamespace(get_client_instance=lambda: client)+ fs = _FS()+ monkeypatch.setattr(media_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ media_utils,+ "get_openviking_config",+ lambda: _config(model_config),+ )++ result = await media_utils.generate_video_summary(+ "viking://resources/video/quarterly.mov",+ "quarterly.mov",+ )++ assert result["summary"].startswith("# quarterly\n\nA useful overview paragraph.")+ assert "### quarterly.mov" in result["summary"]+ assert result["summary"].index("### quarterly.mov") < result["summary"].index("## Facts")+ client.get_media_completion_async.assert_awaited_once()+++async def test_provider_failure_returns_empty_summary(monkeypatch):+ client = _lazy_client(side_effect=RuntimeError("status code 401"))+ model_config = SimpleNamespace(+ model="doubao-seed-2-0-lite-260428",+ get_client_instance=lambda: client,+ )+ fs = _FS()+ monkeypatch.setattr(media_utils, "get_viking_fs", lambda: fs)+ monkeypatch.setattr(+ media_utils,+ "get_openviking_config",+ lambda: _config(model_config),+ )++ result = await media_utils.generate_video_summary(+ "viking://resources/video/clip.mp4",+ "clip.mp4",+ )++ assert result == {"name": "clip.mp4", "summary": ""}+++def test_refusal_provider_output_is_rejected():+ for raw in (+ "Unable to analyze this media.",+ "抱歉,我无法理解该视频内容。",+ "# Provider response\n\nI'm unable to analyze this media.",+ ):+ assert (+ media_utils._normalize_media_markdown(+ raw,+ filename="clip.mp4",+ overview_max_chars=4000,+ abstract_max_chars=256,+ )+ == ""+ )diff --git a/tests/storage/test_semantic_dag_media_overview.py b/tests/storage/test_semantic_dag_media_overview.pynew file mode 100644index 000000000..0f6521660--- /dev/null+++ b/tests/storage/test_semantic_dag_media_overview.py@@ -0,0 +1,66 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++from types import SimpleNamespace+from unittest.mock import patch++from openviking.server.identity import RequestContext, Role+from openviking.storage.queuefs.semantic_dag import (+ DirNode,+ SemanticDagExecutor,+)+from openviking_cli.session.user_id import UserIdentifier+++def _executor():+ ctx = RequestContext(+ user=UserIdentifier("acc", "user"),+ role=Role.USER,+ )+ with patch(+ "openviking.storage.queuefs.semantic_dag.get_viking_fs",+ return_value=SimpleNamespace(),+ ):+ return SemanticDagExecutor(SimpleNamespace(), "resource", 2, ctx)+++def _node(+ filename,+ summary,+):+ uri = "viking://resources/media"+ files = [(filename, summary)]+ file_paths = [f"{uri}/{name}" for name, _ in files]+ return DirNode(+ uri=uri,+ children_dirs=[],+ file_paths=file_paths,+ file_index={file_path: index for index, file_path in enumerate(file_paths)},+ child_index={},+ file_summaries=[{"name": name, "summary": item_summary} for name, item_summary in files],+ children_abstracts=[],+ pending=0,+ )+++def test_valid_single_media_summary_is_direct_overview():+ executor = _executor()++ for filename in ("demo.mp4", "recording.mp3"):+ summary = f"# Demo\n\nDense brief.\n\n### {filename}\n\nDetailed event."+ node = _node(filename, summary)+ assert executor._select_direct_media_overview(node, node.file_summaries) == summary+++def test_malformed_media_summary_is_not_direct():+ executor = _executor()+ cases = [+ _node("demo.mp4", ""),+ _node(+ "demo.mp4",+ "# Demo\n\nBrief without filename heading",+ ),+ ]++ for node in cases:+ assert executor._select_direct_media_overview(node, node.file_summaries) is Nonediff --git a/tests/unit/config/test_vlm_media_config.py b/tests/unit/config/test_vlm_media_config.pynew file mode 100644index 000000000..6eeec5ba9--- /dev/null+++ b/tests/unit/config/test_vlm_media_config.py@@ -0,0 +1,63 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++import pytest+from pydantic import ValidationError++from openviking_cli.utils.config.vlm_config import VLMConfig+++def _configured_vlm(**overrides) -> VLMConfig:+ values = {+ "provider": "volcengine",+ "api_key": "test-key",+ "model": "test-model",+ }+ values.update(overrides)+ return VLMConfig(**values)+++def test_vlm_media_is_disabled_by_default():+ config = VLMConfig()++ assert config.media.enabled is False+ assert config.media.max_concurrent == 2+ assert config.media.file_processing_timeout == 1800.0+ assert config.media.file_poll_interval == 3.0+ assert config.media.video_fps == 1.0+++def test_vlm_media_rejects_invalid_runtime_limits():+ invalid_values = [+ ({"max_concurrent": 0}, "max_concurrent"),+ ({"file_processing_timeout": 0}, "file_processing_timeout"),+ ({"file_poll_interval": 0}, "file_poll_interval"),+ ({"video_fps": 0.19}, "video_fps"),+ ({"video_fps": 5.01}, "video_fps"),+ ]++ for media, field in invalid_values:+ with pytest.raises(ValidationError, match=field):+ VLMConfig(media=media)+++def test_vlm_media_options_reach_backend_config():+ config = _configured_vlm(+ media={+ "enabled": True,+ "max_concurrent": 3,+ "file_processing_timeout": 900,+ "file_poll_interval": 1.5,+ "video_fps": 0.5,+ }+ )++ backend_config = config._build_vlm_config_dict_for_credential(config.credentials[0])++ assert backend_config["media"] == {+ "enabled": True,+ "max_concurrent": 3,+ "file_processing_timeout": 900.0,+ "file_poll_interval": 1.5,+ "video_fps": 0.5,+ }diff --git a/tests/unit/test_vectorize_file_strategy.py b/tests/unit/test_vectorize_file_strategy.pyindex 35df6ac22..a60f94844 100644--- a/tests/unit/test_vectorize_file_strategy.py+++ b/tests/unit/test_vectorize_file_strategy.py@@ -4,6 +4,10 @@ from unittest.mock import AsyncMockimport pytestfrom openviking.core.context import Context, ResourceContentType+from openviking.parse.parsers.media.utils import (+ MPEG_TS_PACKET_SIZE,+ MPEG_TS_PROBE_BYTES,+)from openviking.utils import embedding_utilsfrom openviking.utils.ingest_options import IngestOptions@@ -41,9 +45,15 @@ class DummyQueueManager:class DummyFS:def __init__(self, content):self.content = content+ self.read_calls = []self.read_file_calls = 0self.read_file_bytes_calls = 0+ async def read(self, _path, offset=0, size=-1, ctx=None):+ self.read_calls.append((offset, size))+ raw = self.content if isinstance(self.content, bytes) else str(self.content).encode("utf-8")+ return raw[offset : offset + size]+async def read_file(self, _path, ctx=None):self.read_file_calls += 1return self.content@@ -78,34 +88,61 @@ class DummyReq:self.account_id = "default"[email protected]("extension", [".ogg", ".m4a", ".opus", ".ac3"])-def test_get_resource_content_type_recognizes_supported_audio_extensions(extension):- assert (- embedding_utils.get_resource_content_type(f"recording{extension}")- == ResourceContentType.AUDIO- )+def test_get_resource_content_type_recognizes_media_extensions():+ expected = {+ "recording.ogg": ResourceContentType.AUDIO,+ "RECORDING.OPUS": ResourceContentType.AUDIO,+ "recording.mkv": ResourceContentType.VIDEO,+ "RECORDING.WEBM": ResourceContentType.VIDEO,+ "source.ts": ResourceContentType.TEXT,+ }+ for filename, content_type in expected.items():+ assert embedding_utils.get_resource_content_type(filename) == content_type[email protected]("extension", [".mkv", ".webm"])-def test_get_resource_content_type_recognizes_supported_video_extensions(extension):- assert (- embedding_utils.get_resource_content_type(f"recording{extension}")- == ResourceContentType.VIDEO- )+def _mpeg_ts_bytes() -> bytes:+ content = bytearray(MPEG_TS_PROBE_BYTES)+ for offset in range(0, MPEG_TS_PROBE_BYTES, MPEG_TS_PACKET_SIZE):+ content[offset] = 0x47+ return bytes(content)-def test_get_resource_content_type_media_extensions_are_case_insensitive():- assert embedding_utils.get_resource_content_type("RECORDING.OPUS") == ResourceContentType.AUDIO- assert embedding_utils.get_resource_content_type("RECORDING.WEBM") == ResourceContentType.VIDEO+async def test_vectorize_disambiguates_typescript_and_mpeg_ts(monkeypatch):+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(text_source="content_only", max_input_tokens=1000)+ ),+ )[email protected]("extension", [".cu", ".cuh"])-def test_get_resource_content_type_recognizes_cuda_extensions_as_text(extension):- assert embedding_utils.get_resource_content_type(f"kernel{extension}") == ResourceContentType.TEXT+ async def vectorize(filename, content, summary):+ queue = DummyQueue()+ fs = DummyFS(content)+ monkeypatch.setattr(+ embedding_utils,+ "get_queue_manager",+ lambda: DummyQueueManager(queue),+ )+ monkeypatch.setattr(embedding_utils, "get_viking_fs", lambda: fs)+ await embedding_utils.vectorize_file(+ file_path=f"viking://user/default/resources/{filename}",+ summary_dict={"name": filename, "summary": summary},+ parent_uri="viking://user/default/resources",+ ctx=DummyReq(),+ )+ return queue, fs+ source = "export const answer: number = 42;"+ text_queue, text_fs = await vectorize("source.ts", source, "TypeScript source")+ video_queue, video_fs = await vectorize("broadcast.ts", _mpeg_ts_bytes(), "")-def test_get_resource_content_type_keeps_ts_as_text():- assert embedding_utils.get_resource_content_type("source.ts") == ResourceContentType.TEXT+ assert text_fs.read_file_calls == 1+ assert text_queue.items[0].context_data["content"] == source+ assert video_fs.read_file_calls == 0+ assert video_queue.items[0].context_data["content"] == "broadcast.ts"@pytest.mark.asyncio@@ -852,6 +889,60 @@ async def test_vectorize_file_truncates_oversized_abstract(monkeypatch):assert abstract.encode("utf-8").decode("utf-8") == abstract # valid UTF-8+async def test_empty_media_uses_filename_but_unknown_binary_skips(monkeypatch):+ monkeypatch.setattr(+ embedding_utils,+ "get_openviking_config",+ lambda: types.SimpleNamespace(+ embedding=types.SimpleNamespace(+ text_source="content_only",+ max_input_tokens=1000,+ )+ ),+ )++ queue = DummyQueue()+ monkeypatch.setattr(+ embedding_utils,+ "get_queue_manager",+ lambda: DummyQueueManager(queue),+ )+ monkeypatch.setattr(+ embedding_utils,+ "get_viking_fs",+ lambda: DummyFS(b"media"),+ )+ await embedding_utils.vectorize_file(+ file_path="viking://resources/media/meeting.mp3",+ summary_dict={"name": "meeting.mp3", "summary": ""},+ parent_uri="viking://resources/media",+ ctx=DummyReq(),+ )++ assert queue.items[0].context_data["content"] == "meeting.mp3"++ queue = DummyQueue()+ monkeypatch.setattr(+ embedding_utils,+ "get_queue_manager",+ lambda: DummyQueueManager(queue),+ )+ monkeypatch.setattr(+ embedding_utils,+ "get_viking_fs",+ lambda: DummyFS(b"binary"),+ )+ await embedding_utils.vectorize_file(+ file_path="viking://resources/media/archive.bin",+ summary_dict={"name": "archive.bin", "summary": ""},+ parent_uri="viking://resources/media",+ ctx=DummyReq(),+ )++ assert queue.items == []++@pytest.mark.asyncioasync def test_vectorize_directory_meta_truncates_oversized_abstract(monkeypatch):"""The directory-meta path (fed by index_resource reading .abstract.md) must