Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(vlm): forward extra body to Ark media responses (#5440)
Evodae543498e46
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.mdindex fa1ac0ec6..2f9a82af8 100644--- a/docs/en/guides/01-configuration.md+++ b/docs/en/guides/01-configuration.md@@ -847,6 +847,8 @@ Audio and video understanding is an optional capability of the configured VLM. I}```+`vlm.extra_request_body` also applies to the Ark Responses request for audio/video, for example `{"service_tier": "flex", "thinking": {"type": "disabled"}}`. Use fields supported by the selected model and Responses API. Explicit extra body fields override generated request fields through the SDK, including `store`; response storage defaults to `false` unless you explicitly override it. These fields are not forwarded to file upload, status, or deletion requests.+The VLM `model` value is the corresponding Ark model endpoint ID. `video_fps` applies only to video and controls the frame sampling rate sent to Ark.The recommended starting models for audio and video understanding are `doubao-seed-2-0-lite-260428` and `doubao-seed-2-0-mini-260428`. These are recommended examples, not an exhaustive compatibility list; Ark continues to update its models and input capabilities. See Ark's official [video input capability list](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#ff5ef604) and [audio input capability list](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#9619c0ba) for other supported models. If `model` is an `ep-*` inference endpoint ID, verify that its underlying foundation model supports the corresponding media input. OpenViking does not validate audio or video model capabilities while loading configuration.@@ -860,7 +862,7 @@ The recommended starting models for audio and video understanding are `doubao-seFormats outside the understanding column continue to follow the existing parser and storage behavior; OpenViking does not transcode them or send them to the understanding model. When such a file is recognized as an audio or video leaf, an empty media summary is indexed using its filename.-For a supported file, OpenViking uploads the media to the Ark Files API without explicitly setting `expire_at`, so file retention follows Ark's default policy. After processing completes, OpenViking references the file's `file_id` from the Responses API with response storage disabled, then attempts to delete the Ark file under a short cleanup deadline. Remote deletion is best-effort and does not replace an otherwise successful result if cleanup fails; a file whose deletion fails or times out continues to follow Ark's default retention policy. Local temporary files are removed independently even when remote cleanup fails or is cancelled.+For a supported file, OpenViking uploads the media to the Ark Files API without explicitly setting `expire_at`, so file retention follows Ark's default policy. After processing completes, OpenViking references the file's `file_id` from the Responses API with response storage disabled by default, then attempts to delete the Ark file under a short cleanup deadline. Remote deletion is best-effort and does not replace an otherwise successful result if cleanup fails; a file whose deletion fails or times out continues to follow Ark's default retention policy. Local temporary files are removed independently even when remote cleanup fails or is cancelled.- A successful summary for a directory containing exactly one audio or video file becomes that directory's L1 directly, with L0 derived through the existing semantic path. No second generic VLM summarization is performed.- Media in a mixed directory contributes its summary to the existing generic VLM aggregation.diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.mdindex e8cbbdfae..8233a2c1c 100644--- a/docs/zh/guides/01-configuration.md+++ b/docs/zh/guides/01-configuration.md@@ -812,6 +812,8 @@ LiteLLM 的 Bedrock bearer-token API-key 鉴权,请设置 `forward_api_key=tru}```+`vlm.extra_request_body` 也适用于音视频的方舟 Responses 请求,例如 `{"service_tier": "flex", "thinking": {"type": "disabled"}}`。请使用所选模型和 Responses API 支持的字段。显式配置的额外字段通过 SDK 覆盖自动生成的请求字段,包括 `store`;响应存储默认设为 `false`,仅在显式配置时覆盖。这些字段不会传给文件上传、状态查询或删除请求。+VLM 的 `model` 填写对应的方舟模型 endpoint ID。`video_fps` 仅用于视频,控制发送给方舟的视频采样帧率。推荐使用 `doubao-seed-2-0-lite-260428` 或 `doubao-seed-2-0-mini-260428` 作为音视频理解模型。它们是可直接采用的推荐示例,并非完整的支持模型列表;方舟会持续更新模型及其输入能力。视频理解的可选模型请参考方舟官方[视频输入能力列表](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#ff5ef604),音频理解的可选模型请参考方舟官方[音频输入能力列表](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#9619c0ba)。如果 `model` 填写的是 `ep-*` 推理接入点 ID,请确认该接入点背后的基础模型支持对应的媒体输入。OpenViking 不会在配置加载时校验模型的音频或视频能力。@@ -825,7 +827,7 @@ VLM 的 `model` 填写对应的方舟模型 endpoint ID。`video_fps` 仅用于不在“可理解”列中的格式继续沿用现有 Parser 和存储行为;OpenViking 不会对这些文件转码,也不会把它们发送给理解模型。当文件被识别为音频或视频叶子节点时,空媒体摘要会使用文件名入库。-对于支持的文件,OpenViking 将媒体上传到方舟 Files API,且不显式指定 `expire_at`,因此文件保留时间遵循方舟的默认策略。文件处理完成后,OpenViking 通过禁用响应存储的 Responses API 请求引用其 `file_id`,最后在较短的清理超时内尝试删除方舟文件。远端删除属于 best-effort;如果删除失败或超时,不会覆盖已经成功的理解结果,文件将继续遵循方舟的默认保留策略。本地临时文件独立清理,即使远端清理失败或请求被取消也会删除。+对于支持的文件,OpenViking 将媒体上传到方舟 Files API,且不显式指定 `expire_at`,因此文件保留时间遵循方舟的默认策略。文件处理完成后,OpenViking 通过默认禁用响应存储的 Responses API 请求引用其 `file_id`,最后在较短的清理超时内尝试删除方舟文件。远端删除属于 best-effort;如果删除失败或超时,不会覆盖已经成功的理解结果,文件将继续遵循方舟的默认保留策略。本地临时文件独立清理,即使远端清理失败或请求被取消也会删除。- 目录中只有一个音频或视频文件且理解成功时,该摘要直接成为目录 L1,并通过现有语义链路派生 L0,不再调用通用 VLM 做第二次总结。- 媒体位于混合目录时,其摘要仍参与现有通用 VLM 聚合。diff --git a/openviking/models/vlm/backends/volcengine_media.py b/openviking/models/vlm/backends/volcengine_media.pyindex 40c1e1700..3debdea48 100644--- a/openviking/models/vlm/backends/volcengine_media.py+++ b/openviking/models/vlm/backends/volcengine_media.py@@ -295,6 +295,8 @@ async def _attempt(}if vlm.max_tokens is not None:request["max_output_tokens"] = vlm.max_tokens+ if vlm.extra_request_body:+ request["extra_body"] = dict(vlm.extra_request_body)started = time.monotonic()response = await client.responses.create(**request)diff --git a/tests/models/vlm/test_volcengine_media.py b/tests/models/vlm/test_volcengine_media.pyindex a75613872..105908cad 100644--- a/tests/models/vlm/test_volcengine_media.py+++ b/tests/models/vlm/test_volcengine_media.py@@ -1,10 +1,13 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+import jsonfrom types import SimpleNamespacefrom unittest.mock import AsyncMock+import httpximport pytest+from volcenginesdkarkruntime import AsyncArkfrom openviking.models.vlm.backends.volcengine_vlm import VolcEngineVLM@@ -71,42 +74,70 @@ def test_volcengine_advertises_only_supported_media_inputs():)-async def test_audio_media_request_and_remote_cleanup(tmp_path, monkeypatch):- path = tmp_path / "meeting.mp3"- path.write_bytes(b"ID3-audio")- ark = _ark("audio result")- vlm = _vlm(extra_headers={"x-request-id": "request-1"})- monkeypatch.setattr(vlm, "get_async_client", lambda: ark)-- result = await vlm.get_media_completion_async(- prompt="analyze audio",- media_path=path,- filename=path.name,- media_type="audio",- )[email protected]("media_type, suffix", [("audio", ".mp3"), ("video", ".mov")])+ "extra_body",+ [None, {"service_tier": "flex", "thinking": {"type": "disabled"}}, {"store": True}],+)+async def test_media_request_and_remote_cleanup(+ tmp_path, monkeypatch, media_type, suffix, extra_body+):+ path = tmp_path / f"media{suffix}"+ path.write_bytes(b"local media fixture")+ requests = []++ def send(request):+ requests.append(request)+ if request.url.path == "/v3/responses":+ return httpx.Response(+ 200,+ json={+ "id": "response-1",+ "status": "completed",+ "output": [+ {+ "type": "message",+ "content": [{"type": "output_text", "text": "media result"}],+ }+ ],+ },+ )+ return httpx.Response(200, json={"id": "file-1", "status": "active", "deleted": True})++ vlm = _vlm(extra_headers={"x-request-id": "request-1"}, extra_request_body=extra_body)+ async with AsyncArk(+ api_key="key",+ base_url="https://ark.example/v3",+ max_retries=0,+ http_client=httpx.AsyncClient(transport=httpx.MockTransport(send)),+ ) as ark:+ monkeypatch.setattr(vlm, "get_async_client", lambda: ark)+ result = await vlm.get_media_completion_async(+ prompt="analyze media", media_path=path, filename=path.name, media_type=media_type+ )- assert result == "audio result"- upload = ark.files.create.await_args.kwargs- assert upload["purpose"] == "user_data"- assert upload["preprocess_configs"] is None- assert upload["extra_headers"]["x-request-id"] == "request-1"- ark.files.wait_for_processing.assert_awaited_once_with(- "file-1",- poll_interval=1.5,- max_wait_seconds=900.0,- )- request = ark.responses.create.await_args.kwargs+ assert result == "media result"+ assert [(r.method, r.url.path) for r in requests] == [+ ("POST", "/v3/files"),+ ("GET", "/v3/files/file-1"),+ ("POST", "/v3/responses"),+ ("DELETE", "/v3/files/file-1"),+ ]+ request = json.loads(requests[2].content)assert request["model"] == "media-model"assert request["input"][0]["content"] == [- {"type": "input_audio", "file_id": "file-1"},- {"type": "input_text", "text": "analyze audio"},+ {"type": f"input_{media_type}", "file_id": "file-1"},+ {"type": "input_text", "text": "analyze media"},]assert request["max_output_tokens"] == 1024- assert request["store"] is False- ark.files.delete.assert_awaited_once_with(- "file-1",- extra_headers=upload["extra_headers"],- )+ assert request["store"] is (extra_body or {}).get("store", False)+ for key, value in (extra_body or {}).items():+ assert request[key] == value+ assert key.encode() not in requests[0].content+ assert vlm.extra_request_body == (extra_body or {})+ for sent in (requests[0], requests[2], requests[3]):+ assert sent.headers["x-request-id"] == "request-1"+ assert not requests[1].content and not json.loads(requests[3].content)async def test_video_media_request_uses_configured_fps(tmp_path, monkeypatch):