Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(vlm): forward extra body to Ark media responses (#5440)

Evodae543498e46
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.md
index fa1ac0ec6..2f9a82af8 100644
--- a/docs/en/guides/01-configuration.md
+++ b/docs/en/guides/01-configuration.md
@@ -847,6 +847,8 @@ Audio and video understanding is an optional capability of the configured VLM. I
}
```
+`vlm.extra_request_body` also applies to the Ark Responses request for audio/video, for example `{"service_tier": "flex", "thinking": {"type": "disabled"}}`. Use fields supported by the selected model and Responses API. Explicit extra body fields override generated request fields through the SDK, including `store`; response storage defaults to `false` unless you explicitly override it. These fields are not forwarded to file upload, status, or deletion requests.
+
The VLM `model` value is the corresponding Ark model endpoint ID. `video_fps` applies only to video and controls the frame sampling rate sent to Ark.
The recommended starting models for audio and video understanding are `doubao-seed-2-0-lite-260428` and `doubao-seed-2-0-mini-260428`. These are recommended examples, not an exhaustive compatibility list; Ark continues to update its models and input capabilities. See Ark's official [video input capability list](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#ff5ef604) and [audio input capability list](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#9619c0ba) for other supported models. If `model` is an `ep-*` inference endpoint ID, verify that its underlying foundation model supports the corresponding media input. OpenViking does not validate audio or video model capabilities while loading configuration.
@@ -860,7 +862,7 @@ The recommended starting models for audio and video understanding are `doubao-se
Formats outside the understanding column continue to follow the existing parser and storage behavior; OpenViking does not transcode them or send them to the understanding model. When such a file is recognized as an audio or video leaf, an empty media summary is indexed using its filename.
-For a supported file, OpenViking uploads the media to the Ark Files API without explicitly setting `expire_at`, so file retention follows Ark's default policy. After processing completes, OpenViking references the file's `file_id` from the Responses API with response storage disabled, then attempts to delete the Ark file under a short cleanup deadline. Remote deletion is best-effort and does not replace an otherwise successful result if cleanup fails; a file whose deletion fails or times out continues to follow Ark's default retention policy. Local temporary files are removed independently even when remote cleanup fails or is cancelled.
+For a supported file, OpenViking uploads the media to the Ark Files API without explicitly setting `expire_at`, so file retention follows Ark's default policy. After processing completes, OpenViking references the file's `file_id` from the Responses API with response storage disabled by default, then attempts to delete the Ark file under a short cleanup deadline. Remote deletion is best-effort and does not replace an otherwise successful result if cleanup fails; a file whose deletion fails or times out continues to follow Ark's default retention policy. Local temporary files are removed independently even when remote cleanup fails or is cancelled.
- A successful summary for a directory containing exactly one audio or video file becomes that directory's L1 directly, with L0 derived through the existing semantic path. No second generic VLM summarization is performed.
- Media in a mixed directory contributes its summary to the existing generic VLM aggregation.
diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.md
index e8cbbdfae..8233a2c1c 100644
--- a/docs/zh/guides/01-configuration.md
+++ b/docs/zh/guides/01-configuration.md
@@ -812,6 +812,8 @@ LiteLLM 的 Bedrock bearer-token API-key 鉴权,请设置 `forward_api_key=tru
}
```
+`vlm.extra_request_body` 也适用于音视频的方舟 Responses 请求,例如 `{"service_tier": "flex", "thinking": {"type": "disabled"}}`。请使用所选模型和 Responses API 支持的字段。显式配置的额外字段通过 SDK 覆盖自动生成的请求字段,包括 `store`;响应存储默认设为 `false`,仅在显式配置时覆盖。这些字段不会传给文件上传、状态查询或删除请求。
+
VLM 的 `model` 填写对应的方舟模型 endpoint ID。`video_fps` 仅用于视频,控制发送给方舟的视频采样帧率。
推荐使用 `doubao-seed-2-0-lite-260428` 或 `doubao-seed-2-0-mini-260428` 作为音视频理解模型。它们是可直接采用的推荐示例,并非完整的支持模型列表;方舟会持续更新模型及其输入能力。视频理解的可选模型请参考方舟官方[视频输入能力列表](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#ff5ef604),音频理解的可选模型请参考方舟官方[音频输入能力列表](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1330310?lang=zh#9619c0ba)。如果 `model` 填写的是 `ep-*` 推理接入点 ID,请确认该接入点背后的基础模型支持对应的媒体输入。OpenViking 不会在配置加载时校验模型的音频或视频能力。
@@ -825,7 +827,7 @@ VLM 的 `model` 填写对应的方舟模型 endpoint ID。`video_fps` 仅用于
不在“可理解”列中的格式继续沿用现有 Parser 和存储行为;OpenViking 不会对这些文件转码,也不会把它们发送给理解模型。当文件被识别为音频或视频叶子节点时,空媒体摘要会使用文件名入库。
-对于支持的文件,OpenViking 将媒体上传到方舟 Files API,且不显式指定 `expire_at`,因此文件保留时间遵循方舟的默认策略。文件处理完成后,OpenViking 通过禁用响应存储的 Responses API 请求引用其 `file_id`,最后在较短的清理超时内尝试删除方舟文件。远端删除属于 best-effort;如果删除失败或超时,不会覆盖已经成功的理解结果,文件将继续遵循方舟的默认保留策略。本地临时文件独立清理,即使远端清理失败或请求被取消也会删除。
+对于支持的文件,OpenViking 将媒体上传到方舟 Files API,且不显式指定 `expire_at`,因此文件保留时间遵循方舟的默认策略。文件处理完成后,OpenViking 通过默认禁用响应存储的 Responses API 请求引用其 `file_id`,最后在较短的清理超时内尝试删除方舟文件。远端删除属于 best-effort;如果删除失败或超时,不会覆盖已经成功的理解结果,文件将继续遵循方舟的默认保留策略。本地临时文件独立清理,即使远端清理失败或请求被取消也会删除。
- 目录中只有一个音频或视频文件且理解成功时,该摘要直接成为目录 L1,并通过现有语义链路派生 L0,不再调用通用 VLM 做第二次总结。
- 媒体位于混合目录时,其摘要仍参与现有通用 VLM 聚合。
diff --git a/openviking/models/vlm/backends/volcengine_media.py b/openviking/models/vlm/backends/volcengine_media.py
index 40c1e1700..3debdea48 100644
--- a/openviking/models/vlm/backends/volcengine_media.py
+++ b/openviking/models/vlm/backends/volcengine_media.py
@@ -295,6 +295,8 @@ async def _attempt(
}
if vlm.max_tokens is not None:
request["max_output_tokens"] = vlm.max_tokens
+ if vlm.extra_request_body:
+ request["extra_body"] = dict(vlm.extra_request_body)
started = time.monotonic()
response = await client.responses.create(**request)
diff --git a/tests/models/vlm/test_volcengine_media.py b/tests/models/vlm/test_volcengine_media.py
index a75613872..105908cad 100644
--- a/tests/models/vlm/test_volcengine_media.py
+++ b/tests/models/vlm/test_volcengine_media.py
@@ -1,10 +1,13 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+import json
from types import SimpleNamespace
from unittest.mock import AsyncMock
+import httpx
import pytest
+from volcenginesdkarkruntime import AsyncArk
from openviking.models.vlm.backends.volcengine_vlm import VolcEngineVLM
@@ -71,42 +74,70 @@ def test_volcengine_advertises_only_supported_media_inputs():
)
-async def test_audio_media_request_and_remote_cleanup(tmp_path, monkeypatch):
- path = tmp_path / "meeting.mp3"
- path.write_bytes(b"ID3-audio")
- ark = _ark("audio result")
- vlm = _vlm(extra_headers={"x-request-id": "request-1"})
- monkeypatch.setattr(vlm, "get_async_client", lambda: ark)
-
- result = await vlm.get_media_completion_async(
- prompt="analyze audio",
- media_path=path,
- filename=path.name,
- media_type="audio",
- )
[email protected]("media_type, suffix", [("audio", ".mp3"), ("video", ".mov")])
+ "extra_body",
+ [None, {"service_tier": "flex", "thinking": {"type": "disabled"}}, {"store": True}],
+)
+async def test_media_request_and_remote_cleanup(
+ tmp_path, monkeypatch, media_type, suffix, extra_body
+):
+ path = tmp_path / f"media{suffix}"
+ path.write_bytes(b"local media fixture")
+ requests = []
+
+ def send(request):
+ requests.append(request)
+ if request.url.path == "/v3/responses":
+ return httpx.Response(
+ 200,
+ json={
+ "id": "response-1",
+ "status": "completed",
+ "output": [
+ {
+ "type": "message",
+ "content": [{"type": "output_text", "text": "media result"}],
+ }
+ ],
+ },
+ )
+ return httpx.Response(200, json={"id": "file-1", "status": "active", "deleted": True})
+
+ vlm = _vlm(extra_headers={"x-request-id": "request-1"}, extra_request_body=extra_body)
+ async with AsyncArk(
+ api_key="key",
+ base_url="https://ark.example/v3",
+ max_retries=0,
+ http_client=httpx.AsyncClient(transport=httpx.MockTransport(send)),
+ ) as ark:
+ monkeypatch.setattr(vlm, "get_async_client", lambda: ark)
+ result = await vlm.get_media_completion_async(
+ prompt="analyze media", media_path=path, filename=path.name, media_type=media_type
+ )
- assert result == "audio result"
- upload = ark.files.create.await_args.kwargs
- assert upload["purpose"] == "user_data"
- assert upload["preprocess_configs"] is None
- assert upload["extra_headers"]["x-request-id"] == "request-1"
- ark.files.wait_for_processing.assert_awaited_once_with(
- "file-1",
- poll_interval=1.5,
- max_wait_seconds=900.0,
- )
- request = ark.responses.create.await_args.kwargs
+ assert result == "media result"
+ assert [(r.method, r.url.path) for r in requests] == [
+ ("POST", "/v3/files"),
+ ("GET", "/v3/files/file-1"),
+ ("POST", "/v3/responses"),
+ ("DELETE", "/v3/files/file-1"),
+ ]
+ request = json.loads(requests[2].content)
assert request["model"] == "media-model"
assert request["input"][0]["content"] == [
- {"type": "input_audio", "file_id": "file-1"},
- {"type": "input_text", "text": "analyze audio"},
+ {"type": f"input_{media_type}", "file_id": "file-1"},
+ {"type": "input_text", "text": "analyze media"},
]
assert request["max_output_tokens"] == 1024
- assert request["store"] is False
- ark.files.delete.assert_awaited_once_with(
- "file-1",
- extra_headers=upload["extra_headers"],
- )
+ assert request["store"] is (extra_body or {}).get("store", False)
+ for key, value in (extra_body or {}).items():
+ assert request[key] == value
+ assert key.encode() not in requests[0].content
+ assert vlm.extra_request_body == (extra_body or {})
+ for sent in (requests[0], requests[2], requests[3]):
+ assert sent.headers["x-request-id"] == "request-1"
+ assert not requests[1].content and not json.loads(requests[3].content)
async def test_video_media_request_uses_configured_fps(tmp_path, monkeypatch):