Skip to content

Agent Zero AI framework

Integrate vision sidecar routing with model presets

Alessandro59f5ae03ea61
diff --git a/extensions/python/system_prompt/_11_tools_prompt.py b/extensions/python/system_prompt/_11_tools_prompt.py
index 6349a862c..a395d6c3b 100644
--- a/extensions/python/system_prompt/_11_tools_prompt.py
+++ b/extensions/python/system_prompt/_11_tools_prompt.py
@@ -51,10 +51,15 @@ async def build_prompt(agent: Agent) -> str:
prompt = agent.read_prompt("agent.system.tools.md", tools=tools_str)
# vision support
- from plugins._model_config.helpers.model_config import get_chat_model_config
+ from plugins._model_config.helpers.model_config import (
+ get_chat_model_config,
+ use_vision_sidecar,
+ )
chat_cfg = get_chat_model_config(agent)
- if chat_cfg.get("vision", False):
+ if use_vision_sidecar(agent):
+ prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision_sidecar.md")
+ elif chat_cfg.get("vision", False):
prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision.md")
return prompt
diff --git a/helpers/responses_tools.py b/helpers/responses_tools.py
index 93778d652..877f03d79 100644
--- a/helpers/responses_tools.py
+++ b/helpers/responses_tools.py
@@ -114,8 +114,13 @@ def _local_tool_prompts(agent: Any) -> list[tuple[str, str]]:
def _vision_tool_prompt(agent: Any) -> str:
try:
- from plugins._model_config.helpers.model_config import get_chat_model_config
+ from plugins._model_config.helpers.model_config import (
+ get_chat_model_config,
+ use_vision_sidecar,
+ )
+ if use_vision_sidecar(agent):
+ return agent.read_prompt("agent.system.tools_vision_sidecar.md")
if not get_chat_model_config(agent).get("vision", False):
return ""
return agent.read_prompt("agent.system.tools_vision.md")
diff --git a/helpers/responses_tools.py.dox.md b/helpers/responses_tools.py.dox.md
index 7c2631867..041919146 100644
--- a/helpers/responses_tools.py.dox.md
+++ b/helpers/responses_tools.py.dox.md
@@ -12,7 +12,7 @@
## Local Contracts
-- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` only when the active chat model enables the matching vision prompt.
+- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` when either Main native vision or the effective preset's Vision Model enables the matching prompt.
- Discover local prompt files through `helpers.subagents.get_paths`; this module
owns the Responses-specific prompt-name compatibility rules.
- Local prompt-derived function names use existing bullet declarations that pair a backticked name with `arg` or `args` for multi-tool prompt files, otherwise prefer explicit `"tool_name"` examples, then the first prompt heading, and finally the prompt filename.
@@ -25,8 +25,8 @@
- Preserve original Agent Zero tool names through the native Responses name map.
- Keep MCP tool schemas merged after local prompt-derived tools.
- Apply `helpers.tool_policy` before emitting local or MCP schemas; a blocked
- capability is absent from provider-native tool definitions. Vision remains
- controlled solely by the active chat model configuration.
+ capability is absent from provider-native tool definitions. Vision routing
+ is controlled by the effective model preset rather than Agent Editor.
- Connector remote tools are advertised only when `_a0_connector` runtime metadata says the matching connected CLI capability is currently available.
## Work Guidance
diff --git a/plugins/_model_config/AGENTS.md b/plugins/_model_config/AGENTS.md
index f3ba7d529..28ec60ee9 100644
--- a/plugins/_model_config/AGENTS.md
+++ b/plugins/_model_config/AGENTS.md
@@ -14,7 +14,7 @@
## Local Contracts
-- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline.
+- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline; its Vision Model slot is optional.
- Preset definitions are global. Global, project, agent-profile, and project/profile plugin configs persist only `model_preset`; chats may persist a preset reference as their explicit override.
- Preserve scoped plugin resolution order and fall back invalid or missing scope/chat references to `Default`.
- Project Settings `llm` payloads are owned here through the generic `helpers.projects` project extension-data hooks; keep project helper code agnostic to `_model_config` paths, presets, and inheritance rules.
@@ -22,7 +22,12 @@
- Check API-key readiness only for the effective model configuration; unused global presets must not produce Welcome-screen warnings.
- Coordinate OAuth-backed providers with `_oauth` instead of hardcoding provider-specific auth here.
- `model_config_get` exposes `model_configured` as a derived chat-model readiness flag from provider, model name, and API-key availability.
-- Non-default presets may inherit omitted slots or durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.
+- Non-default presets may inherit omitted main, utility, or embedding slots and durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.
+- The optional `vision` slot is strictly per preset and never inherited from `Default`; an empty slot disables sidecar vision for that preset.
+- Main native vision wins by default. A configured Vision Model handles `vision_load` when Main lacks vision, or when that preset explicitly enables `override_main`.
+- Keep the optional Vision provider/model selector inside the Main Model card and flush with Main's field alignment, without a nested left inset. Show it only while Main vision is disabled or `override_main` is enabled; do not render a standalone Vision Model card.
+- Show `Use separate Vision Model` immediately below `Supports Vision` while Main vision is enabled, not inside Advanced Settings; describe the disabled state as using Main's native vision.
+- In model overviews, render the effective Vision Model as a text-only `Vision override / Provider / Model` child aligned with Main's provider column, not as an icon-bearing peer row.
- Changing a model provider in the settings UI must clear `api_base` and `kwargs` because both may be provider-specific.
- Repair provider-specific model-config aliases at the model-config read/build boundary; keep provider-specific repairs out of provider-agnostic core wrappers such as `models.py`.
- `modelConfig.createPresetEditor()` owns local preset drafts, row actions, and stable UI-only row keys so deletion or renaming cannot rebind nested model fields.
diff --git a/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html b/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html
index 460bdfe72..0940d547f 100644
--- a/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html
+++ b/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html
@@ -65,6 +65,16 @@
</span>
</div>
</template>
+ <template x-if="preset.vision?.name && (!preset.chat?.vision || preset.vision?.override_main)">
+ <div class="model-switcher-model-row">
+ <span class="model-switcher-model-label">Vision</span>
+ <span class="model-switcher-model-value">
+ <span x-text="preset.vision.provider" style="opacity:0.5;"></span>
+ <span style="opacity:0.3; margin:0 3px;">/</span>
+ <span x-text="preset.vision.name"></span>
+ </span>
+ </div>
+ </template>
<template x-if="preset.utility?.name">
<div class="model-switcher-model-row">
<span class="model-switcher-model-label">Utility</span>
diff --git a/plugins/_model_config/helpers/model_config.py b/plugins/_model_config/helpers/model_config.py
index 6a1386451..3f6c36f93 100644
--- a/plugins/_model_config/helpers/model_config.py
+++ b/plugins/_model_config/helpers/model_config.py
@@ -16,11 +16,21 @@ PRESET_SCOPE_GLOBAL = "global"
PRESET_SCOPE_PROJECT = "project"
PRESET_SLOT_CONFIG_SECTIONS = {
"chat": "chat_model",
+ "vision": "vision_model",
"utility": "utility_model",
"embedding": "embedding_model",
}
MODEL_SLOT_PRESET_REPLACE_FIELDS = {"kwargs"}
IMPLICIT_PRESET_SLOT_DEFAULTS = {
+ "vision": {
+ "vision": True,
+ "max_embeds": 10,
+ "override_main": False,
+ "rl_requests": 0,
+ "rl_input": 0,
+ "rl_output": 0,
+ "kwargs": {},
+ },
"utility": {
"ctx_length": 128000,
"ctx_input": 0.7,
@@ -267,6 +277,9 @@ def _clean_preset_for_file(preset: dict) -> dict:
slot_config = preset.get(slot)
if isinstance(slot_config, dict):
slot_clean = _strip_ui_fields(slot_config, strip_api_key=True)
+ if slot == "vision" and _slot_has_identity(slot_clean):
+ slot_clean["vision"] = True
+ slot_clean.setdefault("max_embeds", 10)
cleaned[slot] = (
slot_clean
if name == DEFAULT_PRESET_NAME
@@ -339,7 +352,12 @@ def validate_presets(presets: list, *, require_default: bool = True) -> list:
def normalize_config_for_save(config: dict) -> dict:
"""Remove UI-only fields and inline API keys before storing scoped config."""
cleaned = deepcopy(config or {})
- for section_name in ("chat_model", "utility_model", "embedding_model"):
+ for section_name in (
+ "chat_model",
+ "vision_model",
+ "utility_model",
+ "embedding_model",
+ ):
section = cleaned.get(section_name)
if isinstance(section, dict):
cleaned[section_name] = _strip_ui_fields(section, strip_api_key=True)
@@ -659,10 +677,12 @@ def build_config_from_preset(
continue
slot_config = _get_preset_slot_config(preset, slot)
if not _should_apply_preset_slot(slot, slot_config):
+ if slot == "vision":
+ config[section] = {}
continue
config[section] = _merge_model_slot(
slot,
- config.get(section, {}),
+ {} if slot == "vision" else config.get(section, {}),
slot_config,
strip_api_key=strip_api_key,
)
@@ -738,6 +758,23 @@ def get_chat_model_config(agent=None) -> dict:
return get_effective_config(agent).get("chat_model", {})
+def get_vision_model_config(agent=None) -> dict:
+ """Get the optional, strictly per-preset Vision Model config."""
+ return get_effective_config(agent).get("vision_model", {})
+
+
+def use_vision_sidecar(agent=None) -> bool:
+ """Return whether vision_load should delegate to the preset's Vision Model."""
+ vision_cfg = get_vision_model_config(agent)
+ if not (
+ str(vision_cfg.get("provider") or "").strip()
+ and str(vision_cfg.get("name") or "").strip()
+ ):
+ return False
+ chat_cfg = get_chat_model_config(agent)
+ return not bool(chat_cfg.get("vision")) or bool(vision_cfg.get("override_main"))
+
+
def get_utility_model_config(agent=None) -> dict:
"""Get utility model config, with per-chat override if active."""
return get_effective_config(agent).get("utility_model", {})
@@ -835,6 +872,16 @@ def build_utility_model(agent=None):
)
+def build_vision_model(agent=None):
+ """Build the optional Vision Model selected by the effective preset."""
+ cfg = get_vision_model_config(agent)
+ mc = build_model_config(cfg, models.ModelType.CHAT)
+ mc.vision = True
+ return models.get_chat_model(
+ mc.provider, mc.name, model_config=mc, **mc.build_kwargs()
+ )
+
+
def build_embedding_model(agent=None):
"""Build and return an embedding model wrapper."""
cfg = get_embedding_model_config(agent)
@@ -881,6 +928,8 @@ def get_missing_api_key_providers(agent=None) -> list[dict]:
("Utility Model", cfg.get("utility_model", {})),
("Embedding Model", get_embedding_model_config(agent)),
]
+ if use_vision_sidecar(agent):
+ checks.insert(1, ("Vision Model", cfg.get("vision_model", {})))
for label, model_cfg in checks:
provider = model_cfg.get("provider", "")
diff --git a/plugins/_model_config/webui/main.html b/plugins/_model_config/webui/main.html
index 7aa9908e4..83c58e6e1 100644
--- a/plugins/_model_config/webui/main.html
+++ b/plugins/_model_config/webui/main.html
@@ -21,7 +21,7 @@
<div class="preset-editor-intro">
<div class="field-title">Model Presets</div>
<div class="field-description">
- Each preset contains the complete main, utility, and embedding model setup. Default is always available.
+ Each preset contains the main, utility, embedding, and optional vision-sidecar setup. Default is always available.
</div>
</div>
@@ -67,9 +67,21 @@
<div class="section-title">Main Model</div>
<div class="section-description">Primary model for conversations, reasoning, and tools.</div>
</div>
- <div x-data="{ get model() { return selectedPreset.chat; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
+ <div x-data="{ get model() { return selectedPreset.chat; }, get visionModel() { return selectedPreset.vision; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
<x-component path="/plugins/_model_config/webui/model-field.html"></x-component>
</div>
+ <template x-if="!selectedPreset.chat.vision || selectedPreset.vision.override_main">
+ <div class="vision-sidecar-selector">
+ <div class="field-title">Vision sidecar</div>
+ <div class="field-description"
+ x-text="selectedPreset.chat.vision
+ ? 'Overrides Main model vision for vision_load.'
+ : 'Used by vision_load while Main model vision is disabled. Leave empty to disable image analysis.'"></div>
+ <div x-data="{ get model() { return selectedPreset.vision; }, modelType: 'vision', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
+ <x-component path="/plugins/_model_config/webui/model-field.html"></x-component>
+ </div>
+ </div>
+ </template>
</section>
<section class="preset-model-section">
@@ -166,6 +178,11 @@
margin-top: 0;
}
+ .vision-sidecar-selector {
+ margin: 0.75rem 0 0;
+ padding: 0.25rem 0 0;
+ }
+
.preset-editor-secondary-actions {
display: flex;
flex-wrap: wrap;
diff --git a/plugins/_model_config/webui/model-config-store.js b/plugins/_model_config/webui/model-config-store.js
index 5e69bde20..1ab6d8c8e 100644
--- a/plugins/_model_config/webui/model-config-store.js
+++ b/plugins/_model_config/webui/model-config-store.js
@@ -7,6 +7,7 @@ import { switcherState, switcherMethods } from "/plugins/_model_config/webui/swi
export const MODEL_SECTIONS = [
{ key: 'chat_model', title: 'Main Model', desc: 'Primary model for chat, reasoning, and browser tasks.' },
+ { key: 'vision_model', title: 'Vision Model', desc: 'Optional model used by vision_load when Main vision is unavailable or overridden.' },
{ key: 'utility_model', title: 'Utility Model', desc: 'Lightweight model for background tasks: memory management, prompt preparation, summarization.' },
{ key: 'embedding_model', title: 'Embedding Model', desc: 'Model for generating vector embeddings used in knowledge retrieval.' }
];
@@ -59,6 +60,15 @@ function isBlankPresetValue(value) {
}
const IMPLICIT_PRESET_SLOT_DEFAULTS = {
+ vision: {
+ vision: true,
+ max_embeds: 10,
+ override_main: false,
+ rl_requests: 0,
+ rl_input: 0,
+ rl_output: 0,
+ kwargs: {},
+ },
utility: {
ctx_length: 128000,
ctx_input: 0.7,
@@ -143,15 +153,21 @@ export function configFromPreset(preset, baseConfig, stripApiKey = true) {
const config = clonePlain(baseConfig || {});
const slots = [
['chat', 'chat_model'],
+ ['vision', 'vision_model'],
['utility', 'utility_model'],
['embedding', 'embedding_model'],
];
for (const [slotKey, sectionKey] of slots) {
const slot = preset?.[slotKey];
- if (!slot || typeof slot !== 'object') continue;
- if (!hasModelIdentity(slot)) continue;
- config[sectionKey] = mergeModelSlot(config[sectionKey] || {}, slot, stripApiKey, slotKey);
+ if (slotKey === 'vision') config[sectionKey] = {};
+ if (!slot || typeof slot !== 'object' || !hasModelIdentity(slot)) continue;
+ config[sectionKey] = mergeModelSlot(
+ slotKey === 'vision' ? {} : (config[sectionKey] || {}),
+ slot,
+ stripApiKey,
+ slotKey,
+ );
}
return config;
@@ -206,8 +222,10 @@ export const store = createStore("modelConfig", {
const source = (rawPresets || []).filter(p => p && typeof p === 'object');
const rawDefault = source.find(p => String(p.name || '').toLowerCase() === 'default') || {};
const slot = value => ({ provider: '', name: '', api_key: '', api_base: '', kwargs: {}, ...(value || {}) });
+ const visionSlot = value => ({ ...slot(value), vision: true, max_embeds: Number(value?.max_embeds ?? 10), override_main: !!value?.override_main });
const defaultConfig = {
chat_model: slot(rawDefault.chat),
+ vision_model: hasModelIdentity(rawDefault.vision) ? visionSlot(rawDefault.vision) : {},
utility_model: slot(rawDefault.utility),
embedding_model: slot(rawDefault.embedding),
};
@@ -219,6 +237,7 @@ export const store = createStore("modelConfig", {
return {
name: p.name || '',
chat: { ...slot(effective.chat_model), _kwargs_text: kwargsToText(effective.chat_model?.kwargs) },
+ vision: { ...visionSlot(effective.vision_model), _kwargs_text: kwargsToText(effective.vision_model?.kwargs) },
utility: { ...slot(effective.utility_model), _kwargs_text: kwargsToText(effective.utility_model?.kwargs) },
embedding: { ...slot(effective.embedding_model), _kwargs_text: kwargsToText(effective.embedding_model?.kwargs) },
};
@@ -273,6 +292,7 @@ export const store = createStore("modelConfig", {
// Config field initialization (converts kwargs dicts to editable text)
initConfigFields(config) {
if (config?.chat_model) config.chat_model._kwargs_text = kwargsToText(config.chat_model.kwargs);
+ if (config?.vision_model) config.vision_model._kwargs_text = kwargsToText(config.vision_model.kwargs);
if (config?.utility_model) config.utility_model._kwargs_text = kwargsToText(config.utility_model.kwargs);
if (config?.embedding_model) config.embedding_model._kwargs_text = kwargsToText(config.embedding_model.kwargs);
},
@@ -340,6 +360,7 @@ export const store = createStore("modelConfig", {
|| this.presets[0]
|| {
chat: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
+ vision: { provider: '', name: '', api_base: '', vision: true, max_embeds: 10, override_main: false, kwargs: {}, _kwargs_text: '' },
utility: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
embedding: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
}
@@ -420,9 +441,10 @@ export const store = createStore("modelConfig", {
const clean = presets.map(p => {
const c = { name: p.name };
const isDefault = p._originalName === DEFAULT_PRESET_NAME;
- for (const slot of ['chat', 'utility']) {
+ for (const slot of ['chat', 'vision', 'utility']) {
if (p[slot]) {
const rest = cleanPresetSlot(p[slot], true, slot, isDefault);
+ if (slot === 'vision' && hasModelIdentity(rest)) rest.vision = true;
if (hasModelIdentity(rest)) c[slot] = rest;
}
}
@@ -608,9 +630,15 @@ export const store = createStore("modelConfig", {
const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';
return [
{ icon: 'chat', title: 'Main', cfg: preset?.chat, pList: chatP },
+ { icon: 'eye', title: 'Vision', cfg: preset?.vision, pList: chatP },
{ icon: 'manufacturing', title: 'Utility', cfg: preset?.utility, pList: chatP },
{ icon: 'database', title: 'Embedding', cfg: preset?.embedding, pList: embedP },
- ].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
+ ]
+ .filter(s => s.title !== 'Vision' || (
+ hasModelIdentity(s.cfg)
+ && (!preset?.chat?.vision || s.cfg?.override_main)
+ ))
+ .map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
},
getPreset(name) {
@@ -747,9 +775,15 @@ export const store = createStore("modelConfig", {
const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';
return [
{ icon: 'chat', title: 'Main', cfg: cfg.chat_model, pList: chatP },
+ { icon: 'eye', title: 'Vision', cfg: cfg.vision_model, pList: chatP },
{ icon: 'manufacturing', title: 'Utility', cfg: cfg.utility_model, pList: chatP },
{ icon: 'database', title: 'Embedding', cfg: cfg.embedding_model, pList: embedP },
- ].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
+ ]
+ .filter(s => s.title !== 'Vision' || (
+ hasModelIdentity(s.cfg)
+ && (!cfg.chat_model?.vision || s.cfg?.override_main)
+ ))
+ .map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
},
async refreshModelsSummary(contextId = '') {
diff --git a/plugins/_model_config/webui/model-field.html b/plugins/_model_config/webui/model-field.html
index e88b905f8..51c70741a 100644
--- a/plugins/_model_config/webui/model-field.html
+++ b/plugins/_model_config/webui/model-field.html
@@ -10,13 +10,14 @@
Reusable model configuration field set.
Parent x-data scope must provide:
model — reactive object with provider, name, api_key, api_base, ctx_length, ctx_history, ctx_input, vision, max_embeds, rl_requests, rl_input, rl_output, kwargs, _kwargs_text
- modelType — 'chat' | 'utility' | 'embedding'
+ modelType — 'chat' | 'vision' | 'utility' | 'embedding'
providers — array of { value, label }
searchType — 'chat' | 'embedding'
apiKeyMode — 'store' (key lives in $store.modelConfig.apiKeyValues), 'inline' (key lives in model.api_key), or 'none'
Optional:
providerFallback — fallback provider string for search/API key status (e.g. preset.chat.provider for utility slot)
apiBaseFallback — fallback api_base string for search (e.g. preset.chat.api_base for utility slot)
+ visionModel — vision slot object, required by the chat model's sidecar switch
-->
<div x-data="{ get _prov() { return model.provider || (typeof providerFallback !== 'undefined' ? providerFallback : ''); }, get _apiBase() { return model.api_base || (typeof apiBaseFallback !== 'undefined' ? apiBaseFallback : ''); } }">
<!-- Provider -->
@@ -146,8 +147,23 @@
</div>
</template>
+ <template x-if="modelType === 'chat' && model.vision">
+ <div class="field">
+ <div class="field-label">
+ <div class="field-title">Use separate Vision Model</div>
+ <div class="field-description">When disabled, vision_load uses this model's native vision.</div>
+ </div>
+ <div class="field-control">
+ <label class="toggle">
+ <input type="checkbox" x-model="visionModel.override_main" />
+ <span class="toggler"></span>
+ </label>
+ </div>
+ </div>
+ </template>
+
<!-- Context window size (main and utility only) -->
- <template x-if="modelType !== 'embedding'">
+ <template x-if="modelType === 'chat' || modelType === 'utility'">
<div class="field">
<div class="field-label">
<div class="field-title">Context window size</div>
@@ -207,6 +223,18 @@
</div>
</template>
+ <template x-if="modelType === 'vision'">
+ <div class="field">
+ <div class="field-label">
+ <div class="field-title">Max embeds</div>
+ <div class="field-description">Maximum number of images sent to the Vision Model in one call. Set to 0 for unlimited.</div>
+ </div>
+ <div class="field-control">
+ <input type="number" min="0" x-model.number="model.max_embeds" />
+ </div>
+ </div>
+ </template>
+
<!-- Utility-specific: ctx_input slider -->
<template x-if="modelType === 'utility'">
<div class="field">
diff --git a/plugins/_model_config/webui/preset-overview.html b/plugins/_model_config/webui/preset-overview.html
index a77467fa4..75ef0750d 100644
--- a/plugins/_model_config/webui/preset-overview.html
+++ b/plugins/_model_config/webui/preset-overview.html
@@ -17,10 +17,22 @@
<div class="model-preset-rows" aria-live="polite">
<template x-for="model in modelRows" :key="model.title">
- <div class="model-preset-row">
- <x-icon class="model-preset-icon" :name="model.icon"></x-icon>
- <span class="model-preset-role" x-text="model.title"></span>
- <span class="model-preset-identity">
+ <div class="model-preset-row"
+ :class="{ 'model-preset-row-nested': model.title === 'Vision' }">
+ <template x-if="model.title !== 'Vision'">
+ <x-icon class="model-preset-icon" :name="model.icon"></x-icon>
+ </template>
+ <template x-if="model.title !== 'Vision'">
+ <span class="model-preset-role" x-text="model.title"></span>
+ </template>
+ <span class="model-preset-identity"
+ :class="{ 'model-preset-identity-vision': model.title === 'Vision' }">
+ <template x-if="model.title === 'Vision'">
+ <span>
+ <span class="model-preset-role">Vision override</span>
+ <span class="model-preset-separator">/</span>
+ </span>
+ </template>
<span class="model-preset-provider" x-text="model.provider"></span>
<span class="model-preset-separator">/</span>
<span x-text="model.name"></span>
@@ -86,6 +98,16 @@
border-top: 1px solid var(--color-border);
}
+ .model-preset-row-nested {
+ border-top: 0 !important;
+ padding-top: var(--spacing-xs);
+ font-size: 0.76rem;
+ }
+
+ .model-preset-identity-vision {
+ grid-column: 3;
+ }
+
.model-preset-icon,
.model-preset-provider,
.model-preset-separator {
diff --git a/prompts/AGENTS.md b/prompts/AGENTS.md
index a3b9fb85e..3119674eb 100644
--- a/prompts/AGENTS.md
+++ b/prompts/AGENTS.md
@@ -25,6 +25,7 @@
- Read the rendering path before changing placeholders or filenames.
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
+- Keep native and sidecar vision prompts synchronized with `vision_load`: related images belong in one call, while a sidecar result is text-only unless Main explicitly requests its native raw route.
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
## Verification
diff --git a/prompts/agent.system.tools_vision_sidecar.md b/prompts/agent.system.tools_vision_sidecar.md
new file mode 100644
index 000000000..73a84ee23
--- /dev/null
+++ b/prompts/agent.system.tools_vision_sidecar.md
@@ -0,0 +1,48 @@
+## multimodal vision tools
+
+### vision_load
+analyze images with the separate Vision Model and return a text result
+args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction, `raw` optional boolean
+Input schema for tool_args:
+```json
+{
+ "type": "object",
+ "properties": {
+ "paths": {
+ "type": "array",
+ "items": {"type": "string"},
+ "description": "Absolute image paths or ephemeral image refs."
+ },
+ "query": {
+ "type": "string",
+ "description": "What the Vision Model should inspect, compare, locate, or read."
+ },
+ "raw": {
+ "type": "boolean",
+ "description": "Use the Main model's native vision instead, when Main supports vision."
+ }
+ },
+ "required": ["paths"],
+ "additionalProperties": false
+}
+```
+rules:
+- put all images needed for one comparison or visual task in the same `paths` array; they are sent in one Vision Model call
+- use a focused `query`; if omitted, the Vision Model returns a concise general description
+- the result is a text capsule; the Main model does not receive the raw images
+- use `raw=true` only when Main supports vision and must inspect the pixels itself
+- only bitmaps are supported
+example:
+```json
+{
+ "thoughts": [
+ "I need to compare both screenshots before answering."
+ ],
+ "headline": "Comparing screenshots",
+ "tool_name": "vision_load",
+ "tool_args": {
+ "paths": ["/path/to/before.png", "/path/to/after.png"],
+ "query": "Compare the error banners and describe what changed."
+ }
+}
+```
diff --git a/tests/test_browser_agent_regressions.py b/tests/test_browser_agent_regressions.py
index 7fba2ba48..a924ec00b 100644
--- a/tests/test_browser_agent_regressions.py
+++ b/tests/test_browser_agent_regressions.py
@@ -36,8 +36,13 @@ class _TestAgentContextType:
class _TestResponse(SimpleNamespace):
- def __init__(self, message="", break_loop=False, **kwargs):
- super().__init__(message=message, break_loop=break_loop, **kwargs)
+ def __init__(self, message="", break_loop=False, additional=None, **kwargs):
+ super().__init__(
+ message=message,
+ break_loop=break_loop,
+ additional=additional,
+ **kwargs,
+ )
class _TestTool:
@@ -96,6 +101,9 @@ _model_config_stub = ModuleType("plugins._model_config.helpers.model_config")
_model_config_stub.get_presets = lambda: []
_model_config_stub.get_preset_by_name = lambda name: None
_model_config_stub.get_chat_model_config = lambda agent=None: {}
+_model_config_stub.get_vision_model_config = lambda agent=None: {}
+_model_config_stub.use_vision_sidecar = lambda agent=None: False
+_model_config_stub.build_vision_model = lambda agent=None: None
sys.modules.setdefault("plugins._model_config.helpers.model_config", _model_config_stub)
@@ -4297,11 +4305,9 @@ async def test_vision_load_materializes_ephemeral_browser_refs(monkeypatch, tmp_
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
- monkeypatch.setattr(
- vision_load_module.plugins,
- "get_plugin_config",
- lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
- )
+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
tool_results = []
messages = []
diff --git a/tests/test_model_config_api_keys.py b/tests/test_model_config_api_keys.py
index eb779aba1..a52f5583a 100644
--- a/tests/test_model_config_api_keys.py
+++ b/tests/test_model_config_api_keys.py
@@ -149,7 +149,7 @@ def test_model_config_frontend_tracks_provider_api_key_edits():
assert "/plugins/_model_config/missing_api_key_status" not in model_gate_content
assert '@input="$store.modelConfig.setApiKeyValue(_prov, $el.value)"' in config_content
assert "apiKeyMode: 'none'" not in preset_modal_content
- assert preset_modal_content.count("apiKeyMode: 'store'") == 3
+ assert preset_modal_content.count("apiKeyMode: 'store'") == 4
assert "$store.modelConfig.resetApiKeyDrafts();" in preset_modal_content
assert "await $store.modelConfig.refreshApiKeyStatus();" in preset_modal_content
assert "await store.persistAllDirtyApiKeys();" in store_content
diff --git a/tests/test_model_config_project_presets.py b/tests/test_model_config_project_presets.py
index bb8c67045..1c972b43c 100644
--- a/tests/test_model_config_project_presets.py
+++ b/tests/test_model_config_project_presets.py
@@ -1092,6 +1092,44 @@ def test_preset_application_inherits_optional_slots(monkeypatch, tmp_path):
assert config["embedding_model"] == base_config["embedding_model"]
+def test_preset_vision_slot_is_optional_and_never_inherited(monkeypatch, tmp_path):
+ _prepare_a0_tree(monkeypatch, tmp_path)
+
+ from plugins._model_config.helpers import model_config
+
+ base_config = {
+ "chat_model": {"provider": "openrouter", "name": "main", "vision": False},
+ "vision_model": {
+ "provider": "openrouter",
+ "name": "default-vision",
+ "max_embeds": 2,
+ },
+ }
+
+ without_sidecar = model_config.build_config_from_preset(
+ {"name": "Text only", "chat": {"provider": "openrouter", "name": "text"}},
+ base_config,
+ )
+ with_sidecar = model_config.build_config_from_preset(
+ {
+ "name": "Visual",
+ "chat": {"provider": "openrouter", "name": "text"},
+ "vision": {
+ "provider": "anthropic",
+ "name": "visual",
+ "max_embeds": 5,
+ },
+ },
+ base_config,
+ )
+
+ assert without_sidecar["vision_model"] == {}
+ assert with_sidecar["vision_model"]["provider"] == "anthropic"
+ assert with_sidecar["vision_model"]["name"] == "visual"
+ assert with_sidecar["vision_model"]["max_embeds"] == 5
+ assert "default-vision" not in str(with_sidecar["vision_model"])
+
+
def test_legacy_utility_preset_defaults_preserve_tuning_but_clear_kwargs(
monkeypatch,
tmp_path,
diff --git a/tests/test_model_config_ui.py b/tests/test_model_config_ui.py
index 1feb61061..5c6d54aa8 100644
--- a/tests/test_model_config_ui.py
+++ b/tests/test_model_config_ui.py
@@ -85,6 +85,49 @@ def test_preset_editor_uses_standard_modal_footer_buttons() -> None:
assert "preset-editor-footer" not in preset_modal
+def test_preset_editor_nests_one_conditional_vision_selector_in_main() -> None:
+ preset_modal = read("plugins", "_model_config", "webui", "main.html")
+ model_field = read("plugins", "_model_config", "webui", "model-field.html")
+ preset_overview = read("plugins", "_model_config", "webui", "preset-overview.html")
+ preset_store = read("plugins", "_model_config", "webui", "model-config-store.js")
+
+ main_start = preset_modal.index('<div class="section-title">Main Model</div>')
+ utility_start = preset_modal.index('<div class="section-title">Utility Model</div>')
+ selector_start = preset_modal.index('class="vision-sidecar-selector"')
+ supports_start = model_field.index('<div class="field-title">Supports Vision</div>')
+ override_start = model_field.index('<div class="field-title">Use separate Vision Model</div>')
+ context_start = model_field.index('<div class="field-title">Context window size</div>')
+ advanced_start = model_field.index('<!-- Advanced Settings (collapsed by default) -->')
+
+ assert '<div class="section-title">Vision Model</div>' not in preset_modal
+ assert preset_modal.count('class="vision-sidecar-selector"') == 1
+ assert main_start < selector_start < utility_start
+ assert '!selectedPreset.chat.vision || selectedPreset.vision.override_main' in preset_modal
+ assert "get visionModel() { return selectedPreset.vision; }" in preset_modal
+ assert "modelType: 'vision'" in preset_modal
+ assert preset_modal.count("apiKeyMode: 'store'") == 4
+ assert "margin: 0.75rem 0 0;" in preset_modal
+ assert "padding: 0.25rem 0 0;" in preset_modal
+ assert "border-left: 2px solid var(--color-border);" not in preset_modal
+ assert "Use separate Vision Model" in model_field
+ assert supports_start < override_start < context_start < advanced_start
+ assert "When disabled, vision_load uses this model's native vision." in model_field
+ assert "When enabled, vision_load uses the preset's Vision Model" not in model_field
+ assert 'x-model="visionModel.override_main"' in model_field
+ assert '<div class="advanced-section" x-data="{ advOpen: false }">' in model_field
+ assert "model-preset-row-nested" in preset_overview
+ assert "model.title === 'Vision'" in preset_overview
+ assert preset_overview.count("model.title !== 'Vision'") == 2
+ assert '<span class="model-preset-role">Vision override</span>' in preset_overview
+ assert "'model-preset-identity-vision': model.title === 'Vision'" in preset_overview
+ assert "grid-column: 3;" in preset_overview
+ assert "padding-top: var(--spacing-xs);" in preset_overview
+ assert "margin-left: 2rem;" not in preset_overview
+ assert "border-left: 1px solid var(--color-border);" not in preset_overview
+ assert "if (slotKey === 'vision') config[sectionKey] = {};" in preset_store
+ assert "['chat', 'vision', 'utility']" in preset_store
+
+
def test_plugin_settings_reset_is_explicit_and_does_not_capture_toast_early() -> None:
settings_modal = read("webui", "components", "plugins", "plugin-settings.html")
settings_store = read("webui", "components", "plugins", "plugin-settings-store.js")
diff --git a/tests/test_parallel_tool.py b/tests/test_parallel_tool.py
index dc1a91199..575f21f51 100644
--- a/tests/test_parallel_tool.py
+++ b/tests/test_parallel_tool.py
@@ -160,6 +160,42 @@ def test_normalize_parallel_tool_calls_accepts_json_string_array() -> None:
assert calls[1].tool_args["message"] == "Research nuclear fusion news in Italian."
+def test_parallel_keeps_mixed_tools_and_batched_vision_paths() -> None:
+ calls = parallel_tools.normalize_parallel_tool_calls(
+ [
+ {"tool_name": "search_a", "tool_args": {"query": "one"}},
+ {"tool_name": "search_b", "tool_args": {"query": "two"}},
+ {"tool_name": "browser_agent", "tool_args": {"message": "open page"}},
+ {
+ "tool_name": "vision_load",
+ "tool_args": {
+ "paths": ["/tmp/before.png", "/tmp/after.png"],
+ "query": "compare",
+ },
+ },
+ ]
+ )
+
+ assert [call.tool_name for call in calls] == [
+ "search_a",
+ "search_b",
+ "browser_agent",
+ "vision_load",
+ ]
+ assert calls[3].tool_args["paths"] == ["/tmp/before.png", "/tmp/after.png"]
+
+
+def test_parallel_allows_multiple_independent_vision_calls() -> None:
+ calls = parallel_tools.normalize_parallel_tool_calls(
+ [
+ {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/a.png"]}},
+ {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/b.png"]}},
+ ]
+ )
+
+ assert [call.tool_name for call in calls] == ["vision_load", "vision_load"]
+
+
def test_subordinate_prompts_share_reusable_tree_contract() -> None:
call_prompt = (PROJECT_ROOT / "prompts/agent.system.tool.call_sub.md").read_text(
encoding="utf-8"
diff --git a/tests/test_tool_policy.py b/tests/test_tool_policy.py
index 2db20350c..3d3447325 100644
--- a/tests/test_tool_policy.py
+++ b/tests/test_tool_policy.py
@@ -546,6 +546,59 @@ async def test_vision_tool_follows_chat_config_not_profile_policy(
assert tool_policy.resolve_tool(agent, "vision_load").source == "runtime-config"
+async def test_vision_sidecar_prompt_replaces_native_vision_prompt(
+ monkeypatch, tmp_path: Path
+) -> None:
+ _write_prompt(tmp_path, "agent.system.tools.md", "TOOLS\n{{tools}}")
+ _write_prompt(
+ tmp_path,
+ "agent.system.tools_vision.md",
+ "### vision_load\nnative pixels\nargs: `paths`",
+ )
+ _write_prompt(
+ tmp_path,
+ "agent.system.tools_vision_sidecar.md",
+ "### vision_load\nsidecar capsule\nargs: `paths`, `query`",
+ )
+ monkeypatch.setattr(tool_policy.subagents, "get_paths", _prompt_paths(tmp_path))
+ monkeypatch.setattr(
+ "plugins._model_config.helpers.model_config.get_chat_model_config",
+ lambda agent: {"vision": True},
+ )
+ monkeypatch.setattr(
+ "plugins._model_config.helpers.model_config.use_vision_sidecar",
+ lambda agent: True,
+ )
+ monkeypatch.setattr(responses_tools, "_mcp_tools", lambda agent: [])
+ agent = _Agent(tmp_path)
+
+ prompt = await _11_tools_prompt.build_prompt(agent)
+ schemas, _name_map = responses_tools.build_responses_function_tools(agent)
+
+ assert "sidecar capsule" in prompt
+ assert "native pixels" not in prompt
+ assert schemas[0]["name"] == "vision_load"
+ assert schemas[0]["description"] == "sidecar capsule"
+
+
+def test_vision_sidecar_prompt_declares_multi_image_native_schema() -> None:
+ prompt = (
+ Path(__file__).resolve().parents[1]
+ / "prompts"
+ / "agent.system.tools_vision_sidecar.md"
+ ).read_text(encoding="utf-8")
+ schema = responses_tools._schema_from_prompt(prompt)
+
+ assert schema["required"] == ["paths"]
+ assert schema["properties"]["paths"] == {
+ "type": "array",
+ "items": {"type": "string"},
+ "description": "Absolute image paths or ephemeral image refs.",
+ }
+ assert {"query", "raw"} <= schema["properties"].keys()
+
+
def test_mcp_prompt_and_native_schema_omit_blocked_tool(
monkeypatch, tmp_path: Path
) -> None:
diff --git a/tests/test_vision_load_image_refs.py b/tests/test_vision_load_image_refs.py
index 2842a5e1e..619901aa0 100644
--- a/tests/test_vision_load_image_refs.py
+++ b/tests/test_vision_load_image_refs.py
@@ -1,3 +1,4 @@
+import asyncio
import types
from types import SimpleNamespace
import sys
@@ -13,8 +14,13 @@ from helpers import images
class _TestResponse(SimpleNamespace):
- def __init__(self, message="", break_loop=False, **kwargs):
- super().__init__(message=message, break_loop=break_loop, **kwargs)
+ def __init__(self, message="", break_loop=False, additional=None, **kwargs):
+ super().__init__(
+ message=message,
+ break_loop=break_loop,
+ additional=additional,
+ **kwargs,
+ )
class _TestTool:
@@ -74,11 +80,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
- monkeypatch.setattr(
- vision_load_module.plugins,
- "get_plugin_config",
- lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
- )
+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
async def direct_call(func, *args, **kwargs):
return func(*args, **kwargs)
@@ -121,3 +125,215 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
stored_path = tmp_path / stored_ref.removeprefix("/a0/")
assert stored_path.read_bytes() == b"png-data"
assert updates[-1]["result"] == "1 images loaded, 0 skipped"
+
+
+def test_vision_sidecar_route_matrix_prefers_main_native_vision(monkeypatch):
+ from plugins._model_config.helpers import model_config
+
+ cases = [
+ ({"vision": False}, {}, False),
+ ({"vision": True}, {"provider": "p", "name": "v"}, False),
+ ({"vision": False}, {"provider": "p", "name": "v"}, True),
+ (
+ {"vision": True},
+ {"provider": "p", "name": "v", "override_main": True},
+ True,
+ ),
+ ]
+ for chat, vision, expected in cases:
+ monkeypatch.setattr(
+ model_config,
+ "get_effective_config",
+ lambda _agent=None, chat=chat, vision=vision: {
+ "chat_model": chat,
+ "vision_model": vision,
+ },
+ )
+ assert model_config.use_vision_sidecar() is expected
+
+
+async def test_vision_sidecar_sends_multiple_images_once_and_keeps_history_text_only(
+ monkeypatch,
+ tmp_path,
+):
+ _install_tool_stub(monkeypatch)
+ import tools.vision_load as vision_load_module
+
+ async def direct_call(func, *args, **kwargs):
+ return func(*args, **kwargs)
+
+ calls = []
+
+ class FakeVisionModel:
+ async def unified_call(self, **kwargs):
+ calls.append(kwargs)
+ return "The second screenshot fixes the red login error.", ""
+
+ monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
+ monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_chat_model_config",
+ lambda _agent: {"vision": True, "max_embeds": 1},
+ )
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_vision_model_config",
+ lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 5},
+ )
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
+
+ image_paths = [tmp_path / "before.png", tmp_path / "after.png"]
+ for path in image_paths:
+ path.write_bytes(b"png-data")
+
+ tool_results = []
+ raw_messages = []
+ agent = SimpleNamespace(
+ context=SimpleNamespace(id=""),
+ agent_name="Agent 0",
+ hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),
+ hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),
+ )
+ tool = vision_load_module.VisionLoad(
+ agent=agent,
+ name="vision_load",
+ method=None,
+ args={"paths": [str(path) for path in image_paths]},
+ message="",
+ loop_data=None,
+ )
+ tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)
+
+ response = await tool.execute(
+ paths=[str(path) for path in image_paths],
+ query="Compare the login errors.",
+ )
+ response.additional = {"_responses_output_item": {"output": response.message}}
+ await tool.after_execution(response)
+
+ assert len(calls) == 1
+ content = calls[0]["messages"][1].content
+ assert content[0] == {"type": "text", "text": "Compare the login errors."}
+ assert [item["type"] for item in content].count("image_url") == 2
+ assert "fixes the red login error" in response.message
+ assert response.message != "dummy"
+ assert raw_messages == []
+ assert tool.loaded_paths == [str(path) for path in image_paths]
+ assert tool_results[0][1]["_responses_output_item"]["output"] == response.message
+
+
+async def test_parallel_worker_consumes_parent_ephemeral_image(monkeypatch, tmp_path):
+ _install_tool_stub(monkeypatch)
+ import tools.vision_load as vision_load_module
+
+ def fake_get_abs_path(*parts):
+ return str(tmp_path.joinpath(*parts))
+
+ def fake_normalize_a0_path(path):
+ return "/a0/" + str(Path(path).relative_to(tmp_path)).replace("\\", "/")
+
+ monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
+ monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
+ monkeypatch.setattr(vision_load_module.VisionLoad, "_config_agent", lambda self: self.agent)
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_chat_model_config",
+ lambda _agent: {"vision": True, "max_embeds": 10},
+ )
+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
+
+ parent_id = "parent-vision"
+ ref = vision_load_module.ephemeral_images.put_image_bytes(
+ context_id=parent_id,
+ mime="image/png",
+ payload=b"png-data",
+ name="shot.png",
+ )
+ context = SimpleNamespace(
+ id="parallel-worker",
+ get_data=lambda key: parent_id
+ if key == vision_load_module.PARALLEL_WORKER_PARENT_CONTEXT_KEY
+ else None,
+ )
+ agent = SimpleNamespace(context=context, agent_name="Agent 0")
+ tool = vision_load_module.VisionLoad(
+ agent=agent,
+ name="vision_load",
+ method=None,
+ args={"paths": [ref]},
+ message="",
+ loop_data=None,
+ )
+
+ await tool.execute(paths=[ref])
+
+ assert tool.loaded_paths == ["shot.png"]
+ assert vision_load_module.ephemeral_images.get_image(ref, context_id=parent_id) is None
+ stored_ref = tool.images_dict["shot.png"]
+ assert stored_ref.startswith("/a0/usr/chats/parent-vision/images/vision-load/shot-")
+
+
+async def test_independent_vision_sidecar_calls_can_run_concurrently(monkeypatch, tmp_path):
+ _install_tool_stub(monkeypatch)
+ import tools.vision_load as vision_load_module
+
+ active = 0
+ max_active = 0
+ call_count = 0
+
+ class FakeVisionModel:
+ async def unified_call(self, **kwargs):
+ nonlocal active, max_active, call_count
+ active += 1
+ call_count += 1
+ max_active = max(max_active, active)
+ await asyncio.sleep(0.02)
+ active -= 1
+ return "done", ""
+
+ async def direct_call(func, *args, **kwargs):
+ return func(*args, **kwargs)
+
+ monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
+ monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": False})
+ monkeypatch.setattr(
+ vision_load_module,
+ "get_vision_model_config",
+ lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 10},
+ )
+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
+
+ image_paths = [tmp_path / "one.png", tmp_path / "two.png"]
+ for path in image_paths:
+ path.write_bytes(b"png-data")
+
+ def make_tool(index):
+ agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")
+ return vision_load_module.VisionLoad(
+ agent=agent,
+ name="vision_load",
+ method=None,
+ args={"paths": [str(path) for path in image_paths]},
+ message="",
+ loop_data=None,
+ )
+
+ responses = await asyncio.gather(
+ *(
+ make_tool(index).execute(
+ paths=[str(path) for path in image_paths],
+ query=f"inspection {index}",
+ )
+ for index in range(4)
+ )
+ )
+
+ assert call_count == 4
+ assert max_active == 4
+ assert all("done" in response.message for response in responses)
diff --git a/tools/vision_load.py b/tools/vision_load.py
index 7358afbde..b29ff931f 100644
--- a/tools/vision_load.py
+++ b/tools/vision_load.py
@@ -1,89 +1,219 @@
-from helpers.print_style import PrintStyle
-from helpers.tool import Tool, Response
-from helpers import runtime, files, plugins, ephemeral_images, images, chat_media
+import asyncio
+import json
from mimetypes import guess_type
-from helpers import history
-# image token estimation for context window
+from langchain_core.messages import HumanMessage, SystemMessage
+
+from helpers import chat_media, ephemeral_images, files, history, images, runtime
+from helpers.parallel_tools import PARALLEL_WORKER_PARENT_CONTEXT_KEY, coerce_bool
+from helpers.print_style import PrintStyle
+from helpers.tool import Response, Tool
+from plugins._model_config.helpers.model_config import (
+ build_vision_model,
+ get_chat_model_config,
+ get_vision_model_config,
+ use_vision_sidecar,
+)
+
TOKENS_ESTIMATE = 1500
+VISION_TIMEOUT_SECONDS = 300
+VISION_SYSTEM_PROMPT = (
+ "You are a precise vision analyst. Answer only what was asked about the images. "
+ "Be concise and factual. Preserve exact visible text when asked to read it."
+)
+DEFAULT_VISION_QUERY = (
+ "Describe the images precisely, including the key objects, visible text, and layout."
+)
class VisionLoad(Tool):
- async def execute(self, paths: list[str] = [], **kwargs) -> Response:
-
- self.images_dict = {}
+ async def execute(
+ self,
+ paths: list[str] | str | None = None,
+ query: str = "",
+ raw: bool = False,
+ **kwargs,
+ ) -> Response:
+ self.images_dict: dict[str, str] = {}
self.loaded_paths: list[str] = []
self.skipped_paths: list[str] = []
+ self._config_owner = self._config_agent()
+ self._main_has_vision = bool(
+ get_chat_model_config(self._config_owner).get("vision", False)
+ )
+ self._delegated = use_vision_sidecar(self._config_owner) and not (
+ coerce_bool(raw, False) and self._main_has_vision
+ )
+ self._max_embeds = self._get_max_embeds()
+
+ normalized = self._normalize_paths(paths)
+ if isinstance(normalized, str):
+ self._history_result = normalized
+ return Response(message=normalized, break_loop=False)
- max_embeds = self._get_max_embeds()
requested = [
- (str(path or "").strip(), self._display_input_path(str(path or "").strip(), idx + 1))
- for idx, path in enumerate(paths)
+ (path.strip(), self._display_input_path(path.strip(), index + 1))
+ for index, path in enumerate(normalized)
]
- limited_paths = requested if max_embeds <= 0 else requested[-max_embeds:]
- self.skipped_paths = (
- [display for _, display in requested[:-max_embeds]]
- if max_embeds > 0 and len(requested) > max_embeds
- else []
- )
+ limited = requested if self._max_embeds <= 0 else requested[-self._max_embeds :]
+ if self._max_embeds > 0 and len(requested) > self._max_embeds:
+ self.skipped_paths = [display for _, display in requested[: -self._max_embeds]]
- for idx, (path, display_path) in enumerate(limited_paths):
+ for index, (path, display_path) in enumerate(limited):
if not path:
continue
if ephemeral_images.is_ref(path):
- image = ephemeral_images.consume_image(
- path,
- context_id=self._context_id(),
- )
+ image = ephemeral_images.consume_image(path, context_id=self._context_id())
if image is None:
continue
- display = image.display_name or display_path
+ display_path = image.display_name or display_path
stored_ref = self._store_ephemeral_image(image)
- if stored_ref:
- self.images_dict[display] = stored_ref
- self.loaded_paths.append(display)
- continue
- if self._is_data_image_url(path):
- stored_ref = self._store_data_url(path, preferred_name=f"vision-load-{idx + 1}.png")
- if stored_ref:
- self.images_dict[display_path] = stored_ref
- self.loaded_paths.append(display_path)
- continue
- if not await runtime.call_development_function(files.exists, str(path)):
+ elif self._is_data_image_url(path):
+ stored_ref = self._store_data_url(
+ path, preferred_name=f"vision-load-{index + 1}.png"
+ )
+ elif await runtime.call_development_function(files.exists, path):
+ mime_type, _ = guess_type(path)
+ if not mime_type or not mime_type.startswith("image/"):
+ continue
+ try:
+ stored_ref = self._store_local_image(
+ path, preferred_name=files.basename(path)
+ )
+ except (FileNotFoundError, OSError, ValueError):
+ continue
+ else:
continue
- if path not in self.images_dict:
- mime_type, _ = guess_type(str(path))
- if mime_type and mime_type.startswith("image/"):
- try:
- stored_ref = self._store_local_image(path, preferred_name=files.basename(path))
- self.images_dict[display_path] = stored_ref
- self.loaded_paths.append(display_path)
- except (FileNotFoundError, OSError, ValueError):
- continue
+ if stored_ref:
+ self.images_dict[display_path] = stored_ref
+ self.loaded_paths.append(display_path)
- return Response(message="dummy", break_loop=False)
+ summary = self._summary()
+ if self._delegated and self.images_dict:
+ try:
+ capsule = await self._call_vision_model(
+ list(self.images_dict.values()), self._query(query, kwargs)
+ )
+ message = (
+ f"Vision Model analyzed {len(self.images_dict)} image(s)"
+ f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"
+ )
+ self._history_result = message
+ return Response(message=message, break_loop=False)
+ except Exception as exc:
+ message = f"Vision Model error: {str(exc)[:1000]}"
+ self._history_result = f"{summary}\n\n{message}"
+ return Response(message=message, break_loop=False)
+
+ if self.images_dict and not self._main_has_vision:
+ summary += (
+ "\n\nImages were not injected because neither Main native vision nor "
+ "a usable Vision Model is active."
+ )
+ self._history_result = (
+ summary if self.images_dict or self.skipped_paths else "No images processed"
+ )
+ message = (
+ "No images processed"
+ if not self.images_dict and not self.skipped_paths
+ else f"{len(self.images_dict)} images loaded, {len(self.skipped_paths)} skipped"
+ )
+ return Response(message=message, break_loop=False)
+
+ async def after_execution(self, response: Response, **kwargs):
+ log_id = str(getattr(getattr(self, "log", None), "id", "") or "")
+ self.agent.hist_add_tool_result(
+ self.name,
+ self._history_result,
+ id=log_id,
+ **(response.additional or {}),
+ )
+
+ if self.images_dict and self._main_has_vision and not self._delegated:
+ content = [
+ {"type": "image_url", "image_url": {"url": image_path}}
+ for image_path in self.images_dict.values()
+ ]
+ self.agent.hist_add_message(
+ False,
+ content=history.RawMessage(
+ raw_content=content,
+ preview="<Image attachments loaded by path>",
+ ),
+ tokens=TOKENS_ESTIMATE * len(content),
+ )
+
+ PrintStyle(
+ font_color="#1B4F72", background_color="white", padding=True, bold=True
+ ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
+ PrintStyle(font_color="#85C1E9").print(response.message)
+ if getattr(self, "log", None):
+ self.log.update(result=response.message)
def _get_max_embeds(self) -> int:
- cfg = plugins.get_plugin_config("_model_config", agent=self.agent) or {}
- chat_cfg = cfg.get("chat_model", {})
- max_embeds = chat_cfg.get("max_embeds", 10)
- return int(max_embeds or 0)
+ cfg = (
+ get_vision_model_config(self._config_owner)
+ if self._delegated
+ else get_chat_model_config(self._config_owner)
+ )
+ try:
+ return int(cfg.get("max_embeds", 10) or 0)
+ except (TypeError, ValueError):
+ return 10
def _context_id(self) -> str:
- return str(getattr(getattr(self.agent, "context", None), "id", "") or "").strip()
+ context = getattr(self.agent, "context", None)
+ if not context:
+ return ""
+ get_data = getattr(context, "get_data", None)
+ parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
+ return str(parent_id or getattr(context, "id", "") or "").strip()
+
+ def _config_agent(self):
+ context = getattr(self.agent, "context", None)
+ get_data = getattr(context, "get_data", None)
+ parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
+ if parent_id:
+ from agent import AgentContext
+
+ parent = AgentContext.get(str(parent_id))
+ if parent:
+ return parent.agent0
+ return self.agent
+
+ async def _call_vision_model(self, image_paths: list[str], query: str) -> str:
+ model = build_vision_model(self._config_owner)
+ content = [{"type": "text", "text": query or DEFAULT_VISION_QUERY}]
+ content.extend(
+ {"type": "image_url", "image_url": {"url": path}}
+ for path in image_paths
+ )
+ response, _ = await asyncio.wait_for(
+ model.unified_call(
+ messages=[
+ SystemMessage(content=VISION_SYSTEM_PROMPT),
+ HumanMessage(content=content),
+ ],
+ explicit_caching=False,
+ max_tokens=2000,
+ ),
+ timeout=VISION_TIMEOUT_SECONDS,
+ )
+ if not str(response or "").strip():
+ raise RuntimeError("Vision Model returned an empty response.")
+ return str(response)
def _store_ephemeral_image(self, image: ephemeral_images.EphemeralImage) -> str:
context_id = self._context_id()
if not context_id:
return image.data_url
source = chat_media.infer_source(image.ref, image.display_name)
- category = chat_media.category_for_source(source)
saved = chat_media.save_image_base64(
context_id=context_id,
data=image.data,
mime_type=image.mime,
- category=category,
+ category=chat_media.category_for_source(source),
source=source,
preferred_name=image.display_name,
)
@@ -94,11 +224,10 @@ class VisionLoad(Tool):
if not context_id:
return data_url
source = chat_media.infer_source(data_url, preferred_name)
- category = chat_media.category_for_source(source)
saved = chat_media.save_image_data_url(
context_id=context_id,
data_url=data_url,
- category=category,
+ category=chat_media.category_for_source(source),
source=source,
preferred_name=preferred_name,
)
@@ -115,6 +244,37 @@ class VisionLoad(Tool):
preferred_name=preferred_name,
)
+ def _summary(self) -> str:
+ loaded = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
+ skipped = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
+ return (
+ f"Loaded images ({len(self.loaded_paths)}):\n{loaded}\n\n"
+ f"Skipped images ({len(self.skipped_paths)}, max {self._max_embeds}):\n{skipped}"
+ )
+
+ @staticmethod
+ def _normalize_paths(paths: list[str] | str | None) -> list[str] | str:
+ if isinstance(paths, str):
+ try:
+ decoded = json.loads(paths)
+ except json.JSONDecodeError:
+ decoded = paths
+ paths = decoded if isinstance(decoded, list) else [paths]
+ if paths is None:
+ return []
+ if not isinstance(paths, (list, tuple)):
+ return "vision_load error: `paths` must be an array of image paths."
+ return [str(path or "").strip() for path in paths]
+
+ @staticmethod
+ def _query(query: str, kwargs: dict) -> str:
+ if str(query or "").strip():
+ return str(query).strip()
+ for key in ("prompt", "question", "instruction", "focus", "request"):
+ if str(kwargs.get(key) or "").strip():
+ return str(kwargs[key]).strip()
+ return DEFAULT_VISION_QUERY
+
@staticmethod
def _is_data_image_url(value: str) -> bool:
normalized = str(value or "").strip().lower()
@@ -125,57 +285,5 @@ class VisionLoad(Tool):
if ephemeral_images.is_ref(value):
return ephemeral_images.display_ref(value)
if cls._is_data_image_url(value):
- prefix = value.split(",", 1)[0]
- return f"{prefix},<ephemeral-image-{index}>"
+ return f"{value.split(',', 1)[0]},<ephemeral-image-{index}>"
return value
-
- async def after_execution(self, response: Response, **kwargs):
-
- # build image data messages for LLMs, or error message
- content = []
- loaded_count = len(self.loaded_paths)
- skipped_count = len(self.skipped_paths)
- loaded_summary = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
- skipped_summary = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
- summary = (
- f"Loaded images: {loaded_count}\n"
- f"Loaded images:\n{loaded_summary}\n\n"
- f"Skipped images: {skipped_count}\n"
- f"Skipped images (max {self._get_max_embeds()} loaded at a time according to model configuration):\n{skipped_summary}"
- )
- if self.images_dict:
- self.agent.hist_add_tool_result(self.name, summary, id=self.log.id if self.log else "")
- for path, image_path in self.images_dict.items():
- if image_path:
- content.append(
- {
- "type": "image_url",
- "image_url": {"url": image_path},
- }
- )
- else:
- content.append(
- {
- "type": "text",
- "text": "Error processing image " + path,
- }
- )
- # append as raw message content for LLMs with vision tokens estimate
- msg = history.RawMessage(raw_content=content, preview="<Image attachments loaded by path>")
- self.agent.hist_add_message(
- False, content=msg, tokens=TOKENS_ESTIMATE * len(content)
- )
- else:
- self.agent.hist_add_tool_result(self.name, summary if self.skipped_paths else "No images processed", id=self.log.id if self.log else "")
-
- # print and log short version
- message = (
- "No images processed"
- if not self.images_dict and not self.skipped_paths
- else f"{loaded_count} images loaded, {skipped_count} skipped"
- )
- PrintStyle(
- font_color="#1B4F72", background_color="white", padding=True, bold=True
- ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
- PrintStyle(font_color="#85C1E9").print(message)
- self.log.update(result=message)
diff --git a/tools/vision_load.py.dox.md b/tools/vision_load.py.dox.md
index da9531242..2269eab56 100644
--- a/tools/vision_load.py.dox.md
+++ b/tools/vision_load.py.dox.md
@@ -3,7 +3,7 @@
## Purpose
- Own the `vision_load.py` agent tool.
-- This module loads images into model-visible content for vision-capable models.
+- This module routes images either into Main model-visible content or through the preset's optional Vision Model.
- Keep this file-level DOX profile synchronized with `vision_load.py` because this directory is intentionally flat.
## Ownership
@@ -12,13 +12,19 @@
- `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
- Classes:
- `VisionLoad` (`Tool`)
- - `async execute(self, paths: list[str]=..., **kwargs) -> Response`
+ - `async execute(self, paths, query="", raw=False, **kwargs) -> Response`
- `async after_execution(self, response: Response, **kwargs)`
- Notable constants/configuration names: `TOKENS_ESTIMATE`.
## Runtime Contracts
- Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.
+- One call may contain multiple paths. The delegated route sends every selected path in one Vision Model request and returns one textual capsule.
+- Main native vision wins unless the effective preset selects the sidecar route; `raw=true` returns to Main native vision only when Main supports it.
+- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.
+- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.
+- Direct parallel workers resolve ephemeral refs, model routing, and durable chat-media storage against their recorded parent context; independent vision jobs remain generic parallel jobs.
+- `max_embeds` comes from the model that actually receives the images.
- Update this file whenever tool arguments, output shape, `break_loop` behavior, intervention handling, prompt instructions, or side effects change.
- `VisionLoad` is a `Tool`.
- `VisionLoad` defines `execute(...)`.
@@ -27,7 +33,7 @@
## Key Concepts
-- Important called helpers/classes observed in the source: `self._get_max_embeds`, `Response`, `str.strip`, `self._context_id`, `chat_media.infer_source`, `chat_media.category_for_source`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `str.strip.lower`, `ephemeral_images.is_ref`, `cls._is_data_image_url`, `self._is_data_image_url`, `plugins.get_plugin_config`, `images.to_data_url`, `normalized.startswith`, `ephemeral_images.display_ref`, `join`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
+- Important called helpers/classes observed in the source: `build_vision_model`, `use_vision_sidecar`, `self._get_max_embeds`, `Response`, `self._context_id`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `ephemeral_images.consume_image`, `images.to_data_url`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
- Keep request/response, tool, or helper semantics documented here at the same time as source changes.
## Work Guidance