Agent Zero AI framework
Integrate vision sidecar routing with model presets
Alessandro59f5ae03ea61
diff --git a/extensions/python/system_prompt/_11_tools_prompt.py b/extensions/python/system_prompt/_11_tools_prompt.pyindex 6349a862c..a395d6c3b 100644--- a/extensions/python/system_prompt/_11_tools_prompt.py+++ b/extensions/python/system_prompt/_11_tools_prompt.py@@ -51,10 +51,15 @@ async def build_prompt(agent: Agent) -> str:prompt = agent.read_prompt("agent.system.tools.md", tools=tools_str)# vision support- from plugins._model_config.helpers.model_config import get_chat_model_config+ from plugins._model_config.helpers.model_config import (+ get_chat_model_config,+ use_vision_sidecar,+ )chat_cfg = get_chat_model_config(agent)- if chat_cfg.get("vision", False):+ if use_vision_sidecar(agent):+ prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision_sidecar.md")+ elif chat_cfg.get("vision", False):prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision.md")return promptdiff --git a/helpers/responses_tools.py b/helpers/responses_tools.pyindex 93778d652..877f03d79 100644--- a/helpers/responses_tools.py+++ b/helpers/responses_tools.py@@ -114,8 +114,13 @@ def _local_tool_prompts(agent: Any) -> list[tuple[str, str]]:def _vision_tool_prompt(agent: Any) -> str:try:- from plugins._model_config.helpers.model_config import get_chat_model_config+ from plugins._model_config.helpers.model_config import (+ get_chat_model_config,+ use_vision_sidecar,+ )+ if use_vision_sidecar(agent):+ return agent.read_prompt("agent.system.tools_vision_sidecar.md")if not get_chat_model_config(agent).get("vision", False):return ""return agent.read_prompt("agent.system.tools_vision.md")diff --git a/helpers/responses_tools.py.dox.md b/helpers/responses_tools.py.dox.mdindex 7c2631867..041919146 100644--- a/helpers/responses_tools.py.dox.md+++ b/helpers/responses_tools.py.dox.md@@ -12,7 +12,7 @@## Local Contracts-- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` only when the active chat model enables the matching vision prompt.+- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` when either Main native vision or the effective preset's Vision Model enables the matching prompt.- Discover local prompt files through `helpers.subagents.get_paths`; this moduleowns the Responses-specific prompt-name compatibility rules.- Local prompt-derived function names use existing bullet declarations that pair a backticked name with `arg` or `args` for multi-tool prompt files, otherwise prefer explicit `"tool_name"` examples, then the first prompt heading, and finally the prompt filename.@@ -25,8 +25,8 @@- Preserve original Agent Zero tool names through the native Responses name map.- Keep MCP tool schemas merged after local prompt-derived tools.- Apply `helpers.tool_policy` before emitting local or MCP schemas; a blocked- capability is absent from provider-native tool definitions. Vision remains- controlled solely by the active chat model configuration.+ capability is absent from provider-native tool definitions. Vision routing+ is controlled by the effective model preset rather than Agent Editor.- Connector remote tools are advertised only when `_a0_connector` runtime metadata says the matching connected CLI capability is currently available.## Work Guidancediff --git a/plugins/_model_config/AGENTS.md b/plugins/_model_config/AGENTS.mdindex f3ba7d529..28ec60ee9 100644--- a/plugins/_model_config/AGENTS.md+++ b/plugins/_model_config/AGENTS.md@@ -14,7 +14,7 @@## Local Contracts-- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline.+- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline; its Vision Model slot is optional.- Preset definitions are global. Global, project, agent-profile, and project/profile plugin configs persist only `model_preset`; chats may persist a preset reference as their explicit override.- Preserve scoped plugin resolution order and fall back invalid or missing scope/chat references to `Default`.- Project Settings `llm` payloads are owned here through the generic `helpers.projects` project extension-data hooks; keep project helper code agnostic to `_model_config` paths, presets, and inheritance rules.@@ -22,7 +22,12 @@- Check API-key readiness only for the effective model configuration; unused global presets must not produce Welcome-screen warnings.- Coordinate OAuth-backed providers with `_oauth` instead of hardcoding provider-specific auth here.- `model_config_get` exposes `model_configured` as a derived chat-model readiness flag from provider, model name, and API-key availability.-- Non-default presets may inherit omitted slots or durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.+- Non-default presets may inherit omitted main, utility, or embedding slots and durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.+- The optional `vision` slot is strictly per preset and never inherited from `Default`; an empty slot disables sidecar vision for that preset.+- Main native vision wins by default. A configured Vision Model handles `vision_load` when Main lacks vision, or when that preset explicitly enables `override_main`.+- Keep the optional Vision provider/model selector inside the Main Model card and flush with Main's field alignment, without a nested left inset. Show it only while Main vision is disabled or `override_main` is enabled; do not render a standalone Vision Model card.+- Show `Use separate Vision Model` immediately below `Supports Vision` while Main vision is enabled, not inside Advanced Settings; describe the disabled state as using Main's native vision.+- In model overviews, render the effective Vision Model as a text-only `Vision override / Provider / Model` child aligned with Main's provider column, not as an icon-bearing peer row.- Changing a model provider in the settings UI must clear `api_base` and `kwargs` because both may be provider-specific.- Repair provider-specific model-config aliases at the model-config read/build boundary; keep provider-specific repairs out of provider-agnostic core wrappers such as `models.py`.- `modelConfig.createPresetEditor()` owns local preset drafts, row actions, and stable UI-only row keys so deletion or renaming cannot rebind nested model fields.diff --git a/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html b/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.htmlindex 460bdfe72..0940d547f 100644--- a/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html+++ b/plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html@@ -65,6 +65,16 @@</span></div></template>+ <template x-if="preset.vision?.name && (!preset.chat?.vision || preset.vision?.override_main)">+ <div class="model-switcher-model-row">+ <span class="model-switcher-model-label">Vision</span>+ <span class="model-switcher-model-value">+ <span x-text="preset.vision.provider" style="opacity:0.5;"></span>+ <span style="opacity:0.3; margin:0 3px;">/</span>+ <span x-text="preset.vision.name"></span>+ </span>+ </div>+ </template><template x-if="preset.utility?.name"><div class="model-switcher-model-row"><span class="model-switcher-model-label">Utility</span>diff --git a/plugins/_model_config/helpers/model_config.py b/plugins/_model_config/helpers/model_config.pyindex 6a1386451..3f6c36f93 100644--- a/plugins/_model_config/helpers/model_config.py+++ b/plugins/_model_config/helpers/model_config.py@@ -16,11 +16,21 @@ PRESET_SCOPE_GLOBAL = "global"PRESET_SCOPE_PROJECT = "project"PRESET_SLOT_CONFIG_SECTIONS = {"chat": "chat_model",+ "vision": "vision_model","utility": "utility_model","embedding": "embedding_model",}MODEL_SLOT_PRESET_REPLACE_FIELDS = {"kwargs"}IMPLICIT_PRESET_SLOT_DEFAULTS = {+ "vision": {+ "vision": True,+ "max_embeds": 10,+ "override_main": False,+ "rl_requests": 0,+ "rl_input": 0,+ "rl_output": 0,+ "kwargs": {},+ },"utility": {"ctx_length": 128000,"ctx_input": 0.7,@@ -267,6 +277,9 @@ def _clean_preset_for_file(preset: dict) -> dict:slot_config = preset.get(slot)if isinstance(slot_config, dict):slot_clean = _strip_ui_fields(slot_config, strip_api_key=True)+ if slot == "vision" and _slot_has_identity(slot_clean):+ slot_clean["vision"] = True+ slot_clean.setdefault("max_embeds", 10)cleaned[slot] = (slot_cleanif name == DEFAULT_PRESET_NAME@@ -339,7 +352,12 @@ def validate_presets(presets: list, *, require_default: bool = True) -> list:def normalize_config_for_save(config: dict) -> dict:"""Remove UI-only fields and inline API keys before storing scoped config."""cleaned = deepcopy(config or {})- for section_name in ("chat_model", "utility_model", "embedding_model"):+ for section_name in (+ "chat_model",+ "vision_model",+ "utility_model",+ "embedding_model",+ ):section = cleaned.get(section_name)if isinstance(section, dict):cleaned[section_name] = _strip_ui_fields(section, strip_api_key=True)@@ -659,10 +677,12 @@ def build_config_from_preset(continueslot_config = _get_preset_slot_config(preset, slot)if not _should_apply_preset_slot(slot, slot_config):+ if slot == "vision":+ config[section] = {}continueconfig[section] = _merge_model_slot(slot,- config.get(section, {}),+ {} if slot == "vision" else config.get(section, {}),slot_config,strip_api_key=strip_api_key,)@@ -738,6 +758,23 @@ def get_chat_model_config(agent=None) -> dict:return get_effective_config(agent).get("chat_model", {})+def get_vision_model_config(agent=None) -> dict:+ """Get the optional, strictly per-preset Vision Model config."""+ return get_effective_config(agent).get("vision_model", {})+++def use_vision_sidecar(agent=None) -> bool:+ """Return whether vision_load should delegate to the preset's Vision Model."""+ vision_cfg = get_vision_model_config(agent)+ if not (+ str(vision_cfg.get("provider") or "").strip()+ and str(vision_cfg.get("name") or "").strip()+ ):+ return False+ chat_cfg = get_chat_model_config(agent)+ return not bool(chat_cfg.get("vision")) or bool(vision_cfg.get("override_main"))++def get_utility_model_config(agent=None) -> dict:"""Get utility model config, with per-chat override if active."""return get_effective_config(agent).get("utility_model", {})@@ -835,6 +872,16 @@ def build_utility_model(agent=None):)+def build_vision_model(agent=None):+ """Build the optional Vision Model selected by the effective preset."""+ cfg = get_vision_model_config(agent)+ mc = build_model_config(cfg, models.ModelType.CHAT)+ mc.vision = True+ return models.get_chat_model(+ mc.provider, mc.name, model_config=mc, **mc.build_kwargs()+ )++def build_embedding_model(agent=None):"""Build and return an embedding model wrapper."""cfg = get_embedding_model_config(agent)@@ -881,6 +928,8 @@ def get_missing_api_key_providers(agent=None) -> list[dict]:("Utility Model", cfg.get("utility_model", {})),("Embedding Model", get_embedding_model_config(agent)),]+ if use_vision_sidecar(agent):+ checks.insert(1, ("Vision Model", cfg.get("vision_model", {})))for label, model_cfg in checks:provider = model_cfg.get("provider", "")diff --git a/plugins/_model_config/webui/main.html b/plugins/_model_config/webui/main.htmlindex 7aa9908e4..83c58e6e1 100644--- a/plugins/_model_config/webui/main.html+++ b/plugins/_model_config/webui/main.html@@ -21,7 +21,7 @@<div class="preset-editor-intro"><div class="field-title">Model Presets</div><div class="field-description">- Each preset contains the complete main, utility, and embedding model setup. Default is always available.+ Each preset contains the main, utility, embedding, and optional vision-sidecar setup. Default is always available.</div></div>@@ -67,9 +67,21 @@<div class="section-title">Main Model</div><div class="section-description">Primary model for conversations, reasoning, and tools.</div></div>- <div x-data="{ get model() { return selectedPreset.chat; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">+ <div x-data="{ get model() { return selectedPreset.chat; }, get visionModel() { return selectedPreset.vision; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }"><x-component path="/plugins/_model_config/webui/model-field.html"></x-component></div>+ <template x-if="!selectedPreset.chat.vision || selectedPreset.vision.override_main">+ <div class="vision-sidecar-selector">+ <div class="field-title">Vision sidecar</div>+ <div class="field-description"+ x-text="selectedPreset.chat.vision+ ? 'Overrides Main model vision for vision_load.'+ : 'Used by vision_load while Main model vision is disabled. Leave empty to disable image analysis.'"></div>+ <div x-data="{ get model() { return selectedPreset.vision; }, modelType: 'vision', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">+ <x-component path="/plugins/_model_config/webui/model-field.html"></x-component>+ </div>+ </div>+ </template></section><section class="preset-model-section">@@ -166,6 +178,11 @@margin-top: 0;}+ .vision-sidecar-selector {+ margin: 0.75rem 0 0;+ padding: 0.25rem 0 0;+ }+.preset-editor-secondary-actions {display: flex;flex-wrap: wrap;diff --git a/plugins/_model_config/webui/model-config-store.js b/plugins/_model_config/webui/model-config-store.jsindex 5e69bde20..1ab6d8c8e 100644--- a/plugins/_model_config/webui/model-config-store.js+++ b/plugins/_model_config/webui/model-config-store.js@@ -7,6 +7,7 @@ import { switcherState, switcherMethods } from "/plugins/_model_config/webui/swiexport const MODEL_SECTIONS = [{ key: 'chat_model', title: 'Main Model', desc: 'Primary model for chat, reasoning, and browser tasks.' },+ { key: 'vision_model', title: 'Vision Model', desc: 'Optional model used by vision_load when Main vision is unavailable or overridden.' },{ key: 'utility_model', title: 'Utility Model', desc: 'Lightweight model for background tasks: memory management, prompt preparation, summarization.' },{ key: 'embedding_model', title: 'Embedding Model', desc: 'Model for generating vector embeddings used in knowledge retrieval.' }];@@ -59,6 +60,15 @@ function isBlankPresetValue(value) {}const IMPLICIT_PRESET_SLOT_DEFAULTS = {+ vision: {+ vision: true,+ max_embeds: 10,+ override_main: false,+ rl_requests: 0,+ rl_input: 0,+ rl_output: 0,+ kwargs: {},+ },utility: {ctx_length: 128000,ctx_input: 0.7,@@ -143,15 +153,21 @@ export function configFromPreset(preset, baseConfig, stripApiKey = true) {const config = clonePlain(baseConfig || {});const slots = [['chat', 'chat_model'],+ ['vision', 'vision_model'],['utility', 'utility_model'],['embedding', 'embedding_model'],];for (const [slotKey, sectionKey] of slots) {const slot = preset?.[slotKey];- if (!slot || typeof slot !== 'object') continue;- if (!hasModelIdentity(slot)) continue;- config[sectionKey] = mergeModelSlot(config[sectionKey] || {}, slot, stripApiKey, slotKey);+ if (slotKey === 'vision') config[sectionKey] = {};+ if (!slot || typeof slot !== 'object' || !hasModelIdentity(slot)) continue;+ config[sectionKey] = mergeModelSlot(+ slotKey === 'vision' ? {} : (config[sectionKey] || {}),+ slot,+ stripApiKey,+ slotKey,+ );}return config;@@ -206,8 +222,10 @@ export const store = createStore("modelConfig", {const source = (rawPresets || []).filter(p => p && typeof p === 'object');const rawDefault = source.find(p => String(p.name || '').toLowerCase() === 'default') || {};const slot = value => ({ provider: '', name: '', api_key: '', api_base: '', kwargs: {}, ...(value || {}) });+ const visionSlot = value => ({ ...slot(value), vision: true, max_embeds: Number(value?.max_embeds ?? 10), override_main: !!value?.override_main });const defaultConfig = {chat_model: slot(rawDefault.chat),+ vision_model: hasModelIdentity(rawDefault.vision) ? visionSlot(rawDefault.vision) : {},utility_model: slot(rawDefault.utility),embedding_model: slot(rawDefault.embedding),};@@ -219,6 +237,7 @@ export const store = createStore("modelConfig", {return {name: p.name || '',chat: { ...slot(effective.chat_model), _kwargs_text: kwargsToText(effective.chat_model?.kwargs) },+ vision: { ...visionSlot(effective.vision_model), _kwargs_text: kwargsToText(effective.vision_model?.kwargs) },utility: { ...slot(effective.utility_model), _kwargs_text: kwargsToText(effective.utility_model?.kwargs) },embedding: { ...slot(effective.embedding_model), _kwargs_text: kwargsToText(effective.embedding_model?.kwargs) },};@@ -273,6 +292,7 @@ export const store = createStore("modelConfig", {// Config field initialization (converts kwargs dicts to editable text)initConfigFields(config) {if (config?.chat_model) config.chat_model._kwargs_text = kwargsToText(config.chat_model.kwargs);+ if (config?.vision_model) config.vision_model._kwargs_text = kwargsToText(config.vision_model.kwargs);if (config?.utility_model) config.utility_model._kwargs_text = kwargsToText(config.utility_model.kwargs);if (config?.embedding_model) config.embedding_model._kwargs_text = kwargsToText(config.embedding_model.kwargs);},@@ -340,6 +360,7 @@ export const store = createStore("modelConfig", {|| this.presets[0]|| {chat: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },+ vision: { provider: '', name: '', api_base: '', vision: true, max_embeds: 10, override_main: false, kwargs: {}, _kwargs_text: '' },utility: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },embedding: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },}@@ -420,9 +441,10 @@ export const store = createStore("modelConfig", {const clean = presets.map(p => {const c = { name: p.name };const isDefault = p._originalName === DEFAULT_PRESET_NAME;- for (const slot of ['chat', 'utility']) {+ for (const slot of ['chat', 'vision', 'utility']) {if (p[slot]) {const rest = cleanPresetSlot(p[slot], true, slot, isDefault);+ if (slot === 'vision' && hasModelIdentity(rest)) rest.vision = true;if (hasModelIdentity(rest)) c[slot] = rest;}}@@ -608,9 +630,15 @@ export const store = createStore("modelConfig", {const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';return [{ icon: 'chat', title: 'Main', cfg: preset?.chat, pList: chatP },+ { icon: 'eye', title: 'Vision', cfg: preset?.vision, pList: chatP },{ icon: 'manufacturing', title: 'Utility', cfg: preset?.utility, pList: chatP },{ icon: 'database', title: 'Embedding', cfg: preset?.embedding, pList: embedP },- ].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));+ ]+ .filter(s => s.title !== 'Vision' || (+ hasModelIdentity(s.cfg)+ && (!preset?.chat?.vision || s.cfg?.override_main)+ ))+ .map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));},getPreset(name) {@@ -747,9 +775,15 @@ export const store = createStore("modelConfig", {const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';return [{ icon: 'chat', title: 'Main', cfg: cfg.chat_model, pList: chatP },+ { icon: 'eye', title: 'Vision', cfg: cfg.vision_model, pList: chatP },{ icon: 'manufacturing', title: 'Utility', cfg: cfg.utility_model, pList: chatP },{ icon: 'database', title: 'Embedding', cfg: cfg.embedding_model, pList: embedP },- ].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));+ ]+ .filter(s => s.title !== 'Vision' || (+ hasModelIdentity(s.cfg)+ && (!cfg.chat_model?.vision || s.cfg?.override_main)+ ))+ .map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));},async refreshModelsSummary(contextId = '') {diff --git a/plugins/_model_config/webui/model-field.html b/plugins/_model_config/webui/model-field.htmlindex e88b905f8..51c70741a 100644--- a/plugins/_model_config/webui/model-field.html+++ b/plugins/_model_config/webui/model-field.html@@ -10,13 +10,14 @@Reusable model configuration field set.Parent x-data scope must provide:model — reactive object with provider, name, api_key, api_base, ctx_length, ctx_history, ctx_input, vision, max_embeds, rl_requests, rl_input, rl_output, kwargs, _kwargs_text- modelType — 'chat' | 'utility' | 'embedding'+ modelType — 'chat' | 'vision' | 'utility' | 'embedding'providers — array of { value, label }searchType — 'chat' | 'embedding'apiKeyMode — 'store' (key lives in $store.modelConfig.apiKeyValues), 'inline' (key lives in model.api_key), or 'none'Optional:providerFallback — fallback provider string for search/API key status (e.g. preset.chat.provider for utility slot)apiBaseFallback — fallback api_base string for search (e.g. preset.chat.api_base for utility slot)+ visionModel — vision slot object, required by the chat model's sidecar switch--><div x-data="{ get _prov() { return model.provider || (typeof providerFallback !== 'undefined' ? providerFallback : ''); }, get _apiBase() { return model.api_base || (typeof apiBaseFallback !== 'undefined' ? apiBaseFallback : ''); } }"><!-- Provider -->@@ -146,8 +147,23 @@</div></template>+ <template x-if="modelType === 'chat' && model.vision">+ <div class="field">+ <div class="field-label">+ <div class="field-title">Use separate Vision Model</div>+ <div class="field-description">When disabled, vision_load uses this model's native vision.</div>+ </div>+ <div class="field-control">+ <label class="toggle">+ <input type="checkbox" x-model="visionModel.override_main" />+ <span class="toggler"></span>+ </label>+ </div>+ </div>+ </template>+<!-- Context window size (main and utility only) -->- <template x-if="modelType !== 'embedding'">+ <template x-if="modelType === 'chat' || modelType === 'utility'"><div class="field"><div class="field-label"><div class="field-title">Context window size</div>@@ -207,6 +223,18 @@</div></template>+ <template x-if="modelType === 'vision'">+ <div class="field">+ <div class="field-label">+ <div class="field-title">Max embeds</div>+ <div class="field-description">Maximum number of images sent to the Vision Model in one call. Set to 0 for unlimited.</div>+ </div>+ <div class="field-control">+ <input type="number" min="0" x-model.number="model.max_embeds" />+ </div>+ </div>+ </template>+<!-- Utility-specific: ctx_input slider --><template x-if="modelType === 'utility'"><div class="field">diff --git a/plugins/_model_config/webui/preset-overview.html b/plugins/_model_config/webui/preset-overview.htmlindex a77467fa4..75ef0750d 100644--- a/plugins/_model_config/webui/preset-overview.html+++ b/plugins/_model_config/webui/preset-overview.html@@ -17,10 +17,22 @@<div class="model-preset-rows" aria-live="polite"><template x-for="model in modelRows" :key="model.title">- <div class="model-preset-row">- <x-icon class="model-preset-icon" :name="model.icon"></x-icon>- <span class="model-preset-role" x-text="model.title"></span>- <span class="model-preset-identity">+ <div class="model-preset-row"+ :class="{ 'model-preset-row-nested': model.title === 'Vision' }">+ <template x-if="model.title !== 'Vision'">+ <x-icon class="model-preset-icon" :name="model.icon"></x-icon>+ </template>+ <template x-if="model.title !== 'Vision'">+ <span class="model-preset-role" x-text="model.title"></span>+ </template>+ <span class="model-preset-identity"+ :class="{ 'model-preset-identity-vision': model.title === 'Vision' }">+ <template x-if="model.title === 'Vision'">+ <span>+ <span class="model-preset-role">Vision override</span>+ <span class="model-preset-separator">/</span>+ </span>+ </template><span class="model-preset-provider" x-text="model.provider"></span><span class="model-preset-separator">/</span><span x-text="model.name"></span>@@ -86,6 +98,16 @@border-top: 1px solid var(--color-border);}+ .model-preset-row-nested {+ border-top: 0 !important;+ padding-top: var(--spacing-xs);+ font-size: 0.76rem;+ }++ .model-preset-identity-vision {+ grid-column: 3;+ }+.model-preset-icon,.model-preset-provider,.model-preset-separator {diff --git a/prompts/AGENTS.md b/prompts/AGENTS.mdindex a3b9fb85e..3119674eb 100644--- a/prompts/AGENTS.md+++ b/prompts/AGENTS.md@@ -25,6 +25,7 @@- Read the rendering path before changing placeholders or filenames.- Prefer small prompt additions over broad rewrites when fixing a specific behavior.- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.+- Keep native and sidecar vision prompts synchronized with `vision_load`: related images belong in one call, while a sidecar result is text-only unless Main explicitly requests its native raw route.- Update tests or snapshots when prompt budget, required sections, or generated system content changes.## Verificationdiff --git a/prompts/agent.system.tools_vision_sidecar.md b/prompts/agent.system.tools_vision_sidecar.mdnew file mode 100644index 000000000..73a84ee23--- /dev/null+++ b/prompts/agent.system.tools_vision_sidecar.md@@ -0,0 +1,48 @@+## multimodal vision tools++### vision_load+analyze images with the separate Vision Model and return a text result+args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction, `raw` optional boolean+Input schema for tool_args:+```json+{+ "type": "object",+ "properties": {+ "paths": {+ "type": "array",+ "items": {"type": "string"},+ "description": "Absolute image paths or ephemeral image refs."+ },+ "query": {+ "type": "string",+ "description": "What the Vision Model should inspect, compare, locate, or read."+ },+ "raw": {+ "type": "boolean",+ "description": "Use the Main model's native vision instead, when Main supports vision."+ }+ },+ "required": ["paths"],+ "additionalProperties": false+}+```+rules:+- put all images needed for one comparison or visual task in the same `paths` array; they are sent in one Vision Model call+- use a focused `query`; if omitted, the Vision Model returns a concise general description+- the result is a text capsule; the Main model does not receive the raw images+- use `raw=true` only when Main supports vision and must inspect the pixels itself+- only bitmaps are supported+example:+```json+{+ "thoughts": [+ "I need to compare both screenshots before answering."+ ],+ "headline": "Comparing screenshots",+ "tool_name": "vision_load",+ "tool_args": {+ "paths": ["/path/to/before.png", "/path/to/after.png"],+ "query": "Compare the error banners and describe what changed."+ }+}+```diff --git a/tests/test_browser_agent_regressions.py b/tests/test_browser_agent_regressions.pyindex 7fba2ba48..a924ec00b 100644--- a/tests/test_browser_agent_regressions.py+++ b/tests/test_browser_agent_regressions.py@@ -36,8 +36,13 @@ class _TestAgentContextType:class _TestResponse(SimpleNamespace):- def __init__(self, message="", break_loop=False, **kwargs):- super().__init__(message=message, break_loop=break_loop, **kwargs)+ def __init__(self, message="", break_loop=False, additional=None, **kwargs):+ super().__init__(+ message=message,+ break_loop=break_loop,+ additional=additional,+ **kwargs,+ )class _TestTool:@@ -96,6 +101,9 @@ _model_config_stub = ModuleType("plugins._model_config.helpers.model_config")_model_config_stub.get_presets = lambda: []_model_config_stub.get_preset_by_name = lambda name: None_model_config_stub.get_chat_model_config = lambda agent=None: {}+_model_config_stub.get_vision_model_config = lambda agent=None: {}+_model_config_stub.use_vision_sidecar = lambda agent=None: False+_model_config_stub.build_vision_model = lambda agent=None: Nonesys.modules.setdefault("plugins._model_config.helpers.model_config", _model_config_stub)@@ -4297,11 +4305,9 @@ async def test_vision_load_materializes_ephemeral_browser_refs(monkeypatch, tmp_monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)- monkeypatch.setattr(- vision_load_module.plugins,- "get_plugin_config",- lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},- )+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)tool_results = []messages = []diff --git a/tests/test_model_config_api_keys.py b/tests/test_model_config_api_keys.pyindex eb779aba1..a52f5583a 100644--- a/tests/test_model_config_api_keys.py+++ b/tests/test_model_config_api_keys.py@@ -149,7 +149,7 @@ def test_model_config_frontend_tracks_provider_api_key_edits():assert "/plugins/_model_config/missing_api_key_status" not in model_gate_contentassert '@input="$store.modelConfig.setApiKeyValue(_prov, $el.value)"' in config_contentassert "apiKeyMode: 'none'" not in preset_modal_content- assert preset_modal_content.count("apiKeyMode: 'store'") == 3+ assert preset_modal_content.count("apiKeyMode: 'store'") == 4assert "$store.modelConfig.resetApiKeyDrafts();" in preset_modal_contentassert "await $store.modelConfig.refreshApiKeyStatus();" in preset_modal_contentassert "await store.persistAllDirtyApiKeys();" in store_contentdiff --git a/tests/test_model_config_project_presets.py b/tests/test_model_config_project_presets.pyindex bb8c67045..1c972b43c 100644--- a/tests/test_model_config_project_presets.py+++ b/tests/test_model_config_project_presets.py@@ -1092,6 +1092,44 @@ def test_preset_application_inherits_optional_slots(monkeypatch, tmp_path):assert config["embedding_model"] == base_config["embedding_model"]+def test_preset_vision_slot_is_optional_and_never_inherited(monkeypatch, tmp_path):+ _prepare_a0_tree(monkeypatch, tmp_path)++ from plugins._model_config.helpers import model_config++ base_config = {+ "chat_model": {"provider": "openrouter", "name": "main", "vision": False},+ "vision_model": {+ "provider": "openrouter",+ "name": "default-vision",+ "max_embeds": 2,+ },+ }++ without_sidecar = model_config.build_config_from_preset(+ {"name": "Text only", "chat": {"provider": "openrouter", "name": "text"}},+ base_config,+ )+ with_sidecar = model_config.build_config_from_preset(+ {+ "name": "Visual",+ "chat": {"provider": "openrouter", "name": "text"},+ "vision": {+ "provider": "anthropic",+ "name": "visual",+ "max_embeds": 5,+ },+ },+ base_config,+ )++ assert without_sidecar["vision_model"] == {}+ assert with_sidecar["vision_model"]["provider"] == "anthropic"+ assert with_sidecar["vision_model"]["name"] == "visual"+ assert with_sidecar["vision_model"]["max_embeds"] == 5+ assert "default-vision" not in str(with_sidecar["vision_model"])++def test_legacy_utility_preset_defaults_preserve_tuning_but_clear_kwargs(monkeypatch,tmp_path,diff --git a/tests/test_model_config_ui.py b/tests/test_model_config_ui.pyindex 1feb61061..5c6d54aa8 100644--- a/tests/test_model_config_ui.py+++ b/tests/test_model_config_ui.py@@ -85,6 +85,49 @@ def test_preset_editor_uses_standard_modal_footer_buttons() -> None:assert "preset-editor-footer" not in preset_modal+def test_preset_editor_nests_one_conditional_vision_selector_in_main() -> None:+ preset_modal = read("plugins", "_model_config", "webui", "main.html")+ model_field = read("plugins", "_model_config", "webui", "model-field.html")+ preset_overview = read("plugins", "_model_config", "webui", "preset-overview.html")+ preset_store = read("plugins", "_model_config", "webui", "model-config-store.js")++ main_start = preset_modal.index('<div class="section-title">Main Model</div>')+ utility_start = preset_modal.index('<div class="section-title">Utility Model</div>')+ selector_start = preset_modal.index('class="vision-sidecar-selector"')+ supports_start = model_field.index('<div class="field-title">Supports Vision</div>')+ override_start = model_field.index('<div class="field-title">Use separate Vision Model</div>')+ context_start = model_field.index('<div class="field-title">Context window size</div>')+ advanced_start = model_field.index('<!-- Advanced Settings (collapsed by default) -->')++ assert '<div class="section-title">Vision Model</div>' not in preset_modal+ assert preset_modal.count('class="vision-sidecar-selector"') == 1+ assert main_start < selector_start < utility_start+ assert '!selectedPreset.chat.vision || selectedPreset.vision.override_main' in preset_modal+ assert "get visionModel() { return selectedPreset.vision; }" in preset_modal+ assert "modelType: 'vision'" in preset_modal+ assert preset_modal.count("apiKeyMode: 'store'") == 4+ assert "margin: 0.75rem 0 0;" in preset_modal+ assert "padding: 0.25rem 0 0;" in preset_modal+ assert "border-left: 2px solid var(--color-border);" not in preset_modal+ assert "Use separate Vision Model" in model_field+ assert supports_start < override_start < context_start < advanced_start+ assert "When disabled, vision_load uses this model's native vision." in model_field+ assert "When enabled, vision_load uses the preset's Vision Model" not in model_field+ assert 'x-model="visionModel.override_main"' in model_field+ assert '<div class="advanced-section" x-data="{ advOpen: false }">' in model_field+ assert "model-preset-row-nested" in preset_overview+ assert "model.title === 'Vision'" in preset_overview+ assert preset_overview.count("model.title !== 'Vision'") == 2+ assert '<span class="model-preset-role">Vision override</span>' in preset_overview+ assert "'model-preset-identity-vision': model.title === 'Vision'" in preset_overview+ assert "grid-column: 3;" in preset_overview+ assert "padding-top: var(--spacing-xs);" in preset_overview+ assert "margin-left: 2rem;" not in preset_overview+ assert "border-left: 1px solid var(--color-border);" not in preset_overview+ assert "if (slotKey === 'vision') config[sectionKey] = {};" in preset_store+ assert "['chat', 'vision', 'utility']" in preset_store++def test_plugin_settings_reset_is_explicit_and_does_not_capture_toast_early() -> None:settings_modal = read("webui", "components", "plugins", "plugin-settings.html")settings_store = read("webui", "components", "plugins", "plugin-settings-store.js")diff --git a/tests/test_parallel_tool.py b/tests/test_parallel_tool.pyindex dc1a91199..575f21f51 100644--- a/tests/test_parallel_tool.py+++ b/tests/test_parallel_tool.py@@ -160,6 +160,42 @@ def test_normalize_parallel_tool_calls_accepts_json_string_array() -> None:assert calls[1].tool_args["message"] == "Research nuclear fusion news in Italian."+def test_parallel_keeps_mixed_tools_and_batched_vision_paths() -> None:+ calls = parallel_tools.normalize_parallel_tool_calls(+ [+ {"tool_name": "search_a", "tool_args": {"query": "one"}},+ {"tool_name": "search_b", "tool_args": {"query": "two"}},+ {"tool_name": "browser_agent", "tool_args": {"message": "open page"}},+ {+ "tool_name": "vision_load",+ "tool_args": {+ "paths": ["/tmp/before.png", "/tmp/after.png"],+ "query": "compare",+ },+ },+ ]+ )++ assert [call.tool_name for call in calls] == [+ "search_a",+ "search_b",+ "browser_agent",+ "vision_load",+ ]+ assert calls[3].tool_args["paths"] == ["/tmp/before.png", "/tmp/after.png"]+++def test_parallel_allows_multiple_independent_vision_calls() -> None:+ calls = parallel_tools.normalize_parallel_tool_calls(+ [+ {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/a.png"]}},+ {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/b.png"]}},+ ]+ )++ assert [call.tool_name for call in calls] == ["vision_load", "vision_load"]++def test_subordinate_prompts_share_reusable_tree_contract() -> None:call_prompt = (PROJECT_ROOT / "prompts/agent.system.tool.call_sub.md").read_text(encoding="utf-8"diff --git a/tests/test_tool_policy.py b/tests/test_tool_policy.pyindex 2db20350c..3d3447325 100644--- a/tests/test_tool_policy.py+++ b/tests/test_tool_policy.py@@ -546,6 +546,59 @@ async def test_vision_tool_follows_chat_config_not_profile_policy(assert tool_policy.resolve_tool(agent, "vision_load").source == "runtime-config"+async def test_vision_sidecar_prompt_replaces_native_vision_prompt(+ monkeypatch, tmp_path: Path+) -> None:+ _write_prompt(tmp_path, "agent.system.tools.md", "TOOLS\n{{tools}}")+ _write_prompt(+ tmp_path,+ "agent.system.tools_vision.md",+ "### vision_load\nnative pixels\nargs: `paths`",+ )+ _write_prompt(+ tmp_path,+ "agent.system.tools_vision_sidecar.md",+ "### vision_load\nsidecar capsule\nargs: `paths`, `query`",+ )+ monkeypatch.setattr(tool_policy.subagents, "get_paths", _prompt_paths(tmp_path))+ monkeypatch.setattr(+ "plugins._model_config.helpers.model_config.get_chat_model_config",+ lambda agent: {"vision": True},+ )+ monkeypatch.setattr(+ "plugins._model_config.helpers.model_config.use_vision_sidecar",+ lambda agent: True,+ )+ monkeypatch.setattr(responses_tools, "_mcp_tools", lambda agent: [])+ agent = _Agent(tmp_path)++ prompt = await _11_tools_prompt.build_prompt(agent)+ schemas, _name_map = responses_tools.build_responses_function_tools(agent)++ assert "sidecar capsule" in prompt+ assert "native pixels" not in prompt+ assert schemas[0]["name"] == "vision_load"+ assert schemas[0]["description"] == "sidecar capsule"+++def test_vision_sidecar_prompt_declares_multi_image_native_schema() -> None:+ prompt = (+ Path(__file__).resolve().parents[1]+ / "prompts"+ / "agent.system.tools_vision_sidecar.md"+ ).read_text(encoding="utf-8")+ schema = responses_tools._schema_from_prompt(prompt)++ assert schema["required"] == ["paths"]+ assert schema["properties"]["paths"] == {+ "type": "array",+ "items": {"type": "string"},+ "description": "Absolute image paths or ephemeral image refs.",+ }+ assert {"query", "raw"} <= schema["properties"].keys()++def test_mcp_prompt_and_native_schema_omit_blocked_tool(monkeypatch, tmp_path: Path) -> None:diff --git a/tests/test_vision_load_image_refs.py b/tests/test_vision_load_image_refs.pyindex 2842a5e1e..619901aa0 100644--- a/tests/test_vision_load_image_refs.py+++ b/tests/test_vision_load_image_refs.py@@ -1,3 +1,4 @@+import asyncioimport typesfrom types import SimpleNamespaceimport sys@@ -13,8 +14,13 @@ from helpers import imagesclass _TestResponse(SimpleNamespace):- def __init__(self, message="", break_loop=False, **kwargs):- super().__init__(message=message, break_loop=break_loop, **kwargs)+ def __init__(self, message="", break_loop=False, additional=None, **kwargs):+ super().__init__(+ message=message,+ break_loop=break_loop,+ additional=additional,+ **kwargs,+ )class _TestTool:@@ -74,11 +80,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatchmonkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)- monkeypatch.setattr(- vision_load_module.plugins,- "get_plugin_config",- lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},- )+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)async def direct_call(func, *args, **kwargs):return func(*args, **kwargs)@@ -121,3 +125,215 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatchstored_path = tmp_path / stored_ref.removeprefix("/a0/")assert stored_path.read_bytes() == b"png-data"assert updates[-1]["result"] == "1 images loaded, 0 skipped"+++def test_vision_sidecar_route_matrix_prefers_main_native_vision(monkeypatch):+ from plugins._model_config.helpers import model_config++ cases = [+ ({"vision": False}, {}, False),+ ({"vision": True}, {"provider": "p", "name": "v"}, False),+ ({"vision": False}, {"provider": "p", "name": "v"}, True),+ (+ {"vision": True},+ {"provider": "p", "name": "v", "override_main": True},+ True,+ ),+ ]+ for chat, vision, expected in cases:+ monkeypatch.setattr(+ model_config,+ "get_effective_config",+ lambda _agent=None, chat=chat, vision=vision: {+ "chat_model": chat,+ "vision_model": vision,+ },+ )+ assert model_config.use_vision_sidecar() is expected+++async def test_vision_sidecar_sends_multiple_images_once_and_keeps_history_text_only(+ monkeypatch,+ tmp_path,+):+ _install_tool_stub(monkeypatch)+ import tools.vision_load as vision_load_module++ async def direct_call(func, *args, **kwargs):+ return func(*args, **kwargs)++ calls = []++ class FakeVisionModel:+ async def unified_call(self, **kwargs):+ calls.append(kwargs)+ return "The second screenshot fixes the red login error.", ""++ monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)+ monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())+ monkeypatch.setattr(+ vision_load_module,+ "get_chat_model_config",+ lambda _agent: {"vision": True, "max_embeds": 1},+ )+ monkeypatch.setattr(+ vision_load_module,+ "get_vision_model_config",+ lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 5},+ )+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)++ image_paths = [tmp_path / "before.png", tmp_path / "after.png"]+ for path in image_paths:+ path.write_bytes(b"png-data")++ tool_results = []+ raw_messages = []+ agent = SimpleNamespace(+ context=SimpleNamespace(id=""),+ agent_name="Agent 0",+ hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),+ hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),+ )+ tool = vision_load_module.VisionLoad(+ agent=agent,+ name="vision_load",+ method=None,+ args={"paths": [str(path) for path in image_paths]},+ message="",+ loop_data=None,+ )+ tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)++ response = await tool.execute(+ paths=[str(path) for path in image_paths],+ query="Compare the login errors.",+ )+ response.additional = {"_responses_output_item": {"output": response.message}}+ await tool.after_execution(response)++ assert len(calls) == 1+ content = calls[0]["messages"][1].content+ assert content[0] == {"type": "text", "text": "Compare the login errors."}+ assert [item["type"] for item in content].count("image_url") == 2+ assert "fixes the red login error" in response.message+ assert response.message != "dummy"+ assert raw_messages == []+ assert tool.loaded_paths == [str(path) for path in image_paths]+ assert tool_results[0][1]["_responses_output_item"]["output"] == response.message+++async def test_parallel_worker_consumes_parent_ephemeral_image(monkeypatch, tmp_path):+ _install_tool_stub(monkeypatch)+ import tools.vision_load as vision_load_module++ def fake_get_abs_path(*parts):+ return str(tmp_path.joinpath(*parts))++ def fake_normalize_a0_path(path):+ return "/a0/" + str(Path(path).relative_to(tmp_path)).replace("\\", "/")++ monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)+ monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)+ monkeypatch.setattr(vision_load_module.VisionLoad, "_config_agent", lambda self: self.agent)+ monkeypatch.setattr(+ vision_load_module,+ "get_chat_model_config",+ lambda _agent: {"vision": True, "max_embeds": 10},+ )+ monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)++ parent_id = "parent-vision"+ ref = vision_load_module.ephemeral_images.put_image_bytes(+ context_id=parent_id,+ mime="image/png",+ payload=b"png-data",+ name="shot.png",+ )+ context = SimpleNamespace(+ id="parallel-worker",+ get_data=lambda key: parent_id+ if key == vision_load_module.PARALLEL_WORKER_PARENT_CONTEXT_KEY+ else None,+ )+ agent = SimpleNamespace(context=context, agent_name="Agent 0")+ tool = vision_load_module.VisionLoad(+ agent=agent,+ name="vision_load",+ method=None,+ args={"paths": [ref]},+ message="",+ loop_data=None,+ )++ await tool.execute(paths=[ref])++ assert tool.loaded_paths == ["shot.png"]+ assert vision_load_module.ephemeral_images.get_image(ref, context_id=parent_id) is None+ stored_ref = tool.images_dict["shot.png"]+ assert stored_ref.startswith("/a0/usr/chats/parent-vision/images/vision-load/shot-")+++async def test_independent_vision_sidecar_calls_can_run_concurrently(monkeypatch, tmp_path):+ _install_tool_stub(monkeypatch)+ import tools.vision_load as vision_load_module++ active = 0+ max_active = 0+ call_count = 0++ class FakeVisionModel:+ async def unified_call(self, **kwargs):+ nonlocal active, max_active, call_count+ active += 1+ call_count += 1+ max_active = max(max_active, active)+ await asyncio.sleep(0.02)+ active -= 1+ return "done", ""++ async def direct_call(func, *args, **kwargs):+ return func(*args, **kwargs)++ monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)+ monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())+ monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": False})+ monkeypatch.setattr(+ vision_load_module,+ "get_vision_model_config",+ lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 10},+ )+ monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)++ image_paths = [tmp_path / "one.png", tmp_path / "two.png"]+ for path in image_paths:+ path.write_bytes(b"png-data")++ def make_tool(index):+ agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")+ return vision_load_module.VisionLoad(+ agent=agent,+ name="vision_load",+ method=None,+ args={"paths": [str(path) for path in image_paths]},+ message="",+ loop_data=None,+ )++ responses = await asyncio.gather(+ *(+ make_tool(index).execute(+ paths=[str(path) for path in image_paths],+ query=f"inspection {index}",+ )+ for index in range(4)+ )+ )++ assert call_count == 4+ assert max_active == 4+ assert all("done" in response.message for response in responses)diff --git a/tools/vision_load.py b/tools/vision_load.pyindex 7358afbde..b29ff931f 100644--- a/tools/vision_load.py+++ b/tools/vision_load.py@@ -1,89 +1,219 @@-from helpers.print_style import PrintStyle-from helpers.tool import Tool, Response-from helpers import runtime, files, plugins, ephemeral_images, images, chat_media+import asyncio+import jsonfrom mimetypes import guess_type-from helpers import history-# image token estimation for context window+from langchain_core.messages import HumanMessage, SystemMessage++from helpers import chat_media, ephemeral_images, files, history, images, runtime+from helpers.parallel_tools import PARALLEL_WORKER_PARENT_CONTEXT_KEY, coerce_bool+from helpers.print_style import PrintStyle+from helpers.tool import Response, Tool+from plugins._model_config.helpers.model_config import (+ build_vision_model,+ get_chat_model_config,+ get_vision_model_config,+ use_vision_sidecar,+)+TOKENS_ESTIMATE = 1500+VISION_TIMEOUT_SECONDS = 300+VISION_SYSTEM_PROMPT = (+ "You are a precise vision analyst. Answer only what was asked about the images. "+ "Be concise and factual. Preserve exact visible text when asked to read it."+)+DEFAULT_VISION_QUERY = (+ "Describe the images precisely, including the key objects, visible text, and layout."+)class VisionLoad(Tool):- async def execute(self, paths: list[str] = [], **kwargs) -> Response:-- self.images_dict = {}+ async def execute(+ self,+ paths: list[str] | str | None = None,+ query: str = "",+ raw: bool = False,+ **kwargs,+ ) -> Response:+ self.images_dict: dict[str, str] = {}self.loaded_paths: list[str] = []self.skipped_paths: list[str] = []+ self._config_owner = self._config_agent()+ self._main_has_vision = bool(+ get_chat_model_config(self._config_owner).get("vision", False)+ )+ self._delegated = use_vision_sidecar(self._config_owner) and not (+ coerce_bool(raw, False) and self._main_has_vision+ )+ self._max_embeds = self._get_max_embeds()++ normalized = self._normalize_paths(paths)+ if isinstance(normalized, str):+ self._history_result = normalized+ return Response(message=normalized, break_loop=False)- max_embeds = self._get_max_embeds()requested = [- (str(path or "").strip(), self._display_input_path(str(path or "").strip(), idx + 1))- for idx, path in enumerate(paths)+ (path.strip(), self._display_input_path(path.strip(), index + 1))+ for index, path in enumerate(normalized)]- limited_paths = requested if max_embeds <= 0 else requested[-max_embeds:]- self.skipped_paths = (- [display for _, display in requested[:-max_embeds]]- if max_embeds > 0 and len(requested) > max_embeds- else []- )+ limited = requested if self._max_embeds <= 0 else requested[-self._max_embeds :]+ if self._max_embeds > 0 and len(requested) > self._max_embeds:+ self.skipped_paths = [display for _, display in requested[: -self._max_embeds]]- for idx, (path, display_path) in enumerate(limited_paths):+ for index, (path, display_path) in enumerate(limited):if not path:continueif ephemeral_images.is_ref(path):- image = ephemeral_images.consume_image(- path,- context_id=self._context_id(),- )+ image = ephemeral_images.consume_image(path, context_id=self._context_id())if image is None:continue- display = image.display_name or display_path+ display_path = image.display_name or display_pathstored_ref = self._store_ephemeral_image(image)- if stored_ref:- self.images_dict[display] = stored_ref- self.loaded_paths.append(display)- continue- if self._is_data_image_url(path):- stored_ref = self._store_data_url(path, preferred_name=f"vision-load-{idx + 1}.png")- if stored_ref:- self.images_dict[display_path] = stored_ref- self.loaded_paths.append(display_path)- continue- if not await runtime.call_development_function(files.exists, str(path)):+ elif self._is_data_image_url(path):+ stored_ref = self._store_data_url(+ path, preferred_name=f"vision-load-{index + 1}.png"+ )+ elif await runtime.call_development_function(files.exists, path):+ mime_type, _ = guess_type(path)+ if not mime_type or not mime_type.startswith("image/"):+ continue+ try:+ stored_ref = self._store_local_image(+ path, preferred_name=files.basename(path)+ )+ except (FileNotFoundError, OSError, ValueError):+ continue+ else:continue- if path not in self.images_dict:- mime_type, _ = guess_type(str(path))- if mime_type and mime_type.startswith("image/"):- try:- stored_ref = self._store_local_image(path, preferred_name=files.basename(path))- self.images_dict[display_path] = stored_ref- self.loaded_paths.append(display_path)- except (FileNotFoundError, OSError, ValueError):- continue+ if stored_ref:+ self.images_dict[display_path] = stored_ref+ self.loaded_paths.append(display_path)- return Response(message="dummy", break_loop=False)+ summary = self._summary()+ if self._delegated and self.images_dict:+ try:+ capsule = await self._call_vision_model(+ list(self.images_dict.values()), self._query(query, kwargs)+ )+ message = (+ f"Vision Model analyzed {len(self.images_dict)} image(s)"+ f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"+ )+ self._history_result = message+ return Response(message=message, break_loop=False)+ except Exception as exc:+ message = f"Vision Model error: {str(exc)[:1000]}"+ self._history_result = f"{summary}\n\n{message}"+ return Response(message=message, break_loop=False)++ if self.images_dict and not self._main_has_vision:+ summary += (+ "\n\nImages were not injected because neither Main native vision nor "+ "a usable Vision Model is active."+ )+ self._history_result = (+ summary if self.images_dict or self.skipped_paths else "No images processed"+ )+ message = (+ "No images processed"+ if not self.images_dict and not self.skipped_paths+ else f"{len(self.images_dict)} images loaded, {len(self.skipped_paths)} skipped"+ )+ return Response(message=message, break_loop=False)++ async def after_execution(self, response: Response, **kwargs):+ log_id = str(getattr(getattr(self, "log", None), "id", "") or "")+ self.agent.hist_add_tool_result(+ self.name,+ self._history_result,+ id=log_id,+ **(response.additional or {}),+ )++ if self.images_dict and self._main_has_vision and not self._delegated:+ content = [+ {"type": "image_url", "image_url": {"url": image_path}}+ for image_path in self.images_dict.values()+ ]+ self.agent.hist_add_message(+ False,+ content=history.RawMessage(+ raw_content=content,+ preview="<Image attachments loaded by path>",+ ),+ tokens=TOKENS_ESTIMATE * len(content),+ )++ PrintStyle(+ font_color="#1B4F72", background_color="white", padding=True, bold=True+ ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")+ PrintStyle(font_color="#85C1E9").print(response.message)+ if getattr(self, "log", None):+ self.log.update(result=response.message)def _get_max_embeds(self) -> int:- cfg = plugins.get_plugin_config("_model_config", agent=self.agent) or {}- chat_cfg = cfg.get("chat_model", {})- max_embeds = chat_cfg.get("max_embeds", 10)- return int(max_embeds or 0)+ cfg = (+ get_vision_model_config(self._config_owner)+ if self._delegated+ else get_chat_model_config(self._config_owner)+ )+ try:+ return int(cfg.get("max_embeds", 10) or 0)+ except (TypeError, ValueError):+ return 10def _context_id(self) -> str:- return str(getattr(getattr(self.agent, "context", None), "id", "") or "").strip()+ context = getattr(self.agent, "context", None)+ if not context:+ return ""+ get_data = getattr(context, "get_data", None)+ parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""+ return str(parent_id or getattr(context, "id", "") or "").strip()++ def _config_agent(self):+ context = getattr(self.agent, "context", None)+ get_data = getattr(context, "get_data", None)+ parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""+ if parent_id:+ from agent import AgentContext++ parent = AgentContext.get(str(parent_id))+ if parent:+ return parent.agent0+ return self.agent++ async def _call_vision_model(self, image_paths: list[str], query: str) -> str:+ model = build_vision_model(self._config_owner)+ content = [{"type": "text", "text": query or DEFAULT_VISION_QUERY}]+ content.extend(+ {"type": "image_url", "image_url": {"url": path}}+ for path in image_paths+ )+ response, _ = await asyncio.wait_for(+ model.unified_call(+ messages=[+ SystemMessage(content=VISION_SYSTEM_PROMPT),+ HumanMessage(content=content),+ ],+ explicit_caching=False,+ max_tokens=2000,+ ),+ timeout=VISION_TIMEOUT_SECONDS,+ )+ if not str(response or "").strip():+ raise RuntimeError("Vision Model returned an empty response.")+ return str(response)def _store_ephemeral_image(self, image: ephemeral_images.EphemeralImage) -> str:context_id = self._context_id()if not context_id:return image.data_urlsource = chat_media.infer_source(image.ref, image.display_name)- category = chat_media.category_for_source(source)saved = chat_media.save_image_base64(context_id=context_id,data=image.data,mime_type=image.mime,- category=category,+ category=chat_media.category_for_source(source),source=source,preferred_name=image.display_name,)@@ -94,11 +224,10 @@ class VisionLoad(Tool):if not context_id:return data_urlsource = chat_media.infer_source(data_url, preferred_name)- category = chat_media.category_for_source(source)saved = chat_media.save_image_data_url(context_id=context_id,data_url=data_url,- category=category,+ category=chat_media.category_for_source(source),source=source,preferred_name=preferred_name,)@@ -115,6 +244,37 @@ class VisionLoad(Tool):preferred_name=preferred_name,)+ def _summary(self) -> str:+ loaded = "\n".join(self.loaded_paths) if self.loaded_paths else "none"+ skipped = "\n".join(self.skipped_paths) if self.skipped_paths else "none"+ return (+ f"Loaded images ({len(self.loaded_paths)}):\n{loaded}\n\n"+ f"Skipped images ({len(self.skipped_paths)}, max {self._max_embeds}):\n{skipped}"+ )++ @staticmethod+ def _normalize_paths(paths: list[str] | str | None) -> list[str] | str:+ if isinstance(paths, str):+ try:+ decoded = json.loads(paths)+ except json.JSONDecodeError:+ decoded = paths+ paths = decoded if isinstance(decoded, list) else [paths]+ if paths is None:+ return []+ if not isinstance(paths, (list, tuple)):+ return "vision_load error: `paths` must be an array of image paths."+ return [str(path or "").strip() for path in paths]++ @staticmethod+ def _query(query: str, kwargs: dict) -> str:+ if str(query or "").strip():+ return str(query).strip()+ for key in ("prompt", "question", "instruction", "focus", "request"):+ if str(kwargs.get(key) or "").strip():+ return str(kwargs[key]).strip()+ return DEFAULT_VISION_QUERY+@staticmethoddef _is_data_image_url(value: str) -> bool:normalized = str(value or "").strip().lower()@@ -125,57 +285,5 @@ class VisionLoad(Tool):if ephemeral_images.is_ref(value):return ephemeral_images.display_ref(value)if cls._is_data_image_url(value):- prefix = value.split(",", 1)[0]- return f"{prefix},<ephemeral-image-{index}>"+ return f"{value.split(',', 1)[0]},<ephemeral-image-{index}>"return value-- async def after_execution(self, response: Response, **kwargs):-- # build image data messages for LLMs, or error message- content = []- loaded_count = len(self.loaded_paths)- skipped_count = len(self.skipped_paths)- loaded_summary = "\n".join(self.loaded_paths) if self.loaded_paths else "none"- skipped_summary = "\n".join(self.skipped_paths) if self.skipped_paths else "none"- summary = (- f"Loaded images: {loaded_count}\n"- f"Loaded images:\n{loaded_summary}\n\n"- f"Skipped images: {skipped_count}\n"- f"Skipped images (max {self._get_max_embeds()} loaded at a time according to model configuration):\n{skipped_summary}"- )- if self.images_dict:- self.agent.hist_add_tool_result(self.name, summary, id=self.log.id if self.log else "")- for path, image_path in self.images_dict.items():- if image_path:- content.append(- {- "type": "image_url",- "image_url": {"url": image_path},- }- )- else:- content.append(- {- "type": "text",- "text": "Error processing image " + path,- }- )- # append as raw message content for LLMs with vision tokens estimate- msg = history.RawMessage(raw_content=content, preview="<Image attachments loaded by path>")- self.agent.hist_add_message(- False, content=msg, tokens=TOKENS_ESTIMATE * len(content)- )- else:- self.agent.hist_add_tool_result(self.name, summary if self.skipped_paths else "No images processed", id=self.log.id if self.log else "")-- # print and log short version- message = (- "No images processed"- if not self.images_dict and not self.skipped_paths- else f"{loaded_count} images loaded, {skipped_count} skipped"- )- PrintStyle(- font_color="#1B4F72", background_color="white", padding=True, bold=True- ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")- PrintStyle(font_color="#85C1E9").print(message)- self.log.update(result=message)diff --git a/tools/vision_load.py.dox.md b/tools/vision_load.py.dox.mdindex da9531242..2269eab56 100644--- a/tools/vision_load.py.dox.md+++ b/tools/vision_load.py.dox.md@@ -3,7 +3,7 @@## Purpose- Own the `vision_load.py` agent tool.-- This module loads images into model-visible content for vision-capable models.+- This module routes images either into Main model-visible content or through the preset's optional Vision Model.- Keep this file-level DOX profile synchronized with `vision_load.py` because this directory is intentionally flat.## Ownership@@ -12,13 +12,19 @@- `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.- Classes:- `VisionLoad` (`Tool`)- - `async execute(self, paths: list[str]=..., **kwargs) -> Response`+ - `async execute(self, paths, query="", raw=False, **kwargs) -> Response`- `async after_execution(self, response: Response, **kwargs)`- Notable constants/configuration names: `TOKENS_ESTIMATE`.## Runtime Contracts- Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.+- One call may contain multiple paths. The delegated route sends every selected path in one Vision Model request and returns one textual capsule.+- Main native vision wins unless the effective preset selects the sidecar route; `raw=true` returns to Main native vision only when Main supports it.+- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.+- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.+- Direct parallel workers resolve ephemeral refs, model routing, and durable chat-media storage against their recorded parent context; independent vision jobs remain generic parallel jobs.+- `max_embeds` comes from the model that actually receives the images.- Update this file whenever tool arguments, output shape, `break_loop` behavior, intervention handling, prompt instructions, or side effects change.- `VisionLoad` is a `Tool`.- `VisionLoad` defines `execute(...)`.@@ -27,7 +33,7 @@## Key Concepts-- Important called helpers/classes observed in the source: `self._get_max_embeds`, `Response`, `str.strip`, `self._context_id`, `chat_media.infer_source`, `chat_media.category_for_source`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `str.strip.lower`, `ephemeral_images.is_ref`, `cls._is_data_image_url`, `self._is_data_image_url`, `plugins.get_plugin_config`, `images.to_data_url`, `normalized.startswith`, `ephemeral_images.display_ref`, `join`, `self.agent.hist_add_tool_result`, `history.RawMessage`.+- Important called helpers/classes observed in the source: `build_vision_model`, `use_vision_sidecar`, `self._get_max_embeds`, `Response`, `self._context_id`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `ephemeral_images.consume_image`, `images.to_data_url`, `self.agent.hist_add_tool_result`, `history.RawMessage`.- Keep request/response, tool, or helper semantics documented here at the same time as source changes.## Work Guidance