Self-improving AI Employees for everyone
Vision Analyze (visionAnalyze)
| Field | Value |
|---|---|
| Category | tool / ai (palette group tool; plugin folder nodes/vision/) |
| Backend | server/nodes/vision/vision_analyze/__init__.py::VisionAnalyzeNode.vision (dispatch via execute_as_tool -> BaseNode.execute() + @Operation("vision"); budget fitting in services/media/image_fit.py; contained read via services/media/workspace.py::read_media_bytes) |
| Tests | server/tests/nodes/test_vision_analyze.py (TestSmartResize, TestFitImageBytes, TestSpecInvariants, TestProviderRequestShapes, TestSafety) |
| Skill (if any) | none |
| Dual-purpose tool | ToolNode - tool name vision (tool_schema_locked = True, split schema: LLM sees VisionAnalyzeToolInput; provider and vision_model are server_controlled_fields) |
Purpose
Gives every agent — including ones running text-only models — real image
understanding by delegating to a vision-capable model. The tool loads a
workspace image at the provider boundary (bytes exist only inside the call
and never enter results), fits it to a visual-token budget, sends it to the
operator-chosen provider with an official SDK request shape, and returns
text: describe (optionally steered by question) or extract_text
(OCR-style). This is deliberately the speech-node shape rather than a
ChatUnifier call, so it works regardless of the host agent's provider and
remains the fallback rung for text-only host models once native image
blocks land.
Inputs (handles)
| Handle | Connection type | Required | Purpose |
|---|---|---|---|
output-tool (source, top, label "Vision") |
tools | yes | Connect to an agent's input-tools; the agent calls vision(...) |
No input-main. uiHints: hideInputSection,
hideOutputSection, hideRunButton; isConfigNode auto-derived. A direct
execution (Params only, no tool args) raises NodeUserError("The vision tool needs an image argument; connect it to an agent's tools ...").
Parameters
Operator config, VisionAnalyzeParams (extra="ignore"):
| Name | Type | Default | Required | displayOptions.show | Description |
|---|---|---|---|---|---|
provider |
enum openai | anthropic | gemini |
openai |
no | - | Which vision-capable provider answers the delegate call |
vision_model |
string | "" |
no | - | Vision model id; empty uses the provider's default_model from llm_defaults.json (NodeUserError if none is configured). Deliberately NOT named model: with a sibling provider field ParameterRenderer.tsx would overwrite a field literally named model with the chat-model list |
LLM-visible arguments, VisionAnalyzeToolInput (extra="forbid"):
| Name | Type | Default | Required | Description |
|---|---|---|---|---|
operation |
enum describe | extract_text |
- | yes | Describe the image, or extract visible text |
image |
string, min_length=1, max_length=4096 (whitespace-only rejected) |
- | yes | Workspace-relative path (e.g. imports/chart.png); use the data tool to discover files |
question |
string | null, max_length=4000 |
null |
no | describe only: a specific question about the image (ignored by extract_text) |
budget |
enum small | normal | large |
normal |
no | Visual-token budget: 256 / 1024 / 2048 tokens -> tokens x 28^2 pixels (200,704 / 802,816 / 1,605,632) |
Outputs (handles)
| Handle | Shape | Description |
|---|---|---|
| (tool result) | object | VisionAnalyzeOutput (extra="allow") |
Output payload (TypeScript shape)
{
operation: 'describe' | 'extract_text';
provider: 'openai' | 'anthropic' | 'gemini';
vision_model: string; // resolved model id
text: string; // provider text ("" if the provider returned nothing)
image: { width: number; height: number; source: string }; // fitted dimensions + the requested path; never bytes
}
Logic Flow
flowchart TD
A[execute_as_tool: ToolInput + Params validated separately] --> B{params is bare Params?}
B -- yes - direct run --> E1[NodeUserError: needs an image argument]
B -- no --> C[config = ctx.raw._tool_config; provider; model = vision_model or _default_model]
C -- no default model --> E2[NodeUserError: set vision_model]
C --> D[read_media_bytes image, ctx, max MEDIA_MAX_READ_BYTES - containment via resolve_media]
D -- ValueError --> E3[NodeUserError str exc]
D -- missing / empty / oversize --> E4[NodeUserError from read_media_bytes]
D --> F[fit_image_bytes budget: smart_resize into min..max pixel window, snap to 28 px grid, LANCZOS; alpha -> PNG else JPEG q90]
F -- any exception --> E5[NodeUserError: not a readable image]
F --> G[prompt = question if describe and question else canned prompt]
G --> H[_api_key: ctx.connection provider .credentials .api_key]
H -- empty --> E6[NodeUserError: No API key stored for provider]
H --> I{provider}
I -- openai --> J[chat.completions.create model, max_completion_tokens 1024, text + image_url data URL detail auto]
I -- anthropic --> K[messages.create model, max_tokens 1024, image base64 block then text]
I -- gemini --> L[models.generate_content model, Part.from_bytes + prompt]
J & K & L --> M[VisionAnalyzeOutput text + fitted width/height + source path]
Decision Logic
- Validation:
imagemust be non-blank;questionis only consulted fordescribe;extra="forbid"rejects any unknown argument (ValidationError->{"error": "Invalid tool input/configuration: ..."}). - Config precedence (
_config):ctx.raw["_tool_config"](validated persisted Params fromexecute_as_tool) > a bareParamsinstance > defaults. Model arguments can never setprovider/vision_model. - Model resolution:
vision_model.strip()orget_provider_config(provider).default_model; a provider without a configured default raisesNodeUserError. - Fitting (
image_fit.smart_resize): one aspect-preserving scale into[min_pixels, max_pixels](floor = thesmallbudget, 200,704 px, so tiny images are scaled UP), both dimensions rounded to multiples of 28 (minimum 28); resize only when the target differs; alpha (RGBA/LA/PA/palette with transparency) keeps PNG, otherwise JPEG quality 90. - Error paths: containment/
ValueErrorfrom the media reader ->NodeUserError; not-an-image ->NodeUserError; missing credential row ->Connection.credentials()raisesPermissionError(rendered byBaseNode.execute()as the annotated credential envelope), an empty stored key ->NodeUserError; provider SDK exceptions are not caught here — they propagate as genuine errors (full traceback), not asLLMError->NodeUserError(this path bypassesChatUnifier).
Side Effects
- Database writes: none. Token usage is deliberately NOT recorded through the pricing service (same stance as the LLM-backed translate providers): this path bills provider tokens the LLM pricing layer has no per-call meter for.
- Broadcasts: none.
- External API calls: exactly one — OpenAI
POST /v1/chat/completions(AsyncOpenAI(api_key)), Anthropicmessages.create(AsyncAnthropic(api_key)), or Geminigenerate_content(genai.Client(api_key)). The image is sent inline as base64 (data URL for OpenAI,source.type = base64for Anthropic,Part.from_bytesfor Gemini);detailis set explicitly toautofor OpenAI because omitting it has caused order-of-magnitude token regressions elsewhere. - File I/O: one contained read of the workspace image; nothing written.
- Subprocess: none.
External Dependencies
- Credentials: the provider's stored API key resolved through
ctx.connection(<provider>)->Credential.resolvefor the credential whose id equals the provider string (OpenAICredential/AnthropicCredential/GeminiCredentialinnodes/model/_credentials.py). The node declares nocredentialstuple of its own. - Services:
services.media(read_media_bytes,resolve_mediacontainment,MEDIA_MAX_READ_BYTES),services.media.image_fit,services.llm.config.get_provider_config. - Python packages:
Pillow;openai,anthropic,google-genai(imported lazily per provider). - Task queue:
TaskQueue.AI_HEAVY. Annotations:readonly,open_world(image content leaves the machine).
Edge cases & known limits
- Workspace-relative paths only: a
mnt/...mount path from thedatatool is not resolvable here —copy_to_workspacefirst. - Output is capped at 1,024 tokens per call (
_MAX_OUTPUT_TOKENS), not configurable. budgetcontrols the pixel area sent, not the provider's own tiling: 28 px is the Anthropic patch edge used as a single default for every provider.llm_defaults.json providers.<p>.vision.enabled(the native-image-block gate) is irrelevant to this tool — it always sends the image, regardless of that flag.- Gemini uses the plain
AIzaAPI-key client only; Vertex/ADC project auth is not supported on this path. - An empty provider response yields
text: ""withsuccess: true— no retry, no error. - Every call re-reads and re-encodes the image; there is no cache across repeated questions about the same file.
Related
- Skills using this as a tool: none.
- Other nodes that consume this output: none directly; pairs with
dataSource(path discovery,llm_mediaopt-in for natively vision-capable hosts) andcanvas/ browser screenshot FileRefs as inputs. - Architecture docs: data_node.md (vision section + native image blocks), media_transport.md, native_llm_sdk.md.