Self-improving AI Employees for everyone
Data (dataSource)
| Field | Value |
|---|---|
| Category | tool / ai (palette group tool) |
| Backend | server/nodes/tool/data_source/__init__.py::DataSourceNode.data (dispatch via execute_as_tool -> BaseNode.execute() + @Operation("data"); plugin folder: _paths.py two-namespace resolution, _readers.py bounded typed readers, _handlers.py Data-panel WS API; machine-wide allowlist in services/data/mount_store.py) |
| WS handlers | data_list_mounts / data_add_mount / data_update_mount / data_remove_mount / data_browse (self-registered via register_ws_handlers; @ws_response, external-socket + authenticated-owner checks, browse additionally requires a saved workflow that owns exactly one dataSource node with the given id) |
| Tests | server/tests/nodes/test_data_source_node.py (spec, ToolInput validation, path security, workspace tiers, mount flows, handler security); server/tests/services/data/test_mount_store.py (mount validation matrix) |
| Skill (if any) | none |
| Dual-purpose tool | ToolNode - tool name data (tool_schema_locked = True, split schema: LLM sees DataToolInput, operator config DataToolParams is server_controlled_fields = {mounts}) |
Purpose
An agent-facing raw-data reader/writer over two path namespaces: the
per-workflow workspace (reports/q3.csv) and operator-approved external
mounts (mnt/<mount_name>/<rel>). Reads dispatch to typed, bounded tiers
(text / csv / json / pdf / html / xlsx / image metadata / binary) so a 20 MB
spreadsheet cannot become a blown Temporal payload; binary content travels
as references, never bytes. Writes go to the workspace or to mounts whose
operator flipped the writable flag. There is deliberately no delete —
deletion stays a human action in the gallery panel. copy_to_workspace
imports a mount file as a real FileRef, which is the bridge to previews,
drags and the visionAnalyze tool.
Inputs (handles)
| Handle | Connection type | Required | Purpose |
|---|---|---|---|
output-tool (source, top, label "Data") |
tools | yes (only way to use it) | Connect to an agent's input-tools; the agent calls data(...) |
No input-main. uiHints: isDataPanel (renders
client/src/components/parameterPanel/DataPanel.tsx), hideInputSection,
hideOutputSection, hideRunButton; isConfigNode auto-derived from the
tool group. A framework-side execution with only Params (no tool args)
is treated as a harmless root list.
Parameters
Operator config, DataToolParams (extra="ignore"; persisted via the Data
panel, never exposed to the model):
| Name | Type | Default | Required | displayOptions.show | Description |
|---|---|---|---|---|---|
mounts |
string[] (title "Enabled Mounts") | [] |
no | - | Names of machine-wide mounts this node exposes. A "" or JSON-string value is coerced (""/None -> [], JSON array string -> list, other string -> [string]) |
LLM-visible arguments, DataToolInput (extra="forbid"):
| Name | Type | Default | Required | Description |
|---|---|---|---|---|
operation |
enum list | read | search | metadata | write | append | copy_to_workspace |
- | yes | Operation |
path |
string, max_length=4096 |
"" |
required (non-empty) by read, metadata, write, append, copy_to_workspace |
Workspace-relative path or mnt/<mount_name>/<file>; empty list = workspace root + enabled mounts |
pattern |
string | null, max_length=256 |
null |
required by search |
Filename glob or bare term (a term without glob metacharacters becomes *term* via search_to_pattern) |
as_type |
enum auto | text | csv | json | pdf | html | xlsx | image | binary |
auto |
no | read: force a tier instead of extension detection |
offset |
integer, ge=0 |
0 |
no | read: rows/lines/pages to skip |
limit |
integer, ge=1, le=500 |
100 |
no | rows/lines/pages/entries cap (shared by list, read, search) |
sheet |
string | null, max_length=128 |
null |
no | read xlsx: sheet name (default: active sheet) |
encoding |
string | null, max_length=32 |
null |
no | read: force text encoding (else utf-8 -> utf-8-sig -> latin-1 with encoding_guessed: true) |
content |
string | null, max_length=200000 |
null |
required by write, append |
UTF-8 text |
dest |
string | null, max_length=4096 |
null |
no | copy_to_workspace destination (default imports/<filename>) |
Per-operation required-field failures raise ValueError("<op> requires a, b") at validation, returned to the model as
{"error": "Invalid tool input/configuration: ...", "error_type": "ValidationError"}.
Outputs (handles)
| Handle | Shape | Description |
|---|---|---|
| (tool result) | object | DataToolOutput (extra="allow"), passed through bound_result (200,000-byte serialized ceiling) |
Output payload (TypeScript shape)
type Common = {
operation: string;
source: 'workspace' | 'mount';
path: string; // canonical round-trippable virtual path
mount?: string; // mount name (mount source only)
truncated?: boolean; // set by any cap, including bound_result
};
type WorkspaceEntry = { /* gallery row */ name: string; path: string; is_dir: boolean; mime_type: string | null;
size_bytes: number; modified_at: string | null; ref: FileRef | null; preview: string | null; url?: string | null };
type MountEntry = { location: string; mount: string; name: string; is_dir: boolean; mime_type: string | null;
size_bytes: number; modified_at: number | null; writable: boolean }; // never a FileRef, never a host path
// list
Common & { entries: WorkspaceEntry[] | MountEntry[]; count: number;
mounts?: Array<{ name: string; path: string; writable: boolean; available: boolean }> }; // root listing only
// search
Common & { pattern: string; entries: WorkspaceEntry[] | MountEntry[]; count: number };
// read (tier-dependent)
Common & (
| { type: 'text'; text: string; lines_total: number; offset: number; limit: number; encoding: string; encoding_guessed: boolean }
| { type: 'csv'; columns: string[]; rows: string[][]; rows_total: number; offset: number; delimiter: string; encoding: string; encoding_guessed: boolean; columns_truncated: boolean }
| { type: 'json'; data: unknown; pruned_paths: string[]; encoding: string; encoding_guessed: boolean }
| { type: 'pdf'; pages: Array<{ page: number; text: string; truncated: boolean }>; pages_total: number; offset: number }
| { type: 'html'; title: string | null; text: string; lines_total: number; offset: number; encoding: string; encoding_guessed: boolean }
| { type: 'xlsx'; sheet: string; sheets: string[]; columns: unknown[]; rows: unknown[][]; rows_total: number; offset: number; columns_truncated: boolean }
| { type: 'image'; image: { width: number; height: number; format: string; mode: string }; ref?: FileRef; location?: string;
llm_media?: [{ ref: FileRef; detail: 'auto' }] } // workspace refs with png/jpeg/webp/gif mime only
| { type: 'binary'; mime_type: string; size_bytes: number; sha256: string; ref?: FileRef; location?: string }
);
// metadata
Common & { type: string; name: string; mime_type: string; size_bytes: number; modified_at: string; sha256: string;
ref?: FileRef; location?: string; image?: { width: number; height: number; format: string; mode: string } };
// write / append
Common & { bytes_written: number; created: boolean; appended: boolean };
// copy_to_workspace
{ operation: 'copy_to_workspace'; source: 'workspace'; path: string; copied_from: string; ref: FileRef };
Logic Flow
flowchart TD
A[execute_as_tool: validate ToolInput, Params separately; _tool_config on ctx] --> B[data op: mounts = _tool_config.mounts]
B --> C{operation}
C -- list --> L1{path is mnt/...?}
L1 -- yes --> L2[resolve_data_path -> must be dir -> os.scandir -> mount_entry rows, dirs first, cap limit]
L1 -- no --> L3[gallery list_directory on workspace_root; root path also appends _mount_summaries]
C -- search --> S1[search_to_pattern; empty -> NodeUserError]
S1 --> S2{mount?}
S2 -- yes --> S3[walk_mount: os.walk no symlink follow, depth 12, 5000 visited, 200 results, fnmatch on name]
S2 -- no --> S4[gallery list_matching recursive glob]
C -- read --> R1[_resolve_file: exists, not dir, <= MEDIA_MAX_READ_BYTES]
R1 --> R2[detect_tier: as_type wins, else extension, else mime]
R2 -- image --> R3[Pillow header probe + ref/location; llm_media when workspace ref mime allowlisted]
R2 -- binary --> R4[sha256 + mime + size + ref/location - never bytes]
R2 -- text/csv/json/pdf/html/xlsx --> R5[bounded reader in asyncio.to_thread]
C -- metadata --> M1[_resolve_file -> stat, tier, mime, sha256, ref/location, image dims if image]
C -- write/append --> W1[resolve_data_path for_write: mount must be writable, workspace always]
W1 --> W2[path lock; append = open ab, write = atomic_write_bytes; mkdir parents]
C -- copy_to_workspace --> K1[path must be mnt/...; dest workspace-relative, never mnt/]
K1 --> K2[never overwrite: suffix -1..-100; size <= MEDIA_MAX_READ_BYTES]
K2 --> K3[shutil.copyfile under path lock -> FileRef]
L2 & L3 & S3 & S4 & R3 & R4 & R5 & M1 & W2 & K3 --> Z[setdefault operation; bound_result; DataToolOutput]
Decision Logic
- Namespace resolution (
resolve_data_path): first segmentmntis reserved. A mount path needs a name (NodeUserErrorlisting the enabled mounts), the name must be in the node'smountssubset AND still present in thedata_mountstable for the owner (owner_id=ctx.user_id/ctx.raw.user_id/"owner"), otherwiseNodeUserError. Containment: reads useresolve_within, mutationsresolve_entry_within(parent proven, basename appended unresolved); both roots are symlink-safe;ValueErrorfrom the backend helpers is translated toNodeUserError. - Write gate: mounts require
writable; writing to a mount root or the workspace root is refused; a directory target is refused. The workspace is always writable. - Tier detection (
detect_tier): explicitas_typewins;.csv/.tsv-> csv,.json-> json,.pdf-> pdf,.html/.htm/.xhtml-> html,.xlsx/.xlsm-> xlsx, a fixed text-extension set -> text; then mimeimage/*-> image,text/*-> text; else binary. - Bounds (
_readers.py): text window 65,536 bytes; csv 500 rows / 100 cols / 2,000 chars per cell (delimiter sniffed from the first 16 KiB,\tfallback for.tsv); json source <= 5 MiB, depth 8 withpruned_paths(50 max); pdf 20 pages per call, 20,000 chars per page; xlsx 500 rows (read_only,data_only,rows_totalfrommax_row - 1); mount search depth 12 / 5,000 visited / 200 results;bound_resulthalvesrows/pages/entries/matches/itemsthen trimstext/datauntil the envelope is <= 200,000 bytes and setstruncated. - References: workspace targets get
ref(a serializedFileRef, alwayskind: "file"viato_file_ref); mount targets getlocationonly — host paths never leave the server. - Vision opt-in: an image-tier
readof a workspace file whose mime is inIMAGE_MIME_ALLOWLIST(png,jpeg,webp,gif) addsllm_media: [{ref, detail: "auto"}], which the agent loop turns into a ref-only image block; mount images must becopy_to_workspaced first. - Error paths: every user-correctable failure is a
NodeUserError(missing file, directory-vs-file confusion, oversize, read-only mount, revoked mount, invalid JSON, bad encoding, missing optional dependency with the install hint, unknown sheet, too many copies).execute_as_toolrenders it as{"error": ...}for the model.
Side Effects
- Database reads:
data_mountsviaDataMountStore.list_mounts/get_mount(root listing and every mount resolution). - Database writes: none from the tool operation. The panel handlers write
data_mountsrows (addvalidates: absolute, exists, directory, not a filesystem root, not the home dir, no overlap withDATA_DIRor a legacy~/.machina, readable, writable-if-flagged, unique name matching^[a-z0-9][a-z0-9_-]{0,63}$, unique root;updatetoggles onlywritable). - Broadcasts: none.
- External API calls: none.
- File I/O: reads under the workspace or mount roots;
write/appendcreate parent directories,writeis atomic (atomic_write_bytes),appendopens inab;copy_to_workspacecopies bytes into the workspace; all mutations take the per-path lock fromnodes/filesystem/_backend. - Subprocess: none.
External Dependencies
- Credentials: none.
- Services:
services.media.workspace(workspace_root,workspace_file_url),services.media.limits.MEDIA_MAX_READ_BYTES,nodes/filesystem/_backend(resolve_within,resolve_entry_within,atomic_write_bytes,get_path_lock),nodes/filesystem/gallery/_service(list_directory,list_matching,search_to_pattern,to_file_ref),services.data.mount_store,services.llm.media.IMAGE_MIME_ALLOWLIST. - Python packages: stdlib for text/csv/json/binary;
pypdf(docsextra) for pdf;beautifulsoup4(docsextra) for html;openpyxlfor xlsx;Pillowfor image metadata. Each missing package surfaces as aNodeUserErrorwith the install hint (html additionally suggestsas_type='text'). - Environment variables: none directly (
DATA_DIRshapes the protected roots viacore.paths). - Task queue:
TaskQueue.DEFAULT. Annotations: not destructive, not readonly, not open_world.
Edge cases & known limits
limitis one knob for three different things (listing entries, read rows/lines/pages, search results); a model paging a CSV withlimit=500also caps its next listing at 500.readandmetadatacompute a full SHA-256 of binary files (up to 25 MiB) on every call.liston the workspace passeslimitto the gallery'slist_directory, which applies its ownWORKSPACE_LIST_LIMITcap in addition.- Mount
searchmatches the file name only (fnmatchonname.lower()), never the relative directory; workspacesearchuses the backend's recursive glob. - Mount listings expose symlinked files (walks do not follow symlinked directories); reading one later still goes through
resolve_within, which rejects escapes. copy_to_workspacesuffixing stops after 100 attempts with aNodeUserError.writeof an emptycontentis rejected by validation (required fields are checked with truthiness), so an agent cannot create an empty file.- Windows backslashes are normalized to
/;_cleanalso strips leading/trailing slashes, so/mnt/x/fandmnt/x/fare the same path. - Mount availability is re-checked on every call: a mount removed in the panel fails immediately even if a stale node still lists it (
available: falsein the root listing). - No delete, no rename, no move — by design.
Related
- Skills using this as a tool: none (the
tool_descriptioncarries the usage contract). - Other nodes that consume this output:
visionAnalyze(takes workspace-relative image paths discovered here),canvas(displaysrefs),gallery(human-side deletion). - Architecture docs: data_node.md, media_transport.md, plugin_system.md.