Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
title: "ADR-0002: UIA-Tree-Pruning" slug: adr-0002-uia-tree-pruning diataxis: adr status: active owner: maintainers last_reviewed: 2026-04-29 phase: 5 audience: developer
ADR-0002 — UIAutomation tree-pruning strategy
Status: Accepted (2026-04-22) Phase: 5 — Vision Capability
Context
The CU harness needs structured UI access so that vision hallucinations ("click OK at (340,512)") do not become the default case — mandate §150. Naive UIA tree traversal (via pywinauto.uia_element_info) returns 3,000–8,000 nodes for Chrome/VSCode/Slack, each with 20+ properties. Serializing alone takes seconds, and sending a 5,000-node tree to Sonnet costs ~60k input tokens — per step.
Decision
Three-stage pruning before handing off to the LLM, order of application:
- Depth limit 6 — from the root window. Deeper children are collapsed as
{"children": "…N nodes omitted"}. Rationale: relevant controls rarely sit deeper than 6 levels. - Interesting-role filter — whitelist:
Button, Edit, ComboBox, List, ListItem, Tab, MenuItem, CheckBox, RadioButton, Hyperlink, Text(only whenName != ""). Everything else (panes, groups, decorations, separators) is passed through as a container but not emitted itself, unless it carries a non-trivialAutomationId. - OnScreen-rect filter — an element is only included if its
BoundingRectanglelies at least 50% within the primary monitor and is not completely covered by higher Z-order windows (heuristic:IsOffscreen == False).
Goal: ≤ 150 nodes per observation. If exceeded: a second pruning round with depth 5, then depth 4. If still > 150 → fallback_to_screenshot_only = True in the observation event.
Serialized format: compact, one node = {"r":"Button","n":"Save","id":"btnSave","b":[100,200,180,32],"p":<parent_idx>}. No redundancy, no whitespace.
Consequences
- Token budget per observation ~3k–5k instead of 60k → the CU loop becomes economically viable.
- Deterministic, no ML training needed.
- The pruning stages are independently configurable in
jarvis.toml:[vision.pruning].
- False negatives: very deep, rarely used controls (e.g. 8 levels deep in a TreeView) are missed. Mitigation: if the LLM says "I don't see the element I'm looking for", the depth limit temporarily goes to 10 (one retry pass).
- Dependency on
pywinauto.uia_element_info.UIAElementInfo— the alternativeuiautomationpackage is NOT needed inrequirements, but might be more robust for modern WinUI3 apps. Worth keeping an eye on.
Alternatives Considered
- Full tree to the LLM: Cost/latency unacceptable, mandate §150 effectively forbids it.
- Screenshot-only: loses actionable IDs, vision hallucinates more. A single screenshot is a fallback, not the default.
- ML-based pruning (e.g. a small local model that classifies „interesting" nodes): overhead (training data, model loading, additional dep). YAGNI for Phase 5.
- Screenpipe-style OCR+layout: needs external infrastructure, no structured click handle. Rejected.
Configurability
[vision.pruning]
max_depth = 6
max_nodes = 150
interesting_roles = ["Button", "Edit", "ComboBox", "List", "ListItem",
"Tab", "MenuItem", "CheckBox", "RadioButton",
"Hyperlink", "Text"]
fallback_to_screenshot_if_overflow = true