👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
State Rendering Notes for Computer Use 1.0.770
Date: 2026-05-07
This note records local reverse-engineering findings from the bundled official computer-use 1.0.770 app. The goal is to keep future open-computer-use state-rendering changes tied to observed evidence rather than one-off chat context.
Evidence Sources
- Bundle inspected:
Codex Computer Use.appSkyComputerUseClient.appSkyComputerUseService
- Commands used:
strings -a .../SkyComputerUseClientstrings -a .../SkyComputerUseServiceotool -Iv .../SkyComputerUseServicenm -m .../SkyComputerUseService
Observed Facts
SkyComputerUseClientcontains the visible host wrapper prefixComputer Use state (CUA App Version: ...).- This suggests the
<app_state>style wrapping is at least partly client / host-side, not necessarily the raw MCP state renderer.
- This suggests the
SkyComputerUseServicecontains renderer-related field names:rolesubroleroleDescriptiontitledescriptionvaluetruncationRangevalueDescriptionvalueTypeNameplaceholderValuehelpidentifierdomClassListactionDescriptionsisSelectedisSelectableisFocusedisFocusableisEnabledisDisclosingisDisclosableisValueSettableattributedStringtransformedString
SkyComputerUseServicecontains AX attribute strings used by the renderer / tree builder:AXPlaceholderValueAXValueDescriptionAXRoleDescriptionAXSelectedChildrenAXFocusedUIElementAXManualAccessibilityAXEnhancedUserInterfaceAXAttributedStringForTextMarkerRangeAXStringForTextMarkerRangeAXTextualContextAXVisibleCharacterRangeAXVisibleChildren
SkyComputerUseServicecontains tree transform names:filterActionsassociateTitleUIElementsflattenIntoSelectableAncestorpruneNonDescriptiveSubtreespruneEmptyDisabledElementsmergeSingleItemGroupsflattenRedundantHierarchyflattenRepetitiveStaticTextflattenLinksIntoMarkdownTextmergeTextOnlySiblingsremoveElementsUnderCalendarEventsretrieveAttributedTextFromTextAreasretrieveAttributedTextFromWebAreas
SkyComputerUseServiceimports AX and window APIs used by the open implementation too:AXUIElementCopyAttributeValueAXUIElementCopyMultipleAttributeValuesAXUIElementPerformActionAXUIElementSetAttributeValueCGWindowListCopyWindowInfokCGWindowIsOnscreenSCWindow
SkyComputerUseServicecontains window error strings including:noWindowsAvailablecgWindowNotFoundnoMatchingWindow
SkyComputerUseServicecontains interaction / action strings including:Secondary ActionsAXRaiseAXPressAXShowMenuAXScrollToVisibleAXScrollLeftByPageAXScrollRightByPageAXScrollUpByPageAXScrollDownByPage
Current Open Implementation Follow-Ups
Already aligned in recent local changes:
- Hidden Lark / Electron windows are recovered before returning
cgWindowNotFound. - Real AX tree root lines render at depth 0, matching official
0 standard window ...shape. AXPlaceholderValue/AXPlaceholderis rendered asPlaceholder: ...when not duplicating title, description, or value.
Promising next targets:
- Add a careful
AXValueDescriptionpath for controls where rawAXValueis missing, numeric, or not human-readable. - Compare official link flattening with current
AXLinkrendering; official strings explicitly mentionflattenLinksIntoMarkdownText. - Compare text-area and web-area extraction against official
retrieveAttributedTextFromTextAreas/retrieveAttributedTextFromWebAreas. - Revisit generic tree transforms against official transform names, especially
flattenRedundantHierarchy,flattenRepetitiveStaticText, andmergeTextOnlySiblings.
Caution
The official binary strings identify available fields and transforms, but not the exact rendering order or all branch conditions. Treat this document as evidence for choosing the next investigation target, not as a complete source-level reconstruction.