👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Document official state renderer findings
Leof42aad254b11
diff --git a/docs/histories/2026-05/20260507-2015-document-official-state-rendering.md b/docs/histories/2026-05/20260507-2015-document-official-state-rendering.mdnew file mode 100644index 0000000..a798009--- /dev/null+++ b/docs/histories/2026-05/20260507-2015-document-official-state-rendering.md@@ -0,0 +1,20 @@+# 记录官方 state renderer 逆向线索++## 用户诉求++继续结合逆向结果优化 `open-computer-use`,让 Lark / Electron app 的工具返回逐步接近官方 `computer-use`。++## 主要改动++- 新增 `docs/references/codex-computer-use-reverse-engineering/state-rendering-1.0.770.md`。+- 记录官方 `computer-use` 1.0.770 client / service binary 中和 state rendering 相关的字符串、AX 字段、tree transform、窗口错误与 action 线索。+- 更新 reverse-engineering README,把新文档纳入导航。++## 设计动机++后续 renderer 对齐不应该只依赖一次工具输出的肉眼对比。把官方 binary 中稳定可观察的字段和 transform 名称沉到仓库里,可以帮助后续 agent 选择更有证据的迭代方向。++## 受影响文件++- `docs/references/codex-computer-use-reverse-engineering/README.md`+- `docs/references/codex-computer-use-reverse-engineering/state-rendering-1.0.770.md`diff --git a/docs/references/codex-computer-use-reverse-engineering/README.md b/docs/references/codex-computer-use-reverse-engineering/README.mdindex faf4552..45f09c6 100644--- a/docs/references/codex-computer-use-reverse-engineering/README.md+++ b/docs/references/codex-computer-use-reverse-engineering/README.md@@ -26,6 +26,8 @@- 针对视频里 5 个 slider,记录 shipping bundle 中是否还带对应 label phrase、它们与当前 binary-confirmed 几何 / spring 量的映射关系,以及对实际曲线的敏感性分析。- `permission-onboarding.md`- 对 Accessibility / Screen Recording 权限引导和 System Settings accessory window 的分析。+- `state-rendering-1.0.770.md`+ - 对官方 `computer-use` 1.0.770 state renderer 字符串、AX 字段、tree transform 和窗口恢复线索的整理,用于继续收敛 Lark / Electron app state 输出。- `assets/README.md`- 直接从官方 bundle 导出的可视资源归档,包括 `SoftwareCursor`、`HintArrow` 和 cursor icon 资产。diff --git a/docs/references/codex-computer-use-reverse-engineering/state-rendering-1.0.770.md b/docs/references/codex-computer-use-reverse-engineering/state-rendering-1.0.770.mdnew file mode 100644index 0000000..28f4ce8--- /dev/null+++ b/docs/references/codex-computer-use-reverse-engineering/state-rendering-1.0.770.md@@ -0,0 +1,115 @@+# State Rendering Notes for Computer Use 1.0.770++Date: 2026-05-07++This note records local reverse-engineering findings from the bundled official `computer-use` 1.0.770 app. The goal is to keep future `open-computer-use` state-rendering changes tied to observed evidence rather than one-off chat context.++## Evidence Sources++- Bundle inspected:+ - `Codex Computer Use.app`+ - `SkyComputerUseClient.app`+ - `SkyComputerUseService`+- Commands used:+ - `strings -a .../SkyComputerUseClient`+ - `strings -a .../SkyComputerUseService`+ - `otool -Iv .../SkyComputerUseService`+ - `nm -m .../SkyComputerUseService`++## Observed Facts++- `SkyComputerUseClient` contains the visible host wrapper prefix `Computer Use state (CUA App Version: ...)`.+ - This suggests the `<app_state>` style wrapping is at least partly client / host-side, not necessarily the raw MCP state renderer.+- `SkyComputerUseService` contains renderer-related field names:+ - `role`+ - `subrole`+ - `roleDescription`+ - `title`+ - `description`+ - `value`+ - `truncationRange`+ - `valueDescription`+ - `valueTypeName`+ - `placeholderValue`+ - `help`+ - `identifier`+ - `domClassList`+ - `actionDescriptions`+ - `isSelected`+ - `isSelectable`+ - `isFocused`+ - `isFocusable`+ - `isEnabled`+ - `isDisclosing`+ - `isDisclosable`+ - `isValueSettable`+ - `attributedString`+ - `transformedString`+- `SkyComputerUseService` contains AX attribute strings used by the renderer / tree builder:+ - `AXPlaceholderValue`+ - `AXValueDescription`+ - `AXRoleDescription`+ - `AXSelectedChildren`+ - `AXFocusedUIElement`+ - `AXManualAccessibility`+ - `AXEnhancedUserInterface`+ - `AXAttributedStringForTextMarkerRange`+ - `AXStringForTextMarkerRange`+ - `AXTextualContext`+ - `AXVisibleCharacterRange`+ - `AXVisibleChildren`+- `SkyComputerUseService` contains tree transform names:+ - `filterActions`+ - `associateTitleUIElements`+ - `flattenIntoSelectableAncestor`+ - `pruneNonDescriptiveSubtrees`+ - `pruneEmptyDisabledElements`+ - `mergeSingleItemGroups`+ - `flattenRedundantHierarchy`+ - `flattenRepetitiveStaticText`+ - `flattenLinksIntoMarkdownText`+ - `mergeTextOnlySiblings`+ - `removeElementsUnderCalendarEvents`+ - `retrieveAttributedTextFromTextAreas`+ - `retrieveAttributedTextFromWebAreas`+- `SkyComputerUseService` imports AX and window APIs used by the open implementation too:+ - `AXUIElementCopyAttributeValue`+ - `AXUIElementCopyMultipleAttributeValues`+ - `AXUIElementPerformAction`+ - `AXUIElementSetAttributeValue`+ - `CGWindowListCopyWindowInfo`+ - `kCGWindowIsOnscreen`+ - `SCWindow`+- `SkyComputerUseService` contains window error strings including:+ - `noWindowsAvailable`+ - `cgWindowNotFound`+ - `noMatchingWindow`+- `SkyComputerUseService` contains interaction / action strings including:+ - `Secondary Actions`+ - `AXRaise`+ - `AXPress`+ - `AXShowMenu`+ - `AXScrollToVisible`+ - `AXScrollLeftByPage`+ - `AXScrollRightByPage`+ - `AXScrollUpByPage`+ - `AXScrollDownByPage`++## Current Open Implementation Follow-Ups++Already aligned in recent local changes:++- Hidden Lark / Electron windows are recovered before returning `cgWindowNotFound`.+- Real AX tree root lines render at depth 0, matching official `0 standard window ...` shape.+- `AXPlaceholderValue` / `AXPlaceholder` is rendered as `Placeholder: ...` when not duplicating title, description, or value.++Promising next targets:++- Add a careful `AXValueDescription` path for controls where raw `AXValue` is missing, numeric, or not human-readable.+- Compare official link flattening with current `AXLink` rendering; official strings explicitly mention `flattenLinksIntoMarkdownText`.+- Compare text-area and web-area extraction against official `retrieveAttributedTextFromTextAreas` / `retrieveAttributedTextFromWebAreas`.+- Revisit generic tree transforms against official transform names, especially `flattenRedundantHierarchy`, `flattenRepetitiveStaticText`, and `mergeTextOnlySiblings`.++## Caution++The official binary strings identify available fields and transforms, but not the exact rendering order or all branch conditions. Treat this document as evidence for choosing the next investigation target, not as a complete source-level reconstruction.