👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Align set_value with settable elements
Leo8e9731fa70c5
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex a3ba47a..be23ba5 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -67,7 +67,7 @@- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:- `AXUIElementPerformAction`- - `AXUIElementSetAttributeValue`+ - `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口- element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 会优先重复可用的 AX action,而不是直接退到全局鼠标事件- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台diff --git a/docs/histories/2026-04/20260422-1050-align-set-value-settable-boundary.md b/docs/histories/2026-04/20260422-1050-align-set-value-settable-boundary.mdnew file mode 100644index 0000000..f4cd05e--- /dev/null+++ b/docs/histories/2026-04/20260422-1050-align-set-value-settable-boundary.md@@ -0,0 +1,22 @@+## [2026-04-22 10:50] | Task: 对齐官方 `set_value` 的 settable 边界++### 用户诉求++> 按照官方姿势处理 Sublime 这类 `set_value` 失败路径。++### 本次改动++- **收敛 `set_value` 边界**: 真实 app 的 `set_value` 现在先检查 `AXUIElementIsAttributeSettable(kAXValueAttribute)`,只有目标确认为 settable 才调用 `AXUIElementSetAttributeValue`。+- **官方风格错误**: 对 Sublime 这类可读 `AXValue` 但不可设置的元素,返回 `Cannot set a value for an element that is not settable`,不再裸露 `AXUIElementSetAttributeValue failed with -25200`。+- **避免语义漂移**: 没有在 `set_value` 内部 fallback 到 `type_text`、剪贴板或未公开的 `AXReplaceRangeWithText`,保持它是“设置 settable accessibility element”的语义。+- **补充回归测试和架构说明**: 新增 settable gate 的单元测试,并同步 `docs/ARCHITECTURE.md` 的 action tool 边界。++### 设计动机++官方 bundled app 的 tool 描述和二进制错误文案都显示 `set_value` 面向 settable accessibility element。Sublime 的正文节点虽然能读到 `AXValue`,但 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 返回 success + false;此时继续强行 `AXUIElementSetAttributeValue` 只会得到底层 `kAXErrorFailure(-25200)`。先做 settable gate 能把失败解释为能力边界,而不是伪装成输入模拟失败。++### 影响文件++- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex d3758df..94cb289 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -61,6 +61,16 @@ func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {return ["1", "true", "yes", "on"].contains(rawValue)}+let nonSettableSetValueErrorMessage = "Cannot set a value for an element that is not settable"++func setValueAttributeIsSettable(result: AXError, settable: Bool, attribute: String) throws -> Bool {+ guard result == .success else {+ throw ComputerUseError.message("AXUIElementIsAttributeSettable(\(attribute)) failed with \(result.rawValue)")+ }++ return settable+}+public final class ComputerUseService {private var snapshotsByApp: [String: AppSnapshot] = [:]@@ -288,6 +298,10 @@ public final class ComputerUseService {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no backing accessibility object")}+ guard try isSettableForSetValue(element: element, attribute: kAXValueAttribute) else {+ throw ComputerUseError.message(nonSettableSetValueErrorMessage)+ }+let cursorTarget = visualCursorTarget(for: record, snapshot: snapshot)moveVisualCursor(to: cursorTarget)@@ -441,6 +455,16 @@ public final class ComputerUseService {return result == .success && settable.boolValue}+ private func isSettableForSetValue(element: AXUIElement, attribute: String) throws -> Bool {+ var settable = DarwinBoolean(false)+ let result = AXUIElementIsAttributeSettable(element, attribute as CFString, &settable)+ return try setValueAttributeIsSettable(+ result: result,+ settable: settable.boolValue,+ attribute: attribute+ )+ }+private func bestElement(containing point: CGPoint, in snapshot: AppSnapshot) -> ElementRecord? {snapshot.elements.values.filter { $0.localFrame?.contains(point) ?? false }diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 58e066f..565bcb3 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -363,6 +363,21 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "false"]))}+ func testSetValueAttributeGateMatchesOfficialSettableBoundary() throws {+ XCTAssertTrue(try setValueAttributeIsSettable(result: .success, settable: true, attribute: kAXValueAttribute))+ XCTAssertFalse(try setValueAttributeIsSettable(result: .success, settable: false, attribute: kAXValueAttribute))+ XCTAssertEqual(nonSettableSetValueErrorMessage, "Cannot set a value for an element that is not settable")++ XCTAssertThrowsError(+ try setValueAttributeIsSettable(result: .attributeUnsupported, settable: false, attribute: kAXValueAttribute)+ ) { error in+ XCTAssertEqual(+ (error as? ComputerUseError)?.errorDescription,+ "AXUIElementIsAttributeSettable(AXValue) failed with -25205"+ )+ }+ }+func testMakeVisualCursorTargetUsesWindowRelativeElementCenter() {let target = makeVisualCursorTarget(localFrame: CGRect(x: 24, y: 32, width: 120, height: 48),