Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Align set_value with settable elements

Leo8e9731fa70c5
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index a3ba47a..be23ba5 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -67,7 +67,7 @@
- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。
- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:
- `AXUIElementPerformAction`
- - `AXUIElementSetAttributeValue`
+ - `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口
- element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 会优先重复可用的 AX action,而不是直接退到全局鼠标事件
- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素
- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台
diff --git a/docs/histories/2026-04/20260422-1050-align-set-value-settable-boundary.md b/docs/histories/2026-04/20260422-1050-align-set-value-settable-boundary.md
new file mode 100644
index 0000000..f4cd05e
--- /dev/null
+++ b/docs/histories/2026-04/20260422-1050-align-set-value-settable-boundary.md
@@ -0,0 +1,22 @@
+## [2026-04-22 10:50] | Task: 对齐官方 `set_value` 的 settable 边界
+
+### 用户诉求
+
+> 按照官方姿势处理 Sublime 这类 `set_value` 失败路径。
+
+### 本次改动
+
+- **收敛 `set_value` 边界**: 真实 app 的 `set_value` 现在先检查 `AXUIElementIsAttributeSettable(kAXValueAttribute)`,只有目标确认为 settable 才调用 `AXUIElementSetAttributeValue`。
+- **官方风格错误**: 对 Sublime 这类可读 `AXValue` 但不可设置的元素,返回 `Cannot set a value for an element that is not settable`,不再裸露 `AXUIElementSetAttributeValue failed with -25200`。
+- **避免语义漂移**: 没有在 `set_value` 内部 fallback 到 `type_text`、剪贴板或未公开的 `AXReplaceRangeWithText`,保持它是“设置 settable accessibility element”的语义。
+- **补充回归测试和架构说明**: 新增 settable gate 的单元测试,并同步 `docs/ARCHITECTURE.md` 的 action tool 边界。
+
+### 设计动机
+
+官方 bundled app 的 tool 描述和二进制错误文案都显示 `set_value` 面向 settable accessibility element。Sublime 的正文节点虽然能读到 `AXValue`,但 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 返回 success + false;此时继续强行 `AXUIElementSetAttributeValue` 只会得到底层 `kAXErrorFailure(-25200)`。先做 settable gate 能把失败解释为能力边界,而不是伪装成输入模拟失败。
+
+### 影响文件
+
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index d3758df..94cb289 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -61,6 +61,16 @@ func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {
return ["1", "true", "yes", "on"].contains(rawValue)
}
+let nonSettableSetValueErrorMessage = "Cannot set a value for an element that is not settable"
+
+func setValueAttributeIsSettable(result: AXError, settable: Bool, attribute: String) throws -> Bool {
+ guard result == .success else {
+ throw ComputerUseError.message("AXUIElementIsAttributeSettable(\(attribute)) failed with \(result.rawValue)")
+ }
+
+ return settable
+}
+
public final class ComputerUseService {
private var snapshotsByApp: [String: AppSnapshot] = [:]
@@ -288,6 +298,10 @@ public final class ComputerUseService {
throw ComputerUseError.stateUnavailable("element \(elementIndex) has no backing accessibility object")
}
+ guard try isSettableForSetValue(element: element, attribute: kAXValueAttribute) else {
+ throw ComputerUseError.message(nonSettableSetValueErrorMessage)
+ }
+
let cursorTarget = visualCursorTarget(for: record, snapshot: snapshot)
moveVisualCursor(to: cursorTarget)
@@ -441,6 +455,16 @@ public final class ComputerUseService {
return result == .success && settable.boolValue
}
+ private func isSettableForSetValue(element: AXUIElement, attribute: String) throws -> Bool {
+ var settable = DarwinBoolean(false)
+ let result = AXUIElementIsAttributeSettable(element, attribute as CFString, &settable)
+ return try setValueAttributeIsSettable(
+ result: result,
+ settable: settable.boolValue,
+ attribute: attribute
+ )
+ }
+
private func bestElement(containing point: CGPoint, in snapshot: AppSnapshot) -> ElementRecord? {
snapshot.elements.values
.filter { $0.localFrame?.contains(point) ?? false }
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 58e066f..565bcb3 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -363,6 +363,21 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "false"]))
}
+ func testSetValueAttributeGateMatchesOfficialSettableBoundary() throws {
+ XCTAssertTrue(try setValueAttributeIsSettable(result: .success, settable: true, attribute: kAXValueAttribute))
+ XCTAssertFalse(try setValueAttributeIsSettable(result: .success, settable: false, attribute: kAXValueAttribute))
+ XCTAssertEqual(nonSettableSetValueErrorMessage, "Cannot set a value for an element that is not settable")
+
+ XCTAssertThrowsError(
+ try setValueAttributeIsSettable(result: .attributeUnsupported, settable: false, attribute: kAXValueAttribute)
+ ) { error in
+ XCTAssertEqual(
+ (error as? ComputerUseError)?.errorDescription,
+ "AXUIElementIsAttributeSettable(AXValue) failed with -25205"
+ )
+ }
+ }
+
func testMakeVisualCursorTargetUsesWindowRelativeElementCenter() {
let target = makeVisualCursorTarget(
localFrame: CGRect(x: 24, y: 32, width: 120, height: 48),