👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
feat(macos): complete background operation: sky_key, covered/other-Space windows, agent display (#65)
ATG281bbbb8b39d
diff --git a/Package.swift b/Package.swiftindex 32ecdf1..e43a5a6 100644--- a/Package.swift+++ b/Package.swift@@ -34,8 +34,17 @@ let package = Package(),],targets: [+ .target(+ name: "OpenComputerUseVirtualDisplayShim",+ path: "packages/OpenComputerUseVirtualDisplayShim",+ publicHeadersPath: "include",+ linkerSettings: [+ .linkedFramework("CoreGraphics"),+ ]+ ),.target(name: "OpenComputerUseKit",+ dependencies: ["OpenComputerUseVirtualDisplayShim"],path: "packages/OpenComputerUseKit/Sources/OpenComputerUseKit"),.executableTarget(diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdnew file mode 100644index 0000000..38cd295--- /dev/null+++ b/THIRD_PARTY_NOTICES.md@@ -0,0 +1,61 @@+# Third-Party Notices++## Cua Driver++The macOS `sky_click` event recipe, the private SkyLight bridge shared by `sky_click` and `sky_key`, and the target-only activation record are derived from Cua Driver:++- Project: https://github.com/trycua/cua+- Source revision: `b8a0f32a06c75225ba24ebb5ab14f6507fa90d15`+- License: MIT++MIT License++Copyright (c) 2025 Cua AI, Inc.++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.++## yabai++The macOS focus-without-raise event-record pattern and the `sky_key` make-key-window record pair (`window_manager_make_key_window`) are derived from yabai:++- Project: https://github.com/asmvik/yabai+- Source revision: `dd845723416f5fe92af49fad5ebab00369e07edd`+- License: MIT++The MIT License (MIT)++Copyright (c) 2019 Åsmund Vikane++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift b/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swiftindex cc6d657..f7b28bc 100644--- a/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift+++ b/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift@@ -32,6 +32,7 @@ final class MCPAppRuntime: NSObject, NSApplicationDelegate {queue: .main) { _ inTask { @MainActor in+ resetOpenComputerUseBackgroundWindowState()resetOpenComputerUseVisualCursor()}}@@ -39,6 +40,7 @@ final class MCPAppRuntime: NSObject, NSApplicationDelegate {}func applicationWillTerminate(_ notification: Notification) {+ resetOpenComputerUseBackgroundWindowState()if let turnEndedObserver {DistributedNotificationCenter.default().removeObserver(turnEndedObserver)}diff --git a/apps/OpenComputerUse/Sources/OpenComputerUse/MacOSAppAgentProxy.swift b/apps/OpenComputerUse/Sources/OpenComputerUse/MacOSAppAgentProxy.swiftindex bceef58..e4e764b 100644--- a/apps/OpenComputerUse/Sources/OpenComputerUse/MacOSAppAgentProxy.swift+++ b/apps/OpenComputerUse/Sources/OpenComputerUse/MacOSAppAgentProxy.swift@@ -182,6 +182,7 @@ private final class MacOSAppAgentRuntime: NSObject, NSApplicationDelegate {queue: .main) { _ inTask { @MainActor in+ resetOpenComputerUseBackgroundWindowState()resetOpenComputerUseVisualCursor()}}@@ -197,6 +198,7 @@ private final class MacOSAppAgentRuntime: NSObject, NSApplicationDelegate {}func applicationWillTerminate(_ notification: Notification) {+ resetOpenComputerUseBackgroundWindowState()if let turnEndedObserver {DistributedNotificationCenter.default().removeObserver(turnEndedObserver)}@@ -307,7 +309,10 @@ private final class AppAgentConnection: @unchecked Sendable {close(fileDescriptor)return}- defer { fclose(file) }+ defer {+ server.endSession()+ fclose(file)+ }while let line = readAgentLine(file) {let response = handle(requestLine: line)diff --git a/apps/OpenComputerUseLinux/main.go b/apps/OpenComputerUseLinux/main.goindex 783a59b..6186130 100644--- a/apps/OpenComputerUseLinux/main.go+++ b/apps/OpenComputerUseLinux/main.go@@ -24,6 +24,10 @@ var version = "0.3.6"var clickMethodValues = []string{"auto", "accessibility", "app_post", "sky_click", "global"}+var keyMethodValues = []string{"auto", "sky_key"}++var windowPlacementValues = []string{"keep", "agent_display", "restore"}+//go:embed runtime.pyvar linuxRuntimeScript string@@ -203,6 +207,13 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult {if err != nil {return textResult(err.Error(), true)}+ windowPlacement, err := parseWindowPlacement(optionalString(args, "window_placement"))+ if err != nil {+ return textResult(err.Error(), true)+ }+ if windowPlacement != "keep" {+ return textResult("window_placement '"+windowPlacement+"' is not supported on Linux", true)+ }return s.getAppState(requiredString(args, "app"), textLimit, maxTreeNodes, maxTreeDepth)case "click":clickMethod, err := parseClickMethod(optionalString(args, "click_method"))@@ -240,9 +251,17 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult {requiredFloat(args, "to_y"),)case "type_text":- return s.typeText(requiredString(args, "app"), requiredString(args, "text"))+ keyMethod, err := parseKeyMethod(optionalString(args, "key_method"))+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.typeText(requiredString(args, "app"), requiredString(args, "text"), keyMethod)case "press_key":- return s.pressKey(requiredString(args, "app"), requiredString(args, "key"))+ keyMethod, err := parseKeyMethod(optionalString(args, "key_method"))+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.pressKey(requiredString(args, "app"), requiredString(args, "key"), keyMethod)case "set_value":return s.setValue(requiredString(args, "app"), requiredElementIndex(args), requiredString(args, "value"))default:@@ -397,26 +416,32 @@ func (s *service) drag(app string, fromX, fromY, toX, toY *float64) toolCallResureturn s.actionResult(app, linuxRequest{Tool: "drag", App: app, FromX: fromX, FromY: fromY, ToX: toX, ToY: toY, WindowBounds: snapshot.WindowBounds})}-func (s *service) typeText(app, text string) toolCallResult {+func (s *service) typeText(app, text, keyMethod string) toolCallResult {if app == "" {return textResult("Missing required argument: app", true)}if text == "" {return textResult("Missing required argument: text", true)}+ if keyMethod == "sky_key" {+ return textResult("key_method 'sky_key' is not supported on Linux", true)+ }if s.currentSnapshot(app) == nil {return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)}return s.actionResult(app, linuxRequest{Tool: "type_text", App: app, Text: text})}-func (s *service) pressKey(app, key string) toolCallResult {+func (s *service) pressKey(app, key, keyMethod string) toolCallResult {if app == "" {return textResult("Missing required argument: app", true)}if key == "" {return textResult("Missing required argument: key", true)}+ if keyMethod == "sky_key" {+ return textResult("key_method 'sky_key' is not supported on Linux", true)+ }if s.currentSnapshot(app) == nil {return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)}@@ -1114,6 +1139,32 @@ func defaultString(value, fallback string) string {return value}+func parseWindowPlacement(value string) (string, error) {+ normalized := strings.ToLower(strings.TrimSpace(value))+ if normalized == "" {+ return "keep", nil+ }+ for _, candidate := range windowPlacementValues {+ if normalized == candidate {+ return normalized, nil+ }+ }+ return "", fmt.Errorf("Invalid window_placement %q. Expected one of: %s", value, strings.Join(windowPlacementValues, ", "))+}++func parseKeyMethod(value string) (string, error) {+ normalized := strings.ToLower(strings.TrimSpace(value))+ if normalized == "" {+ return "auto", nil+ }+ for _, candidate := range keyMethodValues {+ if normalized == candidate {+ return normalized, nil+ }+ }+ return "", fmt.Errorf("Invalid key_method %q. Expected one of: %s", value, strings.Join(keyMethodValues, ", "))+}+func parseClickMethod(value string) (string, error) {normalized := strings.ToLower(strings.TrimSpace(value))if normalized == "" {@@ -1169,10 +1220,11 @@ func toolDefinitions() []toolDefinition {Description: "Get the state of an already running app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`.",Annotations: readOnlyAnnotations(),InputSchema: objectSchema(map[string]any{- "app": stringProperty("App name or bundle identifier"),- "text_limit": textLimitProperty("Maximum text characters to return. Use \"max\" for full text. Defaults to 500."),- "max_tree_nodes": positiveIntegerProperty("Maximum accessibility tree nodes to render. Defaults to 1200."),- "max_tree_depth": positiveIntegerProperty("Maximum accessibility tree depth to render. Defaults to 64."),+ "app": stringProperty("App name or bundle identifier"),+ "text_limit": textLimitProperty("Maximum text characters to return. Use \"max\" for full text. Defaults to 500."),+ "max_tree_nodes": positiveIntegerProperty("Maximum accessibility tree nodes to render. Defaults to 1200."),+ "max_tree_depth": positiveIntegerProperty("Maximum accessibility tree depth to render. Defaults to 64."),+ "window_placement": enumStringProperty("keep (default) leaves the window where it is. agent_display and restore are macOS-only and not supported on Linux.", windowPlacementValues),}, []string{"app"}),},{@@ -1196,8 +1248,9 @@ func toolDefinitions() []toolDefinition {Description: "Press a key or key-combination on the keyboard, including modifier and navigation keys.\n - This supports xdotool's `key` syntax.\n - Examples: \"a\", \"Return\", \"Tab\", \"super+c\", \"Up\", \"KP_0\" (for the numpad 0). This tool is part of plugin `Computer Use`.",Annotations: defaultAnnotations(),InputSchema: objectSchema(map[string]any{- "app": stringProperty("App name or bundle identifier"),- "key": stringProperty("Key or key-combination to press"),+ "app": stringProperty("App name or bundle identifier"),+ "key": stringProperty("Key or key-combination to press"),+ "key_method": enumStringProperty("Keyboard delivery: auto (default) or sky_key. sky_key is the macOS SkyLight background-window path and is not supported on Linux.", keyMethodValues),}, []string{"app", "key"}),},{@@ -1226,8 +1279,9 @@ func toolDefinitions() []toolDefinition {Description: "Type literal text using keyboard input. This tool is part of plugin `Computer Use`.",Annotations: defaultAnnotations(),InputSchema: objectSchema(map[string]any{- "app": stringProperty("App name or bundle identifier"),- "text": stringProperty("Literal text to type"),+ "app": stringProperty("App name or bundle identifier"),+ "text": stringProperty("Literal text to type"),+ "key_method": enumStringProperty("Keyboard delivery: auto (default) or sky_key. sky_key is the macOS SkyLight background-window path and is not supported on Linux.", keyMethodValues),}, []string{"app", "text"}),},}diff --git a/apps/OpenComputerUseLinux/main_test.go b/apps/OpenComputerUseLinux/main_test.goindex 73ef1db..4039d71 100644--- a/apps/OpenComputerUseLinux/main_test.go+++ b/apps/OpenComputerUseLinux/main_test.go@@ -383,3 +383,54 @@ func shortTempDir(t *testing.T) string {})return path}++func TestLinuxKeyMethodSchemaAndUnsupportedSkyKey(t *testing.T) {+ for _, name := range []string{"type_text", "press_key"} {+ tool := findToolDefinition(t, name)+ properties := tool.InputSchema["properties"].(map[string]any)+ method := properties["key_method"].(map[string]any)+ values := method["enum"].([]string)+ if strings.Join(values, ",") != "auto,sky_key" {+ t.Fatalf("%s key_method enum = %#v", name, values)+ }+ }++ for input, want := range map[string]string{"": "auto", " SKY_KEY ": "sky_key"} {+ got, err := parseKeyMethod(input)+ if err != nil || got != want {+ t.Fatalf("parseKeyMethod(%q) = %q, %v", input, got, err)+ }+ }+ if _, err := parseKeyMethod("global"); err == nil || !strings.Contains(err.Error(), "Expected one of: auto, sky_key") {+ t.Fatalf("parseKeyMethod(global) error = %v", err)+ }++ service := newService()+ result := service.typeText("Text Editor", "hello", "sky_key")+ if !result.IsError || result.Content[0].Text != "key_method 'sky_key' is not supported on Linux" {+ t.Fatalf("type_text sky_key result = %#v", result)+ }+ result = service.pressKey("Text Editor", "Return", "sky_key")+ if !result.IsError || result.Content[0].Text != "key_method 'sky_key' is not supported on Linux" {+ t.Fatalf("press_key sky_key result = %#v", result)+ }+}++func TestLinuxWindowPlacementSchemaAndUnsupportedAgentDisplay(t *testing.T) {+ tool := findToolDefinition(t, "get_app_state")+ properties := tool.InputSchema["properties"].(map[string]any)+ placement := properties["window_placement"].(map[string]any)+ if strings.Join(placement["enum"].([]string), ",") != "keep,agent_display,restore" {+ t.Fatalf("window_placement enum = %#v", placement["enum"])+ }+ if got, err := parseWindowPlacement(" AGENT_DISPLAY "); err != nil || got != "agent_display" {+ t.Fatalf("parseWindowPlacement = %q, %v", got, err)+ }+ if _, err := parseWindowPlacement("park"); err == nil {+ t.Fatal("parseWindowPlacement(park) should fail")+ }+ result := newService().callTool("get_app_state", map[string]any{"app": "Text Editor", "window_placement": "agent_display"})+ if !result.IsError || result.Content[0].Text != "window_placement 'agent_display' is not supported on Linux" {+ t.Fatalf("agent_display result = %#v", result)+ }+}diff --git a/apps/OpenComputerUseWindows/main.go b/apps/OpenComputerUseWindows/main.goindex fb51a73..aa5d236 100644--- a/apps/OpenComputerUseWindows/main.go+++ b/apps/OpenComputerUseWindows/main.go@@ -24,6 +24,10 @@ var version = "0.3.6"var clickMethodValues = []string{"auto", "accessibility", "app_post", "sky_click", "global"}+var keyMethodValues = []string{"auto", "sky_key"}++var windowPlacementValues = []string{"keep", "agent_display", "restore"}+const (minimumScreenshotDimension = 64minimumScreenshotArea = 20_000@@ -262,6 +266,13 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult {if err != nil {return textResult(err.Error(), true)}+ windowPlacement, err := parseWindowPlacement(optionalString(args, "window_placement"))+ if err != nil {+ return textResult(err.Error(), true)+ }+ if windowPlacement != "keep" {+ return textResult("window_placement '"+windowPlacement+"' is not supported on Windows", true)+ }return s.getAppState(requiredString(args, "app"), textLimit, maxTreeNodes, maxTreeDepth)case "click":clickMethod, err := parseClickMethod(optionalString(args, "click_method"))@@ -299,9 +310,17 @@ func (s *service) callTool(name string, args map[string]any) toolCallResult {requiredFloat(args, "to_y"),)case "type_text":- return s.typeText(requiredString(args, "app"), requiredString(args, "text"))+ keyMethod, err := parseKeyMethod(optionalString(args, "key_method"))+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.typeText(requiredString(args, "app"), requiredString(args, "text"), keyMethod)case "press_key":- return s.pressKey(requiredString(args, "app"), requiredString(args, "key"))+ keyMethod, err := parseKeyMethod(optionalString(args, "key_method"))+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.pressKey(requiredString(args, "app"), requiredString(args, "key"), keyMethod)case "set_value":return s.setValue(requiredString(args, "app"), requiredElementIndex(args), requiredString(args, "value"))default:@@ -463,26 +482,32 @@ func screenshotCoordinatesUnavailableResult(app string) toolCallResult {return textResult("No usable screenshot is available for "+app+". Restore a visible, non-minimized, on-screen window and run get_app_state again before using screenshot coordinates.", true)}-func (s *service) typeText(app, text string) toolCallResult {+func (s *service) typeText(app, text, keyMethod string) toolCallResult {if app == "" {return textResult("Missing required argument: app", true)}if text == "" {return textResult("Missing required argument: text", true)}+ if keyMethod == "sky_key" {+ return textResult("key_method 'sky_key' is not supported on Windows", true)+ }if s.currentSnapshot(app) == nil {return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)}return s.actionResult(app, psRequest{Tool: "type_text", App: app, Text: text})}-func (s *service) pressKey(app, key string) toolCallResult {+func (s *service) pressKey(app, key, keyMethod string) toolCallResult {if app == "" {return textResult("Missing required argument: app", true)}if key == "" {return textResult("Missing required argument: key", true)}+ if keyMethod == "sky_key" {+ return textResult("key_method 'sky_key' is not supported on Windows", true)+ }if s.currentSnapshot(app) == nil {return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)}@@ -767,6 +792,32 @@ func defaultString(value, fallback string) string {return value}+func parseWindowPlacement(value string) (string, error) {+ normalized := strings.ToLower(strings.TrimSpace(value))+ if normalized == "" {+ return "keep", nil+ }+ for _, candidate := range windowPlacementValues {+ if normalized == candidate {+ return normalized, nil+ }+ }+ return "", fmt.Errorf("Invalid window_placement %q. Expected one of: %s", value, strings.Join(windowPlacementValues, ", "))+}++func parseKeyMethod(value string) (string, error) {+ normalized := strings.ToLower(strings.TrimSpace(value))+ if normalized == "" {+ return "auto", nil+ }+ for _, candidate := range keyMethodValues {+ if normalized == candidate {+ return normalized, nil+ }+ }+ return "", fmt.Errorf("Invalid key_method %q. Expected one of: %s", value, strings.Join(keyMethodValues, ", "))+}+func parseClickMethod(value string) (string, error) {normalized := strings.ToLower(strings.TrimSpace(value))if normalized == "" {@@ -813,10 +864,11 @@ func toolDefinitions() []toolDefinition {Description: "Get the state of an already running app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`.",Annotations: readOnlyAnnotations(),InputSchema: objectSchema(map[string]any{- "app": stringProperty("App name or bundle identifier"),- "text_limit": textLimitProperty("Maximum text characters to return. Use \"max\" for full text. Defaults to 500."),- "max_tree_nodes": positiveIntegerProperty("Maximum accessibility tree nodes to render. Defaults to 1200."),- "max_tree_depth": positiveIntegerProperty("Maximum accessibility tree depth to render. Defaults to 64."),+ "app": stringProperty("App name or bundle identifier"),+ "text_limit": textLimitProperty("Maximum text characters to return. Use \"max\" for full text. Defaults to 500."),+ "max_tree_nodes": positiveIntegerProperty("Maximum accessibility tree nodes to render. Defaults to 1200."),+ "max_tree_depth": positiveIntegerProperty("Maximum accessibility tree depth to render. Defaults to 64."),+ "window_placement": enumStringProperty("keep (default) leaves the window where it is. agent_display and restore are macOS-only and not supported on Windows.", windowPlacementValues),}, []string{"app"}),},{@@ -840,8 +892,9 @@ func toolDefinitions() []toolDefinition {Description: "Press a key or key-combination on the keyboard, including modifier and navigation keys.\n - This supports xdotool's `key` syntax.\n - Examples: \"a\", \"Return\", \"Tab\", \"super+c\", \"Up\", \"KP_0\" (for the numpad 0). This tool is part of plugin `Computer Use`.",Annotations: defaultAnnotations(),InputSchema: objectSchema(map[string]any{- "app": stringProperty("App name or bundle identifier"),- "key": stringProperty("Key or key-combination to press"),+ "app": stringProperty("App name or bundle identifier"),+ "key": stringProperty("Key or key-combination to press"),+ "key_method": enumStringProperty("Keyboard delivery: auto (default) or sky_key. sky_key is the macOS SkyLight background-window path and is not supported on Windows.", keyMethodValues),}, []string{"app", "key"}),},{@@ -870,8 +923,9 @@ func toolDefinitions() []toolDefinition {Description: "Type literal text using keyboard input. This tool is part of plugin `Computer Use`.",Annotations: defaultAnnotations(),InputSchema: objectSchema(map[string]any{- "app": stringProperty("App name or bundle identifier"),- "text": stringProperty("Literal text to type"),+ "app": stringProperty("App name or bundle identifier"),+ "text": stringProperty("Literal text to type"),+ "key_method": enumStringProperty("Keyboard delivery: auto (default) or sky_key. sky_key is the macOS SkyLight background-window path and is not supported on Windows.", keyMethodValues),}, []string{"app", "text"}),},}diff --git a/apps/OpenComputerUseWindows/main_test.go b/apps/OpenComputerUseWindows/main_test.goindex 8bc1700..8600c61 100644--- a/apps/OpenComputerUseWindows/main_test.go+++ b/apps/OpenComputerUseWindows/main_test.go@@ -404,3 +404,54 @@ func findToolDefinition(t *testing.T, name string) toolDefinition {t.Fatalf("missing tool definition %q", name)return toolDefinition{}}++func TestWindowsKeyMethodSchemaAndUnsupportedSkyKey(t *testing.T) {+ for _, name := range []string{"type_text", "press_key"} {+ tool := findToolDefinition(t, name)+ properties := tool.InputSchema["properties"].(map[string]any)+ method := properties["key_method"].(map[string]any)+ values := method["enum"].([]string)+ if strings.Join(values, ",") != "auto,sky_key" {+ t.Fatalf("%s key_method enum = %#v", name, values)+ }+ }++ for input, want := range map[string]string{"": "auto", " SKY_KEY ": "sky_key"} {+ got, err := parseKeyMethod(input)+ if err != nil || got != want {+ t.Fatalf("parseKeyMethod(%q) = %q, %v", input, got, err)+ }+ }+ if _, err := parseKeyMethod("global"); err == nil || !strings.Contains(err.Error(), "Expected one of: auto, sky_key") {+ t.Fatalf("parseKeyMethod(global) error = %v", err)+ }++ service := newService()+ result := service.typeText("Notepad", "hello", "sky_key")+ if !result.IsError || result.Content[0].Text != "key_method 'sky_key' is not supported on Windows" {+ t.Fatalf("type_text sky_key result = %#v", result)+ }+ result = service.pressKey("Notepad", "Return", "sky_key")+ if !result.IsError || result.Content[0].Text != "key_method 'sky_key' is not supported on Windows" {+ t.Fatalf("press_key sky_key result = %#v", result)+ }+}++func TestWindowsWindowPlacementSchemaAndUnsupportedAgentDisplay(t *testing.T) {+ tool := findToolDefinition(t, "get_app_state")+ properties := tool.InputSchema["properties"].(map[string]any)+ placement := properties["window_placement"].(map[string]any)+ if strings.Join(placement["enum"].([]string), ",") != "keep,agent_display,restore" {+ t.Fatalf("window_placement enum = %#v", placement["enum"])+ }+ if got, err := parseWindowPlacement(" AGENT_DISPLAY "); err != nil || got != "agent_display" {+ t.Fatalf("parseWindowPlacement = %q, %v", got, err)+ }+ if _, err := parseWindowPlacement("park"); err == nil {+ t.Fatal("parseWindowPlacement(park) should fail")+ }+ result := newService().callTool("get_app_state", map[string]any{"app": "Notepad", "window_placement": "agent_display"})+ if !result.IsError || result.Content[0].Text != "window_placement 'agent_display' is not supported on Windows" {+ t.Fatalf("agent_display result = %#v", result)+ }+}diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 2ed833b..c3f6541 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -57,7 +57,7 @@- `ping`- `tools/list`- `tools/call`-- `notifications/turn-ended` 是开源版显式的 turn boundary hook;收到后会清理当前进程里的 visual cursor overlay。CLI `open-computer-use turn-ended [payload]` 也会通过 macOS distributed notification 通知正在运行的 AppKit MCP 进程执行同一类清理,用于接 Codex legacy notify 的 after-agent payload。+- `notifications/turn-ended` 是开源版显式的 turn boundary hook;收到后会清理当前进程里的 visual cursor overlay、occlusion keep-alive 和 agent display。CLI `open-computer-use turn-ended [payload]` 也会通过 macOS distributed notification 通知正在运行的 AppKit MCP / app-agent 进程执行同一类清理,用于接 Codex legacy notify 的 after-agent payload。stdio server 退出和 app-agent client connection 关闭也会执行该 session cleanup。### 3. Tool Service 层@@ -70,7 +70,7 @@- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑且不包含带 URL 的 `AXLink` 后代,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`;包含带 URL 的 `AXLink` 后代时保留通用 wrapper 和链接子节点,避免导航链接被摘要吞掉。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。-- JS REPL 的 API 和当前官方 code-first 形态对齐到异步 app binding,而不是把离散 tools 简单包成同步函数。实现不复制或运行 proprietary `@oai/*` package;协议和行为记录在 `docs/references/js-repl.md`。+- JS REPL 的 API 和当前官方 code-first 形态对齐到异步 app binding,而不是把离散 tools 简单包成同步函数。snapshot 的 `windowPlacement` 以及 keyboard action 的 `keyMethod` 会转换成 native tool 的 snake_case 参数。实现不复制或运行 proprietary `@oai/*` package;协议和行为记录在 `docs/references/js-repl.md`。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。@@ -87,6 +87,10 @@- `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post`、`sky_click` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;其余 mouse 路径可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。- element-targeted `click` 的 `auto` 左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后进入现有 non-AX 分支:未开启全局指针环境变量时使用 `postToPid` 定向鼠标事件,开启时直接使用全局 HID 事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action。- 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=sky_click` 只在 macOS 接受左键单击/双击,它验证当前 snapshot 的 `CGWindowID` 仍为同一 PID 所有且 on-screen,仅通过 `SLPSPostEventRecordTo` 让目标应用短暂进入 synthetic-active 状态,绝不向真实前台应用发送 defocus record,再按 Chromium-compatible recipe 发送 target move、`(-1,-1)` primer 和真实 down/up。每个鼠标事件同时经动态解析的 `SLEventPostToPid` 与公开 `postToPid` 投递,并带 window-local location、PID/window 和 click-group 字段;renderer settle 后只撤销目标的 synthetic-active 状态。`sky_click` 的 action-result snapshot 刷新也禁止 activate / `AXRaise` 恢复,目标状态瞬时不可读时直接失败。它不改变 `auto`,任一步失败也不 fallback。显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。+ - `get_app_state` 解析目标窗口时先看 on-screen 窗口,再退到所有窗口,所以位于其他 Space 的窗口不会触发 activate / raise 恢复;拍到未被遮挡的窗口后会关闭它的 WindowServer occlusion 通知(`WindowOcclusionKeepAlive`),之后被遮挡或切换 Space 时 Chromium / Electron / WebKit 仍保留页面与 `AXWebArea`。keep-alive 保存 SPI 返回的原始开关状态,并在 turn-ended、MCP/app-agent connection 关闭、server 正常关闭或进程退出时恢复;恢复失败的记录保留到后续重试。因为 snapshot 会维护该状态,macOS `get_app_state` 不声明 `readOnlyHint`。Chromium 系 app 的 web AX tree 是懒加载的,窗口可见时最多重走 3 秒,被遮挡时在 tree 末尾追加说明。`sky_click` 不再要求窗口 on-screen。详见 `docs/references/macos-window-visibility-and-spaces.md`。+ - 窗口截图主路径是 WindowServer 的 `SLSHWCaptureWindowList`(任何 Space、被遮挡均可,15–45 ms),ScreenCaptureKit 为 fallback;AX root window 通过 `_AXUIElementGetWindow` 直接绑定 `CGWindowID`。输入 recipe 的固定间隔都是环境变量可调的校准旋钮,默认值来自基准测试(见 `docs/references/macos-window-visibility-and-spaces.md`)。+ - `get_app_state.window_placement` 是开源版的可选扩展:`keep`(默认)、`agent_display`、`restore`。`agent_display` 通过 `AgentDisplay` + `packages/OpenComputerUseVirtualDisplayShim` 创建 agent 专用 virtual display,用 AX 位置把目标窗口的 frame 移过去(不激活、不抬升),让已被遮挡或在其他 Space 的窗口对其 app 真正可见;`restore` 按 app PID 恢复当前 runtime 停放的全部窗口,session cleanup 或进程退出也会恢复全部窗口并销毁显示器。恢复失败时保留原位置记录和 virtual display,避免把未恢复窗口误记为成功并允许后续重试;已关闭的窗口则直接清除记录。shim 捕获私有 KVC/selector 的 Objective-C exception,并把不兼容系统收敛为创建失败。Windows / Linux 对非 `keep` 值返回 unsupported。+ - `type_text.key_method` / `press_key.key_method` 是开源版的可选扩展,支持 `auto`(默认)和 macOS-only 的 `sky_key`。`sky_key` 复用 `sky_click` 的 target-only synthetic-active record,再投递 yabai 的 make-key-window record 让目标 app 在进程内把当前 snapshot 窗口设为 key window,之后走现有 `CGEvent.postToPid` 键盘路径;带 `cmd` 的单字符组合键改为在目标 AX 菜单栏找到对应菜单项并 `AXPress`。它不要求窗口 on-screen(可用于其他 Space),但对隐藏 app fail closed;action-result snapshot 与 `sky_click` 一样使用 read-only recovery policy。- 对 renderer 合成的 summary `text` 行,`click` 不再默认点击父容器中心;这类元素使用左侧安全锚点。对普通 row/container/text 的后代候选,也会过滤右侧紧凑 hover action(例如 Lark / Electron 会话列表里的“完成”勾),避免把主行点击误操作成 side action;Electron/Lark 这类 app 的 WebArea 合成文本会优先寻找紧邻的行级 `AXPress` 祖先并静默执行,避免用物理鼠标点行。浏览器 WebArea(例如 Chrome/GitHub)不走这条 Electron-scoped 行级祖先点击优化,仍保留通用 link/container 点击路径。命中点如果只反查到覆盖整页的 Electron/WebArea 级 AX 元素,不再继续扫描整个大容器的子孙候选,避免一次行点击被远处的可点击元素截走。- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`type_text` 会把文本按 Unicode extended grapheme cluster 聚合成小批量 `keyboardSetUnicodeString` 事件,避免中文标点、emoji / 代理对和组合字符被逐个 UTF-16 code unit 拆开后在 Electron 富文本输入框里乱序或变形。如果当前 focused element 的 `AXValue` 可设置,`type_text` 会优先按可编辑内容追加并写回 `AXValue`,这覆盖 Feishu / Electron 富文本输入框不可靠接收后台键盘事件的场景,并会过滤已知占位提示,避免把 placeholder 拼进草稿;如果当前 focused element 不是可编辑文本目标,`type_text` 会报错要求先 click 文本输入区或使用 `set_value`,不再把无效果的后台键盘投递当成成功;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias@@ -119,7 +123,7 @@- Windows runtime 默认只连接已经运行的 app,不会在 `get_app_state` 找不到进程时自动 `Start-Process`,也不会默认允许 `SetFocus` secondary action;这两条前台抢占路径分别需要 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_APP_LAUNCH=1` 和 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOCUS_ACTIONS=1` 显式打开。`type_text` 默认优先对可写文本控件的 child HWND 发送 `EM_SETSEL` / `EM_REPLACESEL`,不再默认走可能触发前台激活的 UIA `ValuePattern.SetValue` fallback;需要旧行为时必须设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK=1`。UIA pattern 和 Win32 message fallback 本身仍是 best-effort:很多控件可以在后台响应,但 Windows 没有一套对所有 GUI toolkit 都等价于 macOS AX 的后台键鼠输入模型。- Windows screenshot 会先检查主窗口可见、未最小化、未被 DWM cloaked、与 virtual screen 相交,并要求窗口宽高至少 64 像素且面积至少 20,000 像素;`list_apps` 也只列出通过这组可捕获性检查的窗口。PowerShell bridge 通过后,Go 层还会解码 PNG header 并用同一尺寸门槛复核。任一检查失败时 snapshot 仍可返回 UI Automation tree,但省略 MCP `image` block;coordinate click / drag 会明确失败,element-targeted action 仍可尝试。实现不会为退化窗口放大、补边或生成占位截图。- 这 9 个 tool 的协议面与 macOS 主线保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime id / automation metadata,coordinate action 使用 screenshot/window-relative 坐标。-- Windows `click_method=accessibility` 映射到 UI Automation pattern,`app_post` 映射到 HWND `PostMessage`;macOS-only 的 `sky_click` 和没有实现的 `global` 都会在 snapshot lookup 前明确返回 unsupported。`auto` 仍保持 UIA 优先、window message fallback 的现有行为。+- Windows `click_method=accessibility` 映射到 UI Automation pattern,`app_post` 映射到 HWND `PostMessage`;macOS-only 的 `sky_click` 和没有实现的 `global` 都会在 snapshot lookup 前明确返回 unsupported。`type_text` / `press_key` 的 `key_method=sky_key` 同样在 snapshot lookup 前返回 unsupported。`auto` 仍保持 UIA 优先、window message fallback 的现有行为。- Windows UI Automation 需要运行在已登录用户的桌面 session 里。通过 SSH 作为脱离桌面的后台进程运行时,PowerShell 可以启动并返回 JSON,但系统可能不给它暴露顶层窗口;这种情况下 `list_apps` 会是空,`get_app_state` 可能返回 `appNotFound(...)`。- 当前 Windows 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/onboarding、没有 code signing,也没有独立的 Windows smoke fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-windows-computer-use-runtime.md`。@@ -132,7 +136,7 @@- Linux runtime 需要运行在已登录桌面用户 session 里。缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,Go runtime 会在启动 Python AT-SPI bridge 前尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session bus、display / Wayland 值;纯 SSH tty 如果找不到已登录桌面 session,可以启动二进制,但不能直接 inspect 或操作 GUI session。- `get_app_state` 的 accessibility tree 在 GTK/GNOME app 上可能很深,Linux bridge 使用与 macOS / Windows 一致的 1200 节点、64 层默认 tree budget,并支持显式提高 `max_tree_nodes` / `max_tree_depth`。截图通过 GDK root window best-effort capture;GNOME Wayland 可能返回黑图,bridge 会检测全黑采样并省略 image block。- 这 9 个 tool 的协议面与 macOS / Windows 保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime path metadata,coordinate action 使用 screenshot/window-relative 坐标。-- Linux `click_method=accessibility` 映射到 AT-SPI action,`global` 映射到 AT-SPI mouse synthesis 并要求全局指针环境变量;AT-SPI 没有等价的进程定向 mouse dispatch,因此 `app_post` 和 macOS-only 的 `sky_click` 会在 snapshot lookup 前明确返回 unsupported。`auto` 仍保持 AT-SPI action 优先、mouse synthesis fallback 的现有行为。+- Linux `click_method=accessibility` 映射到 AT-SPI action,`global` 映射到 AT-SPI mouse synthesis 并要求全局指针环境变量;AT-SPI 没有等价的进程定向 mouse dispatch,因此 `app_post` 和 macOS-only 的 `sky_click` 会在 snapshot lookup 前明确返回 unsupported,`key_method=sky_key` 也一样。`auto` 仍保持 AT-SPI action 优先、mouse synthesis fallback 的现有行为。- 当前 Linux 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/desktop entry,也没有独立 Linux fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-linux-computer-use-runtime.md`。## 关键边界diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.mdindex 5cd8610..b0c3c6c 100644--- a/docs/RELIABILITY.md+++ b/docs/RELIABILITY.md@@ -7,6 +7,16 @@- JS REPL 与 CLI:`node --test scripts/node-repl/*.test.mjs`- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`- macOS SkyLight 实机回归:`OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`+- macOS SkyLight 后台键盘实机回归:`OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST=1 swift test --filter SkyKeyboardLiveTests`+- macOS 被遮挡窗口 keep-alive 实机回归:`OPEN_COMPUTER_USE_RUN_OCCLUSION_LIVE_TEST=1 swift test --filter OcclusionKeepAliveLiveTests`+- macOS 多 app 扫描(对运行中的每个 GUI app 做 read-only snapshot,并在空白文本框里输入再删除一个标记;会向真实 app 输入,只在无人使用时跑):`OPEN_COMPUTER_USE_RUN_APP_MATRIX=1 swift test --filter AppMatrixLiveTests`;加 `OPEN_COMPUTER_USE_APP_MATRIX_PARK=1` 会把每个非全屏窗口停靠到 agent display 上做同样的事再移回原位。+- macOS 后台输入基准(25 轮 `sky_click` + `sky_key`,输出成功数与 p50 / p95 / max):`OPEN_COMPUTER_USE_RUN_BACKGROUND_BENCH=1 OPEN_COMPUTER_USE_BENCH_CYCLES=25 swift test --filter BackgroundInputBenchmarkLiveTests`+- 基准结果与复现命令汇总在 `docs/references/background-input-benchmarks.md`。+- 输入默认值以跨版本可靠性优先:普通 `auto` 的 type chunk / press-key settle 保持 20 / 100 ms,`sky_key` 的 key-window / release settle 为 10 / 10 ms;六个 timing knob 均可通过对应 `OPEN_COMPUTER_USE_*` 环境变量显式覆盖。+- 逐阶段耗时:任何命令前加 `OPEN_COMPUTER_USE_DEBUG_TIMING=1`,stderr 输出 `[open-computer-use] timing <阶段> <ms>ms`(`sky_key.*`、`sky_click.total`、`snapshot.window_capture`、`snapshot.tree_walk`、`agent_display.*`)。+- 跑实机回归时不要在这台 Mac 上打字或点鼠标:测试会短暂把目标窗口放到前台,用户的按键会落进去,前台 fixture 也会因用户操作失去 active。+- macOS agent display 实机回归(会短暂创建并销毁一块 virtual display):`OPEN_COMPUTER_USE_RUN_AGENT_DISPLAY_LIVE_TEST=1 swift test --filter AgentDisplayLiveTests`+- macOS 其他 Desktop 实机回归(需要第二个 Desktop,测试会短暂切换 Space 启动目标):`OPEN_COMPUTER_USE_RUN_CROSS_SPACE_LIVE_TEST=1 swift test --filter CrossSpaceLiveTests`- Linux runtime:`(cd apps/OpenComputerUseLinux && go test ./...)`、`./scripts/build-open-computer-use-linux.sh --arch arm64`- Windows runtime:`(cd apps/OpenComputerUseWindows && go test ./...)`、`./scripts/build-open-computer-use-windows.sh --arch amd64`;截图相关变更还要在 Windows 交互桌面验证正常窗口有 image、1x1/minimized/cloaked 窗口无 image。- 本地诊断:@@ -19,7 +29,8 @@- Codex plugin 的 code-first surface 需要可执行的 `node`。REPL adapter 会为 native MCP 保留独立 child lifecycle,JavaScript kernel 则放在 Worker 中;脚本超时会丢弃该 Worker 的全部 bindings,并由下一次调用使用新的 kernel。- npm CLI 本身也由 Node 启动。`ocu js` 只在一次执行期间保留 Worker/native MCP child;`ocu repl` 在当前 terminal session 内保留它们;`ocu mcp` 则跟随 stdio connection。macOS 的隐藏 app agent 是独立的权限身份,可能在这些前台命令退出后继续驻留。- `ocu --help` 始终展示 `js` / `repl`;`ocu capabilities --json` 可在不启动 native MCP 的情况下检查 Node、adapter、kernel 和 native artifact。当前 npm shebang 仍要求 shell 能从 PATH 启动 Node,所以“完全无 Node”必须在更外层用 native bootstrap 解决。-- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。受控实机回归除 DOM、前台 PID、鼠标和 z-order 外,还必须验证前台 AppKit active、key window、first responder 以及 resign/key-loss 计数。+- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo` 和 `GetProcessForPID`。截图主路径依赖可选的 `SLSHWCaptureWindowList`,窗口绑定依赖可选的 `_AXUIElementGetWindow`,缺失时分别退回 ScreenCaptureKit 与标题启发式。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。受控实机回归除 DOM、前台 PID、鼠标和 z-order 外,还必须验证前台 AppKit active、key window、first responder 以及 resign/key-loss 计数。`key_method=sky_key` 复用同一组符号,实机回归还要确认被遮挡 Chrome 的输入框收到文字、`cmd+a` 菜单快捷键生效,以及投递结束后 Chrome 页面重新 blur。+- agent display 依赖私有 `CGVirtualDisplay` 类、KVC key 与 selector。shim 会捕获 Objective-C exception 并把创建失败收敛成 capability unavailable;不允许私有 API 漂移直接终止 host 进程。显式 `restore` 按 app PID 恢复该 runtime 停放的全部窗口;turn-ended 和 MCP/REPL connection 关闭也会释放 occlusion keep-alive、恢复窗口并销毁空闲 display。- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异;Electron/WebView app 的 AX tree 通常很深,当前会压缩空 wrapper 并放宽遍历深度,以优先保留可操作文本、按钮和输入框。- Linux runtime 依赖已登录桌面用户 session;缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,会尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session env。纯 SSH tty 如果找不到桌面 session 仍不能直接访问 AT-SPI GUI tree。@@ -30,11 +41,13 @@1. 先跑 `open-computer-use doctor`,确认权限状态;如果缺权限,命令会通过 `.app` app agent 拉起权限 onboarding 窗口,已全部授权则只打印状态并退出。2. 用 `open-computer-use list-apps` 确认目标 app 是否被发现。3. 用 `open-computer-use snapshot <app>` 看是 transport 问题还是 snapshot / action 问题。-4. 如果只有 `sky_click` 失败,先重新执行 `get_app_state`,确认窗口仍为 on-screen、未隐藏/最小化且没有切换 Space;错误里出现 `missing SkyLight symbols` 时不要改用隐式 fallback,应按当前 macOS 版本重新验证私有 SPI。被遮挡的 Chromium 页面仍无效果时,再用受控页面区分 renderer 策略变化与坐标/window-local 映射问题。-5. 如果只想验证仓库基线,直接跑 fixture + smoke,不要先在复杂第三方 app 上排查。-6. 排查 Linux runtime 时,先确认目标命令是否由桌面用户运行,再用 `open-computer-use call list_apps` 和 `open-computer-use snapshot <app>` 区分 session/env 问题与 AT-SPI tree/action 问题。如果是 Codex MCP,重新执行 `open-computer-use install-codex-mcp` 后重启 Codex,确认配置仍是 `open-computer-use mcp`。-7. Windows snapshot 返回 accessibility tree 但没有 image 时,先确认窗口可见、未最小化、未 cloaked、位于 virtual screen 内且不是退化小窗口;无 image 时不要调用 coordinate click / drag,优先使用 `element_index`。-8. 排查 `ocu js` / `ocu repl` 时先执行 `ocu capabilities --json`。缺少 adapter、kernel 或 native artifact 时重新安装 npm 包;若 shell 报 `node: command not found`,说明 npm launcher 本身尚未启动,需要先安装 Node 或等待未来 native bootstrap 方案。+4. 如果 `sky_key` 输入没有落到目标:先确认目标不是隐藏 app、窗口仍属于同一 PID;Chromium 目标可以在页面里观察 `document.hasFocus()`,投递期间应为 true。跑实机回归时不要同时在其他窗口打字,否则前台 fixture 会因为用户自己的操作失去 active。+5. 如果被遮挡或其他 Space 的 Chromium / Electron 窗口 tree 里没有网页内容,看 tree 末尾是否有 “window is covered” 说明:说明 agent 第一次看到它时它已经被遮挡。让窗口露出一次再 `get_app_state`,之后再遮挡也会保留内容;或者在同一 session 里显式调用 `get_app_state` 并传 `window_placement=agent_display` 把窗口停靠到 agent 显示器,用完 `restore`。若 session 已结束,runtime 会自动恢复,不需要另开连接补 `restore`。+6. 如果只有 `sky_click` 失败,先重新执行 `get_app_state`,确认窗口仍属于同一进程且 app 未被隐藏;错误里出现 `missing SkyLight symbols` 时不要改用隐式 fallback,应按当前 macOS 版本重新验证私有 SPI。被遮挡的 Chromium 页面仍无效果时,再用受控页面区分 renderer 策略变化与坐标/window-local 映射问题。+7. 如果只想验证仓库基线,直接跑 fixture + smoke,不要先在复杂第三方 app 上排查。+8. 排查 Linux runtime 时,先确认目标命令是否由桌面用户运行,再用 `open-computer-use call list_apps` 和 `open-computer-use snapshot <app>` 区分 session/env 问题与 AT-SPI tree/action 问题。如果是 Codex MCP,重新执行 `open-computer-use install-codex-mcp` 后重启 Codex,确认配置仍是 `open-computer-use mcp`。+9. Windows snapshot 返回 accessibility tree 但没有 image 时,先确认窗口可见、未最小化、未 cloaked、位于 virtual screen 内且不是退化小窗口;无 image 时不要调用 coordinate click / drag,优先使用 `element_index`。+10. 排查 `ocu js` / `ocu repl` 时先执行 `ocu capabilities --json`。缺少 adapter、kernel 或 native artifact 时重新安装 npm 包;若 shell 报 `node: command not found`,说明 npm launcher 本身尚未启动,需要先安装 Node 或等待未来 native bootstrap 方案。## 后续补强方向diff --git a/docs/SECURITY.md b/docs/SECURITY.mdindex b98c84b..c485dd6 100644--- a/docs/SECURITY.md+++ b/docs/SECURITY.md@@ -33,7 +33,10 @@- `click_method=global` 是显式的系统级指针路径,可能移动真实鼠标、改变前台焦点或命中坐标处的其他窗口。调用参数本身不视为足够授权;macOS 和支持该模式的 Linux runtime 还要求进程环境中设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`。未设置时必须在任何可见 cursor 移动或真实输入事件之前拒绝请求。- `click_method=app_post`、`sky_click` 与 `accessibility` 不允许静默切换到 `global`。这保证调用方选择的非侵入边界在失败时仍然成立。- `click_method=sky_click` 是显式 macOS 私有 SPI 能力,不进入 `auto`。它不移动系统指针、不改变 WindowServer frontmost app,也不 raise 或切换目标窗口;内部只让目标应用短暂进入 synthetic-active 状态,绝不向真实前台应用发送 defocus record,renderer settle 后也只撤销目标的合成状态。点击后的 action-result snapshot 禁止 activate / `AXRaise` 恢复。它仍会向指定 PID/window 注入真实输入语义,因此只允许使用当前 snapshot 的 on-screen、同 PID 窗口,并在窗口身份不匹配、target-focus record 失败或私有符号缺失时 fail closed。第一版仅支持同一 Space 内的左键单击/双击。-- SkyLight ABI、raw event field 和 Chromium 接收行为都不受 Apple 公共兼容性承诺保护。系统升级后的失败不得触发静默 global fallback;应先重新验证符号和受控目标,再决定是否更新实现。+- `key_method=sky_key` 与 `sky_click` 共用同一套边界:显式 macOS 私有 SPI 能力,不进入 `auto`,失败不 fallback;只让目标 app 短暂进入 synthetic-active 并把目标窗口设为其进程内的 key window,绝不向真实前台应用发送任何 record,不移动指针,不 raise,不切 Space。它会向目标 PID 注入真实键盘语义,并可能通过 AX 按下目标菜单项(例如 `cmd+q` 就会退出目标 app),所以只允许当前 snapshot 的同 PID 窗口,隐藏 app 与私有符号缺失时 fail closed。+- `get_app_state` 的 occlusion keep-alive 会关闭目标窗口的 WindowServer occlusion 通知,让 app 在被遮挡时继续渲染和暴露内容。因此 macOS 工具定义不把它声明为 read-only,即使默认 `keep` 不移动窗口。它不改变窗口可见性、层级、Space 或焦点,只作用于当前 snapshot 的窗口;运行时保存原始通知状态,并在 turn-ended、MCP/REPL connection 关闭、server 正常关闭或进程退出时恢复,失败项保留以便重试。副作用是 session 内被遮挡的窗口会继续消耗渲染资源;无法捕获的强制终止仍不具备进程内清理机会。+- `window_placement=agent_display` 是唯一会改变用户可见状态的显式模式:它创建一个用户看不到的 virtual display(显示器排列会多出一块,鼠标可能滑入),并把目标窗口移到那里,窗口在停靠期间不在用户桌面上。它不激活、不抬升、不切换 Space、不移动指针;显式 `restore` 会恢复该 app 在当前 runtime 中停放的全部窗口,turn-ended、MCP/REPL connection 关闭、server 正常关闭或进程退出也会恢复全部窗口并销毁显示器。恢复无法确认时保留记录和显示器以便重试,不会把失败误报成成功;无法捕获的强制终止仍不具备进程内清理机会。默认 `keep` 不做窗口移动,Windows / Linux 拒绝非 `keep` 值。+- SkyLight ABI、raw event field、`CGVirtualDisplay` KVC/selector surface 和 Chromium 接收行为都不受 Apple 公共兼容性承诺保护。系统升级后的私有 virtual-display 调用异常会被 shim 捕获并按 capability unavailable 失败;其他私有能力失败也不得触发静默 global fallback。应先重新验证符号和受控目标,再决定是否更新实现。- 下一阶段应优先补:- session 级审批- 更清楚的敏感 app / 系统设置防护策略diff --git a/docs/exec-plans/completed/20260908-add-sky-key-background-keyboard.md b/docs/exec-plans/completed/20260908-add-sky-key-background-keyboard.mdnew file mode 100644index 0000000..335957a--- /dev/null+++ b/docs/exec-plans/completed/20260908-add-sky-key-background-keyboard.md@@ -0,0 +1,58 @@+# Add macOS SkyLight background keyboard (`sky_key`)++## 目标++为 `type_text` / `press_key` 增加显式 `key_method=sky_key`,让 macOS 能向被遮挡或位于其他 Space 的目标窗口(重点是 Chromium / Electron)输入文字与快捷键,同时保持真实前台应用的 active / key / first responder、指针位置、窗口层级和当前 Space 不变。++## 范围++- 包含:+ - 实机验证 mac-cua / Cua Driver 的认证键盘路径是否可用,并找出 Chromium 真正的门槛。+ - macOS `sky_key` 路由:synthetic-active record + yabai make-key-window record + 现有 `postToPid` 键盘路径 + AX 菜单快捷键。+ - Windows / Linux 公共枚举同步与 unsupported 结果。+ - Swift / Go 测试、skill usage、架构、安全、可靠性、参考资料与 history。+- 不包含:+ - 修改 `auto` 或让 `sky_click` / 快照 / 截图支持其他 Space(后续计划)。+ - `SLPSSetFrontProcessWithOptions` foreground assist、自动 unhide 隐藏 app。+ - 发布、打 tag、提交或推送远端。++## 背景++- 相关文档:`docs/references/macos-skylight-background-click.md`、`docs/references/macos-skylight-background-keyboard.md`。+- 相关代码路径:`SkyLightSPI`、`SkyKeyboardSimulation`、`ComputerUseService.typeText/pressKey`、三平台 `key_method` parser / schema。+- 已知约束:显式方法不能静默 fallback;私有 SPI 必须运行时探测;绝不向真实前台应用发送 record。++## 风险++- 风险:make-key-window record 是未公开 ABI,macOS 更新可能失效。+- 缓解方式:集中在 `SkyLightSPI.swift`,缺符号 fail closed,实机回归纳入 `docs/RELIABILITY.md`。+- 风险:AX 菜单项按下等价于用户按快捷键,`cmd+q` 之类会真的执行。+- 缓解方式:语义与用户按键一致,在 `docs/SECURITY.md` 明确说明;不做额外过滤以免与 `auto` 语义分叉。+- 风险:Chrome 变 key 的等待时间因机器不同而变化。+- 缓解方式:固定 300ms(实测 ~215ms),后续可改为 AX focused-window 轮询。++## 里程碑++1. 实机验证并找到可行序列。+2. 实现 macOS 路由、跨平台协议与测试。+3. 文档、history、归档计划。++## 验证方式++- 命令:`swift build`、`swift test`、`(cd apps/OpenComputerUseWindows && go test ./...)`、`(cd apps/OpenComputerUseLinux && go test ./...)`、`./scripts/run-tool-smoke-tests.sh`、`make check-docs`。+- 命令:`OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST=1 swift test --filter SkyKeyboardLiveTests`。+- 实机检查:被遮挡 Chrome 收到文字与 `cmd+a`;前台 fixture active / key / first responder 与 resign 计数不变;目标在全屏 Space 时 active Space 不切换。++## 进度记录++- [x] 验证 mac-cua 认证路径无效,定位 Chromium `isKeyWindow` 门槛。+- [x] 验证 yabai make-key-window record + synthetic-active record 可让被遮挡 Chrome 接受输入。+- [x] 验证菜单快捷键经 AX 菜单项生效,验证跨 Space 输入。+- [x] 实现 `sky_key`、三平台协议、测试与文档。++## 决策记录++- 2026-09-08:不引入 `SLSEventAuthenticationMessage` 认证信封,实测它既非必要也不充分。+- 2026-09-08:`cmd` 单字符组合键走目标 AX 菜单项;导航类组合键(`cmd+shift+left` 等)保持键盘事件。+- 2026-09-08:`sky_key` 允许 off-screen 窗口(其他 Space),但隐藏 app fail closed;不代替用户 unhide。+- 2026-09-08:否定 `SLPSSetFrontProcessWithOptions` 与 `CGSSetConnectionProperty SetFrontmost`,前者让用户应用 resign active,后者在 macOS 27 无效。diff --git a/docs/exec-plans/completed/20260908-cross-space-background-state.md b/docs/exec-plans/completed/20260908-cross-space-background-state.mdnew file mode 100644index 0000000..515eb9e--- /dev/null+++ b/docs/exec-plans/completed/20260908-cross-space-background-state.md@@ -0,0 +1,60 @@+# Cross-Space background state: read, click and type anywhere without taking focus++## 目标++让 `get_app_state`、`click`(`sky_click`)、`type_text` / `press_key`(`sky_key`)对被遮挡的窗口和位于其他 Space 的窗口都可靠工作,全程不激活、不抬升、不切换 Space、不移动指针,并且用 macOS / SkyLight 原语实现,而不是按 app 打补丁。++## 范围++- 包含:+ - 窗口解析不再要求 on-screen;SCK 按 window id 捕获 off-screen 窗口。+ - WindowServer occlusion keep-alive(已完成)与 Chromium 系懒加载 AX tree 的重走(已完成)。+ - `sky_click` / `sky_key` 接受 off-screen 窗口(已完成)。+ - 在真实的第二个桌面 Space 上做端到端回归(待机器上存在第二个 Desktop)。+ - 已经被遮挡 / 已在其他 Space 的 Chromium 窗口:agent 专用 virtual display(`CGVirtualDisplay`)做成显式 `window_placement`(已完成)。+- 不包含:+ - 任何会显示窗口给用户、切换用户 Space 或前台 app 的 fallback。+ - Windows / Linux runtime。++## 背景++- 相关文档:`docs/references/macos-window-visibility-and-spaces.md`、`docs/references/macos-skylight-background-keyboard.md`、`docs/references/macos-skylight-background-click.md`。+- 相关代码路径:`WindowOcclusionKeepAlive`、`SnapshotBuilder`、`WindowCapture`、`SkyClickDispatcher`、`SkyKeyboardDispatcher`。+- 已知约束:app 的“可见”只来自 WindowServer 通知;已进入 hidden 的 Chromium 页面无法在不显示窗口的情况下被外部改回 visible。++## 风险++- 风险:keep-alive 让被遮挡窗口持续渲染,消耗资源。+- 缓解方式:只作用于 agent 驱动过的窗口,进程退出时恢复;后续可在 session 结束时释放。+- 风险:virtual display 会改变显示器排列,鼠标可能滑入不可见显示器,窗口会离开用户当前位置。+- 缓解方式:先做 spike 评估,明确 opt-in,结束后把窗口移回原 Space 与位置。+- 风险:私有 SPI 签名猜错会写坏内存(已在 `SLSLockWindowVisibleRegion` 上发生过)。+- 缓解方式:新增符号前用 lldb 反汇编确认参数,记录在参考文档里。++## 里程碑++1. occlusion keep-alive、off-screen 解析与捕获、`sky_click` 放宽校验(完成)。+2. 第二个桌面 Space 上的端到端回归:AX tree、截图、`sky_click`、`sky_key`。+3. virtual display spike:创建 / 销毁、把窗口移入移出、验证 Chromium 可见性,评估用户可感知副作用。++## 验证方式++- 命令:`swift test`、`OPEN_COMPUTER_USE_RUN_OCCLUSION_LIVE_TEST=1 swift test --filter OcclusionKeepAliveLiveTests`。+- 手工检查:创建 Desktop 2,把 Chrome 窗口放过去,在 Desktop 1 执行 `get_app_state` / `click` / `type_text`,确认 active Space 与前台 app 不变。+- 观测检查:进程退出后目标窗口重新收到 occlusion 通知(遮挡后页面变 hidden)。++## 进度记录++- [x] 定位根因:occlusion 通知,而非 Space。+- [x] keep-alive、off-screen 捕获、懒加载重走、`sky_click` 放宽校验,单元与实机回归通过。+- [x] Desktop 2 端到端回归:`CrossSpaceLiveTests` 在真实第二个 Desktop 上通过(snapshot / 截图 / `sky_click` / `sky_key`,active Space 与前台不变)。+- [x] virtual display spike:已被遮挡的 Chrome 窗口放到 agent virtual display 后页面可见、snapshot / sky_click / sky_key 全部生效,用户 Space、前台与鼠标不变。+- [x] virtual display 产品化:`get_app_state.window_placement=agent_display / restore`,ObjC shim、`AgentDisplay`、三平台协议、`AgentDisplayLiveTests`。排列与鼠标滑入未做额外处理,记录在参考文档。++## 决策记录++- 2026-09-08:不用 bundle id 列表判断 web app;keep-alive 对所有被驱动窗口生效,懒加载重走按 bundle 内的 Chromium / Electron / CEF 框架或 `AXManualAccessibility` 判定。+- 2026-09-08:已被遮挡的窗口不做任何显示式 fallback,只在 tree 里说明;下一步评估 virtual display。+- 2026-09-08:第三方进程不能跨 Space 移动别的 app 的窗口(window-management bridge gate),运行时也不需要;测试改为在目标 Desktop 上启动目标。+- 2026-09-08:virtual display spike 证明 macOS 层面可行(细节见 `docs/references/macos-window-visibility-and-spaces.md`);产品化作为独立后续,不混进本轮 PR。+- 2026-09-08:agent display 做成显式 `window_placement`,默认 `keep` 不移动任何窗口;`restore` 与进程退出恢复原位。计划归档。diff --git a/docs/exec-plans/completed/20260924-harden-pr65-background-operation.md b/docs/exec-plans/completed/20260924-harden-pr65-background-operation.mdnew file mode 100644index 0000000..7ef88dc--- /dev/null+++ b/docs/exec-plans/completed/20260924-harden-pr65-background-operation.md@@ -0,0 +1,56 @@+# 收敛 PR #65 后台操作能力++## 目标++把 PR #65 的后台键盘、跨 Space snapshot 与 agent display 能力同步到最新 `main`,修复评审发现的生命周期、恢复目标、MCP annotation、默认 JS REPL 接入和私有 API 兼容性问题,并形成可验证、可继续评审的提交。++## 范围++- 包含:合并最新 `main`、后台状态清理、按 app 恢复 parked windows、JS REPL 参数透传、私有 virtual display shim fail-closed、测试与文档。+- 不包含:改变 `sky_click` / `sky_key` 的底层事件 recipe、运行会干扰当前桌面的 opt-in live tests、直接合并 PR。++## 背景++- 相关文档:`docs/RELIABILITY.md`、`docs/SECURITY.md`、`docs/references/background-input-benchmarks.md`。+- 相关代码路径:`packages/OpenComputerUseKit/`、`apps/OpenComputerUse/`、`scripts/node-repl/`。+- 已知约束:macOS app agent 会跨前台 CLI/MCP 连接驻留;默认 Codex plugin 只暴露 JS REPL;SkyLight 和 CGVirtualDisplay 均为私有 API。++## 风险++- 风险:连接结束后窗口仍停在虚拟显示器,或 occlusion notifications 未恢复。+- 缓解方式:在 app-agent connection 和 turn boundary 显式清理,并补生命周期测试。+- 风险:静态 `readOnlyHint` 与可变窗口状态不一致。+- 缓解方式:将 macOS `get_app_state` 标为非只读并同步测试。+- 风险:私有 Objective-C API 变化抛出异常导致进程退出。+- 缓解方式:在 shim 内捕获 Objective-C exception 并返回不可用结果。++## 里程碑++1. 合并最新 `main` 并解决文档与许可冲突。+2. 修复 lifecycle、restore、annotation 和 JS REPL 集成。+3. 补齐测试与文档,运行仓库验证后推送 PR head。++## 验证方式++- `swift test`+- `node --test scripts/node-repl/*.test.mjs`+- `(cd apps/OpenComputerUseWindows && go test ./...)`+- `(cd apps/OpenComputerUseLinux && go test ./...)`+- `./scripts/run-tool-smoke-tests.sh`+- `./scripts/check-docs.sh`++## 进度记录++- [x] 完成代码审查并确认阻塞项。+- [x] 合并最新 `main`,保留新增 derived implementation 对应的许可声明。+- [x] 完成实现与测试。+- [x] 完成验证、history 与待推送提交。++## 决策记录++- 2026-09-24:采用 merge commit 同步 `main`,避免改写贡献者已有提交历史。+- 2026-09-24:保留 `THIRD_PARTY_NOTICES.md`,因为 PR 明确包含 derived implementation,不能在冲突解决时静默丢弃许可文本。+- 2026-09-24:不运行 opt-in GUI live tests;它们会切换窗口/Space 或注入真实输入,需要单独安排无人使用桌面的验证窗口。+- 2026-09-24:一过性的 CLI connection 会在关闭时清理 background state;需要连续 park/action/restore 时使用同一个 JS、REPL、MCP 或 `call --calls` session。+- 2026-09-24:无侵入验证完成;Node 全套唯一失败是最新 `main` 已复现的 Worker late-error timing case,不归因于本次改动。+- 2026-09-24:实现与文档收敛为 maintainer commit `f97e890`;plan 随完成记录归档,推送后由 PR 继续承载 review 状态。diff --git a/docs/histories/2026-09/20260908-2230-add-sky-key-background-keyboard.md b/docs/histories/2026-09/20260908-2230-add-sky-key-background-keyboard.mdnew file mode 100644index 0000000..1fdb979--- /dev/null+++ b/docs/histories/2026-09/20260908-2230-add-sky-key-background-keyboard.md@@ -0,0 +1,45 @@+## [2026-09-08 22:30] | Task: 添加 macOS SkyLight 后台键盘输入++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Fable 5.1`+* **Runtime**: `Claude Code CLI / macOS 27.0 arm64`++### 📥 User Query+> 参考 mac-cua swift 分支里的 SkyLight keyboard 实现,判断能否给本仓库贡献;先实机验证是否真的可用,再按 `sky_click` 的约定实现。后续追加:不要把限制当作结论,要做到后台、跨 Space、不抢用户焦点的可靠键盘输入。++### 🛠 Changes Overview+**Scope:** macOS keyboard runtime、三平台工具协议、测试、skill 文档、参考资料与第三方归属。++**Key Actions:**+- **Verified first**: mac-cua 的 `deliverKeyboard`(`SLSEventAuthenticationMessage` 认证信封)在 macOS 27 实测无效且从未被调用;被遮挡 Chrome 收得到 `keydown` 但因 NSWindow 不是 key window 不插入文字。原生 AppKit 应用后台本来就接受 `postToPid` 键盘事件。+- **sky_key route**: 复用 `sky_click` 的 target-only synthetic-active record,再投递 yabai `window_manager_make_key_window` 的两条 record 让目标窗口在其进程内成为 key window,等待 300ms 后走现有 `CGEvent.postToPid` 键盘路径,最后撤销 synthetic 状态。+- **Menu chords**: 带 `cmd` 的单字符组合键在目标 AX 菜单栏按 `AXMenuItemCmdChar` / `AXMenuItemCmdModifiers` 匹配并 `AXPress`,解决后台 Chromium 不分发 NSMenu key equivalent 的问题。+- **Safety boundary**: 显式 opt-in、不进入 `auto`、不 fallback;窗口可 off-screen(其他 Space),隐藏 app 与私有符号缺失 fail closed;snapshot refresh 使用 read-only recovery policy。+- **Cross-platform contract**: `type_text` / `press_key` 新增 `key_method` 枚举,Windows / Linux 在 snapshot lookup 前返回 unsupported。+- **Verification**: 新增纯逻辑单元测试与默认跳过的隔离 Chrome 实机回归;参考文档记录被否定的方案(front-process 切换、`SetFrontmost` 连接属性、认证信封、window field 打标、AX 窗口焦点)。++### 🧠 Design Intent (Why)+*Chromium 页面焦点由 `OnWindowIsKeyChanged(isKeyWindow)` 决定,而不是由事件是否“认证”决定。yabai 的 make-key-window record 能让目标 app 在进程内把窗口设为 key window,既不改变 WindowServer frontmost、z-order 或 Space,也不需要 `SLPSSetFrontProcessWithOptions` 这类会让用户应用 resign active 的调用。菜单快捷键走 AX 菜单项则等价于用户按键触发的动作,且在后台可用。*++### ✅ Verification+- `swift build`、`swift test`:168 tests,2 个默认跳过的实机 live test,0 failures。+- `OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST=1 swift test --filter SkyKeyboardLiveTests`:被 fixture 完全遮挡的隔离 Chrome 收到 `hello`、`backspace`、`shift+a`、`cmd+a` + 替换文字;前台 fixture 保持 active / key / first responder,resign 计数 0,指针与 z-order 不变,投递结束后 Chrome 页面 blur。+- `OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests` 仍通过。+- 一次性实验:目标窗口进入全屏 Space、桌面 Space 保持 active 时,`type_text` 与 `cmd+a` 仍生效且 Space 不切换;`cmd+c` / `cmd+v` / `cmd+x` / `cmd+z` 经菜单项生效。+- Windows / Linux:`go vet ./...`、`go test ./...` 通过。+- `./scripts/run-tool-smoke-tests.sh` 通过完整 9-tool smoke 与 cursor idle smoke;`make check-docs` 通过。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyKeyboardSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`+- `apps/OpenComputerUseWindows/`、`apps/OpenComputerUseLinux/`+- `skills/open-computer-use/references/usage.md`+- `docs/ARCHITECTURE.md`、`docs/SECURITY.md`、`docs/RELIABILITY.md`+- `docs/references/macos-skylight-background-keyboard.md`、`docs/references/README.md`+- `docs/releases/feature-release-notes.md`、`THIRD_PARTY_NOTICES.md`diff --git a/docs/histories/2026-09/20260908-2300-background-window-visibility-and-spaces.md b/docs/histories/2026-09/20260908-2300-background-window-visibility-and-spaces.mdnew file mode 100644index 0000000..a0f8157--- /dev/null+++ b/docs/histories/2026-09/20260908-2300-background-window-visibility-and-spaces.md@@ -0,0 +1,40 @@+## [2026-09-08 23:00] | Task: 后台与其他 Space 窗口的状态读取、点击和可见性 keep-alive++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Fable 5.1`+* **Runtime**: `Claude Code CLI / macOS 27.0 arm64`++### 📥 User Query+> 完整目标:能从任何地方(包括其他 Space)拿到 AX tree、点击、输入,极其可靠且完全不抢用户焦点。要按 macOS / SkyLight 原语来设计,不要按 app 逐个打补丁。++### 🛠 Changes Overview+**Scope:** macOS snapshot / capture、SkyLight SPI、`sky_click` 校验、测试、参考文档。++**Key Actions:**+- **Root cause**: 被遮挡或在其他 Space 的 Chromium / Electron 窗口 AX tree 没有网页内容,是因为 app 收到 WindowServer occlusion 通知后把页面设为 hidden(Chromium 以 `NSWindow.occlusionState` 为准),与 Space 本身无关。+- **Occlusion keep-alive**: 新增 `WindowOcclusionKeepAlive`,在窗口未被遮挡时通过 `SLSPackagesEnableWindowOcclusionNotifications(cid, wid, false, &previous)` 固定 app 的“可见”状态,进程退出时恢复;对所有被驱动的窗口生效,不依赖 bundle id。+- **Off-screen windows**: `WindowCapture` 退到 `.optionAll`,SCK 捕获用 `onScreenWindowsOnly: false`,其他 Space 的窗口不再触发 activate / raise 恢复;`sky_click` 校验改为“窗口仍属于目标进程且 app 未隐藏”。+- **Lazy web AX tree**: Chromium 系 app(按 `Contents/Frameworks` 中的 Chromium / Electron / CEF 框架或 `AXManualAccessibility` 成功判定)第一次走 tree 若无 `AXWebArea`,窗口可见时最多重走 3 秒;被遮挡时在 tree 末尾追加说明。+- **Verification**: 新增 `OcclusionKeepAliveLiveTests` 与纯逻辑单元测试;参考文档记录 SPI 签名(lldb 反汇编)、否定方案与未解决边界。++### 🧠 Design Intent (Why)+*可见性是 WindowServer 的概念,app 只是通知的接收方;关闭一个窗口的 occlusion 通知就是在 macOS 层面告诉 app “你一直可见”,对任何 app 都成立。已经被遮挡的窗口没有原语能在不显示的情况下变可见,macOS 自己的答案是 virtual display,留到后续计划。*++### ✅ Verification+- `swift test`:171 tests,3 个默认跳过的实机 live test,0 failures。+- `OPEN_COMPUTER_USE_RUN_OCCLUSION_LIVE_TEST=1 swift test --filter OcclusionKeepAliveLiveTests`:可见时的 snapshot 包含网页内容并固定窗口;被 fixture 完全遮挡后页面仍 `visible`,第二次 snapshot 仍包含网页内容与截图,Chrome 未被激活。+- `SkyKeyboardLiveTests`、`SkyClickLiveTests` 通过;`./scripts/run-tool-smoke-tests.sh` 通过完整 9-tool smoke 与 cursor idle smoke;`make check-docs` 通过。+- 一次性实验:被遮挡的 Chrome 在 `sky_click` 后截图内容变化(渲染未冻结);全屏 Space 上的窗口 `sky_click` / `sky_key` 生效且 active Space 不切换;virtual display spike 证明已被遮挡的窗口放到 agent 显示器后页面可见、snapshot / click / type 生效。+- `OPEN_COMPUTER_USE_RUN_CROSS_SPACE_LIVE_TEST=1 swift test --filter CrossSpaceLiveTests`(需要第二个 Desktop):Chrome 在 Desktop 2、用户在 Desktop 1 时 snapshot 含网页内容与截图,`sky_click` / `sky_key` 生效,active Space 不变,Chrome 未被激活。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/WindowOcclusionKeepAlive.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`(`OcclusionKeepAliveLiveTests`、`CrossSpaceLiveTests`)+- `docs/references/macos-window-visibility-and-spaces.md`、`docs/references/README.md`+- `docs/ARCHITECTURE.md`、`docs/SECURITY.md`、`docs/RELIABILITY.md`+- `skills/open-computer-use/references/usage.md`、`docs/releases/feature-release-notes.md`+- `docs/exec-plans/active/20260908-cross-space-background-state.md`diff --git a/docs/histories/2026-09/20260908-2330-agent-display-window-placement.md b/docs/histories/2026-09/20260908-2330-agent-display-window-placement.mdnew file mode 100644index 0000000..ac2ecbe--- /dev/null+++ b/docs/histories/2026-09/20260908-2330-agent-display-window-placement.md@@ -0,0 +1,36 @@+## [2026-09-08 23:30] | Task: agent display:把已被隐藏的窗口停靠到 agent 专用 virtual display++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Fable 5.1`+* **Runtime**: `Claude Code CLI / macOS 27.0 arm64`++### 📥 User Query+> 把 virtual display 方案做成正式功能。++### 🛠 Changes Overview+**Scope:** 新 ObjC shim target、macOS snapshot 路由、三平台工具协议、测试、文档。++**Key Actions:**+- **Shim**: 新增 `packages/OpenComputerUseVirtualDisplayShim`(ObjC,ARC),用 `NSClassFromString` 解析 `CGVirtualDisplayDescriptor` / `CGVirtualDisplay` / `CGVirtualDisplaySettings` / `CGVirtualDisplayMode`,暴露 `ocu_virtual_display_create / destroy / is_supported` 三个 C 函数;标量属性经 KVC 设置,避免猜错 C 类型。+- **AgentDisplay**: 单例管理显示器与停靠窗口:按需创建 1920×1080 显示器,用 AX `kAXPosition` 把窗口 frame 移进显示器(纯 `agentDisplayPlacement` 计算落点),记录原位置;`restore` 移回并在无窗口停靠时销毁显示器;`atexit` 兜底恢复。+- **Protocol**: `get_app_state` 新增 `window_placement`(`keep` / `agent_display` / `restore`),非 `keep` 时 snapshot 使用 read-only recovery policy;fixture 拒绝;Windows / Linux 在 snapshot 前返回 unsupported。+- **Verification**: 纯逻辑单元测试 + `AgentDisplayLiveTests`:被完全遮挡、页面 hidden 的 Chrome 经 `park` 后页面 `visible`,snapshot 含网页内容与截图,`sky_click` / `sky_key` 生效,前台与鼠标不变;`restore` 后窗口回到原位、显示器数量恢复。++### 🧠 Design Intent (Why)+*已进入 hidden 的页面没有 WindowServer 原语能在不显示的情况下改回 visible;让窗口“真的在一块屏幕上”是 macOS 自己的做法(Screen Sharing headless 会话)。所以做成显式 opt-in:它改变窗口位置这件事必须由调用方决定,并且总能恢复。*++### ✅ Verification+- `swift test`:174 tests,5 个默认跳过的实机 live test,0 failures。+- `OPEN_COMPUTER_USE_RUN_AGENT_DISPLAY_LIVE_TEST=1 swift test --filter AgentDisplayLiveTests` 通过;其他四个 live test、`./scripts/run-tool-smoke-tests.sh`、`make check-docs` 通过。+- Windows / Linux:`go vet ./...`、`go test ./...` 通过。++### 📁 Files Modified+- `Package.swift`、`packages/OpenComputerUseVirtualDisplayShim/`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AgentDisplay.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`、`ComputerUseService.swift`、`ComputerUseToolDispatcher.swift`、`ToolDefinitions.swift`+- `apps/OpenComputerUseWindows/`、`apps/OpenComputerUseLinux/`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`(`AgentDisplayLiveTests` 等)+- `docs/references/macos-window-visibility-and-spaces.md`、`docs/ARCHITECTURE.md`、`docs/SECURITY.md`、`docs/RELIABILITY.md`+- `skills/open-computer-use/references/usage.md`、`docs/releases/feature-release-notes.md`+- `docs/exec-plans/completed/20260908-cross-space-background-state.md`diff --git a/docs/histories/2026-09/20260908-2355-background-input-timing-and-benchmark.md b/docs/histories/2026-09/20260908-2355-background-input-timing-and-benchmark.mdnew file mode 100644index 0000000..5f103b9--- /dev/null+++ b/docs/histories/2026-09/20260908-2355-background-input-timing-and-benchmark.md@@ -0,0 +1,33 @@+## [2026-09-08 23:55] | Task: 后台输入耗时观测、轮询替代固定等待、基准测试++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Fable 5.1`+* **Runtime**: `Claude Code CLI / macOS 27.0 arm64`++### 📥 User Query+> 后台点击 / 键盘到底多可靠?把所有环节都加上观测,把耗时压到毫秒级。++### 🛠 Changes Overview+**Scope:** SkyLight SPI 观测符号、`AgentDisplay` 轮询、`sky_key` / `sky_click` / snapshot 耗时日志、基准 live test、文档。++**Key Actions:**+- **TimingLog**: `OPEN_COMPUTER_USE_DEBUG_TIMING=1` 时向 stderr 输出各阶段毫秒耗时;`waitUntil` 轮询工具。+- **观测 SPI**: `SkyLightSPI` 新增只读的 `SLSCopySpacesForWindows`、`SLSCopyManagedDisplaySpaces`、`_AXUIElementGetWindow`,缺失时返回 nil。+- **AgentDisplay**: 显示器就绪改为轮询 bounds + 新 Space 出现;停靠改为轮询窗口 frame 进入显示器且 Space 归属更新;恢复改为轮询 frame 回到原位;SPI 缺失时退回固定等待。+- **sky_key**: 记录 activate / key_window / deliver / total;尝试用 AX focused window 与 focused element 观测 Chrome 变 key 均提前报告,实测导致输入丢失,因此保留 300 ms 固定 settle。+- **Benchmark**: `BackgroundInputBenchmarkLiveTests` 对被遮挡 Chrome 跑 N 轮 `sky_click` + `sky_key`,输出成功数与 p50 / p95 / max。++### 🧠 Design Intent (Why)+*能观测的等待用轮询,观测不到的保留实测出来的固定值并记录下来,不猜。*++### ✅ Verification+- `swift test`:175 tests,6 个默认跳过的实机 live test,0 failures。+- 基准:25/25 `sky_click`、25/25 `sky_key` 成功,前台不变;`sky_click` 观察到 p50 389 ms / p95 398 ms,`sky_key` 观察到 p50 270 ms / p95 279 ms。+- `AgentDisplayLiveTests` 通过:创建 334 ms、就绪 407 ms、停靠 230 ms、恢复 260 ms。+- 一次性实验:目标处于 synthetic key 状态时 HID 键盘事件仍送到真实前台 app。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TimingLog.swift`、`SkyLightSPI.swift`、`AgentDisplay.swift`、`SkyKeyboardSimulation.swift`、`SkyClickSimulation.swift`、`AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/BackgroundInputBenchmarkLiveTests.swift`+- `docs/RELIABILITY.md`、`docs/references/macos-window-visibility-and-spaces.md`diff --git a/docs/histories/2026-09/20260909-0030-latency-floor-hw-capture-app-matrix.md b/docs/histories/2026-09/20260909-0030-latency-floor-hw-capture-app-matrix.mdnew file mode 100644index 0000000..f20bb10--- /dev/null+++ b/docs/histories/2026-09/20260909-0030-latency-floor-hw-capture-app-matrix.md@@ -0,0 +1,35 @@+## [2026-09-09 00:30] | Task: 把后台输入间隔压到基准下限、HW 窗口截图、多 app 扫描++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Fable 5.1`+* **Runtime**: `Claude Code CLI / macOS 27.0 arm64`++### 📥 User Query+> 所有动作要最低延迟;不要机械地按 app 打补丁,从系统角度想;尽量多测不同技术栈的 app。++### 🛠 Changes Overview+**Scope:** 输入 recipe 的时间参数、窗口截图路径、AX 窗口绑定、基准与扫描 live test、文档。++**Key Actions:**+- **InputTiming knobs**: 所有固定间隔改为环境变量可调,默认值由 `BackgroundInputBenchmarkLiveTests` 扫描决定:focus record 40→10 ms,`sky_click` recipe scale 1→0.2,type_text chunk 20→0 ms,press_key 收尾 100→0 ms,`sky_key` key-window settle 300→0 ms、release 100→0 ms。结果:`sky_click` 调用返回 323→82 ms,`sky_key` 526→30 ms,50/50 成功、每轮恰好一次点击。+- **Compatibility correction (2026-09-09)**: macOS 26.6.2 的独立复验显示 `sky_key` 零 settle/release 会偶发丢键(18/20、49/50),各恢复 10 ms 后为 50/50;普通 `auto` 输入不在该后台基准覆盖范围内,因此 type_text chunk / press-key settle 恢复既有 20 / 100 ms 默认,仍可用环境变量显式调优。+- **系统事实**: 同一事件队列内不需要间隔;`SLPSPostEventRecordTo` 与 `CGEventPostToPid` 是两条通道,跨通道保留小间隔;`sky_click` 内部 down/up 间隔不能为 0(scale 0 全部失败)。+- **HW capture**: 截图主路径改为 `SLSHWCaptureWindowList`(15–45 ms,非活动全屏 Space 的窗口也能截),ScreenCaptureKit 为 fallback(它对这类窗口返回 -3811)。+- **AX↔CGWindow 绑定**: `_AXUIElementGetWindow` 直接把 AX root window 绑到 `CGWindowID`,消除 tree 与截图/坐标指向不同窗口的一类错误。+- **AppMatrixLiveTests**: 扫描运行中的全部 GUI app,read-only snapshot + 在空白文本框输入并删除标记,输出每 app 一行。++### 🧠 Design Intent (Why)+*间隔不是靠继承的数字,而是靠测量:能证明有顺序保证的地方归零,跨通道的地方留 2 倍余量,并把每个数字留成旋钮。截图与绑定改用 WindowServer / AX 的直接原语,对所有 app 一致。*++### ✅ Verification+- `swift test`:176 tests,7 个默认跳过的实机 live test,0 failures。+- 基准(50 轮,未 pin,默认参数):`sky_click` 50/50、`sky_key` 50/50,前台不变。+- 五个实机 live test(sky_key、sky_click、keep-alive、Desktop 2、agent display)在新默认值下全部通过。+- 多 app 扫描:有窗口的 9 个 app 全部拿到 tree 与截图(含全屏 Space 的 VS Code、Blender);System Settings 与 Slack 输入并清理成功。+- agent display 模式扫描(`OPEN_COMPUTER_USE_APP_MATRIX_PARK=1`):8 个非全屏窗口停靠、snapshot、输入、恢复全部成功,前台与鼠标不变,显示器结束后移除。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TimingLog.swift`、`SkyLightSPI.swift`、`SkyClickSimulation.swift`、`SkyKeyboardSimulation.swift`、`InputSimulation.swift`、`AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/BackgroundInputBenchmarkLiveTests.swift`、`AppMatrixLiveTests.swift`+- `docs/references/macos-window-visibility-and-spaces.md`、`docs/RELIABILITY.md`、`docs/ARCHITECTURE.md`diff --git a/docs/histories/2026-09/20260909-1551-pr65-compatibility-follow-up.md b/docs/histories/2026-09/20260909-1551-pr65-compatibility-follow-up.mdnew file mode 100644index 0000000..3cc9ee5--- /dev/null+++ b/docs/histories/2026-09/20260909-1551-pr65-compatibility-follow-up.md@@ -0,0 +1,43 @@+## [2026-09-09 15:51] | Task: 收紧 PR 65 的跨版本输入与窗口状态恢复++### 🤖 Execution Context+* **Agent ID**: TraeCode+* **Base Model**: GPT-5+* **Runtime**: TraeCode CLI / macOS 26.6.2 arm64++### 📥 User Query+> 审查 PR 65 后直接补充修订并提交到该 PR。++### 🛠 Changes Overview+**Scope:** PR 与 main 集成、macOS 输入时序、occlusion / agent display 生命周期、测试与文档。++**Key Actions:**+- 合入当前 `main`,保留已经发布的 drag 投递修复与文档。+- 普通 `auto` 输入恢复 20 ms type chunk 和 100 ms press-key settle;`sky_key` settle / release 改为跨版本实测稳定的 10 / 10 ms。+- occlusion cleanup 保存并恢复 SPI 返回的原始通知状态,正常 server shutdown 时主动清理,失败项保留供重试。+- agent display 只在确认窗口恢复后删除记录;失败时保留原坐标和显示器供重试,窗口已经关闭时安全清除。+- 补默认时序与恢复坐标单元测试,并更新 benchmark、架构、安全和发布说明。++### 🧠 Design Intent (Why)+*新能力可以允许旧系统兼容性继续增量完善,但不能把只在一个系统版本验证过的零延迟结论扩散到普通默认输入,也不能在恢复失败时丢失用户窗口的恢复信息。*++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TimingLog.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyKeyboardSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/WindowOcclusionKeepAlive.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AgentDisplay.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`+- `apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift`+- `apps/OpenComputerUse/Sources/OpenComputerUse/MacOSAppAgentProxy.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`、`docs/SECURITY.md`、`docs/RELIABILITY.md`+- `docs/references/background-input-benchmarks.md`、`docs/references/macos-window-visibility-and-spaces.md`+- `docs/releases/feature-release-notes.md`++### ✅ Verification+- `swift test`:185 tests,7 个 opt-in live tests 默认跳过,0 failures。+- `swift build -c release`:通过(仅有仓库既存 warning)。+- Linux / Windows:`go test ./... && go vet ./...` 均通过。+- `make check-docs`、action pinning、shell syntax、tool smoke、cursor idle smoke 与 `git diff --check` 均通过。+- macOS 26.6.2 后台输入基准:零 settle/release 为 49/50;10 / 10 ms 为 50/50。+- occlusion 与 agent-display 的 opt-in live fixture 在双显示器机器上无法满足其固定窗口布局前置条件,因此未把 setup failure 当成实现失败,也没有放宽断言;agent display 的创建、停靠、操作和恢复路径在现场均实际执行。独立 `sky_click` 偶发失败也可在当前 main 复现,不属于本 PR 引入的回归。diff --git a/docs/histories/2026-09/20260924-1740-harden-pr65-background-sessions.md b/docs/histories/2026-09/20260924-1740-harden-pr65-background-sessions.mdnew file mode 100644index 0000000..a231d17--- /dev/null+++ b/docs/histories/2026-09/20260924-1740-harden-pr65-background-sessions.md@@ -0,0 +1,55 @@+## [2026-09-24 17:40] | Task: 收敛 PR 65 的后台操作 session 边界++### 🤖 Execution Context+* **Agent ID**: TraeCode+* **Base Model**: GPT-5+* **Runtime**: TraeCode CLI / local macOS repository workspace++### 📥 User Query+> 评估 PR 65 后,直接在贡献者的 PR 上迭代并补充 maintainer commits。++### 🛠 Changes Overview+**Scope:** PR 与最新 main 集成、macOS 后台状态生命周期、agent display 恢复、JS REPL 参数、私有 API 防护、测试与文档。++**Key Actions:**+- 用 merge commit 同步最新 `main`,保留贡献者历史以及 derived implementation 对应的第三方许可声明。+- MCP turn-ended、stdio shutdown 与 app-agent connection close 统一清理 visual cursor、occlusion keep-alive 和 agent display;app runtime 的 distributed turn-ended 通知也执行同一 session cleanup。+- `window_placement=restore` 改为按 app PID 恢复该 runtime 停放的全部窗口,不再误用 snapshot 当前选中的 window ID。+- macOS `get_app_state` 不再声明 `readOnlyHint`,因为默认 snapshot 会维护 WindowServer occlusion 状态,显式 placement 还会移动窗口。+- JS app binding 将 `windowPlacement` 转发给 snapshot,将 `keyMethod` 转发给 `typeText` / `pressKey`,并补参数映射测试和同一 session 的用法。+- `CGVirtualDisplay` shim 捕获私有 KVC / selector 抛出的 Objective-C exception,系统 API 漂移时返回不可用结果而不是终止 host 进程。++### 🧠 Design Intent (Why)+后台能力会暂时改变进程级 WindowServer 状态,所以它的安全边界必须跟 MCP/REPL session 一致;正常 turn 或 connection 结束不能把窗口遗留在 virtual display。显式 restore 应按调用者指定的 app 恢复真实 parked set,而不是依赖移动后可能变化的 preferred-window 选择。默认 Codex 入口是 JS REPL,因此 native 新参数也必须在 code-first API 中可达。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AgentDisplay.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`+- `apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift`+- `apps/OpenComputerUse/Sources/OpenComputerUse/MacOSAppAgentProxy.swift`+- `packages/OpenComputerUseVirtualDisplayShim/OpenComputerUseVirtualDisplayShim.m`+- `scripts/node-repl/open-computer-use-repl.mjs`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`+- `scripts/node-repl/open-computer-use-repl.test.mjs`+- `docs/ARCHITECTURE.md`、`docs/SECURITY.md`、`docs/RELIABILITY.md`、`docs/references/js-repl.md`+- `skills/open-computer-use/references/usage.md`、`docs/releases/feature-release-notes.md`++### ✅ Verification+- `swift test`:186 tests,7 个 opt-in live tests 跳过,0 failures。+- `swift build -c release`:通过。+- 新增 JS background-option 映射测试通过;Node 全套只剩最新 `main` 已可复现的 Worker late-error timing failure。+- Linux / Windows:`go test ./...` 均通过。+- `./scripts/run-tool-smoke-tests.sh`:native tool 与 cursor idle smoke 均通过。+- `./scripts/check-docs.sh`、`./scripts/check-action-pinning.sh` 与 `git diff --check`:通过。+- 按仓库约束未运行会切换窗口 / Space 或注入真实输入的 opt-in live tests;agent-display live test 已改为覆盖按 PID 恢复入口,留待空闲桌面窗口执行。++### 🔄 2026-10-10 main 同步补充++- 再次把最新 `main` 合入贡献者分支,保留主线新增的 Windows 退化截图 fail-closed 防护,以及 PR 的 `key_method` / `window_placement` 跨平台参数兼容。+- Windows runtime 同时支持无效 PNG / 过小窗口过滤、无可用截图时拒绝 coordinate action,并继续对 macOS-only 的 `sky_key` 和 agent-display placement 返回明确的 unsupported 结果。+- `docs/RELIABILITY.md` 同时保留 macOS 后台输入、agent display、session cleanup 排障说明和 Windows screenshot-only failure 排障说明。+- 合并结果通过 `swift test`(186 tests,7 个 opt-in live tests 跳过)、Node 22 tests、Windows / Linux `go test ./...`、release build、Windows / Linux 制品构建、headless tool/cursor smoke、文档骨架与 action pinning 检查。+- 未运行会切换窗口、Space 或向真实应用注入输入的 opt-in macOS live tests。+- 首次推送后 `main` 又前进到 Linux `press_key` keysym 修复;再次同步该提交,合并树无内容冲突,并重新验证最终 head,避免 GitHub mergeability 计算与快速变化的 base 产生竞态。diff --git a/docs/references/README.md b/docs/references/README.mdindex 4291d10..febf97d 100644--- a/docs/references/README.md+++ b/docs/references/README.md@@ -21,3 +21,9 @@- 当抓包目录里的 `websocket/` + `local-sessions/` 仍不足以解释本地 tool / MCP 行为时,再补查 Codex 本地 `logs_2.sqlite`。- `codex-computer-use-cli.md`- 仓库内 `scripts/computer-use-cli/` 的用途、使用方法,以及为什么探测官方 bundled `computer-use` 时要优先走 `codex app-server` 代理而不是 direct stdio。+- `background-input-benchmarks.md`+ - 后台点击 / 键盘 / snapshot / agent display 的实测数字:环境、方法、每张表的 n 与分位数、决定默认间隔的扫描、多 app 扫描结果和复现命令(英文)。+- `macos-window-visibility-and-spaces.md`+ - 被遮挡 / 其他 Space 窗口的 AX tree 与截图为什么会丢、WindowServer occlusion 通知 keep-alive、Chromium 懒加载 AX tree、已验证与未解决的边界。+- `macos-skylight-background-keyboard.md`+ - `key_method=sky_key` 的实机研究记录:为什么后台 Chromium 收得到按键却不输入文字、yabai key-window record、菜单快捷键走 AX、跨 Space 验证,以及哪些方案被实测否定。diff --git a/docs/references/background-input-benchmarks.md b/docs/references/background-input-benchmarks.mdnew file mode 100644index 0000000..1064cb0--- /dev/null+++ b/docs/references/background-input-benchmarks.md@@ -0,0 +1,216 @@+# Background input benchmarks (macOS)++Measured numbers for the background paths in this repo: `sky_click`, `key_method=sky_key`,+`get_app_state` snapshots, the occlusion keep-alive, and `window_placement=agent_display`.+Every number below comes from a test that is in the repo and can be re-run with the commands at+the end. Nothing here is estimated.++## Environment++| Item | Value |+| --- | --- |+| Machine | Apple M5, 16 GB;另在 Apple Silicon macOS 26.6.2 复验跨版本默认值 |+| macOS | 27.0 (26A5425a), single built-in display 3024x1964 Retina;26.6.2 (25G83) compatibility run |+| Chrome | Google Chrome 152.0.7977.82, isolated `--user-data-dir`, `--app=` window 600x420 |+| Date | 2026-09-08 / 2026-09-09 |+| Foreground during tests | `OpenComputerUseFixture` window fully covering the target, its text field first responder |+| Machine idle | Yes for every table below. Runs with a user at the keyboard were discarded (their keys landed in the test windows). |++## 1. Background click and key latency++`BackgroundInputBenchmarkLiveTests`, target = isolated Chrome fully covered by the fixture.+"Returns" is the time until the dispatcher returns; "observed" is until the page reflected the+action (polled every 3 ms via the window title, so it includes Chromium's own processing plus+up to a few ms of polling).++### Final defaults, unpinned window, 50 cycles++| Action | Success | Returns p50 | Returns p95 | Returns max | Observed p50 | Observed p95 | Observed max |+| --- | --- | --- | --- | --- | --- | --- | --- |+| `sky_click` (single, exact-once) | 50/50 | 81.8 ms | 87.1 ms | 111.2 ms | 260.2 ms | 271.4 ms | 289.5 ms |+| `sky_key` `type_text` | 50/50 | 30.1 ms | 35.4 ms | 36.4 ms | 236.7 ms | 243.7 ms | 247.3 ms |++Frontmost app unchanged in every cycle. "Exact-once" means the page's click counter advanced by+exactly one per cycle; a double-fire would fail the run.++### Before tuning (inherited fixed gaps), unpinned, 25 to 50 cycles++| Action | Success | Returns p50 | Observed p50 |+| --- | --- | --- | --- |+| `sky_click` | 50/50 | 322.7 ms | 390.1 ms |+| `sky_key` `type_text` (300 ms key settle + 100 ms release) | 50/50 | 525.8 ms | 567.1 ms |++Observed latency barely moved (Chromium-internal work dominates it); the returned-call latency+is what the tuning removed.++## 2. Which gaps can be zero: the sweep that set the defaults++Same benchmark, 30 cycles per row, unpinned. Knobs: `FOCUS` = gap after each synthetic+focus/defocus record (`OPEN_COMPUTER_USE_FOCUS_RECORD_SETTLE_MS`); `SCALE` = multiplier on the+`sky_click` recipe gaps and renderer settle (`OPEN_COMPUTER_USE_SKY_CLICK_DELAY_SCALE`); type_text+chunk gap and press_key tail were 0 in every row except the last.++| FOCUS | SCALE | sky_click | sky_key | sky_click returns p50 | sky_key returns p50 |+| --- | --- | --- | --- | --- | --- |+| 0 ms | 0 | 0/30 | 5/30 | 5.6 ms | 8.3 ms |+| 20 ms | 0 | 0/30 | 7/30 | 53.9 ms | 55.8 ms |+| 20 ms | 0.05 | 30/30 | 30/30 | 67.1 ms | 54.5 ms |+| 0 ms | 0.1 | 30/30 | 30/30 | 33.3 ms | 6.5 ms |+| 10 ms | 0.1 | 30/30 | 30/30 | 57.6 ms | 29.8 ms |+| 20 ms | 0.1 | 30/30 | 30/30 | 79.8 ms | 53.7 ms |+| 0 ms | 0.25 | 30/30 | 30/30 | 69.6 ms | 5.7 ms |+| 0 ms | 1 | 30/30 | 30/30 | 234.1 ms | 5.7 ms |+| 40 ms | 1 (old defaults, with 20 ms chunk + 100 ms press tail) | 30/30 | 30/30 | 323.0 ms | 119.1 ms |++Readings:++- macOS 27 accepted zero-delay events in this sweep, but this does not establish a portable+ default. On macOS 26.6.2 the same pinned benchmark produced 18/20 and then 49/50 successful+ `sky_key` cycles with zero settle/release; setting both to 10 ms produced 50/50. Ordinary+ `auto` input was not covered by this background-only benchmark, so its established 20 ms+ type chunk gap and 100 ms press-key tail remain unchanged.+- The gap after a focus record is cross-channel: focus records travel `SLPSPostEventRecordTo`,+ mouse and key events travel `CGEventPostToPid`. 0 ms passed in every row that had non-zero+ click gaps, but the default is 10 ms (was 40) to keep margin.+- The `sky_click` recipe gaps cannot be 0. At scale 0 every click failed and the following key+ cycle mostly failed too; scale 0.05 passed everything. Default is 0.2 (2x the passing minimum),+ which is 20 ms after the primer and 20 ms renderer settle instead of 100 ms each.++Current defaults: `FOCUS=10`, `SCALE=0.2`, `TYPE_CHUNK=20`, `PRESS_KEY=100`, `SKY_KEY_SETTLE=10`,+`SKY_KEY_RELEASE=10`. All six are environment variables; see `docs/RELIABILITY.md`.++## 3. Pinned versus unpinned window++Same benchmark, 15 cycles per row, `sky_key` only. "Pinned" = the occlusion keep-alive had+disabled the window's occlusion notifications while it was visible (what `get_app_state` does),+so Chromium still considered the covered page visible.++| Window | Key settle | Success | Returns p50 | Observed p50 |+| --- | --- | --- | --- | --- |+| pinned | 0 ms | 15/15 | 220.9 ms* | 271.2 ms |+| unpinned | 0 ms | 15/15 | 221.9 ms* | 267.0 ms |+| unpinned | 20 ms | 15/15 | 246.9 ms* | 288.7 ms |+| unpinned | 100 ms | 15/15 | 325.9 ms* | 367.3 ms |+| unpinned | 300 ms | 15/15 | 526.5 ms* | 569.7 ms |++Compatibility follow-up on macOS 26.6.2, pinned isolated Chrome, 50 cycles: zero settle/release+gave 49/50 keys, while 10 ms settle/release gave 50/50 keys (clicks were 50/50 in both runs).++\* these rows still carried the old 40 ms focus gaps, 20 ms chunk gap and 100 ms release. The+point of the table is that the key settle does not affect success in either state.++## 4. Snapshot (`get_app_state`) phases++`OPEN_COMPUTER_USE_DEBUG_TIMING=1`, real apps on this machine, read-only policy.++| Phase | Measured |+| --- | --- |+| Window capture via `SLSHWCaptureWindowList` (now primary) | 14.8 to 32.6 ms across 9 apps, retina resolution |+| Window capture via ScreenCaptureKit (fallback) | 86.9 to 154.6 ms; fails with -3811 for windows on inactive fullscreen Spaces |+| AX tree walk | 26.4 to 81.4 ms (Chrome, 300 to 600 nodes) |+| Whole snapshot incl. PNG encoding | 47 ms (5-node app) to 1044 ms (Blender, 5 nodes but slow AX) |++## 5. Agent display (`window_placement=agent_display`)++`AgentDisplayLiveTests` and the parked survey, poll-based waits (no fixed sleeps).++| Step | Measured |+| --- | --- |+| Create the virtual display (`applySettings:` returns) | 333 to 353 ms |+| Display ready (bounds valid and its Space listed by WindowServer) | 407 to 572 ms from creation |+| Park a window (AX position set, frame inside the display, Space updated) | 144 to 230 ms with the display already up; 518 to 911 ms when the display is created for it |+| Restore a window to its original origin | 260 ms (live test); 496 to 1149 ms in the survey, dominated by waiting for the frame to report the old origin |+| Display removed after last restore | yes; online display count returned to 1 every time |++Chromium page on the parked window: `visibilityState` becomes `visible`, the full web tree is+present, capture and click/type work. Frontmost app and pointer unchanged in every run.++## 6. Reliability counts++| Test | Result |+| --- | --- |+| `BackgroundInputBenchmarkLiveTests`, 50 cycles, final defaults | 50/50 clicks, 50/50 keys, exact-once, frontmost unchanged |+| `SkyKeyboardLiveTests` (typing, backspace, `shift+a`, `cmd+a` via menu, release) | pass |+| `SkyClickLiveTests` (exactly one DOM click, no foreground side effects) | pass |+| `OcclusionKeepAliveLiveTests` (covered page stays visible, tree and screenshot intact) | pass |+| `CrossSpaceLiveTests` (window on a real second Desktop; snapshot, click, type; active Space unchanged) | pass |+| `AgentDisplayLiveTests` (hidden window parked, driven, restored, display removed) | pass |+| Physical (HID) keystrokes while a target holds synthetic key state | delivered to the real frontmost app, not to the target |+| `swift test` | 176 tests, 0 failures, 7 live tests skipped by default |++## 7. Multi-app survey (`AppMatrixLiveTests`)++27 running GUI apps, read-only snapshot each, plus click and type into an empty text field+where one exists. Input only targets fields whose current value is empty and whose center lies+inside the window, and the marker is removed afterwards.++### On the desktop (windows left where they were)++| App | Engine | Snapshot | Nodes | Web tree | Screenshot | Input |+| --- | --- | --- | --- | --- | --- | --- |+| Zed | native | 156 ms | 14 | | 306 KB | no empty field |+| TextEdit | native | 105 ms | 53 | | 63 KB | no empty field |+| System Settings | native | 213 ms | 98 | | 321 KB | typed and cleared |+| KiCad | native (wx) | 159 ms | 33 | | 424 KB | no empty field |+| Alma | native | 47 ms | 5 | | 5 KB | no empty field |+| Blender | GL | 1044 ms | 5 | | 555 KB (fullscreen Space; SCK failed, HW capture worked) | no empty field |+| VS Code | Electron | 434 ms | 310 | yes | 244 KB (fullscreen Space; SCK failed, HW capture worked) | no empty field |+| Slack | Electron | 375 ms | 261 | yes | 388 KB | typed and cleared |+| Google Chrome | Chromium | 772 ms | 610 | yes | 199 KB | no empty field |+| Safari | WebKit | 409 ms | 394 | yes | 269 KB | candidate was a popover child outside the window; rejected by design |+| Notes, Mail, Excel, Activity Monitor, Terminal, Helium, Figma, Canva, Linear, Devin | various | `cgWindowNotFound` | | | | apps running with no window; read-only policy does not activate them |++### Parked on the agent display, then restored++| App | Park | Snapshot on display | Nodes | Web tree | Screenshot | Input while parked | Restore | Back at origin |+| --- | --- | --- | --- | --- | --- | --- | --- | --- |+| Zed | 911 ms | 98 ms | 14 | | 335 KB | no empty field | 638 ms | yes |+| TextEdit | 652 ms | 61 ms | 53 | | 25 KB | no empty field | 550 ms | yes |+| Safari | 635 ms | 359 ms | 394 | yes | 280 KB | no empty field | 887 ms | yes |+| Alma | 518 ms | 24 ms | 5 | | 4 KB | no empty field | 496 ms | yes |+| Slack | 522 ms | 365 ms | 261 | yes | 250 KB | typed and cleared, 132 ms | 983 ms | yes |+| Google Chrome | 606 ms | 647 ms | 610 | yes | 215 KB | no empty field | 1149 ms | yes |+| System Settings | 557 ms | 154 ms | 101 | | 152 KB | typed and cleared, 108 ms | 624 ms | yes |+| KiCad | 689 ms | 74 ms | 27 | | 88 KB | no empty field | 619 ms | yes |+| VS Code, Blender | skipped | | | | | fullscreen windows cannot be repositioned | | |++Frontmost app and pointer unchanged for the whole run; the display was removed at the end.++## 8. What the numbers do not cover++- One machine, one macOS build (a beta). Private SPI behaviour must be re-measured per release.+- "Observed" latency includes Chromium's internal processing and the title polling; it is a+ ceiling, not a measurement of the runtime.+- Background typing was verified in one native app, one Electron app and Chromium. WebKit and+ native document editors were only read, not typed into, because the survey refuses to type+ into non-empty fields.++## Reproduce++```sh+# unit suite+swift test++# 50-cycle latency benchmark (defaults), window not pinned+OPEN_COMPUTER_USE_RUN_BACKGROUND_BENCH=1 OPEN_COMPUTER_USE_BENCH_CYCLES=50 OPEN_COMPUTER_USE_BENCH_UNPINNED=1 \+ swift test --filter BackgroundInputBenchmarkLiveTests++# sweep example: zero focus gap, tenth click gaps+OPEN_COMPUTER_USE_RUN_BACKGROUND_BENCH=1 OPEN_COMPUTER_USE_BENCH_CYCLES=30 OPEN_COMPUTER_USE_BENCH_UNPINNED=1 \+ OPEN_COMPUTER_USE_FOCUS_RECORD_SETTLE_MS=0 OPEN_COMPUTER_USE_SKY_CLICK_DELAY_SCALE=0.1 \+ swift test --filter BackgroundInputBenchmarkLiveTests++# live regressions+OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST=1 swift test --filter SkyKeyboardLiveTests+OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests+OPEN_COMPUTER_USE_RUN_OCCLUSION_LIVE_TEST=1 swift test --filter OcclusionKeepAliveLiveTests+OPEN_COMPUTER_USE_RUN_CROSS_SPACE_LIVE_TEST=1 swift test --filter CrossSpaceLiveTests # needs a second Desktop+OPEN_COMPUTER_USE_RUN_AGENT_DISPLAY_LIVE_TEST=1 swift test --filter AgentDisplayLiveTests++# app survey (types into real apps; run only when nobody is using the Mac)+OPEN_COMPUTER_USE_RUN_APP_MATRIX=1 swift test --filter AppMatrixLiveTests+OPEN_COMPUTER_USE_RUN_APP_MATRIX=1 OPEN_COMPUTER_USE_APP_MATRIX_PARK=1 swift test --filter AppMatrixLiveTests++# per-phase timings on any command+OPEN_COMPUTER_USE_DEBUG_TIMING=1 open-computer-use call get_app_state --args '{"app":"Google Chrome"}'+```diff --git a/docs/references/js-repl.md b/docs/references/js-repl.mdindex 99c5a2d..8323545 100644--- a/docs/references/js-repl.md+++ b/docs/references/js-repl.md@@ -41,15 +41,15 @@ await app.getAXState();runtime and emits the initial accessibility state. The returned binding exposes:```text-getAXState(options?)-getScreenshot(options?)-getAXStateAndScreenshot(options?)+getAXState({ emit?, textLimit?, maxTreeNodes?, maxTreeDepth?, windowPlacement? })+getScreenshot({ emit?, windowPlacement? })+getAXStateAndScreenshot({ emit?, textLimit?, maxTreeNodes?, maxTreeDepth?, windowPlacement? })click(elementIndexOrPoint, options?)drag([fromX, fromY], [toX, toY])-pressKey(key)+pressKey(key, options?)scroll(elementIndex, direction, pages?)setValue(elementIndex, value)-typeText(text)+typeText(text, options?)performSecondaryAction(elementIndex, action)```@@ -62,6 +62,24 @@ methods emit their result by default; passit. Use `nodeRepl.write(value)` for extra text and`await nodeRepl.emitImage(image)` for extra images.+Snapshot options use camelCase and map to the native tool arguments:+`textLimit`, `maxTreeNodes`, `maxTreeDepth`, and `windowPlacement`. Keyboard+options accept `keyMethod`. For example, one macOS session can park a window,+operate on it without taking foreground focus, and restore every window parked+for that app:++```js+var chrome = await cua.getApp("Google Chrome");+await chrome.getAXState({ windowPlacement: "agent_display" });+await chrome.typeText("hello", { keyMethod: "sky_key" });+await chrome.pressKey("cmd+a", { keyMethod: "sky_key" });+await chrome.getAXState({ windowPlacement: "restore" });+```++`windowPlacement` and `keyMethod: "sky_key"` are macOS-only native+capabilities. A parked window is also restored when the MCP/REPL connection+closes or the host reports the end of the assistant turn.+Top-level bindings persist across calls. Prefer `var` for reusable names, or use`js_reset` when the session really needs a fresh lexical scope. Resetting theREPL does not close applications or erase their UI state.diff --git a/docs/references/macos-skylight-background-keyboard.md b/docs/references/macos-skylight-background-keyboard.mdnew file mode 100644index 0000000..f2f3b2f--- /dev/null+++ b/docs/references/macos-skylight-background-keyboard.md@@ -0,0 +1,43 @@+# macOS SkyLight 后台键盘参考++## 用途++这份笔记记录 `key_method=sky_key` 的机制、实机验证结论和被否定的方案。所有结论来自 2026-09-08 在 macOS 27.0(build 26A5425a)上对被完全遮挡的隔离 Chrome `--app` 窗口和本仓库 fixture app 的受控测试;测试代码是 `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyKeyboardLiveTests.swift`。++## 问题++- 普通 AppKit 应用在后台就能接受 `CGEvent.postToPid` 键盘事件(fixture 在 inactive、非 key 状态下照常收到文字)。+- Chromium / Electron 不行:页面能收到 `keydown` / `keyup`,但 `document.hasFocus()` 为 false,不会触发 `input`,文字不会插入。原因在 Chromium 源码 `content/app_shim_remote_cocoa/render_widget_host_view_cocoa.mm`:页面焦点由 `OnWindowIsKeyChanged(keyTrackingWindow.isKeyWindow)` 驱动,只要 NSWindow 不是 AppKit 意义上的 key window,渲染进程就把页面当作未聚焦。++## `sky_key` 采用的序列++1. 复用 `sky_click` 的 target-only synthetic-active record(`[0x08]=0x0D`、`[0x8a]=0x01`),只发给目标 PSN,不碰真实前台应用。+2. 向目标 PSN 投递 yabai `window_manager_make_key_window` 的两条 record:`[0x04]=0xF8`、`[0x3a]=0x10`、`[0x20..0x30)=0xFF`、window id 在 `0x3c..0x3f`,`[0x08]` 先 `0x01` 再 `0x02`。目标 app 在进程内把该窗口设为 key window,不改变 WindowServer frontmost、z-order 或 Space。+3. 等待 300ms(Chrome 实测约 215ms 后 `document.hasFocus()` 变 true)。+4. 通过现有的公开 `CGEvent.postToPid` 路径发送键盘事件(`InputSimulation.typeText` / `pressKey`)。带 `cmd` 的单字符组合键改为在目标 app 的 AX 菜单栏里按 `AXMenuItemCmdChar` + `AXMenuItemCmdModifiers` 找到对应菜单项并 `AXPress`。+5. 再发 `[0x8a]=0x02` 的 deactivate record 撤销 synthetic 状态;Chrome 页面随即 blur。++## 实机验证结论++- 只发 synthetic-active record 不会让 Chrome 变 key;record 顺序必须是先 activation 后 key-window,反过来无效。+- 变 key 之后,公开 `postToPid` 就能输入;Cua Driver 的 `SLSEventAuthenticationMessage` 认证信封不是必要条件,窗口不是 key 时带认证也同样无效,所以本仓库没有引入它。+- 菜单栏快捷键(`cmd+a` / `cmd+c` / `cmd+v` / `cmd+x` / `cmd+z`)以键盘事件形式发到后台 Chromium 不会生效,因为 AppKit 只为 active app 分发 NSMenu key equivalent;但目标的 Edit 菜单项在后台是 enabled 的,`AXPress` 可以正确执行。`cmd+shift+left` 这类 key-binding 组合键仍走键盘事件。+- 目标窗口在另一个 Space(实测用全屏 Space)时同样可用:active Space 不切换,frontmost 不变。+- 前台 fixture 在整个过程中保持 active、key、first responder,`resignActive` / `resignKey` 计数为 0,真实指针不动,目标窗口 z-order 不变。+- 被 `cmd+h` 隐藏的 app 不接受 key-window record,`sky_key` 对隐藏 app fail closed,不代替用户 unhide。++## 明确否定的方案++- `SLPSSetFrontProcessWithOptions`(Cua `with_foreground_assist` / `with_menu_shortcut_activation`):会让真实前台应用 resign active,被前台 fixture 计数器捕获。+- `CGSSetConnectionProperty(cid, targetCid, "SetFrontmost", true)`(mac-cua Python 的 micro-activate):macOS 27 返回 1000,对 Chrome 无效。+- 只靠 `SLSEventAuthenticationMessage` + `SLEventPostToPid`、给键盘事件打 window field 40/51/91/92、先 `sky_click` 再输入、AX `AXFocused` / `AXMain` 设到窗口上:都无法让 Chrome 页面聚焦。++## 参考来源++- yabai:[asmvik/yabai](https://github.com/asmvik/yabai/tree/dd845723416f5fe92af49fad5ebab00369e07edd) `src/window_manager.c` 的 `window_manager_make_key_window`。+- Cua Driver:[trycua/cua](https://github.com/trycua/cua/tree/b8a0f32a06c75225ba24ebb5ab14f6507fa90d15) `libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs`、`skylight.rs`。+- Chromium:`content/app_shim_remote_cocoa/render_widget_host_view_cocoa.mm`(`isKeyTrackingWindowKey`、`OnWindowIsKeyChanged`、`performKeyEquivalent:`)。++## 兼容性检查++`sky_key` 复用 `sky_click` 的五个私有符号,缺任一个都 fail closed。macOS 更新后重新跑 `OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST=1 swift test --filter SkyKeyboardLiveTests`。diff --git a/docs/references/macos-window-visibility-and-spaces.md b/docs/references/macos-window-visibility-and-spaces.mdnew file mode 100644index 0000000..000f35c--- /dev/null+++ b/docs/references/macos-window-visibility-and-spaces.md@@ -0,0 +1,92 @@+# macOS 后台窗口可见性与 Space 参考++## 用途++这份笔记记录 `get_app_state` / `sky_click` / `sky_key` 如何处理被遮挡、位于其他 Space 的窗口,以及为什么这些问题要在 WindowServer(SkyLight)层面解决而不是逐个 app 打补丁。结论来自 2026-09-08 在 macOS 27.0 上对隔离 Chrome、Electron app 和本仓库 fixture 的受控测试;实机回归是 `OcclusionKeepAliveLiveTests`。++## 从 macOS 的角度看问题++一个“不抢用户焦点的 computer-use agent”要做三件事,每件在 macOS 里都有对应的原语:++| 需求 | macOS 原语 | 本仓库的用法 |+| --- | --- | --- |+| 读取窗口状态 | Accessibility API(按进程,与 Space 无关)+ ScreenCaptureKit 按 window id 捕获 backing store | `SnapshotBuilder`、`WindowCapture` |+| 窗口“是否可见” | WindowServer occlusion 状态,经 `SLSPackagesEnableWindowOcclusionNotifications` 注册的通知送达 app,AppKit 暴露为 `NSWindow.occlusionState` | `WindowOcclusionKeepAlive` |+| 向后台进程投递输入 | `SLPSPostEventRecordTo`(activation / key-window record)、`SLEventPostToPid` / `CGEventPostToPid` | `sky_click`、`sky_key` |+| Space | `SLSCopyManagedDisplaySpaces`、`SLSMoveWindowsToManagedSpace`、`SLSManagedDisplaySetCurrentSpace` | 只用于实验与验证,运行时不移动窗口 |++app 自己不会“知道”被遮挡;它只会收到 WindowServer 的 occlusion 通知。Chromium 收到通知后把页面设为 hidden:渲染进程停止出帧,`AXWebArea` 子树从 AX tree 消失。Electron、WebKit 同理。原生 AppKit 窗口不受影响。所以“其他 Space 的 AX tree 拿不到”的根因是 occlusion,不是 Space 本身。++## 已验证的行为++- 被完全遮挡的 Chrome:`document.visibilityState` 变为 `hidden`,AX tree 只剩窗口 chrome,没有 `AXWebArea`;`SCScreenshotManager` 仍能按 window id 捕获,`sky_click` 仍能触发 DOM click。+- 对窗口调用 `SLSPackagesEnableWindowOcclusionNotifications(cid, wid, false, &previous)`(跨进程可用,返回 0)后再遮挡:页面保持 `visible`,`AXWebArea` 保留,点击与输入正常,用户前台应用不受影响。重新启用通知不会补发当前状态,直到下一次真实变化。+- 已经被遮挡再关闭通知无效:app 会一直停留在 hidden。所以 keep-alive 只在窗口当前未被遮挡时启用,判定用 `CGWindowListCopyWindowInfo(.optionOnScreenAboveWindow)` 的几何采样。+- `SLSPackagesEnableWindowOcclusionNotifications` 的第四个参数是可选的 `uint8_t *previous` 输出指针;`SLSLockWindowVisibleRegion(cid, wid, uint64_t *seed)` 也有输出指针,且要求窗口属于本连接(返回 1000)。签名来自 lldb 反汇编,猜错会写坏内存。+- Chromium 系(Chrome、Electron、CEF)在第一个 AX client 请求后才异步构建 web AX tree,Chrome 需要最多约 2 秒;`AXManualAccessibility` 对 Electron 返回成功,对 Chrome 返回 `attributeUnsupported`,所以不能作为唯一信号。运行时改为检查 bundle 的 `Contents/Frameworks` 是否包含 Chromium / Electron / CEF 框架。+- 全屏 Space 里的 Chrome 窗口:`visibilityState` 仍为 `visible`,`sky_click` / `sky_key` 生效,但 fullscreen 会重建窗口结构,不能作为“另一个桌面 Space”的代理测试。+- 真实的第二个桌面(`CrossSpaceLiveTests`):Chrome 在 Desktop 2 上、用户停在 Desktop 1 且 fixture 在前台时,`SnapshotBuilder` 解析到该窗口并返回网页内容与截图,`sky_click` / `sky_key` 生效,active Space 不变,Chrome 未被激活。其他 Desktop 上的窗口 `kCGWindowIsOnscreen` 仍为 true,判断窗口所在 Space 要用 `SLSCopySpacesForWindows(cid, 0x7, [wid])`。+- macOS 26+ 第三方进程不能把别的 app 的窗口移到其他 Space:`SLSMoveWindowsToManagedSpace` / `SLSAddWindowsToSpaces` / `SLSSpaceCreate` 等都先检查 `SLSWindowManagementClientOperationsEnabled()`(依赖 window-management bridge delegate),不满足时静默返回。`SLSManagedDisplaySetCurrentSpace` 不受此限制。本仓库运行时也不需要移动窗口。++## 运行时行为++1. `get_app_state` 解析目标窗口时先看 on-screen 列表,再退到 `.optionAll`,因此其他 Space 的窗口不会再触发 activate / raise 的恢复逻辑。+2. 拍到窗口后,若窗口当前未被遮挡,就关闭它的 occlusion 通知(keep-alive),进程退出时统一恢复。之后用户遮挡它或切换 Space,app 仍认为自己可见。+3. Chromium 系 app 第一次走 tree 若没有 `AXWebArea`,在窗口可见时最多重走 3 秒;窗口被遮挡则在 tree 末尾追加说明,不等待、不激活。+4. `sky_click` / `sky_key` 只要求窗口仍属于目标进程且 app 未被隐藏,不再要求 on-screen。++## 已经被遮挡的窗口:agent display(`window_placement=agent_display`)++agent 第一次接触时就已经被遮挡或已在其他 Space 的 Chromium 窗口,没有 WindowServer 原语能在不显示的情况下把它“变可见”。macOS 自己的答案是 virtual display(`CGVirtualDisplay` 私有 ObjC 类,Screen Sharing 的 headless 会话就是这样做的)。本仓库把它做成 `get_app_state` 的显式 `window_placement=agent_display`:`AgentDisplay` 通过 `packages/OpenComputerUseVirtualDisplayShim`(ObjC,`NSClassFromString` 解析、无硬链接依赖)创建 1920×1080 的 agent 显示器,用 AX `kAXPosition` 把目标窗口的 frame 移进该显示器(不激活、不抬升),窗口随之落到该显示器的 Space 上并对其 app 真正可见;`window_placement=restore` 或进程退出时把窗口移回原位置,没有窗口停靠时销毁显示器。实机回归是 `AgentDisplayLiveTests`。spike 阶段的观察:++- `CGVirtualDisplayDescriptor` → `-[CGVirtualDisplay initWithDescriptor:]` → `CGVirtualDisplaySettings`(`hiDPI`、`modes` = `-[CGVirtualDisplayMode initWithWidth:height:refreshRate:]`)→ `applySettings:` 返回 true,得到新的 `displayID`,`SLSCopyManagedDisplaySpaces` 立刻多出一个显示器和它自己的 Space;用户的 active Space、frontmost app、鼠标位置都不变。新显示器排在主屏右侧(bounds 从 x=1512 开始),鼠标理论上可以滑进去。+- `SLSMoveWindowsToManagedSpace` 单独不能把窗口跨显示器移到那个 Space;先用 AX `kAXPosition` 把窗口 frame 移进 virtual display 的 bounds(不激活),再移 Space,`SLSCopySpacesForWindows` 就报告它在新 Space。+- 一个此前被完全遮挡、页面 `hidden`、没有 `AXWebArea` 的 Chrome 窗口,放到 virtual display 上后页面立刻 `visible`;生产 `SnapshotBuilder` 返回网页内容与截图,`sky_click` / `sky_key` 正常;结束后把窗口移回原位置与原 Space,进程退出后显示器消失。+- spike 里用 Swift KVC / IMP 直接驱动这些类会在 dealloc 崩溃;正式实现放在 ObjC shim 里,ARC 管理对象生命周期,`AgentDisplayLiveTests` 验证创建、停靠、点击输入、恢复与销毁全程无崩溃,显示器数量恢复。+- 已知副作用:agent 显示器排在主屏右侧,鼠标可能滑入;停靠期间窗口不在用户桌面上。所以它是显式 opt-in,不进入默认路径。++## 实测耗时(macOS 27.0,M 系列,2026-09-09)++被完全遮挡的隔离 Chrome,50 轮,全部成功且每轮恰好一次点击,前台不变:++| 动作 | 调用返回 p50 | 页面观察到 p50 | p95(观察到) | 调整前的调用返回 |+| --- | --- | --- | --- | --- |+| `sky_click` | 82 ms | 260 ms | 271 ms | 323 ms |+| `sky_key` `type_text` | 30 ms | 237 ms | 244 ms | 526 ms |++“页面观察到”包含 Chromium 自己的处理和测试轮询窗口标题的延迟,是外部无法压缩的部分。++固定间隔的取舍来自 `BackgroundInputBenchmarkLiveTests` 的扫描,而不是沿用继承来的数字:++- macOS 27 上同一事件队列曾在 0 ms 下达到 50/50;macOS 26.6.2 复验时,`sky_key` 先后只有 18/20、49/50 成功,settle / release 各 10 ms 后恢复为 50/50。因此 `sky_key` 默认保留 10 ms 跨版本余量;普通 `auto` 输入不参与这项优化,继续保持 type_text chunk 20 ms、press_key 收尾 100 ms 的既有默认。+- 跨通道的地方仍要间隔:activation / key-window record 走 `SLPSPostEventRecordTo`,鼠标键盘事件走 `CGEventPostToPid`,两条通道没有顺序保证。focus record 后的间隔 0 ms 实测也通过,默认取 10 ms(原 40 ms)留余量。+- `sky_click` recipe 内部的间隔不能为 0:scale 0 时 30 轮点击全部失败并把后续按键也带坏,scale 0.05 起全部成功,默认取 0.2(primer 后 20 ms、renderer settle 20 ms)。+- 所有间隔都是环境变量可调的校准旋钮(`OPEN_COMPUTER_USE_FOCUS_RECORD_SETTLE_MS`、`OPEN_COMPUTER_USE_SKY_CLICK_DELAY_SCALE`、`OPEN_COMPUTER_USE_TYPE_CHUNK_DELAY_MS`、`OPEN_COMPUTER_USE_PRESS_KEY_SETTLE_MS`、`OPEN_COMPUTER_USE_SKY_KEY_SETTLE_MS`、`OPEN_COMPUTER_USE_SKY_KEY_RELEASE_MS`)。当前默认依次为 10 ms、0.2、20 ms、100 ms、10 ms、10 ms。++agent display:创建显示器约 330–350 ms,WindowServer 登记 Space 后就绪约 400–570 ms,停靠窗口约 140–230 ms,恢复约 260 ms(都是轮询到条件满足即返回)。++窗口截图:主路径改为 `SLSHWCaptureWindowList(cid, &wid, 1, 0x800)`(WindowServer 直接读窗口 backing store,retina 分辨率,15–45 ms),ScreenCaptureKit 只做 fallback(90–155 ms)。ScreenCaptureKit 对处于非活动全屏 Space 的窗口返回 `-3811`,HW capture 对它们同样有效(实测 VS Code、Blender)。++窗口绑定:AX root window 用 `_AXUIElementGetWindow` 直接绑定到 `CGWindowID`,tree、截图、元素坐标指向同一个窗口;标题 / 面积启发式只做 fallback。++## 多 app 扫描(`AppMatrixLiveTests`,2026-09-09)++对本机运行中的 27 个 GUI app 各做一次 read-only snapshot,并在暴露空白文本框 / 搜索框的 app 上 `sky_click` + `sky_key` 输入标记再删除,全程前台不变:++- 有窗口的 app 全部拿到 tree 与截图:Zed、TextEdit、System Settings、KiCad(原生)、VS Code、Slack、Chrome(Chromium / Electron)、Safari(WebKit)、Blender(GL,tree 只有 5 个节点,截图正常)。+- 输入验证:System Settings 搜索框(原生)与 Slack 的 “Find a conversation” 搜索框(Electron)都收到标记并可删除;Safari 的候选是 popover 子元素,坐标在主窗口之外,`sky_click` 按设计拒绝。+- “cgWindowNotFound” 的行是没有窗口的 app(Notes、Mail、Excel、Activity Monitor 等只在后台运行),read-only policy 下不会去激活它们,这是预期结果。++同一扫描加 `OPEN_COMPUTER_USE_APP_MATRIX_PARK=1`(agent display 模式):把每个非全屏窗口停靠到 agent display,在那里 snapshot、点击、输入、清理,再移回原位。8 个 app(Zed、TextEdit、Safari、Alma、Slack、Chrome、System Settings、KiCad)全部:停靠后窗口位于 agent display、tree 与截图完整、`RESTORE back=true`(回到原坐标 2 px 内);Slack 与 System Settings 在停靠状态下输入并清理成功,click+type 108–132 ms;前台 app、鼠标位置不变,结束后显示器消失。停靠耗时 520–910 ms(每个 app 都重新创建显示器;显示器已存在时约 150–230 ms),恢复 500–1150 ms(轮询窗口 frame 回到原位)。全屏 Space 里的窗口(VS Code、Blender)不能移动 frame,跳过。++物理键盘路由:目标处于 synthetic key 状态时,HID 键盘事件仍然送到真实前台 app,不会进入目标窗口(`KeyRoutingExperiment` 实验,未入库)。++## 还没有解决的部分++- `CrossSpaceLiveTests` 需要机器上有第二个 Desktop(Mission Control 里手动创建;`SLSSpaceCreate` 造的 Space 不是 managed Desktop,Dock 的 Mission Control AX tree 只在鼠标悬停时才暴露 Spaces bar)。测试通过 `SLSManagedDisplaySetCurrentSpace` 切到 Desktop 2 启动 Chrome 再切回来,因为窗口不能被第三方进程跨 Space 移动。+- agent display 的鼠标滑入与显示器排列没有额外处理;`SLSSpaceCreate` 类跨 Space 移动窗口的 API 对第三方进程无效,所以停靠只能靠移动 frame。++## 参考来源++- Chromium `content/app_shim_remote_cocoa/web_contents_occlusion_checker_mac.mm`、`web_contents_view_cocoa.mm`(`isWindowOccluded:` 以 `NSWindow.occlusionState` 为准)。+- yabai `src/window_manager.c`(event record 布局)、Cua Driver(SkyLight 桥接)。diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.mdindex d6383cd..ab69021 100644--- a/docs/releases/feature-release-notes.md+++ b/docs/releases/feature-release-notes.md@@ -12,11 +12,15 @@| 日期 | 功能域 | 用户价值 | 变更摘要 || --- | --- | --- | --- || 2026-09-29 | 跨平台分发与 JavaScript 自动化 | Agent 可以通过持久 JavaScript 会话批量编排桌面操作,延迟异步错误不会污染下一次调用;DSH 用户获得一键兼容安装;不使用 npm 或 Homebrew 的用户也可直接下载三端 runtime 和 skill。 | 发布 `0.3.6`:新增 plugin `js` / `js_reset`、`ocu js` / `ocu repl`、DSH 验证式安装,并在 tag CI 完成后统一发布 macOS universal app、Linux / Windows 双架构 CLI、skill、SHA-256 校验和与 manifest。 |+| 2026-09-24 | macOS 后台操作收敛 | assistant turn 或 MCP/REPL connection 结束时会自动恢复窗口与 occlusion 状态;JS API 可直接传 `windowPlacement` 和 `keyMethod`。 | `restore` 改为按 app 恢复当前 runtime 停放的全部窗口;`get_app_state` 不再错误标为 read-only;私有 `CGVirtualDisplay` API 变化会 fail closed 而不是以 Objective-C exception 终止 host。一次性 CLI 的 park/action/restore 需放在同一个 `call --calls` 中。 || 2026-09-22 | JS REPL CLI | 用户无需先配置 MCP host,也可以直接用 `ocu js` 执行一次性 Computer Use JavaScript,或用 `ocu repl` 在终端中复用持久 binding;Agent 还能先读取结构化 capability 状态。 | npm launcher 新增 positional/stdin/file 三种 `js` 输入、持久 terminal REPL、`capabilities [--json]` 和稳定 help availability;`ocu mcp` 仍保持原生 9-tool surface。 || 2026-09-22 | JS REPL 错误传播与 read-back | `js` 代码抛出的异常立即以 `isError` 返回且保留 bindings,不再等到 30 s 超时并丢弃 session;macOS 上一批 action 不再为每个动作等待 150 ms 并重新读取整棵 AX tree,十次点击从数秒降到约一次 state read。 | REPL adapter 从 Node 的 `handleError`(Node 26)或 domain(Node 22)路径 settle 当前 evaluation;adapter 以 `OPEN_COMPUTER_USE_ACTION_READ_BACK=0` 启动 native runtime,macOS action 在该模式下返回短状态、跳过 settle 与 snapshot;默认 9-tool 行为不变,Linux / Windows 暂时忽略该变量。 || 2026-09-21 | JS REPL Computer Use | Agent 可以用异步 JavaScript 绑定 app,并在一个 tool round trip 内完成 inspect、action、条件和最终验证,不必在多轮间搬运所有中间状态。 | Codex plugin 通过持久 Node.js Worker 和 app-bound `cua` API 暴露 `js` / `js_reset`;`open-computer-use mcp` 继续保留既有 9-tool compatibility surface。 || 2026-09-17 | DeepSeek Harness 集成 | DSH 用户可以用一条命令安装 Open Computer Use,并在写入配置前发现错误的可执行文件或不兼容的 MCP server。 | 新增 DSH MCP 兼容安装器、turn-boundary 光标清理 hook 和非破坏性 skill 安装;安装前校验 server identity 与非空工具目录但不锁死工具数量,显式安装在 MCP 启动失败时 fail loud,并明确该路径不注册 DSH 的 first-class computer-use provider。 || 2026-09-10 | macOS 拖拽修复 | 显式开启全局指针路径后,窗口移动、文本拖选与 Finder 拖放可收到启动真实拖拽所需的事件;默认路径的限制也会明确呈现。 | 发布 `0.3.5`:补齐拖拽位移、手势事件编号与 HID 投递时序,增加投递路径说明;同步升级 Go MCP SDK 到 `v1.4.1`。 |+| 2026-09-08 | macOS agent display | 已经被遮挡或在其他 Space、内容被 app 隐藏的窗口,可以显式停靠到 agent 专用的不可见显示器上读取、点击和输入,用户的 Space、焦点、指针不受影响,用完可恢复原位。 | `get_app_state` 新增 `window_placement`(`keep` / `agent_display` / `restore`):通过 `CGVirtualDisplay` 创建 agent 显示器并用 AX 位置移动窗口 frame,`restore`、MCP server 正常关闭或进程退出时移回并销毁显示器;失败时保留恢复记录供重试;Windows / Linux 返回 unsupported。 |+| 2026-09-08 | macOS 被遮挡 / 其他 Space 窗口 | `get_app_state` 不再为其他 Space 的窗口激活或抬升应用;先看过一次的 Chromium / Electron 窗口在被遮挡或切换 Space 后仍能读到网页内容和实时截图;`sky_click` 可以点击被遮挡或其他 Space 的窗口。 | 窗口解析退到所有窗口列表;拍到未被遮挡的窗口后关闭其 WindowServer occlusion 通知,保存并在 MCP server 正常关闭或进程退出时恢复原始状态;Chromium 系 app 懒加载 AX tree 时最多重走 3 秒;`sky_click` 校验只要求窗口仍属于目标进程。 |+| 2026-09-08 | macOS 后台键盘输入 | Agent 可以向被遮挡或位于其他 Space 的 Chromium / Electron 窗口输入文字和快捷键,用户当前的应用、焦点、指针和 Space 都不受影响。 | 为 `type_text` / `press_key` 新增 `key_method`,提供 `auto`(默认)和 macOS-only 的 `sky_key`:复用 `sky_click` 的 target-only synthetic focus,加上 make-key-window record 让目标窗口在其进程内成为 key window,`cmd` 组合键改走目标菜单栏。Windows / Linux 返回稳定 unsupported。 || 2026-09-08 | drag 投递路径可见性 | 在默认安全配置下调用 `drag` 做窗口移动、文本拖选或 Finder 拖放时,结果会明确说明事件只投递给目标进程、为何没有效果,以及需要设置哪个环境变量,不再表现为“成功但无效果”。 | `drag` 结果新增一条 `Drag delivered via app_post` / `Drag delivered via global pointer path` 文本项;tool description 补充默认路径限制与 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 说明;skill 文档新增 Drag Delivery 章节;默认行为与安全门本身不变。 || 2026-09-08 | macOS 应用定位与 secondary action | 按名称定位应用时减少误选后台 helper,执行 secondary action 时避免动作错配。 | 发布 `0.3.4`:优先匹配普通应用;支持 Safari custom action 短名称,并修复过滤后的动作映射和名称冲突。 || 2026-09-06 | macOS secondary action | Agent 可以通过 Safari 暴露的 `close tab` secondary action 可靠关闭标签页,不会因内部动作被过滤而误执行另一项动作。 | macOS renderer 将 `Name:close tab ...` 形式的 AppKit custom action descriptor 显示为短名称,executor 从同一组过滤后的 raw actions 做等价名称匹配,并拒绝歧义匹配;短名称与其他原始 action 冲突时保留完整 descriptor,避免误执行同名原始动作。 |diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex e0a56c0..83cd477 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -97,6 +97,27 @@ let screenshotResultMaxDimension: CGFloat = 1280let screenshotResultMinScale: CGFloat = 0.25private let windowVisibilityRecoveryDelay: TimeInterval = 0.7private let axWebAreaRole = "AXWebArea"+// Chrome needs up to ~2s after AXManualAccessibility before the web tree exists.+private let webAreaRewalkAttempts = 30+private let webAreaRewalkInterval: TimeInterval = 0.1++/// Layer-0 windows ordered above `windowID` on screen, excluding this process+/// (the visual cursor overlay must not count as cover).+func coveringWindowBounds(above windowID: CGWindowID) -> [CGRect] {+ let infoList = CGWindowListCopyWindowInfo([.optionOnScreenAboveWindow, .excludeDesktopElements], windowID) as? [[String: Any]] ?? []+ let ownPID = ProcessInfo.processInfo.processIdentifier+ return infoList.compactMap { info in+ guard+ let ownerPID = info[kCGWindowOwnerPID as String] as? pid_t, ownerPID != ownPID,+ let layer = info[kCGWindowLayer as String] as? Int, layer == 0,+ let boundsDictionary = info[kCGWindowBounds as String] as? NSDictionary,+ let bounds = CGRect(dictionaryRepresentation: boundsDictionary)+ else {+ return nil+ }+ return bounds+ }+}private let axContentsAttribute = "AXContents"private let axVisibleChildrenAttribute = "AXVisibleChildren"private let compactGenericActionTargetMaxWidth: CGFloat = 240@@ -114,6 +135,8 @@ public struct AppSnapshot {let focusedSummary: String?let focusedElement: AXUIElement?let selectedText: String?+ /// The AX window the tree was rendered from (nil for fixture snapshots).+ let windowElement: AXUIElement?let elements: [Int: ElementRecord]@@ -164,7 +187,8 @@ enum SnapshotBuilder {}let appElement = AXUIElementCreateApplication(app.pid)- enableBestEffortAccessibilityModes(appElement)+ let lazyWebAccessibility = enableBestEffortAccessibilityModes(appElement)+ || appHasLazyWebAccessibility(bundleURL: app.runningApplication.bundleURL)let systemWide = AXUIElementCreateSystemWide()var focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)var focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)@@ -182,7 +206,13 @@ enum SnapshotBuilder {rootWindow = resolvedFocusedWindowvar windowTitle = stringValue(of: rootWindow, attribute: kAXTitleAttribute)- var windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)+ // Bind the AX window to its CGWindowID directly when the SPI allows it, so+ // the tree, the capture and every element frame refer to the same window;+ // the title/area heuristics remain the fallback.+ var windowCapture = TimingLog.measure("snapshot.window_capture") {+ WindowCapture.resolve(for: app.pid, exactWindowID: SkyLightSPI.shared.windowID(for: rootWindow))+ ?? WindowCapture.resolve(for: app.pid, titleHint: windowTitle)+ }if windowCapture == nil,recoveryPolicy == .allowActivation,recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {@@ -190,7 +220,8 @@ enum SnapshotBuilder {if let recoveredWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) {rootWindow = recoveredWindowwindowTitle = stringValue(of: recoveredWindow, attribute: kAXTitleAttribute)- windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)+ windowCapture = WindowCapture.resolve(for: app.pid, exactWindowID: SkyLightSPI.shared.windowID(for: recoveredWindow))+ ?? WindowCapture.resolve(for: app.pid, titleHint: windowTitle)}}@@ -207,7 +238,8 @@ enum SnapshotBuilder {focusedApplication: focusedApplication,systemWide: systemWide,textLimit: textLimit,- treeLimits: treeLimits+ treeLimits: treeLimits,+ lazyWebAccessibility: lazyWebAccessibility)}@@ -220,7 +252,8 @@ enum SnapshotBuilder {focusedApplication: AXUIElement?,systemWide: AXUIElement,textLimit: SnapshotTextLimit,- treeLimits: AccessibilityTreeLimits+ treeLimits: AccessibilityTreeLimits,+ lazyWebAccessibility: Bool) -> AppSnapshot {let windowBounds = windowCapture.boundslet screenshotPNGData = windowCapture.pngDataIfAvailable()@@ -234,13 +267,43 @@ enum SnapshotBuilder {)var renderer = TreeRenderer(context: context)- renderer.render(rootElement)+ TimingLog.measure("snapshot.tree_walk") { renderer.render(rootElement) }++ // Pin the window's visible state (WindowServer occlusion notifications+ // off) so covering it or moving it to another Space later does not hide+ // its content. Only possible while it is currently unoccluded.+ let pinned = WindowOcclusionKeepAlive.shared.keepVisible(windowID: windowCapture.windowID, bounds: windowBounds)++ // Engines that accept AXManualAccessibility build their web tree lazily+ // and drop it while hidden. Re-walk briefly while the window is visible;+ // when it is hidden, say so instead of waiting.+ var occlusionNote: String?+ if lazyWebAccessibility, !renderer.hasWebArea {+ if pinned {+ for _ in 0..<webAreaRewalkAttempts {+ Thread.sleep(forTimeInterval: webAreaRewalkInterval)+ renderer = TreeRenderer(context: context)+ renderer.render(rootElement)+ if renderer.hasWebArea {+ break+ }+ }+ } else {+ occlusionNote = "Note: this window is covered or on another Space, so the app hides its web content and it is not in the tree. Bring the window into view once (covering it afterwards keeps working), or call get_app_state with window_placement=agent_display to park it on the agent's own display."+ }+ }+if let menuBar = copyElement(appElement, attribute: kAXMenuBarAttribute),!CFEqual(menuBar, rootElement){renderer.render(menuBar)}+ var treeLines = renderer.lines+ if let occlusionNote {+ treeLines.append(occlusionNote)+ }+return AppSnapshot(app: app,windowTitle: windowTitle,@@ -249,10 +312,11 @@ enum SnapshotBuilder {targetWindowLayer: windowCapture.layer,screenshotPNGData: screenshotPNGData,mode: .accessibility,- treeLines: renderer.lines,+ treeLines: treeLines,focusedSummary: renderer.focusedSummary,focusedElement: focusedElement,selectedText: selectedText,+ windowElement: rootElement,elements: renderer.records)}@@ -409,17 +473,24 @@ enum SnapshotBuilder {focusedSummary: focusedSummary,focusedElement: nil,selectedText: nil,+ windowElement: nil,elements: records)}}-private func enableBestEffortAccessibilityModes(_ appElement: AXUIElement) {+/// Returns `true` when the app accepts `AXManualAccessibility`, i.e. it builds+/// its (web) accessibility tree lazily on request. Native AppKit apps report+/// the attribute as unsupported, which is the app-agnostic signal used instead+/// of bundle-identifier lists.+@discardableResult+private func enableBestEffortAccessibilityModes(_ appElement: AXUIElement) -> Bool {// Chromium/Electron apps may withhold parts of their AX tree until manual// accessibility is enabled. These private attributes are best-effort and// harmlessly fail on apps that do not support them.- _ = AXUIElementSetAttributeValue(appElement, "AXManualAccessibility" as CFString, kCFBooleanTrue)+ let manual = AXUIElementSetAttributeValue(appElement, "AXManualAccessibility" as CFString, kCFBooleanTrue)_ = AXUIElementSetAttributeValue(appElement, "AXEnhancedUserInterface" as CFString, kCFBooleanTrue)+ return manual == .success}private struct WindowCapture {@@ -428,8 +499,38 @@ private struct WindowCapture {let bounds: CGRectlet image: CGImage?+ /// Exact binding: the window the AX tree was walked from.+ static func resolve(for pid: pid_t, exactWindowID: CGWindowID?) -> WindowCapture? {+ guard let exactWindowID,+ let infoList = CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]]+ else {+ return nil+ }+ for info in infoList {+ guard+ let number = info[kCGWindowNumber as String] as? NSNumber, number.uint32Value == exactWindowID,+ let ownerPID = info[kCGWindowOwnerPID as String] as? pid_t, ownerPID == pid,+ let layer = info[kCGWindowLayer as String] as? Int,+ let boundsDictionary = info[kCGWindowBounds as String] as? NSDictionary,+ let bounds = CGRect(dictionaryRepresentation: boundsDictionary), !bounds.isEmpty+ else {+ continue+ }+ return WindowCapture(windowID: exactWindowID, layer: layer, bounds: bounds, image: captureImage(windowID: exactWindowID, bounds: bounds))+ }+ return nil+ }+static func resolve(for pid: pid_t, titleHint: String?) -> WindowCapture? {- guard let infoList = CGWindowListCopyWindowInfo([.optionOnScreenOnly], kCGNullWindowID) as? [[String: Any]] else {+ // On-screen windows first (front-to-back order is meaningful there);+ // fall back to every window of the pid so a window on another Space+ // still resolves instead of triggering the activate-and-raise recovery.+ resolve(for: pid, titleHint: titleHint, options: [.optionOnScreenOnly])+ ?? resolve(for: pid, titleHint: titleHint, options: [.optionAll])+ }++ private static func resolve(for pid: pid_t, titleHint: String?, options: CGWindowListOption) -> WindowCapture? {+ guard let infoList = CGWindowListCopyWindowInfo(options, kCGNullWindowID) as? [[String: Any]] else {return nil}@@ -466,9 +567,18 @@ private struct WindowCapture {return WindowCapture(windowID: best.windowID, layer: best.layer, bounds: best.bounds, image: image)}+ /// WindowServer's hardware window capture first (any Space, covered or not,+ /// ~10-40ms), ScreenCaptureKit when it is unavailable.private static func captureImage(windowID: CGWindowID, bounds: CGRect) -> CGImage? {+ if let image = TimingLog.measure("snapshot.capture_hw") { SkyLightSPI.shared.hardwareCaptureWindow(windowID) } {+ return image+ }+ return TimingLog.measure("snapshot.capture_sck") { captureImageWithScreenCaptureKit(windowID: windowID, bounds: bounds) }+ }++ private static func captureImageWithScreenCaptureKit(windowID: CGWindowID, bounds: CGRect) -> CGImage? {try? BlockingAsyncBridge.run(timeout: screenshotCaptureTimeout) {- let shareableContent = try await SCShareableContent.current+ let shareableContent = try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: false)guard let window = shareableContent.windows.first(where: { $0.windowID == windowID }) else {return nil}@@ -669,6 +779,7 @@ private struct RenderContext {private struct TreeRenderer {let context: RenderContext+ var hasWebArea = falsevar nextIndex = 0var lines: [String] = []var records: [Int: ElementRecord] = [:]@@ -692,6 +803,9 @@ private struct TreeRenderer {let index = nextIndexlet role = stringValue(of: root, attribute: kAXRoleAttribute) ?? "AXUnknown"+ if role == axWebAreaRole {+ hasWebArea = true+ }let subrole = stringValue(of: root, attribute: kAXSubroleAttribute)let baseRoleText = roleDescription(of: root, role: role, subrole: subrole)let label = stringValue(of: root, attribute: kAXDescriptionAttribute)diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AgentDisplay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AgentDisplay.swiftnew file mode 100644index 0000000..a5a5517--- /dev/null+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AgentDisplay.swift@@ -0,0 +1,315 @@+import AppKit+import ApplicationServices+import CoreGraphics+import Foundation+import OpenComputerUseVirtualDisplayShim++public enum WindowPlacement: String, CaseIterable, Sendable {+ case keep+ case agentDisplay = "agent_display"+ case restore+}++func parseWindowPlacement(_ rawValue: String?) throws -> WindowPlacement {+ let normalized = rawValue?+ .trimmingCharacters(in: .whitespacesAndNewlines)+ .lowercased() ?? WindowPlacement.keep.rawValue++ guard let placement = WindowPlacement(rawValue: normalized) else {+ let expected = WindowPlacement.allCases.map(\.rawValue).joined(separator: ", ")+ throw ComputerUseError.message(+ "Invalid window_placement '\(rawValue ?? "")'. Expected one of: \(expected)"+ )+ }++ return placement+}++/// Where a parked window goes on the agent display: inset from the top-left+/// corner, clamped so at least the top-left `minVisible` points stay inside the+/// display when the window is larger than it.+func agentDisplayPlacement(windowSize: CGSize, displayBounds: CGRect, inset: CGFloat = 40, minVisible: CGFloat = 200) -> CGPoint {+ let maxX = max(displayBounds.minX, displayBounds.maxX - max(windowSize.width, minVisible))+ let maxY = max(displayBounds.minY, displayBounds.maxY - max(windowSize.height, minVisible))+ return CGPoint(+ x: min(displayBounds.minX + inset, maxX),+ y: min(displayBounds.minY + inset, maxY)+ )+}++/// The agent's own display: a virtual display (the mechanism behind Screen+/// Sharing's headless sessions) that the user never sees. A window parked on it+/// is genuinely on screen for WindowServer, so its app renders it, exposes its+/// full accessibility tree and accepts input, while the user's Space, foreground+/// app and pointer are untouched. Parking moves the window's frame (AX+/// position, no activation); the original position is restored on `restore`+/// and at process exit, and the display is removed once nothing is parked.+final class AgentDisplay: @unchecked Sendable {+ static let shared = AgentDisplay()++ static let displaySize = CGSize(width: 1920, height: 1080)+ // Upper bounds for the polls; the observed values are far lower and are+ // logged with OPEN_COMPUTER_USE_DEBUG_TIMING=1.+ static let displaySettle: TimeInterval = 3.0+ static let moveSettle: TimeInterval = 2.0+ // Fallback sleeps when the observation SPIs are unavailable.+ static let displayFallbackSettle: TimeInterval = 0.5+ static let moveFallbackSettle: TimeInterval = 1.0++ struct ParkedWindow {+ let pid: pid_t+ let element: AXUIElement+ let originalPosition: CGPoint+ }++ private let lock = NSLock()+ private var handle: UnsafeMutableRawPointer?+ private(set) var displayID: CGDirectDisplayID = 0+ private(set) var displaySpaceIDs: Set<UInt64> = []+ private var parked: [CGWindowID: ParkedWindow] = [:]+ private var exitHookInstalled = false++ var isSupported: Bool {+ ocu_virtual_display_is_supported() != 0+ }++ var displayBounds: CGRect? {+ displayID == 0 ? nil : CGDisplayBounds(displayID)+ }++ func isParked(windowID: CGWindowID) -> Bool {+ lock.lock()+ defer { lock.unlock() }+ return parked[windowID] != nil+ }++ func parkedWindowIDs(pid: pid_t) -> [CGWindowID] {+ lock.lock()+ defer { lock.unlock() }+ return parked.filter { $0.value.pid == pid }.map(\.key)+ }++ /// Move `window` onto the agent display. Returns the display bounds.+ @discardableResult+ func park(windowID: CGWindowID, pid: pid_t, window: AXUIElement) throws -> CGRect {+ lock.lock()+ defer { lock.unlock() }++ let bounds = try ensureDisplay()+ if parked[windowID] != nil {+ return bounds+ }++ guard let position = axPoint(window, kAXPositionAttribute as String) else {+ throw ComputerUseError.stateUnavailable("window_placement 'agent_display' could not read the window position")+ }+ let size = axSize(window) ?? CGSize(width: 800, height: 600)+ let target = agentDisplayPlacement(windowSize: size, displayBounds: bounds)+ let start = TimingLog.now()+ try setAXPosition(window, target)+ parked[windowID] = ParkedWindow(pid: pid, element: window, originalPosition: position)+ installExitHookIfNeeded()+ waitForWindow(windowID, toLandOn: displaySpaceIDs, or: bounds)+ TimingLog.log("agent_display.park", since: start)+ return bounds+ }++ /// Wait until the window's frame sits inside the display and, when the+ /// Space SPI is available, WindowServer also reports it on one of the+ /// display's Spaces. Space membership alone precedes the frame update.+ private func waitForWindow(_ windowID: CGWindowID, toLandOn spaceIDs: Set<UInt64>, or bounds: CGRect) {+ let spi = SkyLightSPI.shared+ let landed = waitUntil(timeout: Self.moveSettle) {+ guard let frame = Self.windowFrame(windowID),+ bounds.contains(CGPoint(x: frame.minX + 1, y: frame.minY + 1))+ else { return false }+ if !spaceIDs.isEmpty, let spaces = spi.spaces(forWindow: windowID) {+ return spaces.contains(where: spaceIDs.contains)+ }+ return true+ }+ if landed == nil {+ Thread.sleep(forTimeInterval: Self.moveFallbackSettle)+ }+ }++ private static func windowFrame(_ windowID: CGWindowID) -> CGRect? {+ let infoList = CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]] ?? []+ for info in infoList {+ guard let number = info[kCGWindowNumber as String] as? NSNumber, number.uint32Value == windowID,+ let dictionary = info[kCGWindowBounds as String] as? NSDictionary,+ let bounds = CGRect(dictionaryRepresentation: dictionary) else { continue }+ return bounds+ }+ return nil+ }++ static func isRestored(frame: CGRect, to originalPosition: CGPoint, tolerance: CGFloat = 1) -> Bool {+ abs(frame.minX - originalPosition.x) < tolerance+ && abs(frame.minY - originalPosition.y) < tolerance+ }++ /// Put a parked window back where it was.+ func restore(windowID: CGWindowID) throws {+ lock.lock()+ defer { lock.unlock() }+ guard let entry = parked[windowID] else {+ return+ }+ let start = TimingLog.now()+ do {+ try setAXPosition(entry.element, entry.originalPosition)+ } catch {+ // A closed target no longer needs restoration. Forget it so the+ // virtual display can still be removed. Keep live failures for a+ // later retry instead of losing the original position.+ if Self.windowFrame(windowID) == nil {+ parked.removeValue(forKey: windowID)+ destroyDisplayIfIdle()+ return+ }+ throw error+ }+ let back = waitUntil(timeout: Self.moveSettle) {+ guard let frame = Self.windowFrame(windowID) else { return false }+ return Self.isRestored(frame: frame, to: entry.originalPosition)+ }+ if back == nil {+ Thread.sleep(forTimeInterval: Self.moveFallbackSettle)+ guard let frame = Self.windowFrame(windowID) else {+ parked.removeValue(forKey: windowID)+ destroyDisplayIfIdle()+ return+ }+ guard Self.isRestored(frame: frame, to: entry.originalPosition) else {+ throw ComputerUseError.stateUnavailable(+ "window_placement 'restore' could not confirm that the window returned to its original position"+ )+ }+ }+ parked.removeValue(forKey: windowID)+ TimingLog.log("agent_display.restore", since: start)+ destroyDisplayIfIdle()+ }++ func restoreAll() {+ lock.lock()+ let windowIDs = Array(parked.keys)+ lock.unlock()+ for windowID in windowIDs {+ try? restore(windowID: windowID)+ }+ }++ /// Restore every window parked for one application. Unlike process-exit+ /// cleanup, an explicit restore reports a live failure to the caller after+ /// attempting the remaining windows.+ func restoreAll(pid: pid_t) throws {+ let windowIDs = parkedWindowIDs(pid: pid)+ var firstError: Error?+ for windowID in windowIDs {+ do {+ try restore(windowID: windowID)+ } catch {+ firstError = firstError ?? error+ }+ }+ if let firstError {+ throw firstError+ }+ }++ // MARK: - Display lifecycle (call with lock held)++ private func ensureDisplay() throws -> CGRect {+ if displayID != 0, let bounds = displayBounds, !bounds.isEmpty {+ return bounds+ }+ guard isSupported else {+ throw ComputerUseError.message("window_placement 'agent_display' is unavailable: CGVirtualDisplay is not present on this macOS")+ }+ let spi = SkyLightSPI.shared+ let spacesBefore = Set((spi.managedDisplaySpaces() ?? [:]).values.flatMap { $0 })+ let start = TimingLog.now()+ var newHandle: UnsafeMutableRawPointer?+ let id = ocu_virtual_display_create("Open Computer Use", UInt32(Self.displaySize.width), UInt32(Self.displaySize.height), 60, &newHandle)+ guard id != 0, let newHandle else {+ throw ComputerUseError.message("window_placement 'agent_display' could not create the agent display")+ }+ handle = newHandle+ displayID = id+ TimingLog.log("agent_display.create", since: start)++ // Ready means WindowServer reports the display's bounds and has given+ // it a Space; without the Space SPI, fall back to a fixed settle.+ var newSpaces: Set<UInt64> = []+ let ready = waitUntil(timeout: Self.displaySettle) {+ guard let bounds = self.displayBounds, !bounds.isEmpty else { return false }+ guard let displays = spi.managedDisplaySpaces() else { return true }+ newSpaces = Set(displays.values.flatMap { $0 }).subtracting(spacesBefore)+ return !newSpaces.isEmpty+ }+ if ready == nil, displayBounds?.isEmpty != false {+ destroyDisplay()+ throw ComputerUseError.message("window_placement 'agent_display' timed out waiting for the agent display")+ }+ if newSpaces.isEmpty {+ Thread.sleep(forTimeInterval: Self.displayFallbackSettle)+ }+ displaySpaceIDs = newSpaces+ TimingLog.log("agent_display.ready", since: start)+ return displayBounds ?? .zero+ }++ private func destroyDisplayIfIdle() {+ if parked.isEmpty {+ destroyDisplay()+ }+ }++ private func destroyDisplay() {+ if let handle {+ ocu_virtual_display_destroy(handle)+ }+ handle = nil+ displayID = 0+ displaySpaceIDs = []+ }++ private func installExitHookIfNeeded() {+ guard !exitHookInstalled else { return }+ exitHookInstalled = true+ atexit {+ AgentDisplay.shared.restoreAll()+ }+ }++ // MARK: - AX helpers++ private func axPoint(_ element: AXUIElement, _ attribute: String) -> CGPoint? {+ var value: CFTypeRef?+ guard AXUIElementCopyAttributeValue(element, attribute as CFString, &value) == .success, let value else { return nil }+ var point = CGPoint.zero+ guard AXValueGetValue(value as! AXValue, .cgPoint, &point) else { return nil }+ return point+ }++ private func axSize(_ element: AXUIElement) -> CGSize? {+ var value: CFTypeRef?+ guard AXUIElementCopyAttributeValue(element, kAXSizeAttribute as CFString, &value) == .success, let value else { return nil }+ var size = CGSize.zero+ guard AXValueGetValue(value as! AXValue, .cgSize, &size) else { return nil }+ return size+ }++ private func setAXPosition(_ element: AXUIElement, _ point: CGPoint) throws {+ var mutable = point+ guard let value = AXValueCreate(.cgPoint, &mutable) else {+ throw ComputerUseError.message("could not encode the window position")+ }+ let result = AXUIElementSetAttributeValue(element, kAXPositionAttribute as CFString, value)+ guard result == .success else {+ throw ComputerUseError.stateUnavailable("window_placement could not move the window (AXError \(result.rawValue))")+ }+ }+}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 07e3376..4349590 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -51,6 +51,30 @@ func validateClickMethod(}}+public enum KeyMethod: String, CaseIterable, Sendable {+ case auto+ case skyKey = "sky_key"+}++func keyActionSnapshotRecoveryPolicy(for method: KeyMethod) -> SnapshotRecoveryPolicy {+ method == .skyKey ? .readOnly : .allowActivation+}++func parseKeyMethod(_ rawValue: String?) throws -> KeyMethod {+ let normalized = rawValue?+ .trimmingCharacters(in: .whitespacesAndNewlines)+ .lowercased() ?? KeyMethod.auto.rawValue++ guard let method = KeyMethod(rawValue: normalized) else {+ let expected = KeyMethod.allCases.map(\.rawValue).joined(separator: ", ")+ throw ComputerUseError.message(+ "Invalid key_method '\(rawValue ?? "")'. Expected one of: \(expected)"+ )+ }++ return method+}+func validateSkyClickArguments(method: ClickMethod,mouseButton: String,@@ -499,9 +523,31 @@ public final class ComputerUseService {public func getAppState(app query: String,textLimit: SnapshotTextLimit = .defaults,- treeLimits: AccessibilityTreeLimits = .defaults+ treeLimits: AccessibilityTreeLimits = .defaults,+ windowPlacement: WindowPlacement = .keep) throws -> ToolCallResult {- snapshotResult(for: try refreshSnapshot(for: query, textLimit: textLimit, treeLimits: treeLimits), style: .fullState)+ guard windowPlacement != .keep else {+ return snapshotResult(for: try refreshSnapshot(for: query, textLimit: textLimit, treeLimits: treeLimits), style: .fullState)+ }++ // Explicit placements never activate or raise; they only move the frame.+ let snapshot = try refreshSnapshot(for: query, textLimit: textLimit, treeLimits: treeLimits, recoveryPolicy: .readOnly)+ guard snapshot.mode != .fixture else {+ throw ComputerUseError.message("window_placement '\(windowPlacement.rawValue)' is not supported for fixture apps")+ }+ switch windowPlacement {+ case .agentDisplay:+ guard let windowID = snapshot.targetWindowID, let windowElement = snapshot.windowElement else {+ throw ComputerUseError.stateUnavailable("window_placement 'agent_display' requires a current target window. Run get_app_state again.")+ }+ try AgentDisplay.shared.park(windowID: windowID, pid: snapshot.app.pid, window: windowElement)+ case .restore:+ try AgentDisplay.shared.restoreAll(pid: snapshot.app.pid)+ case .keep:+ break+ }++ return snapshotResult(for: try refreshSnapshot(for: query, textLimit: textLimit, treeLimits: treeLimits, recoveryPolicy: .readOnly), style: .fullState)}public func click(@@ -779,14 +825,24 @@ public final class ComputerUseService {)}- public func typeText(app query: String, text: String) throws -> ToolCallResult {+ public func typeText(app query: String, text: String, keyMethod: KeyMethod = .auto) throws -> ToolCallResult {let snapshot = try currentSnapshot(for: query)if snapshot.mode == .fixture {+ try requireAutoKeyMethodForFixture(keyMethod)try FixtureBridge.post(FixtureCommand(kind: "type_text", identifier: "fixture-input", value: text))Thread.sleep(forTimeInterval: 0.15)return try actionResult(for: query)}+ if keyMethod == .skyKey {+ let target = try skyKeyTarget(for: snapshot)+ try InputSimulation.typeTextWithSkyLight(text, windowID: target.windowID, pid: target.pid)+ return snapshotResult(+ for: try refreshSnapshot(for: query, recoveryPolicy: keyActionSnapshotRecoveryPolicy(for: keyMethod)),+ style: .actionResult+ )+ }+if try typeTextBySettingFocusedValueIfAvailable(text, in: snapshot) {pauseBeforeReadBack(0.1)return try actionResult(for: query)@@ -800,18 +856,46 @@ public final class ComputerUseService {return try actionResult(for: query)}- public func pressKey(app query: String, key: String) throws -> ToolCallResult {+ public func pressKey(app query: String, key: String, keyMethod: KeyMethod = .auto) throws -> ToolCallResult {let snapshot = try currentSnapshot(for: query)if snapshot.mode == .fixture {+ try requireAutoKeyMethodForFixture(keyMethod)try FixtureBridge.post(FixtureCommand(kind: "press_key", identifier: "fixture-key-capture", value: key))Thread.sleep(forTimeInterval: 0.15)return try actionResult(for: query)}+ if keyMethod == .skyKey {+ let target = try skyKeyTarget(for: snapshot)+ try InputSimulation.pressKeyWithSkyLight(key, windowID: target.windowID, pid: target.pid)+ return snapshotResult(+ for: try refreshSnapshot(for: query, recoveryPolicy: keyActionSnapshotRecoveryPolicy(for: keyMethod)),+ style: .actionResult+ )+ }+try InputSimulation.pressKey(key, pid: snapshot.app.pid)return try actionResult(for: query)}+ private func requireAutoKeyMethodForFixture(_ keyMethod: KeyMethod) throws {+ guard keyMethod == .auto else {+ throw ComputerUseError.message(+ "key_method '\(keyMethod.rawValue)' is not supported for fixture apps"+ )+ }+ }++ private func skyKeyTarget(for snapshot: AppSnapshot) throws -> SkyKeyboardTarget {+ guard let windowID = snapshot.targetWindowID else {+ throw ComputerUseError.stateUnavailable(+ "key_method 'sky_key' requires a current on-screen target window. Run get_app_state again."+ )+ }++ return SkyKeyboardTarget(windowID: windowID, pid: snapshot.app.pid)+ }+public func setValue(app query: String, elementIndex: String, value: String) throws -> ToolCallResult {let snapshot = try currentSnapshot(for: query)let record = try lookupElement(snapshot: snapshot, index: elementIndex)diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swiftindex c7e5770..23a6b12 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift@@ -54,7 +54,8 @@ public final class ComputerUseToolDispatcher {treeLimits: AccessibilityTreeLimits.defaults.replacing(maxNodeCount: try optionalPositiveInt("max_tree_nodes", in: arguments),maxDepth: try optionalPositiveInt("max_tree_depth", in: arguments)- )+ ),+ windowPlacement: try parseWindowPlacement(optionalString("window_placement", in: arguments)))case "click":return try service.click(@@ -90,12 +91,14 @@ public final class ComputerUseToolDispatcher {case "type_text":return try service.typeText(app: requireString("app", in: arguments),- text: requireString("text", in: arguments)+ text: requireString("text", in: arguments),+ keyMethod: try parseKeyMethod(optionalString("key_method", in: arguments)))case "press_key":return try service.pressKey(app: requireString("app", in: arguments),- key: requireString("key", in: arguments)+ key: requireString("key", in: arguments),+ keyMethod: try parseKeyMethod(optionalString("key_method", in: arguments)))case "set_value":return try service.setValue(diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swiftindex 5a06785..4e569a6 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift@@ -99,6 +99,20 @@ enum InputSimulation {)}+ static func typeTextWithSkyLight(_ text: String, windowID: CGWindowID, pid: pid_t) throws {+ try SkyKeyboardDispatcher.typeText(+ target: SkyKeyboardTarget(windowID: windowID, pid: pid),+ text: text+ )+ }++ static func pressKeyWithSkyLight(_ specification: String, windowID: CGWindowID, pid: pid_t) throws {+ try SkyKeyboardDispatcher.pressKey(+ target: SkyKeyboardTarget(windowID: windowID, pid: pid),+ key: specification+ )+ }+static func scrollTargeted(at point: CGPoint, direction: String, pages: Double, pid: pid_t) throws {guard let event = CGEvent(scrollWheelEvent2Source: nil, units: .line, wheelCount: 2, wheel1: wheel1(direction: direction, pages: pages), wheel2: wheel2(direction: direction, pages: pages), wheel3: 0) else {throw ComputerUseError.message("Failed to create scroll event.")@@ -183,7 +197,9 @@ enum InputSimulation {}down.postToPid(pid)up.postToPid(pid)- Thread.sleep(forTimeInterval: 0.02)+ if InputTiming.typeChunkDelay > 0 {+ Thread.sleep(forTimeInterval: InputTiming.typeChunkDelay)+ }}}@@ -244,7 +260,9 @@ enum InputSimulation {activeFlags.remove(modifier.flag)}- Thread.sleep(forTimeInterval: 0.1)+ if InputTiming.pressKeySettle > 0 {+ Thread.sleep(forTimeInterval: InputTiming.pressKeySettle)+ }}private static func postMouseEvent(type: CGEventType, source: CGEventSource?, point: CGPoint, button: CGMouseButton, clickState: Int, delta: CGSize? = nil, eventNumber: Int64? = nil) throws {diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swiftindex 575bc71..dc7c873 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift@@ -19,12 +19,18 @@ Ask the user before taking destructive or externally visible actions such as senpublic final class StdioMCPServer {private let dispatcher: ComputerUseToolDispatcher+ private let backgroundStateReset: () -> Void- public init(service: ComputerUseService = ComputerUseService()) {+ public init(+ service: ComputerUseService = ComputerUseService(),+ backgroundStateReset: @escaping () -> Void = resetOpenComputerUseBackgroundWindowState+ ) {self.dispatcher = ComputerUseToolDispatcher(service: service)+ self.backgroundStateReset = backgroundStateReset}public func run() throws {+ defer { endSession() }while let line = readLine(strippingNewline: true) {guard !line.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty else {continue@@ -67,9 +73,7 @@ public final class StdioMCPServer {case "notifications/initialized":return nilcase "notifications/turn-ended":- VisualCursorSupport.performOnMain {- SoftwareCursorOverlay.reset()- }+ endSession()return nilcase "ping":return try encodeJSONRPCResult(id: id, result: [:])@@ -119,6 +123,16 @@ public final class StdioMCPServer {}}+ /// Release state that is scoped to one MCP client session. App-hosted MCP+ /// connections call this explicitly because they use `handle(line:)`+ /// instead of the stdio `run()` loop.+ public func endSession() {+ VisualCursorSupport.performOnMain {+ SoftwareCursorOverlay.reset()+ }+ backgroundStateReset()+ }+private func encodeJSONRPCResult(id: Any?, result: [String: Any]) throws -> String {try encode(["jsonrpc": "2.0",diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swiftindex 878d357..51b52cb 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift@@ -77,24 +77,15 @@ func skyClickEventRecipe(clickCount: Int) throws -> [SkyClickEventStep] {return steps}+/// The window must still exist and belong to the snapshot's process. It does+/// not need to be on-screen: SkyLight routes by window id, so a covered window+/// or one on another Space is a valid target (verified live on macOS 27).func skyClickWindowMatchesTarget(windowInfo: [[String: Any]],windowID: CGWindowID,pid: pid_t) -> Bool {- windowInfo.contains { info in- guard- let number = info[kCGWindowNumber as String] as? NSNumber,- number.uint32Value == windowID,- let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,- ownerPID.int32Value == pid,- let onScreen = info[kCGWindowIsOnscreen as String] as? NSNumber- else {- return false- }-- return onScreen.boolValue- }+ skyKeyWindowMatchesTarget(windowInfo: windowInfo, windowID: windowID, pid: pid)}enum SkyClickDispatcher {@@ -127,7 +118,9 @@ enum SkyClickDispatcher {}dispatchLock.lock()+ let start = TimingLog.now()defer {+ TimingLog.log("sky_click.total", since: start)dispatchLock.unlock()}@@ -185,8 +178,9 @@ enum SkyClickDispatcher {try spi.postToPid(event, pid: target.pid)event.postToPid(target.pid)- if step.delayAfter > 0 {- Thread.sleep(forTimeInterval: step.delayAfter)+ let delay = step.delayAfter * InputTiming.skyClickDelayScale+ if delay > 0 {+ Thread.sleep(forTimeInterval: delay)}}} catch {@@ -200,7 +194,10 @@ enum SkyClickDispatcher {// SkyLight delivery is asynchronous. Keep the target's AppKit// synthetic active state long enough for Chromium's renderer hop// to consume the final mouse-up before deactivating only the target.- Thread.sleep(forTimeInterval: 0.100)+ let rendererSettle = 0.100 * InputTiming.skyClickDelayScale+ if rendererSettle > 0 {+ Thread.sleep(forTimeInterval: rendererSettle)+ }try spi.endSyntheticTargetFocus(focusContext)}}@@ -222,17 +219,19 @@ enum SkyClickDispatcher {throw ComputerUseError.message("sky_click target is outside the snapshot window bounds")}- let windowInfo = CGWindowListCopyWindowInfo(- [.optionIncludingWindow],- target.windowID- ) as? [[String: Any]] ?? []+ if NSRunningApplication(processIdentifier: target.pid)?.isHidden == true {+ throw ComputerUseError.stateUnavailable(+ "sky_click target app is hidden. Unhide it first; sky_click does not change window visibility."+ )+ }+ let windowInfo = CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]] ?? []guard skyClickWindowMatchesTarget(windowInfo: windowInfo,windowID: target.windowID,pid: target.pid) else {throw ComputerUseError.stateUnavailable(- "sky_click target window is stale, off-screen, or no longer owned by the target app. Run get_app_state again."+ "sky_click target window no longer exists or is no longer owned by the target app. Run get_app_state again.")}}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyKeyboardSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyKeyboardSimulation.swiftnew file mode 100644index 0000000..8e940c3--- /dev/null+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyKeyboardSimulation.swift@@ -0,0 +1,262 @@+import AppKit+import ApplicationServices+import CoreGraphics+import Foundation++struct SkyKeyboardTarget {+ let windowID: CGWindowID+ let pid: pid_t+}++/// A `cmd`-chord expressed the way AppKit menus advertise their key+/// equivalents: `AXMenuItemCmdChar` plus the `AXMenuItemCmdModifiers` bitmask+/// (1 = shift, 2 = option, 4 = control; 8 marks a non-command equivalent).+struct SkyMenuKeyEquivalent: Equatable, Sendable {+ let character: String+ let modifiers: Int+}++/// Menu-bar key equivalents (`cmd+a`, `cmd+c`, `cmd+t`, ...) are dispatched by+/// AppKit through `NSMenu`, which only fires for the active app. A background+/// target instead needs the menu item invoked directly, which is exactly what+/// the user's keypress would have triggered. Only single-character keys with+/// a command modifier map to a menu equivalent; everything else (arrows,+/// return, plain characters, non-command chords) is delivered as key events.+func skyMenuKeyEquivalent(for parsed: ParsedKeyPress) -> SkyMenuKeyEquivalent? {+ guard parsed.modifiers.contains(where: { $0.flag == .maskCommand }),+ parsed.displayValue.count == 1+ else {+ return nil+ }++ var modifiers = 0+ for modifier in parsed.modifiers {+ switch modifier.flag {+ case .maskShift:+ modifiers |= 1+ case .maskAlternate:+ modifiers |= 2+ case .maskControl:+ modifiers |= 4+ default:+ break+ }+ }+ return SkyMenuKeyEquivalent(character: parsed.displayValue.uppercased(), modifiers: modifiers)+}++func skyMenuItemMatches(cmdChar: String?, cmdModifiers: Int?, enabled: Bool?, equivalent: SkyMenuKeyEquivalent) -> Bool {+ guard let cmdChar, enabled != false else {+ return false+ }+ return cmdChar.uppercased() == equivalent.character && (cmdModifiers ?? 0) == equivalent.modifiers+}++/// Owner check for keyboard targets. Unlike `sky_click`, the window does not+/// need to be on-screen: keyboard delivery is process-addressed, so a window on+/// another Space or hidden app is a valid target as long as it still exists+/// and belongs to the snapshot's process.+func skyKeyWindowMatchesTarget(+ windowInfo: [[String: Any]],+ windowID: CGWindowID,+ pid: pid_t+) -> Bool {+ windowInfo.contains { info in+ guard+ let number = info[kCGWindowNumber as String] as? NSNumber,+ number.uint32Value == windowID,+ let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber+ else {+ return false+ }++ return ownerPID.int32Value == pid+ }+}++/// `sky_key`: deliver keyboard input to a background window without changing+/// the real foreground app.+///+/// Keyboard events posted to a pid reach any AppKit window, but Chromium and+/// Electron only insert text while their NSWindow is key (page focus is driven+/// by `OnWindowIsKeyChanged`). The recipe therefore reuses the `sky_click`+/// target-only synthetic-active state, then posts yabai's key-window record+/// pair so the target app makes the window key internally, waits for it to+/// take effect, delivers the ordinary `CGEvent.postToPid` keyboard events (or+/// presses the matching menu item for command chords), and releases the+/// synthetic state again. The real frontmost app keeps its active, key and+/// first-responder state throughout; the target is never raised and the+/// pointer never moves.+enum SkyKeyboardDispatcher {+ private static let dispatchLock = NSLock()+ static let keyWindowFallbackSettleDefaultMilliseconds = 10.0+ static let releaseSettleDefaultMilliseconds = 10.0+ /// The records and key events are ordered, but target applications can+ /// process them asynchronously. macOS 26 live testing dropped occasional+ /// keys with zero delay, so retain a small cross-version safety margin.+ static let keyWindowFallbackSettle = InputTiming.milliseconds(+ "OPEN_COMPUTER_USE_SKY_KEY_SETTLE_MS",+ default: keyWindowFallbackSettleDefaultMilliseconds+ )+ static let releaseSettle = InputTiming.milliseconds(+ "OPEN_COMPUTER_USE_SKY_KEY_RELEASE_MS",+ default: releaseSettleDefaultMilliseconds+ )++ static func typeText(+ target: SkyKeyboardTarget,+ text: String,+ spi: SkyLightSPI = .shared+ ) throws {+ try deliver(to: target, spi: spi) {+ try InputSimulation.typeText(text, pid: target.pid)+ }+ }++ static func pressKey(+ target: SkyKeyboardTarget,+ key: String,+ spi: SkyLightSPI = .shared+ ) throws {+ // Reject bad key specs before touching the target's focus state.+ let parsed = try KeyPressParser.parse(key)+ try deliver(to: target, spi: spi) {+ if let equivalent = skyMenuKeyEquivalent(for: parsed),+ let item = menuItem(matching: equivalent, pid: target.pid) {+ let result = AXUIElementPerformAction(item, kAXPressAction as CFString)+ guard result == .success else {+ throw ComputerUseError.message(+ "sky_key could not press the menu item for '\(key)' (AXError \(result.rawValue))"+ )+ }+ Thread.sleep(forTimeInterval: 0.1)+ return+ }++ try InputSimulation.pressKey(key, pid: target.pid)+ }+ }++ private static func deliver(+ to target: SkyKeyboardTarget,+ spi: SkyLightSPI,+ _ body: () throws -> Void+ ) throws {+ guard spi.capability.isAvailable else {+ throw ComputerUseError.message(+ "key_method 'sky_key' is unavailable: \(spi.capability.unavailableReason)"+ )+ }++ dispatchLock.lock()+ defer {+ dispatchLock.unlock()+ }++ try validate(target: target)++ if NSWorkspace.shared.frontmostApplication?.processIdentifier == target.pid {+ try body()+ return+ }++ let start = TimingLog.now()+ let focusContext = try spi.beginSyntheticTargetFocus(+ targetPID: target.pid,+ targetWindowID: target.windowID+ )+ do {+ try spi.makeSyntheticTargetWindowKey(focusContext)+ TimingLog.log("sky_key.activate", since: start)+ let keyStart = TimingLog.now()+ waitForKeyWindow(target, spi: spi)+ TimingLog.log("sky_key.key_window", since: keyStart)+ let deliverStart = TimingLog.now()+ try body()+ TimingLog.log("sky_key.deliver", since: deliverStart)+ if releaseSettle > 0 {+ Thread.sleep(forTimeInterval: releaseSettle)+ }+ } catch {+ try? spi.endSyntheticTargetFocus(focusContext)+ throw error+ }+ try spi.endSyntheticTargetFocus(focusContext)+ TimingLog.log("sky_key.total", since: start)+ }++ private static func waitForKeyWindow(_ target: SkyKeyboardTarget, spi: SkyLightSPI) {+ if keyWindowFallbackSettle > 0 {+ Thread.sleep(forTimeInterval: keyWindowFallbackSettle)+ }+ }++ private static func validate(target: SkyKeyboardTarget) throws {+ // A hidden app (cmd+h) never makes its windows key, so keys would be+ // silently dropped. Fail closed instead of unhiding on the user's behalf.+ if NSRunningApplication(processIdentifier: target.pid)?.isHidden == true {+ throw ComputerUseError.stateUnavailable(+ "sky_key target app is hidden. Unhide it first; sky_key does not change window visibility."+ )+ }++ // `.optionAll`, not `.optionIncludingWindow`: the latter omits windows+ // that are off-screen (hidden app, minimized, or on another Space).+ let windowInfo = CGWindowListCopyWindowInfo(+ [.optionAll],+ kCGNullWindowID+ ) as? [[String: Any]] ?? []+ guard skyKeyWindowMatchesTarget(+ windowInfo: windowInfo,+ windowID: target.windowID,+ pid: target.pid+ ) else {+ throw ComputerUseError.stateUnavailable(+ "sky_key target window no longer exists or is no longer owned by the target app. Run get_app_state again."+ )+ }+ }++ // MARK: - Menu-bar key equivalents++ private static func menuItem(matching equivalent: SkyMenuKeyEquivalent, pid: pid_t) -> AXUIElement? {+ let application = AXUIElementCreateApplication(pid)+ guard let menuBar = copyValue(application, kAXMenuBarAttribute) else {+ return nil+ }+ return firstMenuItem(in: menuBar as! AXUIElement, matching: equivalent, depth: 0)+ }++ private static func firstMenuItem(in element: AXUIElement, matching equivalent: SkyMenuKeyEquivalent, depth: Int) -> AXUIElement? {+ guard depth < 6 else {+ return nil+ }++ for child in copyValue(element, kAXChildrenAttribute) as? [AXUIElement] ?? [] {+ let role = copyValue(child, kAXRoleAttribute) as? String+ if role == kAXMenuItemRole as String,+ skyMenuItemMatches(+ cmdChar: copyValue(child, "AXMenuItemCmdChar") as? String,+ cmdModifiers: (copyValue(child, "AXMenuItemCmdModifiers") as? NSNumber)?.intValue,+ enabled: (copyValue(child, kAXEnabledAttribute) as? NSNumber)?.boolValue,+ equivalent: equivalent+ ) {+ return child+ }++ if let nested = firstMenuItem(in: child, matching: equivalent, depth: depth + 1) {+ return nested+ }+ }++ return nil+ }++ private static func copyValue(_ element: AXUIElement, _ attribute: String) -> CFTypeRef? {+ var value: CFTypeRef?+ guard AXUIElementCopyAttributeValue(element, attribute as CFString, &value) == .success else {+ return nil+ }+ return value+ }+}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swiftindex 4f14930..6786031 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift@@ -1,3 +1,4 @@+import ApplicationServicesimport CoreGraphicsimport Darwinimport Foundation@@ -47,8 +48,32 @@ func skyLightActivationRecord(windowID: CGWindowID, focused: Bool) -> [UInt8] {return record}+/// yabai's `window_manager_make_key_window` record pair. Posted to the target+/// PSN after the app-activation record, it makes AppKit inside the target treat+/// `windowID` as its key window without a front-process switch or raise. This+/// is what Chromium's page focus keys off (`OnWindowIsKeyChanged`), so it is+/// the difference between key events being received and text being inserted.+func skyLightKeyWindowRecords(windowID: CGWindowID) -> [[UInt8]] {+ var record = [UInt8](repeating: 0, count: 0xF8)+ record[0x04] = 0xF8+ record[0x3A] = 0x10+ record[0x3C] = UInt8(truncatingIfNeeded: windowID)+ record[0x3D] = UInt8(truncatingIfNeeded: windowID >> 8)+ record[0x3E] = UInt8(truncatingIfNeeded: windowID >> 16)+ record[0x3F] = UInt8(truncatingIfNeeded: windowID >> 24)+ for index in 0x20..<0x30 {+ record[index] = 0xFF+ }+ return [UInt8(0x01), UInt8(0x02)].map { eventKind in+ var copy = record+ copy[0x08] = eventKind+ return copy+ }+}+struct SkyLightSPICapability: Equatable, Sendable {let missingSymbols: [String]+ var feature: String = "click"var isAvailable: Bool {missingSymbols.isEmpty@@ -59,16 +84,25 @@ struct SkyLightSPICapability: Equatable, Sendable {return "available"}- return "missing private click symbols: \(missingSymbols.joined(separator: ", "))"+ return "missing private \(feature) symbols: \(missingSymbols.joined(separator: ", "))"}}-/// Runtime-only bridge for the private SkyLight functions used by `sky_click`.+protocol WindowOcclusionControlling: AnyObject {+ var occlusionCapability: SkyLightSPICapability { get }++ @discardableResult+ func setWindowOcclusionNotificationsEnabled(_ enabled: Bool, windowID: CGWindowID) throws -> Bool+}++/// Runtime-only bridge for the private SkyLight functions used by `sky_click`+/// and `sky_key`.///-/// The declarations and event-field recipe are derived from the MIT-licensed-/// Cua Driver and yabai implementations. Keep all undocumented ABI in this-/// file so a future macOS compatibility change has one review boundary.-final class SkyLightSPI: @unchecked Sendable {+/// The declarations, event-field recipe and key-window records are derived+/// from the MIT-licensed Cua Driver and yabai implementations. Keep all+/// undocumented ABI in this file so a future macOS compatibility change has+/// one review boundary.+final class SkyLightSPI: WindowOcclusionControlling, @unchecked Sendable {static let shared = SkyLightSPI()private static let frameworkPath = "/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight"@@ -77,6 +111,18 @@ final class SkyLightSPI: @unchecked Sendable {private static let setWindowLocationSymbol = "CGEventSetWindowLocation"private static let postEventRecordSymbol = "SLPSPostEventRecordTo"private static let getProcessForPIDSymbol = "GetProcessForPID"+ private static let mainConnectionSymbol = "SLSMainConnectionID"+ private static let enableWindowOcclusionNotificationsSymbol = "SLSPackagesEnableWindowOcclusionNotifications"+ // Read-only observation symbols used to replace fixed sleeps with polls.+ private static let copySpacesForWindowsSymbol = "SLSCopySpacesForWindows"+ private static let copyManagedDisplaySpacesSymbol = "SLSCopyManagedDisplaySpaces"+ private static let axElementGetWindowSymbol = "_AXUIElementGetWindow"+ // WindowServer reads the window's backing store on the GPU: works for+ // covered windows and for windows on inactive (fullscreen) Spaces where+ // ScreenCaptureKit fails, in ~10-40ms. The wrapper is the 4-argument form+ // (cid, windowIDs, count, options); 0x800 ignores the global clip shape.+ private static let hardwareCaptureWindowListSymbol = "SLSHWCaptureWindowList"+ private static let hardwareCaptureOptions: UInt32 = 0x800private static let applicationServicesPath = "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices"private typealias PostToPidFunction = @convention(c) (pid_t, UnsafeMutableRawPointer?) -> Void@@ -87,6 +133,14 @@ final class SkyLightSPI: @unchecked Sendable {private typealias SetWindowLocationFunction = @convention(c) (UnsafeMutableRawPointer?, Double, Double) -> Voidprivate typealias PostEventRecordFunction = @convention(c) (UnsafeRawPointer?, UnsafePointer<UInt8>?) -> Int32private typealias GetProcessForPIDFunction = @convention(c) (pid_t, UnsafeMutableRawPointer?) -> Int32+ private typealias MainConnectionFunction = @convention(c) () -> UInt32+ // (cid, wid, enable, previousStateOut) — the out pointer is optional; the+ // ABI was read from the function prologue on macOS 27 (strb w21 / cbz x19).+ private typealias EnableWindowOcclusionNotificationsFunction = @convention(c) (UInt32, CGWindowID, UInt8, UnsafeMutablePointer<UInt8>?) -> Int32+ private typealias CopySpacesForWindowsFunction = @convention(c) (UInt32, Int32, CFArray) -> Unmanaged<CFArray>?+ private typealias CopyManagedDisplaySpacesFunction = @convention(c) (UInt32) -> Unmanaged<CFArray>?+ private typealias AXElementGetWindowFunction = @convention(c) (AXUIElement, UnsafeMutablePointer<CGWindowID>) -> AXError+ private typealias HardwareCaptureWindowListFunction = @convention(c) (UInt32, UnsafePointer<CGWindowID>, Int32, UInt32) -> Unmanaged<CFArray>?private let frameworkHandle: UnsafeMutableRawPointer?private let applicationServicesHandle: UnsafeMutableRawPointer?@@ -95,8 +149,17 @@ final class SkyLightSPI: @unchecked Sendable {private let setWindowLocationFunction: SetWindowLocationFunction?private let postEventRecordFunction: PostEventRecordFunction?private let getProcessForPIDFunction: GetProcessForPIDFunction?+ private let mainConnectionFunction: MainConnectionFunction?+ private let enableWindowOcclusionNotificationsFunction: EnableWindowOcclusionNotificationsFunction?+ private let copySpacesForWindowsFunction: CopySpacesForWindowsFunction?+ private let copyManagedDisplaySpacesFunction: CopyManagedDisplaySpacesFunction?+ private let axElementGetWindowFunction: AXElementGetWindowFunction?+ private let hardwareCaptureWindowListFunction: HardwareCaptureWindowListFunction?let capability: SkyLightSPICapability+ /// Occlusion keep-alive (`WindowOcclusionKeepAlive`) is a separate optional+ /// capability so a missing symbol never disables `sky_click` / `sky_key`.+ let occlusionCapability: SkyLightSPICapabilityprivate init() {let handle = dlopen(Self.frameworkPath, RTLD_LAZY | RTLD_GLOBAL)@@ -108,6 +171,12 @@ final class SkyLightSPI: @unchecked Sendable {setWindowLocationFunction = Self.resolve(handle: handle, symbol: Self.setWindowLocationSymbol)postEventRecordFunction = Self.resolve(handle: handle, symbol: Self.postEventRecordSymbol)getProcessForPIDFunction = Self.resolve(handle: appServicesHandle, symbol: Self.getProcessForPIDSymbol)+ mainConnectionFunction = Self.resolve(handle: handle, symbol: Self.mainConnectionSymbol)+ enableWindowOcclusionNotificationsFunction = Self.resolve(handle: handle, symbol: Self.enableWindowOcclusionNotificationsSymbol)+ copySpacesForWindowsFunction = Self.resolve(handle: handle, symbol: Self.copySpacesForWindowsSymbol)+ copyManagedDisplaySpacesFunction = Self.resolve(handle: handle, symbol: Self.copyManagedDisplaySpacesSymbol)+ axElementGetWindowFunction = Self.resolve(handle: appServicesHandle, symbol: Self.axElementGetWindowSymbol)+ hardwareCaptureWindowListFunction = Self.resolve(handle: handle, symbol: Self.hardwareCaptureWindowListSymbol)var missingSymbols: [String] = []if postToPidFunction == nil {@@ -126,6 +195,76 @@ final class SkyLightSPI: @unchecked Sendable {missingSymbols.append(Self.getProcessForPIDSymbol)}capability = SkyLightSPICapability(missingSymbols: missingSymbols)++ var missingOcclusionSymbols: [String] = []+ if mainConnectionFunction == nil {+ missingOcclusionSymbols.append(Self.mainConnectionSymbol)+ }+ if enableWindowOcclusionNotificationsFunction == nil {+ missingOcclusionSymbols.append(Self.enableWindowOcclusionNotificationsSymbol)+ }+ occlusionCapability = SkyLightSPICapability(missingSymbols: missingOcclusionSymbols, feature: "occlusion")+ }++ // MARK: - Observation (read-only; nil when the symbol is absent)++ /// Space ids the window currently belongs to.+ func spaces(forWindow windowID: CGWindowID) -> [UInt64]? {+ guard let mainConnectionFunction, let copySpacesForWindowsFunction else { return nil }+ let result = copySpacesForWindowsFunction(mainConnectionFunction(), 0x7, [NSNumber(value: windowID)] as CFArray)?.takeRetainedValue()+ return (result as? [NSNumber])?.map(\.uint64Value)+ }++ /// User (type 0) Space ids per display, keyed by display identifier.+ func managedDisplaySpaces() -> [String: [UInt64]]? {+ guard let mainConnectionFunction, let copyManagedDisplaySpacesFunction else { return nil }+ guard let displays = copyManagedDisplaySpacesFunction(mainConnectionFunction())?.takeRetainedValue() as? [[String: Any]] else { return nil }+ var result: [String: [UInt64]] = [:]+ for display in displays {+ let identifier = display["Display Identifier"] as? String ?? UUID().uuidString+ result[identifier] = ((display["Spaces"] as? [[String: Any]]) ?? []).compactMap { space in+ guard (space["type"] as? NSNumber)?.intValue == 0 else { return nil }+ return (space["id64"] as? NSNumber)?.uint64Value+ }+ }+ return result+ }++ /// Capture one window's backing store (retina scale). nil when the symbol is+ /// absent or WindowServer returned nothing; callers fall back to SCK.+ func hardwareCaptureWindow(_ windowID: CGWindowID) -> CGImage? {+ guard let mainConnectionFunction, let hardwareCaptureWindowListFunction else { return nil }+ var id = windowID+ guard let array = hardwareCaptureWindowListFunction(mainConnectionFunction(), &id, 1, Self.hardwareCaptureOptions)?.takeRetainedValue(),+ CFArrayGetCount(array) > 0+ else { return nil }+ let image = unsafeBitCast(CFArrayGetValueAtIndex(array, 0), to: CGImage.self)+ return image.width > 0 && image.height > 0 ? image : nil+ }++ /// CGWindowID behind an AX window element.+ func windowID(for element: AXUIElement) -> CGWindowID? {+ guard let axElementGetWindowFunction else { return nil }+ var windowID: CGWindowID = 0+ guard axElementGetWindowFunction(element, &windowID) == .success, windowID != 0 else { return nil }+ return windowID+ }++ /// Enable or disable WindowServer occlusion notifications for one window of+ /// another process. Returns the previous state. Disabling pins the target's+ /// AppKit `occlusionState` at its current value.+ @discardableResult+ func setWindowOcclusionNotificationsEnabled(_ enabled: Bool, windowID: CGWindowID) throws -> Bool {+ guard let mainConnectionFunction, let enableWindowOcclusionNotificationsFunction else {+ throw ComputerUseError.message("occlusion keep-alive is unavailable: \(occlusionCapability.unavailableReason)")+ }++ var previous: UInt8 = 0+ let status = enableWindowOcclusionNotificationsFunction(mainConnectionFunction(), windowID, enabled ? 1 : 0, &previous)+ guard status == 0 else {+ throw ComputerUseError.message("SLSPackagesEnableWindowOcclusionNotifications failed (CGError \(status))")+ }+ return previous != 0}func postToPid(_ event: CGEvent, pid: pid_t) throws {@@ -175,16 +314,44 @@ final class SkyLightSPI: @unchecked Sendable {targetWindowID: targetWindowID)try postActivationCommand(plan.activateTarget)- Thread.sleep(forTimeInterval: 0.040)+ if InputTiming.focusRecordSettle > 0 {+ Thread.sleep(forTimeInterval: InputTiming.focusRecordSettle)+ }return SkyLightSyntheticFocusContext(deactivateTarget: plan.deactivateTarget)}+ /// Make the target window key inside the target app while it is in the+ /// synthetic-active state. Must follow `beginSyntheticTargetFocus`; posted+ /// the other way round the records are ignored. `endSyntheticTargetFocus`+ /// releases the key state again (the target sees resignKey/blur).+ func makeSyntheticTargetWindowKey(_ context: SkyLightSyntheticFocusContext) throws {+ guard let postEventRecordFunction else {+ throw unavailableError()+ }++ let command = context.deactivateTarget+ for record in skyLightKeyWindowRecords(windowID: command.windowID) {+ let status = command.psn.withUnsafeBytes { psnBytes in+ record.withUnsafeBufferPointer { recordBytes in+ postEventRecordFunction(psnBytes.baseAddress, recordBytes.baseAddress)+ }+ }+ guard status == 0 else {+ throw ComputerUseError.message(+ "key_method 'sky_key' synthetic key-window event failed (OSStatus \(status))"+ )+ }+ }+ }+func endSyntheticTargetFocus(_ context: SkyLightSyntheticFocusContext) throws {try postActivationCommand(context.deactivateTarget)- Thread.sleep(forTimeInterval: 0.040)+ if InputTiming.focusRecordSettle > 0 {+ Thread.sleep(forTimeInterval: InputTiming.focusRecordSettle)+ }}private func unavailableError() -> ComputerUseError {diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TimingLog.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TimingLog.swiftnew file mode 100644index 0000000..34e49ac--- /dev/null+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TimingLog.swift@@ -0,0 +1,84 @@+import Foundation++/// Millisecond timing for the background-input and window-placement paths.+/// Enabled with `OPEN_COMPUTER_USE_DEBUG_TIMING=1`; lines go to stderr as+/// `[open-computer-use] timing <label> <ms>ms` so a tool session can be+/// profiled without changing tool output.+enum TimingLog {+ nonisolated(unsafe) static var enabled: Bool = {+ let value = ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_DEBUG_TIMING"]?+ .trimmingCharacters(in: .whitespacesAndNewlines).lowercased() ?? ""+ return ["1", "true", "yes", "on"].contains(value)+ }()++ static func now() -> TimeInterval {+ ProcessInfo.processInfo.systemUptime+ }++ static func log(_ label: String, since start: TimeInterval) {+ guard enabled else { return }+ fputs(String(format: "[open-computer-use] timing %@ %.1fms\n", label, (now() - start) * 1000), stderr)+ }++ static func measure<T>(_ label: String, _ body: () throws -> T) rethrows -> T {+ let start = now()+ defer { log(label, since: start) }+ return try body()+ }+}++/// Calibration knobs for the fixed gaps in the input recipes. Every gap+/// defaults to the value verified by the benchmark on macOS 27 and can be+/// overridden per environment (milliseconds) when a target needs more slack.+enum InputTiming {+ static let typeChunkDelayDefaultMilliseconds = 20.0+ static let pressKeySettleDefaultMilliseconds = 100.0++ static func milliseconds(+ _ variable: String,+ default fallback: Double,+ environment: [String: String] = ProcessInfo.processInfo.environment+ ) -> TimeInterval {+ guard let raw = environment[variable], let ms = Double(raw) else { return fallback / 1000 }+ return max(0, ms) / 1000+ }++ // Benchmark on macOS 27 (30-50 cycles each, covered Chrome, exact-once+ // clicks): focus settle 0 ms and click scale 0.05 still pass; click scale 0+ // fails every click and poisons the next key cycle. Defaults keep a 2x+ // margin over the passing minimum. Focus records and CGEvents travel+ // different channels, which is why a small gap between them stays.+ /// Gap after a synthetic focus / defocus record (sky_click and sky_key). Was 40 ms.+ static let focusRecordSettle = milliseconds("OPEN_COMPUTER_USE_FOCUS_RECORD_SETTLE_MS", default: 10)+ /// Multiplier for the sky_click recipe gaps (move/primer/click pairs) and the renderer settle. Was 1.+ static let skyClickDelayScale: Double = {+ guard let raw = ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_SKY_CLICK_DELAY_SCALE"], let scale = Double(raw) else { return 0.2 }+ return max(0, scale)+ }()+ /// Gap between Unicode chunks in type_text. Keep the established default for+ /// app/toolkit compatibility; callers can explicitly tune it after testing.+ static let typeChunkDelay = milliseconds(+ "OPEN_COMPUTER_USE_TYPE_CHUNK_DELAY_MS",+ default: typeChunkDelayDefaultMilliseconds+ )+ /// Gap after a press_key chord. Keep the established default because the+ /// target may process its event queue asynchronously.+ static let pressKeySettle = milliseconds(+ "OPEN_COMPUTER_USE_PRESS_KEY_SETTLE_MS",+ default: pressKeySettleDefaultMilliseconds+ )+}++/// Poll `condition` every `interval` until it holds or `timeout` elapses.+/// Returns the elapsed time when it held, or nil on timeout.+@discardableResult+func waitUntil(timeout: TimeInterval, interval: TimeInterval = 0.01, _ condition: () -> Bool) -> TimeInterval? {+ let start = TimingLog.now()+ while TimingLog.now() - start < timeout {+ if condition() {+ return TimingLog.now() - start+ }+ Thread.sleep(forTimeInterval: interval)+ }+ return nil+}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swiftindex 35dff4b..11a8f97 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift@@ -71,13 +71,17 @@ public enum ToolDefinitions {ToolDefinition(name: "get_app_state",description: "Start an app use session if needed, then get the state of the app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`.",- annotations: readOnlyAnnotations(),+ annotations: defaultAnnotations(),inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier"),"text_limit": textLimitProperty(description: "Maximum text characters to return. Use \"max\" for full text. Defaults to 500."),"max_tree_nodes": positiveIntegerProperty(description: "Maximum accessibility tree nodes to render. Defaults to 1200."),"max_tree_depth": positiveIntegerProperty(description: "Maximum accessibility tree depth to render. Defaults to 64."),+ "window_placement": stringProperty(+ description: "keep (default) leaves the window where it is. agent_display (macOS) parks the app's window on an invisible display owned by the agent so a covered or other-Space window renders, exposes its full tree and accepts input without touching the user's Space, focus or pointer; the window disappears from the user's desktop until restore or process exit. restore puts it back.",+ enumValues: WindowPlacement.allCases.map(\.rawValue)+ ),],required: ["app"])@@ -109,6 +113,7 @@ public enum ToolDefinitions {properties: ["app": stringProperty(description: "App name or bundle identifier"),"key": stringProperty(description: "Key or key combination to press"),+ "key_method": keyMethodProperty(),],required: ["app", "key"])@@ -148,6 +153,7 @@ public enum ToolDefinitions {properties: ["app": stringProperty(description: "App name or bundle identifier"),"text": stringProperty(description: "Literal text to type"),+ "key_method": keyMethodProperty(),],required: ["app", "text"])@@ -185,6 +191,13 @@ private func readOnlyAnnotations() -> [String: Any] {]}+private func keyMethodProperty() -> [String: Any] {+ stringProperty(+ description: "Keyboard delivery: auto (default) or sky_key. sky_key posts authenticated keyboard events through the macOS SkyLight background-window path to the current target window without changing foreground focus. Requires a current on-screen target window from get_app_state.",+ enumValues: KeyMethod.allCases.map(\.rawValue)+ )+}+private func stringProperty(description: String, enumValues: [String]? = nil) -> [String: Any] {var property: [String: Any] = ["type": "string",diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/WindowOcclusionKeepAlive.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/WindowOcclusionKeepAlive.swiftnew file mode 100644index 0000000..83df3c9--- /dev/null+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/WindowOcclusionKeepAlive.swift@@ -0,0 +1,135 @@+import AppKit+import CoreGraphics+import Foundation++/// Sample a grid over `bounds`; the window looks unoccluded when at least one+/// sample point is not covered by any window above it. Mirrors Chromium's own+/// manual occlusion check closely enough to decide whether freezing the+/// visible state is safe (freezing while occluded would pin the page hidden).+func windowLooksUnoccluded(bounds: CGRect, coveringBounds: [CGRect], grid: Int = 16) -> Bool {+ guard bounds.width > 0, bounds.height > 0 else {+ return false+ }+ for row in 0..<grid {+ for column in 0..<grid {+ let point = CGPoint(+ x: bounds.minX + bounds.width * (CGFloat(column) + 0.5) / CGFloat(grid),+ y: bounds.minY + bounds.height * (CGFloat(row) + 0.5) / CGFloat(grid)+ )+ if !coveringBounds.contains(where: { $0.contains(point) }) {+ return true+ }+ }+ }+ return false+}++/// Chromium-derived engines (Chrome, Electron, CEF hosts) build their web+/// accessibility tree lazily after the first AX client request. Detect the+/// engine from the frameworks the app ships instead of a bundle-identifier+/// list; WebKit hosts expose their web area immediately and need no wait.+func appHasLazyWebAccessibility(bundleURL: URL?) -> Bool {+ guard let bundleURL else {+ return false+ }+ let frameworks = bundleURL.appendingPathComponent("Contents/Frameworks")+ let names = (try? FileManager.default.contentsOfDirectory(atPath: frameworks.path)) ?? []+ return names.contains { name in+ let lowered = name.lowercased()+ return lowered.contains("chrom") || lowered.contains("electron") || lowered.contains("cef")+ }+}++/// Keeps a driven window "visible" from its own app's point of view while the+/// agent works on it in the background.+///+/// Apps learn that a window is covered only through WindowServer occlusion+/// notifications (`NSWindow.occlusionState`). Web engines react by hiding the+/// page: Chromium, Electron and WebKit drop the web accessibility subtree and+/// pause rendering. Disabling those notifications for one window while it is+/// visible pins the app's notion at "visible", so covering the window later or+/// moving it to another Space no longer hides its content. This is a+/// WindowServer-level, app-agnostic switch, applied to every window the agent+/// drives; its prior state is restored when the server shuts down normally.+/// Nothing here activates, raises, or moves a window, and it cannot un-hide a+/// window that is already occluded when the agent first sees it.+final class WindowOcclusionKeepAlive: @unchecked Sendable {+ static let shared = WindowOcclusionKeepAlive()++ private let lock = NSLock()+ /// Preserve the state reported by the SPI so cleanup does not enable+ /// notifications that another owner had already disabled.+ private var frozenWindowPreviousStates: [CGWindowID: Bool] = [:]+ private var exitHookInstalled = false++ /// Returns `true` when the window's visible state is pinned (now or already).+ @discardableResult+ func keepVisible(+ windowID: CGWindowID,+ bounds: CGRect,+ spi: any WindowOcclusionControlling = SkyLightSPI.shared+ ) -> Bool {+ guard spi.occlusionCapability.isAvailable else {+ return false+ }++ lock.lock()+ defer {+ lock.unlock()+ }++ if frozenWindowPreviousStates[windowID] != nil {+ return true+ }+ guard windowLooksUnoccluded(bounds: bounds, coveringBounds: coveringWindowBounds(above: windowID)) else {+ return false+ }+ do {+ let previousState = try spi.setWindowOcclusionNotificationsEnabled(false, windowID: windowID)+ frozenWindowPreviousStates[windowID] = previousState+ } catch {+ return false+ }+ installExitHookIfNeeded()+ return true+ }++ func isPinned(windowID: CGWindowID) -> Bool {+ lock.lock()+ defer {+ lock.unlock()+ }+ return frozenWindowPreviousStates[windowID] != nil+ }++ func releaseAll(spi: any WindowOcclusionControlling = SkyLightSPI.shared) {+ lock.lock()+ defer { lock.unlock() }+ for (windowID, previousState) in Array(frozenWindowPreviousStates) {+ do {+ try spi.setWindowOcclusionNotificationsEnabled(previousState, windowID: windowID)+ frozenWindowPreviousStates.removeValue(forKey: windowID)+ } catch {+ // Retain failed entries so a later cleanup can retry.+ }+ }+ }++ private func installExitHookIfNeeded() {+ guard !exitHookInstalled else {+ return+ }+ exitHookInstalled = true+ atexit {+ WindowOcclusionKeepAlive.shared.releaseAll()+ }+ }++}++/// Release process-owned background window state at the host's session+/// boundary. The exit hooks remain a final best-effort fallback.+public func resetOpenComputerUseBackgroundWindowState() {+ WindowOcclusionKeepAlive.shared.releaseAll()+ AgentDisplay.shared.restoreAll()+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/AgentDisplayLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/AgentDisplayLiveTests.swiftnew file mode 100644index 0000000..4dc9739--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/AgentDisplayLiveTests.swift@@ -0,0 +1,160 @@+import AppKit+import ApplicationServices+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++/// Opt-in live regression: a Chrome window that is already hidden (fully+/// covered) when the agent first sees it is parked on the agent display via+/// `window_placement=agent_display`; the next snapshot has the web content and+/// a screenshot, clicks and keys land, the user's Space, frontmost app and+/// pointer are untouched, and `restore` puts the window back and removes the+/// display.+@MainActor+final class AgentDisplayLiveTests: XCTestCase {+ func testHiddenChromeIsParkedOnTheAgentDisplayAndRestored() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_AGENT_DISPLAY_LIVE_TEST"] == "1" else {+ throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_AGENT_DISPLAY_LIVE_TEST=1 to run the agent display live test")+ }+ setvbuf(stdout, nil, _IONBF, 0)+ guard AgentDisplay.shared.isSupported else {+ throw XCTSkip("CGVirtualDisplay is unavailable on this macOS")+ }+ let chromeURL = URL(fileURLWithPath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")+ guard FileManager.default.isExecutableFile(atPath: chromeURL.path) else {+ throw XCTSkip("Google Chrome is not installed at the standard path")+ }+ let originalFrontApp = NSWorkspace.shared.frontmostApplication+ defer {+ if let originalFrontApp { _ = originalFrontApp.activate(options: [.activateAllWindows]) }+ }+ let testRoot = Self.packageRoot.appendingPathComponent(".build/ocu-agent-display-\(UUID().uuidString)", isDirectory: true)+ let profileURL = testRoot.appendingPathComponent("chrome-profile", isDirectory: true)+ let pageURL = testRoot.appendingPathComponent("index.html")+ try FileManager.default.createDirectory(at: profileURL, withIntermediateDirectories: true)+ try Self.liveTestHTML.write(to: pageURL, atomically: false, encoding: .utf8)+ defer { try? FileManager.default.removeItem(at: testRoot) }+ let chrome = Process()+ chrome.executableURL = chromeURL+ chrome.arguments = [+ "--user-data-dir=\(profileURL.path)", "--no-first-run", "--no-default-browser-check",+ "--disable-background-networking", "--disable-component-update",+ "--window-position=200,200", "--window-size=600,420", "--app=\(pageURL.absoluteString)",+ ]+ chrome.standardOutput = FileHandle.nullDevice+ chrome.standardError = FileHandle.nullDevice+ try chrome.run()+ defer { stop(chrome) }+ let window = try waitForWindow(pid: chrome.processIdentifier, nameContaining: "ocu-agent-", onScreenOnly: true)+ guard let runningChrome = NSRunningApplication(processIdentifier: window.pid) else { return XCTFail("Chrome not running") }+ let descriptor = RunningAppDescriptor(name: runningChrome.localizedName ?? "Google Chrome", bundleIdentifier: runningChrome.bundleIdentifier, pid: window.pid, runningApplication: runningChrome)++ // Cover Chrome before the agent ever looks at it: worst case, the page is hidden.+ let cover = try launch(executable: Self.packageRoot.appendingPathComponent(".build/debug/OpenComputerUseFixture"))+ defer { stop(cover) }+ _ = try waitForWindow(pid: cover.processIdentifier, nameContaining: "OpenComputerUseFixture", onScreenOnly: true)+ RunLoop.current.run(until: Date().addingTimeInterval(2.5))+ let hiddenTitle = try waitForWindow(pid: window.pid, nameContaining: "ocu-agent-", onScreenOnly: false).name+ XCTAssertTrue(hiddenTitle.hasSuffix("hidden"), "Chrome must have hidden the covered page, got \(hiddenTitle)")+ let hiddenSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ XCTAssertFalse(hiddenSnapshot.treeLines.contains { $0.contains("sky probe paragraph") }, "hidden page must not expose web content")+ XCTAssertTrue(hiddenSnapshot.treeLines.contains { $0.hasPrefix("Note: this window is covered") })++ let frontBefore = NSWorkspace.shared.frontmostApplication?.processIdentifier+ let mouseBefore = NSEvent.mouseLocation+ var displayCountBefore: UInt32 = 0+ CGGetOnlineDisplayList(0, nil, &displayCountBefore)++ // Park on the agent display through the same call the tool uses.+ let windowElement = try XCTUnwrap(hiddenSnapshot.windowElement)+ let displayBounds = try AgentDisplay.shared.park(windowID: window.id, pid: window.pid, window: windowElement)+ defer { AgentDisplay.shared.restoreAll() }+ let parked = try waitForWindow(pid: window.pid, nameContaining: "ocu-agent-", onScreenOnly: false)+ print("agent display live test: display=\(displayBounds) parked=\(parked.bounds) title=\(parked.name)")+ XCTAssertTrue(displayBounds.contains(CGPoint(x: parked.bounds.minX + 1, y: parked.bounds.minY + 1)), "the window must sit on the agent display")+ let visibleTitle = try waitForWindow(pid: window.pid, nameContaining: "visible", onScreenOnly: false).name+ XCTAssertTrue(visibleTitle.hasSuffix("visible"), "Chrome must consider the parked page visible")++ let parkedSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ XCTAssertEqual(parkedSnapshot.targetWindowID, window.id)+ XCTAssertTrue(parkedSnapshot.treeLines.contains { $0.contains("sky probe paragraph") }, "parked snapshot must include web content")+ XCTAssertNotNil(parkedSnapshot.screenshotPNGData)+ XCTAssertEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, frontBefore, "parking must not change the frontmost app")+ XCTAssertEqual(NSEvent.mouseLocation, mouseBefore, "parking must not move the pointer")++ let spi = SkyLightSPI.shared+ let windowPoint = CGPoint(x: parked.bounds.width / 2, y: parked.bounds.height - 60)+ try SkyClickDispatcher.click(+ target: SkyClickTarget(+ screenPoint: CGPoint(x: parked.bounds.minX + windowPoint.x, y: parked.bounds.minY + windowPoint.y),+ windowPoint: windowPoint, windowBounds: parked.bounds, windowID: window.id, pid: window.pid),+ clickCount: 1, spi: spi)+ try SkyKeyboardDispatcher.typeText(target: SkyKeyboardTarget(windowID: window.id, pid: window.pid), text: "ok", spi: spi)+ let acted = try waitForWindow(pid: window.pid, nameContaining: "ocu-agent-ok|clicks=1|", onScreenOnly: false)+ XCTAssertTrue(acted.name.hasPrefix("ocu-agent-ok|clicks=1|"), "click and keys must reach the parked window")++ // Restore: window back where it was, display gone.+ try AgentDisplay.shared.restoreAll(pid: window.pid)+ let restored = try waitForWindow(pid: window.pid, nameContaining: "ocu-agent-", onScreenOnly: false)+ XCTAssertEqual(restored.bounds.origin, window.bounds.origin, "restore must put the window back")+ RunLoop.current.run(until: Date().addingTimeInterval(2.0))+ var displayCountAfter: UInt32 = 0+ CGGetOnlineDisplayList(0, nil, &displayCountAfter)+ XCTAssertEqual(displayCountAfter, displayCountBefore, "the agent display must be removed after restore")+ XCTAssertEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, frontBefore)+ }++ private struct WindowRecord { let id: CGWindowID; let pid: pid_t; let bounds: CGRect; let name: String }++ private func waitForWindow(pid: pid_t, nameContaining marker: String, onScreenOnly: Bool) throws -> WindowRecord {+ let deadline = Date().addingTimeInterval(10)+ while Date() < deadline {+ let options: CGWindowListOption = onScreenOnly ? [.optionOnScreenOnly, .excludeDesktopElements] : [.optionAll]+ let raw = CGWindowListCopyWindowInfo(options, kCGNullWindowID) as? [[String: Any]] ?? []+ for info in raw {+ guard let number = info[kCGWindowNumber as String] as? NSNumber,+ let owner = info[kCGWindowOwnerPID as String] as? NSNumber, owner.int32Value == pid,+ let name = info[kCGWindowName as String] as? String, name.contains(marker),+ let bounds = CGRect(dictionaryRepresentation: (info[kCGWindowBounds as String] as? NSDictionary) ?? [:]), bounds.width > 0+ else { continue }+ return WindowRecord(id: number.uint32Value, pid: pid, bounds: bounds, name: name)+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.1))+ }+ throw ComputerUseError.message("Timed out waiting for window title containing \(marker)")+ }++ private func launch(executable: URL) throws -> Process {+ let process = Process(); process.executableURL = executable+ process.standardOutput = FileHandle.nullDevice; process.standardError = FileHandle.nullDevice+ try process.run(); return process+ }++ private func stop(_ process: Process) {+ if process.isRunning {+ process.terminate()+ let deadline = Date().addingTimeInterval(5)+ while process.isRunning, Date() < deadline { RunLoop.current.run(until: Date().addingTimeInterval(0.05)) }+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.25))+ }++ private static let liveTestHTML = #"""+ <!doctype html><meta charset="utf-8"><title>ocu-agent-ready</title>+ <style>html,body{margin:0;height:100%} input{width:90%;font:22px system-ui;margin:20px} button{position:absolute;left:0;right:0;bottom:0;height:120px;font:22px system-ui;background:#5b8def;color:#fff;border:0}</style>+ <p>sky probe paragraph</p><input id="t" autofocus placeholder="sky probe"><button id="b">click probe</button>+ <script>+ let clicks = 0; const input = document.querySelector('#t');+ const update = () => { document.title = `ocu-agent-${input.value}|clicks=${clicks}|${document.visibilityState}`; };+ document.querySelector('#b').addEventListener('click', () => { clicks += 1; update(); });+ input.addEventListener('input', update); document.addEventListener('visibilitychange', update); update();+ </script>+ """#++ private static let packageRoot: URL = {+ var url = URL(fileURLWithPath: #filePath)+ for _ in 0..<5 { url.deleteLastPathComponent() }+ return url+ }()+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/AppMatrixLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/AppMatrixLiveTests.swiftnew file mode 100644index 0000000..179aa14--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/AppMatrixLiveTests.swift@@ -0,0 +1,173 @@+import AppKit+import ApplicationServices+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++/// Opt-in survey across whatever GUI apps are running: snapshot each app+/// without activating it, then, when it exposes an EMPTY text or search field,+/// sky_click that field, sky_key a marker into it, verify the marker via a+/// fresh snapshot, and delete exactly the marker again. Prints one row per app+/// and never asserts per app; the only assertion is that the frontmost app is+/// unchanged at the end. Set OPEN_COMPUTER_USE_APP_MATRIX_SKIP to a comma list+/// of app names to exclude. With OPEN_COMPUTER_USE_APP_MATRIX_PARK=1 each+/// non-fullscreen window is additionally parked on the agent display, surveyed+/// and driven there, then restored to its original position.+@MainActor+final class AppMatrixLiveTests: XCTestCase {+ func testSurveyRunningApps() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_APP_MATRIX"] == "1" else { throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_APP_MATRIX=1") }+ setvbuf(stdout, nil, _IONBF, 0)+ let spi = SkyLightSPI.shared+ let skip = Set((ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_APP_MATRIX_SKIP"] ?? "").split(separator: ",").map { $0.trimmingCharacters(in: .whitespaces).lowercased() })+ let ownPID = ProcessInfo.processInfo.processIdentifier+ let frontBefore = NSWorkspace.shared.frontmostApplication+ let marker = "zq7x"+ let park = ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_APP_MATRIX_PARK"] == "1"+ let mouseBefore = NSEvent.mouseLocation+ var rows: [String] = []+ func inputRound(_ snapshot: AppSnapshot, _ descriptor: RunningAppDescriptor, _ pid: pid_t) -> String {+ guard let windowID = snapshot.targetWindowID, let bounds = snapshot.windowBounds else { return "no window id" }+ let candidates = snapshot.elements.values.filter { record in+ guard let role = record.role, let element = record.element else { return false }+ guard role == kAXTextFieldRole as String || role == "AXSearchField" || role == kAXTextAreaRole as String || role == kAXComboBoxRole as String else { return false }+ var value: CFTypeRef?+ AXUIElementCopyAttributeValue(element, kAXValueAttribute as CFString, &value)+ guard ((value as? String) ?? "").isEmpty, let frame = record.localFrame, frame.width >= 40, frame.height >= 12 else { return false }+ return CGRect(origin: .zero, size: bounds.size).contains(CGPoint(x: frame.midX, y: frame.midY))+ }.sorted { ($0.role == "AXSearchField" ? 0 : 1, $0.index) < ($1.role == "AXSearchField" ? 0 : 1, $1.index) }+ guard let field = candidates.first, let frame = field.localFrame else { return "no empty text field" }+ let windowPoint = CGPoint(x: frame.midX, y: frame.midY)+ let target = SkyKeyboardTarget(windowID: windowID, pid: pid)+ do {+ let c0 = TimingLog.now()+ try SkyClickDispatcher.click(target: SkyClickTarget(screenPoint: CGPoint(x: bounds.minX + windowPoint.x, y: bounds.minY + windowPoint.y), windowPoint: windowPoint, windowBounds: bounds, windowID: windowID, pid: pid), clickCount: 1, spi: spi)+ try SkyKeyboardDispatcher.typeText(target: target, text: marker, spi: spi)+ let ms = Int((TimingLog.now() - c0) * 1000)+ Thread.sleep(forTimeInterval: 0.3)+ let inTree = (try? SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly))?.treeLines.contains { $0.contains(marker) } ?? false+ for _ in 0..<marker.count { try SkyKeyboardDispatcher.pressKey(target: target, key: "backspace", spi: spi) }+ Thread.sleep(forTimeInterval: 0.2)+ let cleared = !((try? SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly))?.treeLines.contains { $0.contains(marker) } ?? true)+ return "field=\(field.role ?? "?")#\(field.index) click+type \(ms)ms typed=\(inTree) cleared=\(cleared)"+ } catch {+ return "input FAILED: \(short(error))"+ }+ }+ func isFullScreen(_ window: AXUIElement?) -> Bool {+ guard let window else { return false }+ var value: CFTypeRef?+ AXUIElementCopyAttributeValue(window, "AXFullScreen" as CFString, &value)+ return (value as? Bool) ?? false+ }+ let apps = NSWorkspace.shared.runningApplications.filter { app in+ app.activationPolicy == .regular && app.processIdentifier != ownPID && app.processIdentifier != frontBefore?.processIdentifier+ && !skip.contains((app.localizedName ?? "").lowercased())+ }+ for app in apps {+ let name = app.localizedName ?? "pid \(app.processIdentifier)"+ let engine = appHasLazyWebAccessibility(bundleURL: app.bundleURL) ? "chromium" : (appLooksWebKit(app) ? "webkit" : "native")+ let descriptor = RunningAppDescriptor(name: name, bundleIdentifier: app.bundleIdentifier, pid: app.processIdentifier, runningApplication: app)+ let t0 = TimingLog.now()+ let snapshot: AppSnapshot+ do {+ snapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ } catch {+ rows.append(row(name, engine, "snapshot FAILED: \(short(error))"))+ continue+ }+ let snapMs = Int((TimingLog.now() - t0) * 1000)+ let nodes = snapshot.elements.count+ let hasWeb = snapshot.treeLines.contains { $0.contains("HTML content") }+ let covered = snapshot.treeLines.contains { $0.hasPrefix("Note: this window is covered") }+ let shot = snapshot.screenshotPNGData?.count ?? 0+ guard let windowID = snapshot.targetWindowID, let bounds = snapshot.windowBounds else {+ rows.append(row(name, engine, "snapshot ok nodes=\(nodes) web=\(hasWeb) shot=\(shot)B \(snapMs)ms; no window id"))+ continue+ }+ var detail = "snapshot \(snapMs)ms nodes=\(nodes) web=\(hasWeb) covered=\(covered) shot=\(shot)B"+ if park {+ if isFullScreen(snapshot.windowElement) {+ rows.append(row(name, engine, detail + "; PARK skipped (fullscreen window)"))+ continue+ }+ guard let windowElement = snapshot.windowElement else {+ rows.append(row(name, engine, detail + "; PARK skipped (no window element)")); continue+ }+ do {+ let p0 = TimingLog.now()+ _ = try AgentDisplay.shared.park(windowID: windowID, pid: app.processIdentifier, window: windowElement)+ let parkMs = Int((TimingLog.now() - p0) * 1000)+ let s0 = TimingLog.now()+ let parkedSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ let parkedMs = Int((TimingLog.now() - s0) * 1000)+ let parkedWeb = parkedSnapshot.treeLines.contains { $0.contains("HTML content") }+ let onDisplay = AgentDisplay.shared.displayBounds.map { $0.contains(CGPoint(x: (parkedSnapshot.windowBounds?.minX ?? -1) + 1, y: (parkedSnapshot.windowBounds?.minY ?? -1) + 1)) } ?? false+ var parkDetail = "PARK \(parkMs)ms onDisplay=\(onDisplay) snapshot \(parkedMs)ms nodes=\(parkedSnapshot.elements.count) web=\(parkedWeb) shot=\(parkedSnapshot.screenshotPNGData?.count ?? 0)B; " + inputRound(parkedSnapshot, descriptor, app.processIdentifier)+ let r0 = TimingLog.now()+ try AgentDisplay.shared.restore(windowID: windowID)+ let restoredBounds = (try? SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly))?.windowBounds+ parkDetail += "; RESTORE \(Int((TimingLog.now() - r0) * 1000))ms back=\(restoredBounds.map { abs($0.minX - bounds.minX) < 2 && abs($0.minY - bounds.minY) < 2 } ?? false)"+ rows.append(row(name, engine, detail + "; " + parkDetail))+ } catch {+ try? AgentDisplay.shared.restore(windowID: windowID)+ rows.append(row(name, engine, detail + "; PARK FAILED: \(short(error))"))+ }+ continue+ }+ // Find an empty text/search field.+ let candidates = snapshot.elements.values.filter { record in+ guard let role = record.role, let element = record.element else { return false }+ guard role == kAXTextFieldRole as String || role == "AXSearchField" || role == kAXTextAreaRole as String || role == kAXComboBoxRole as String else { return false }+ var value: CFTypeRef?+ AXUIElementCopyAttributeValue(element, kAXValueAttribute as CFString, &value)+ let text = (value as? String) ?? ""+ guard text.isEmpty, let frame = record.localFrame, frame.width >= 40, frame.height >= 12 else { return false }+ // Popover children carry frames outside the window; sky_click rejects those by design.+ return CGRect(origin: .zero, size: bounds.size).contains(CGPoint(x: frame.midX, y: frame.midY))+ }.sorted { ($0.role == "AXSearchField" ? 0 : 1, $0.index) < ($1.role == "AXSearchField" ? 0 : 1, $1.index) }+ guard let field = candidates.first, let frame = field.localFrame else {+ rows.append(row(name, engine, detail + "; no empty text field -> no input test"))+ continue+ }+ let windowPoint = CGPoint(x: frame.midX, y: frame.midY)+ let screenPoint = CGPoint(x: bounds.minX + windowPoint.x, y: bounds.minY + windowPoint.y)+ let target = SkyKeyboardTarget(windowID: windowID, pid: app.processIdentifier)+ do {+ let c0 = TimingLog.now()+ try SkyClickDispatcher.click(target: SkyClickTarget(screenPoint: screenPoint, windowPoint: windowPoint, windowBounds: bounds, windowID: windowID, pid: app.processIdentifier), clickCount: 1, spi: spi)+ let k0 = TimingLog.now()+ try SkyKeyboardDispatcher.typeText(target: target, text: marker, spi: spi)+ let kMs = Int((TimingLog.now() - k0) * 1000)+ Thread.sleep(forTimeInterval: 0.3)+ var value: CFTypeRef?+ AXUIElementCopyAttributeValue(field.element!, kAXValueAttribute as CFString, &value)+ let inField = ((value as? String) ?? "").contains(marker)+ // Some search bars open an overlay: look for the marker anywhere in a fresh tree.+ let afterSnapshot = try? SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ let inTree = afterSnapshot?.treeLines.contains { $0.contains(marker) } ?? false+ for _ in 0..<marker.count { try SkyKeyboardDispatcher.pressKey(target: target, key: "backspace", spi: spi) }+ Thread.sleep(forTimeInterval: 0.2)+ let clearedSnapshot = try? SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ let cleared = !(clearedSnapshot?.treeLines.contains { $0.contains(marker) } ?? true)+ detail += "; field=\(field.role ?? "?")#\(field.index) click+type \(Int((TimingLog.now() - c0) * 1000))ms (type \(kMs)ms) typedInField=\(inField) typedInTree=\(inTree) cleared=\(cleared)"+ } catch {+ detail += "; input FAILED: \(short(error))"+ }+ rows.append(row(name, engine, detail))+ }+ print("MATRIX frontmost=\(frontBefore?.localizedName ?? "?") apps=\(apps.count) park=\(park)")+ rows.forEach { print($0) }+ XCTAssertEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, frontBefore?.processIdentifier, "the survey must not change the frontmost app")+ XCTAssertEqual(NSEvent.mouseLocation, mouseBefore, "the survey must not move the pointer")+ AgentDisplay.shared.restoreAll()+ WindowOcclusionKeepAlive.shared.releaseAll()+ }++ private func row(_ name: String, _ engine: String, _ detail: String) -> String { "MATRIX \(name) [\(engine)] \(detail)" }+ private func short(_ error: Error) -> String { String(((error as? ComputerUseError)?.errorDescription ?? "\(error)").prefix(90)) }+ private func appLooksWebKit(_ app: NSRunningApplication) -> Bool {+ (app.bundleIdentifier ?? "").lowercased().contains("safari")+ }+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/BackgroundInputBenchmarkLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/BackgroundInputBenchmarkLiveTests.swiftnew file mode 100644index 0000000..71bf2e6--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/BackgroundInputBenchmarkLiveTests.swift@@ -0,0 +1,134 @@+import AppKit+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++/// Opt-in benchmark: N cycles of sky_click and sky_key against a fully covered+/// isolated Chrome window. Reports success counts and latency percentiles+/// (dispatch = time until the call returns; observed = time until the page+/// reflected the action). `OPEN_COMPUTER_USE_BENCH_CYCLES` sets N (default 20).+@MainActor+final class BackgroundInputBenchmarkLiveTests: XCTestCase {+ func testCoveredChromeClickAndKeyLatency() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_BACKGROUND_BENCH"] == "1" else {+ throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_BACKGROUND_BENCH=1 to run the background input benchmark")+ }+ setvbuf(stdout, nil, _IONBF, 0)+ let cycles = Int(ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_BENCH_CYCLES"] ?? "") ?? 20+ let spi = SkyLightSPI.shared+ guard spi.capability.isAvailable else { throw XCTSkip("SkyLight SPI unavailable") }+ let chromeURL = URL(fileURLWithPath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")+ guard FileManager.default.isExecutableFile(atPath: chromeURL.path) else { throw XCTSkip("Chrome missing") }+ let originalFrontApp = NSWorkspace.shared.frontmostApplication+ defer { if let originalFrontApp { _ = originalFrontApp.activate(options: [.activateAllWindows]) } }++ let testRoot = Self.packageRoot.appendingPathComponent(".build/ocu-bench-\(UUID().uuidString)", isDirectory: true)+ let profileURL = testRoot.appendingPathComponent("chrome-profile", isDirectory: true)+ let pageURL = testRoot.appendingPathComponent("index.html")+ try FileManager.default.createDirectory(at: profileURL, withIntermediateDirectories: true)+ try Self.html.write(to: pageURL, atomically: false, encoding: .utf8)+ defer { try? FileManager.default.removeItem(at: testRoot) }+ let chrome = Process()+ chrome.executableURL = chromeURL+ chrome.arguments = ["--user-data-dir=\(profileURL.path)", "--no-first-run", "--no-default-browser-check", "--disable-background-networking", "--disable-component-update", "--window-position=200,200", "--window-size=600,420", "--app=\(pageURL.absoluteString)"]+ chrome.standardOutput = FileHandle.nullDevice; chrome.standardError = FileHandle.nullDevice+ try chrome.run()+ defer { stop(chrome) }+ let window = try waitForWindow(pid: chrome.processIdentifier, nameContaining: "ocu-bench-")+ // Pin visibility while visible (what get_app_state does), then cover it.+ let pinned = ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_BENCH_UNPINNED"] != "1"+ if pinned { WindowOcclusionKeepAlive.shared.keepVisible(windowID: window.id, bounds: window.bounds, spi: spi) }+ defer { WindowOcclusionKeepAlive.shared.releaseAll() }+ let cover = try launch(executable: Self.packageRoot.appendingPathComponent(".build/debug/OpenComputerUseFixture"))+ defer { stop(cover) }+ _ = try waitForWindow(pid: cover.processIdentifier, nameContaining: "OpenComputerUseFixture")+ RunLoop.current.run(until: Date().addingTimeInterval(1.0))+ XCTAssertNotEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, window.pid)++ func title() -> String {+ let raw = CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]] ?? []+ for info in raw {+ if let n = info[kCGWindowNumber as String] as? NSNumber, n.uint32Value == window.id { return info[kCGWindowName as String] as? String ?? "" }+ }+ return ""+ }+ func observe(_ marker: String, timeout: TimeInterval = 3) -> TimeInterval? {+ let start = ProcessInfo.processInfo.systemUptime+ while ProcessInfo.processInfo.systemUptime - start < timeout {+ if title().contains(marker) { return ProcessInfo.processInfo.systemUptime - start }+ Thread.sleep(forTimeInterval: 0.003)+ }+ return nil+ }+ let windowPoint = CGPoint(x: window.bounds.width / 2, y: window.bounds.height - 60)+ let clickTarget = SkyClickTarget(screenPoint: CGPoint(x: window.bounds.minX + windowPoint.x, y: window.bounds.minY + windowPoint.y), windowPoint: windowPoint, windowBounds: window.bounds, windowID: window.id, pid: window.pid)+ let keyTarget = SkyKeyboardTarget(windowID: window.id, pid: window.pid)++ var clickDispatch: [Double] = [], clickObserved: [Double] = [], keyDispatch: [Double] = [], keyObserved: [Double] = []+ var clickOK = 0, keyOK = 0+ let frontBefore = NSWorkspace.shared.frontmostApplication?.processIdentifier+ for cycle in 1...cycles {+ let c0 = ProcessInfo.processInfo.systemUptime+ try SkyClickDispatcher.click(target: clickTarget, clickCount: 1, spi: spi)+ let cDispatch = ProcessInfo.processInfo.systemUptime - c0+ if let seen = observe("clicks=\(cycle)|") { clickOK += 1; clickObserved.append((cDispatch + seen) * 1000) }+ clickDispatch.append(cDispatch * 1000)++ let k0 = ProcessInfo.processInfo.systemUptime+ try SkyKeyboardDispatcher.typeText(target: keyTarget, text: "k\(cycle).", spi: spi)+ let kDispatch = ProcessInfo.processInfo.systemUptime - k0+ if let seen = observe("k\(cycle).|") { keyOK += 1; keyObserved.append((kDispatch + seen) * 1000) }+ keyDispatch.append(kDispatch * 1000)+ }+ let frontAfter = NSWorkspace.shared.frontmostApplication?.processIdentifier+ func pct(_ values: [Double], _ p: Double) -> Double {+ guard !values.isEmpty else { return .nan }+ let sorted = values.sorted(); let index = min(sorted.count - 1, Int((Double(sorted.count - 1) * p).rounded()))+ return sorted[index]+ }+ func row(_ name: String, _ values: [Double]) -> String {+ String(format: "BENCH %-22@ n=%2d p50=%7.1fms p95=%7.1fms max=%7.1fms", name, values.count, pct(values, 0.5), pct(values, 0.95), values.max() ?? .nan)+ }+ print("BENCH cycles=\(cycles) pinned=\(pinned) keySettle=\(Int(SkyKeyboardDispatcher.keyWindowFallbackSettle * 1000))ms release=\(Int(SkyKeyboardDispatcher.releaseSettle * 1000))ms target=covered isolated Chrome (macOS \(ProcessInfo.processInfo.operatingSystemVersionString))")+ print("BENCH sky_click success \(clickOK)/\(cycles) sky_key success \(keyOK)/\(cycles) frontmost unchanged=\(frontBefore == frontAfter)")+ print(row("sky_click dispatch", clickDispatch)); print(row("sky_click observed", clickObserved))+ print(row("sky_key dispatch", keyDispatch)); print(row("sky_key observed", keyObserved))+ XCTAssertEqual(clickOK, cycles, "every background click must land")+ XCTAssertEqual(keyOK, cycles, "every background key must land")+ XCTAssertEqual(frontBefore, frontAfter)+ }++ private struct WindowRecord { let id: CGWindowID; let pid: pid_t; let bounds: CGRect; let name: String }+ private func waitForWindow(pid: pid_t, nameContaining marker: String) throws -> WindowRecord {+ let deadline = Date().addingTimeInterval(10)+ while Date() < deadline {+ let raw = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]] ?? []+ for info in raw {+ guard let n = info[kCGWindowNumber as String] as? NSNumber, let owner = info[kCGWindowOwnerPID as String] as? NSNumber, owner.int32Value == pid,+ let name = info[kCGWindowName as String] as? String, name.contains(marker),+ let b = CGRect(dictionaryRepresentation: (info[kCGWindowBounds as String] as? NSDictionary) ?? [:]), b.width > 0 else { continue }+ return WindowRecord(id: n.uint32Value, pid: pid, bounds: b, name: name)+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.1))+ }+ throw ComputerUseError.message("timeout \(marker)")+ }+ private func launch(executable: URL) throws -> Process { let p = Process(); p.executableURL = executable; p.standardOutput = FileHandle.nullDevice; p.standardError = FileHandle.nullDevice; try p.run(); return p }+ private func stop(_ process: Process) {+ if process.isRunning { process.terminate(); let d = Date().addingTimeInterval(5); while process.isRunning, Date() < d { RunLoop.current.run(until: Date().addingTimeInterval(0.05)) } }+ RunLoop.current.run(until: Date().addingTimeInterval(0.25))+ }+ private static let html = #"""+ <!doctype html><meta charset="utf-8"><title>ocu-bench-ready</title>+ <style>html,body{margin:0;height:100%} input{width:90%;font:22px system-ui;margin:20px} button{position:absolute;left:0;right:0;bottom:0;height:120px;font:22px system-ui;background:#5b8def;color:#fff;border:0}</style>+ <input id="t" autofocus><button id="b">probe</button>+ <script>+ let clicks = 0; const input = document.querySelector('#t');+ const update = () => { document.title = `ocu-bench-${input.value.slice(-12)}|clicks=${clicks}|`; };+ document.querySelector('#b').addEventListener('click', () => { clicks += 1; update(); });+ input.addEventListener('input', update); update();+ </script>+ """#+ private static let packageRoot: URL = { var url = URL(fileURLWithPath: #filePath); for _ in 0..<5 { url.deleteLastPathComponent() }; return url }()+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CrossSpaceLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CrossSpaceLiveTests.swiftnew file mode 100644index 0000000..646799f--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CrossSpaceLiveTests.swift@@ -0,0 +1,204 @@+import AppKit+import ApplicationServices+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++/// Opt-in live regression: a Chrome window parked on another Space (a+/// temporary WindowServer Space when the machine only has one) can still be+/// snapshotted, clicked and typed into from the active Space without changing+/// the active Space or the frontmost app.+@MainActor+final class CrossSpaceLiveTests: XCTestCase {+ private typealias MainCID = @convention(c) () -> UInt32+ private typealias CopySpaces = @convention(c) (UInt32) -> Unmanaged<CFArray>?+ private typealias ActiveSpace = @convention(c) (UInt32) -> UInt64+ private typealias CopySpacesForWindows = @convention(c) (UInt32, Int32, CFArray) -> Unmanaged<CFArray>?+ private typealias SetCurrentSpace = @convention(c) (UInt32, CFString, UInt64) -> Void++ func testWindowOnAnotherSpaceIsReadableClickableAndTypable() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_CROSS_SPACE_LIVE_TEST"] == "1" else {+ throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_CROSS_SPACE_LIVE_TEST=1 to run the cross-Space live test")+ }+ setvbuf(stdout, nil, _IONBF, 0)+ let spi = SkyLightSPI.shared+ guard spi.capability.isAvailable, spi.occlusionCapability.isAvailable else {+ throw XCTSkip("SkyLight SPI unavailable")+ }+ guard let sky = dlopen("/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight", RTLD_LAZY),+ let cidPtr = dlsym(sky, "SLSMainConnectionID"), let copyPtr = dlsym(sky, "SLSCopyManagedDisplaySpaces"),+ let activePtr = dlsym(sky, "SLSGetActiveSpace"), let setCurrentPtr = dlsym(sky, "SLSManagedDisplaySetCurrentSpace"),+ let spacesForPtr = dlsym(sky, "SLSCopySpacesForWindows")+ else {+ throw XCTSkip("Space SPIs unavailable")+ }+ let cid = unsafeBitCast(cidPtr, to: MainCID.self)()+ let copySpaces = unsafeBitCast(copyPtr, to: CopySpaces.self)+ let activeSpace = unsafeBitCast(activePtr, to: ActiveSpace.self)+ let setCurrentSpace = unsafeBitCast(setCurrentPtr, to: SetCurrentSpace.self)+ let copySpacesForWindows = unsafeBitCast(spacesForPtr, to: CopySpacesForWindows.self)+ var displayIdentifier = ""+ if let list = copySpaces(cid)?.takeRetainedValue() as? [[String: Any]], let first = list.first {+ displayIdentifier = first["Display Identifier"] as? String ?? ""+ }+ func spacesOfWindow(_ id: CGWindowID) -> [UInt64] {+ ((copySpacesForWindows(cid, 0x7, [NSNumber(value: id)] as CFArray)?.takeRetainedValue() as? [NSNumber]) ?? []).map(\.uint64Value)+ }+ func userSpaces() -> [UInt64] {+ var ids: [UInt64] = []+ if let displays = copySpaces(cid)?.takeRetainedValue() as? [[String: Any]] {+ for display in displays {+ for space in (display["Spaces"] as? [[String: Any]]) ?? [] where (space["type"] as? NSNumber)?.intValue == 0 {+ if let id = (space["id64"] as? NSNumber)?.uint64Value { ids.append(id) }+ }+ }+ }+ return ids+ }++ let chromeURL = URL(fileURLWithPath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")+ guard FileManager.default.isExecutableFile(atPath: chromeURL.path) else {+ throw XCTSkip("Google Chrome is not installed at the standard path")+ }+ // Third-party processes cannot move other apps' windows between Spaces on+ // macOS 26+ (SLSMoveWindowsToManagedSpace is gated on the window-management+ // bridge), so launch the target while the other Desktop is active instead.+ let homeSpace = activeSpace(cid)+ guard let otherSpace = userSpaces().first(where: { $0 != homeSpace }) else {+ throw XCTSkip("Create a second Desktop in Mission Control to run the cross-Space live test")+ }+ setCurrentSpace(cid, displayIdentifier as CFString, otherSpace)+ RunLoop.current.run(until: Date().addingTimeInterval(1.5))+ XCTAssertEqual(activeSpace(cid), otherSpace, "test setup must switch to the second Desktop")+ let originalFrontApp = NSWorkspace.shared.frontmostApplication+ defer {+ if let originalFrontApp { _ = originalFrontApp.activate(options: [.activateAllWindows]) }+ }+ let testRoot = Self.packageRoot.appendingPathComponent(".build/ocu-cross-space-\(UUID().uuidString)", isDirectory: true)+ let profileURL = testRoot.appendingPathComponent("chrome-profile", isDirectory: true)+ let pageURL = testRoot.appendingPathComponent("index.html")+ try FileManager.default.createDirectory(at: profileURL, withIntermediateDirectories: true)+ try Self.liveTestHTML.write(to: pageURL, atomically: false, encoding: .utf8)+ defer { try? FileManager.default.removeItem(at: testRoot) }+ let chrome = Process()+ chrome.executableURL = chromeURL+ chrome.arguments = [+ "--user-data-dir=\(profileURL.path)", "--no-first-run", "--no-default-browser-check",+ "--disable-background-networking", "--disable-component-update",+ "--window-position=200,200", "--window-size=600,420", "--app=\(pageURL.absoluteString)",+ ]+ chrome.standardOutput = FileHandle.nullDevice+ chrome.standardError = FileHandle.nullDevice+ try chrome.run()+ defer { stop(chrome) }+ let window = try waitForWindow(pid: chrome.processIdentifier, nameContaining: "ocu-space-", onScreenOnly: true)+ guard let runningChrome = NSRunningApplication(processIdentifier: window.pid) else { return XCTFail("Chrome not running") }+ let descriptor = RunningAppDescriptor(name: runningChrome.localizedName ?? "Google Chrome", bundleIdentifier: runningChrome.bundleIdentifier, pid: window.pid, runningApplication: runningChrome)+ _ = runningChrome.activate(options: [.activateAllWindows])+ RunLoop.current.run(until: Date().addingTimeInterval(1.5))+ XCTAssertEqual(spacesOfWindow(window.id), [otherSpace], "Chrome must have opened on the second Desktop")++ // Phase 1: visible snapshot (on the second Desktop) pins the window.+ let visibleSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ XCTAssertTrue(visibleSnapshot.treeLines.contains { $0.contains("sky probe") }, "visible snapshot must include web content")+ XCTAssertTrue(WindowOcclusionKeepAlive.shared.isPinned(windowID: window.id))++ // Back to the user's Desktop; bring the fixture to the front there.+ setCurrentSpace(cid, displayIdentifier as CFString, homeSpace)+ RunLoop.current.run(until: Date().addingTimeInterval(1.5))+ XCTAssertEqual(activeSpace(cid), homeSpace)+ let fixture = try launch(executable: Self.packageRoot.appendingPathComponent(".build/debug/OpenComputerUseFixture"))+ defer { stop(fixture) }+ _ = try waitForWindow(pid: fixture.processIdentifier, nameContaining: "OpenComputerUseFixture", onScreenOnly: true)+ RunLoop.current.run(until: Date().addingTimeInterval(1.0))+ let parked = try waitForWindow(pid: window.pid, nameContaining: "ocu-space-", onScreenOnly: false)+ print("cross-space live test: parked window spaces=\(spacesOfWindow(window.id)) onScreen=\(parked.onScreen) title=\(parked.name) activeSpace=\(activeSpace(cid)) home=\(homeSpace) other=\(otherSpace)")+ XCTAssertEqual(spacesOfWindow(window.id), [otherSpace], "the window must live on the other Space")+ XCTAssertEqual(activeSpace(cid), homeSpace, "parking the window must not switch Spaces")+ let frontBefore = NSWorkspace.shared.frontmostApplication?.processIdentifier+ XCTAssertNotEqual(frontBefore, window.pid)++ // Phase 2: production paths against the parked window.+ let parkedSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)+ XCTAssertEqual(parkedSnapshot.targetWindowID, window.id, "snapshot must resolve the off-screen window")+ XCTAssertTrue(parkedSnapshot.treeLines.contains { $0.contains("sky probe") }, "parked snapshot must include web content")+ XCTAssertNotNil(parkedSnapshot.screenshotPNGData, "parked window must still be captured")+ XCTAssertEqual(activeSpace(cid), homeSpace, "snapshot must not switch Spaces")+ XCTAssertNotEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, window.pid, "snapshot must not activate Chrome")++ let windowPoint = CGPoint(x: parked.bounds.width / 2, y: parked.bounds.height - 60)+ try SkyClickDispatcher.click(+ target: SkyClickTarget(+ screenPoint: CGPoint(x: parked.bounds.minX + windowPoint.x, y: parked.bounds.minY + windowPoint.y),+ windowPoint: windowPoint, windowBounds: parked.bounds, windowID: window.id, pid: window.pid),+ clickCount: 1, spi: spi)+ let clicked = try waitForWindow(pid: window.pid, nameContaining: "clicks=1", onScreenOnly: false)+ XCTAssertTrue(clicked.name.contains("clicks=1"), "sky_click must reach the parked window")++ try SkyKeyboardDispatcher.typeText(target: SkyKeyboardTarget(windowID: window.id, pid: window.pid), text: "s2", spi: spi)+ try SkyKeyboardDispatcher.pressKey(target: SkyKeyboardTarget(windowID: window.id, pid: window.pid), key: "cmd+a", spi: spi)+ try SkyKeyboardDispatcher.typeText(target: SkyKeyboardTarget(windowID: window.id, pid: window.pid), text: "Ok", spi: spi)+ let typed = try waitForWindow(pid: window.pid, nameContaining: "ocu-space-Ok|", onScreenOnly: false)+ print("cross-space live test: after typing title=\(typed.name)")+ XCTAssertTrue(typed.name.hasPrefix("ocu-space-Ok|clicks=1|"), "sky_key must type and select-all on the parked window")++ XCTAssertEqual(activeSpace(cid), homeSpace, "actions must not switch Spaces")+ XCTAssertNotEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, window.pid, "actions must not activate Chrome")+ XCTAssertEqual(spacesOfWindow(window.id), [otherSpace], "the window must stay on the other Space")+ WindowOcclusionKeepAlive.shared.releaseAll()+ }++ private struct WindowRecord { let id: CGWindowID; let pid: pid_t; let bounds: CGRect; let name: String; let onScreen: Bool }++ private func waitForWindow(pid: pid_t, nameContaining marker: String, onScreenOnly: Bool) throws -> WindowRecord {+ let deadline = Date().addingTimeInterval(10)+ while Date() < deadline {+ let options: CGWindowListOption = onScreenOnly ? [.optionOnScreenOnly, .excludeDesktopElements] : [.optionAll]+ let raw = CGWindowListCopyWindowInfo(options, kCGNullWindowID) as? [[String: Any]] ?? []+ for info in raw {+ guard let number = info[kCGWindowNumber as String] as? NSNumber,+ let owner = info[kCGWindowOwnerPID as String] as? NSNumber, owner.int32Value == pid,+ let name = info[kCGWindowName as String] as? String, name.contains(marker),+ let bounds = CGRect(dictionaryRepresentation: (info[kCGWindowBounds as String] as? NSDictionary) ?? [:]), bounds.width > 0+ else { continue }+ return WindowRecord(id: number.uint32Value, pid: pid, bounds: bounds, name: name, onScreen: (info[kCGWindowIsOnscreen as String] as? NSNumber)?.boolValue ?? false)+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.1))+ }+ throw ComputerUseError.message("Timed out waiting for window title containing \(marker)")+ }++ private func launch(executable: URL) throws -> Process {+ let process = Process(); process.executableURL = executable+ process.standardOutput = FileHandle.nullDevice; process.standardError = FileHandle.nullDevice+ try process.run(); return process+ }++ private func stop(_ process: Process) {+ if process.isRunning {+ process.terminate()+ let deadline = Date().addingTimeInterval(5)+ while process.isRunning, Date() < deadline { RunLoop.current.run(until: Date().addingTimeInterval(0.05)) }+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.25))+ }++ private static let liveTestHTML = #"""+ <!doctype html><meta charset="utf-8"><title>ocu-space-ready</title>+ <style>html,body{margin:0;height:100%} input{width:90%;font:22px system-ui;margin:20px} button{position:absolute;left:0;right:0;bottom:0;height:120px;font:22px system-ui;background:#5b8def;color:#fff;border:0}</style>+ <p>sky probe paragraph</p><input id="t" autofocus placeholder="sky probe"><button id="b">click probe</button>+ <script>+ let clicks = 0; const input = document.querySelector('#t');+ const update = () => { document.title = `ocu-space-${input.value}|clicks=${clicks}|${document.visibilityState}`; };+ document.querySelector('#b').addEventListener('click', () => { clicks += 1; update(); });+ input.addEventListener('input', update); document.addEventListener('visibilitychange', update); update();+ </script>+ """#++ private static let packageRoot: URL = {+ var url = URL(fileURLWithPath: #filePath)+ for _ in 0..<5 { url.deleteLastPathComponent() }+ return url+ }()+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OcclusionKeepAliveLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OcclusionKeepAliveLiveTests.swiftnew file mode 100644index 0000000..0278104--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OcclusionKeepAliveLiveTests.swift@@ -0,0 +1,170 @@+import AppKit+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++/// Opt-in live regression: a snapshot taken while an isolated Chrome window is+/// visible pins its visible state, so after another window fully covers it the+/// next snapshot still contains the web content and the page stays visible.+@MainActor+final class OcclusionKeepAliveLiveTests: XCTestCase {+ func testCoveredChromeKeepsWebContentAfterVisibleSnapshot() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_OCCLUSION_LIVE_TEST"] == "1" else {+ throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_OCCLUSION_LIVE_TEST=1 to run the isolated Chrome live test")+ }+ let spi = SkyLightSPI.shared+ guard spi.occlusionCapability.isAvailable else {+ throw XCTSkip("SkyLight occlusion SPI unavailable: \(spi.occlusionCapability.unavailableReason)")+ }+ let chromeURL = URL(fileURLWithPath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")+ guard FileManager.default.isExecutableFile(atPath: chromeURL.path) else {+ throw XCTSkip("Google Chrome is not installed at the standard path")+ }+ let coverExecutable = Self.packageRoot.appendingPathComponent(".build/debug/OpenComputerUseFixture")+ guard FileManager.default.isExecutableFile(atPath: coverExecutable.path) else {+ throw XCTSkip("Build OpenComputerUseFixture before running the live test")+ }+ let originalFrontApp = NSWorkspace.shared.frontmostApplication+ defer {+ if let originalFrontApp {+ _ = originalFrontApp.activate(options: [.activateAllWindows])+ }+ }++ let testRoot = Self.packageRoot.appendingPathComponent(".build/ocu-occlusion-live-\(UUID().uuidString)", isDirectory: true)+ let profileURL = testRoot.appendingPathComponent("chrome-profile", isDirectory: true)+ let pageURL = testRoot.appendingPathComponent("index.html")+ try FileManager.default.createDirectory(at: profileURL, withIntermediateDirectories: true)+ try Self.liveTestHTML.write(to: pageURL, atomically: false, encoding: .utf8)+ defer {+ try? FileManager.default.removeItem(at: testRoot)+ }++ let chrome = Process()+ chrome.executableURL = chromeURL+ chrome.arguments = [+ "--user-data-dir=\(profileURL.path)", "--no-first-run", "--no-default-browser-check",+ "--disable-background-networking", "--disable-component-update",+ "--window-position=200,200", "--window-size=600,420", "--app=\(pageURL.absoluteString)",+ ]+ chrome.standardOutput = FileHandle.nullDevice+ chrome.standardError = FileHandle.nullDevice+ try chrome.run()+ defer {+ stop(chrome)+ }+ let window = try waitForWindow(pid: chrome.processIdentifier, nameContaining: "ocu-occlusion-", onScreenOnly: true)+ guard let runningChrome = NSRunningApplication(processIdentifier: window.pid) else {+ return XCTFail("Chrome is not a running application")+ }+ let descriptor = RunningAppDescriptor(+ name: runningChrome.localizedName ?? "Google Chrome",+ bundleIdentifier: runningChrome.bundleIdentifier,+ pid: window.pid,+ runningApplication: runningChrome+ )++ // Chrome is visible: this snapshot must see web content and pin the window.+ _ = runningChrome.activate(options: [.activateAllWindows])+ RunLoop.current.run(until: Date().addingTimeInterval(1.5))+ let visibleSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)++ XCTAssertTrue(visibleSnapshot.treeLines.contains { $0.contains("sky probe") }, "visible snapshot must include the web content")+ XCTAssertTrue(WindowOcclusionKeepAlive.shared.isPinned(windowID: window.id), "the visible window must be pinned")+ XCTAssertEqual(visibleSnapshot.targetWindowID, window.id)++ let cover = try launch(executable: coverExecutable)+ defer {+ stop(cover)+ }+ _ = try waitForWindow(pid: cover.processIdentifier, nameContaining: "OpenComputerUseFixture", onScreenOnly: true)+ RunLoop.current.run(until: Date().addingTimeInterval(2.5))+ let coverBounds = try XCTUnwrap(waitForWindow(pid: cover.processIdentifier, nameContaining: "OpenComputerUseFixture", onScreenOnly: true).bounds)+ XCTAssertTrue(coverBounds.contains(window.bounds), "the fixture must fully cover Chrome")+ XCTAssertNotEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, window.pid)++ let pageTitle = try waitForWindow(pid: window.pid, nameContaining: "ocu-occlusion-", onScreenOnly: false).name+ XCTAssertTrue(pageTitle.hasSuffix("vis=visible"), "Chrome must still consider the covered page visible, got \(pageTitle)")++ let coveredSnapshot = try SnapshotBuilder.build(for: descriptor, recoveryPolicy: .readOnly)++ XCTAssertTrue(coveredSnapshot.treeLines.contains { $0.contains("sky probe") }, "covered snapshot must still include the web content")+ XCTAssertFalse(coveredSnapshot.treeLines.contains { $0.hasPrefix("Note: this window is covered") })+ XCTAssertNotNil(coveredSnapshot.screenshotPNGData)+ XCTAssertNotEqual(NSWorkspace.shared.frontmostApplication?.processIdentifier, window.pid, "snapshots must not activate Chrome")++ WindowOcclusionKeepAlive.shared.releaseAll()+ XCTAssertFalse(WindowOcclusionKeepAlive.shared.isPinned(windowID: window.id))+ }++ private struct WindowRecord {+ let id: CGWindowID+ let pid: pid_t+ let bounds: CGRect+ let name: String+ }++ private func waitForWindow(pid: pid_t, nameContaining marker: String, onScreenOnly: Bool) throws -> WindowRecord {+ let deadline = Date().addingTimeInterval(10)+ while Date() < deadline {+ let options: CGWindowListOption = onScreenOnly ? [.optionOnScreenOnly, .excludeDesktopElements] : [.optionAll]+ let raw = CGWindowListCopyWindowInfo(options, kCGNullWindowID) as? [[String: Any]] ?? []+ for info in raw {+ guard+ let number = info[kCGWindowNumber as String] as? NSNumber,+ let owner = info[kCGWindowOwnerPID as String] as? NSNumber, owner.int32Value == pid,+ let name = info[kCGWindowName as String] as? String, name.contains(marker),+ let bounds = CGRect(dictionaryRepresentation: (info[kCGWindowBounds as String] as? NSDictionary) ?? [:]),+ bounds.width > 0+ else {+ continue+ }+ return WindowRecord(id: number.uint32Value, pid: pid, bounds: bounds, name: name)+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.1))+ }+ throw ComputerUseError.message("Timed out waiting for window title containing \(marker)")+ }++ private func launch(executable: URL) throws -> Process {+ let process = Process()+ process.executableURL = executable+ process.standardOutput = FileHandle.nullDevice+ process.standardError = FileHandle.nullDevice+ try process.run()+ return process+ }++ private func stop(_ process: Process) {+ if process.isRunning {+ process.terminate()+ let deadline = Date().addingTimeInterval(5)+ while process.isRunning, Date() < deadline {+ RunLoop.current.run(until: Date().addingTimeInterval(0.05))+ }+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.25))+ }++ private static let liveTestHTML = #"""+ <!doctype html>+ <meta charset="utf-8">+ <title>ocu-occlusion-ready</title>+ <p>sky probe paragraph</p>+ <input id="target" autofocus placeholder="sky probe">+ <script>+ const update = () => { document.title = `ocu-occlusion-${document.visibilityState === 'visible' ? 'vis=visible' : 'vis=hidden'}`; };+ document.addEventListener('visibilitychange', update);+ update();+ </script>+ """#++ private static let packageRoot: URL = {+ var url = URL(fileURLWithPath: #filePath)+ for _ in 0..<5 {+ url.deleteLastPathComponent()+ }+ return url+ }()+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 09c45ce..86ed3f3 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -623,11 +623,16 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(instructions, computerUseServerInstructions)}- func testMCPAcceptsTurnEndedNotificationWithoutResponse() {- let server = StdioMCPServer(service: ComputerUseService())+ func testMCPTurnEndedNotificationResetsBackgroundStateWithoutResponse() {+ var resetCount = 0+ let server = StdioMCPServer(+ service: ComputerUseService(),+ backgroundStateReset: { resetCount += 1 }+ )let response = server.handle(line: #"{"jsonrpc":"2.0","method":"notifications/turn-ended","params":{"type":"agent-turn-complete"}}"#)XCTAssertNil(response)+ XCTAssertEqual(resetCount, 1)}func testWindowRelativeFrameUsesSharedGlobalCoordinates() {@@ -654,7 +659,7 @@ final class OpenComputerUseKitTests: XCTestCase {)XCTAssertEqual(tools["get_app_state"]?.annotations["readOnlyHint"] as? Bool,- true+ nil)XCTAssertEqual(tools["click"]?.inputSchema["additionalProperties"] as? Bool,@@ -1682,7 +1687,7 @@ final class OpenComputerUseKitTests: XCTestCase {}}- func testSkyClickWindowValidationRequiresMatchingOnScreenOwner() {+ func testSkyClickWindowValidationRequiresMatchingOwner() {let matching: [String: Any] = [kCGWindowNumber as String: NSNumber(value: UInt32(321)),kCGWindowOwnerPID as String: NSNumber(value: Int32(1234)),@@ -1697,12 +1702,17 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(skyClickWindowMatchesTarget(windowInfo: [matching], windowID: 321, pid: 1235))+ }- var offScreen = matching- offScreen[kCGWindowIsOnscreen as String] = NSNumber(value: false)- XCTAssertFalse(- skyClickWindowMatchesTarget(windowInfo: [offScreen], windowID: 321, pid: 1234)- )+ func testWindowLooksUnoccludedSamplesAGrid() {+ let bounds = CGRect(x: 100, y: 100, width: 400, height: 300)+ XCTAssertTrue(windowLooksUnoccluded(bounds: bounds, coveringBounds: []))+ XCTAssertFalse(windowLooksUnoccluded(bounds: bounds, coveringBounds: [bounds.insetBy(dx: -10, dy: -10)]))+ XCTAssertFalse(windowLooksUnoccluded(bounds: bounds, coveringBounds: [+ CGRect(x: 0, y: 0, width: 300, height: 1000), CGRect(x: 300, y: 0, width: 1000, height: 1000),+ ]), "two windows that jointly cover the target count as occlusion")+ XCTAssertTrue(windowLooksUnoccluded(bounds: bounds, coveringBounds: [CGRect(x: 100, y: 100, width: 360, height: 300)]), "an exposed strip keeps it visible")+ XCTAssertFalse(windowLooksUnoccluded(bounds: .zero, coveringBounds: []))}func testSkyLightCapabilityReportsMissingSymbols() {@@ -1752,6 +1762,170 @@ final class OpenComputerUseKitTests: XCTestCase {)}+ func testSkyLightKeyWindowRecordsFollowYabaiLayout() {+ let records = skyLightKeyWindowRecords(windowID: 0x1234_5678)+ XCTAssertEqual(records.map(\.count), [0xF8, 0xF8])+ XCTAssertEqual(records.map { $0[0x08] }, [0x01, 0x02])+ for record in records {+ XCTAssertEqual(record[0x04], 0xF8)+ XCTAssertEqual(record[0x3A], 0x10)+ XCTAssertEqual(Array(record[0x3C...0x3F]), [0x78, 0x56, 0x34, 0x12])+ XCTAssertEqual(Array(record[0x20..<0x30]), [UInt8](repeating: 0xFF, count: 0x10))+ XCTAssertEqual(record[0x8A], 0, "key-window records must not carry the activation focus marker")+ }+ }++ func testSkyMenuKeyEquivalentOnlyCoversCommandCharacterChords() throws {+ XCTAssertEqual(+ skyMenuKeyEquivalent(for: try KeyPressParser.parse("cmd+a")),+ SkyMenuKeyEquivalent(character: "A", modifiers: 0)+ )+ XCTAssertEqual(+ skyMenuKeyEquivalent(for: try KeyPressParser.parse("cmd+shift+option+ctrl+z")),+ SkyMenuKeyEquivalent(character: "Z", modifiers: 7)+ )+ XCTAssertNil(skyMenuKeyEquivalent(for: try KeyPressParser.parse("shift+a")), "non-command chords are key events")+ XCTAssertNil(skyMenuKeyEquivalent(for: try KeyPressParser.parse("cmd+shift+left")), "navigation chords are key bindings, not menu items")+ XCTAssertNil(skyMenuKeyEquivalent(for: try KeyPressParser.parse("return")))+ }++ func testSkyMenuItemMatchingIsCaseInsensitiveAndSkipsDisabledItems() {+ let selectAll = SkyMenuKeyEquivalent(character: "A", modifiers: 0)+ XCTAssertTrue(skyMenuItemMatches(cmdChar: "a", cmdModifiers: nil, enabled: nil, equivalent: selectAll))+ XCTAssertTrue(skyMenuItemMatches(cmdChar: "A", cmdModifiers: 0, enabled: true, equivalent: selectAll))+ XCTAssertFalse(skyMenuItemMatches(cmdChar: "A", cmdModifiers: 1, enabled: true, equivalent: selectAll))+ XCTAssertFalse(skyMenuItemMatches(cmdChar: "A", cmdModifiers: 0, enabled: false, equivalent: selectAll))+ XCTAssertFalse(skyMenuItemMatches(cmdChar: nil, cmdModifiers: 0, enabled: true, equivalent: selectAll))+ }++ func testSkyKeyWindowValidationAllowsOffScreenWindowsOfTheSameOwner() {+ let offScreen: [String: Any] = [+ kCGWindowNumber as String: NSNumber(value: UInt32(321)),+ kCGWindowOwnerPID as String: NSNumber(value: Int32(1234)),+ kCGWindowIsOnscreen as String: NSNumber(value: false),+ ]+ XCTAssertTrue(skyKeyWindowMatchesTarget(windowInfo: [offScreen], windowID: 321, pid: 1234))+ XCTAssertFalse(skyKeyWindowMatchesTarget(windowInfo: [offScreen], windowID: 321, pid: 1235))+ XCTAssertFalse(skyKeyWindowMatchesTarget(windowInfo: [offScreen], windowID: 322, pid: 1234))+ XCTAssertTrue(skyClickWindowMatchesTarget(windowInfo: [offScreen], windowID: 321, pid: 1234), "sky_click also accepts covered or other-Space windows")+ }++ func testLazyWebAccessibilityDetectionUsesShippedFrameworks() throws {+ let root = FileManager.default.temporaryDirectory.appendingPathComponent("ocu-lazy-ax-\(UUID().uuidString)")+ defer { try? FileManager.default.removeItem(at: root) }+ let chromium = root.appendingPathComponent("Chromium.app/Contents/Frameworks/Chromium Framework.framework")+ try FileManager.default.createDirectory(at: chromium, withIntermediateDirectories: true)+ let native = root.appendingPathComponent("Native.app/Contents/Frameworks/Sparkle.framework")+ try FileManager.default.createDirectory(at: native, withIntermediateDirectories: true)+ XCTAssertTrue(appHasLazyWebAccessibility(bundleURL: root.appendingPathComponent("Chromium.app")))+ XCTAssertFalse(appHasLazyWebAccessibility(bundleURL: root.appendingPathComponent("Native.app")))+ XCTAssertFalse(appHasLazyWebAccessibility(bundleURL: nil))+ }++ func testWindowPlacementParsingAndAgentDisplayGeometry() throws {+ XCTAssertEqual(try parseWindowPlacement(nil), .keep)+ XCTAssertEqual(try parseWindowPlacement(" Agent_Display "), .agentDisplay)+ XCTAssertEqual(try parseWindowPlacement("restore"), .restore)+ XCTAssertThrowsError(try parseWindowPlacement("park")) { error in+ XCTAssertEqual(+ (error as? ComputerUseError)?.errorDescription,+ "Invalid window_placement 'park'. Expected one of: keep, agent_display, restore"+ )+ }++ let display = CGRect(x: 1512, y: 0, width: 1920, height: 1080)+ XCTAssertEqual(agentDisplayPlacement(windowSize: CGSize(width: 600, height: 420), displayBounds: display), CGPoint(x: 1552, y: 40))+ let huge = agentDisplayPlacement(windowSize: CGSize(width: 4000, height: 3000), displayBounds: display)+ XCTAssertEqual(huge, CGPoint(x: 1512, y: 0), "oversized windows are pinned to the display origin")+ XCTAssertTrue(display.contains(huge))+ for name in ["get_app_state"] {+ let tool = ToolDefinitions.all.first { $0.name == name }+ let properties = tool?.inputSchema["properties"] as? [String: Any]+ XCTAssertEqual((properties?["window_placement"] as? [String: Any])?["enum"] as? [String], ["keep", "agent_display", "restore"])+ }+ }++ func testBackgroundInputTimingDefaultsKeepCompatibilityMargins() {+ XCTAssertEqual(InputTiming.typeChunkDelayDefaultMilliseconds, 20)+ XCTAssertEqual(InputTiming.pressKeySettleDefaultMilliseconds, 100)+ XCTAssertEqual(SkyKeyboardDispatcher.keyWindowFallbackSettleDefaultMilliseconds, 10)+ XCTAssertEqual(SkyKeyboardDispatcher.releaseSettleDefaultMilliseconds, 10)+ XCTAssertEqual(+ InputTiming.milliseconds("OCU_TEST_TIMING", default: 20, environment: [:]),+ 0.020,+ accuracy: 0.000_001+ )+ XCTAssertEqual(+ InputTiming.milliseconds("OCU_TEST_TIMING", default: 20, environment: ["OCU_TEST_TIMING": "7.5"]),+ 0.0075,+ accuracy: 0.000_001+ )+ }++ func testAgentDisplayRestoreGeometryUsesATolerance() {+ let original = CGPoint(x: 200, y: 300)+ XCTAssertTrue(AgentDisplay.isRestored(frame: CGRect(x: 200.5, y: 299.5, width: 600, height: 420), to: original))+ XCTAssertFalse(AgentDisplay.isRestored(frame: CGRect(x: 202, y: 300, width: 600, height: 420), to: original))+ }++ func testOcclusionKeepAliveRestoresPriorStateAndCanRetryFailures() {+ final class FakeOcclusionSPI: WindowOcclusionControlling {+ let occlusionCapability = SkyLightSPICapability(missingSymbols: [], feature: "occlusion")+ var states: [CGWindowID: Bool] = [101: true, 102: false]+ var failedRestores: Set<CGWindowID> = [101]+ var calls: [CGWindowID: Int] = [:]++ func setWindowOcclusionNotificationsEnabled(_ enabled: Bool, windowID: CGWindowID) throws -> Bool {+ calls[windowID, default: 0] += 1+ let previous = states[windowID] ?? true+ if calls[windowID, default: 0] > 1, failedRestores.remove(windowID) != nil {+ throw ComputerUseError.message("injected restore failure")+ }+ states[windowID] = enabled+ return previous+ }+ }++ let keepAlive = WindowOcclusionKeepAlive()+ let spi = FakeOcclusionSPI()+ let visibleBounds = CGRect(x: 10, y: 10, width: 100, height: 100)+ XCTAssertTrue(keepAlive.keepVisible(windowID: 101, bounds: visibleBounds, spi: spi))+ XCTAssertTrue(keepAlive.keepVisible(windowID: 102, bounds: visibleBounds, spi: spi))+ XCTAssertEqual(spi.states[101], false)+ XCTAssertEqual(spi.states[102], false)++ keepAlive.releaseAll(spi: spi)+ XCTAssertTrue(keepAlive.isPinned(windowID: 101), "failed restores must remain retryable")+ XCTAssertFalse(keepAlive.isPinned(windowID: 102))+ XCTAssertEqual(spi.states[102], false, "restore the original disabled state, not always true")++ keepAlive.releaseAll(spi: spi)+ XCTAssertFalse(keepAlive.isPinned(windowID: 101))+ XCTAssertEqual(spi.states[101], true)+ }++ func testKeyMethodParsingAndPolicy() throws {+ XCTAssertEqual(try parseKeyMethod(nil), .auto)+ XCTAssertEqual(try parseKeyMethod(" SKY_KEY "), .skyKey)+ XCTAssertThrowsError(try parseKeyMethod("global")) { error in+ XCTAssertEqual(+ (error as? ComputerUseError)?.errorDescription,+ "Invalid key_method 'global'. Expected one of: auto, sky_key"+ )+ }+ XCTAssertEqual(keyActionSnapshotRecoveryPolicy(for: .skyKey), .readOnly)+ XCTAssertEqual(keyActionSnapshotRecoveryPolicy(for: .auto), .allowActivation)+ }++ func testKeyboardToolSchemasExposeKeyMethod() {+ for name in ["type_text", "press_key"] {+ let tool = ToolDefinitions.all.first { $0.name == name }+ let properties = tool?.inputSchema["properties"] as? [String: Any]+ let keyMethod = properties?["key_method"] as? [String: Any]+ XCTAssertEqual(keyMethod?["enum"] as? [String], ["auto", "sky_key"], name)+ }+ }+func testSkyLightRuntimeSPIProbeCanStampEventWithoutPosting() throws {let spi = SkyLightSPI.sharedguard spi.capability.isAvailable else {@@ -2334,6 +2508,7 @@ final class OpenComputerUseKitTests: XCTestCase {focusedSummary: focusedSummary,focusedElement: nil,selectedText: selectedText,+ windowElement: nil,elements: [:])}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyKeyboardLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyKeyboardLiveTests.swiftnew file mode 100644index 0000000..7c49f0a--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyKeyboardLiveTests.swift@@ -0,0 +1,365 @@+import AppKit+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++@MainActor+final class SkyKeyboardLiveTests: XCTestCase {+ private struct WindowRecord {+ let id: CGWindowID+ let pid: pid_t+ let bounds: CGRect+ let name: String+ }++ func testCoveredChromeReceivesSkyKeyTypingWithoutForegroundSideEffects() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST"] == "1" else {+ throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_SKY_KEY_LIVE_TEST=1 to run the isolated Chrome live test")+ }+ let spi = SkyLightSPI.shared+ guard spi.capability.isAvailable else {+ throw XCTSkip("SkyLight SPI unavailable: \(spi.capability.unavailableReason)")+ }++ let chromeURL = URL(fileURLWithPath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")+ guard FileManager.default.isExecutableFile(atPath: chromeURL.path) else {+ throw XCTSkip("Google Chrome is not installed at the standard path")+ }+ let originalFrontApp = NSWorkspace.shared.frontmostApplication+ defer {+ if let originalFrontApp {+ _ = originalFrontApp.activate(options: [.activateAllWindows])+ }+ }++ let coverExecutable = Self.packageRoot+ .appendingPathComponent(".build/debug/OpenComputerUseFixture")+ guard FileManager.default.isExecutableFile(atPath: coverExecutable.path) else {+ throw XCTSkip("Build OpenComputerUseFixture before running the live test")+ }+ let coverProbe = try launch(executable: coverExecutable)+ defer {+ stop(coverProbe)+ }+ let coverProbeWindow = try waitForWindow(+ pid: coverProbe.processIdentifier,+ nameContaining: "OpenComputerUseFixture"+ )++ let testRoot = Self.packageRoot.appendingPathComponent(".build", isDirectory: true)+ .appendingPathComponent("ocu-sky-key-live-\(UUID().uuidString)", isDirectory: true)+ let profileURL = testRoot.appendingPathComponent("chrome-profile", isDirectory: true)+ let pageURL = testRoot.appendingPathComponent("index.html")+ try FileManager.default.createDirectory(at: profileURL, withIntermediateDirectories: true)+ try Self.liveTestHTML.write(to: pageURL, atomically: false, encoding: .utf8)+ defer {+ do {+ try FileManager.default.removeItem(at: testRoot)+ } catch {+ print("sky_key live test cleanup warning: \(error)")+ }+ }++ let targetWidth = Int(min(640, coverProbeWindow.bounds.width - 80))+ let targetHeight = Int(min(480, coverProbeWindow.bounds.height - 80))+ let targetX = Int(coverProbeWindow.bounds.minX + 40)+ let targetY = Int(coverProbeWindow.bounds.minY + 40)++ let chrome = Process()+ chrome.executableURL = chromeURL+ chrome.arguments = [+ "--user-data-dir=\(profileURL.path)",+ "--no-first-run",+ "--no-default-browser-check",+ "--disable-background-networking",+ "--disable-component-update",+ "--window-position=\(targetX),\(targetY)",+ "--window-size=\(targetWidth),\(targetHeight)",+ "--app=\(pageURL.absoluteString)",+ ]+ chrome.standardOutput = FileHandle.nullDevice+ chrome.standardError = FileHandle.nullDevice+ try chrome.run()+ defer {+ stop(chrome)+ }++ let readyWindow = try waitForWindow(+ pid: chrome.processIdentifier,+ nameContaining: "ocu-sky-key-ready"+ )+ print("sky_key live test: target window=\(readyWindow.id) owner pid=\(readyWindow.pid)")+ stop(coverProbe)+ let cover = try launch(executable: coverExecutable)+ defer {+ stop(cover)+ }+ let coverWindow = try waitForWindow(+ pid: cover.processIdentifier,+ nameContaining: "OpenComputerUseFixture"+ )+ try waitUntil(timeout: 5, failure: "Isolated Chrome remained the frontmost app") {+ guard let frontPID = NSWorkspace.shared.frontmostApplication?.processIdentifier else {+ return false+ }+ return frontPID != readyWindow.pid+ }+ try waitUntil(timeout: 5, failure: "The cover window did not move above isolated Chrome") {+ let ordered = windows()+ guard+ let currentCoverIndex = ordered.firstIndex(where: { $0.id == coverWindow.id }),+ let currentTargetIndex = ordered.firstIndex(where: { $0.id == readyWindow.id })+ else {+ return false+ }+ return currentCoverIndex < currentTargetIndex+ }++ let orderedBefore = windows()+ guard+ let coveredWindow = orderedBefore.first(where: { $0.id == readyWindow.id }),+ let freshCoverWindow = orderedBefore.first(where: { $0.id == coverWindow.id }),+ let coverIndex = orderedBefore.firstIndex(where: { $0.id == coverWindow.id }),+ let targetIndex = orderedBefore.firstIndex(where: { $0.id == readyWindow.id })+ else {+ return XCTFail("Could not re-read the controlled Chrome and cover windows")+ }+ XCTAssertTrue(+ freshCoverWindow.bounds.contains(coveredWindow.bounds),+ "The controlled Chrome window must be fully covered before sky_key"+ )+ XCTAssertLessThan(coverIndex, targetIndex, "The cover window must be above Chrome in z-order")+ try FixtureBridge.post(+ FixtureCommand(kind: "click", identifier: "fixture-input")+ )+ let foregroundStateBefore = try waitForFixtureState(+ pid: cover.processIdentifier,+ failure: "The foreground fixture did not keep its text field focused"+ ) { state in+ state.isActive == true+ && state.isKeyWindow == true+ && state.focusedIdentifier == "fixture-input"+ }++ let frontPIDBefore = NSWorkspace.shared.frontmostApplication?.processIdentifier+ let cursorBefore = CGEvent(source: nil)?.location+ let target = SkyKeyboardTarget(windowID: coveredWindow.id, pid: coveredWindow.pid)++ try SkyKeyboardDispatcher.typeText(target: target, text: "hello", spi: spi)+ let typedWindow = try waitForWindow(pid: readyWindow.pid, nameContaining: "ocu-sky-key-value-hello|")+ XCTAssertTrue(+ typedWindow.name.hasPrefix("ocu-sky-key-value-hello|"),+ "sky_key typing must land in the covered Chrome input exactly once"+ )++ try SkyKeyboardDispatcher.pressKey(target: target, key: "backspace", spi: spi)+ try SkyKeyboardDispatcher.pressKey(target: target, key: "shift+a", spi: spi)+ _ = try waitForWindow(pid: readyWindow.pid, nameContaining: "ocu-sky-key-value-hellA|")++ // Menu-bar key equivalents go through the target's Edit menu.+ try SkyKeyboardDispatcher.pressKey(target: target, key: "cmd+a", spi: spi)+ try SkyKeyboardDispatcher.typeText(target: target, text: "Ok!", spi: spi)+ let editedWindow = try waitForWindow(pid: readyWindow.pid, nameContaining: "ocu-sky-key-value-Ok!|")+ RunLoop.current.run(until: Date().addingTimeInterval(0.5))+ let finalWindow = try XCTUnwrap(windows().first(where: { $0.id == editedWindow.id }))+ XCTAssertEqual(+ finalWindow.name,+ "ocu-sky-key-value-Ok!|unfocused",+ "sky_key must deliver keys, key-binding chords and menu chords, then release Chrome's synthetic key state"+ )++ // Control: the plain CGEvent.postToPid keyboard path that `auto` uses.+ // Chromium receives the key events but refuses text insertion while its+ // window is not key. Observed, not asserted, so a future Chromium change+ // cannot fail the sky_key regression.+ try InputSimulation.typeText("plain", pid: coveredWindow.pid)+ RunLoop.current.run(until: Date().addingTimeInterval(0.5))+ let controlTitle = windows().first(where: { $0.id == editedWindow.id })?.name ?? ""+ print("sky_key live test: public postToPid control title after release=\(controlTitle)")++ XCTAssertEqual(+ NSWorkspace.shared.frontmostApplication?.processIdentifier,+ frontPIDBefore,+ "sky_key must not change the frontmost app"+ )+ let foregroundStateAfter = try waitForFixtureState(+ pid: cover.processIdentifier,+ failure: "The foreground fixture state was unavailable after sky_key"+ ) { _ in true }+ XCTAssertEqual(foregroundStateAfter.isActive, true, "sky_key must keep the foreground app active")+ XCTAssertEqual(foregroundStateAfter.isKeyWindow, true, "sky_key must keep the foreground window key")+ XCTAssertEqual(+ foregroundStateAfter.focusedIdentifier,+ foregroundStateBefore.focusedIdentifier,+ "sky_key must preserve the foreground first responder"+ )+ XCTAssertEqual(+ foregroundStateAfter.activationLossCount,+ foregroundStateBefore.activationLossCount,+ "sky_key must not transiently resign the foreground app"+ )+ XCTAssertEqual(+ foregroundStateAfter.keyWindowLossCount,+ foregroundStateBefore.keyWindowLossCount,+ "sky_key must not transiently resign the foreground key window"+ )+ XCTAssertEqual(+ foregroundStateAfter.elements.first(where: { $0.identifier == "fixture-input" })?.value,+ foregroundStateBefore.elements.first(where: { $0.identifier == "fixture-input" })?.value,+ "sky_key keystrokes must not leak into the foreground text field"+ )++ if let cursorBefore, let cursorAfter = CGEvent(source: nil)?.location {+ print("sky_key live test: cursor before=\(cursorBefore) after=\(cursorAfter)")+ XCTAssertLessThan(hypot(cursorAfter.x - cursorBefore.x, cursorAfter.y - cursorBefore.y), 0.5, "sky_key must not move the real pointer (a user moving the mouse during the run also trips this)")+ }++ let orderedAfter = windows()+ guard+ let coverIndexAfter = orderedAfter.firstIndex(where: { $0.id == coverWindow.id }),+ let targetIndexAfter = orderedAfter.firstIndex(where: { $0.id == editedWindow.id })+ else {+ return XCTFail("Could not verify final window z-order")+ }+ XCTAssertLessThan(coverIndexAfter, targetIndexAfter, "sky_key must not raise the Chrome window")+ }++ private func waitForWindow(+ pid: pid_t? = nil,+ nameContaining marker: String,+ timeout: TimeInterval = 10+ ) throws -> WindowRecord {+ var match: WindowRecord?+ do {+ try waitUntil(timeout: timeout, failure: "Timed out waiting for Chrome window title containing \(marker)") {+ match = windows().first(where: { window in+ (pid == nil || window.pid == pid) && window.name.contains(marker)+ })+ return match != nil+ }+ } catch {+ let observed = windows()+ .filter { pid == nil || $0.pid == pid }+ .map { "\($0.id):\($0.name)" }+ .joined(separator: ", ")+ throw ComputerUseError.message(+ "Timed out waiting for Chrome window title containing \(marker); observed [\(observed)]"+ )+ }+ return try XCTUnwrap(match)+ }++ private func waitUntil(+ timeout: TimeInterval,+ failure: String,+ condition: () -> Bool+ ) throws {+ let deadline = Date().addingTimeInterval(timeout)+ while Date() < deadline {+ if condition() {+ return+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.05))+ }+ throw ComputerUseError.message(failure)+ }++ private func waitForFixtureState(+ pid: pid_t,+ failure: String,+ condition: (FixtureAppState) -> Bool+ ) throws -> FixtureAppState {+ var match: FixtureAppState?+ try waitUntil(timeout: 5, failure: failure) {+ guard+ let state = try? FixtureBridge.readState(),+ state.processIdentifier == pid,+ condition(state)+ else {+ return false+ }+ match = state+ return true+ }+ return try XCTUnwrap(match)+ }++ private func stop(_ process: Process) {+ if process.isRunning {+ process.terminate()+ let deadline = Date().addingTimeInterval(5)+ while process.isRunning, Date() < deadline {+ RunLoop.current.run(until: Date().addingTimeInterval(0.05))+ }+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.25))+ }++ private func launch(executable: URL) throws -> Process {+ let process = Process()+ process.executableURL = executable+ process.standardOutput = FileHandle.nullDevice+ process.standardError = FileHandle.nullDevice+ try process.run()+ return process+ }++ private func windows() -> [WindowRecord] {+ let rawWindows = CGWindowListCopyWindowInfo(+ [.optionOnScreenOnly, .excludeDesktopElements],+ kCGNullWindowID+ ) as? [[String: Any]] ?? []++ return rawWindows.compactMap { info in+ guard+ let number = info[kCGWindowNumber as String] as? NSNumber,+ let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,+ let layer = info[kCGWindowLayer as String] as? NSNumber,+ layer.intValue == 0,+ let boundsDictionary = info[kCGWindowBounds as String] as? [String: Any],+ let bounds = CGRect(dictionaryRepresentation: boundsDictionary as CFDictionary),+ bounds.width > 0,+ bounds.height > 0+ else {+ return nil+ }++ return WindowRecord(+ id: number.uint32Value,+ pid: ownerPID.int32Value,+ bounds: bounds,+ name: info[kCGWindowName as String] as? String ?? ""+ )+ }+ }++ private static let liveTestHTML = #"""+ <!doctype html>+ <meta charset="utf-8">+ <title>ocu-sky-key-ready</title>+ <style>+ html, body { width: 100%; height: 100%; margin: 0; }+ input { width: 90%; font: 24px system-ui; margin: 24px; }+ </style>+ <input id="target" autofocus placeholder="sky_key probe">+ <script>+ const input = document.querySelector('#target');+ input.focus();+ const update = () => {+ document.title = `ocu-sky-key-value-${input.value}|${document.hasFocus() ? 'focused' : 'unfocused'}`;+ };+ input.addEventListener('input', update);+ window.addEventListener('focus', update);+ window.addEventListener('blur', update);+ </script>+ """#++ private static let packageRoot: URL = {+ var url = URL(fileURLWithPath: #filePath)+ for _ in 0..<5 {+ url.deleteLastPathComponent()+ }+ return url+ }()+}diff --git a/packages/OpenComputerUseVirtualDisplayShim/OpenComputerUseVirtualDisplayShim.m b/packages/OpenComputerUseVirtualDisplayShim/OpenComputerUseVirtualDisplayShim.mnew file mode 100644index 0000000..cb98938--- /dev/null+++ b/packages/OpenComputerUseVirtualDisplayShim/OpenComputerUseVirtualDisplayShim.m@@ -0,0 +1,91 @@+#import "OpenComputerUseVirtualDisplayShim.h"++#import <CoreGraphics/CoreGraphics.h>+#import <Foundation/Foundation.h>++// Method surfaces of the private classes, declared as protocols so messaging+// is typed for ARC without creating link-time class references.+@protocol OCUVirtualDisplayMode <NSObject>+- (instancetype)initWithWidth:(NSUInteger)width height:(NSUInteger)height refreshRate:(double)refreshRate;+@end++@protocol OCUVirtualDisplay <NSObject>+- (instancetype)initWithDescriptor:(id)descriptor;+- (BOOL)applySettings:(id)settings;+@end++int ocu_virtual_display_is_supported(void) {+ return NSClassFromString(@"CGVirtualDisplayDescriptor") != nil+ && NSClassFromString(@"CGVirtualDisplay") != nil+ && NSClassFromString(@"CGVirtualDisplaySettings") != nil+ && NSClassFromString(@"CGVirtualDisplayMode") != nil;+}++uint32_t ocu_virtual_display_create(const char *name, uint32_t width, uint32_t height, double refresh_rate, void **handle_out) {+ if (handle_out == NULL || !ocu_virtual_display_is_supported()) {+ return 0;+ }+ *handle_out = NULL;++ @try {+ @autoreleasepool {+ Class descriptorClass = NSClassFromString(@"CGVirtualDisplayDescriptor");+ Class displayClass = NSClassFromString(@"CGVirtualDisplay");+ Class settingsClass = NSClassFromString(@"CGVirtualDisplaySettings");+ Class modeClass = NSClassFromString(@"CGVirtualDisplayMode");++ // Scalars go through KVC so the runtime boxes them with the property's+ // real type encoding instead of a guessed C type.+ NSObject *descriptor = [[descriptorClass alloc] init];+ [descriptor setValue:[NSString stringWithUTF8String:name ?: "Open Computer Use"] forKey:@"name"];+ [descriptor setValue:@(width) forKey:@"maxPixelsWide"];+ [descriptor setValue:@(height) forKey:@"maxPixelsHigh"];+ [descriptor setValue:[NSValue valueWithSize:NSMakeSize(width * 0.26, height * 0.26)] forKey:@"sizeInMillimeters"];+ [descriptor setValue:@(0x0CAC) forKey:@"productID"];+ [descriptor setValue:@(0x0CAC) forKey:@"vendorID"];+ [descriptor setValue:@(1) forKey:@"serialNum"];+ [descriptor setValue:dispatch_get_main_queue() forKey:@"queue"];++ id<OCUVirtualDisplay> display = [[displayClass alloc] initWithDescriptor:descriptor];+ if (display == nil) {+ return 0;+ }++ id<OCUVirtualDisplayMode> mode = [[modeClass alloc] initWithWidth:width height:height refreshRate:refresh_rate];+ NSObject *settings = [[settingsClass alloc] init];+ [settings setValue:@(1) forKey:@"hiDPI"];+ [settings setValue:@[mode] forKey:@"modes"];+ if (![display applySettings:settings]) {+ return 0;+ }++ uint32_t displayID = [[(NSObject *)display valueForKey:@"displayID"] unsignedIntValue];+ if (displayID == 0) {+ return 0;+ }+ *handle_out = (void *)CFBridgingRetain(display);+ return displayID;+ }+ } @catch (NSException *exception) {+ // Private KVC keys and selector signatures can change between macOS+ // releases. Treat that as an unavailable capability, not a crash.+ (void)exception;+ return 0;+ }+}++void ocu_virtual_display_destroy(void *handle) {+ if (handle == NULL) {+ return;+ }+ @try {+ @autoreleasepool {+ id display = CFBridgingRelease(handle);+ display = nil;+ }+ } @catch (NSException *exception) {+ // Destruction is best-effort; there is no safe recovery operation for+ // an opaque private object after its teardown throws.+ (void)exception;+ }+}diff --git a/packages/OpenComputerUseVirtualDisplayShim/include/OpenComputerUseVirtualDisplayShim.h b/packages/OpenComputerUseVirtualDisplayShim/include/OpenComputerUseVirtualDisplayShim.hnew file mode 100644index 0000000..f404cca--- /dev/null+++ b/packages/OpenComputerUseVirtualDisplayShim/include/OpenComputerUseVirtualDisplayShim.h@@ -0,0 +1,30 @@+// OpenComputerUseVirtualDisplayShim — the only place that touches the private+// CGVirtualDisplay classes (CoreGraphics). Every class is resolved with+// NSClassFromString at runtime, so nothing here is a hard link dependency; a+// macOS without the classes reports "unsupported" and the caller fails closed.+#ifndef OPEN_COMPUTER_USE_VIRTUAL_DISPLAY_SHIM_H+#define OPEN_COMPUTER_USE_VIRTUAL_DISPLAY_SHIM_H++#include <stdint.h>++#ifdef __cplusplus+extern "C" {+#endif++/// 1 when the private CGVirtualDisplay classes exist on this system.+int ocu_virtual_display_is_supported(void);++/// Create a virtual display of `width` x `height` points at `refresh_rate` Hz.+/// Returns the CGDirectDisplayID (0 on failure) and stores an opaque retained+/// handle in `handle_out`; pass it to `ocu_virtual_display_destroy` to remove+/// the display again.+uint32_t ocu_virtual_display_create(const char *name, uint32_t width, uint32_t height, double refresh_rate, void **handle_out);++/// Remove the display and release the handle.+void ocu_virtual_display_destroy(void *handle);++#ifdef __cplusplus+}+#endif++#endifdiff --git a/scripts/node-repl/open-computer-use-repl.mjs b/scripts/node-repl/open-computer-use-repl.mjsindex 822fac8..2bbb122 100755--- a/scripts/node-repl/open-computer-use-repl.mjs+++ b/scripts/node-repl/open-computer-use-repl.mjs@@ -37,14 +37,14 @@ The runtime exposes an asynchronous app-bound API:- \`await cua.getState({ emit? })\`: list current apps.- \`await cua.listApps({ emit? })\`: list current apps.- \`await cua.getApp(nameOrBundleID)\`: bind an app and emit its initial accessibility state.-- \`await app.getAXState({ emit?, textLimit?, maxTreeNodes?, maxTreeDepth? })\`+- \`await app.getAXState({ emit?, textLimit?, maxTreeNodes?, maxTreeDepth?, windowPlacement? })\`- \`await app.getScreenshot({ emit? })\`- \`await app.getAXStateAndScreenshot(options?)\`- \`await app.click(elementIndexOrPoint, { mouseButton?, clickCount?, clickMethod? })\`- \`await app.scroll(elementIndex, direction, pages?)\`- \`await app.drag([fromX, fromY], [toX, toY])\`-- \`await app.typeText(text)\`-- \`await app.pressKey(key)\`+- \`await app.typeText(text, { keyMethod? })\`+- \`await app.pressKey(key, { keyMethod? })\`- \`await app.setValue(elementIndex, value)\`- \`await app.performSecondaryAction(elementIndex, action)\`@@ -220,6 +220,7 @@ function optionsToSnapshotArgs(options = {}) {if (options.textLimit !== undefined) args.text_limit = options.textLimit;if (options.maxTreeNodes !== undefined) args.max_tree_nodes = options.maxTreeNodes;if (options.maxTreeDepth !== undefined) args.max_tree_depth = options.maxTreeDepth;+ if (options.windowPlacement !== undefined) args.window_placement = options.windowPlacement;return args;}@@ -231,6 +232,12 @@ function optionsToClickArgs(options = {}) {return args;}+function optionsToKeyArgs(options = {}) {+ const args = {};+ if (options.keyMethod !== undefined) args.key_method = options.keyMethod;+ return args;+}+export function createCuaApi(native, activeOutput) {let docsEmitted = false;async function call(tool, args = {}) {@@ -260,7 +267,7 @@ export function createCuaApi(native, activeOutput) {return text;},async getScreenshot(options = {}) {- const result = await call("get_app_state", { app, text_limit: 1 });+ const result = await call("get_app_state", { app, ...optionsToSnapshotArgs(options), text_limit: 1 });const image = toolResultImages(result)[0];if (!image) throw new Error(`Screenshot unavailable for ${app}`);const bytes = Buffer.from(image.data, "base64");@@ -280,13 +287,13 @@ export function createCuaApi(native, activeOutput) {await call("click", { app, ...targetArgs, ...optionsToClickArgs(options) });},async drag(from, to) { await call("drag", { app, from_x: from[0], from_y: from[1], to_x: to[0], to_y: to[1] }); },- async pressKey(key) { await call("press_key", { app, key }); },+ async pressKey(key, options = {}) { await call("press_key", { app, key, ...optionsToKeyArgs(options) }); },async scroll(target, direction, pages = 1) {if (Array.isArray(target)) throw new Error("coordinate scroll is not supported by this Open Computer Use runtime");await call("scroll", { app, element_index: target, direction, pages });},async setValue(elementIndex, value) { await call("set_value", { app, element_index: elementIndex, value }); },- async typeText(text) { await call("type_text", { app, text }); },+ async typeText(text, options = {}) { await call("type_text", { app, text, ...optionsToKeyArgs(options) }); },async performSecondaryAction(elementIndex, action) { await call("perform_secondary_action", { app, element_index: elementIndex, action }); },});}diff --git a/scripts/node-repl/open-computer-use-repl.test.mjs b/scripts/node-repl/open-computer-use-repl.test.mjsindex 8293c7f..fd1d4b6 100644--- a/scripts/node-repl/open-computer-use-repl.test.mjs+++ b/scripts/node-repl/open-computer-use-repl.test.mjs@@ -102,6 +102,47 @@ test("app-bound API composes actions and state in one js call", async () => {assert.ok(result.content.some(item => item.type === "text" && item.text === "state:Text"));});+test("app-bound API forwards background-operation options", async () => {+ const native = mockNative();+ const session = new PersistentJavaScriptSession({ native });+ const result = await session.run(`+ var backgroundApp = await cua.getApp("Text");+ await backgroundApp.getAXState({ emit: false, textLimit: "max", maxTreeNodes: 42, maxTreeDepth: 7, windowPlacement: "agent_display" });+ await backgroundApp.getScreenshot({ emit: false, windowPlacement: "keep" });+ await backgroundApp.getAXStateAndScreenshot({ emit: false, windowPlacement: "restore" });+ await backgroundApp.typeText("hello", { keyMethod: "sky_key" });+ await backgroundApp.pressKey("cmd+a", { keyMethod: "sky_key" });+ `);++ assert.equal(result.isError, false);+ assert.deepEqual(native.calls[1], {+ name: "get_app_state",+ arguments: {+ app: "Text",+ text_limit: "max",+ max_tree_nodes: 42,+ max_tree_depth: 7,+ window_placement: "agent_display",+ },+ });+ assert.deepEqual(native.calls[2], {+ name: "get_app_state",+ arguments: { app: "Text", window_placement: "keep", text_limit: 1 },+ });+ assert.deepEqual(native.calls[3], {+ name: "get_app_state",+ arguments: { app: "Text", window_placement: "restore" },+ });+ assert.deepEqual(native.calls[4], {+ name: "type_text",+ arguments: { app: "Text", text: "hello", key_method: "sky_key" },+ });+ assert.deepEqual(native.calls[5], {+ name: "press_key",+ arguments: { app: "Text", key: "cmd+a", key_method: "sky_key" },+ });+});+test("tool errors are catchable in JavaScript", async () => {const session = new PersistentJavaScriptSession({ native: mockNative() });const result = await session.run(`diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.mdindex 5bff5f2..d21ea36 100644--- a/skills/open-computer-use/references/usage.md+++ b/skills/open-computer-use/references/usage.md@@ -51,6 +51,16 @@ await app.typeText("Hello");await app.getAXState();```+The code-first surface also forwards macOS background-operation options:++```js+var chrome = await cua.getApp("Google Chrome");+await chrome.getAXState({ windowPlacement: "agent_display" });+await chrome.typeText("hello", { keyMethod: "sky_key" });+await chrome.pressKey("cmd+a", { keyMethod: "sky_key" });+await chrome.getAXState({ windowPlacement: "restore" });+```+Bindings persist across `js` calls until `js_reset`. See`docs/references/js-repl.md` for the full API and security boundary.@@ -170,6 +180,34 @@ Keep the environment override scoped as narrowly as possible. While it remains eWindows returns an unsupported error for `sky_click` and `global`; Linux returns an unsupported error for `app_post` and `sky_click`. An unsupported or failed explicit method does not fall back to `auto`.+## Choosing a Keyboard Method++`key_method` on `type_text` and `press_key` is optional. Omitting it uses `auto`, which keeps the existing accessibility-first typing and app-posted key events. Explicit methods never fall back:++- `sky_key`: use the macOS private SkyLight background-window path. It makes the current target window the key window inside its own app (without raising it, moving the pointer, switching Spaces, or deactivating the foreground app), delivers the keys, and releases that state again. Command chords such as `cmd+a`, `cmd+c` or `cmd+v` are performed through the target's own menu bar. Supported on macOS only.++Use `sky_key` when a Chromium or Electron window is covered by another window or lives on another Space and `auto` typing does not reach it:++```sh+open-computer-use call get_app_state --args '{"app":"Google Chrome"}'+open-computer-use call click --args '{"app":"Google Chrome","x":420,"y":260,"click_method":"sky_click"}'+open-computer-use call type_text --args '{"app":"Google Chrome","text":"hello","key_method":"sky_key"}'+open-computer-use call press_key --args '{"app":"Google Chrome","key":"cmd+a","key_method":"sky_key"}'+```++Covered windows and windows on other Spaces: `get_app_state` never activates or raises a window that is off-screen, and it pins the visibility of every window it captures while that window is unoccluded, so covering it or switching Spaces afterwards keeps Chromium/Electron content in the tree and screenshots live. If the tree ends with a note that the window is covered, the app had already hidden its content before the first snapshot. Either bring the window into view once and snapshot again, or park it on the agent's own invisible display:++```sh+open-computer-use call --calls '[+ {"tool":"get_app_state","args":{"app":"Google Chrome","window_placement":"agent_display"}},+ {"tool":"get_app_state","args":{"app":"Google Chrome","window_placement":"restore"}}+]'+```++`agent_display` moves the window's frame onto a virtual display the user never sees (macOS only): the app renders it, exposes its full tree and accepts `sky_click` / `sky_key`, while the user's Space, focus and pointer stay untouched. The window is gone from the user's desktop until `restore`, the assistant turn ends, the MCP/REPL connection closes, or the runtime exits, so treat it as a deliberate, temporary step. `restore` restores every window parked by the runtime for that app. Keep park/action/restore calls in one `js`, REPL, MCP, or `call --calls` session; a one-shot CLI connection restores its background state when it closes. `sky_click` and `sky_key` accept covered and other-Space windows.++`sky_key` types into whatever element the target window has focused, so click or AX-focus the field first. It fails closed for hidden apps (`cmd+h`), stale window ids, and missing private symbols; Windows and Linux return an unsupported error.+## Drag Delivery`drag` has no method parameter. On macOS the path it takes is decided by the same process-level gate that authorizes `click_method: "global"`:@@ -185,7 +223,7 @@ When the gate is not enabled, treat window-server drags as unavailable and reach### macOS-The macOS runtime uses Accessibility, ScreenCaptureKit, app-posted input events, and an explicit private-SkyLight `sky_click` route. It normally avoids moving the user's real pointer. The visual cursor overlay is part of the Open Computer Use experience and can be disabled by the surrounding runtime only when needed. Private SkyLight symbols and raw event fields are not API-stable; re-validate `sky_click` after macOS upgrades.+The macOS runtime uses Accessibility, ScreenCaptureKit, app-posted input events, and explicit private-SkyLight `sky_click` / `sky_key` routes. It normally avoids moving the user's real pointer. The visual cursor overlay is part of the Open Computer Use experience and can be disabled by the surrounding runtime only when needed. Private SkyLight symbols and raw event fields are not API-stable; re-validate `sky_click` after macOS upgrades.### Windows