👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Add Linux Computer Use runtime
Leobdbcd33525b8
diff --git a/README.md b/README.mdindex 14dcbde..a1ac7f9 100644--- a/README.md+++ b/README.md@@ -8,7 +8,7 @@ https://github.com/user-attachments/assets/eacb3b15-f939-46c7-b3b3-6f876977a58d<sub><em>Gemini CLI connected to `open-computer-use` through MCP, driving real Computer Use actions end to end.</em></sub>-`open-computer-use` is an open-source `Computer Use` service exposed over `MCP`, so any AI agent or MCP client can call it directly and use computer interaction capabilities on macOS. An experimental Windows runtime now lives in this repo too, with the same 9-tool surface implemented as a standalone Go-built `.exe`.+`open-computer-use` is an open-source `Computer Use` service exposed over `MCP`, so any AI agent or MCP client can call it directly and use computer interaction capabilities on macOS. Experimental Windows and Linux runtimes now live in this repo too, each exposing the same 9-tool surface through standalone Go-built binaries.This project was inspired by OpenAI's recently released [Codex Computer Use](https://openai.com/index/codex-for-almost-everything/). It showed that non-intrusive CUA can be built on top of macOS Accessibility, which is why I decided to build an open-source version.@@ -91,6 +91,22 @@ Run the `.exe` in the signed-in desktop session. Running it as a Windows serviceBy default, the Windows runtime only attaches to already running apps, does not call `SetFocus`, and avoids the UIA `ValuePattern.SetValue` fallback for `type_text` because some apps bring themselves forward from that path. If you explicitly want the old foreground behavior, set `OPEN_COMPUTER_USE_WINDOWS_ALLOW_APP_LAUNCH=1` to allow app launch fallback, `OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOCUS_ACTIONS=1` to allow the `SetFocus` secondary action, and `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK=1` to allow UIA text fallback.+## Linux Runtime++The Linux runtime is also separate from the macOS Swift `.app`. It is built from `apps/OpenComputerUseLinux` and uses AT-SPI2 accessibility over the signed-in desktop session's D-Bus bus. Semantic actions, editable text, and value updates are preferred; coordinate mouse, drag, and key synthesis are best-effort fallbacks rather than a universal Wayland background input model.++```bash+# Build a Linux arm64 binary from this repo+./scripts/build-open-computer-use-linux.sh --arch arm64++# On Linux, run it in the signed-in desktop session+open-computer-use mcp+open-computer-use call list_apps+open-computer-use call --calls '[{"tool":"get_app_state","args":{"app":"gnome-text-editor"}},{"tool":"type_text","args":{"app":"gnome-text-editor","text":"hello"}}]'+```++The process needs the desktop user's `XDG_RUNTIME_DIR`, `DBUS_SESSION_BUS_ADDRESS`, and display environment. A pure SSH tty without those variables can build and launch the binary, but it cannot inspect or operate the GUI session. Screenshot capture is best-effort on GNOME Wayland and may be omitted when the compositor returns a black frame.+## Cursor MotionCursor Motion is an open-source cursor motion system for macOS, based on public information shared by members of the Software.Inc team. You can run it from source or download the app from the [Releases page](https://github.com/iFurySt/open-codex-computer-use/releases).diff --git a/README.zh-CN.md b/README.zh-CN.mdindex fa13e67..0185349 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -8,7 +8,7 @@ https://github.com/user-attachments/assets/eacb3b15-f939-46c7-b3b3-6f876977a58d<sub><em>Gemini CLI 作为 host 接入 `open-computer-use` MCP,并完整触发真实的 Computer Use 操作。</em></sub>-`open-computer-use` 是一个开源的 `Computer Use` 服务,已经包装成 `MCP` 协议,支持所有的 AI Agent 或 MCP Client 快速调用,实现 macOS 上的 `Computer Use` 能力。仓库里也已经新增实验性的 Windows runtime,用 Go 生成独立 `.exe`,暴露同样的 9 个 tool。+`open-computer-use` 是一个开源的 `Computer Use` 服务,已经包装成 `MCP` 协议,支持所有的 AI Agent 或 MCP Client 快速调用,实现 macOS 上的 `Computer Use` 能力。仓库里也已经新增实验性的 Windows 和 Linux runtime,用 Go 生成独立二进制,暴露同样的 9 个 tool。项目的背后是 OpenAI 刚发布的 [Codex Computer Use](https://openai.com/index/codex-for-almost-everything/),让我看到了基于 Accessibility 可以实现非抢占式 CUA 能力,因此决定复刻一个开源版本@@ -101,6 +101,22 @@ open-computer-use.exe call --calls "[{\"tool\":\"get_app_state\",\"args\":{\"app默认情况下,Windows runtime 只连接已经运行的 app,不会自动启动目标 app,也不会执行 `SetFocus`;`type_text` 也会避开 UIA `ValuePattern.SetValue` fallback,因为有些 app 会在这条路径里主动把自己带到前台。如果确实需要旧的前台行为,可以设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_APP_LAUNCH=1` 允许启动 fallback,设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOCUS_ACTIONS=1` 允许 `SetFocus` secondary action,设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK=1` 允许 UIA text fallback。+## Linux Runtime++Linux 侧也不复用 Swift `.app`,而是放在 `apps/OpenComputerUseLinux` 里独立构建;执行时走已登录桌面 session 的 AT-SPI2 / D-Bus accessibility。默认优先用语义化 action、editable text 和 value 接口;坐标鼠标、拖拽和键盘合成只是 best-effort fallback,不等价于一套通用的 Wayland 后台输入模型。++```bash+# 从仓库里构建 Linux arm64 binary+./scripts/build-open-computer-use-linux.sh --arch arm64++# 在 Linux 已登录桌面 session 里直接运行+open-computer-use mcp+open-computer-use call list_apps+open-computer-use call --calls '[{"tool":"get_app_state","args":{"app":"gnome-text-editor"}},{"tool":"type_text","args":{"app":"gnome-text-editor","text":"hello"}}]'+```++这个进程需要桌面用户的 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 和 display 环境。纯 SSH tty 可以构建和启动二进制,但不能直接 inspect 或操作 GUI session。GNOME Wayland 下截图是 best-effort,如果 compositor 返回黑图,Linux bridge 会省略 image block。+## Cursor MotionCursor Motion 是一个面向 macOS 的开源光标运动系统,基于 Software.Inc 几位大佬的公开信息实现的开源版本,可以代码里运行,也可以到 [Releases 页面](https://github.com/iFurySt/open-codex-computer-use/releases) 下载 app 运行。diff --git a/apps/OpenComputerUseLinux/go.mod b/apps/OpenComputerUseLinux/go.modnew file mode 100644index 0000000..5a944d8--- /dev/null+++ b/apps/OpenComputerUseLinux/go.mod@@ -0,0 +1,3 @@+module github.com/iFurySt/open-codex-computer-use/apps/opencomputeruselinux++go 1.22diff --git a/apps/OpenComputerUseLinux/main.go b/apps/OpenComputerUseLinux/main.gonew file mode 100644index 0000000..0aaa53d--- /dev/null+++ b/apps/OpenComputerUseLinux/main.go@@ -0,0 +1,984 @@+package main++import (+ "bytes"+ "context"+ _ "embed"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "os"+ "os/exec"+ "path/filepath"+ "runtime"+ "strconv"+ "strings"+ "time"+)++var version = "0.1.28"++//go:embed runtime.py+var linuxRuntimeScript string++const serverInstructions = "Computer Use tools let you interact with Linux desktop apps by performing UI actions.\n\nBegin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value.\n\nPrefer element-targeted interactions over coordinate clicks when an index for the targeted element is available. Linux actions use AT-SPI2 semantic actions and editable text APIs first. Coordinate mouse and key synthesis are best-effort fallbacks and are not a universal Wayland background input model."++type toolDefinition struct {+ Name string `json:"name"`+ Description string `json:"description"`+ Annotations map[string]any `json:"annotations,omitempty"`+ InputSchema map[string]any `json:"inputSchema"`+}++type contentItem struct {+ Type string `json:"type"`+ Text string `json:"text,omitempty"`+ Data string `json:"data,omitempty"`+ MimeType string `json:"mimeType,omitempty"`+}++type toolCallResult struct {+ Content []contentItem `json:"content"`+ IsError bool `json:"isError"`+}++func textResult(text string, isError bool) toolCallResult {+ return toolCallResult{Content: []contentItem{{Type: "text", Text: text}}, IsError: isError}+}++type appDescriptor struct {+ Name string `json:"name"`+ BundleIdentifier string `json:"bundleIdentifier,omitempty"`+ PID int `json:"pid"`+}++type frame struct {+ X float64 `json:"x"`+ Y float64 `json:"y"`+ Width float64 `json:"width"`+ Height float64 `json:"height"`+}++func (f frame) renderedLocalFrame() string {+ return fmt.Sprintf("{{x: %.0f, y: %.0f, width: %.0f, height: %.0f}}", f.X, f.Y, f.Width, f.Height)+}++type elementRecord struct {+ Index int `json:"index"`+ RuntimeID []int `json:"runtimeId,omitempty"`+ AutomationID string `json:"automationId,omitempty"`+ Name string `json:"name,omitempty"`+ ControlType string `json:"controlType,omitempty"`+ LocalizedControlType string `json:"localizedControlType,omitempty"`+ ClassName string `json:"className,omitempty"`+ Value string `json:"value,omitempty"`+ NativeWindowHandle int64 `json:"nativeWindowHandle,omitempty"`+ Frame *frame `json:"frame,omitempty"`+ Actions []string `json:"actions,omitempty"`+}++type appSnapshot struct {+ App appDescriptor `json:"app"`+ WindowTitle string `json:"windowTitle,omitempty"`+ WindowBounds *frame `json:"windowBounds,omitempty"`+ ScreenshotPNGBase64 string `json:"screenshotPngBase64,omitempty"`+ TreeLines []string `json:"treeLines,omitempty"`+ FocusedSummary string `json:"focusedSummary,omitempty"`+ SelectedText string `json:"selectedText,omitempty"`+ Elements []elementRecord `json:"elements,omitempty"`+}++func (s *appSnapshot) renderedText() string {+ if s == nil {+ return ""+ }+ appRef := s.App.BundleIdentifier+ if appRef == "" {+ appRef = s.App.Name+ }+ title := s.WindowTitle+ if strings.TrimSpace(title) == "" {+ title = s.App.Name+ }++ lines := []string{+ fmt.Sprintf("App=%s (pid %d)", appRef, s.App.PID),+ fmt.Sprintf("Window: %q, App: %s.", title, s.App.Name),+ }+ lines = append(lines, s.TreeLines...)+ if strings.TrimSpace(s.SelectedText) != "" {+ lines = append(lines, "", fmt.Sprintf("Selected text: [%s]", s.SelectedText))+ } else if strings.TrimSpace(s.FocusedSummary) != "" {+ lines = append(lines, "", fmt.Sprintf("The focused UI element is %s.", s.FocusedSummary))+ }+ return strings.Join(lines, "\n")+}++func (s *appSnapshot) result() toolCallResult {+ result := toolCallResult{+ Content: []contentItem{{Type: "text", Text: s.renderedText()}},+ }+ if s != nil && s.ScreenshotPNGBase64 != "" {+ result.Content = append(result.Content, contentItem{+ Type: "image",+ Data: s.ScreenshotPNGBase64,+ MimeType: "image/png",+ })+ }+ return result+}++type linuxRequest struct {+ Tool string `json:"tool"`+ App string `json:"app,omitempty"`+ Element *elementRecord `json:"element,omitempty"`+ X *float64 `json:"x,omitempty"`+ Y *float64 `json:"y,omitempty"`+ FromX *float64 `json:"from_x,omitempty"`+ FromY *float64 `json:"from_y,omitempty"`+ ToX *float64 `json:"to_x,omitempty"`+ ToY *float64 `json:"to_y,omitempty"`+ ClickCount int `json:"click_count,omitempty"`+ MouseButton string `json:"mouse_button,omitempty"`+ Action string `json:"action,omitempty"`+ Direction string `json:"direction,omitempty"`+ Pages float64 `json:"pages,omitempty"`+ Text string `json:"text,omitempty"`+ Key string `json:"key,omitempty"`+ Value string `json:"value,omitempty"`+ WindowBounds *frame `json:"windowBounds,omitempty"`+}++type linuxResponse struct {+ OK bool `json:"ok"`+ Text string `json:"text,omitempty"`+ Error string `json:"error,omitempty"`+ Snapshot *appSnapshot `json:"snapshot,omitempty"`+}++type service struct {+ snapshots map[string]*appSnapshot+}++func newService() *service {+ return &service{snapshots: map[string]*appSnapshot{}}+}++func (s *service) callTool(name string, args map[string]any) toolCallResult {+ switch name {+ case "list_apps":+ return s.listApps()+ case "get_app_state":+ return s.getAppState(requiredString(args, "app"))+ case "click":+ return s.click(+ requiredString(args, "app"),+ optionalString(args, "element_index"),+ optionalFloat(args, "x"),+ optionalFloat(args, "y"),+ intValue(optionalFloat(args, "click_count"), 1),+ defaultString(optionalString(args, "mouse_button"), "left"),+ )+ case "perform_secondary_action":+ return s.performSecondaryAction(+ requiredString(args, "app"),+ requiredString(args, "element_index"),+ requiredString(args, "action"),+ )+ case "scroll":+ return s.scroll(+ requiredString(args, "app"),+ requiredString(args, "direction"),+ requiredString(args, "element_index"),+ floatValue(optionalFloat(args, "pages"), 1),+ )+ case "drag":+ return s.drag(+ requiredString(args, "app"),+ requiredFloat(args, "from_x"),+ requiredFloat(args, "from_y"),+ requiredFloat(args, "to_x"),+ requiredFloat(args, "to_y"),+ )+ case "type_text":+ return s.typeText(requiredString(args, "app"), requiredString(args, "text"))+ case "press_key":+ return s.pressKey(requiredString(args, "app"), requiredString(args, "key"))+ case "set_value":+ return s.setValue(requiredString(args, "app"), requiredString(args, "element_index"), requiredString(args, "value"))+ default:+ return textResult(fmt.Sprintf("unsupportedTool(%q)", name), true)+ }+}++func (s *service) listApps() toolCallResult {+ response, err := runPython(linuxRequest{Tool: "list_apps"})+ if err != nil {+ return textResult(err.Error(), true)+ }+ if !response.OK {+ return textResult(response.Error, true)+ }+ if strings.TrimSpace(response.Text) == "" {+ response.Text = "No running top-level apps are visible to this Linux runtime."+ }+ return textResult(response.Text, false)+}++func (s *service) getAppState(app string) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ snapshot, result := s.refreshSnapshot(app, linuxRequest{Tool: "get_app_state", App: app})+ if result.IsError {+ return result+ }+ return snapshot.result()+}++func (s *service) click(app, elementIndex string, x, y *float64, clickCount int, mouseButton string) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if elementIndex == "" && (x == nil || y == nil) {+ return textResult("click requires either element_index or x/y", true)+ }+ snapshot := s.currentSnapshot(app)+ if snapshot == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ request := linuxRequest{+ Tool: "click",+ App: app,+ X: x,+ Y: y,+ ClickCount: clickCount,+ MouseButton: mouseButton,+ WindowBounds: snapshot.WindowBounds,+ }+ if elementIndex != "" {+ record, err := lookupElement(snapshot, elementIndex)+ if err != nil {+ return textResult(err.Error(), true)+ }+ request.Element = record+ }+ return s.actionResult(app, request)+}++func (s *service) performSecondaryAction(app, elementIndex, action string) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if elementIndex == "" {+ return textResult("Missing required argument: element_index", true)+ }+ if action == "" {+ return textResult("Missing required argument: action", true)+ }+ snapshot := s.currentSnapshot(app)+ if snapshot == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ record, err := lookupElement(snapshot, elementIndex)+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.actionResult(app, linuxRequest{Tool: "perform_secondary_action", App: app, Element: record, Action: action})+}++func (s *service) scroll(app, direction, elementIndex string, pages float64) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if elementIndex == "" {+ return textResult("Missing required argument: element_index", true)+ }+ normalized := strings.ToLower(direction)+ if normalized != "up" && normalized != "down" && normalized != "left" && normalized != "right" {+ return textResult("Invalid scroll direction: "+direction, true)+ }+ if pages <= 0 {+ return textResult("pages must be > 0", true)+ }+ snapshot := s.currentSnapshot(app)+ if snapshot == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ record, err := lookupElement(snapshot, elementIndex)+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.actionResult(app, linuxRequest{Tool: "scroll", App: app, Element: record, Direction: normalized, Pages: pages})+}++func (s *service) drag(app string, fromX, fromY, toX, toY *float64) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if fromX == nil {+ return textResult("Missing required argument: from_x", true)+ }+ if fromY == nil {+ return textResult("Missing required argument: from_y", true)+ }+ if toX == nil {+ return textResult("Missing required argument: to_x", true)+ }+ if toY == nil {+ return textResult("Missing required argument: to_y", true)+ }+ snapshot := s.currentSnapshot(app)+ if snapshot == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ return s.actionResult(app, linuxRequest{Tool: "drag", App: app, FromX: fromX, FromY: fromY, ToX: toX, ToY: toY, WindowBounds: snapshot.WindowBounds})+}++func (s *service) typeText(app, text string) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if text == "" {+ return textResult("Missing required argument: text", true)+ }+ if s.currentSnapshot(app) == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ return s.actionResult(app, linuxRequest{Tool: "type_text", App: app, Text: text})+}++func (s *service) pressKey(app, key string) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if key == "" {+ return textResult("Missing required argument: key", true)+ }+ if s.currentSnapshot(app) == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ return s.actionResult(app, linuxRequest{Tool: "press_key", App: app, Key: key})+}++func (s *service) setValue(app, elementIndex, value string) toolCallResult {+ if app == "" {+ return textResult("Missing required argument: app", true)+ }+ if elementIndex == "" {+ return textResult("Missing required argument: element_index", true)+ }+ snapshot := s.currentSnapshot(app)+ if snapshot == nil {+ return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)+ }+ record, err := lookupElement(snapshot, elementIndex)+ if err != nil {+ return textResult(err.Error(), true)+ }+ return s.actionResult(app, linuxRequest{Tool: "set_value", App: app, Element: record, Value: value})+}++func (s *service) actionResult(app string, request linuxRequest) toolCallResult {+ snapshot, result := s.refreshSnapshot(app, request)+ if result.IsError {+ return result+ }+ return snapshot.result()+}++func (s *service) currentSnapshot(app string) *appSnapshot {+ return s.snapshots[strings.ToLower(app)]+}++func (s *service) refreshSnapshot(app string, request linuxRequest) (*appSnapshot, toolCallResult) {+ response, err := runPython(request)+ if err != nil {+ return nil, textResult(err.Error(), true)+ }+ if !response.OK {+ return nil, textResult(response.Error, true)+ }+ if response.Snapshot == nil {+ return nil, textResult("Linux runtime did not return an app snapshot.", true)+ }+ s.rememberSnapshot(app, response.Snapshot)+ return response.Snapshot, toolCallResult{}+}++func (s *service) rememberSnapshot(query string, snapshot *appSnapshot) {+ keys := []string{query, snapshot.App.Name, snapshot.App.BundleIdentifier, strconv.Itoa(snapshot.App.PID)}+ for _, key := range keys {+ key = strings.ToLower(strings.TrimSpace(key))+ if key != "" {+ s.snapshots[key] = snapshot+ }+ }+}++func lookupElement(snapshot *appSnapshot, elementIndex string) (*elementRecord, error) {+ index, err := strconv.Atoi(elementIndex)+ if err != nil {+ return nil, fmt.Errorf("unknown element_index %q", elementIndex)+ }+ for _, record := range snapshot.Elements {+ if record.Index == index {+ copy := record+ return ©, nil+ }+ }+ return nil, fmt.Errorf("unknown element_index %q", elementIndex)+}++func runPython(request linuxRequest) (*linuxResponse, error) {+ if runtime.GOOS != "linux" {+ return nil, errors.New("Linux Computer Use runtime requires python3 on Linux")+ }++ tempDir, err := os.MkdirTemp("", "open-computer-use-linux-*")+ if err != nil {+ return nil, err+ }+ defer os.RemoveAll(tempDir)++ scriptPath := filepath.Join(tempDir, "runtime.py")+ operationPath := filepath.Join(tempDir, "operation.json")+ if err := os.WriteFile(scriptPath, []byte(linuxRuntimeScript), 0o600); err != nil {+ return nil, err+ }+ operationData, err := json.Marshal(request)+ if err != nil {+ return nil, err+ }+ if err := os.WriteFile(operationPath, operationData, 0o600); err != nil {+ return nil, err+ }++ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)+ defer cancel()++ cmd := exec.CommandContext(ctx, "python3", scriptPath, operationPath)+ var stderr bytes.Buffer+ cmd.Stderr = &stderr+ output, err := cmd.Output()+ if ctx.Err() == context.DeadlineExceeded {+ return nil, errors.New("Linux runtime timed out after 30s")+ }+ if err != nil {+ text := strings.TrimSpace(stderr.String())+ if text == "" {+ text = strings.TrimSpace(string(output))+ }+ if text == "" {+ text = err.Error()+ }+ return nil, fmt.Errorf("Linux runtime failed: %s", text)+ }++ var response linuxResponse+ if err := json.Unmarshal(output, &response); err != nil {+ return nil, fmt.Errorf("Linux runtime returned invalid JSON: %w: %s", err, strings.TrimSpace(string(output)))+ }+ return &response, nil+}++func requiredString(args map[string]any, key string) string {+ value, _ := args[key].(string)+ return strings.TrimSpace(value)+}++func optionalString(args map[string]any, key string) string {+ value, _ := args[key].(string)+ return value+}++func requiredFloat(args map[string]any, key string) *float64 {+ return optionalFloat(args, key)+}++func optionalFloat(args map[string]any, key string) *float64 {+ switch value := args[key].(type) {+ case float64:+ return &value+ case int:+ float := float64(value)+ return &float+ case json.Number:+ float, err := value.Float64()+ if err == nil {+ return &float+ }+ }+ return nil+}++func intValue(value *float64, fallback int) int {+ if value == nil {+ return fallback+ }+ return int(*value)+}++func floatValue(value *float64, fallback float64) float64 {+ if value == nil {+ return fallback+ }+ return *value+}++func defaultString(value, fallback string) string {+ if strings.TrimSpace(value) == "" {+ return fallback+ }+ return value+}++func toolDefinitions() []toolDefinition {+ return []toolDefinition{+ {+ Name: "click",+ Description: "Click an element by index or pixel coordinates from screenshot. This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "element_index": stringProperty("Element index to click"),+ "x": numberProperty("X coordinate in screenshot pixel coordinates"),+ "y": numberProperty("Y coordinate in screenshot pixel coordinates"),+ "click_count": integerProperty("Number of clicks. Defaults to 1"),+ "mouse_button": enumStringProperty("Mouse button to click. Defaults to left.", []string{"left", "right", "middle"}),+ }, []string{"app"}),+ },+ {+ Name: "drag",+ Description: "Drag from one point to another using pixel coordinates. This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "from_x": numberProperty("Start X coordinate"),+ "from_y": numberProperty("Start Y coordinate"),+ "to_x": numberProperty("End X coordinate"),+ "to_y": numberProperty("End Y coordinate"),+ }, []string{"app", "from_x", "from_y", "to_x", "to_y"}),+ },+ {+ Name: "get_app_state",+ Description: "Get the state of an already running app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`.",+ Annotations: readOnlyAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ }, []string{"app"}),+ },+ {+ Name: "list_apps",+ Description: "List the apps on this computer. Returns the set of apps that are currently running, as well as any that have been used in the last 14 days, including details on usage frequency. This tool is part of plugin `Computer Use`.",+ Annotations: readOnlyAnnotations(),+ InputSchema: objectSchema(map[string]any{}, nil),+ },+ {+ Name: "perform_secondary_action",+ Description: "Invoke a secondary accessibility action exposed by an element. This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "element_index": stringProperty("Element identifier"),+ "action": stringProperty("Secondary accessibility action name"),+ }, []string{"app", "element_index", "action"}),+ },+ {+ Name: "press_key",+ Description: "Press a key or key-combination on the keyboard, including modifier and navigation keys.\n - This supports xdotool's `key` syntax.\n - Examples: \"a\", \"Return\", \"Tab\", \"super+c\", \"Up\", \"KP_0\" (for the numpad 0). This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "key": stringProperty("Key or key-combination to press"),+ }, []string{"app", "key"}),+ },+ {+ Name: "scroll",+ Description: "Scroll an element in a direction by a number of pages. This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "direction": stringProperty("Scroll direction: up, down, left, or right"),+ "element_index": stringProperty("Element identifier"),+ "pages": numberProperty("Number of pages to scroll. Fractional values are supported. Defaults to 1"),+ }, []string{"app", "element_index", "direction"}),+ },+ {+ Name: "set_value",+ Description: "Set the value of a settable accessibility element. This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "element_index": stringProperty("Element identifier"),+ "value": stringProperty("Value to assign"),+ }, []string{"app", "element_index", "value"}),+ },+ {+ Name: "type_text",+ Description: "Type literal text using keyboard input. This tool is part of plugin `Computer Use`.",+ Annotations: defaultAnnotations(),+ InputSchema: objectSchema(map[string]any{+ "app": stringProperty("App name or bundle identifier"),+ "text": stringProperty("Literal text to type"),+ }, []string{"app", "text"}),+ },+ }+}++func objectSchema(properties map[string]any, required []string) map[string]any {+ schema := map[string]any{+ "type": "object",+ "properties": properties,+ "additionalProperties": false,+ }+ if len(required) > 0 {+ schema["required"] = required+ }+ return schema+}++func defaultAnnotations() map[string]any {+ return map[string]any{"destructiveHint": false, "openWorldHint": false}+}++func readOnlyAnnotations() map[string]any {+ return map[string]any{"destructiveHint": false, "idempotentHint": true, "openWorldHint": false, "readOnlyHint": true}+}++func stringProperty(description string) map[string]any {+ return map[string]any{"type": "string", "description": description}+}++func enumStringProperty(description string, values []string) map[string]any {+ property := stringProperty(description)+ property["enum"] = values+ return property+}++func numberProperty(description string) map[string]any {+ return map[string]any{"type": "number", "description": description}+}++func integerProperty(description string) map[string]any {+ return map[string]any{"type": "integer", "description": description}+}++func main() {+ if err := runCLI(os.Args[1:], os.Stdout); err != nil {+ fmt.Fprintln(os.Stderr, err)+ os.Exit(1)+ }+}++func runCLI(args []string, stdout io.Writer) error {+ if len(args) == 0 {+ fmt.Fprint(stdout, helpText(""))+ return nil+ }++ switch args[0] {+ case "-h", "--help", "help":+ topic := ""+ if len(args) > 1 {+ topic = args[1]+ }+ fmt.Fprint(stdout, helpText(topic))+ return nil+ case "-v", "--version", "version":+ fmt.Fprintln(stdout, version)+ return nil+ case "mcp":+ return runMCP(os.Stdin, stdout)+ case "doctor":+ fmt.Fprintln(stdout, "Linux runtime: AT-SPI2 and GDK are available when this process runs in the signed-in desktop session with XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS set.")+ return nil+ case "list-apps":+ result := newService().callTool("list_apps", map[string]any{})+ if result.IsError {+ return errors.New(result.Content[0].Text)+ }+ fmt.Fprintln(stdout, result.Content[0].Text)+ return nil+ case "snapshot":+ if len(args) != 2 {+ return errors.New("snapshot requires an app name, process name, window title, or pid")+ }+ result := newService().callTool("get_app_state", map[string]any{"app": args[1]})+ if result.IsError {+ return errors.New(result.Content[0].Text)+ }+ fmt.Fprintln(stdout, result.Content[0].Text)+ return nil+ case "call":+ output, hasError, err := runCallCommand(args[1:], newService())+ if err != nil {+ return err+ }+ encoded, err := json.MarshalIndent(output, "", " ")+ if err != nil {+ return err+ }+ fmt.Fprintln(stdout, string(encoded))+ if hasError {+ return errors.New("tool call returned isError=true")+ }+ return nil+ default:+ return fmt.Errorf("unknown command: %s\n\n%s", args[0], helpText(""))+ }+}++func runCallCommand(args []string, svc *service) (any, bool, error) {+ if len(args) == 0 {+ return nil, false, errors.New("call requires a tool name or --calls/--calls-file")+ }++ var toolName, argsJSON, argsFile, callsJSON, callsFile string+ for index := 0; index < len(args); index++ {+ arg := args[index]+ switch arg {+ case "--args":+ index+++ if index >= len(args) {+ return nil, false, errors.New("--args requires a value")+ }+ argsJSON = args[index]+ case "--args-file":+ index+++ if index >= len(args) {+ return nil, false, errors.New("--args-file requires a value")+ }+ argsFile = args[index]+ case "--calls":+ index+++ if index >= len(args) {+ return nil, false, errors.New("--calls requires a value")+ }+ callsJSON = args[index]+ case "--calls-file":+ index+++ if index >= len(args) {+ return nil, false, errors.New("--calls-file requires a value")+ }+ callsFile = args[index]+ default:+ if strings.HasPrefix(arg, "-") {+ return nil, false, fmt.Errorf("unknown call option: %s", arg)+ }+ if toolName != "" {+ return nil, false, errors.New("call accepts at most one tool name")+ }+ toolName = arg+ }+ }++ if callsJSON != "" || callsFile != "" {+ if toolName != "" || argsJSON != "" || argsFile != "" {+ return nil, false, errors.New("call sequence does not accept a tool name, --args, or --args-file")+ }+ calls, err := readCallSequence(callsJSON, callsFile)+ if err != nil {+ return nil, false, err+ }+ var outputs []map[string]any+ hasError := false+ for _, call := range calls {+ result := svc.callTool(call.Tool, call.Args)+ outputs = append(outputs, map[string]any{"tool": call.Tool, "result": result})+ if result.IsError {+ hasError = true+ break+ }+ }+ return outputs, hasError, nil+ }++ if toolName == "" {+ return nil, false, errors.New("call requires a tool name or --calls/--calls-file")+ }+ arguments, err := readArguments(argsJSON, argsFile)+ if err != nil {+ return nil, false, err+ }+ result := svc.callTool(toolName, arguments)+ return result, result.IsError, nil+}++type callSpec struct {+ Tool string+ Args map[string]any+}++func readArguments(inline, file string) (map[string]any, error) {+ if inline != "" && file != "" {+ return nil, errors.New("Use either inline JSON or a JSON file, not both")+ }+ if inline == "" && file == "" {+ return map[string]any{}, nil+ }+ source, err := readJSONSource(inline, file)+ if err != nil {+ return nil, err+ }+ var args map[string]any+ decoder := json.NewDecoder(strings.NewReader(source))+ decoder.UseNumber()+ if err := decoder.Decode(&args); err != nil {+ return nil, fmt.Errorf("Invalid JSON input: %w", err)+ }+ if args == nil {+ return nil, errors.New("--args must be a JSON object")+ }+ return args, nil+}++func readCallSequence(inline, file string) ([]callSpec, error) {+ if inline != "" && file != "" {+ return nil, errors.New("Use either --calls or --calls-file, not both")+ }+ source, err := readJSONSource(inline, file)+ if err != nil {+ return nil, err+ }+ var raw []map[string]any+ decoder := json.NewDecoder(strings.NewReader(source))+ decoder.UseNumber()+ if err := decoder.Decode(&raw); err != nil {+ return nil, fmt.Errorf("Invalid JSON input: %w", err)+ }+ calls := make([]callSpec, 0, len(raw))+ for index, item := range raw {+ name, _ := item["tool"].(string)+ if name == "" {+ name, _ = item["name"].(string)+ }+ if name == "" {+ return nil, fmt.Errorf("call sequence item #%d requires a non-empty tool", index+1)+ }+ args, _ := item["args"].(map[string]any)+ if args == nil {+ args, _ = item["arguments"].(map[string]any)+ }+ if args == nil {+ args = map[string]any{}+ }+ calls = append(calls, callSpec{Tool: name, Args: args})+ }+ return calls, nil+}++func readJSONSource(inline, file string) (string, error) {+ if inline != "" {+ return inline, nil+ }+ if file == "" {+ return "", errors.New("JSON input is required")+ }+ data, err := os.ReadFile(file)+ if err != nil {+ return "", err+ }+ return string(data), nil+}++func runMCP(stdin io.Reader, stdout io.Writer) error {+ svc := newService()+ decoder := json.NewDecoder(stdin)+ encoder := json.NewEncoder(stdout)+ for {+ var request map[string]any+ if err := decoder.Decode(&request); err != nil {+ if errors.Is(err, io.EOF) {+ return nil+ }+ _ = encoder.Encode(jsonRPCError(nil, -32700, "Invalid JSON-RPC payload"))+ continue+ }+ response := handleMCPRequest(request, svc)+ if response != nil {+ if err := encoder.Encode(response); err != nil {+ return err+ }+ }+ }+}++func handleMCPRequest(request map[string]any, svc *service) map[string]any {+ id := request["id"]+ method, _ := request["method"].(string)+ params, _ := request["params"].(map[string]any)+ switch method {+ case "initialize":+ return jsonRPCResult(id, map[string]any{+ "protocolVersion": "2025-03-26",+ "serverInfo": map[string]any{+ "name": "open-computer-use",+ "version": version,+ },+ "capabilities": map[string]any{"tools": map[string]any{"listChanged": false}},+ "instructions": serverInstructions,+ })+ case "notifications/initialized", "notifications/turn-ended":+ return nil+ case "ping":+ return jsonRPCResult(id, map[string]any{})+ case "tools/list":+ return jsonRPCResult(id, map[string]any{"tools": toolDefinitions()})+ case "tools/call":+ name, _ := params["name"].(string)+ arguments, _ := params["arguments"].(map[string]any)+ if arguments == nil {+ arguments = map[string]any{}+ }+ return jsonRPCResult(id, svc.callTool(name, arguments))+ default:+ if method == "" {+ return nil+ }+ return jsonRPCError(id, -32601, "Method not found: "+method)+ }+}++func jsonRPCResult(id any, result any) map[string]any {+ return map[string]any{"jsonrpc": "2.0", "id": id, "result": result}+}++func jsonRPCError(id any, code int, message string) map[string]any {+ return map[string]any{+ "jsonrpc": "2.0",+ "id": id,+ "error": map[string]any{"code": code, "message": message},+ }+}++func helpText(command string) string {+ switch command {+ case "mcp":+ return "Usage:\n open-computer-use mcp\n\nStart the stdio MCP server.\n"+ case "call":+ return "Usage:\n open-computer-use call <tool> [--args '<json-object>']\n open-computer-use call --calls '<json-array>'\n\nThe JSON array form keeps all calls in one process so element_index state can be reused.\n"+ case "snapshot":+ return "Usage:\n open-computer-use snapshot <app>\n\nPrint the current Linux AT-SPI snapshot for the target app.\n"+ default:+ return `Open Computer Use for Linux++Usage:+ open-computer-use [command] [options]++Commands:+ mcp Start the stdio MCP server.+ doctor Print Linux runtime notes.+ list-apps Print running apps with top-level windows.+ snapshot <app> Print the current AT-SPI snapshot for an app.+ call <tool> Call one tool, or run a JSON array of tool calls.+ help [command] Show general or command-specific help.+ version Print the CLI version.++Notes:+ The Linux runtime uses AT-SPI2 semantic actions first, then best-effort+ coordinate/key synthesis. Run it in the signed-in desktop session.+`+ }+}diff --git a/apps/OpenComputerUseLinux/main_test.go b/apps/OpenComputerUseLinux/main_test.gonew file mode 100644index 0000000..516d635--- /dev/null+++ b/apps/OpenComputerUseLinux/main_test.go@@ -0,0 +1,87 @@+package main++import (+ "bytes"+ "encoding/json"+ "strings"+ "testing"+)++func TestToolDefinitionCount(t *testing.T) {+ if got := len(toolDefinitions()); got != 9 {+ t.Fatalf("toolDefinitions() count = %d, want 9", got)+ }+}++func TestCallSequenceStopsAfterFirstToolError(t *testing.T) {+ output, hasError, err := runCallCommand([]string{+ "--calls",+ `[{"tool":"not_a_tool"},{"tool":"list_apps"}]`,+ }, newService())+ if err != nil {+ t.Fatal(err)+ }+ if !hasError {+ t.Fatal("expected hasError")+ }+ items, ok := output.([]map[string]any)+ if !ok {+ t.Fatalf("output type = %T", output)+ }+ if len(items) != 1 {+ t.Fatalf("sequence output count = %d, want 1", len(items))+ }+}++func TestReadArgumentsAcceptsJSONObject(t *testing.T) {+ args, err := readArguments(`{"app":"Text Editor","pages":2}`, "")+ if err != nil {+ t.Fatal(err)+ }+ if args["app"] != "Text Editor" {+ t.Fatalf("app = %v", args["app"])+ }+ if args["pages"].(json.Number).String() != "2" {+ t.Fatalf("pages = %v", args["pages"])+ }+}++func TestMCPInitializeResponseContainsToolsCapability(t *testing.T) {+ request := map[string]any{+ "jsonrpc": "2.0",+ "id": float64(1),+ "method": "initialize",+ "params": map[string]any{},+ }+ response := handleMCPRequest(request, newService())+ result, ok := response["result"].(map[string]any)+ if !ok {+ t.Fatalf("missing result: %#v", response)+ }+ capabilities := result["capabilities"].(map[string]any)+ if _, ok := capabilities["tools"]; !ok {+ t.Fatalf("missing tools capability: %#v", capabilities)+ }+}++func TestCLIHelpMentionsLinuxRuntime(t *testing.T) {+ var out bytes.Buffer+ if err := runCLI([]string{"--help"}, &out); err != nil {+ t.Fatal(err)+ }+ if !strings.Contains(out.String(), "Open Computer Use for Linux") {+ t.Fatalf("help text did not mention Linux runtime:\n%s", out.String())+ }+}++func TestLinuxRuntimeDocumentsATSPIAndFallbackBoundary(t *testing.T) {+ if !strings.Contains(linuxRuntimeScript, "Atspi") {+ t.Fatal("Linux runtime must use AT-SPI")+ }+ if !strings.Contains(linuxRuntimeScript, "generate_mouse_event") {+ t.Fatal("Linux runtime should keep coordinate input explicit and visible in the bridge")+ }+ if !strings.Contains(serverInstructions, "not a universal Wayland background input model") {+ t.Fatal("MCP instructions must document the Linux background-input boundary")+ }+}diff --git a/apps/OpenComputerUseLinux/runtime.py b/apps/OpenComputerUseLinux/runtime.pynew file mode 100644index 0000000..24bb994--- /dev/null+++ b/apps/OpenComputerUseLinux/runtime.py@@ -0,0 +1,801 @@+#!/usr/bin/env python3++import base64+import json+import math+import os+import sys+import time+import traceback+import warnings++warnings.filterwarnings("ignore", category=DeprecationWarning)++import gi++gi.require_version("Atspi", "2.0")++try:+ gi.require_version("Gdk", "3.0")+ from gi.repository import Gdk+except (ImportError, ValueError):+ Gdk = None++from gi.repository import Atspi+++MAX_ELEMENTS = 500+MAX_DEPTH = 64+++def frame(x, y, width, height):+ if width is None or height is None or width < 0 or height < 0:+ return None+ return {+ "x": float(x),+ "y": float(y),+ "width": float(width),+ "height": float(height),+ }+++def safe(call, default=None):+ try:+ value = call()+ if value is None:+ return default+ return value+ except Exception:+ return default+++def require_desktop_session():+ missing = []+ if not os.environ.get("XDG_RUNTIME_DIR"):+ missing.append("XDG_RUNTIME_DIR")+ if not os.environ.get("DBUS_SESSION_BUS_ADDRESS"):+ missing.append("DBUS_SESSION_BUS_ADDRESS")+ if missing:+ raise RuntimeError(+ "Linux runtime requires an active desktop session; missing "+ + ", ".join(missing)+ )+++def desktop():+ return Atspi.get_desktop(0)+++def child_count(node):+ return int(safe(node.get_child_count, 0) or 0)+++def child_at(node, index):+ return safe(lambda: node.get_child_at_index(index))+++def node_name(node):+ return str(safe(node.get_name, "") or "")+++def node_role(node):+ return str(safe(node.get_role_name, "") or "")+++def node_pid(node):+ value = safe(node.get_process_id, 0)+ try:+ return int(value or 0)+ except Exception:+ return 0+++def state_contains(node, state):+ state_set = safe(node.get_state_set)+ if state_set is None:+ return False+ return bool(safe(lambda: state_set.contains(state), False))+++def extents(node):+ component = safe(node.get_component_iface)+ if component is None:+ return None+ rect = safe(lambda: Atspi.Component.get_extents(component, Atspi.CoordType.SCREEN))+ if (+ rect is None+ or rect.width <= 0+ or rect.height <= 0+ or rect.width > 100000+ or rect.height > 100000+ ):+ return None+ return frame(rect.x, rect.y, rect.width, rect.height)+++def relative_frame(node, window_bounds):+ bounds = extents(node)+ if bounds is None:+ return None+ if window_bounds is None:+ return bounds+ return frame(+ bounds["x"] - window_bounds["x"],+ bounds["y"] - window_bounds["y"],+ bounds["width"],+ bounds["height"],+ )+++def iter_apps():+ root = desktop()+ apps = []+ for index in range(child_count(root)):+ app = child_at(root, index)+ if app is not None and node_name(app):+ apps.append(app)+ return apps+++def app_windows(app):+ windows = []+ for index in range(child_count(app)):+ child = child_at(app, index)+ if child is None:+ continue+ role = node_role(child).lower()+ bounds = extents(child)+ if role in {"frame", "window", "dialog", "alert"} or bounds is not None:+ windows.append((index, child))+ return windows+++def main_window(app):+ windows = app_windows(app)+ if not windows:+ raise RuntimeError(+ "No top-level AT-SPI window is available for " + node_name(app)+ )+ for index, window in windows:+ if state_contains(window, Atspi.StateType.ACTIVE):+ return index, window+ for index, window in windows:+ if state_contains(window, Atspi.StateType.SHOWING):+ return index, window+ return windows[0]+++def matches_query(app, query):+ normalized = query.strip().lower()+ if not normalized:+ return False+ if normalized.isdigit() and node_pid(app) == int(normalized):+ return True+ app_name = node_name(app).lower()+ if app_name == normalized or normalized in app_name:+ return True+ for _, window in app_windows(app):+ title = node_name(window).lower()+ if title == normalized or normalized in title:+ return True+ return False+++def resolve_app(query):+ for app in iter_apps():+ if matches_query(app, query):+ return app+ raise RuntimeError('appNotFound("{}")'.format(query))+++def action_names(node):+ names = []+ count = int(safe(node.get_n_actions, 0) or 0)+ for index in range(count):+ name = str(safe(lambda i=index: node.get_action_name(i), "") or "")+ description = str(+ safe(lambda i=index: node.get_action_description(i), "") or ""+ )+ label = name or description+ if label and label not in names:+ names.append(label)+ return names+++def accessible_id(node):+ return str(safe(node.get_accessible_id, "") or "")+++def text_value(node):+ if not bool(safe(node.is_text, False)):+ return ""+ text_iface = safe(node.get_text_iface)+ if text_iface is None:+ return ""+ count = int(safe(lambda: Atspi.Text.get_character_count(text_iface), 0) or 0)+ if count <= 0:+ return ""+ value = str(safe(lambda: Atspi.Text.get_text(text_iface, 0, min(count, 500)), "") or "")+ if count > 500:+ return value + "..."+ return value+++def numeric_value(node):+ value_iface = safe(node.get_value_iface)+ if value_iface is None:+ return ""+ current = safe(lambda: Atspi.Value.get_current_value(value_iface))+ if current is None:+ return ""+ return str(current)+++def element_value(node):+ return text_value(node) or numeric_value(node)+++def record_for(node, index, path, window_bounds):+ bounds = relative_frame(node, window_bounds)+ role = node_role(node)+ return {+ "index": index,+ "runtimeId": path[:],+ "automationId": accessible_id(node),+ "name": node_name(node),+ "controlType": role,+ "localizedControlType": role,+ "className": str(safe(node.get_toolkit_name, "") or ""),+ "value": element_value(node),+ "nativeWindowHandle": 0,+ "frame": bounds,+ "actions": action_names(node),+ }+++def render_tree(root, window_bounds, root_path):+ records = []+ lines = []++ def visit(node, depth, path):+ if len(records) >= MAX_ELEMENTS or depth > MAX_DEPTH or node is None:+ return+ index = len(records)+ record = record_for(node, index, path, window_bounds)+ records.append(record)++ role = record["localizedControlType"] or record["controlType"] or "element"+ title = record["name"] or record["automationId"] or ""+ value_segment = ""+ if record["value"] and record["value"] != title:+ safe_value = record["value"].replace("\r", "\\r").replace("\n", "\\n")+ value_segment = " Value: " + safe_value+ actions_segment = ""+ if record["actions"]:+ actions_segment = " Secondary Actions: " + ", ".join(record["actions"])+ frame_segment = ""+ if record["frame"] is not None:+ f = record["frame"]+ frame_segment = " Frame: {{x: {0}, y: {1}, width: {2}, height: {3}}}".format(+ round(f["x"]),+ round(f["y"]),+ round(f["width"]),+ round(f["height"]),+ )+ lines.append(+ ("\t" * (depth + 1))+ + "{} {} {}{}{}{}".format(+ index, role, title, value_segment, actions_segment, frame_segment+ ).rstrip()+ )++ for child_index in range(child_count(node)):+ child = child_at(node, child_index)+ visit(child, depth + 1, path + [child_index])++ visit(root, 0, root_path)+ return records, lines+++def capture_window_png(bounds):+ if Gdk is None or bounds is None:+ return None+ try:+ screen = Gdk.Screen.get_default()+ if screen is None:+ return None+ root = screen.get_root_window()+ pixbuf = Gdk.pixbuf_get_from_window(+ root,+ int(round(bounds["x"])),+ int(round(bounds["y"])),+ max(1, int(round(bounds["width"]))),+ max(1, int(round(bounds["height"]))),+ )+ if pixbuf is None:+ return None+ if pixbuf_looks_black(pixbuf):+ return None+ ok, data = pixbuf.save_to_bufferv("png", [], [])+ if not ok:+ return None+ return base64.b64encode(bytes(data)).decode("ascii")+ except Exception:+ return None+++def pixbuf_looks_black(pixbuf):+ try:+ pixels = pixbuf.get_pixels()+ channels = pixbuf.get_n_channels()+ rowstride = pixbuf.get_rowstride()+ width = pixbuf.get_width()+ height = pixbuf.get_height()+ if width <= 0 or height <= 0 or channels < 3:+ return True+ step_x = max(1, width // 16)+ step_y = max(1, height // 16)+ checked = 0+ for y in range(0, height, step_y):+ row = y * rowstride+ for x in range(0, width, step_x):+ offset = row + (x * channels)+ if (+ pixels[offset] > 3+ or pixels[offset + 1] > 3+ or pixels[offset + 2] > 3+ ):+ return False+ checked += 1+ return checked > 0+ except Exception:+ return False+++def focused_summary(app_pid):+ try:+ root = desktop()+ for app in iter_apps():+ if node_pid(app) != app_pid:+ continue+ _, win = main_window(app)+ focused = find_first(+ win, lambda node: state_contains(node, Atspi.StateType.FOCUSED)+ )+ if focused is None:+ return None+ role = node_role(focused)+ name = node_name(focused)+ return (role + " " + name).strip()+ except Exception:+ return None+++def selected_text(app_pid):+ try:+ for app in iter_apps():+ if node_pid(app) != app_pid:+ continue+ _, win = main_window(app)+ focused = find_first(+ win, lambda node: state_contains(node, Atspi.StateType.FOCUSED)+ )+ if focused is None or not bool(safe(focused.is_text, False)):+ return None+ text_iface = safe(focused.get_text_iface)+ selections = safe(lambda: Atspi.Text.get_text_selections(text_iface), [])+ if selections:+ selection = selections[0]+ return Atspi.Text.get_text(+ text_iface, selection.start_offset, selection.end_offset+ )+ except Exception:+ return None+ return None+++def build_snapshot(query):+ app = resolve_app(query)+ window_index, window = main_window(app)+ bounds = extents(window)+ records, lines = render_tree(window, bounds, [window_index])+ pid = node_pid(app)+ return {+ "app": {+ "name": node_name(app),+ "bundleIdentifier": node_name(app),+ "pid": pid,+ },+ "windowTitle": node_name(window),+ "windowBounds": bounds,+ "screenshotPngBase64": capture_window_png(bounds),+ "treeLines": lines,+ "focusedSummary": focused_summary(pid),+ "selectedText": selected_text(pid),+ "elements": records,+ }+++def list_apps_text():+ lines = []+ for app in sorted(iter_apps(), key=lambda item: (node_name(item).lower(), node_pid(item))):+ windows = app_windows(app)+ if not windows:+ continue+ title = node_name(windows[0][1]) or "untitled"+ name = node_name(app)+ lines.append(+ "{} -- {} [running, pid={}, window={}]".format(+ name, name, node_pid(app), title+ )+ )+ return "\n".join(lines)+++def find_first(root, predicate):+ if root is None:+ return None+ if predicate(root):+ return root+ for index in range(child_count(root)):+ found = find_first(child_at(root, index), predicate)+ if found is not None:+ return found+ return None+++def iter_all(root):+ items = []++ def visit(node):+ if node is None or len(items) >= MAX_ELEMENTS:+ return+ items.append(node)+ for index in range(child_count(node)):+ visit(child_at(node, index))++ visit(root)+ return items+++def resolve_path(app, path):+ if not path:+ return None+ node = app+ for index in path:+ node = child_at(node, int(index))+ if node is None:+ return None+ return node+++def same_frame(record_frame, node_frame):+ if record_frame is None or node_frame is None:+ return False+ for key in ("x", "y", "width", "height"):+ if abs(float(record_frame.get(key, 0)) - float(node_frame.get(key, 0))) > 3:+ return False+ return True+++def find_element(app, record):+ if not record:+ return None+ node = resolve_path(app, record.get("runtimeId") or [])+ if node is not None:+ return node++ _, window = main_window(app)+ target_name = str(record.get("name") or "")+ target_id = str(record.get("automationId") or "")+ target_role = str(record.get("controlType") or "")+ window_bounds = extents(window)+ for candidate in iter_all(window):+ if target_id and accessible_id(candidate) == target_id:+ return candidate+ if target_name and node_name(candidate) == target_name and node_role(candidate) == target_role:+ return candidate+ if target_role and node_role(candidate) == target_role:+ if same_frame(record.get("frame"), relative_frame(candidate, window_bounds)):+ return candidate+ return None+++def preferred_action_index(node):+ preferred_exact = {+ "click",+ "press",+ "activate",+ "default.activate",+ "invoke",+ "select",+ "toggle",+ "open",+ }+ count = int(safe(node.get_n_actions, 0) or 0)+ fallback = None+ for index in range(count):+ name = str(safe(lambda i=index: node.get_action_name(i), "") or "")+ description = str(safe(lambda i=index: node.get_action_description(i), "") or "")+ lower = (name or description).lower()+ if lower in preferred_exact:+ return index+ if fallback is None and (+ "activate" in lower or "click" in lower or "press" in lower+ ):+ fallback = index+ return fallback+++def do_action_by_index(node, index):+ if index is None:+ return False+ return bool(safe(lambda: node.do_action(int(index)), False))+++def screen_point(window_bounds, element=None, x=None, y=None):+ if element is not None:+ f = element.get("frame")+ if f is not None and window_bounds is not None:+ return (+ window_bounds["x"] + f["x"] + f["width"] / 2,+ window_bounds["y"] + f["y"] + f["height"] / 2,+ )+ if x is None or y is None or window_bounds is None:+ raise RuntimeError("coordinate action requires window bounds and x/y")+ return window_bounds["x"] + float(x), window_bounds["y"] + float(y)+++def mouse_button_events(button):+ normalized = (button or "left").lower()+ if normalized == "right":+ return "b3p", "b3r"+ if normalized == "middle":+ return "b2p", "b2r"+ return "b1p", "b1r"+++def send_mouse_click(x, y, button, count):+ down, up = mouse_button_events(button)+ repeat = max(1, int(count or 1))+ for _ in range(repeat):+ Atspi.generate_mouse_event(int(round(x)), int(round(y)), "abs")+ Atspi.generate_mouse_event(int(round(x)), int(round(y)), down)+ time.sleep(0.035)+ Atspi.generate_mouse_event(int(round(x)), int(round(y)), up)+ time.sleep(0.05)+++def send_drag(from_x, from_y, to_x, to_y):+ Atspi.generate_mouse_event(int(round(from_x)), int(round(from_y)), "abs")+ Atspi.generate_mouse_event(int(round(from_x)), int(round(from_y)), "b1p")+ steps = 12+ for step in range(1, steps + 1):+ x = from_x + ((to_x - from_x) * step / steps)+ y = from_y + ((to_y - from_y) * step / steps)+ Atspi.generate_mouse_event(int(round(x)), int(round(y)), "abs")+ time.sleep(0.02)+ Atspi.generate_mouse_event(int(round(to_x)), int(round(to_y)), "b1r")+++KEY_ALIASES = {+ "return": "Return",+ "enter": "Return",+ "tab": "Tab",+ "escape": "Escape",+ "esc": "Escape",+ "backspace": "BackSpace",+ "back_space": "BackSpace",+ "delete": "Delete",+ "space": "space",+ "left": "Left",+ "up": "Up",+ "right": "Right",+ "down": "Down",+ "home": "Home",+ "end": "End",+ "page_up": "Page_Up",+ "prior": "Page_Up",+ "page_down": "Page_Down",+ "next": "Page_Down",+}++MODIFIER_KEYS = {+ "ctrl": "Control_L",+ "control": "Control_L",+ "shift": "Shift_L",+ "alt": "Alt_L",+ "super": "Super_L",+ "win": "Super_L",+ "cmd": "Super_L",+}+++def keyval(name):+ if Gdk is not None:+ value = Gdk.keyval_from_name(name)+ if value:+ return int(value)+ if len(name) == 1:+ return ord(name)+ raise RuntimeError("Unsupported key: " + name)+++def send_key(key):+ parts = [part for part in str(key).split("+") if part]+ if not parts:+ raise RuntimeError("Unsupported key: " + str(key))+ main = parts[-1]+ modifiers = parts[:-1]+ pressed = []+ for modifier in modifiers:+ name = MODIFIER_KEYS.get(modifier.lower())+ if name is None:+ continue+ value = keyval(name)+ Atspi.generate_keyboard_event(value, None, Atspi.KeySynthType.PRESS)+ pressed.append(value)+ normalized = KEY_ALIASES.get(main.lower(), main)+ if len(normalized) == 1:+ Atspi.generate_keyboard_event(0, normalized, Atspi.KeySynthType.STRING)+ else:+ Atspi.generate_keyboard_event(+ keyval(normalized), None, Atspi.KeySynthType.PRESSRELEASE+ )+ for value in reversed(pressed):+ Atspi.generate_keyboard_event(value, None, Atspi.KeySynthType.RELEASE)+++def send_text(text):+ Atspi.generate_keyboard_event(0, str(text), Atspi.KeySynthType.STRING)+++def find_editable_text(root):+ def is_editable(node):+ return bool(safe(node.is_editable_text, False)) and bool(+ safe(node.is_text, False)+ )++ return find_first(root, is_editable)+++def insert_text(root, text):+ node = find_editable_text(root)+ if node is None:+ return False+ editable = safe(node.get_editable_text_iface)+ text_iface = safe(node.get_text_iface)+ if editable is None or text_iface is None:+ return False+ offset = int(safe(lambda: Atspi.Text.get_character_count(text_iface), 0) or 0)+ return bool(+ safe(+ lambda: Atspi.EditableText.insert_text(+ editable, offset, str(text), len(str(text))+ ),+ False,+ )+ )+++def set_element_value(node, value):+ if node is not None and bool(safe(node.is_editable_text, False)):+ editable = safe(node.get_editable_text_iface)+ if editable is not None:+ return bool(+ safe(+ lambda: Atspi.EditableText.set_text_contents(editable, str(value)),+ False,+ )+ )+ value_iface = safe(node.get_value_iface) if node is not None else None+ if value_iface is not None:+ try:+ return bool(Atspi.Value.set_current_value(value_iface, float(value)))+ except Exception:+ pass+ return False+++def invoke_secondary_action(node, action):+ if node is None:+ raise RuntimeError("unknown element_index")+ normalized = str(action).lower()+ count = int(safe(node.get_n_actions, 0) or 0)+ for index in range(count):+ name = str(safe(lambda i=index: node.get_action_name(i), "") or "")+ description = str(safe(lambda i=index: node.get_action_description(i), "") or "")+ if normalized in {name.lower(), description.lower()}:+ if do_action_by_index(node, index):+ return+ break+ raise RuntimeError("{} is not a valid secondary action for element".format(action))+++def scroll_element(direction, pages):+ key = "Page_Down"+ if direction == "up":+ key = "Page_Up"+ elif direction == "left":+ key = "Left"+ elif direction == "right":+ key = "Right"+ repeat = max(1, int(math.ceil(float(pages or 1))))+ for _ in range(repeat):+ send_key(key)+ time.sleep(0.04)+++def perform_operation(operation):+ tool = operation.get("tool")+ if tool == "list_apps":+ return {"ok": True, "text": list_apps_text()}+ if tool == "get_app_state":+ return {"ok": True, "snapshot": build_snapshot(operation.get("app", ""))}++ app = resolve_app(operation.get("app", ""))+ _, window = main_window(app)+ bounds = operation.get("windowBounds") or extents(window)+ element_record = operation.get("element")+ element = find_element(app, element_record)++ if tool == "click":+ handled = False+ if element is not None and operation.get("mouse_button", "left") == "left":+ handled = do_action_by_index(element, preferred_action_index(element))+ if not handled:+ x, y = screen_point(+ bounds,+ element_record,+ operation.get("x"),+ operation.get("y"),+ )+ send_mouse_click(+ x, y, operation.get("mouse_button", "left"), operation.get("click_count", 1)+ )+ elif tool == "perform_secondary_action":+ invoke_secondary_action(element, operation.get("action", ""))+ elif tool == "scroll":+ scroll_element(operation.get("direction", "down"), operation.get("pages", 1))+ elif tool == "drag":+ from_x, from_y = screen_point(+ bounds, None, operation.get("from_x"), operation.get("from_y")+ )+ to_x, to_y = screen_point(bounds, None, operation.get("to_x"), operation.get("to_y"))+ send_drag(from_x, from_y, to_x, to_y)+ elif tool == "type_text":+ if not insert_text(window, operation.get("text", "")):+ send_text(operation.get("text", ""))+ elif tool == "press_key":+ send_key(operation.get("key", ""))+ elif tool == "set_value":+ if element is None:+ raise RuntimeError("unknown element_index")+ if not set_element_value(element, operation.get("value", "")):+ raise RuntimeError("Cannot set a value for an element that is not settable")+ else:+ raise RuntimeError('unsupportedTool("{}")'.format(tool))++ time.sleep(0.12)+ return {"ok": True, "snapshot": build_snapshot(operation.get("app", ""))}+++def main():+ if len(sys.argv) != 2:+ raise RuntimeError("runtime.py requires an operation JSON path")+ require_desktop_session()+ Atspi.init()+ with open(sys.argv[1], "r", encoding="utf-8") as file:+ operation = json.load(file)+ try:+ response = perform_operation(operation)+ except Exception as exc:+ response = {"ok": False, "error": str(exc)}+ print(json.dumps(response, separators=(",", ":")))+++if __name__ == "__main__":+ try:+ main()+ except Exception:+ print(json.dumps({"ok": False, "error": traceback.format_exc()}))diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 3a94278..8fef15f 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -1,6 +1,6 @@# 架构总览-这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。主线仍是 Swift 实现的 macOS automation MCP server,同时新增了实验性的 Windows runtime,用独立 Go `.exe` 暴露同一组 9 个 Computer Use tools。+这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。主线仍是 Swift 实现的 macOS automation MCP server,同时新增了实验性的 Windows 和 Linux runtime,用独立 Go 二进制暴露同一组 9 个 Computer Use tools。## 当前目录结构@@ -12,6 +12,8 @@端到端 smoke runner,会拉起 fixture 和 MCP server,并通过 JSON-RPC 真实调用 9 个 tools;同时也支持单独的 visual cursor idle smoke,用跨进程 observation file 断言等待下一次 move 时是 anchored tip + tiny rotate wobble,而不是横向漂移。- `apps/OpenComputerUseWindows`实验性 Windows runtime。它不依赖 Swift 或 `.app` bundle,Go CLI/MCP 入口会嵌入 PowerShell UI Automation bridge,构建产物是 `open-computer-use.exe`。+- `apps/OpenComputerUseLinux`+ 实验性 Linux runtime。它不依赖 Swift 或 `.app` bundle,Go CLI/MCP 入口会嵌入 Python AT-SPI bridge,构建产物是 `open-computer-use`。- `packages/OpenComputerUseKit`核心库,包含:- MCP stdio transport 与 tool registry@@ -25,7 +27,7 @@- `experiments/StandaloneCursor`新的独立 Swift cursor viewer,直接复用 `scripts/cursor-motion-re/official_cursor_motion.py` 里收敛出来的候选路径、score 与 raw spring timeline,用来观察更贴近 binary lift 的表现。- `scripts/`- 仓库级自动化命令,包括 smoke test、`.app` 打包入口、Windows `.exe` 构建入口、npm 分发脚本,以及 `scripts/computer-use-cli/` 这个用于探测官方 bundled `computer-use` 的 Go helper。+ 仓库级自动化命令,包括 smoke test、`.app` 打包入口、Windows `.exe` / Linux binary 构建入口、npm 分发脚本,以及 `scripts/computer-use-cli/` 这个用于探测官方 bundled `computer-use` 的 Go helper。- `docs/`逆向分析、执行计划、history 和项目约束。@@ -106,6 +108,17 @@- Windows UI Automation 需要运行在已登录用户的桌面 session 里。通过 SSH 作为脱离桌面的后台进程运行时,PowerShell 可以启动并返回 JSON,但系统可能不给它暴露顶层窗口;这种情况下 `list_apps` 会是空,`get_app_state` 可能返回 `appNotFound(...)`。- 当前 Windows 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/onboarding、没有 code signing,也没有独立的 Windows smoke fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-windows-computer-use-runtime.md`。+### 7. Linux Runtime++- Linux runtime 位于 `apps/OpenComputerUseLinux`,以 Go 维护 CLI、`call --calls` sequence、MCP JSON-RPC、tool schema 和进程内 snapshot cache。+- 构建入口是 `scripts/build-open-computer-use-linux.sh --arch arm64|amd64`,默认输出到 `dist/linux/<arch>/open-computer-use`;当前 npm 包仍只发布 macOS app bundle,Linux 分发还没有接入 release package。+- Go runtime 通过 `go:embed` 带上 `runtime.py`,执行 tool call 时临时落盘并调用 `python3`。Python bridge 使用 GNOME/GObject Introspection 暴露的 AT-SPI2 接口做 app/window discovery、accessibility tree rendering、semantic action、editable text、value set,以及 best-effort 的 key/mouse fallback。+- Linux 上最接近 macOS AX 的是 AT-SPI2/D-Bus accessibility,而不是一套统一的后台键鼠输入模型。第一版优先使用元素暴露的 AT-SPI action、EditableText 和 Value 接口;coordinate `click` / `drag` 与 `press_key` 使用 AT-SPI event synthesis fallback,在 Wayland 下只能按 best-effort 处理。+- Linux runtime 需要运行在已登录桌面用户 session 里,并带有 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 和 display 环境。纯 SSH tty 可以启动二进制,但不能直接 inspect 或操作 GUI session;实机验证时通过 `runuser -u <desktop-user>` 挂到 `/run/user/<uid>/bus`。+- `get_app_state` 的 accessibility tree 在 GTK/GNOME app 上可能很深,Linux bridge 当前把 traversal depth 放宽到 64。截图通过 GDK root window best-effort capture;GNOME Wayland 可能返回黑图,bridge 会检测全黑采样并省略 image block。+- 这 9 个 tool 的协议面与 macOS / Windows 保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime path metadata,coordinate action 使用 screenshot/window-relative 坐标。+- 当前 Linux 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/desktop entry、没有 release artifact,也没有独立 Linux fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-linux-computer-use-runtime.md`。+## 关键边界- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。@@ -125,6 +138,8 @@- release tgz:`./scripts/release-package.sh`- Windows runtime 单测:`(cd apps/OpenComputerUseWindows && go test ./...)`- Windows exe 构建:`./scripts/build-open-computer-use-windows.sh --arch arm64`+- Linux runtime 单测:`(cd apps/OpenComputerUseLinux && go test ./...)`+- Linux binary 构建:`./scripts/build-open-computer-use-linux.sh --arch arm64`- 对比样本:`artifacts/tool-comparisons/20260417-focus-behavior/`- 手工诊断:- `.build/debug/OpenComputerUse doctor`diff --git a/docs/CICD.md b/docs/CICD.mdindex 8d093a0..bef3c4e 100644--- a/docs/CICD.md+++ b/docs/CICD.md@@ -6,6 +6,7 @@- `scripts/release-package.sh`:构建 universal `Open Computer Use.app`,stage 三个 npm 包目录,并产出 `dist/release/npm/*.tgz` 与 `dist/release/release-manifest.json`。当前 CI 继续显式使用 ad-hoc signing,保持和此前发布链路一致;本地 debug/dev 构建则允许使用开发机自己的签名身份。- `scripts/build-cursor-motion-dmg.sh`:本地构建 `Cursor Motion.app` 并封装 `dist/release/cursor-motion/CursorMotion-<version>.dmg`,支持 `native` / `arm64` / `x86_64` / `universal`。+- `scripts/build-open-computer-use-linux.sh`:本地构建实验性 Linux `open-computer-use` binary,支持 `arm64` / `amd64`;当前还没有接入正式 release package。- `.github/workflows/release.yml`:支持 push semver tag 自动发布,也支持手动触发;tag push 时会同时跑 npm release 打包逻辑与 `Cursor Motion` 的 DMG 打包,并把 `.dmg` 上传到对应的 GitHub Releases 页面。`Open Computer Use` 的 npm 产物默认走 ad-hoc signing;如果配置了 `OPEN_COMPUTER_USE_CODESIGN_*` secrets,则会先导入 `Developer ID Application` 证书,再按同一 identity 对 release `.app` 统一签名。`Cursor Motion` 的 DMG 也会复用同一张 `Developer ID Application` 证书签 app;若同时配置 `APPLE_NOTARY_*` secrets,则会在上传前对 `.dmg` 做 notarization 和 staple。## 设计原则diff --git a/docs/QUALITY_SCORE.md b/docs/QUALITY_SCORE.mdindex 150ae6b..a711120 100644--- a/docs/QUALITY_SCORE.md+++ b/docs/QUALITY_SCORE.md@@ -13,6 +13,7 @@| --- | --- | --- | --- || 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及一轮按官方 surface / result 行为收敛过的 9 个 tools。 | 继续收敛复杂 AX 场景下的 state rendering 细节、权限 UI 和更清晰的用户错误提示。 || Windows runtime | C | 已新增独立 Go `.exe`,通过 Windows UI Automation + Win32 window message 暴露同样 9 个 tools、MCP server 和 `call --calls`;默认不再自动启动 app、执行 `SetFocus`,或让 `type_text` 走可能抢前台的 UIA text fallback,但仍是功能性第一版。 | 补交互式桌面 smoke、Windows fixture、installer/signing/release 分发,以及更原生的 Go UIA 实现或更稳定的 bridge。 |+| Linux runtime | C | 已新增独立 Go binary,通过 Python GI / AT-SPI2 暴露同样 9 个 tools、MCP server 和 `call --calls`;Ubuntu GNOME VM 已跑通 `list_apps`、MCP tools list 和 Text Editor 8-tool sequence,但截图在 GNOME Wayland 下仍只能 best-effort,coordinate input 也不是通用后台模型。 | 补 Linux fixture、可重复 smoke runner、release artifact、portal/compositor screenshot 路径,以及更原生的 Go D-Bus/libatspi bridge。 || 架构文档 | B | 顶层结构、fixture bridge、app 模式和验证路径已经落文档。 | 后续补 release artifact、code signing / notarization 和 host 集成方式。 || 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并新增了针对“前台焦点是否被抢占”的手工对比样本沉淀。 | 增加更多普通 app 的录制回归,减少只依赖 fixture 和一次性手工检查。 || 可观测性 | C | 已有 `doctor`、`snapshot`、smoke 输出,以及一组仓库内留档的官方 `computer-use` / 本仓库实现对比样本。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息,把一次性样本收敛成可重复采集流程。 |diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.mdindex 2c0e86b..c5eda97 100644--- a/docs/RELIABILITY.md+++ b/docs/RELIABILITY.md@@ -5,6 +5,7 @@- 构建:`swift build`- 单元测试:`swift test`- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`+- Linux runtime:`(cd apps/OpenComputerUseLinux && go test ./...)`、`./scripts/build-open-computer-use-linux.sh --arch arm64`- 本地诊断:- `.build/debug/OpenComputerUse doctor`- `.build/debug/OpenComputerUse snapshot <app>`@@ -14,6 +15,8 @@- 宿主必须有 `Accessibility` 与 `Screen Recording` 权限。- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异。+- Linux runtime 依赖已登录桌面用户 session 的 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 和 display 环境;纯 SSH tty 默认不能直接访问 AT-SPI GUI tree。+- GNOME Wayland 截图可能被 compositor 限制,当前 Linux bridge 会把黑图视为无效截图并省略 image block。## 当前故障排查顺序@@ -21,6 +24,7 @@2. 用 `.build/debug/OpenComputerUse list-apps` 确认目标 app 是否被发现。3. 用 `.build/debug/OpenComputerUse snapshot <app>` 看是 transport 问题还是 snapshot / action 问题。4. 如果只想验证仓库基线,直接跑 fixture + smoke,不要先在复杂第三方 app 上排查。+5. 排查 Linux runtime 时,先确认目标命令是否运行在桌面用户 session 中,再用 `open-computer-use call list_apps` 和 `open-computer-use snapshot <app>` 区分 session/env 问题与 AT-SPI tree/action 问题。## 后续补强方向diff --git a/docs/SECURITY.md b/docs/SECURITY.mdindex 9e042ad..744e587 100644--- a/docs/SECURITY.md+++ b/docs/SECURITY.md@@ -5,10 +5,12 @@- 当前 server 只提供本地 `stdio` MCP,不对外监听 TCP/HTTP 端口。- 所有动作都必须显式带 `app` 参数;当前不会在后台自动扫描并控制任意 app。- 真实 app 路径依赖宿主进程已经获得 macOS `Accessibility` 与 `Screen Recording` 权限。+- 实验性 Linux runtime 依赖已登录桌面用户的 AT-SPI2 / D-Bus session;coordinate mouse、drag、keyboard synthesis 只是 best-effort fallback,不应被视为跨 Wayland compositor 的通用后台输入授权。## 数据处理- 普通 app 的 screenshot 默认只在内存中编码成 PNG,并通过 MCP `image` content block 直接回传;默认不长期持久化。+- Linux runtime 的 screenshot 是 best-effort;如果 GNOME Wayland 返回黑图,bridge 会省略 image block,避免把无效截图误当成真实画面。- fixture app 的合成状态只写到本地临时 JSON 文件,目的是支撑 deterministic smoke test;当前写入走原子替换,减少测试期间的读写竞争。- 当前仓库不引入第三方服务,也不上传截图、AX tree 或输入内容。diff --git a/docs/exec-plans/active/20260422-linux-computer-use-runtime.md b/docs/exec-plans/active/20260422-linux-computer-use-runtime.mdnew file mode 100644index 0000000..ece7d1c--- /dev/null+++ b/docs/exec-plans/active/20260422-linux-computer-use-runtime.md@@ -0,0 +1,96 @@+# Linux Computer Use Runtime++## 目标++把 `open-computer-use` 从 macOS / Windows 扩展到 Linux 桌面,优先让同一组 9 个 Computer Use tools 能在 Ubuntu GNOME 桌面 session 里通过独立二进制跑通,并把 Linux 桌面自动化的能力边界记录清楚。++## 范围++- 包含:+ - Linux 独立 runtime,不耦合 Swift `.app`。+ - Go CLI / MCP / `call --calls` 入口。+ - `list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value` 的功能性实现。+ - Linux arm64/amd64 构建脚本和基础 Go 单测。+ - Ubuntu GNOME VM 上的 9-tool 实机 smoke。+ - 架构文档、README、质量说明和 history。+- 不包含:+ - 替换 macOS Swift 主线。+ - Linux installer、desktop entry、system package、code signing 或 npm release artifact。+ - visual cursor overlay。+ - 完整 Linux fixture / 可重复 smoke runner。++## 背景++- 相关文档:+ - `docs/ARCHITECTURE.md`+ - `docs/QUALITY_SCORE.md`+ - `docs/SECURITY.md`+ - `docs/RELIABILITY.md`+- 相关代码路径:+ - `apps/OpenComputerUseLinux/`+ - `scripts/build-open-computer-use-linux.sh`+ - `scripts/ci.sh`+ - `apps/OpenComputerUseWindows/`+- 已知约束:+ - Linux 上最接近 macOS AX 的桌面接口是 AT-SPI2,经由 D-Bus 暴露 app/window/accessibility tree、actions、editable text 和 value 接口。+ - Ubuntu GNOME 默认 Wayland session 下,任意后台坐标键鼠和截图都不是一套等价于 macOS AX 的通用模型。+ - 第一版策略是 AT-SPI semantic action / editable text / value 优先,coordinate click / drag / key synthesis 仅作为 best-effort fallback。+ - SSH tty 默认没有 `XDG_RUNTIME_DIR` / `DBUS_SESSION_BUS_ADDRESS` / display 环境;真实 UI 测试需要跑在已登录桌面用户 session 里。++## 风险++- 风险:不同 toolkit 暴露的 AT-SPI tree 深度、role、action 名称差异大。+ - 缓解方式:Linux bridge 单独放宽 tree traversal depth,并保留字段级容错。+- 风险:Wayland 下 screenshot 可能返回黑图或被 portal/ compositor 拒绝。+ - 缓解方式:截图只做 best-effort;检测到全黑采样时不回传 image block,避免误导调用方。+- 风险:coordinate click / drag / key synthesis 可能影响当前 foreground context。+ - 缓解方式:MCP instructions 和 README 明确写出 Linux background input boundary;优先使用 element-targeted AT-SPI action。++## 里程碑++1. 确认 Linux 可用接口和边界。+2. 完成 Linux Go runtime、Python AT-SPI bridge 和 9-tool 功能性实现。+3. 完成本地单测、交叉编译和 Ubuntu GNOME VM 9-tool smoke。+4. 后续补 Linux fixture、可重复 smoke runner、release artifact 和更稳定截图方案。++## 验证方式++- 命令:+ - `(cd apps/OpenComputerUseLinux && go test ./...)`+ - `./scripts/build-open-computer-use-linux.sh --arch arm64`+ - `./scripts/build-open-computer-use-linux.sh --arch amd64`+ - `open-computer-use mcp`+ - `open-computer-use call list_apps`+ - `open-computer-use call --calls-file <9-tool smoke json>`+- 手工检查:+ - 在 Ubuntu GNOME desktop session 里打开 `gnome-text-editor`。+ - 运行 `get_app_state -> set_value -> type_text -> press_key -> perform_secondary_action -> click -> scroll -> drag` sequence。+ - 确认每个 tool 返回 `isError=false`,并且 Text Editor 内容包含 marker。+- 观测检查:+ - 纯 SSH tty 没有桌面环境变量时应返回明确错误,而不是误判为 AT-SPI 逻辑失败。++## 进度记录++- [x] 确认 Ubuntu GNOME VM 有已登录 `leo` Wayland session、AT-SPI bus、Python GI、Atspi、Gdk/GdkPixbuf。+- [x] 新增 `apps/OpenComputerUseLinux`,用 Go 实现 CLI、MCP、tool schema、`call --calls` 和 snapshot cache。+- [x] 嵌入 Python AT-SPI bridge,实现 app/window discovery、tree rendering、semantic action、editable text、value、key/mouse fallback 和 best-effort screenshot。+- [x] 新增 Linux arm64/amd64 构建脚本。+- [x] 新增 Go 单测,并接入仓库基础 CI。+- [x] 本地通过 `(cd apps/OpenComputerUseLinux && go test ./...)`。+- [x] 本地通过 `./scripts/build-open-computer-use-linux.sh --arch arm64` 和 `--arch amd64`。+- [x] 上传 arm64 二进制到 Ubuntu VM,验证 `--version` 为 `0.1.33`。+- [x] 在 Ubuntu VM 中验证 `call list_apps` 返回 `isError=false` 并包含 `gnome-text-editor`。+- [x] 在 Ubuntu VM 中验证 MCP `initialize` / `tools/list`,tool count 为 9。+- [x] 在 Ubuntu VM 中验证 8-tool sequence:`get_app_state`、`set_value`、`type_text`、`press_key`、`perform_secondary_action`、`click`、`scroll`、`drag` 均 `isError=false`。+- [ ] 增加 Linux fixture 和可重复 smoke runner。+- [ ] 评估 xdg-desktop-portal / compositor-specific screenshot 路径,补非黑图 capture。+- [ ] 将 Linux artifact 接入 release packaging。+- [ ] 评估用原生 Go D-Bus/libatspi 替换 Python GI bridge 的收益和风险。++## 决策记录++- 2026-04-22:Linux runtime 不复用 Swift `.app` 或 Windows `.exe` bridge,采用独立 Go binary,避免把 macOS / Windows 的权限和输入模型强行带到 Linux。+- 2026-04-22:第一版用 Go 管协议、状态和分发边界,用嵌入式 Python GI 调 AT-SPI/GDK,优先完成 9-tool 功能性闭环。+- 2026-04-22:Linux 默认使用 AT-SPI semantic action / editable text / value;coordinate mouse、drag、keyboard synthesis 作为 best-effort fallback,并在 MCP instructions 中明确不是通用 Wayland background input。+- 2026-04-22:GNOME Text Editor 的 AT-SPI tree 深度超过 Windows runtime 沿用的 16 层,Linux bridge 单独把 traversal depth 放宽到 64。+- 2026-04-22:GNOME Wayland 下 GDK root capture 在 VM 上返回黑图;Linux bridge 检测全黑采样后省略 image block,后续再评估 portal/compositor-specific capture。diff --git a/docs/histories/2026-04/20260422-2036-add-linux-computer-use-runtime.md b/docs/histories/2026-04/20260422-2036-add-linux-computer-use-runtime.mdnew file mode 100644index 0000000..f6a4f59--- /dev/null+++ b/docs/histories/2026-04/20260422-2036-add-linux-computer-use-runtime.md@@ -0,0 +1,38 @@+## [2026-04-22 20:36] | Task: Add Linux Computer Use Runtime++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `local repo + Ubuntu GNOME VM over SSH`++### 📥 User Query+> Extend `open-computer-use` to Linux now that macOS and Windows exist. Confirm whether Linux has an AX-like background app automation surface, implement the same 9 tools, upload to the Ubuntu desktop VM, test every tool, and keep a plan/TODO for follow-up tracking.++### 🛠 Changes Overview+**Scope:** Linux runtime, build scripts, CI, docs++**Key Actions:**+- **[Runtime]**: Added `apps/OpenComputerUseLinux`, a Go CLI/MCP runtime exposing the same 9 Computer Use tools, `call --calls`, MCP initialize/tools list, and per-process snapshot cache.+- **[Bridge]**: Embedded a Python GI / AT-SPI2 bridge for app/window discovery, accessibility tree snapshots, semantic actions, editable text, value setting, and best-effort key/mouse fallback.+- **[Build/CI]**: Added `scripts/build-open-computer-use-linux.sh` for arm64/amd64 builds and wired Linux Go tests into `scripts/ci.sh`.+- **[Docs]**: Updated README, architecture, reliability, security, CI/CD, quality score, and added an active Linux runtime execution plan with remaining TODOs.+- **[Validation]**: Built the Linux arm64 binary, uploaded it to the Ubuntu GNOME VM, and verified `list_apps`, MCP `tools/list`, plus a Text Editor sequence covering `get_app_state`, `set_value`, `type_text`, `press_key`, `perform_secondary_action`, `click`, `scroll`, and `drag`.++### 🧠 Design Intent (Why)+Linux desktop automation should not inherit macOS `.app` / TCC assumptions or Windows UIA / Win32 assumptions. AT-SPI2 is the closest Linux equivalent to macOS AX for semantic UI automation, but Wayland does not provide a universal background keyboard/mouse/screenshot model. The first slice keeps the public tool protocol stable, makes the runtime buildable and testable as a standalone Linux binary, and documents coordinate input and screenshot paths as best-effort.++### 📁 Files Modified+- `apps/OpenComputerUseLinux/go.mod`+- `apps/OpenComputerUseLinux/main.go`+- `apps/OpenComputerUseLinux/main_test.go`+- `apps/OpenComputerUseLinux/runtime.py`+- `scripts/build-open-computer-use-linux.sh`+- `scripts/ci.sh`+- `README.md`+- `README.zh-CN.md`+- `docs/ARCHITECTURE.md`+- `docs/CICD.md`+- `docs/QUALITY_SCORE.md`+- `docs/RELIABILITY.md`+- `docs/SECURITY.md`+- `docs/exec-plans/active/20260422-linux-computer-use-runtime.md`diff --git a/scripts/build-open-computer-use-linux.sh b/scripts/build-open-computer-use-linux.shnew file mode 100755index 0000000..6a8e7c5--- /dev/null+++ b/scripts/build-open-computer-use-linux.sh@@ -0,0 +1,71 @@+#!/usr/bin/env bash++set -euo pipefail++repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+arch="arm64"+configuration="release"+out_dir="${repo_root}/dist/linux"++print_help() {+ cat <<'EOF'+Usage:+ scripts/build-open-computer-use-linux.sh [options]++Options:+ --arch arm64|amd64 Linux target architecture. Defaults to arm64.+ --configuration release Reserved for parity with the macOS build script.+ --out-dir <dir> Output directory. Defaults to dist/linux.+ -h, --help Show help.+EOF+}++while [[ $# -gt 0 ]]; do+ case "$1" in+ --arch)+ arch="${2:?--arch requires a value}"+ shift 2+ ;;+ --configuration)+ configuration="${2:?--configuration requires a value}"+ shift 2+ ;;+ --out-dir)+ out_dir="$(cd "${repo_root}" && mkdir -p "$2" && cd "$2" && pwd)"+ shift 2+ ;;+ -h|--help)+ print_help+ exit 0+ ;;+ *)+ echo "Unknown argument: $1" >&2+ print_help >&2+ exit 1+ ;;+ esac+done++case "${arch}" in+ arm64|amd64) ;;+ *)+ echo "Unsupported Linux arch: ${arch}" >&2+ exit 1+ ;;+esac++version="$(node -e "console.log(require('./plugins/open-computer-use/.codex-plugin/plugin.json').version)")"+module_dir="${repo_root}/apps/OpenComputerUseLinux"+output_dir="${out_dir}/${arch}"+mkdir -p "${output_dir}"++(+ cd "${module_dir}"+ GOOS=linux GOARCH="${arch}" CGO_ENABLED=0 go build \+ -trimpath \+ -ldflags "-s -w -X main.version=${version}" \+ -o "${output_dir}/open-computer-use" \+ .+)++echo "Built ${configuration} Linux runtime: ${output_dir}/open-computer-use"diff --git a/scripts/ci.sh b/scripts/ci.shindex 63392c0..f12dc11 100755--- a/scripts/ci.sh+++ b/scripts/ci.sh@@ -21,6 +21,10 @@ if command -v go >/dev/null 2>&1; thencd "${repo_root}/apps/OpenComputerUseWindows"go test ./...)+ (+ cd "${repo_root}/apps/OpenComputerUseLinux"+ go test ./...+ )fiecho "基础 CI 检查通过"