👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Add call-seq helper for computer-use-cli
Leoff33b9849dbc
diff --git a/docs/histories/2026-04/20260420-1416-add-computer-use-cli-call-seq.md b/docs/histories/2026-04/20260420-1416-add-computer-use-cli-call-seq.mdnew file mode 100644index 0000000..f3529c3--- /dev/null+++ b/docs/histories/2026-04/20260420-1416-add-computer-use-cli-call-seq.md@@ -0,0 +1,36 @@+## [2026-04-20 14:16] | Task: 为 computer-use-cli 增加顺序调用能力++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `gpt-5.4`+* **Runtime**: `Codex CLI`++### 📥 User Query+> 想自己通过 `scripts/computer-use-cli/` 连官方 bundled `computer-use`,复现哪些 tool 调用会出现 overlay cursor,并直接用 `go run` 测。++### 🛠 Changes Overview+**Scope:** `scripts/computer-use-cli`++**Key Actions:**+- **新增 `call-seq` 子命令**: 允许在同一条 direct MCP 连接或同一条 app-server ephemeral thread 里顺序执行多个 tool call,解决官方 `computer-use` 动作类工具必须先做 `get_app_state` 的前置约束。+- **补充官方自测样例**: 新增 `examples/textedit-overlay-seq.json`,包含 `get_app_state -> set_value -> scroll -> perform_secondary_action` 的正例序列。+- **更新说明与测试**: README 补充 `call-seq` 用法与限制说明,单测覆盖顺序调用 JSON 解析。++### 🧠 Design Intent (Why)+官方 bundled `computer-use` 在动作类 tool 前要求同线程内已有对应 app 的最新 state。原来的 `call` 每次都会新建一个 app-server 临时 thread,无法直接用 `go run` 复现这条链路。新增 `call-seq` 后,用户可以用一个 JSON 文件稳定复现实测路径,也更适合后续继续做官方行为对比。++### 📁 Files Modified+- `scripts/computer-use-cli/main.go`+- `scripts/computer-use-cli/app_server.go`+- `scripts/computer-use-cli/main_test.go`+- `scripts/computer-use-cli/README.md`+- `scripts/computer-use-cli/examples/textedit-overlay-seq.json`++### 🔁 Follow-up (2026-04-20 15:07)++- **[9 tool coverage sample]**: 将 `examples/textedit-overlay-seq.json` 扩成一条覆盖官方 9 个 tools 的 `TextEdit` 序列,方便直接用 `go run . call-seq` 手工观察整体效果。+- **[官方 stale-state 约束留档]**: 样例在每个会改动 app state 的 action 之间显式插入 `get_app_state`,因为官方 bundled `computer-use` 会在 mutation 后返回“先重新 query 最新 state”的约束。++**Follow-up Files:**+- `scripts/computer-use-cli/README.md`+- `scripts/computer-use-cli/examples/textedit-overlay-seq.json`diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.mdindex 2a519c5..cb072af 100644--- a/scripts/computer-use-cli/README.md+++ b/scripts/computer-use-cli/README.md@@ -8,6 +8,7 @@ This directory is intended to be run from `scripts/computer-use-cli/` inside thecd scripts/computer-use-cligo run . list-toolsgo run . call list_apps+go run . call-seq --calls-file /tmp/calls.json```It supports two transports:@@ -27,6 +28,7 @@ go run . resolve-servergo run . list-toolsgo run . call list_appsgo run . call get_app_state --args '{"app":"Feishu"}'+go run . call-seq --calls-file /tmp/calls.json```Explicit transport examples:@@ -35,6 +37,7 @@ Explicit transport examples:go run . list-tools --transport app-servergo run . call list_apps --transport app-servergo run . call list_apps --transport direct --server-bin /path/to/open-computer-use+go run . call-seq --transport app-server --calls-file /tmp/calls.json```Flags can appear either before or after the tool name for `call`:@@ -44,6 +47,24 @@ go run . call --server-bin /path/to/server list_appsgo run . call list_apps --server-bin /path/to/server```+`call` creates a fresh ephemeral app-server thread per invocation. If you need `get_app_state`+followed by one or more action tools in the same official `computer-use` session, use+`call-seq` with a JSON array:++```json+[+ {"tool": "get_app_state", "args": {"app": "TextEdit"}},+ {"tool": "set_value", "args": {"app": "TextEdit", "element_index": "2", "value": "cursor probe 01\ncursor probe 02"}},+ {"tool": "scroll", "args": {"app": "TextEdit", "element_index": "1", "direction": "down", "pages": 1}}+]+```++This repo includes a ready-to-run example at+`examples/textedit-overlay-seq.json`.+That sample covers all 9 official tools against `TextEdit`, and it intentionally+repeats `get_app_state` between state-changing actions because the official+bundled `computer-use` invalidates the cached app state after each mutation.+## Default targetBy default, the CLI auto-discovers the official bundled `computer-use` plugin under:@@ -92,4 +113,5 @@ Example working invocation against the official bundled `computer-use`:```bashgo run . call list_apps --transport app-server+go run . call-seq --transport app-server --calls-file /tmp/calls.json```diff --git a/scripts/computer-use-cli/app_server.go b/scripts/computer-use-cli/app_server.goindex 6ca55c1..b677cd7 100644--- a/scripts/computer-use-cli/app_server.go+++ b/scripts/computer-use-cli/app_server.go@@ -215,11 +215,20 @@ func (s *appServerSession) callToolViaAppServer(toolName string,toolArgs map[string]any,) (*mcp.CallToolResult, error) {- threadCwd, err := resolveThreadCwd(flags)+ threadID, err := s.startEphemeralThread(ctx, flags)if err != nil {return nil, err}+ return s.callToolOnThread(ctx, threadID, flags.serverName, toolName, toolArgs)+}++func (s *appServerSession) startEphemeralThread(ctx context.Context, flags commonFlags) (string, error) {+ threadCwd, err := resolveThreadCwd(flags)+ if err != nil {+ return "", err+ }+var thread appServerThreadStartResultif err := s.request(ctx, "thread/start", appServerThreadStartParams{ApprovalPolicy: flags.approvalPolicy,@@ -227,16 +236,25 @@ func (s *appServerSession) callToolViaAppServer(Ephemeral: true,Sandbox: flags.sandbox,}, &thread); err != nil {- return nil, fmt.Errorf("start ephemeral thread: %w", err)+ return "", fmt.Errorf("start ephemeral thread: %w", err)}if thread.Thread.ID == "" {- return nil, fmt.Errorf("app-server returned an empty thread id")+ return "", fmt.Errorf("app-server returned an empty thread id")}+ return thread.Thread.ID, nil+}+func (s *appServerSession) callToolOnThread(+ ctx context.Context,+ threadID string,+ serverName string,+ toolName string,+ toolArgs map[string]any,+) (*mcp.CallToolResult, error) {var result mcp.CallToolResultif err := s.request(ctx, "mcpServer/tool/call", appServerToolCallParams{- ThreadID: thread.Thread.ID,- Server: flags.serverName,+ ThreadID: threadID,+ Server: serverName,Tool: toolName,Arguments: toolArgs,}, &result); err != nil {@@ -245,6 +263,35 @@ func (s *appServerSession) callToolViaAppServer(return &result, nil}+func callToolSequenceViaAppServer(+ ctx context.Context,+ session *appServerSession,+ flags commonFlags,+ calls []toolCallSpec,+) ([]toolCallOutput, error) {+ threadCtx, cancel := context.WithTimeout(ctx, flags.timeout)+ threadID, err := session.startEphemeralThread(threadCtx, flags)+ cancel()+ if err != nil {+ return nil, err+ }++ results := make([]toolCallOutput, 0, len(calls))+ for i, call := range calls {+ callCtx, cancel := context.WithTimeout(ctx, flags.timeout)+ result, err := session.callToolOnThread(callCtx, threadID, flags.serverName, call.Tool, call.Args)+ cancel()+ if err != nil {+ return nil, fmt.Errorf("call #%d %q via app-server: %w", i+1, call.Tool, err)+ }+ results = append(results, toolCallOutput{+ Tool: call.Tool,+ Result: result,+ })+ }+ return results, nil+}+func (s *appServerSession) request(ctx context.Context, method string, params any, out any) error {requestID := atomic.AddInt64(&s.nextID, 1)diff --git a/scripts/computer-use-cli/examples/textedit-overlay-seq.json b/scripts/computer-use-cli/examples/textedit-overlay-seq.jsonnew file mode 100644index 0000000..96028d2--- /dev/null+++ b/scripts/computer-use-cli/examples/textedit-overlay-seq.json@@ -0,0 +1,103 @@+[+ {+ "tool": "list_apps"+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "set_value",+ "args": {+ "app": "TextEdit",+ "element_index": "2",+ "value": "line 01\nline 02\nline 03\nline 04\nline 05\nline 06\nline 07\nline 08\nline 09\nline 10\nline 11\nline 12\nline 13\nline 14\nline 15\nline 16\nline 17\nline 18\nline 19\nline 20"+ }+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "click",+ "args": {+ "app": "TextEdit",+ "element_index": "2"+ }+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "type_text",+ "args": {+ "app": "TextEdit",+ "text": "\n[type_text] hello from call-seq"+ }+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "press_key",+ "args": {+ "app": "TextEdit",+ "key": "Return"+ }+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "scroll",+ "args": {+ "app": "TextEdit",+ "element_index": "1",+ "direction": "down",+ "pages": 1+ }+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "drag",+ "args": {+ "app": "TextEdit",+ "from_x": 120,+ "from_y": 120,+ "to_x": 320,+ "to_y": 120+ }+ },+ {+ "tool": "get_app_state",+ "args": {+ "app": "TextEdit"+ }+ },+ {+ "tool": "perform_secondary_action",+ "args": {+ "app": "TextEdit",+ "element_index": "0",+ "action": "Raise"+ }+ }+]diff --git a/scripts/computer-use-cli/main.go b/scripts/computer-use-cli/main.goindex d2fe2ec..bb27c39 100644--- a/scripts/computer-use-cli/main.go+++ b/scripts/computer-use-cli/main.go@@ -47,6 +47,16 @@ type resolvedTarget struct {ServerBin string `json:"serverBin"`}+type toolCallSpec struct {+ Tool string `json:"tool"`+ Args map[string]any `json:"args,omitempty"`+}++type toolCallOutput struct {+ Tool string `json:"tool"`+ Result *mcp.CallToolResult `json:"result"`+}+type rpcSession struct {conn mcp.ConnectionnextID int64@@ -86,6 +96,8 @@ func run(ctx context.Context, args []string, stdout io.Writer, stderr io.Writer)return runListTools(ctx, args[1:], stdout, stderr)case "call":return runCall(ctx, args[1:], stdout, stderr)+ case "call-seq":+ return runCallSeq(ctx, args[1:], stdout, stderr)case "help", "-h", "--help":printRootUsage(stderr)return nil@@ -247,6 +259,66 @@ func runCall(ctx context.Context, args []string, stdout io.Writer, stderr io.Wri}}+func runCallSeq(ctx context.Context, args []string, stdout io.Writer, stderr io.Writer) error {+ var flags commonFlags+ var callsJSON string+ var callsFile string++ fs := flag.NewFlagSet("call-seq", flag.ContinueOnError)+ fs.SetOutput(stderr)+ addCommonFlags(fs, &flags)+ fs.StringVar(&callsJSON, "calls", "", "JSON array of sequential tool calls")+ fs.StringVar(&callsFile, "calls-file", "", "Path to a file containing a JSON array of sequential tool calls")+ if err := fs.Parse(args); err != nil {+ return err+ }+ if fs.NArg() != 0 {+ return fmt.Errorf("usage: %s call-seq [flags]", cliName)+ }+ if callsJSON != "" && callsFile != "" {+ return fmt.Errorf("use only one of --calls or --calls-file")+ }++ calls, err := readToolCallSequence(callsJSON, callsFile)+ if err != nil {+ return err+ }++ transport, err := resolveTransport(flags)+ if err != nil {+ return err+ }++ switch transport {+ case transportDirect:+ session, err := connect(ctx, flags)+ if err != nil {+ return err+ }+ defer session.Close()++ results, err := callToolSequenceDirect(ctx, session, flags.timeout, calls)+ if err != nil {+ return err+ }+ return writeJSON(stdout, results, flags.pretty)+ case transportAppServer:+ session, err := connectAppServer(ctx, flags)+ if err != nil {+ return err+ }+ defer session.Close()++ results, err := callToolSequenceViaAppServer(ctx, session, flags, calls)+ if err != nil {+ return err+ }+ return writeJSON(stdout, results, flags.pretty)+ default:+ return fmt.Errorf("unsupported transport %q", transport)+ }+}+func addCommonFlags(fs *flag.FlagSet, flags *commonFlags) {fs.StringVar(&flags.transport, "transport", transportAuto, "Connection mode: auto, direct, or app-server")fs.StringVar(&flags.pluginRoot, "plugin-root", "", "Path to the installed computer-use plugin root")@@ -324,6 +396,28 @@ func (s *rpcSession) listTools(ctx context.Context) (*mcp.ListToolsResult, errorreturn &result, nil}+func callToolSequenceDirect(+ ctx context.Context,+ session *rpcSession,+ timeout time.Duration,+ calls []toolCallSpec,+) ([]toolCallOutput, error) {+ results := make([]toolCallOutput, 0, len(calls))+ for i, call := range calls {+ callCtx, cancel := context.WithTimeout(ctx, timeout)+ result, err := session.callTool(callCtx, call.Tool, call.Args)+ cancel()+ if err != nil {+ return nil, fmt.Errorf("call #%d %q: %w", i+1, call.Tool, err)+ }+ results = append(results, toolCallOutput{+ Tool: call.Tool,+ Result: result,+ })+ }+ return results, nil+}+func (s *rpcSession) callTool(ctx context.Context,toolName string,@@ -616,6 +710,40 @@ func readToolArgs(argsJSON, argsFile string) (map[string]any, error) {return object, nil}+func readToolCallSequence(callsJSON, callsFile string) ([]toolCallSpec, error) {+ if callsFile != "" {+ data, err := os.ReadFile(callsFile)+ if err != nil {+ return nil, fmt.Errorf("read calls file %q: %w", callsFile, err)+ }+ callsJSON = string(data)+ }++ if strings.TrimSpace(callsJSON) == "" {+ return nil, fmt.Errorf("missing --calls or --calls-file")+ }++ decoder := json.NewDecoder(strings.NewReader(callsJSON))+ decoder.UseNumber()++ var calls []toolCallSpec+ if err := decoder.Decode(&calls); err != nil {+ return nil, fmt.Errorf("decode tool call sequence JSON: %w", err)+ }+ if decoder.More() {+ return nil, fmt.Errorf("tool call sequence JSON must contain exactly one array")+ }+ if len(calls) == 0 {+ return nil, fmt.Errorf("tool call sequence must contain at least one call")+ }+ for i, call := range calls {+ if strings.TrimSpace(call.Tool) == "" {+ return nil, fmt.Errorf("call #%d is missing a non-empty tool name", i+1)+ }+ }+ return calls, nil+}+func writeJSON(w io.Writer, value any, pretty bool) error {var (data []byte@@ -680,6 +808,7 @@ Usage:%s resolve-server [flags]%s list-tools [flags]%s call [flags] <tool-name>+ %s call-seq [flags]Examples:%s resolve-server@@ -688,10 +817,11 @@ Examples:%s call list_apps --transport app-server%s call list_apps --transport direct --server-bin /path/to/open-computer-use%s call get_app_state --args '{"app":"Feishu"}'+ %s call-seq --transport app-server --calls-file /tmp/calls.jsonEnvironment:%s Override the installed plugin root.%s Override the SkyComputerUseClient executable path.%s Override the Codex binary used for app-server mode.-`, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, pluginRootEnvVar, serverBinEnvVar, appServerBinEnvVar)+`, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, pluginRootEnvVar, serverBinEnvVar, appServerBinEnvVar)}diff --git a/scripts/computer-use-cli/main_test.go b/scripts/computer-use-cli/main_test.goindex 0c61cfc..4bbe3bf 100644--- a/scripts/computer-use-cli/main_test.go+++ b/scripts/computer-use-cli/main_test.go@@ -37,6 +37,37 @@ func TestReadToolArgsRejectsNonObject(t *testing.T) {}}+func TestReadToolCallSequenceAcceptsJSONArray(t *testing.T) {+ t.Parallel()++ calls, err := readToolCallSequence(`[{"tool":"get_app_state","args":{"app":"TextEdit"}},{"tool":"scroll","args":{"pages":1}}]`, "")+ if err != nil {+ t.Fatalf("readToolCallSequence returned error: %v", err)+ }+ if got, want := len(calls), 2; got != want {+ t.Fatalf("len(calls) = %d, want %d", got, want)+ }+ if got, want := calls[0].Tool, "get_app_state"; got != want {+ t.Fatalf("calls[0].Tool = %q, want %q", got, want)+ }++ number, ok := calls[1].Args["pages"].(json.Number)+ if !ok {+ t.Fatalf("pages type = %T, want json.Number", calls[1].Args["pages"])+ }+ if got, want := number.String(), "1"; got != want {+ t.Fatalf("pages = %q, want %q", got, want)+ }+}++func TestReadToolCallSequenceRejectsMissingTool(t *testing.T) {+ t.Parallel()++ if _, err := readToolCallSequence(`[{"args":{"app":"TextEdit"}}]`, ""); err == nil {+ t.Fatal("readToolCallSequence should reject a call without a tool name")+ }+}+func TestDiscoverPluginRootChoosesNewestCandidate(t *testing.T) {tempHome := t.TempDir()t.Setenv("HOME", tempHome)