Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Add call-seq helper for computer-use-cli

Leoff33b9849dbc
diff --git a/docs/histories/2026-04/20260420-1416-add-computer-use-cli-call-seq.md b/docs/histories/2026-04/20260420-1416-add-computer-use-cli-call-seq.md
new file mode 100644
index 0000000..f3529c3
--- /dev/null
+++ b/docs/histories/2026-04/20260420-1416-add-computer-use-cli-call-seq.md
@@ -0,0 +1,36 @@
+## [2026-04-20 14:16] | Task: 为 computer-use-cli 增加顺序调用能力
+
+### 🤖 Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `gpt-5.4`
+* **Runtime**: `Codex CLI`
+
+### 📥 User Query
+> 想自己通过 `scripts/computer-use-cli/` 连官方 bundled `computer-use`,复现哪些 tool 调用会出现 overlay cursor,并直接用 `go run` 测。
+
+### 🛠 Changes Overview
+**Scope:** `scripts/computer-use-cli`
+
+**Key Actions:**
+- **新增 `call-seq` 子命令**: 允许在同一条 direct MCP 连接或同一条 app-server ephemeral thread 里顺序执行多个 tool call,解决官方 `computer-use` 动作类工具必须先做 `get_app_state` 的前置约束。
+- **补充官方自测样例**: 新增 `examples/textedit-overlay-seq.json`,包含 `get_app_state -> set_value -> scroll -> perform_secondary_action` 的正例序列。
+- **更新说明与测试**: README 补充 `call-seq` 用法与限制说明,单测覆盖顺序调用 JSON 解析。
+
+### 🧠 Design Intent (Why)
+官方 bundled `computer-use` 在动作类 tool 前要求同线程内已有对应 app 的最新 state。原来的 `call` 每次都会新建一个 app-server 临时 thread,无法直接用 `go run` 复现这条链路。新增 `call-seq` 后,用户可以用一个 JSON 文件稳定复现实测路径,也更适合后续继续做官方行为对比。
+
+### 📁 Files Modified
+- `scripts/computer-use-cli/main.go`
+- `scripts/computer-use-cli/app_server.go`
+- `scripts/computer-use-cli/main_test.go`
+- `scripts/computer-use-cli/README.md`
+- `scripts/computer-use-cli/examples/textedit-overlay-seq.json`
+
+### 🔁 Follow-up (2026-04-20 15:07)
+
+- **[9 tool coverage sample]**: 将 `examples/textedit-overlay-seq.json` 扩成一条覆盖官方 9 个 tools 的 `TextEdit` 序列,方便直接用 `go run . call-seq` 手工观察整体效果。
+- **[官方 stale-state 约束留档]**: 样例在每个会改动 app state 的 action 之间显式插入 `get_app_state`,因为官方 bundled `computer-use` 会在 mutation 后返回“先重新 query 最新 state”的约束。
+
+**Follow-up Files:**
+- `scripts/computer-use-cli/README.md`
+- `scripts/computer-use-cli/examples/textedit-overlay-seq.json`
diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.md
index 2a519c5..cb072af 100644
--- a/scripts/computer-use-cli/README.md
+++ b/scripts/computer-use-cli/README.md
@@ -8,6 +8,7 @@ This directory is intended to be run from `scripts/computer-use-cli/` inside the
cd scripts/computer-use-cli
go run . list-tools
go run . call list_apps
+go run . call-seq --calls-file /tmp/calls.json
```
It supports two transports:
@@ -27,6 +28,7 @@ go run . resolve-server
go run . list-tools
go run . call list_apps
go run . call get_app_state --args '{"app":"Feishu"}'
+go run . call-seq --calls-file /tmp/calls.json
```
Explicit transport examples:
@@ -35,6 +37,7 @@ Explicit transport examples:
go run . list-tools --transport app-server
go run . call list_apps --transport app-server
go run . call list_apps --transport direct --server-bin /path/to/open-computer-use
+go run . call-seq --transport app-server --calls-file /tmp/calls.json
```
Flags can appear either before or after the tool name for `call`:
@@ -44,6 +47,24 @@ go run . call --server-bin /path/to/server list_apps
go run . call list_apps --server-bin /path/to/server
```
+`call` creates a fresh ephemeral app-server thread per invocation. If you need `get_app_state`
+followed by one or more action tools in the same official `computer-use` session, use
+`call-seq` with a JSON array:
+
+```json
+[
+ {"tool": "get_app_state", "args": {"app": "TextEdit"}},
+ {"tool": "set_value", "args": {"app": "TextEdit", "element_index": "2", "value": "cursor probe 01\ncursor probe 02"}},
+ {"tool": "scroll", "args": {"app": "TextEdit", "element_index": "1", "direction": "down", "pages": 1}}
+]
+```
+
+This repo includes a ready-to-run example at
+`examples/textedit-overlay-seq.json`.
+That sample covers all 9 official tools against `TextEdit`, and it intentionally
+repeats `get_app_state` between state-changing actions because the official
+bundled `computer-use` invalidates the cached app state after each mutation.
+
## Default target
By default, the CLI auto-discovers the official bundled `computer-use` plugin under:
@@ -92,4 +113,5 @@ Example working invocation against the official bundled `computer-use`:
```bash
go run . call list_apps --transport app-server
+go run . call-seq --transport app-server --calls-file /tmp/calls.json
```
diff --git a/scripts/computer-use-cli/app_server.go b/scripts/computer-use-cli/app_server.go
index 6ca55c1..b677cd7 100644
--- a/scripts/computer-use-cli/app_server.go
+++ b/scripts/computer-use-cli/app_server.go
@@ -215,11 +215,20 @@ func (s *appServerSession) callToolViaAppServer(
toolName string,
toolArgs map[string]any,
) (*mcp.CallToolResult, error) {
- threadCwd, err := resolveThreadCwd(flags)
+ threadID, err := s.startEphemeralThread(ctx, flags)
if err != nil {
return nil, err
}
+ return s.callToolOnThread(ctx, threadID, flags.serverName, toolName, toolArgs)
+}
+
+func (s *appServerSession) startEphemeralThread(ctx context.Context, flags commonFlags) (string, error) {
+ threadCwd, err := resolveThreadCwd(flags)
+ if err != nil {
+ return "", err
+ }
+
var thread appServerThreadStartResult
if err := s.request(ctx, "thread/start", appServerThreadStartParams{
ApprovalPolicy: flags.approvalPolicy,
@@ -227,16 +236,25 @@ func (s *appServerSession) callToolViaAppServer(
Ephemeral: true,
Sandbox: flags.sandbox,
}, &thread); err != nil {
- return nil, fmt.Errorf("start ephemeral thread: %w", err)
+ return "", fmt.Errorf("start ephemeral thread: %w", err)
}
if thread.Thread.ID == "" {
- return nil, fmt.Errorf("app-server returned an empty thread id")
+ return "", fmt.Errorf("app-server returned an empty thread id")
}
+ return thread.Thread.ID, nil
+}
+func (s *appServerSession) callToolOnThread(
+ ctx context.Context,
+ threadID string,
+ serverName string,
+ toolName string,
+ toolArgs map[string]any,
+) (*mcp.CallToolResult, error) {
var result mcp.CallToolResult
if err := s.request(ctx, "mcpServer/tool/call", appServerToolCallParams{
- ThreadID: thread.Thread.ID,
- Server: flags.serverName,
+ ThreadID: threadID,
+ Server: serverName,
Tool: toolName,
Arguments: toolArgs,
}, &result); err != nil {
@@ -245,6 +263,35 @@ func (s *appServerSession) callToolViaAppServer(
return &result, nil
}
+func callToolSequenceViaAppServer(
+ ctx context.Context,
+ session *appServerSession,
+ flags commonFlags,
+ calls []toolCallSpec,
+) ([]toolCallOutput, error) {
+ threadCtx, cancel := context.WithTimeout(ctx, flags.timeout)
+ threadID, err := session.startEphemeralThread(threadCtx, flags)
+ cancel()
+ if err != nil {
+ return nil, err
+ }
+
+ results := make([]toolCallOutput, 0, len(calls))
+ for i, call := range calls {
+ callCtx, cancel := context.WithTimeout(ctx, flags.timeout)
+ result, err := session.callToolOnThread(callCtx, threadID, flags.serverName, call.Tool, call.Args)
+ cancel()
+ if err != nil {
+ return nil, fmt.Errorf("call #%d %q via app-server: %w", i+1, call.Tool, err)
+ }
+ results = append(results, toolCallOutput{
+ Tool: call.Tool,
+ Result: result,
+ })
+ }
+ return results, nil
+}
+
func (s *appServerSession) request(ctx context.Context, method string, params any, out any) error {
requestID := atomic.AddInt64(&s.nextID, 1)
diff --git a/scripts/computer-use-cli/examples/textedit-overlay-seq.json b/scripts/computer-use-cli/examples/textedit-overlay-seq.json
new file mode 100644
index 0000000..96028d2
--- /dev/null
+++ b/scripts/computer-use-cli/examples/textedit-overlay-seq.json
@@ -0,0 +1,103 @@
+[
+ {
+ "tool": "list_apps"
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "set_value",
+ "args": {
+ "app": "TextEdit",
+ "element_index": "2",
+ "value": "line 01\nline 02\nline 03\nline 04\nline 05\nline 06\nline 07\nline 08\nline 09\nline 10\nline 11\nline 12\nline 13\nline 14\nline 15\nline 16\nline 17\nline 18\nline 19\nline 20"
+ }
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "click",
+ "args": {
+ "app": "TextEdit",
+ "element_index": "2"
+ }
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "type_text",
+ "args": {
+ "app": "TextEdit",
+ "text": "\n[type_text] hello from call-seq"
+ }
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "press_key",
+ "args": {
+ "app": "TextEdit",
+ "key": "Return"
+ }
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "scroll",
+ "args": {
+ "app": "TextEdit",
+ "element_index": "1",
+ "direction": "down",
+ "pages": 1
+ }
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "drag",
+ "args": {
+ "app": "TextEdit",
+ "from_x": 120,
+ "from_y": 120,
+ "to_x": 320,
+ "to_y": 120
+ }
+ },
+ {
+ "tool": "get_app_state",
+ "args": {
+ "app": "TextEdit"
+ }
+ },
+ {
+ "tool": "perform_secondary_action",
+ "args": {
+ "app": "TextEdit",
+ "element_index": "0",
+ "action": "Raise"
+ }
+ }
+]
diff --git a/scripts/computer-use-cli/main.go b/scripts/computer-use-cli/main.go
index d2fe2ec..bb27c39 100644
--- a/scripts/computer-use-cli/main.go
+++ b/scripts/computer-use-cli/main.go
@@ -47,6 +47,16 @@ type resolvedTarget struct {
ServerBin string `json:"serverBin"`
}
+type toolCallSpec struct {
+ Tool string `json:"tool"`
+ Args map[string]any `json:"args,omitempty"`
+}
+
+type toolCallOutput struct {
+ Tool string `json:"tool"`
+ Result *mcp.CallToolResult `json:"result"`
+}
+
type rpcSession struct {
conn mcp.Connection
nextID int64
@@ -86,6 +96,8 @@ func run(ctx context.Context, args []string, stdout io.Writer, stderr io.Writer)
return runListTools(ctx, args[1:], stdout, stderr)
case "call":
return runCall(ctx, args[1:], stdout, stderr)
+ case "call-seq":
+ return runCallSeq(ctx, args[1:], stdout, stderr)
case "help", "-h", "--help":
printRootUsage(stderr)
return nil
@@ -247,6 +259,66 @@ func runCall(ctx context.Context, args []string, stdout io.Writer, stderr io.Wri
}
}
+func runCallSeq(ctx context.Context, args []string, stdout io.Writer, stderr io.Writer) error {
+ var flags commonFlags
+ var callsJSON string
+ var callsFile string
+
+ fs := flag.NewFlagSet("call-seq", flag.ContinueOnError)
+ fs.SetOutput(stderr)
+ addCommonFlags(fs, &flags)
+ fs.StringVar(&callsJSON, "calls", "", "JSON array of sequential tool calls")
+ fs.StringVar(&callsFile, "calls-file", "", "Path to a file containing a JSON array of sequential tool calls")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ if fs.NArg() != 0 {
+ return fmt.Errorf("usage: %s call-seq [flags]", cliName)
+ }
+ if callsJSON != "" && callsFile != "" {
+ return fmt.Errorf("use only one of --calls or --calls-file")
+ }
+
+ calls, err := readToolCallSequence(callsJSON, callsFile)
+ if err != nil {
+ return err
+ }
+
+ transport, err := resolveTransport(flags)
+ if err != nil {
+ return err
+ }
+
+ switch transport {
+ case transportDirect:
+ session, err := connect(ctx, flags)
+ if err != nil {
+ return err
+ }
+ defer session.Close()
+
+ results, err := callToolSequenceDirect(ctx, session, flags.timeout, calls)
+ if err != nil {
+ return err
+ }
+ return writeJSON(stdout, results, flags.pretty)
+ case transportAppServer:
+ session, err := connectAppServer(ctx, flags)
+ if err != nil {
+ return err
+ }
+ defer session.Close()
+
+ results, err := callToolSequenceViaAppServer(ctx, session, flags, calls)
+ if err != nil {
+ return err
+ }
+ return writeJSON(stdout, results, flags.pretty)
+ default:
+ return fmt.Errorf("unsupported transport %q", transport)
+ }
+}
+
func addCommonFlags(fs *flag.FlagSet, flags *commonFlags) {
fs.StringVar(&flags.transport, "transport", transportAuto, "Connection mode: auto, direct, or app-server")
fs.StringVar(&flags.pluginRoot, "plugin-root", "", "Path to the installed computer-use plugin root")
@@ -324,6 +396,28 @@ func (s *rpcSession) listTools(ctx context.Context) (*mcp.ListToolsResult, error
return &result, nil
}
+func callToolSequenceDirect(
+ ctx context.Context,
+ session *rpcSession,
+ timeout time.Duration,
+ calls []toolCallSpec,
+) ([]toolCallOutput, error) {
+ results := make([]toolCallOutput, 0, len(calls))
+ for i, call := range calls {
+ callCtx, cancel := context.WithTimeout(ctx, timeout)
+ result, err := session.callTool(callCtx, call.Tool, call.Args)
+ cancel()
+ if err != nil {
+ return nil, fmt.Errorf("call #%d %q: %w", i+1, call.Tool, err)
+ }
+ results = append(results, toolCallOutput{
+ Tool: call.Tool,
+ Result: result,
+ })
+ }
+ return results, nil
+}
+
func (s *rpcSession) callTool(
ctx context.Context,
toolName string,
@@ -616,6 +710,40 @@ func readToolArgs(argsJSON, argsFile string) (map[string]any, error) {
return object, nil
}
+func readToolCallSequence(callsJSON, callsFile string) ([]toolCallSpec, error) {
+ if callsFile != "" {
+ data, err := os.ReadFile(callsFile)
+ if err != nil {
+ return nil, fmt.Errorf("read calls file %q: %w", callsFile, err)
+ }
+ callsJSON = string(data)
+ }
+
+ if strings.TrimSpace(callsJSON) == "" {
+ return nil, fmt.Errorf("missing --calls or --calls-file")
+ }
+
+ decoder := json.NewDecoder(strings.NewReader(callsJSON))
+ decoder.UseNumber()
+
+ var calls []toolCallSpec
+ if err := decoder.Decode(&calls); err != nil {
+ return nil, fmt.Errorf("decode tool call sequence JSON: %w", err)
+ }
+ if decoder.More() {
+ return nil, fmt.Errorf("tool call sequence JSON must contain exactly one array")
+ }
+ if len(calls) == 0 {
+ return nil, fmt.Errorf("tool call sequence must contain at least one call")
+ }
+ for i, call := range calls {
+ if strings.TrimSpace(call.Tool) == "" {
+ return nil, fmt.Errorf("call #%d is missing a non-empty tool name", i+1)
+ }
+ }
+ return calls, nil
+}
+
func writeJSON(w io.Writer, value any, pretty bool) error {
var (
data []byte
@@ -680,6 +808,7 @@ Usage:
%s resolve-server [flags]
%s list-tools [flags]
%s call [flags] <tool-name>
+ %s call-seq [flags]
Examples:
%s resolve-server
@@ -688,10 +817,11 @@ Examples:
%s call list_apps --transport app-server
%s call list_apps --transport direct --server-bin /path/to/open-computer-use
%s call get_app_state --args '{"app":"Feishu"}'
+ %s call-seq --transport app-server --calls-file /tmp/calls.json
Environment:
%s Override the installed plugin root.
%s Override the SkyComputerUseClient executable path.
%s Override the Codex binary used for app-server mode.
-`, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, pluginRootEnvVar, serverBinEnvVar, appServerBinEnvVar)
+`, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, cliName, pluginRootEnvVar, serverBinEnvVar, appServerBinEnvVar)
}
diff --git a/scripts/computer-use-cli/main_test.go b/scripts/computer-use-cli/main_test.go
index 0c61cfc..4bbe3bf 100644
--- a/scripts/computer-use-cli/main_test.go
+++ b/scripts/computer-use-cli/main_test.go
@@ -37,6 +37,37 @@ func TestReadToolArgsRejectsNonObject(t *testing.T) {
}
}
+func TestReadToolCallSequenceAcceptsJSONArray(t *testing.T) {
+ t.Parallel()
+
+ calls, err := readToolCallSequence(`[{"tool":"get_app_state","args":{"app":"TextEdit"}},{"tool":"scroll","args":{"pages":1}}]`, "")
+ if err != nil {
+ t.Fatalf("readToolCallSequence returned error: %v", err)
+ }
+ if got, want := len(calls), 2; got != want {
+ t.Fatalf("len(calls) = %d, want %d", got, want)
+ }
+ if got, want := calls[0].Tool, "get_app_state"; got != want {
+ t.Fatalf("calls[0].Tool = %q, want %q", got, want)
+ }
+
+ number, ok := calls[1].Args["pages"].(json.Number)
+ if !ok {
+ t.Fatalf("pages type = %T, want json.Number", calls[1].Args["pages"])
+ }
+ if got, want := number.String(), "1"; got != want {
+ t.Fatalf("pages = %q, want %q", got, want)
+ }
+}
+
+func TestReadToolCallSequenceRejectsMissingTool(t *testing.T) {
+ t.Parallel()
+
+ if _, err := readToolCallSequence(`[{"args":{"app":"TextEdit"}}]`, ""); err == nil {
+ t.Fatal("readToolCallSequence should reject a call without a tool name")
+ }
+}
+
func TestDiscoverPluginRootChoosesNewestCandidate(t *testing.T) {
tempHome := t.TempDir()
t.Setenv("HOME", tempHome)