Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

4.1 KB
Raw Blame

Agents in sandboxes

Run coding-agent harness X in image Y over a task file, fire-and-forget, then collect verified results into a JSON report. Uses the cua SDK's agents API (sandbox.agents()); see Run a coding agent in a sandbox.

File What
agents_in_sandboxes.py launch, collect, or run (both)
tasks.jsonl Two tiny tasks, each with a shell check
tour.py The docs page's code as one script
tests/ Hermetic tests for the pure logic

Tasks

One JSON object per line:

{"id": "hello", "prompt": "Create a file hello.txt whose only line is: hello", "check": "grep -qx hello hello.txt"}

Each task runs in its own directory (--workdir, default /tmp/agents-in-sandboxes/<id>). check is the verifier: a shell command run there inside the sandbox after the run finishes; exit 0 passes. mock is a script for the mock provider (see below). timeout_s caps how long collect waits for the task.

Run it with a real key

pip install cua
export ANTHROPIC_API_KEY=...
python agents_in_sandboxes.py launch --image ghcr.io/trycua/linux:24.04 \
  --harness claude-code --tasks tasks.jsonl --key-var ANTHROPIC_API_KEY \
  --mcp docs=https://mcp.example.com/mcp
# ... later, from any process:
python agents_in_sandboxes.py collect --wait 600 --out report.json --delete

launch creates the sandbox, installs the harness, starts one run per task with exit_when_idle and writes runs.json. The runs keep going without it. collect reconnects, finds the runs (agents.list(), by label), waits up to --wait seconds for each, reads run.result() and run.artifacts(), runs each check, and writes report.json:

{"image": "...", "harness": "claude-code", "sandbox": "local:agents-in-sandboxes",
 "summary": {"tasks": 2, "finished": 2, "pending": 0, "passed": 2, "failed": 0, ...},
 "tasks": [{"id": "hello", "run_id": "run-...", "status": "idle", "stop_reason": "end_turn",
            "text": "...", "usage": {"input_tokens": ...}, "artifacts": [...], "verified": true}]}

Other options: --image takes any image with cua-spacesd (a registry image or a benchmark image), --connect local:NAME reuses a sandbox, --harness takes any id from cua agent harnesses, --base-url/--model point it at a proxy or gateway, --no-sandbox-mcp drops the sandbox's own MCP tools. The exit code is 1 when a check failed.

Run it against the mock provider

cua-mock-llm (in libs/cua) is a scripted provider that speaks the Anthropic, OpenAI and Gemini wire formats. With --scripted, each task's mock field is appended to its prompt as a mock: directive, and the mock makes the harness run that shell command for real in the sandbox. The model replies are scripted, not generated; the harness, its install, its tools, the MCP wiring and the verifiers are real.

# build the mock (Linux binary) and start it where the sandbox can reach it
cd libs/cua && cargo zigbuild --release -p cua-mock-llm --target aarch64-unknown-linux-musl
MOCK_KEY=mock-$RANDOM$RANDOM
docker run -d --name cua-e2e-mock-llm --memory=256m -e CUA_MOCK_LLM_KEY=$MOCK_KEY \
  -v "$PWD/target/aarch64-unknown-linux-musl/release/cua-mock-llm:/usr/local/bin/cua-mock-llm:ro" \
  debian:bookworm-slim cua-mock-llm --listen 0.0.0.0:8787
MOCK_IP=$(docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' cua-e2e-mock-llm)

cd examples/agents-in-sandboxes
ANTHROPIC_API_KEY=$MOCK_KEY python agents_in_sandboxes.py run --name cua-e2e-ais \
  --harness claude-code --tasks tasks.jsonl --key-var ANTHROPIC_API_KEY \
  --base-url http://$MOCK_IP:8787 --model claude-mock-1 --scripted --delete
docker rm -f cua-e2e-mock-llm

The same flags run tour.py against the mock: ANTHROPIC_API_KEY=$MOCK_KEY OPENAI_API_KEY=$MOCK_KEY python tour.py --base-url http://$MOCK_IP:8787.

Tests

uv run --no-project --with pytest pytest examples/agents-in-sandboxes/tests -q

They cover task parsing, MCP flags, the verifier line, usage parsing and the report, and check statically that every SDK call here and in tour.py exists in the generated Python binding.