Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(sandbox): add a how-to for running Minecraft in a Windows sandbox (#3131)
r33drichards6db0cabdf0e6
diff --git a/docs/content/docs/how-to-guides/sandbox/meta.json b/docs/content/docs/how-to-guides/sandbox/meta.jsonindex 508b827e7a..3986ad1cf1 100644--- a/docs/content/docs/how-to-guides/sandbox/meta.json+++ b/docs/content/docs/how-to-guides/sandbox/meta.json@@ -8,6 +8,7 @@"scale-out","tunneling","images",- "interactive-shell"+ "interactive-shell",+ "minecraft"]}diff --git a/docs/content/docs/how-to-guides/sandbox/minecraft.mdx b/docs/content/docs/how-to-guides/sandbox/minecraft.mdxnew file mode 100644index 0000000000..e98f2119bc--- /dev/null+++ b/docs/content/docs/how-to-guides/sandbox/minecraft.mdx@@ -0,0 +1,368 @@+---+title: Run Minecraft in a Windows sandbox+description: Boot a Windows sandbox, install Minecraft Java Edition, and drive it with an agent through the cua-driver MCP server running inside the sandbox.+---++import { Callout } from 'fumadocs-ui/components/callout';++Minecraft exercises almost everything a Windows sandbox can do: it needs internet access, a Java runtime, working OpenGL, and a GUI that only clicks can drive. This guide boots a Windows sandbox, installs Minecraft Java Edition, and hands it to an agent that talks to **cua-driver's MCP server inside the sandbox** — the same loop against a local sandbox and against Fleet.++## Before you start++- **cua-sandbox 0.3.2 or newer.** Windows on Fleet needs 0.3.0, `Image.expose()` on the local QEMU runtime landed in 0.3.1, and the `sb.exposed_ports` this guide reads the forwarded port from landed in 0.3.2.+- **A host with hardware virtualisation** for the local path — a Linux x86_64 machine with `/dev/kvm`, or an Intel Mac. This guide passes `-cpu host`, which QEMU only accepts with KVM or HVF. An x86_64 guest on Apple Silicon runs under TCG emulation, where `-cpu host` is rejected outright. The Fleet path runs there instead, including the game, with the one extra environment variable described in the Fleet section below.+- **A Microsoft account that owns Minecraft Java Edition.** Signing in uses Microsoft device authorization, so one step in the middle is manual: a code appears inside the sandbox and you approve it in your own browser.+- **A vision-capable LLM endpoint** for the agent loop.++## Boot a Windows sandbox++`Image.windows()` resolves to a pinned Windows Server 2022 containerDisk. Three things get added on top of the defaults:++- **`.expose(3000)`** publishes cua-driver's MCP server, which already runs inside the guest, so the agent can reach it.+- **A second network interface.** The bare-metal runtime attaches its NIC with `restrict=on`, which isolates the guest. `sb.shell.run()` still works over the forwarded port, but nothing inside Windows can reach the internet — and Minecraft needs to.+- **`-cpu host`.** The default `qemu64` model is too thin for a software OpenGL driver: Minecraft creates its window and then dies while loading resources, with no Java exception and no crash log. The last `-cpu` on the command line wins, so appending it is enough.++```python+import asyncio+from cua import Image, QEMURuntime, Sandbox++EXTRA_ARGS = [+ # a second, unrestricted user-mode NIC — the default one is restrict=on+ '-netdev', 'user,id=net1,net=10.0.3.0/24,host=10.0.3.2,dns=10.0.3.3',+ '-device', 'virtio-net-pci,netdev=net1,mac=52:55:00:d1:55:02',+ # a CPU the software OpenGL driver can actually use+ '-cpu', 'host',+]++async def main():+ sb = await Sandbox.create(+ Image.windows().expose(3000),+ name='mc-win',+ local=True,+ runtime=QEMURuntime(+ mode='bare-metal',+ cpu_count=12,+ memory_mb=16384,+ extra_args=EXTRA_ARGS,+ ),+ )+ mcp_port = sb.exposed_ports[3000]+ print(f'cua-driver MCP on http://127.0.0.1:{mcp_port}/mcp')+ await sb.disconnect() # the sandbox keeps running++asyncio.run(main())+```++A warm boot takes about 30 seconds. `exposed_ports` maps each exposed guest port to the host port it landed on, and `GET /healthz` on that port answers `ok` once cua-driver is up.++<Callout type="warn">+ **Read the port from `sb.exposed_ports`, not from a tunnel.** `sb.tunnel.forward(3000)` — the usual way to get a forwarded port, and the one the Fleet section below uses — raises `NotImplementedError: HTTPTransport does not support port forwarding` on the local transport. `exposed_ports` is the local equivalent: the runtime picks a free host port at boot, so the mapping is only knowable at runtime, and it is saved with the sandbox state so a later `Sandbox.connect()` can read it back. On Fleet the property is empty, because Fleet publishes services instead — use `tunnel.forward()` there.+</Callout>++<Callout type="warn">+ Give the second NIC its own subnet. Both user-mode networks default to `10.0.2.0/24` and both offer the guest `10.0.2.15`, so Windows drops one interface to a `169.254.x.x` link-local address with no gateway and no working DNS.+</Callout>++Confirm the guest really has internet before installing anything.++```python+async with Sandbox.connect('mc-win', local=True) as sb:+ check = await sb.shell.run(+ 'powershell -Command "(Invoke-WebRequest -UseBasicParsing '+ 'https://piston-meta.mojang.com/mc/game/version_manifest.json).StatusCode"'+ )+ print(check.stdout) # 200+```++## Install a launcher and a software OpenGL driver++The sandbox GPU is the *Microsoft Basic Display Adapter*, which offers OpenGL 1.1. Minecraft 1.17 and later need OpenGL 3.2, so the game needs Mesa3D's `opengl32.dll` (llvmpipe), which implements OpenGL in software.++Both downloads below are **MinGW** builds on purpose. The MSVC builds of Prism Launcher and Mesa both depend on the Visual C++ redistributable, which Windows Server 2022 does not ship: Prism then exits silently, and Mesa's DLL fails to load so Windows quietly falls back to the system `opengl32.dll`.++```powershell+$ErrorActionPreference = 'Stop'+$ProgressPreference = 'SilentlyContinue'+New-Item -ItemType Directory -Force -Path C:\mc | Out-Null++# Prism Launcher — signs in with Microsoft device authorization, which needs no browser+Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\prism.zip `+ 'https://github.com/PrismLauncher/PrismLauncher/releases/download/11.0.3/PrismLauncher-Windows-MinGW-w64-Portable-11.0.3.zip'+Expand-Archive C:\mc\prism.zip -DestinationPath C:\mc\prismw -Force++# 7-Zip, because Mesa ships as .7z+Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\7z.msi 'https://www.7-zip.org/a/7z2408-x64.msi'+Start-Process msiexec.exe -ArgumentList '/i','C:\mc\7z.msi','/qn' -Wait++# Mesa3D software OpenGL+Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\mesa.7z `+ 'https://github.com/pal1000/mesa-dist-win/releases/download/26.1.6/mesa3d-26.1.6-release-mingw.7z'+& 'C:\Program Files\7-Zip\7z.exe' x C:\mc\mesa.7z -oC:\mc\mesamw -y | Out-Null++Start-Process -FilePath C:\mc\prismw\prismlauncher.exe -WorkingDirectory C:\mc\prismw+```++Save that as `setup.ps1`, push it into the sandbox, and run it. It downloads roughly 100 MB, so allow a generous timeout.++```python+from pathlib import Path++async with Sandbox.connect('mc-win', local=True) as sb:+ await sb.shell.run('if not exist C:\\mc mkdir C:\\mc')+ await sb.files.write_text('C:\\mc\\setup.ps1', Path('setup.ps1').read_text())+ result = await sb.shell.run(+ 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\mc\\setup.ps1',+ timeout=1800,+ )+ print(result.stdout)+```++## Sign in and create an instance++Prism opens a **Quick Setup** wizard on first run. Screenshot the sandbox, click through it, and stop at the account page.++```python+async with Sandbox.connect('mc-win', local=True) as sb:+ Path('sandbox.png').write_bytes(await sb.screenshot()) # look at it+ await sb.mouse.click(888, 678) # Next+```++1. Work through the wizard to **Accounts → Add Microsoft**. Prism shows a QR code and an eight-character device code.+2. Read the code off a screenshot, open `https://www.microsoft.com/link` in your own browser, enter it, and approve the sign-in. The account then appears with status *Ready*.+3. Click **Add Instance**, search for a version such as `1.20.1`, and click **OK**. Prism downloads the client jar and assets.++<Callout type="info">+ Device codes expire after about fifteen minutes, but Prism issues a fresh one automatically and keeps polling, so the dialog can be left open. Take a new screenshot to read the current code rather than reusing an old one.+</Callout>++## Point the software driver at the launcher's Java++Click **Launch** once. Prism downloads its own Java runtime and the game fails with `GLFW error 65542: WGL: The driver does not appear to support OpenGL` — expected, because Mesa is not in place yet.++Prism may keep using the runtime it downloaded even if you set `JavaPath` in its config, so copy the Mesa DLLs next to *every* `javaw.exe` under the install root. Windows loads `opengl32.dll` from the running executable's directory before the system directory, which is what makes this work.++```powershell+$dirs = Get-ChildItem C:\mc -Recurse -Filter javaw.exe -ErrorAction SilentlyContinue |+ Select-Object -ExpandProperty DirectoryName -Unique+foreach ($d in $dirs) {+ Copy-Item C:\mc\mesamw\x64\opengl32.dll, C:\mc\mesamw\x64\libgallium_wgl.dll $d -Force+ Write-Output "mesa -> $d"+}+```++Deliver it the same way as the first script.++```python+async with Sandbox.connect('mc-win', local=True) as sb:+ await sb.files.write_text('C:\\mc\\mesa.ps1', Path('mesa.ps1').read_text())+ result = await sb.shell.run(+ 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\mc\\mesa.ps1', timeout=600,+ )+ print(result.stdout) # mesa -> C:\mc\prismw\java\java-runtime-gamma\bin+```++Click **Launch** again. The Minecraft title screen appears after a minute or two.++## Drive it with an agent over MCP++The sandbox already runs **cua-driver**, which serves an MCP endpoint on guest port 3000 — that is what `.expose(3000)` published. The agent is a small loop: list the MCP tools, hand them to a model as ordinary function tools, call whichever one it picks, feed the result back.++Three things about cua-driver's tools shape the loop:++- **A YAML policy governs which tools may actually run, and `list_tools()` does not reflect it.** Every cua-driver release to date advertises the full surface and refuses out-of-policy calls only when you make them, with `Permission denied: user policy: tool 'X' is not allowed by the YAML policy`. So the listing is a menu of what exists, not of what you can call. Here that surface was 55 tools, identically over the local and Fleet transports: `get_desktop_state`, `list_apps`, `list_windows`, `get_window_state`, `click`, `double_click`, `type_text`, `press_key`, `hotkey`, `launch_app`, `bring_to_front`, `scroll` and `drag` ran, while `get_screen_size`, `get_accessibility_tree`, `get_config`, `check_permissions`, `get_cursor_position` and `zoom` were refused. Treat that split as something to probe on your own image rather than a fixed list — a denial arrives before the tool executes, so probing is cheap. Later drivers filter the listing through the policy, at which point the two finally agree.+- **Clicks are addressed to an application, not the screen.** `click(pid=..., x=..., y=...)` targets a window belonging to that pid, which you find with `list_windows`. Add `delivery_mode='foreground'` when a background-delivered click does not land.+- **There is no wait tool.** The loop waits by calling `get_desktop_state` again, so say that in the system prompt or the model will invent something worse.++```python+import asyncio, json, os+import litellm+from fastmcp import Client+from fastmcp.client.transports import StreamableHttpTransport++SYSTEM = """You operate a computer through the provided tools.++The desktop is Windows at 1280x800. Work in a loop: look at the screen with+get_desktop_state, decide one action, call one tool, then look again.++ * click / type_text / press_key act on a specific application, addressed by+ `pid`. Use list_windows to find the pid, then pass pid with x/y.+ * There is no wait tool. If something is still loading, call+ get_desktop_state again — repeated looks are how you wait.++Call exactly one tool per turn. When the task is complete, reply DONE."""++async def complete(**kwargs):+ """Two workarounds for the endpoint used here — yours may need neither.++ It is streaming-only (a plain request comes back with empty output), and it+ rejects role=system, so the system prompt travels as the first user turn.+ """+ messages, system = [], []+ for m in kwargs['messages']:+ (system if m.get('role') == 'system' else messages).append(m)+ if system:+ text = '\n\n'.join(m['content'] for m in system)+ messages = [{'role': 'user', 'content': text}] + messages+ kwargs['messages'] = messages++ stream = await litellm.acompletion(**kwargs, stream=True)+ chunks = [c async for c in stream]+ return litellm.stream_chunk_builder(chunks, messages=messages)++def prune_images(messages, keep=3):+ """Each get_desktop_state returns a full screenshot; keep only the newest."""+ seen = 0+ for msg in reversed(messages):+ if not isinstance(msg.get('content'), list):+ continue+ for part in msg['content']:+ if part.get('type') == 'image_url':+ seen += 1+ if seen > keep:+ part.clear()+ part.update({'type': 'text', 'text': '[older screenshot dropped]'})+ return messages++async def run(mcp_url, task, model, max_steps=60, headers=None):+ client = Client(StreamableHttpTransport(mcp_url, headers=headers))+ async with client:+ # list_tools() returned 55 tools on the image used here, most of them+ # browser and recording plumbing this task never needs. Hand the model+ # only what the job requires: dozens of schemas is a lot of context to+ # spend, and a shorter menu is a shorter list of ways to go wrong. Worth+ # doing whether or not your driver already filters denied tools out.+ wanted = {+ 'get_desktop_state', 'list_windows', 'list_apps', 'click',+ 'double_click', 'type_text', 'press_key', 'hotkey', 'launch_app',+ 'bring_to_front', 'scroll',+ }+ mcp_tools = [t for t in await client.list_tools() if t.name in wanted]+ tools = [{+ 'type': 'function',+ 'function': {+ 'name': t.name,+ 'description': (t.description or '')[:800],+ 'parameters': t.inputSchema or {'type': 'object', 'properties': {}},+ },+ } for t in mcp_tools]++ messages = [{'role': 'system', 'content': SYSTEM},+ {'role': 'user', 'content': task}]++ for _ in range(max_steps):+ resp = await complete(+ model=model, messages=prune_images(messages), tools=tools,+ tool_choice='auto', temperature=0.0,+ )+ msg = resp.choices[0].message+ messages.append(msg.model_dump())+ if not msg.tool_calls:+ break # model said DONE++ for call in msg.tool_calls:+ args = json.loads(call.function.arguments or '{}')+ result = await client.call_tool(call.function.name, args,+ raise_on_error=False)+ text = ''.join(getattr(b, 'text', '') for b in (result.content or []))+ messages.append({'role': 'tool', 'tool_call_id': call.id,+ 'name': call.function.name, 'content': text[:1500]})++ shot = next((b.data for b in (result.content or [])+ if getattr(b, 'data', None)), None)+ if shot:+ messages.append({'role': 'user', 'content': [+ {'type': 'text', 'text': 'screenshot after that action:'},+ {'type': 'image_url',+ 'image_url': {'url': f'data:image/png;base64,{shot}'}},+ ]})+```++Point it at the exposed port and give it the task.++```python+TASK = """+Prism Launcher is open, with a Minecraft instance and a signed-in account.+Select the instance and click Launch. Minecraft uses a software renderer, so the+window takes minutes to appear and repaints slowly — keep calling+get_desktop_state to watch it, and do not restart anything. On the title screen+click Singleplayer, then Create New World, then Create New World again. Stop as+soon as you are inside the world (terrain in first person, hotbar and hearts+visible) and reply DONE. Never press Escape while Minecraft is in the foreground.+"""++asyncio.run(run(f'http://127.0.0.1:{mcp_port}/mcp', TASK, 'your-model'))+```++Because the MCP tools are presented as **ordinary function tools**, this works against endpoints that reject the provider-native computer-use tool types. That is not hypothetical: on the gateway used here, the same model with the same image in the same second returned 200 for an ordinary function tool and 403 for Anthropic's `computer_20250124`, and `computer_use_preview` was refused outright.++The `complete()` wrapper above exists only for that gateway — it is streaming-only, and it rejects `role: system`. Against an endpoint without those quirks, call `litellm.acompletion` directly.++A full run — launcher to standing in a new world — took 52 steps locally and 51 on Fleet, roughly twenty minutes, most of it waiting on the software renderer. Expect the model to spend long stretches doing nothing but re-screenshotting.++<Callout type="info">+ Give the model help with coordinates. A vision model without grounding guesses pixel positions and misses: in one run an ungrounded model clicked at (1226, 210) four times, nowhere near the button it wanted, then declared it had no desktop tool. cua-driver's `list_windows` and pid-scoped clicks avoid most of this, and a grounding pass over the screenshot removes the rest.+</Callout>++## Run the same thing on Fleet++The image and the agent loop are identical on Fleet. Two things change: there is no `local=True` and no `runtime=`, and the MCP endpoint is reached through Fleet's service proxy rather than a forwarded localhost port.++`.expose(3000)` becomes a Fleet **service** named `port-3000`, and `sb.tunnel.forward()` hands you its URL — the same call documented in [Forward a port from a sandbox](/how-to-guides/sandbox/tunneling), so there is nothing Fleet-specific to hand-assemble.++```python+import httpx+from cua import Image, Sandbox++sb = await Sandbox.create(Image.windows().expose(3000), name='mc-fleet', time_to_start=900)++tunnel = await sb.tunnel.forward(3000)+# https://run.cua.ai/api/svc/<namespace>/<sandbox>-port-3000/++token = httpx.post(+ 'https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token',+ data={'grant_type': 'client_credentials',+ 'client_id': os.environ['CUA_CLIENT_ID'],+ 'client_secret': os.environ['CUA_CLIENT_SECRET']},+).json()['access_token']++await run(tunnel.url + 'mcp', TASK, 'your-model',+ headers={'Authorization': f'Bearer {token}'})+```++That is the same `run()` as above, with only the URL and an auth header changed. A Fleet Windows sandbox takes about three minutes to become ready, against about thirty seconds for a warm local one, and `GET healthz` on that service URL answers `ok` when cua-driver is up.++Notice there is no `-cpu host` here. That flag exists on the local path because the bare-metal runtime defaults to the thin `qemu64` model; a Fleet sandbox is provisioned for you and already reports a full host CPU — `Intel Xeon Processor (SapphireRapids)`, with AVX, AVX2 and AVX-512 all present — so there is nothing to override, and no QEMU arguments to pass.++`Sandbox.create` also accepts `cpu` and `memory_mb`, but sizing a Fleet sandbox is account-dependent: passing them routes the request through a gated custom-resource path, which returned `403 create pool: k8s request is not allowed` at every size until a card-on-file requirement was lifted for the account. The default sandbox — 4 vCPU and 4 GB — is what this guide was written against, and it is enough.++<Callout type="warn">+ **On Fleet the game needs one extra environment variable: `GALLIUM_DRIVER=softpipe`.** With Mesa's default llvmpipe renderer, Minecraft dies during resource loading every time — `Process crashed with exitcode -2147024809`, no Java exception, no `hs_err` file, nothing in the Windows event log, the log simply stopping after `Reloading ResourceManager`. Switching Mesa to its `softpipe` rasteriser fixes it, and the game runs.++ Set the variable in the process that launches the launcher, so the game inherits it — a machine-level variable does not reach an already-running process, and a test that silently did not apply looks exactly like a test that failed. `setup.ps1` above already started Prism without it, so close the running launcher first and start it again like this.++ ```powershell+ $env:GALLIUM_DRIVER = 'softpipe'+ Start-Process -FilePath C:\mc\prismw\prismlauncher.exe -WorkingDirectory C:\mc\prismw+ ```++ Softpipe is a reference rasteriser with no JIT, so it is **considerably slower than llvmpipe** — allow several minutes for the title screen and longer again for world generation.+</Callout>++<Callout type="warn">+ Use client credentials, not a `cua auth login` session token. The session token is short-lived and is held without refresh, so a long provisioning wait dies partway with `401 auth token is invalid`. Client-credential tokens expire too — the ones issued here came back with `expires_in` of 900 seconds — so a run longer than that has to re-mint the token and rebuild the MCP client.+</Callout>++What that crash is *not*, since each obvious explanation was tested and eliminated: not the Minecraft version or the Java/LWJGL generation (1.20.1 on Java 17 with LWJGL 3 and 1.12.2 on Java 8 with LWJGL 2 fail identically), not the heap (forcing Prism's auto-sized `-Xmx2717m` down to `-Xmx1024m` changed nothing), and not the size of the machine (the same local session disk rebooted with `-m 4096 -smp 4`, matching Fleet exactly, runs the game fine). Narrowing llvmpipe's vectors with `LP_NATIVE_VECTOR_WIDTH=128` did **not** help either, which argues the fault is not simply wide-vector code generation.++## Troubleshooting++| Symptom | Cause | Fix |+|---------|-------|-----|+| QEMU refuses to start with `-cpu host` | no KVM or HVF — for example an x86_64 guest on Apple Silicon, which runs under TCG | use a host with hardware virtualisation, or the Fleet path |+| Launcher never appears, no error | MSVC build without the VC++ redistributable | use the MinGW portable build |+| Guest has an IP address but cannot resolve names | both user-mode NICs offered the same address | give the second NIC its own subnet |+| `GLFW error 65542: WGL: The driver does not appear to support OpenGL` | Mesa DLLs missing beside the `javaw.exe` actually in use, or the MSVC Mesa build failed to load | copy the MinGW Mesa DLLs into every `javaw.exe` directory |+| Game exits during resource loading with no Java exception, **local** | default `qemu64` CPU model | append `-cpu host` to `extra_args` |+| Game exits during resource loading with `exitcode -2147024809`, **Fleet** | Mesa's default llvmpipe renderer | set `GALLIUM_DRIVER=softpipe` in the process that starts the launcher, and restart the launcher if it is already running |+| `Permission denied: user policy: tool 'X' is not allowed` | cua-driver's YAML policy refuses that tool, which `list_tools()` advertises anyway on every driver released so far | use an allowed tool — `get_desktop_state` instead of `get_screen_size`, `list_windows` instead of `get_accessibility_tree` |+| Model replies with empty output on the first call | endpoint is streaming-only | issue `stream=True` and rebuild with `litellm.stream_chunk_builder` |