Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

docs(sandbox): add a how-to for running Minecraft in a Windows sandbox (#3131)

r33drichards6db0cabdf0e6
diff --git a/docs/content/docs/how-to-guides/sandbox/meta.json b/docs/content/docs/how-to-guides/sandbox/meta.json
index 508b827e7a..3986ad1cf1 100644
--- a/docs/content/docs/how-to-guides/sandbox/meta.json
+++ b/docs/content/docs/how-to-guides/sandbox/meta.json
@@ -8,6 +8,7 @@
"scale-out",
"tunneling",
"images",
- "interactive-shell"
+ "interactive-shell",
+ "minecraft"
]
}
diff --git a/docs/content/docs/how-to-guides/sandbox/minecraft.mdx b/docs/content/docs/how-to-guides/sandbox/minecraft.mdx
new file mode 100644
index 0000000000..e98f2119bc
--- /dev/null
+++ b/docs/content/docs/how-to-guides/sandbox/minecraft.mdx
@@ -0,0 +1,368 @@
+---
+title: Run Minecraft in a Windows sandbox
+description: Boot a Windows sandbox, install Minecraft Java Edition, and drive it with an agent through the cua-driver MCP server running inside the sandbox.
+---
+
+import { Callout } from 'fumadocs-ui/components/callout';
+
+Minecraft exercises almost everything a Windows sandbox can do: it needs internet access, a Java runtime, working OpenGL, and a GUI that only clicks can drive. This guide boots a Windows sandbox, installs Minecraft Java Edition, and hands it to an agent that talks to **cua-driver's MCP server inside the sandbox** — the same loop against a local sandbox and against Fleet.
+
+## Before you start
+
+- **cua-sandbox 0.3.2 or newer.** Windows on Fleet needs 0.3.0, `Image.expose()` on the local QEMU runtime landed in 0.3.1, and the `sb.exposed_ports` this guide reads the forwarded port from landed in 0.3.2.
+- **A host with hardware virtualisation** for the local path — a Linux x86_64 machine with `/dev/kvm`, or an Intel Mac. This guide passes `-cpu host`, which QEMU only accepts with KVM or HVF. An x86_64 guest on Apple Silicon runs under TCG emulation, where `-cpu host` is rejected outright. The Fleet path runs there instead, including the game, with the one extra environment variable described in the Fleet section below.
+- **A Microsoft account that owns Minecraft Java Edition.** Signing in uses Microsoft device authorization, so one step in the middle is manual: a code appears inside the sandbox and you approve it in your own browser.
+- **A vision-capable LLM endpoint** for the agent loop.
+
+## Boot a Windows sandbox
+
+`Image.windows()` resolves to a pinned Windows Server 2022 containerDisk. Three things get added on top of the defaults:
+
+- **`.expose(3000)`** publishes cua-driver's MCP server, which already runs inside the guest, so the agent can reach it.
+- **A second network interface.** The bare-metal runtime attaches its NIC with `restrict=on`, which isolates the guest. `sb.shell.run()` still works over the forwarded port, but nothing inside Windows can reach the internet — and Minecraft needs to.
+- **`-cpu host`.** The default `qemu64` model is too thin for a software OpenGL driver: Minecraft creates its window and then dies while loading resources, with no Java exception and no crash log. The last `-cpu` on the command line wins, so appending it is enough.
+
+```python
+import asyncio
+from cua import Image, QEMURuntime, Sandbox
+
+EXTRA_ARGS = [
+ # a second, unrestricted user-mode NIC — the default one is restrict=on
+ '-netdev', 'user,id=net1,net=10.0.3.0/24,host=10.0.3.2,dns=10.0.3.3',
+ '-device', 'virtio-net-pci,netdev=net1,mac=52:55:00:d1:55:02',
+ # a CPU the software OpenGL driver can actually use
+ '-cpu', 'host',
+]
+
+async def main():
+ sb = await Sandbox.create(
+ Image.windows().expose(3000),
+ name='mc-win',
+ local=True,
+ runtime=QEMURuntime(
+ mode='bare-metal',
+ cpu_count=12,
+ memory_mb=16384,
+ extra_args=EXTRA_ARGS,
+ ),
+ )
+ mcp_port = sb.exposed_ports[3000]
+ print(f'cua-driver MCP on http://127.0.0.1:{mcp_port}/mcp')
+ await sb.disconnect() # the sandbox keeps running
+
+asyncio.run(main())
+```
+
+A warm boot takes about 30 seconds. `exposed_ports` maps each exposed guest port to the host port it landed on, and `GET /healthz` on that port answers `ok` once cua-driver is up.
+
+<Callout type="warn">
+ **Read the port from `sb.exposed_ports`, not from a tunnel.** `sb.tunnel.forward(3000)` — the usual way to get a forwarded port, and the one the Fleet section below uses — raises `NotImplementedError: HTTPTransport does not support port forwarding` on the local transport. `exposed_ports` is the local equivalent: the runtime picks a free host port at boot, so the mapping is only knowable at runtime, and it is saved with the sandbox state so a later `Sandbox.connect()` can read it back. On Fleet the property is empty, because Fleet publishes services instead — use `tunnel.forward()` there.
+</Callout>
+
+<Callout type="warn">
+ Give the second NIC its own subnet. Both user-mode networks default to `10.0.2.0/24` and both offer the guest `10.0.2.15`, so Windows drops one interface to a `169.254.x.x` link-local address with no gateway and no working DNS.
+</Callout>
+
+Confirm the guest really has internet before installing anything.
+
+```python
+async with Sandbox.connect('mc-win', local=True) as sb:
+ check = await sb.shell.run(
+ 'powershell -Command "(Invoke-WebRequest -UseBasicParsing '
+ 'https://piston-meta.mojang.com/mc/game/version_manifest.json).StatusCode"'
+ )
+ print(check.stdout) # 200
+```
+
+## Install a launcher and a software OpenGL driver
+
+The sandbox GPU is the *Microsoft Basic Display Adapter*, which offers OpenGL 1.1. Minecraft 1.17 and later need OpenGL 3.2, so the game needs Mesa3D's `opengl32.dll` (llvmpipe), which implements OpenGL in software.
+
+Both downloads below are **MinGW** builds on purpose. The MSVC builds of Prism Launcher and Mesa both depend on the Visual C++ redistributable, which Windows Server 2022 does not ship: Prism then exits silently, and Mesa's DLL fails to load so Windows quietly falls back to the system `opengl32.dll`.
+
+```powershell
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+New-Item -ItemType Directory -Force -Path C:\mc | Out-Null
+
+# Prism Launcher — signs in with Microsoft device authorization, which needs no browser
+Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\prism.zip `
+ 'https://github.com/PrismLauncher/PrismLauncher/releases/download/11.0.3/PrismLauncher-Windows-MinGW-w64-Portable-11.0.3.zip'
+Expand-Archive C:\mc\prism.zip -DestinationPath C:\mc\prismw -Force
+
+# 7-Zip, because Mesa ships as .7z
+Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\7z.msi 'https://www.7-zip.org/a/7z2408-x64.msi'
+Start-Process msiexec.exe -ArgumentList '/i','C:\mc\7z.msi','/qn' -Wait
+
+# Mesa3D software OpenGL
+Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\mesa.7z `
+ 'https://github.com/pal1000/mesa-dist-win/releases/download/26.1.6/mesa3d-26.1.6-release-mingw.7z'
+& 'C:\Program Files\7-Zip\7z.exe' x C:\mc\mesa.7z -oC:\mc\mesamw -y | Out-Null
+
+Start-Process -FilePath C:\mc\prismw\prismlauncher.exe -WorkingDirectory C:\mc\prismw
+```
+
+Save that as `setup.ps1`, push it into the sandbox, and run it. It downloads roughly 100 MB, so allow a generous timeout.
+
+```python
+from pathlib import Path
+
+async with Sandbox.connect('mc-win', local=True) as sb:
+ await sb.shell.run('if not exist C:\\mc mkdir C:\\mc')
+ await sb.files.write_text('C:\\mc\\setup.ps1', Path('setup.ps1').read_text())
+ result = await sb.shell.run(
+ 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\mc\\setup.ps1',
+ timeout=1800,
+ )
+ print(result.stdout)
+```
+
+## Sign in and create an instance
+
+Prism opens a **Quick Setup** wizard on first run. Screenshot the sandbox, click through it, and stop at the account page.
+
+```python
+async with Sandbox.connect('mc-win', local=True) as sb:
+ Path('sandbox.png').write_bytes(await sb.screenshot()) # look at it
+ await sb.mouse.click(888, 678) # Next
+```
+
+1. Work through the wizard to **Accounts → Add Microsoft**. Prism shows a QR code and an eight-character device code.
+2. Read the code off a screenshot, open `https://www.microsoft.com/link` in your own browser, enter it, and approve the sign-in. The account then appears with status *Ready*.
+3. Click **Add Instance**, search for a version such as `1.20.1`, and click **OK**. Prism downloads the client jar and assets.
+
+<Callout type="info">
+ Device codes expire after about fifteen minutes, but Prism issues a fresh one automatically and keeps polling, so the dialog can be left open. Take a new screenshot to read the current code rather than reusing an old one.
+</Callout>
+
+## Point the software driver at the launcher's Java
+
+Click **Launch** once. Prism downloads its own Java runtime and the game fails with `GLFW error 65542: WGL: The driver does not appear to support OpenGL` — expected, because Mesa is not in place yet.
+
+Prism may keep using the runtime it downloaded even if you set `JavaPath` in its config, so copy the Mesa DLLs next to *every* `javaw.exe` under the install root. Windows loads `opengl32.dll` from the running executable's directory before the system directory, which is what makes this work.
+
+```powershell
+$dirs = Get-ChildItem C:\mc -Recurse -Filter javaw.exe -ErrorAction SilentlyContinue |
+ Select-Object -ExpandProperty DirectoryName -Unique
+foreach ($d in $dirs) {
+ Copy-Item C:\mc\mesamw\x64\opengl32.dll, C:\mc\mesamw\x64\libgallium_wgl.dll $d -Force
+ Write-Output "mesa -> $d"
+}
+```
+
+Deliver it the same way as the first script.
+
+```python
+async with Sandbox.connect('mc-win', local=True) as sb:
+ await sb.files.write_text('C:\\mc\\mesa.ps1', Path('mesa.ps1').read_text())
+ result = await sb.shell.run(
+ 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\mc\\mesa.ps1', timeout=600,
+ )
+ print(result.stdout) # mesa -> C:\mc\prismw\java\java-runtime-gamma\bin
+```
+
+Click **Launch** again. The Minecraft title screen appears after a minute or two.
+
+## Drive it with an agent over MCP
+
+The sandbox already runs **cua-driver**, which serves an MCP endpoint on guest port 3000 — that is what `.expose(3000)` published. The agent is a small loop: list the MCP tools, hand them to a model as ordinary function tools, call whichever one it picks, feed the result back.
+
+Three things about cua-driver's tools shape the loop:
+
+- **A YAML policy governs which tools may actually run, and `list_tools()` does not reflect it.** Every cua-driver release to date advertises the full surface and refuses out-of-policy calls only when you make them, with `Permission denied: user policy: tool 'X' is not allowed by the YAML policy`. So the listing is a menu of what exists, not of what you can call. Here that surface was 55 tools, identically over the local and Fleet transports: `get_desktop_state`, `list_apps`, `list_windows`, `get_window_state`, `click`, `double_click`, `type_text`, `press_key`, `hotkey`, `launch_app`, `bring_to_front`, `scroll` and `drag` ran, while `get_screen_size`, `get_accessibility_tree`, `get_config`, `check_permissions`, `get_cursor_position` and `zoom` were refused. Treat that split as something to probe on your own image rather than a fixed list — a denial arrives before the tool executes, so probing is cheap. Later drivers filter the listing through the policy, at which point the two finally agree.
+- **Clicks are addressed to an application, not the screen.** `click(pid=..., x=..., y=...)` targets a window belonging to that pid, which you find with `list_windows`. Add `delivery_mode='foreground'` when a background-delivered click does not land.
+- **There is no wait tool.** The loop waits by calling `get_desktop_state` again, so say that in the system prompt or the model will invent something worse.
+
+```python
+import asyncio, json, os
+import litellm
+from fastmcp import Client
+from fastmcp.client.transports import StreamableHttpTransport
+
+SYSTEM = """You operate a computer through the provided tools.
+
+The desktop is Windows at 1280x800. Work in a loop: look at the screen with
+get_desktop_state, decide one action, call one tool, then look again.
+
+ * click / type_text / press_key act on a specific application, addressed by
+ `pid`. Use list_windows to find the pid, then pass pid with x/y.
+ * There is no wait tool. If something is still loading, call
+ get_desktop_state again — repeated looks are how you wait.
+
+Call exactly one tool per turn. When the task is complete, reply DONE."""
+
+async def complete(**kwargs):
+ """Two workarounds for the endpoint used here — yours may need neither.
+
+ It is streaming-only (a plain request comes back with empty output), and it
+ rejects role=system, so the system prompt travels as the first user turn.
+ """
+ messages, system = [], []
+ for m in kwargs['messages']:
+ (system if m.get('role') == 'system' else messages).append(m)
+ if system:
+ text = '\n\n'.join(m['content'] for m in system)
+ messages = [{'role': 'user', 'content': text}] + messages
+ kwargs['messages'] = messages
+
+ stream = await litellm.acompletion(**kwargs, stream=True)
+ chunks = [c async for c in stream]
+ return litellm.stream_chunk_builder(chunks, messages=messages)
+
+def prune_images(messages, keep=3):
+ """Each get_desktop_state returns a full screenshot; keep only the newest."""
+ seen = 0
+ for msg in reversed(messages):
+ if not isinstance(msg.get('content'), list):
+ continue
+ for part in msg['content']:
+ if part.get('type') == 'image_url':
+ seen += 1
+ if seen > keep:
+ part.clear()
+ part.update({'type': 'text', 'text': '[older screenshot dropped]'})
+ return messages
+
+async def run(mcp_url, task, model, max_steps=60, headers=None):
+ client = Client(StreamableHttpTransport(mcp_url, headers=headers))
+ async with client:
+ # list_tools() returned 55 tools on the image used here, most of them
+ # browser and recording plumbing this task never needs. Hand the model
+ # only what the job requires: dozens of schemas is a lot of context to
+ # spend, and a shorter menu is a shorter list of ways to go wrong. Worth
+ # doing whether or not your driver already filters denied tools out.
+ wanted = {
+ 'get_desktop_state', 'list_windows', 'list_apps', 'click',
+ 'double_click', 'type_text', 'press_key', 'hotkey', 'launch_app',
+ 'bring_to_front', 'scroll',
+ }
+ mcp_tools = [t for t in await client.list_tools() if t.name in wanted]
+ tools = [{
+ 'type': 'function',
+ 'function': {
+ 'name': t.name,
+ 'description': (t.description or '')[:800],
+ 'parameters': t.inputSchema or {'type': 'object', 'properties': {}},
+ },
+ } for t in mcp_tools]
+
+ messages = [{'role': 'system', 'content': SYSTEM},
+ {'role': 'user', 'content': task}]
+
+ for _ in range(max_steps):
+ resp = await complete(
+ model=model, messages=prune_images(messages), tools=tools,
+ tool_choice='auto', temperature=0.0,
+ )
+ msg = resp.choices[0].message
+ messages.append(msg.model_dump())
+ if not msg.tool_calls:
+ break # model said DONE
+
+ for call in msg.tool_calls:
+ args = json.loads(call.function.arguments or '{}')
+ result = await client.call_tool(call.function.name, args,
+ raise_on_error=False)
+ text = ''.join(getattr(b, 'text', '') for b in (result.content or []))
+ messages.append({'role': 'tool', 'tool_call_id': call.id,
+ 'name': call.function.name, 'content': text[:1500]})
+
+ shot = next((b.data for b in (result.content or [])
+ if getattr(b, 'data', None)), None)
+ if shot:
+ messages.append({'role': 'user', 'content': [
+ {'type': 'text', 'text': 'screenshot after that action:'},
+ {'type': 'image_url',
+ 'image_url': {'url': f'data:image/png;base64,{shot}'}},
+ ]})
+```
+
+Point it at the exposed port and give it the task.
+
+```python
+TASK = """
+Prism Launcher is open, with a Minecraft instance and a signed-in account.
+Select the instance and click Launch. Minecraft uses a software renderer, so the
+window takes minutes to appear and repaints slowly — keep calling
+get_desktop_state to watch it, and do not restart anything. On the title screen
+click Singleplayer, then Create New World, then Create New World again. Stop as
+soon as you are inside the world (terrain in first person, hotbar and hearts
+visible) and reply DONE. Never press Escape while Minecraft is in the foreground.
+"""
+
+asyncio.run(run(f'http://127.0.0.1:{mcp_port}/mcp', TASK, 'your-model'))
+```
+
+Because the MCP tools are presented as **ordinary function tools**, this works against endpoints that reject the provider-native computer-use tool types. That is not hypothetical: on the gateway used here, the same model with the same image in the same second returned 200 for an ordinary function tool and 403 for Anthropic's `computer_20250124`, and `computer_use_preview` was refused outright.
+
+The `complete()` wrapper above exists only for that gateway — it is streaming-only, and it rejects `role: system`. Against an endpoint without those quirks, call `litellm.acompletion` directly.
+
+A full run — launcher to standing in a new world — took 52 steps locally and 51 on Fleet, roughly twenty minutes, most of it waiting on the software renderer. Expect the model to spend long stretches doing nothing but re-screenshotting.
+
+<Callout type="info">
+ Give the model help with coordinates. A vision model without grounding guesses pixel positions and misses: in one run an ungrounded model clicked at (1226, 210) four times, nowhere near the button it wanted, then declared it had no desktop tool. cua-driver's `list_windows` and pid-scoped clicks avoid most of this, and a grounding pass over the screenshot removes the rest.
+</Callout>
+
+## Run the same thing on Fleet
+
+The image and the agent loop are identical on Fleet. Two things change: there is no `local=True` and no `runtime=`, and the MCP endpoint is reached through Fleet's service proxy rather than a forwarded localhost port.
+
+`.expose(3000)` becomes a Fleet **service** named `port-3000`, and `sb.tunnel.forward()` hands you its URL — the same call documented in [Forward a port from a sandbox](/how-to-guides/sandbox/tunneling), so there is nothing Fleet-specific to hand-assemble.
+
+```python
+import httpx
+from cua import Image, Sandbox
+
+sb = await Sandbox.create(Image.windows().expose(3000), name='mc-fleet', time_to_start=900)
+
+tunnel = await sb.tunnel.forward(3000)
+# https://run.cua.ai/api/svc/<namespace>/<sandbox>-port-3000/
+
+token = httpx.post(
+ 'https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token',
+ data={'grant_type': 'client_credentials',
+ 'client_id': os.environ['CUA_CLIENT_ID'],
+ 'client_secret': os.environ['CUA_CLIENT_SECRET']},
+).json()['access_token']
+
+await run(tunnel.url + 'mcp', TASK, 'your-model',
+ headers={'Authorization': f'Bearer {token}'})
+```
+
+That is the same `run()` as above, with only the URL and an auth header changed. A Fleet Windows sandbox takes about three minutes to become ready, against about thirty seconds for a warm local one, and `GET healthz` on that service URL answers `ok` when cua-driver is up.
+
+Notice there is no `-cpu host` here. That flag exists on the local path because the bare-metal runtime defaults to the thin `qemu64` model; a Fleet sandbox is provisioned for you and already reports a full host CPU — `Intel Xeon Processor (SapphireRapids)`, with AVX, AVX2 and AVX-512 all present — so there is nothing to override, and no QEMU arguments to pass.
+
+`Sandbox.create` also accepts `cpu` and `memory_mb`, but sizing a Fleet sandbox is account-dependent: passing them routes the request through a gated custom-resource path, which returned `403 create pool: k8s request is not allowed` at every size until a card-on-file requirement was lifted for the account. The default sandbox — 4 vCPU and 4 GB — is what this guide was written against, and it is enough.
+
+<Callout type="warn">
+ **On Fleet the game needs one extra environment variable: `GALLIUM_DRIVER=softpipe`.** With Mesa's default llvmpipe renderer, Minecraft dies during resource loading every time — `Process crashed with exitcode -2147024809`, no Java exception, no `hs_err` file, nothing in the Windows event log, the log simply stopping after `Reloading ResourceManager`. Switching Mesa to its `softpipe` rasteriser fixes it, and the game runs.
+
+ Set the variable in the process that launches the launcher, so the game inherits it — a machine-level variable does not reach an already-running process, and a test that silently did not apply looks exactly like a test that failed. `setup.ps1` above already started Prism without it, so close the running launcher first and start it again like this.
+
+ ```powershell
+ $env:GALLIUM_DRIVER = 'softpipe'
+ Start-Process -FilePath C:\mc\prismw\prismlauncher.exe -WorkingDirectory C:\mc\prismw
+ ```
+
+ Softpipe is a reference rasteriser with no JIT, so it is **considerably slower than llvmpipe** — allow several minutes for the title screen and longer again for world generation.
+</Callout>
+
+<Callout type="warn">
+ Use client credentials, not a `cua auth login` session token. The session token is short-lived and is held without refresh, so a long provisioning wait dies partway with `401 auth token is invalid`. Client-credential tokens expire too — the ones issued here came back with `expires_in` of 900 seconds — so a run longer than that has to re-mint the token and rebuild the MCP client.
+</Callout>
+
+What that crash is *not*, since each obvious explanation was tested and eliminated: not the Minecraft version or the Java/LWJGL generation (1.20.1 on Java 17 with LWJGL 3 and 1.12.2 on Java 8 with LWJGL 2 fail identically), not the heap (forcing Prism's auto-sized `-Xmx2717m` down to `-Xmx1024m` changed nothing), and not the size of the machine (the same local session disk rebooted with `-m 4096 -smp 4`, matching Fleet exactly, runs the game fine). Narrowing llvmpipe's vectors with `LP_NATIVE_VECTOR_WIDTH=128` did **not** help either, which argues the fault is not simply wide-vector code generation.
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| QEMU refuses to start with `-cpu host` | no KVM or HVF — for example an x86_64 guest on Apple Silicon, which runs under TCG | use a host with hardware virtualisation, or the Fleet path |
+| Launcher never appears, no error | MSVC build without the VC++ redistributable | use the MinGW portable build |
+| Guest has an IP address but cannot resolve names | both user-mode NICs offered the same address | give the second NIC its own subnet |
+| `GLFW error 65542: WGL: The driver does not appear to support OpenGL` | Mesa DLLs missing beside the `javaw.exe` actually in use, or the MSVC Mesa build failed to load | copy the MinGW Mesa DLLs into every `javaw.exe` directory |
+| Game exits during resource loading with no Java exception, **local** | default `qemu64` CPU model | append `-cpu host` to `extra_args` |
+| Game exits during resource loading with `exitcode -2147024809`, **Fleet** | Mesa's default llvmpipe renderer | set `GALLIUM_DRIVER=softpipe` in the process that starts the launcher, and restart the launcher if it is already running |
+| `Permission denied: user policy: tool 'X' is not allowed` | cua-driver's YAML policy refuses that tool, which `list_tools()` advertises anyway on every driver released so far | use an allowed tool — `get_desktop_state` instead of `get_screen_size`, `list_windows` instead of `get_accessibility_tree` |
+| Model replies with empty output on the first call | endpoint is streaming-only | issue `stream=True` and rebuild with `litellm.stream_chunk_builder` |