Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs: add Codex native Computer Use Windows Fleet guide (#3865)
r33drichardsd740aa883b43
diff --git a/docs/content/docs/(cloud-fleets)/meta.json b/docs/content/docs/(cloud-fleets)/meta.jsonindex 3a35b28af3..cb77bd5b30 100644--- a/docs/content/docs/(cloud-fleets)/meta.json+++ b/docs/content/docs/(cloud-fleets)/meta.json@@ -22,6 +22,7 @@"[Troubleshoot Fleet pools](/how-to-guides/sandbox/troubleshoot-fleet-pools)","---Use Fleets with---","[OpenAI Agents](/how-to-guides/sandbox/run-openai-agents-api-on-cloud-fleet)",+ "[Codex native Computer Use (Windows)](/how-to-guides/sandbox/run-codex-native-computer-use-on-windows-fleet)","[Cursor](/how-to-guides/sandbox/run-cursor-cloud-agent-on-fleet)","[OpenClaw](/how-to-guides/sandbox/run-openclaw-on-cloud-fleet)","[Omarchy](/how-to-guides/sandbox/run-omarchy-on-cloud-fleet)",diff --git a/docs/content/docs/how-to-guides/sandbox/meta.json b/docs/content/docs/how-to-guides/sandbox/meta.jsonindex 867fea0e4c..12249aa3b0 100644--- a/docs/content/docs/how-to-guides/sandbox/meta.json+++ b/docs/content/docs/how-to-guides/sandbox/meta.json@@ -19,6 +19,7 @@"troubleshoot-fleet-pools","---Cua Fleets: Use Fleets with---","run-openai-agents-api-on-cloud-fleet",+ "run-codex-native-computer-use-on-windows-fleet","run-cursor-cloud-agent-on-fleet","run-openclaw-on-cloud-fleet","run-omarchy-on-cloud-fleet",diff --git a/docs/content/docs/how-to-guides/sandbox/run-codex-native-computer-use-on-windows-fleet.mdx b/docs/content/docs/how-to-guides/sandbox/run-codex-native-computer-use-on-windows-fleet.mdxnew file mode 100644index 0000000000..45109c812a--- /dev/null+++ b/docs/content/docs/how-to-guides/sandbox/run-codex-native-computer-use-on-windows-fleet.mdx@@ -0,0 +1,529 @@+---+title: Run Codex with native Computer Use on a Cua Cloud Fleet (Windows)+description: Run Codex-in-ChatGPT Computer Use inside an isolated Windows Fleet VM.+---++import { Callout } from 'fumadocs-ui/components/callout';++Codex native Computer Use is the first-party Computer Use plugin in the ChatGPT+desktop app (Codex-in-ChatGPT). It is not a standalone headless CLI flag or a+model endpoint that can be mocked. The native backend is available on macOS and+Windows through the `computer_use.macos` and `computer_use.windows`+configuration sections; there is no Linux backend. Availability also depends on+a supported region and a signed-in, eligible ChatGPT account.++Running the ChatGPT desktop app in a Windows Fleet VM lets Computer Use take over+the VM instead of your main desktop. The recipe is: provision a Windows VM,+install ChatGPT desktop, enable its Computer Use plugin, and let Codex drive that+VM's desktop.++<Callout type="warn">+ The Fleet infrastructure and in-VM installation path were exercised, but native Computer Use+ cannot be verified without a real region- and plan-eligible ChatGPT sign-in. App sign-in, plugin+ installation and enablement, and app-control approvals remain interactive.+</Callout>++## What was live-verified++These observations are evidence for the Fleet and guest path only. They do not+establish that ChatGPT installation or native Computer Use completed.++| Check | Live observation |+| --- | --- |+| Image response | `cua-windows-2022:main-bac7daa3` responded; its index digest began `sha256:6d341afc` and matched the SDK pin |+| Pool creation | `Image.windows("2022")` selected UEFI; pool `codex-cu-win-09141933` became ready in 3.3 seconds |+| Cold claim | Claim `cu-demo` bound in 487 seconds |+| Guest | Windows Server 2022 Standard Evaluation, build `10.0.20348.0` |+| Desktop | `computer-server` returned a valid 164,398-byte PNG screenshot |+| Clean image | ChatGPT, Codex, Edge, Chrome, and `winget` were absent; PowerShell 5.1 was present |+| ChatGPT package | The official MSIX installation was launched, but substrate recycle interrupted final package-version verification |++## Prerequisites++You need:++- run.cua.ai OAuth credentials in `CUA_CLIENT_ID` and `CUA_CLIENT_SECRET`;+- Python `>=3.11,<3.14` with [`uv`](https://docs.astral.sh/uv/), or+ Terraform with the [`trycua/fleets` provider](https://registry.terraform.io/providers/trycua/fleets/latest); and+- an eligible ChatGPT account with Codex access in a supported region.++Keep credentials in environment variables or a secret manager. Do not put them+in scripts, Terraform files, or VM images.++## 1. Create a Windows pool++Windows requires UEFI. In `cua-sandbox==0.7.0`,+`Image.windows("2022")` selects the pinned Windows Server 2022 Fleet image and+generates an EFI template.++Save this as `create_win_pool.py`:++```python title="create_win_pool.py"+# /// script+# requires-python = ">=3.11,<3.14"+# dependencies = ["cua-sandbox==0.7.0"]+# ///+import asyncio+from cua_sandbox import Image, Pool++async def main():+ pool = await Pool.apply(+ Image.windows("2022"),+ name="my-windows-pool",+ replicas=1,+ cpu=4,+ memory_mb=4096,+ services={"server": 8000},+ )+ print("ready:", pool.name)++asyncio.run(main())+```++Run it:++```bash+uv run create_win_pool.py+```++Pool names are globally unique lowercase DNS labels. Change the example name if+it is already owned by another account.++### Terraform alternative++The provider uses `CYCLOPS_*` environment variables rather than the Sandbox+SDK's `CUA_*` names. Follow [Configure a sandbox pool with Terraform](/how-to-guides/sandbox/configure-pool-with-terraform#authentication)+to map the same OAuth credentials, pin provider version `0.2.0`, and configure+the provider endpoint before adding this resource:++```hcl+resource "fleets_pool" "windows" {+ name = "windows-pool"+ cpu_cores = 4+ memory = "4Gi"+ container_disk_image = "public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3"+ runtime = "kubevirt"+ firmware = "efi"++ readiness_probe_json = jsonencode({+ tcpSocket = { port = 8000 }+ initialDelaySeconds = 60+ periodSeconds = 5+ timeoutSeconds = 3+ failureThreshold = 120+ })++ service {+ name = "computer-server"+ target_port = 8000+ protocol = "TCP"+ }++ autoscaling {+ min_pool_size = 0+ initial_pool_size = 1+ max_pool_size = 5+ }+}+```++Run `terraform fmt -check` and `terraform validate` before applying it.++## 2. Claim the VM++Look up the pool rather than reconciling configuration owned elsewhere:++```python+pool = await Pool.get("my-windows-pool")+sandbox = await pool.claim(+ name="codex-cu",+ service="server",+ time_to_start=1200,+)+print(sandbox.pool_name, sandbox.claim_name)+await sandbox.screenshot()+```++A cold Windows claim can take about eight minutes. Keep `time_to_start` high+enough for boot and computer-server readiness, and call `await sandbox.close()`+when you are finished with the claim.++On this Windows image, `shell.run` starts commands with `cmd.exe`. Invoke+PowerShell explicitly when you need it:++```python+result = await sandbox.shell.run(+ 'powershell -NoProfile -Command "Get-ComputerInfo | Select-Object WindowsProductName, OsVersion"'+)+```++Long guest commands can outlast the SDK's fixed response read timeout. Start+long work detached with PowerShell `Start-Process`, redirect output to a log,+and poll that log with short `shell.run` calls.++## 3. Run Codex CLI with native Computer Use++This is the corrected workflow based on our live test.++Codex CLI can drive native Windows apps. The tested setup uses the ChatGPT desktop app to install and initialize OpenAI’s native runtime, then runs the task through codex exec.++It uses OpenAI’s @oai/sky through node_repl—not cua-driver MCP. A standalone npm-only installation was not tested.++### Prerequisites++Inside your Windows Fleet VM, you need:++- An unlocked, interactive Windows desktop.+- Node.js and npm.+- PowerShell running as the interactive desktop user.+- Administrator privileges for the machine-wide package provisioning step.+- ChatGPT authentication, or a separately tested model-provider configuration.++Our tested versions were:++| Component | Version |+|---|---|+| Windows | Server 2022 |+| Codex CLI | 0.154.0 |+| Desktop package | OpenAI.Codex 26.903.8094.0 |+| Native Computer Use plugin | 26.903.61454 |++Run the following commands inside the Windows VM, not in a Linux host or WSL session.++### 3.1. Install and launch the desktop runtime++The desktop app supplies the native Computer Use runtime and generates its configuration.++In an elevated PowerShell window:++```powershell+$work = 'C:\codex-e2e'+New-Item -ItemType Directory -Force $work | Out-Null++$ProgressPreference = 'SilentlyContinue'++Invoke-WebRequest `+ 'https://persistent.oaistatic.com/codex-app-prod/ChatGPT-x64.msix' `+ -OutFile "$work\ChatGPT-x64.msix" `+ -UseBasicParsing++Invoke-WebRequest `+ 'https://persistent.oaistatic.com/codex-app-prod/ChatGPT-License.xml' `+ -OutFile "$work\ChatGPT-License.xml" `+ -UseBasicParsing++$packagePath = (Get-Item "$work\ChatGPT-x64.msix").FullName+$licensePath = (Get-Item "$work\ChatGPT-License.xml").FullName++Add-AppxPackage -Path $packagePath++Add-AppxProvisionedPackage `+ -Online `+ -PackagePath $packagePath `+ -LicensePath $licensePath+```++The provisioning command installs the supplied offline license and changes machine-wide deployment state. In our Server 2022 test, installing the package without this step left app activation failing with 0xC0EA0001.++Verify the package, then launch it:++```powershell+$package = Get-AppxPackage -Name OpenAI.Codex++if (-not $package) {+ throw 'OpenAI.Codex is not installed for this user.'+}++$package | Select-Object Name, Version, InstallLocation++Start-Process explorer.exe -ArgumentList (+ 'shell:AppsFolder\' + $package.PackageFamilyName + '!App'+)+```++Wait for the app to initialize. Keep it running during this workflow.++Two corrections to the original guide:++- Add-AppxPackage does not accept -LicenseFile.+- The tested package is named OpenAI.Codex; searching only for *ChatGPT* misses it.++The download URLs are rolling releases. Check the installed version rather than assuming it matches our test.++### 3.2. Install and authenticate Codex CLI++Install the tested CLI version into an isolated directory:++```powershell+npm.cmd install --prefix C:\codex-cli @openai/[email protected]++$Codex = 'C:\codex-cli\node_modules\.bin\codex.cmd'++& $Codex --version+& $Codex login --device-auth+```++Open the URL printed by Codex, enter its generated code, and approve the login.++Then verify:++```powershell+& $Codex login status+```++Expected:++```text+Logged in using ChatGPT+```++CLI login and desktop-app login are separate checks. Our CLI test succeeded even though the desktop app’s earlier browser sign-in attempt timed out.++### 3.3. Enable the native Computer Use plugin++Check the feature and available plugins:++```powershell+& $Codex features list | Select-String 'computer_use'++$catalog = (+ & $Codex plugin list --available --json+) | ConvertFrom-Json++@($catalog.installed) + @($catalog.available) |+ Where-Object { $_.name -match 'computer-use' } |+ Select-Object pluginId, version, installed, enabled+```++Install the official native plugin:++```powershell+& $Codex plugin add computer-use@openai-bundled --json+```++The desktop app generated a configuration containing:++```toml+[mcp_servers.node_repl]+```++Check for that section:++```powershell+$codexHome = if ($env:CODEX_HOME) {+ $env:CODEX_HOME+} else {+ Join-Path $env:USERPROFILE '.codex'+}++Select-String `+ -Path "$codexHome\config.toml" `+ -Pattern '^\[mcp_servers\.node_repl\]'+```++Preserve the app-generated configuration. It contains runtime paths and native-pipe settings specific to this installation. Do not copy those paths or pipe identifiers from another VM.++If the marketplace or runtime configuration is missing, return to the desktop app’s Computer Use setup and check its initialization. Installing the CLI alone is not evidence that the native runtime is ready.++### 3.4. Prepare the foreground desktop++Use one disposable Notepad window for the smoke test:++```powershell+$notepad = Start-Process notepad.exe -PassThru+Start-Sleep -Seconds 1++$shell = New-Object -ComObject WScript.Shell+$shell.AppActivate($notepad.Id)+```++Visually confirm that Notepad is in front. Avoid switching windows while Codex is working.++This matters: our test encountered a native capture that returned Notepad’s accessibility tree but pixels from the overlapping ChatGPT window. A foreground-only retry worked after explicitly activating Notepad.++The agent must stop if the screenshot and target-window identity disagree.++### 3.5. Run a native CLI smoke test++Create a prompt with an unambiguous expected string:++```powershell+$prompt = @'+Read the installed computer-use skill and its required guidance,+API, and confirmation documents.++Use only OpenAI's native @oai/sky API through node_repl for UI actions.++Find the single blank Notepad window. Capture it and verify that the+screenshot actually shows Notepad. If multiple windows exist, the editor+is not blank, or the screenshot shows another app, stop.++Type exactly the characters inside these quotes, including the period:+"Codex native Windows Computer Use E2E - 6 x 7 = 42."++Capture the resulting Notepad window and report its visible contents.+Leave it open and unsaved.++The user authorizes controlling this scratch Notepad window.+Do not inspect credentials or unrelated apps.+Do not use shell UI automation, cua-driver, or another backend.+Shell commands may only read the required plugin documentation.+Stop on permission errors or screenshot/target mismatches.+'@++[IO.File]::WriteAllText(+ "$work\prompt.txt",+ $prompt,+ [Text.UTF8Encoding]::new($false)+)+```++Run it with captured output:++```powershell+$arguments = @(+ '/c'+ $Codex+ '-a never'+ 'exec'+ '--skip-git-repo-check'+ "-C $work"+ '--sandbox danger-full-access'+ '--json'+ "-o $work\final.txt"+ '-'+) -join ' '++$process = Start-Process cmd.exe -WindowStyle Hidden `+ -ArgumentList $arguments `+ -RedirectStandardInput "$work\prompt.txt" `+ -RedirectStandardOutput "$work\events.jsonl" `+ -RedirectStandardError "$work\stderr.log" `+ -PassThru `+ -Wait++$process.ExitCode+Get-Content "$work\final.txt" -Encoding UTF8+```++**Security:** This reproduces our test’s shell execution policy. danger-full-access is appropriate only for an authorized, isolated VM—not your everyday workstation. -a never prevents interactive shell approval requests; it does not establish that every native-app action is permitted.++Noninteractive codex exec works with redirected input/output; it does not require a PTY.++### 3.6. Optional: Use LiteLLM for model inference++Keep the native runtime configuration and add a provider overlay rather than replacing config.toml.++Create `$codexHome\litellm.config.toml`:++```toml+model = "medium"+model_provider = "litellm"++[model_providers.litellm]+name = "LiteLLM"+base_url = "https://litellm-public.tail204509.ts.net/v1"+env_key = "CODEX_GATEWAY_KEY"+wire_api = "responses"+requires_openai_auth = false+supports_websockets = false+```++Supply the credential only to the current process environment:++```powershell+$secret = Read-Host 'LiteLLM key' -AsSecureString++$env:CODEX_GATEWAY_KEY = (+ [System.Net.NetworkCredential]::new('', $secret)+).Password+```++Add `--profile litellm` after `exec` in the previous run command. The child process inherits the environment variable.++After the run:++```powershell+Remove-Item Env:CODEX_GATEWAY_KEY+```++Our medium gateway test successfully exercised native screenshots, keyboard input, and typing with this provider configuration.++That establishes gateway inference with an already provisioned native runtime. It does not establish gateway-only plugin installation or entitlement.++### 3.7. Verify the result++Do not rely solely on Codex saying “passed.”++- Inspect the actual Notepad text.+- Inspect the native screenshot returned in events.jsonl.+- Confirm UI actions used node_repl with @oai/sky, not a substituted backend.+- Compare exact characters if the test requires exact output.+- Preserve the first failure separately from a successful retry.++Our gateway run omitted a requested trailing period. The UI interaction worked, but the strict string assertion remained a failure.++Known warnings: the gateway’s OpenAI-style /models response produced a nonfatal “missing field models” warning in this CLI version. Explicit model selection still worked.++Fleet cleanup: release the claim when finished and disable/delete unused warm capacity. Do not leave a warm pool replenishing VMs after the test.++## 4. Reach the desktop service++Fleet routes a declared guest service at:++```text+https://run.cua.ai/api/svc/<namespace>/<sandbox_name>-port-<N>/+```++Authenticate with the Bearer token returned by the OAuth client-credentials+flow. Use the bound `sandbox.name`, not the claim name. In the Python pool,+`server:8000` is computer-server; the Terraform example names the same target+port `computer-server`.++Declare every additional port when creating the pool. With the Python image API,+`.expose(N)` declares a `port-N` Fleet service, but it does not install or+start VNC/noVNC (or any other guest service):++```python+image = Image.windows("2022").expose(6080)+```++Prefer `sandbox.services.request()` for authenticated named-service requests.+See [Connect to guest services and tunnels](/how-to-guides/sandbox/tunneling)+for the supported routing contract.++## Caveats++- Native Computer Use is gated by ChatGPT account, plan, credentials, and region;+ it cannot be validated with a mocked endpoint.+- Codex Computer Use on Windows does not support multiplayer or background+ Computer Use. Use [Cua Driver](/how-to-guides/sandbox/control-the-desktop-with-cua-driver)+ on Windows when you need either capability.+- Native Computer Use controls only the foreground desktop. Keep the VM unlocked+ and do not switch away from the active session during a task.+- Published Windows images do not include production or commercial Windows+ licensing. Bring and manage your own license for production use.+- Warm replicas consume billable capacity until the pool is deleted or scaled+ down. A stopped controller process cannot run its cleanup code.++## Clean up++Release the claim, then delete a pool you own when no other workload uses it:++```python+await sandbox.close()+await pool.delete()+```++For a pool created by `Pool.apply()`, deletion removes the pool and its owned+template, and namespace deletion follows. Independently rerun+`await Pool.get("my-windows-pool")`; an HTTP 403 or 404 confirms that the+credentials can no longer retrieve it. Do not create a replacement resource+merely to obtain a cleanup handle.++## Related documentation++- [Cua Fleets](/cloud-fleets)+- [Pool and claim reference](/reference/sandbox-sdk/pool)+- [Sandbox runtime support](/reference/sandbox-sdk/runtime-support)+- [Configure a sandbox pool with Terraform](/how-to-guides/sandbox/configure-pool-with-terraform)