Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

chore: remove internal scratchpad / handoff docs from repo root + libs (#1603)

Francesco Bonaccif83c43c7477c
diff --git a/WINDOWS_SESSION0_STATUS.md b/WINDOWS_SESSION0_STATUS.md
deleted file mode 100644
index ff6872f84a..0000000000
--- a/WINDOWS_SESSION0_STATUS.md
+++ /dev/null
@@ -1,158 +0,0 @@
-# Windows Session-0 Stabilization — Status Report
-
-## Good morning — TL;DR
-
-While you slept I stabilized Windows cua-driver-rs end-to-end on the VM
-you provisioned. **All 11 parity examples PASS** (was 6/11 before).
-**Two real bugs fixed** (`doctor` segfault, `launch_app` hang), **one
-focus-steal hardening** (UWP foreground-restore), **one quality filter**
-(opaque system packages out of `list_apps`), **one startup banner**
-(serve in Session 0 warns about GUI-tool limitations).
-
-Branch `stab/windows-session0-fixes` is 7 commits ahead of
-`origin/main`, all local — no PR opened per your ask. To turn into a
-PR when ready:
-
-```bash
-git -C <repo-root> push -u origin stab/windows-session0-fixes
-gh pr create --base main --head stab/windows-session0-fixes \
- --title "fix(windows): Session-0 hardening + UWP focus-restore" \
- --body-file WINDOWS_SESSION0_STATUS.md
-```
-
-**Still wants your eyes:** the UWP focus-restore needs visual
-confirmation in Session 1+ (RDP in, run the snippet under "Visual
-confirmation" below). I made it stronger than my first pass (now uses
-the `keybd_event` workaround to lift Windows' SetForegroundWindow
-restriction) but I can't visually verify from Mac side.
-
----
-
-**Branch:** `stab/windows-session0-fixes` (local-only, no PR yet per your ask)
-**Base:** `origin/main` @ 69a9dbd5 (post-#1547)
-**VM:** `<vm-name>` (<vm-host>) — Rust toolchain + VS Build Tools 2022 installed
-**Dev loop:** edit on Mac → `scp` changed files to `~/cua/...` on VM → `cargo build --release -p cua-driver` (~20s incremental, ~5min cold) → run. Full reference: `libs/cua-driver-rs/DEV_LOOP_WINDOWS_VM.md`.
-
-## TL;DR
-
-**Two real bugs found + fixed**, both Session-0 (services / SSH-launched) edge cases:
-
-1. **`doctor` crashed with `0xC0000005 ACCESS_VIOLATION`** — COM lifecycle bug in `ui_automation_available`: `IUIAutomation` dropped AFTER `CoUninitialize`, so `IUnknown::Release` ran against a torn-down apartment. Fix: flatten probe result before tearing down COM. ✅
-2. **`launch_app` hung indefinitely in Session 0** — two paths: `resolve_aumid_by_name` walked `shell:AppsFolder` via the interactive shell broker (doesn't exist in Session 0 → hang). `launch_uwp` itself called `IApplicationActivationManager::ActivateApplication` which also needs interactive AppX runtime (same hang). Both now short-circuit on Session 0 with `current_session_id() == Some(0)` — `resolve_aumid_by_name` returns None (falls through to ShellExecuteEx PATH lookup, which works), `launch_uwp` fails fast with a descriptive error. ✅
-
-**Plus**: foreground-restore after UWP activation (now with `keybd_event` workaround for Windows' foreground-lock restriction), three parity-example fixes (Session-0 awareness + #1545 contract), opaque-system-family UWP filter for list_apps, serve startup Session-0 banner, and `overlay_dump` example build fix.
-
-## Bugs fixed (commits on `stab/windows-session0-fixes`)
-
-| Commit | Subject |
-|---|---|
-| `a9d9c027` | fix(windows): doctor COM lifecycle + Session 0 hardening for launch paths |
-| `9e91f683` | test(windows): parity examples Session-0 aware + #1545 contract |
-| `7251e73b` | feat(serve): Session-0 warning banner on Windows daemon startup |
-| `2eb3c4d3` | docs(parity): update list_apps + launch_app Windows status with live evidence |
-| `ac70b94d` | feat(installed_apps): filter opaque-system-family UWP packages from list_apps |
-| `6f580201` | fix(installed_apps): drop Package.IsFramework() probe — hangs in Session 0 |
-| `d5b5821c` | fix(launch_uwp): claim 'last input' before SetForegroundWindow restore |
-
-### Surprise bug found during this work (now fixed in `6f580201`)
-
-`Package.IsFramework()` WinRT call **hangs in Session 0** (0 CPU, no progress, no error). I had added it as a "filter out framework packages from list_apps" optimization in `ac70b94d`, which silently broke list_apps for ~3 hours of debug-loop. Worth documenting: avoid WinRT property getters that may touch the AppX activation runtime when running in services context. The opaque-family-name filter still catches most framework noise.
-
-## Validated on the VM (Session 0, SSH from Mac)
-
-| Tool | Status | Notes |
-|---|---|---|
-| `cua-driver --version` | ✅ 0.2.2 | |
-| `cua-driver list-tools` | ✅ 30 tools | matches Swift surface |
-| `cua-driver doctor` | ✅ exit 0 | clean run, all probes report. Session-0 warning surfaced correctly. |
-| `cua-driver call get_screen_size` | ✅ 1024x768 @ 1.0 | |
-| `cua-driver call get_cursor_position` | ✅ 0,0 | |
-| `cua-driver call check_permissions` | ✅ elevated, post_message, uia all true | |
-| `cua-driver call get_config` | ✅ returns full config | |
-| `cua-driver call list_windows` | ✅ empty `[]` (expected — no desktop) | |
-| `cua-driver call list_apps` | ✅ 3.7s cold, returns ~150 apps (running + installed) | |
-| `cua-driver call get_accessibility_tree` | ✅ returns process list | |
-| `launch_app {"name":"notepad"}` | ✅ ShellExecuteEx, pid returned | SW_SHOWNOACTIVATE (no focus steal) |
-| `launch_app {"path":"C:\\..\\calc.exe"}` | ✅ ShellExecuteEx, pid returned | |
-| `launch_app {"aumid":"...Calculator!App"}` | ✅ fails fast with Session-0 message (no hang) | needs interactive session to actually launch |
-
-### Parity tests
-*(`cargo run --example *_parity` against running daemon)*
-
-Pre-fix: 6 PASS / 5 FAIL — Session-0 expected-failures + stale post-#1545 assertions.
-Post-fix: see `b8jvtpz9m.output` (in flight as of this writing).
-
-## Visual confirmation needs RDP
-
-**Background-launch focus invariant for UWP apps** is now stronger than the initial best-effort:
-
-- Win32 path (ShellExecuteEx + SW_SHOWNOACTIVATE): ✅ proven by code — the API contract says SW_SHOWNOACTIVATE never activates.
-- UWP path (`IApplicationActivationManager::ActivateApplication(AO_NONE)`): the AppX runtime brings the app to foreground (= Start-Menu-click semantics). Now does:
- 1. Snapshot prior foreground via `GetForegroundWindow` before the call
- 2. Inject a VK_NONAME (0xFC) `keybd_event` press+release immediately after to claim "owner of last input" — that lifts Windows' foreground-lock restriction
- 3. Call `SetForegroundWindow(prior)` in a 3 × 50ms retry loop, logging each attempt at debug
-
- VK_NONAME (0xFC) doesn't map to any UI action, so no application sees a keystroke. The Session-0 short-circuit higher up bypasses this entire path so it only executes on interactive logons where focus actually matters.
-
-To visually validate when you wake:
-
-```powershell
-# RDP into <vm-host> as <user>, then in a PowerShell on the desktop:
-$exe = "C:\Users\<user>\AppData\Local\Programs\trycua\cua-driver-rs\bin\cua-driver.exe"
-# Or use the dev build:
-# $exe = "$env:USERPROFILE\cua\libs\cua-driver-rs\target\release\cua-driver.exe"
-
-# Open something focusable first (e.g. Notepad++ via Start menu) and start typing into it.
-# Then from a *different* PowerShell window, fire:
-'{"name":"calculator"}' | & $exe call launch_app
-# Calculator should launch into the taskbar/background; your Notepad++ focus
-# should be preserved (no visible focus steal). If focus DOES move to
-# Calculator, the SetForegroundWindow restriction blocked our restore —
-# we have a fallback path (push to HWND_BOTTOM) on standby.
-```
-
-## Other improvements made
-
-1. **`eprintln!` traces in list_apps / installed_apps replaced with `tracing::debug!`** — silent at default log level. Enable with `RUST_LOG=cua_driver=debug,installed_apps=debug,list_apps=debug` when diagnosing.
-2. **Parity tests `list_apps_parity` / `list_windows_parity` / `launch_app_parity`** brought up to current contract (#1545's unified shape, Session-0 awareness, UWP fail-fast acceptance).
-3. **`overlay_dump.rs` example** — fixed `PrintWindow` + `PRINT_WINDOW_FLAGS` imports (moved to `Win32::Storage::Xps` in windows-rs 0.58).
-
-## Known issues NOT fixed (documented for next session)
-
-1. **UWP display names still show family-name fallback for some packages** — system Components and packages with unresolved `ms-resource:` tokens AND no `Properties/DisplayName` in their AppxManifest.xml fall back to their FamilyName (e.g. `1527c705-839a-4832-9118-54d4Bd6a0c89_cw5n1h2txyewy`). #1547 closed the common case; the long tail is system packages users mostly don't care about. Possible follow-ups: (a) filter out packages whose family_name matches GUID pattern AND display name is empty/equal-to-family — they're usually unactivatable system components; (b) try `Package.PublisherDisplayName` as a third fallback.
-
-2. **`SetForegroundWindow` restoration after UWP activation is best-effort** — Windows restricts foreground transfers and may silently ignore the call. Stronger options to consider: (a) `SetWindowPos(spawned_hwnd, HWND_BOTTOM, ..., SWP_NOACTIVATE)` to push the UWP window behind everything (more aggressive than macOS but guaranteed); (b) the `keybd_event` trick to give cua-driver "last input" status so SetForegroundWindow's restriction lifts (visible side effect of pressing Alt — usually OK).
-
-3. **`overlay_dump` builds — but won't run usefully in Session 0** since the overlay needs a desktop. Build-fix only.
-
-4. **Parity tests requiring foreground windows** — `list_windows`, `screenshot`, `get_window_state`, `click`, `type_text`, `hotkey`, `press_key`, `drag`, `scroll` etc. all need interactive desktop. They're tested by being called against the running daemon, but their assertions (e.g. "expected at least one visible window") are Session-0-aware where I touched them; the rest will naturally need Session 1+ to exercise meaningfully.
-
-## How to take the local fixes forward
-
-```bash
-# View the local-only commits
-git -C <repo-root> log --oneline origin/main..stab/windows-session0-fixes
-
-# Bring them up onto a fresh branch for a PR
-git -C <repo-root> checkout -b fix/windows-session0
-git -C <repo-root> merge --ff-only stab/windows-session0-fixes
-
-# Then `gh pr create` as usual
-```
-
-The local branch is rebased on current `origin/main` (post-#1547). No PR opened per your instruction.
-
-## Other PRs of yours seen earlier in the session
-
-- **#1546** — closed as superseded by #1547 (older diff of same domain)
-- **#1547** — merged (UWP ms-resource: display name normalization)
-- **#1523** — *parity(launch_app) macOS port of #1492* — has CR fixes pushed but **needs a port-onto-current-main**, not a simple rebase. Main moved significantly: `platform-macos/src/apps.rs` deleted and reorganized into `apps/mod.rs` + `apps/nsworkspace.rs`, `AppInfo` gained 3 new fields. Estimated ~30 min focused work to rebuild. Parked pending your call.
-
-## Timeline
-
-- Session started ~Sun May 18 00:00 (local)
-- Toolchain install on VM: 23:08 → 23:34
-- Dev loop online + first cargo build: 23:34 → 23:48 (~11min cold build)
-- doctor + launch_app bugs identified + fixed: ~00:00 → ~02:00
-- This report: ~02:35
-- Resuming at next user-online checkpoint
diff --git a/docs/scripts/HANDOFF-modal-s3-sync.md b/docs/scripts/HANDOFF-modal-s3-sync.md
deleted file mode 100644
index b7ab10b12c..0000000000
--- a/docs/scripts/HANDOFF-modal-s3-sync.md
+++ /dev/null
@@ -1,130 +0,0 @@
-# Handoff: Deploy & Test Modal S3 Sync
-
-## Context
-
-PR #1430 (`feat/modal-s3-sync` branch in `trycua/cua`) adds a `sync_to_s3()` function to `docs/scripts/modal_app.py`. This function uploads generated documentation and code search databases from Modal volumes to S3, bridging the gap between Modal (which generates the DBs on a schedule) and a K8s-hosted MCP server (which serves them).
-
-### Architecture
-
-```
-Modal (generates DBs daily) → sync_to_s3() → S3 → K8s CronJob (pulls from S3) → PVC → MCP Server
-```
-
-### What was built
-
-1. **`sync_to_s3()` function** in `docs/scripts/modal_app.py`:
- - Uploads SQLite + LanceDB files from two Modal volumes to `s3://trycua-docs-mcp-data/`
- - Uses **Modal OIDC federation** (no static AWS keys) — Modal auto-injects `MODAL_IDENTITY_TOKEN`, which is exchanged via STS `AssumeRoleWithWebIdentity` for temporary credentials
- - IAM role: `arn:aws:iam::296062593712:role/modal-docs-mcp-write-role` (already provisioned via Terraform in `trycua/cloud` repo)
-
-2. **Integration into scheduled pipelines**:
- - `scheduled_crawl()` (6 AM UTC) — calls `sync_to_s3()` after regenerating docs databases
- - `scheduled_code_index()` (5 AM UTC) — calls `sync_to_s3()` after aggregating code databases
-
-3. **S3 bucket** (`trycua-docs-mcp-data`) already exists with versioning, encryption, and cross-region replication. Currently has `docs_db/` data from a prior manual upload.
-
-### What needs testing
-
-The code has NOT been deployed yet. The previous environment could not reach PyPI (SSL handshake failures), so `modal` CLI was never installed.
-
----
-
-## Your Task
-
-### 1. Install Modal CLI
-
-```bash
-pip install modal
-```
-
-### 2. Authenticate Modal
-
-The Modal token should be available in `~/.modal/config.toml`. If not, ask the user to provide it or run:
-
-```bash
-modal token set --token-id <ask-user> --token-secret <ask-user>
-```
-
-### 3. Checkout the branch
-
-```bash
-cd /path/to/cua # the trycua/cua repo
-git fetch origin
-git checkout feat/modal-s3-sync
-```
-
-### 4. Deploy the Modal app
-
-```bash
-modal deploy docs/scripts/modal_app.py
-```
-
-This should succeed. The app name is `cua-docs-mcp` — it will register all functions including the new `sync_to_s3`.
-
-### 5. Test `sync_to_s3` manually
-
-```bash
-modal run docs/scripts/modal_app.py::sync_to_s3
-```
-
-**Expected output:**
-- "Syncing databases to s3://trycua-docs-mcp-data/ ..."
-- Upload lines for docs SQLite, docs LanceDB directory, code SQLite, and code LanceDB directory
-- "S3 sync complete: N files uploaded to s3://trycua-docs-mcp-data/"
-
-**If OIDC auth fails** with an STS error about the role trust policy, check:
-- The Modal app name must be `cua-docs-mcp` (defined at top of `modal_app.py`)
-- The role trust policy restricts to `app_name:cua-docs-mcp` — if the app name changed, the OIDC claim won't match
-- Verify the role exists: `aws iam get-role --role-name modal-docs-mcp-write-role`
-
-**If volumes are empty** (no databases to upload), you'll need to generate them first:
-```bash
-# Generate docs databases (takes ~30 min)
-modal run docs/scripts/modal_app.py --skip-code
-
-# Generate code databases (takes ~1-2 hours)
-modal run docs/scripts/modal_app.py --skip-docs
-```
-
-### 6. Verify S3 contents
-
-```bash
-aws s3 ls s3://trycua-docs-mcp-data/ --recursive --region us-west-2
-```
-
-**Expected structure:**
-```
-docs_db/docs.sqlite
-docs_db/docs.lance/...
-code_db/code_index.sqlite
-code_db/code_index.lancedb/...
-```
-
-### 7. Report results
-
-Let the user know:
-- Whether `modal deploy` succeeded
-- Whether `sync_to_s3` ran and how many files were uploaded
-- The S3 listing output
-- Any errors encountered
-
----
-
-## Key Files
-
-| File | Purpose |
-|------|---------|
-| `docs/scripts/modal_app.py` | Modal app — crawling, indexing, S3 sync, MCP server |
-| (cloud repo) `terraform/aws/docs-mcp-storage/main.tf` | S3 bucket, OIDC role, readonly IAM user |
-| (cloud repo) `terraform/aws/oidc/modal-oidc.tf` | Modal OIDC provider |
-
-## AWS Details
-
-| Resource | Value |
-|----------|-------|
-| S3 Bucket | `trycua-docs-mcp-data` |
-| Region | `us-west-2` |
-| Write Role ARN | `arn:aws:iam::296062593712:role/modal-docs-mcp-write-role` |
-| Readonly Creds Secret | `docs-mcp/readonly-credentials` (in AWS Secrets Manager) |
-| Modal App Name | `cua-docs-mcp` |
-| Modal Workspace ID | `ac-3LfmQEOVnLXl4ns0YBxNuA` |
diff --git a/libs/cua-bench/plan.md b/libs/cua-bench/plan.md
deleted file mode 100644
index b287106931..0000000000
--- a/libs/cua-bench/plan.md
+++ /dev/null
@@ -1,531 +0,0 @@
-# Plan: Tinker RL Training Integration with cua-bench
-
-## Overview
-
-This plan describes how to integrate [Tinker](https://tinker-docs.thinkingmachines.ai/) as an RL training backend for the OpenCUA model using `cua-bench` as the environment. The result is a closed training loop: `cua-bench` generates rollout trajectories and rewards → Tinker updates model weights via PPO → updated weights reload into vLLM → next rollout round.
-
----
-
-## 1. How Traces Are Collected Today
-
-After running `cb run tasks --agent opencua`, each task execution produces a **HuggingFace Dataset** stored on disk via `cua_bench/tracing.py`.
-
-### Trace event schema
-
-| Field | Type | Description |
-|---|---|---|
-| `event_name` | `str` | One of `"reset"`, `"agent_step"`, `"solve"`, `"evaluate"` |
-| `data_json` | `str` | JSON payload specific to event type |
-| `data_images` | `Sequence[Image]` | Screenshots at that moment |
-| `trajectory_id` | `str` | UUID identifying the full episode |
-| `timestamp` | `str` | ISO-8601 UTC timestamp |
-
-### Events recorded per task
-
-| Event | When | Key `data_json` fields |
-|---|---|---|
-| `reset` | After env setup completes | `{task, task_index, elapsed}` |
-| `agent_step` | After every agent action | `{step, agent, model, usage, output}` |
-| `solve` | After oracle solution (non-eval runs) | `{completed: bool}` |
-| `evaluate` | After evaluation function | `{result: <reward_value>}` |
-
-The `agent_step` `output` field mirrors the raw message list from the ComputerAgent SDK, including typed actions (`click`, `type_text`, etc.) and the final `"DONE"` message.
-
-### Trace storage paths
-
-```
-~/.local/share/cua-bench/runs/<run_id>/
-├── task_0_trace/ # HF Dataset (arrow files)
-│ ├── dataset_info.json
-│ ├── state.json
-│ └── data/
-├── task_0_agent_logs/
-│ └── trajectories/ # ComputerAgent internal trajectories
-└── imgs/ # Optional PNG dumps (--save-pngs flag)
-```
-
-### Loading traces programmatically
-
-```python
-from datasets import load_from_disk
-
-ds = load_from_disk("~/.local/share/cua-bench/runs/<run_id>/task_0_trace/")
-
-# Extract reward from evaluate event
-reward_row = next(r for r in ds if r["event_name"] == "evaluate")
-reward = json.loads(reward_row["data_json"])["result"]
-
-# Extract agent steps in order
-steps = [r for r in ds if r["event_name"] == "agent_step"]
-for step in steps:
- data = json.loads(step["data_json"])
- screenshot = step["data_images"][0] # PIL.Image
-```
-
----
-
-## 2. Tinker API Overview
-
-Tinker is a cloud-hosted RL/SL fine-tuning platform for LLMs. It abstracts distributed GPU training while exposing a simple Python API.
-
-### Core client hierarchy
-
-```
-ServiceClient (entry point)
-├── create_lora_training_client() → TrainingClient
-│ ├── forward_backward(data, loss_fn) → Future
-│ ├── optim_step(AdamParams) → Future
-│ ├── save_state(name) → Future[str] (tinker:// path)
-│ └── save_weights_and_get_sampling_client() → SamplingClient
-├── create_sampling_client(checkpoint) → SamplingClient
-│ ├── sample(prompt, params) → List[str]
-│ └── compute_logprobs(prompt, target) → List[float]
-└── create_rest_client() → RestClient
- ├── list_checkpoints(run_id)
- └── get_checkpoint_archive_url_from_tinker_path(path)
-```
-
-### Supported RL loss functions
-
-| Loss | Use case |
-|---|---|
-| `"importance_sampling"` | Off-policy PPO without clipping |
-| `"ppo"` | Clipped PPO (standard RLHF) |
-| `"cispo"` | Clipped importance sampling coefficients |
-| `"dro"` | Direct reward optimization with quadratic penalty |
-| Custom | Via `forward_backward_custom()` with arbitrary differentiable loss |
-
-### Checkpoint persistence
-
-| Method | Saves | When to use |
-|---|---|---|
-| `save_weights_for_sampler(name)` | Weights only | Quick eval between epochs |
-| `save_state(name)` | Weights + optimizer state | Full resume / multi-phase SL→RL |
-| `save_weights_and_get_sampling_client()` | Weights → new SamplingClient | End of training step, for next rollout |
-
-Checkpoint paths follow the scheme `tinker://<model_id>/<name>`. Default TTL is 7 days; use `RestClient` to extend.
-
----
-
-## 3. Proposed Architecture
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│ RL Training Loop │
-│ │
-│ ┌──────────────┐ rollouts ┌───────────────────────┐ │
-│ │ cua-bench │ ─────────────▶│ Trace Collector │ │
-│ │ (env) │ │ (HF Datasets) │ │
-│ └──────┬───────┘ └──────────┬────────────┘ │
-│ │ │ │
-│ ┌──────▼───────┐ ┌──────────▼────────────┐ │
-│ │ vLLM │ ◀─ weights ── │ Tinker │ │
-│ │ (OpenCUA) │ │ TrainingClient │ │
-│ └──────────────┘ │ (PPO / IS loss) │ │
-│ └───────────────────────┘ │
-└─────────────────────────────────────────────────────────────┘
-```
-
-### Component roles
-
-| Component | Role |
-|---|---|
-| `cua-bench` | Rollout environment; generates trajectories + reward signals |
-| `cua_bench/tracing.py` | Serializes trajectories to HF Datasets on disk |
-| `Trace Collector` | Reads traces, extracts `(prompt, action, reward)` tuples |
-| `Datum Builder` | Converts tuples to Tinker `Datum` objects (token sequences + masks) |
-| `Tinker TrainingClient` | Runs PPO gradient updates on remote GPU cluster |
-| `Tinker SamplingClient` | Not used for generation (vLLM does that), but used for `compute_logprobs` to get reference policy log-probs for IS ratio |
-| `vLLM` | Serves the current policy for cua-bench rollouts |
-| `Weight Sync` | Downloads updated LoRA weights from Tinker checkpoint → hot-reloads into vLLM |
-
----
-
-## 4. Implementation Plan
-
-### Phase 1: Trace extraction pipeline
-
-**New file**: `cua_bench/rl/trace_reader.py`
-
-```python
-"""
-Read completed cua-bench run traces and convert them to RL experience tuples.
-"""
-import json
-from pathlib import Path
-from dataclasses import dataclass
-from typing import List, Optional
-from datasets import load_from_disk
-
-@dataclass
-class RLStep:
- step: int
- prompt_images: list # screenshots before action
- action_text: str # raw action output from model
- reward: float # 0.0 except at terminal step
-
-@dataclass
-class Episode:
- trajectory_id: str
- task_description: str
- steps: List[RLStep]
- terminal_reward: float # from "evaluate" event
-
-def load_episode_from_trace(trace_dir: str) -> Episode:
- ds = load_from_disk(trace_dir)
- rows_by_event = {}
- for row in ds:
- rows_by_event.setdefault(row["event_name"], []).append(row)
-
- reward = float(json.loads(rows_by_event["evaluate"][0]["data_json"])["result"])
- trajectory_id = ds[0]["trajectory_id"]
-
- steps = []
- for row in rows_by_event.get("agent_step", []):
- data = json.loads(row["data_json"])
- steps.append(RLStep(
- step=data["step"],
- prompt_images=row["data_images"],
- action_text=str(data["output"]),
- reward=0.0,
- ))
-
- # Assign terminal reward to last step only (sparse reward)
- if steps:
- steps[-1].reward = reward
-
- return Episode(trajectory_id, task_description="", steps=steps, terminal_reward=reward)
-
-def load_run_episodes(run_dir: str) -> List[Episode]:
- """Load all task traces from a cua-bench run directory."""
- run_path = Path(run_dir)
- episodes = []
- for trace_dir in sorted(run_path.glob("task_*_trace")):
- episodes.append(load_episode_from_trace(str(trace_dir)))
- return episodes
-```
-
-### Phase 2: Datum construction
-
-**New file**: `cua_bench/rl/datum_builder.py`
-
-The Tinker `Datum` format requires tokenized inputs with explicit weight masks. For a vision-language model like OpenCUA (based on Qwen-VL), each step's datum looks like:
-
-```
-[system_tokens] [image_tokens] [user_instruction_tokens] [action_tokens]
- ^^^^^^^^^^^^^^^^
- loss computed here
-```
-
-Key decisions:
-- **Input sequence**: System prompt + screenshot(s) + task description (this is the `model_input`)
-- **Target sequence**: The action the agent took (this is the `loss_fn_inputs` target)
-- **Weight mask**: 1.0 on action tokens, 0.0 on prompt tokens
-- **Advantage weighting**: For PPO, multiply step reward by a per-step advantage estimate; apply as weight multiplier on action tokens
-
-```python
-import tinker.types as tt
-from transformers import AutoTokenizer
-
-def build_datum(step: RLStep, tokenizer, advantage: float) -> tt.Datum:
- """Convert a single RL step to a Tinker Datum."""
- # Tokenize action (target)
- action_ids = tokenizer.encode(step.action_text, add_special_tokens=False)
-
- # Tokenize prompt context (images + instruction handled by rendering pipeline)
- # ... (use Tinker's ImageChunk + EncodedTextChunk for multimodal)
-
- model_input = tt.ModelInput(
- chunks=[
- tt.EncodedTextChunk(token_ids=prompt_ids),
- tt.ImageChunk(image=step.prompt_images[0]), # latest screenshot
- tt.EncodedTextChunk(token_ids=action_ids),
- ]
- )
-
- # Shift targets: model predicts token[i+1] from token[i]
- target_ids = action_ids # shifted internally by Tinker
- weights = [max(0.0, advantage)] * len(action_ids) # zero-out negative advantages
-
- return tt.Datum(
- model_input=model_input,
- loss_fn_inputs=tt.LossFnInputs(
- target_token_ids=target_ids,
- weights=weights,
- )
- )
-
-def build_batch(episode: Episode, tokenizer, gamma=0.99) -> list[tt.Datum]:
- """Convert episode to list of Datums with discounted advantage estimates."""
- # Compute discounted returns (simple Monte Carlo)
- G = 0.0
- returns = []
- for step in reversed(episode.steps):
- G = step.reward + gamma * G
- returns.insert(0, G)
-
- baseline = sum(returns) / len(returns) if returns else 0.0
-
- datums = []
- for step, G in zip(episode.steps, returns):
- advantage = G - baseline
- datums.append(build_datum(step, tokenizer, advantage))
- return datums
-```
-
-### Phase 3: RL training loop
-
-**New file**: `cua_bench/rl/train.py`
-
-```python
-"""
-Main RL training loop using Tinker.
-
-Usage:
- python -m cua_bench.rl.train \
- --base-model Qwen/Qwen3-8B \
- --run-dir ~/.local/share/cua-bench/runs/<run_id> \
- --epochs 10 \
- --lr 1e-5
-"""
-import tinker
-from tinker import ServiceClient
-import tinker.types as tt
-
-def train_epoch(training_client, episodes: list, tokenizer, args):
- """Run one PPO update epoch over a batch of episodes."""
- all_datums = []
- for ep in episodes:
- all_datums.extend(build_batch(ep, tokenizer, gamma=args.gamma))
-
- if not all_datums:
- return
-
- # Submit forward-backward and optim step without awaiting between them
- # (keeps operations in the same ~10s Tinker clock cycle)
- fwdbwd_future = training_client.forward_backward_async(
- data=all_datums,
- loss_fn="ppo",
- loss_fn_params=tt.PpoParams(
- clip_low_threshold=0.8,
- clip_high_threshold=1.2,
- ),
- )
- optim_future = training_client.optim_step_async(
- tt.AdamParams(
- learning_rate=args.lr,
- betas=(0.9, 0.999),
- weight_decay=0.01,
- grad_clip=1.0,
- )
- )
-
- # Collect results
- fwdbwd_result = fwdbwd_future.result()
- optim_future.result()
- return fwdbwd_result
-
-def main(args):
- service_client = ServiceClient(api_key=args.tinker_api_key)
-
- training_client = service_client.create_lora_training_client(
- base_model=args.base_model,
- lora_config=tt.LoraConfig(rank=16, seed=42),
- )
-
- tokenizer = training_client.get_sampling_client().tokenizer
-
- for epoch in range(args.epochs):
- # 1. Run cua-bench rollouts with current policy
- run_id = run_cuabench_rollouts(args)
-
- # 2. Load traces
- run_dir = f"~/.local/share/cua-bench/runs/{run_id}"
- episodes = load_run_episodes(run_dir)
-
- # 3. Log stats
- avg_reward = sum(ep.terminal_reward for ep in episodes) / max(len(episodes), 1)
- print(f"Epoch {epoch}: {len(episodes)} episodes, avg_reward={avg_reward:.3f}")
-
- # 4. Train
- train_epoch(training_client, episodes, tokenizer, args)
-
- # 5. Save weights and get new sampling client
- sampling_client = training_client.save_weights_and_get_sampling_client()
-
- # 6. Checkpoint full state every N epochs
- if (epoch + 1) % args.checkpoint_every == 0:
- ckpt_path = training_client.save_state(name=f"epoch-{epoch+1}").result()
- print(f"Checkpoint saved: {ckpt_path}")
-
- # 7. Hot-reload updated weights into vLLM
- sync_weights_to_vllm(training_client, args.vllm_url)
-
-def run_cuabench_rollouts(args) -> str:
- """Launch a cua-bench dataset run and return the run_id."""
- import subprocess, re
- result = subprocess.run([
- "cb", "run", "dataset", args.tasks_path,
- "--agent", "opencua",
- "--model", args.model,
- "--max-steps", str(args.max_steps),
- ], capture_output=True, text=True)
- run_id = re.search(r"run-[a-f0-9-]+", result.stdout).group(0)
- # Wait for all sessions to complete (cb run watch or poll cb run info)
- wait_for_run(run_id)
- return run_id
-```
-
-### Phase 4: Weight sync to vLLM
-
-After each Tinker training step the updated LoRA weights must reload into the running vLLM server so that subsequent cua-bench rollouts use the new policy.
-
-**New file**: `cua_bench/rl/weight_sync.py`
-
-```python
-"""
-Download LoRA checkpoint from Tinker and hot-reload into a running vLLM instance.
-"""
-import requests
-import tarfile
-import tempfile
-from pathlib import Path
-
-def sync_weights_to_vllm(training_client, vllm_base_url: str):
- """
- 1. Save weights to Tinker checkpoint
- 2. Download tar archive
- 3. POST to vLLM's /load_lora_adapter endpoint
- """
- # Save weights-only checkpoint (faster than save_state)
- ckpt_path = training_client.save_weights_for_sampler(name="current").result()
-
- # Get signed download URL via RestClient
- rest_client = training_client.service_client.create_rest_client()
- url_response = rest_client.get_checkpoint_archive_url_from_tinker_path(ckpt_path)
-
- # Download tar archive
- with tempfile.TemporaryDirectory() as tmpdir:
- archive_path = Path(tmpdir) / "lora.tar"
- r = requests.get(url_response.url, stream=True)
- r.raise_for_status()
- with open(archive_path, "wb") as f:
- for chunk in r.iter_content(chunk_size=8192):
- f.write(chunk)
-
- # Extract LoRA adapter files
- with tarfile.open(archive_path) as tar:
- tar.extractall(tmpdir)
-
- adapter_dir = Path(tmpdir) / "adapter"
-
- # Hot-reload via vLLM dynamic LoRA API
- requests.post(
- f"{vllm_base_url}/load_lora_adapter",
- json={
- "lora_name": "opencua-current",
- "lora_path": str(adapter_dir),
- }
- ).raise_for_status()
-```
-
-### Phase 5: Resume training from checkpoint
-
-```python
-# Resume full training state (weights + optimizer momentum)
-training_client = service_client.create_training_client_from_state_with_optimizer(
- checkpoint_path="tinker://<model_id>/epoch-5"
-)
-
-# Resume weights only (fresh optimizer)
-training_client = service_client.create_lora_training_client(
- base_model=args.base_model,
- lora_config=tt.LoraConfig(rank=16),
- initial_checkpoint="tinker://<model_id>/epoch-5",
-)
-```
-
----
-
-## 5. File Structure
-
-```
-libs/cua-bench/
-├── cua_bench/
-│ └── rl/
-│ ├── __init__.py
-│ ├── trace_reader.py # Load HF Dataset traces → Episode objects
-│ ├── datum_builder.py # Episode → Tinker Datum (tokenized, masked)
-│ ├── train.py # Main RL training loop (entry point)
-│ └── weight_sync.py # Download Tinker checkpoint → reload vLLM
-├── plan.md # This file
-└── pyproject.toml # Add tinker to [project.optional-dependencies]
-```
-
-Add to `pyproject.toml`:
-```toml
-[project.optional-dependencies]
-rl = [
- "tinker",
- "transformers>=4.57.0,<5.0",
-]
-```
-
----
-
-## 6. End-to-End Training Script
-
-```bash
-# 1. Start vLLM with base model
-vllm serve Qwen/Qwen3-8B \
- --served-model-name opencua \
- --enable-lora \
- --host 0.0.0.0 --port 30000
-
-# 2. Set env vars
-export OPENCUA_BASE_URL=http://localhost:30000/v1
-export OPENCUA_API_KEY=EMPTY
-export TINKER_API_KEY=<your_tinker_key>
-
-# 3. Run RL training loop
-python -m cua_bench.rl.train \
- --base-model Qwen/Qwen3-8B \
- --tasks-path tasks/slack_env \
- --model openai/opencua \
- --max-steps 50 \
- --epochs 20 \
- --lr 1e-5 \
- --gamma 0.99 \
- --checkpoint-every 5 \
- --vllm-url http://localhost:30000/v1
-```
-
----
-
-## 7. Key Design Decisions
-
-| Decision | Rationale |
-|---|---|
-| **Sparse reward** | `evaluate_task` returns a single terminal reward per episode; intermediate steps get reward 0.0 and the terminal step gets the full reward |
-| **Monte Carlo returns** with simple baseline | Simple, unbiased, works well with sparse rewards; GAE would need a value network |
-| **PPO loss in Tinker** | Standard choice for LLM RL; clip thresholds prevent oversized updates when off-policy rollouts diverge |
-| **LoRA fine-tuning** | Tinker only supports LoRA currently; keeps checkpoint size small for fast sync to vLLM |
-| **Weights-only sync to vLLM** | `save_weights_for_sampler` is faster than `save_state`; vLLM's dynamic LoRA API avoids server restart |
-| **Async Tinker calls** | Submit `forward_backward_async` and `optim_step_async` without awaiting between them to keep both in the same ~10s Tinker clock cycle |
-| **Screenshot as model input** | Each `agent_step` trace row has a screenshot; use the most recent as the visual context for that step's datum |
-
----
-
-## 8. Open Questions / Future Work
-
-1. **Reference policy for IS ratio**: PPO needs log-probs from both current policy (from Tinker `compute_logprobs`) and sampling policy (from vLLM generation). Need to capture vLLM log-probs at rollout time and store them in trace metadata.
-
-2. **Dense rewards**: Currently uses sparse evaluation reward. Could add partial rewards (e.g., intermediate task completion signals from `evaluate_task` called mid-episode) for faster learning.
-
-3. **Multi-task batching**: Different task types may have very different reward scales. Per-task reward normalization or separate baselines may be needed.
-
-4. **Rollout parallelism**: `cb run dataset` already parallelizes across tasks. Tinker's clock-cycle architecture means training throughput is bounded by how many rollout batches can be collected per ~10s cycle.
-
-5. **Vision input**: OpenCUA is a vision-language model. Tinker supports Qwen3-VL via `ImageChunk`. The datum builder must encode screenshots as Tinker image chunks rather than base64 strings.
-
-6. **SL warm-start before RL**: Use `save_state` after a supervised learning phase, then `create_training_client_from_state_with_optimizer` to continue into RL without losing optimizer momentum.
diff --git a/libs/cua-driver-rs/DEV_LOOP_WINDOWS_VM.md b/libs/cua-driver-rs/DEV_LOOP_WINDOWS_VM.md
deleted file mode 100644
index a71d5e7915..0000000000
--- a/libs/cua-driver-rs/DEV_LOOP_WINDOWS_VM.md
+++ /dev/null
@@ -1,132 +0,0 @@
-# Dev loop — cua-driver-rs on the Windows VM
-
-Quick reference for iterating on Windows-specific cua-driver-rs code
-directly on `<vm-name>` without round-tripping through GitHub
-CD. Set up Sun 2026-05-18 by the autonomous session for the doctor
-segfault / Session-0 hang investigations.
-
-## Connection
-
-- **SSH** (Session 0 — services context, no desktop):
- ```bash
- ssh -i ~/.ssh/<ssh-key> <user>@<vm-host>
- ```
-- **RDP** (Session 1+ with attached desktop, needed for visual / GUI tool validation):
- ```bash
- open <rdp-file-dir>/<vm-name>.rdp
- ```
- Credentials: `<user>` + password from your password manager.
-
-## Toolchain on the VM (already installed)
-
-- Git 2.54.0
-- Rustup 1.29 + `stable-x86_64-pc-windows-msvc` (rustc 1.95.0)
-- Cargo 1.95.0
-- Visual Studio Build Tools 2022 (VCTools + Windows 11 SDK)
-- Clone: `C:\Users\<user>\cua` (HEAD pinned to origin/main at clone time;
- `git pull` to refresh)
-
-## Iteration loop
-
-1. **Edit locally on the Mac** (use Claude Code, your IDE, etc.).
-2. **Push changed files to the VM** via `scp`:
- ```bash
- scp -i ~/.ssh/<ssh-key> \
- <repo-root>/libs/cua-driver-rs/crates/platform-windows/src/<file>.rs \
- <user>@<vm-host>:cua/libs/cua-driver-rs/crates/platform-windows/src/<file>.rs
- ```
-3. **Build on the VM** (~20 s incremental, ~5 min cold):
- ```bash
- ssh -i ~/.ssh/<ssh-key> <user>@<vm-host> \
- "powershell -Command 'cd cua\\libs\\cua-driver-rs; cargo build --release -p cua-driver'"
- ```
-4. **Test on the VM** — use PowerShell `Start-Job` for clean timeouts;
- pipe JSON args via stdin to bypass PowerShell's nested-quoting hell:
- ```powershell
- '{"name":"notepad"}' | & "$env:USERPROFILE\cua\libs\cua-driver-rs\target\release\cua-driver.exe" call launch_app
- ```
-
-## Gotchas the autonomous session hit
-
-1. **PowerShell heredoc quoting** — `Start-Process -ArgumentList` with a JSON
- arg containing `"` will silently strip the quotes. **Workaround:**
- pipe the JSON via stdin to `cua-driver call <tool>` instead of
- passing it as argv.
-
-2. **`Start-Process -RedirectStandardError` buffering** — stderr from
- the child process may not flush to the redirect file until the child
- exits cleanly. If you `Stop-Process` mid-run you lose buffered
- stderr. **Workaround:** use `Start-Job` with `Tee-Object` for live
- capture, or write to a file from inside the script.
-
-3. **PowerShell strings with `(`, `,`, em-dashes** — PowerShell parses
- parentheses and commas inside double-quoted strings as method calls
- in some contexts. Em-dashes get corrupted by `scp`. **Workaround:**
- stick to plain ASCII single-quoted strings in any script you `scp`
- over.
-
-4. **`R` is a PowerShell alias for `Invoke-History`.** Don't name a
- helper function `R`.
-
-5. **Some parity examples hard-code `target/debug/cua-driver.exe`** —
- if you've only run `cargo build --release` you need to also run
- `cargo build` (debug) once, otherwise `list_tools_parity` and
- `describe_parity` fail with "cannot find the path specified".
-
-6. **`SetForegroundWindow` is restricted on Windows** — calling it
- from a non-foreground process silently no-ops most of the time. The
- foreground-restore in `launch_uwp` is best-effort. Plan visual
- confirmation in Session 1+ for any focus-related work.
-
-7. **Visual Studio Build Tools install is ~5 GB and takes ~10-15 min
- over Azure VM network.** Don't reinstall — it's there.
-
-## Running the parity suite
-
-```powershell
-$env:CUA_DRIVER_RS_TELEMETRY_ENABLED = "0"
-$exe = "$env:USERPROFILE\cua\libs\cua-driver-rs\target\release\cua-driver.exe"
-
-# 1. Start the daemon
-Get-Process | Where-Object { $_.Name -eq 'cua-driver' } | Stop-Process -Force
-Start-Process -FilePath $exe -ArgumentList 'serve' -NoNewWindow
-
-# 2. Run all parity examples (skip overlay_dump — it's interactive-overlay-only)
-cd $env:USERPROFILE\cua\libs\cua-driver-rs
-$examples = @('list_tools_parity','describe_parity','get_screen_size_parity',
- 'get_cursor_position_parity','config_parity',
- 'agent_cursor_setters_parity','get_agent_cursor_state_parity',
- 'set_agent_cursor_style_parity','list_windows_parity',
- 'list_apps_parity','launch_app_parity')
-foreach ($ex in $examples) {
- cargo run --release --quiet --example $ex -p platform-windows
- Write-Host "$ex exit=$LASTEXITCODE"
-}
-
-# 3. Cleanup
-Get-Process | Where-Object { $_.Name -eq 'cua-driver' } | Stop-Process -Force
-```
-
-In Session 0 the autonomous session got 11/11 PASS as of 2026-05-18.
-
-## Commands the autonomous session used most
-
-```bash
-# Quick file push
-scp -i ~/.ssh/<ssh-key> <local> <user>@<vm-host>:cua/libs/cua-driver-rs/...
-
-# Quick build + test with timeout (Mac side)
-ssh -o ServerAliveInterval=20 -i ~/.ssh/<ssh-key> <user>@<vm-host> \
- 'powershell -ExecutionPolicy Bypass -File some-script.ps1'
-
-# Quick PSCMD via stdin pipe
-echo '{"name":"notepad"}' | ssh -i ~/.ssh/<ssh-key> <user>@<vm-host> \
- '"C:\\Users\\<user>\\cua\\libs\\cua-driver-rs\\target\\release\\cua-driver.exe" call launch_app'
-
-# Kill all stale cua-driver procs
-ssh -i ~/.ssh/<ssh-key> <user>@<vm-host> \
- "powershell -Command 'Get-Process cua-driver -EA SilentlyContinue | Stop-Process -Force'"
-
-# Check which sessions exist
-ssh -i ~/.ssh/<ssh-key> <user>@<vm-host> "powershell -Command qwinsta"
-```
diff --git a/libs/python/mcp-server/QUICK_TEST_COMMANDS.sh b/libs/python/mcp-server/QUICK_TEST_COMMANDS.sh
deleted file mode 100755
index 3242c610b8..0000000000
--- a/libs/python/mcp-server/QUICK_TEST_COMMANDS.sh
+++ /dev/null
@@ -1,63 +0,0 @@
-#!/bin/bash
-# Quick Test Commands for MCP Server Local Desktop Option
-# Run these commands to test the implementation
-
-set -e # Exit on error
-
-echo "======================================================================"
-echo "Testing MCP Server Local Desktop Option"
-echo "======================================================================"
-echo ""
-
-# Change to repo root
-cd "$(dirname "$0")/.."
-
-# Test 1: Quick Logic Test (No setup required)
-echo "Test 1: Quick Logic Test (No setup required)"
-echo "----------------------------------------------------------------------"
-python tests/quick_test_local_option.py
-echo ""
-
-# Test 2: Automated Tests (Requires pytest and packages)
-echo "Test 2: Automated Tests (Requires pytest and packages installed)"
-echo "----------------------------------------------------------------------"
-if command -v pytest &> /dev/null; then
- echo "Running pytest..."
- pytest tests/test_mcp_server_local_option.py -v || echo "Note: Some tests may require full setup"
-else
- echo "⚠️ pytest not found. Install with: pip install pytest"
-fi
-echo ""
-
-# Test 3: Existing MCP server tests
-echo "Test 3: Existing MCP Server Tests"
-echo "----------------------------------------------------------------------"
-if command -v pytest &> /dev/null; then
- echo "Running existing session management tests..."
- pytest tests/test_mcp_server_session_management.py -v || echo "Note: Some tests may fail if dependencies are missing"
-else
- echo "⚠️ pytest not found. Install with: pip install pytest"
-fi
-echo ""
-
-# Summary
-echo "======================================================================"
-echo "Test Summary"
-echo "======================================================================"
-echo "✅ Quick logic test completed"
-echo ""
-echo "Next steps for comprehensive testing:"
-echo "1. Install dependencies:"
-echo " pip install -e libs/python/core"
-echo " pip install -e libs/python/computer"
-echo " pip install -e libs/python/agent"
-echo " pip install -e libs/python/mcp-server"
-echo " pip install -e libs/python/computer-server"
-echo ""
-echo "2. For manual end-to-end testing, see:"
-echo " tests/MANUAL_TEST_LOCAL_OPTION.md"
-echo ""
-echo "3. For detailed testing info, see:"
-echo " tests/TESTING_SUMMARY.md"
-echo ""
-
diff --git a/libs/python/mcp-server/quick_test_local_option.py b/libs/python/mcp-server/quick_test_local_option.py
deleted file mode 100755
index 6e2caab229..0000000000
--- a/libs/python/mcp-server/quick_test_local_option.py
+++ /dev/null
@@ -1,244 +0,0 @@
-#!/usr/bin/env python3
-"""
-Quick test to verify the local desktop option logic without full setup.
-
-This script tests the environment variable parsing and logic flow
-without requiring VMs, computer-server, or MCP clients to be running.
-"""
-
-import os
-import sys
-
-
-def test_env_var_parsing():
- """Test that environment variable is parsed correctly."""
- print("Testing CUA_USE_HOST_COMPUTER_SERVER environment variable parsing...")
- print("-" * 60)
-
- test_cases = [
- # (env_value, expected_result, description)
- ("true", True, "lowercase 'true'"),
- ("True", True, "capitalized 'True'"),
- ("TRUE", True, "uppercase 'TRUE'"),
- ("1", True, "numeric '1'"),
- ("yes", True, "lowercase 'yes'"),
- ("Yes", True, "capitalized 'Yes'"),
- ("false", False, "lowercase 'false'"),
- ("False", False, "capitalized 'False'"),
- ("FALSE", False, "uppercase 'FALSE'"),
- ("0", False, "numeric '0'"),
- ("no", False, "lowercase 'no'"),
- ("", False, "empty string"),
- ("random", False, "random value"),
- (None, False, "not set (None)"),
- ]
-
- passed = 0
- failed = 0
-
- for env_value, expected, description in test_cases:
- # Simulate the logic from session_manager.py line 59
- if env_value is None:
- actual = os.getenv("CUA_USE_HOST_COMPUTER_SERVER", "false").lower() in (
- "true",
- "1",
- "yes",
- )
- else:
- os.environ["CUA_USE_HOST_COMPUTER_SERVER"] = env_value
- actual = os.getenv("CUA_USE_HOST_COMPUTER_SERVER", "false").lower() in (
- "true",
- "1",
- "yes",
- )
-
- status = "✓ PASS" if actual == expected else "✗ FAIL"
- if actual == expected:
- passed += 1
- else:
- failed += 1
-
- print(
- f"{status} | Value: {env_value!r:15} | Expected: {expected!s:5} | Got: {actual!s:5} | {description}"
- )
-
- # Clean up
- os.environ.pop("CUA_USE_HOST_COMPUTER_SERVER", None)
-
- print("-" * 60)
- print(f"Results: {passed} passed, {failed} failed")
- return failed == 0
-
-
-def test_session_manager_logic():
- """Test the logic flow in session_manager.py without actual Computer creation."""
- print("\nTesting session_manager.py logic flow...")
- print("-" * 60)
-
- # Read the actual session_manager.py to verify the logic
- import pathlib
-
- session_manager_path = (
- pathlib.Path(__file__).parent.parent
- / "libs"
- / "python"
- / "mcp-server"
- / "mcp_server"
- / "session_manager.py"
- )
-
- if not session_manager_path.exists():
- print(f"✗ FAIL | session_manager.py not found at {session_manager_path}")
- return False
-
- content = session_manager_path.read_text()
-
- # Check for the key logic
- checks = [
- ('os.getenv("CUA_USE_HOST_COMPUTER_SERVER"', "Environment variable check present"),
- ("use_host_computer_server=use_host", "use_host_computer_server parameter passed"),
- ("Computer(", "Computer instantiation present"),
- ]
-
- all_checks_passed = True
- for check_str, description in checks:
- if check_str in content:
- print(f"✓ PASS | {description}")
- else:
- print(f"✗ FAIL | {description} - not found")
- all_checks_passed = False
-
- print("-" * 60)
- return all_checks_passed
-
-
-def test_documentation_consistency():
- """Verify documentation mentions the new feature."""
- print("\nTesting documentation consistency...")
- print("-" * 60)
-
- import pathlib
-
- docs_to_check = [
- ("configuration.mdx", "CUA_USE_HOST_COMPUTER_SERVER"),
- ("usage.mdx", "Targeting Your Local Desktop"),
- ]
-
- docs_path = (
- pathlib.Path(__file__).parent.parent
- / "docs"
- / "content"
- / "docs"
- / "libraries"
- / "mcp-server"
- )
-
- all_docs_ok = True
- for doc_file, expected_content in docs_to_check:
- doc_path = docs_path / doc_file
- if not doc_path.exists():
- print(f"✗ FAIL | {doc_file} not found")
- all_docs_ok = False
- continue
-
- content = doc_path.read_text()
- if expected_content in content:
- print(f"✓ PASS | {doc_file} contains '{expected_content}'")
- else:
- print(f"✗ FAIL | {doc_file} missing '{expected_content}'")
- all_docs_ok = False
-
- print("-" * 60)
- return all_docs_ok
-
-
-def print_usage_examples():
- """Print usage examples for both modes."""
- print("\n" + "=" * 60)
- print("USAGE EXAMPLES")
- print("=" * 60)
-
- print("\n1. DEFAULT MODE (VM):")
- print("-" * 60)
- print(
- """
-{
- "mcpServers": {
- "cua-agent": {
- "command": "/bin/bash",
- "args": ["~/.cua/start_mcp_server.sh"],
- "env": {
- "CUA_MODEL_NAME": "anthropic/claude-sonnet-4-5-20250929"
- }
- }
- }
-}
-
-Note: CUA_USE_HOST_COMPUTER_SERVER is not set, so VM mode is used (safe).
-"""
- )
-
- print("\n2. LOCAL DESKTOP MODE:")
- print("-" * 60)
- print(
- """
-Step 1: Start computer-server locally:
- python -m computer_server
-
-Step 2: Configure MCP client:
-{
- "mcpServers": {
- "cua-agent": {
- "command": "/bin/bash",
- "args": ["~/.cua/start_mcp_server.sh"],
- "env": {
- "CUA_MODEL_NAME": "anthropic/claude-sonnet-4-5-20250929",
- "CUA_USE_HOST_COMPUTER_SERVER": "true"
- }
- }
- }
-}
-
-⚠️ WARNING: AI will have direct access to your desktop!
-"""
- )
-
-
-def main():
- """Run all quick tests."""
- print("=" * 60)
- print("QUICK TEST: MCP Server Local Desktop Option")
- print("=" * 60)
- print()
-
- results = []
-
- # Run tests
- results.append(("Environment Variable Parsing", test_env_var_parsing()))
- results.append(("Session Manager Logic", test_session_manager_logic()))
- results.append(("Documentation Consistency", test_documentation_consistency()))
-
- # Print summary
- print("\n" + "=" * 60)
- print("SUMMARY")
- print("=" * 60)
- for test_name, passed in results:
- status = "✓ PASSED" if passed else "✗ FAILED"
- print(f"{status} | {test_name}")
-
- all_passed = all(result for _, result in results)
-
- if all_passed:
- print("\n🎉 All quick tests passed!")
- print_usage_examples()
- print("\nNext steps:")
- print("1. Run full automated tests: pytest tests/test_mcp_server_local_option.py")
- print("2. Follow manual testing guide: tests/MANUAL_TEST_LOCAL_OPTION.md")
- return 0
- else:
- print("\n❌ Some tests failed. Please review the output above.")
- return 1
-
-
-if __name__ == "__main__":
- sys.exit(main())