Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

4.4 KB
Raw Blame

Multi-cursor background computer-use demo: "National Records System"

A set of deliberately legacy-looking government records terminals (navy banner, UNCLASSIFIED // FOR OFFICIAL USE ONLY strip, function-key bar, green-screen records grid, status line): the kind of internal agency app that, in the age of AI, has no automation integration. cua-driver automates them anyway.

One human action in the foreground "master" terminal is replayed onto four background terminals at the same time, each driven by its own cua-driver session = its own uniquely-coloured agent cursor, with no window ever raised and the user's mouse never moved.

It also proves cua-driver works with or without an accessibility tree: the five windows span five UI frameworks, and cua-driver's default dispatch auto-selects UIA-Invoke where an a11y tree exists and falls back to pixel/pointer-injection where it doesn't.

Layout (each window = ½ work-width × ½ work-height)

The four corners tile the taskbar-safe work area into quadrants; the master is centered, overlapping all four:

 ┌────────────────────────┬────────────────────────┐
 │ Win32 GDI (NO a11y)     │ WinForms (.NET)         │
 │            crimson ●    │            amber ●      │
 │           ┌────────────────────────┐             │
 │           │ MASTER - Win32 controls │  ← you      │
 ├───────────│ (foreground, overlaps)  │─────────────┤
 │ WPF (XAML)│                         │ Electron    │
 │           └────────────────────────┘ mint_lime ●  │
 │            aqua ●       │            (Chromium)    │
 └────────────────────────┴────────────────────────┘

Click SUBMIT (or type a subject name then submit) in the center master: four coloured cursors glide onto the four corner terminals and commit the same record there, concurrently, in the background. Watch each corner's green-screen records grid grow and its RECORDS: counter tick up, without any corner ever coming to the front.

Frameworks (and what they exercise)

Window Framework Accessibility cua-driver path
TL Win32 + GDI (custom-drawn) none pixel hit-test → PostMessage / pointer injection
TR .NET WinForms MSAA/UIA UIA Invoke
BL .NET WPF UIA (XAML) UIA Invoke (no foreground steal via WS_EX_NOACTIVATE)
BR Electron UIA (Chromium) UIA Invoke
Center Win32 standard controls MSAA (foreground; the human drives it)

Build

# from this directory
cargo build                                   # legacy-app + orchestrator (Rust)
dotnet build dotnet/winforms/winforms.csproj  # WinForms
dotnet build dotnet/wpf/wpf.csproj            # WPF
npm install --prefix electron                 # Electron (downloads electron once)

Also build the driver once (repo root workspace):

cargo build -p cua-driver --manifest-path ..\..\libs\cua-driver\rust\Cargo.toml

Run

.\target\debug\orchestrator.exe            # human-driven: click/type in the center
.\target\debug\orchestrator.exe --auto     # self-playing: drives a TYPE+CLICK every few seconds

The orchestrator starts the cua-driver daemon, launches + positions all five windows, and fans every center action out to the four corners over four concurrent cua-driver call sessions (crimson / amber / aqua / mint_lime → four cursor colours). Close the center window (or kill the orchestrator) to tear everything down. A Windows Job Object kills the whole tree, so nothing is orphaned.

Env overrides

CUA_DRIVER_EXE, LEGACY_APP_EXE, WINFORMS_EXE, WPF_EXE, ELECTRON_DIR.

How the coloured cursors work

cua-driver assigns each session a cursor colour by name (palette-name sessions like crimson pick that colour directly). Passing "session":"<color>" on each click/type_text call routes it to that session's overlay cursor, which glides to the target. Four sessions → four cursors animating at once. See WINDOWS.md for the no-z-raise mechanism.