Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Add human demonstration recorder to docs (#888)
ddupont6e001b292d4c
diff --git a/docs/content/docs/cua/guide/advanced/demonstration-recorder.mdx b/docs/content/docs/cua/guide/advanced/demonstration-recorder.mdxnew file mode 100644index 0000000000..8dc16ee2ce--- /dev/null+++ b/docs/content/docs/cua/guide/advanced/demonstration-recorder.mdx@@ -0,0 +1,78 @@+---+title: Human Demonstration Recorder+description: Record human demonstrations from CUA sandboxes for training and evaluation+---++CUA sandboxes include a [noVNC fork](https://github.com/trycua/novnc) with built-in recording capabilities. This lets you capture human demonstrations as replayable session files for training data collection, evaluation baselines, or debugging.++<video controls width="100%">+ <source src="/videos/demonstration-recorder.mp4" type="video/mp4" />+</video>++## Manual Recording++### 1. Start a sandbox with the VNC UI++```bash+./utils/novnc_proxy --vnc localhost:5900 --listen localhost:6081+```++### 2. Enable recording++Add `autorecord=1` to the URL to automatically start recording when you connect:++```+http://localhost:6081/vnc.html?host=localhost&port=6081&autoconnect=1&autorecord=1+```++Or click the **record button** in the control bar before connecting.++### 3. Perform your demonstration++Interact with the sandbox normally. All VNC traffic (screen updates and input events) is captured.++### 4. Download the recording++Click **Stop Recording**, then **Download** to save the session as a `.js` file.++### 5. Replay the recording++Open the playback UI with your recording:++```+http://localhost:6081/tests/vnc_playback.html?data=../recordings/your-recording.js+```++## Streaming to External Server++For automated pipelines, stream recordings directly to a custom server using the `record_url` parameter:++```bash+# Start the recording server+python3 utils/recording_server.py --port 6090 --output demo.bin --js++# Connect with streaming enabled+http://localhost:6081/vnc.html?host=localhost&port=6081&autoconnect=1&autorecord=1&record_url=ws://localhost:6090+```++Or use the integrated novnc_proxy flags:++```bash+./utils/novnc_proxy --vnc localhost:5900 --listen localhost:6081 \+ --autoconnect --record-server --record-output demo.bin --record-js+```++Recording data streams to disk in real-time via WebSocket.++## URL Parameters++| Parameter | Description |+| --------------------- | ---------------------------------------------- |+| `autorecord=1` | Start recording when connection is established |+| `record_url=ws://...` | Stream recording to external WebSocket server |++## Using Demonstrations to Prompt Agents++Recordings are raw noVNC data streams, but they can be processed into semantic action traces that improve agent planning and execution. An example of an agent that will support this capability is [ShowUI-Aloha](https://github.com/showlab/ShowUI-Aloha), which learns workflows from human demonstrations and generalizes to new task variants.++_Coming soon_diff --git a/docs/content/docs/cua/guide/advanced/meta.json b/docs/content/docs/cua/guide/advanced/meta.jsonindex 7a47a83863..5638a78b8c 100644--- a/docs/content/docs/cua/guide/advanced/meta.json+++ b/docs/content/docs/cua/guide/advanced/meta.json@@ -8,6 +8,7 @@"sandboxed-python","local-computer-server","human-in-the-loop",+ "demonstration-recorder","trajectories","tracing","chat-history",