Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver): publish Computer History preview docs (#3212)
Francesco Bonacci0225d3653f45
diff --git a/docs/content/docs/how-to-guides/driver/meta.json b/docs/content/docs/how-to-guides/driver/meta.jsonindex d93de66869..dc9feb1138 100644--- a/docs/content/docs/how-to-guides/driver/meta.json+++ b/docs/content/docs/how-to-guides/driver/meta.json@@ -2,6 +2,7 @@"title": "Driver","pages": ["install",+ "use-computer-history","use-sdk-in-process","verify-a-desktop-action","use-claude-agent-sdk",diff --git a/docs/content/docs/how-to-guides/driver/use-computer-history.mdx b/docs/content/docs/how-to-guides/driver/use-computer-history.mdxnew file mode 100644index 0000000000..e04a910c08--- /dev/null+++ b/docs/content/docs/how-to-guides/driver/use-computer-history.mdx@@ -0,0 +1,218 @@+---+title: Use Computer History+description: Enable, inspect, query, and delete the encrypted Computer History preview in Cua Driver nightly builds.+---++import { Callout } from 'fumadocs-ui/components/callout';+import { Tabs, Tab } from 'fumadocs-ui/components/tabs';++Computer History keeps an encrypted local record of actions performed through+Cua Driver. This guide covers the early preview in nightly macOS, Windows, and+Linux builds.++<Callout type="info">+ Computer History is off by default. It records metadata for Cua-mediated actions after you enable+ it. It does not watch unrelated desktop activity.+</Callout>++## Before you start++- Use an interactive macOS, Windows, or Linux desktop session.+- Grant Cua Driver the operating-system permissions needed for the actions you+ want it to perform. Computer History grants no new action permissions.+- On Linux, run an unlocked Secret Service implementation such as GNOME+ Keyring. History fails closed when no unlocked Secret Service is available.++## Install the nightly build++<Tabs items={['macOS and Linux', 'Windows']}>+ <Tab value="macOS and Linux">++```bash+/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" -- --channel nightly+```++ </Tab>+ <Tab value="Windows">++```powershell+& ([scriptblock]::Create((irm https://cua.ai/driver/install.ps1))) -Channel nightly+```++ </Tab>+</Tabs>++To move an existing installation to nightly, run:++```bash+cua-driver channel set nightly+cua-driver update --apply+cua-driver channel status+```++## Enable history++```bash+cua-driver history enable+cua-driver history status+```++`history enable` initializes a device-local native credential, verifies an+encrypted write and read, and starts capture. The default retention period is+7 days, and the encrypted-store quota is 100 MiB.++The store uses the current user's macOS Keychain, Windows Credential Manager,+or Linux Secret Service. There is no plaintext fallback, and capture performs+no network I/O.++## Inspect recent events++List the newest 50 events:++```bash+cua-driver history list 50+```++Show one event by sequence number:++```bash+cua-driver history show 42+```++Add `--json` to `status`, `list`, or `show` when another local program needs a+structured response.++## Let an agent consult history++An admitted Cua Driver runtime exposes two read-only tools:++- `history_status`, authorized by `history.status`;+- `history_query`, authorized by `history.query`.++Connect the agent through the normal Cua Driver MCP or SDK path. The active+permission mode, policy ceiling, and capability manifest authorize every+call. A query returns at most 200 metadata events and appends an encrypted+access-audit event when it returns data.++Tool availability does not make an agent consult history automatically. For+continuation and recent-work requests, give the agent a trusted history-first+instruction or have the host perform the status and bounded-query preflight.+See the [agent integration reference](/reference/cua-driver/computer-history)+for the consultation flow and permission contract.++The agent tools cannot enable, pause, resume, disable, delete, or export+history. They cannot retrieve encryption keys.++## Pause, resume, or disable capture++```bash+cua-driver history pause+cua-driver history resume+cua-driver history disable+```++Pause and disable preserve existing encrypted history. A permitted agent can+still query that history while the runtime admits the preview.++## Delete history++```bash+cua-driver history delete --yes+```++This destroys the exact namespace key and deletes the encrypted store. It+does not claim physical erasure from backups, filesystem snapshots, copied+ciphertext, SSD wear leveling, or memory that a process already decrypted.++To delete history while uninstalling:++<Tabs items={['macOS and Linux', 'Windows']}>+ <Tab value="macOS and Linux">++```bash+/bin/bash -c "$(curl -fsSL https://cua.ai/driver/uninstall.sh)" -- --purge+```++Run this as the interactive login user, without `sudo`, so native credential+cleanup stays in that user's context.++ </Tab>+ <Tab value="Windows">++```powershell+$env:CUA_DRIVER_RS_UNINSTALL_FORCE = '1'+$env:CUA_DRIVER_RS_UNINSTALL_PURGE = '1'+irm https://cua.ai/driver/uninstall.ps1 | iex+```++ </Tab>+</Tabs>++## Return to stable++```bash+cua-driver history disable+cua-driver channel set stable+cua-driver update --apply+```++Changing channels preserves encrypted history unless you explicitly delete or+purge it.++## Data boundaries++History may contain timestamps, opaque session and action IDs, Cua capability+names, fixed application identity fields, and fixed delivery, route, evidence,+effect, lifecycle, access, and health categories.++It never stores screenshots, video, audio, typed text, raw keystrokes,+clipboard contents, raw tool arguments or results, accessibility trees, file+paths, window titles, URLs, or free-form diagnostics.++Default encrypted-store locations are:++- macOS: `~/Library/Application Support/cua-driver/computer-history`;+- Windows: `%LOCALAPPDATA%\cua-driver\computer-history`;+- Linux: `$XDG_STATE_HOME/cua-driver/computer-history`, or+ `~/.local/state/cua-driver/computer-history` when `XDG_STATE_HOME` is unset.++The filesystem can reveal that the directory exists, its total size, and file+modification times. Copying only the encrypted files to another machine does+not provide recovery because the namespace key stays in the current user's+native credential store.++## Troubleshooting++### `history_preview_not_admitted`++The running development daemon lacks preview admission. Restart it with:++```bash+cua-driver serve --experimental-history+```++Installed nightly builds handle the verified relaunch during `history enable`.++### `history_key_locked` or `history_key_unavailable`++Unlock the login session and its native credential store, then retry. On+Linux, confirm that a Secret Service implementation is running. Capture stays+disabled, and Cua Driver creates no plaintext history files.++### `history_quota_reached`++Delete history if you no longer need it. Reaching the quota does not block the+computer action that encountered the full store.++### `history_storage_corrupt`++The reader found an invalid, incomplete, reordered, or unauthenticated record.+It refuses the affected store. If you accept permanent data loss, run+`cua-driver history delete --yes` and enable history again.++## Related documentation++- [Computer History agent integration](/reference/cua-driver/computer-history)+- [Permission modes](/reference/cua-driver/permission-modes)+- [Permission policies](/reference/cua-driver/permission-policies)+- [Install Cua Driver](/how-to-guides/driver/install)diff --git a/docs/content/docs/reference/cua-driver/computer-history.mdx b/docs/content/docs/reference/cua-driver/computer-history.mdxnew file mode 100644index 0000000000..c1468db253--- /dev/null+++ b/docs/content/docs/reference/cua-driver/computer-history.mdx@@ -0,0 +1,265 @@+---+title: Computer History Agent Integration+description: Tool discovery, permissions, query semantics, privacy boundaries, and consultation behavior for agent hosts.+---++import { Callout } from 'fumadocs-ui/components/callout';++Computer History is an opt-in encrypted local record of actions performed+through Cua Driver. Agent runtimes can read bounded metadata through two+permission-gated tools. They cannot control capture or access the encryption+key and encrypted chunks.++<Callout type="info">+ This contract covers the early preview in nightly macOS, Windows, and Linux builds. Discover tools+ at runtime and tolerate their absence.+</Callout>++## Integration flow++```mermaid+sequenceDiagram+ participant A as Agent runtime+ participant T as Cua tool registry+ participant P as Cua permission system+ participant H as Encrypted local history++ A->>T: Discover tools+ alt History tools absent+ T-->>A: Continue without history+ else History tools present+ A->>T: history_status({})+ T->>P: Authorize history.status+ P-->>T: Allow or deny+ T-->>A: Structured status or denial+ opt A bounded read is useful+ A->>T: history_query(filters)+ T->>P: Authorize history.query+ P-->>T: Allow or deny+ T->>H: Decrypt, validate, filter, and bound+ H-->>T: Metadata-only events+ T->>H: Append encrypted access record+ T-->>A: Events and context disclosure+ end+ end+```++The access record is appended only when a successful query returns at least+one event. The response that caused it does not include the new access record.++## Availability and feature detection++The runtime registers the tools only when the desktop daemon admits the+preview. Tool presence does not prove that the user granted the calling agent+access.++| Observed state | Meaning | Host behavior |+| --------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- |+| Neither tool is advertised | The runtime does not admit the preview or the platform does not support it. | Continue without history. |+| `history_status` is advertised | The runtime admits the preview. | Request `history.status` permission before reading status. |+| `enabled: false` | Capture is off. Earlier encrypted history may remain. | Query only when prior history is useful and authorized. |+| `paused: true` | New capture is paused. | Treat history after the pause point as incomplete. |+| Drops or unhealthy storage are reported | The record may contain gaps. | Preserve the warning and avoid completeness claims. |++## Consultation behavior++Tool discovery makes history available to a model. It does not cause the model+to call either tool. A history-aware host should consult history when the user+asks it to continue, resume, recall recent Cua activity, explain a prior Cua+run, or locate where a Cua-mediated workflow stopped.++For a matching request, the host should:++1. discover the history tools before broader desktop inspection;+2. call `history_status` and retain disabled, paused, unhealthy, and+ dropped-event state;+3. when useful and authorized, call `history_query` with a bounded recent+ slice;+4. treat events as metadata evidence rather than a transcript;+5. keep omitted content, geometry, arguments, results, and user intent+ unknown;+6. use the returned application or capability as a lead and verify current+ state through the least intrusive source; and+7. continue without history after absence, denial, empty results, or a+ recoverable history failure.++A trusted system instruction or bundled skill can guide this behavior. A host+may instead perform the status and query preflight before model execution when+deterministic consultation is required.++| Integration level | Required behavior | Accurate claim |+| -------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |+| Tool-capable | The runtime advertises the tools and schemas. | Agents can query Computer History. |+| History-aware | A trusted policy directs consultation for matching requests and handles fallbacks. | The agent checks Computer History for recent-work and continuation requests. |+| Deterministic consultation | The host enforces status and bounded-query preflight before broader discovery. | The agent automatically checks Computer History for matching requests. |++Prompt wording can guide tool selection but does not establish deterministic+consultation.++## `history_status`++Returns operational metadata without returning history events.++**Required capability:** `history.status`++**Properties:** read-only, non-destructive, idempotent, closed-world++**Input:** an empty object. Unknown fields are rejected.++Important response fields are:++| Field | Type | Meaning |+| ---------------- | ------- | ------------------------------------------------------------------ |+| `supported` | boolean | The platform adapter supports the preview. |+| `admitted` | boolean | The daemon admitted the experimental feature. |+| `enabled` | boolean | The user enabled capture. |+| `paused` | boolean | New action capture is paused. |+| `encrypted` | boolean | The encrypted storage profile is active. Preview 0 returns `true`. |+| `profile` | string | Storage profile identifier. |+| `retention_days` | integer | Query-visible retention. Default: `7`. |+| `quota_bytes` | integer | Encrypted-store quota. Default: `104857600`. |+| `bytes_used` | integer | Current encrypted bytes under the history root. |+| `dropped_events` | integer | Events dropped by the nonblocking capture path. |+| `health` | string | Fixed health category. |++Health categories are `ready`, `disabled`, `paused`, `not_admitted`,+`key_unavailable`, `key_locked`, `key_corrupt`, `key_destroy_failed`,+`storage_unavailable`, `storage_corrupt`, `quota_reached`, `events_dropped`, and+`writer_stopped`.++## `history_query`++Returns a bounded event slice that may enter the current model context.++**Required capability:** `history.query`++**Properties:** read-only, non-destructive, closed-world. A successful+non-empty query appends an encrypted access record, so the call is not+idempotent.++| Field | Type | Required | Bounds | Meaning |+| ---------------- | ------- | -------- | ------------------- | ------------------------------------------------------------------------------------------------- |+| `limit` | integer | No | `1..200` | Maximum events. Default: `50`. |+| `session_id` | string | No | `1..128` characters | Opaque ID returned by history, or a caller-known session label resolved in the history namespace. |+| `since_sequence` | integer | No | `>=1` | Inclusive lower sequence bound. |+| `until_sequence` | integer | No | `>=1` | Inclusive upper sequence bound. |++Unknown fields are rejected. When both sequence bounds are present,+`since_sequence` must not exceed `until_sequence`.++Events are ordered by `data.sequence`. The query applies every filter, retains+the newest `limit` matching events, and returns that slice in ascending order.+The preview has no opaque pagination token. Missing sequence numbers are valid+gap evidence because capture uses a bounded nonblocking queue.++Every response includes:++```json+{+ "events": [],+ "metadata_only": true,+ "model_context_disclosure": true+}+```++## Event contract++Events use CloudEvents 1.0 JSON and+`urn:cua-driver:schema:history-event:v0`.++| Event type | Payload kind | Meaning |+| ---------------------------------------- | ------------------ | --------------------------------------------------------- |+| `cua-driver.history.control.v0` | `control` | User lifecycle operation such as enable, pause, or flush. |+| `cua-driver.history.action_started.v0` | `action_started` | A Cua-mediated action began. |+| `cua-driver.history.action_completed.v0` | `action_completed` | The validated action outcome. |+| `cua-driver.history.session_started.v0` | `session` | A Cua Driver lifecycle session began. |+| `cua-driver.history.session_ended.v0` | `session` | A Cua Driver lifecycle session ended. |+| `cua-driver.history.access.v0` | `access` | A local CLI or agent query returned events. |+| `cua-driver.history.health.v0` | `health` | A fixed writer-health or dropped-event marker. |++Clients must branch on both `dataschema` and `type`. Stop interpreting an event+whose schema is unsupported.++## Permission contract++Status and query are separate private-observation operations. Permission for+`history.status` does not grant `history.query`.++In bounded mode, the approved manifest must name the tools and the matching+computer-history operations:++```yaml+version: 3+expires_after: 1h+idle_timeout: 10m+resources:+ computer_history:+ operations:+ - status+ - query+allow:+ tools:+ - history_status+ - history_query+```++An agent may propose this manifest. The trusted launcher selects and approves+it. Clients must surface a denial and continue without history. They must not+read the store directly, change permission modes, or reconstruct denied+history with another observation tool.++## Privacy and storage boundary++Returned events may contain timestamps, opaque identifiers, capability names,+fixed application identity fields, and fixed outcome, route, delivery,+evidence, lifecycle, access, and health categories.++The contract prohibits screenshots, video, audio, accessibility trees, typed+text, raw keystrokes, clipboard contents, raw tool arguments or results, file+paths, window titles, URLs, and free-form diagnostics.++Each CloudEvent is encrypted and authenticated inside a COSE_Encrypt0 record.+Records use RFC 8742 CBOR Sequence framing. A native credential-store key and+per-chunk HKDF keys protect data at rest. History payloads remain local and+history tools are excluded from per-tool and agent-session telemetry.++## Errors++Tool failures return a structured `code`. Authorization may deny the call+before tool execution through the normal authorization envelope.++| Code | Host behavior |+| ------------------------------ | ---------------------------------------------------------------------- |+| `invalid_history_query` | Correct the request once. Do not retry it unchanged. |+| `invalid_history_query_range` | Correct the sequence bounds. |+| `history_preview_not_admitted` | Refresh tool discovery and continue without history. |+| `history_key_unavailable` | Report history as unavailable. |+| `history_key_locked` | Let the user unlock the native credential store. |+| `history_key_corrupt` | Stop querying and direct the user to local recovery controls. |+| `history_storage_unavailable` | Continue the primary task without history. |+| `history_storage_corrupt` | Stop consuming results and direct the user to local recovery controls. |+| `history_quota_reached` | Treat history after that point as incomplete. |+| `history_events_dropped` | Treat the affected interval as incomplete. |+| `history_writer_stopped` | Do not assume new events are recorded. |++## Version identifiers++| Contract | Identifier |+| ----------------- | --------------------------------------------------------------------- |+| Status tool | `history_status` |+| Query tool | `history_query` |+| Status capability | `history.status` |+| Query capability | `history.query` |+| Event schema | `urn:cua-driver:schema:history-event:v0` |+| Storage profile | `cua-history-profile-v1/cbor-sequence+cose-encrypt0+cloudevents-json` |++The tool and capability names are intended to remain stable. The event schema+is experimental. Clients must use runtime tool discovery and advertised input+schemas.++## Source contracts++- [Full agent integration RFC](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-agent-integration-rfc.md)+- [Event schema](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-event-v0.schema.json)+- [Encrypted storage profile](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-profile-v1.cddl)+- [Use Computer History](/how-to-guides/driver/use-computer-history)diff --git a/docs/content/docs/reference/cua-driver/meta.json b/docs/content/docs/reference/cua-driver/meta.jsonindex f7166e3f51..4b3db9e3c9 100644--- a/docs/content/docs/reference/cua-driver/meta.json+++ b/docs/content/docs/reference/cua-driver/meta.json@@ -4,6 +4,7 @@"cli-reference","sdk-reference","telemetry",+ "computer-history","macos-permissions","embedding","mcp-tools",