Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): explain permanent scope recovery (#2857)
Francesco Bonacci9110d9fd69c7
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 49bd4529f6..d6a3da1e86 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -642,7 +642,7 @@ Declare a session — a named, color-coded identity for THIS agent run. Pass a s**Arguments:**-- `capture_scope` (string, optional): Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session. default: `"auto"`+- `capture_scope` (string, optional): Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session. default: `"auto"`- `cursor_theme` (object or null, optional): Optional initial cursor theme. The host applies it before the cursor isfirst made visible, avoiding a flash of the default theme.- `session` (string, required): Stable session id for this run (e.g. "research-run-1").@@ -986,7 +986,7 @@ Perform hover, right-click, double-click, scroll, or drag in an exactly-bound br### `escalate_session`-Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.+Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session, then start_session with a new session id.**Arguments:**diff --git a/libs/cua-driver/contract/manifest.json b/libs/cua-driver/contract/manifest.jsonindex 0860a225f3..389cae4b73 100644--- a/libs/cua-driver/contract/manifest.json+++ b/libs/cua-driver/contract/manifest.json@@ -584,7 +584,7 @@},{"name": "escalate_session",- "description": "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.",+ "description": "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session, then start_session with a new session id.","platforms": ["macos","windows",@@ -2554,7 +2554,7 @@"properties": {"capture_scope": {"default": "auto",- "description": "Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.",+ "description": "Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session.","enum": ["auto","window",diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex f8856ff54f..97cf20a84f 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -366,7 +366,9 @@ sessions may choose different policies safely.- `auto` (default): begins with effective scope `window`. Desktop perceptionand actions are locked until the window ladder is exhausted, each attemptedaction is verified, and the caller explicitly invokes `escalate_session`.- Escalation is one-way for the live session.+ Escalation is permanent for that session, and window-scoped tools remain+ disabled. To recover window scope, call `end_session`, then call+ `start_session` with a new session id.- `window`: strict window-only perception and actions. Desktop tools are alwaysrejected with `desktop_scope_disabled`.- `desktop`: strict full-desktop perception and foreground/system actions.diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rsindex 3d0c8bf810..4cd11eeb1c 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs@@ -266,7 +266,7 @@ impl ScrollBy {pub struct StartSessionInput {/// Stable session id for this run (e.g. "research-run-1").pub session: String,- /// Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.+ /// Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session.#[serde(default, skip_serializing_if = "Option::is_none")]#[schemars(schema_with = "capture_scope_schema")]pub capture_scope: Option<CaptureScope>,diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rsindex 74a9ae3210..739d81a150 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs@@ -52,7 +52,7 @@ fn start() -> ToolContract {fn escalate() -> ToolContract {contract::<EscalateSessionInput, SessionStateOutput>("escalate_session",- "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.",+ "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session, then start_session with a new session id.",&["session.capture_scope.escalate"],ToolAnnotations {read_only: false,diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rsindex 4574758947..19d8d1c0b7 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs@@ -295,13 +295,22 @@ pub fn enforce_tool(tool_name: &str, args: &Value) -> Result<(), ScopeViolation>),state,}),- (EffectiveCaptureScope::Desktop, ToolScope::Window) => Err(ScopeViolation {- code: "window_scope_disabled",- message: format!(- "window-scope tool '{tool_name}' is disabled while session '{session}' is in desktop scope"- ),- state,- }),+ (EffectiveCaptureScope::Desktop, ToolScope::Window) => {+ let message = if state.policy == CaptureScopePolicy::Auto {+ format!(+ "window-scope tool '{tool_name}' is disabled because escalation to desktop scope is permanent for session '{session}'; to recover, call end_session for session '{session}', then call start_session with a new session id"+ )+ } else {+ format!(+ "window-scope tool '{tool_name}' is disabled while session '{session}' is in desktop scope"+ )+ };+ Err(ScopeViolation {+ code: "window_scope_disabled",+ message,+ state,+ })+ }}}@@ -351,6 +360,39 @@ mod tests {);}+ #[test]+ fn escalated_auto_session_reports_permanent_scope_and_recovery() {+ let session = fresh("auto-recovery");+ bind_session(&session, Some(CaptureScopePolicy::Auto)).unwrap();+ escalate_session(+ &session,+ EscalationReason::ForegroundIneffective,+ Some("window ladder exhausted"),+ )+ .unwrap();++ let violation = enforce_tool("get_window_state", &json!({"session": session})).unwrap_err();+ assert_eq!(violation.code, "window_scope_disabled");+ assert_eq!(+ violation.message,+ format!(+ "window-scope tool 'get_window_state' is disabled because escalation to desktop scope is permanent for session '{session}'; to recover, call end_session for session '{session}', then call start_session with a new session id"+ )+ );+ assert_eq!(+ violation.as_json(&session),+ json!({+ "session": session,+ "capture_scope": "auto",+ "effective_scope": "desktop",+ "desktop_unlocked": true,+ "escalation_reason": "foreground_ineffective",+ "escalation_detail": "window ladder exhausted",+ "code": "window_scope_disabled"+ })+ );+ }+#[test]fn conflicting_live_rebind_is_rejected() {let session = fresh("conflict");diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rsindex cd469ab0ba..86ba8146a5 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs@@ -112,6 +112,22 @@ fn policies_are_isolated_immutable_and_enforced_over_mcp() {json!({"session": "scope-auto"}),);assert_eq!(code(&auto_window), Some("window_scope_disabled"));+ assert_eq!(+ auto_window["result"]["structuredContent"],+ json!({+ "session": "scope-auto",+ "capture_scope": "auto",+ "effective_scope": "desktop",+ "desktop_unlocked": true,+ "escalation_reason": "foreground_ineffective",+ "escalation_detail": "window ladder exhausted",+ "code": "window_scope_disabled"+ })+ );+ assert_eq!(+ auto_window["result"]["content"][0]["text"],+ "window-scope tool 'get_window_state' is disabled because escalation to desktop scope is permanent for session 'scope-auto'; to recover, call end_session for session 'scope-auto', then call start_session with a new session id"+ );let conflict = call(&mut driver,diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.tsindex bec6290e08..ff50e94b3b 100644--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts@@ -2791,7 +2791,7 @@ export type StartSessionInput = {*/session: string,/**- * Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.+ * Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session.*/captureScope?: CaptureScope,/**