Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): explain permanent scope recovery (#2857)

Francesco Bonacci9110d9fd69c7
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 49bd4529f6..d6a3da1e86 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -642,7 +642,7 @@ Declare a session — a named, color-coded identity for THIS agent run. Pass a s
**Arguments:**
-- `capture_scope` (string, optional): Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session. default: `"auto"`
+- `capture_scope` (string, optional): Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session. default: `"auto"`
- `cursor_theme` (object or null, optional): Optional initial cursor theme. The host applies it before the cursor is
first made visible, avoiding a flash of the default theme.
- `session` (string, required): Stable session id for this run (e.g. "research-run-1").
@@ -986,7 +986,7 @@ Perform hover, right-click, double-click, scroll, or drag in an exactly-bound br
### `escalate_session`
-Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.
+Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session, then start_session with a new session id.
**Arguments:**
diff --git a/libs/cua-driver/contract/manifest.json b/libs/cua-driver/contract/manifest.json
index 0860a225f3..389cae4b73 100644
--- a/libs/cua-driver/contract/manifest.json
+++ b/libs/cua-driver/contract/manifest.json
@@ -584,7 +584,7 @@
},
{
"name": "escalate_session",
- "description": "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.",
+ "description": "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session, then start_session with a new session id.",
"platforms": [
"macos",
"windows",
@@ -2554,7 +2554,7 @@
"properties": {
"capture_scope": {
"default": "auto",
- "description": "Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.",
+ "description": "Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session.",
"enum": [
"auto",
"window",
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index f8856ff54f..97cf20a84f 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -366,7 +366,9 @@ sessions may choose different policies safely.
- `auto` (default): begins with effective scope `window`. Desktop perception
and actions are locked until the window ladder is exhausted, each attempted
action is verified, and the caller explicitly invokes `escalate_session`.
- Escalation is one-way for the live session.
+ Escalation is permanent for that session, and window-scoped tools remain
+ disabled. To recover window scope, call `end_session`, then call
+ `start_session` with a new session id.
- `window`: strict window-only perception and actions. Desktop tools are always
rejected with `desktop_scope_disabled`.
- `desktop`: strict full-desktop perception and foreground/system actions.
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs
index 3d0c8bf810..4cd11eeb1c 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs
@@ -266,7 +266,7 @@ impl ScrollBy {
pub struct StartSessionInput {
/// Stable session id for this run (e.g. "research-run-1").
pub session: String,
- /// Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.
+ /// Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session.
#[serde(default, skip_serializing_if = "Option::is_none")]
#[schemars(schema_with = "capture_scope_schema")]
pub capture_scope: Option<CaptureScope>,
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs
index 74a9ae3210..739d81a150 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/session.rs
@@ -52,7 +52,7 @@ fn start() -> ToolContract {
fn escalate() -> ToolContract {
contract::<EscalateSessionInput, SessionStateOutput>(
"escalate_session",
- "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.",
+ "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session, then start_session with a new session id.",
&["session.capture_scope.escalate"],
ToolAnnotations {
read_only: false,
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs
index 4574758947..19d8d1c0b7 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs
@@ -295,13 +295,22 @@ pub fn enforce_tool(tool_name: &str, args: &Value) -> Result<(), ScopeViolation>
),
state,
}),
- (EffectiveCaptureScope::Desktop, ToolScope::Window) => Err(ScopeViolation {
- code: "window_scope_disabled",
- message: format!(
- "window-scope tool '{tool_name}' is disabled while session '{session}' is in desktop scope"
- ),
- state,
- }),
+ (EffectiveCaptureScope::Desktop, ToolScope::Window) => {
+ let message = if state.policy == CaptureScopePolicy::Auto {
+ format!(
+ "window-scope tool '{tool_name}' is disabled because escalation to desktop scope is permanent for session '{session}'; to recover, call end_session for session '{session}', then call start_session with a new session id"
+ )
+ } else {
+ format!(
+ "window-scope tool '{tool_name}' is disabled while session '{session}' is in desktop scope"
+ )
+ };
+ Err(ScopeViolation {
+ code: "window_scope_disabled",
+ message,
+ state,
+ })
+ }
}
}
@@ -351,6 +360,39 @@ mod tests {
);
}
+ #[test]
+ fn escalated_auto_session_reports_permanent_scope_and_recovery() {
+ let session = fresh("auto-recovery");
+ bind_session(&session, Some(CaptureScopePolicy::Auto)).unwrap();
+ escalate_session(
+ &session,
+ EscalationReason::ForegroundIneffective,
+ Some("window ladder exhausted"),
+ )
+ .unwrap();
+
+ let violation = enforce_tool("get_window_state", &json!({"session": session})).unwrap_err();
+ assert_eq!(violation.code, "window_scope_disabled");
+ assert_eq!(
+ violation.message,
+ format!(
+ "window-scope tool 'get_window_state' is disabled because escalation to desktop scope is permanent for session '{session}'; to recover, call end_session for session '{session}', then call start_session with a new session id"
+ )
+ );
+ assert_eq!(
+ violation.as_json(&session),
+ json!({
+ "session": session,
+ "capture_scope": "auto",
+ "effective_scope": "desktop",
+ "desktop_unlocked": true,
+ "escalation_reason": "foreground_ineffective",
+ "escalation_detail": "window ladder exhausted",
+ "code": "window_scope_disabled"
+ })
+ );
+ }
+
#[test]
fn conflicting_live_rebind_is_rejected() {
let session = fresh("conflict");
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs
index cd469ab0ba..86ba8146a5 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs
@@ -112,6 +112,22 @@ fn policies_are_isolated_immutable_and_enforced_over_mcp() {
json!({"session": "scope-auto"}),
);
assert_eq!(code(&auto_window), Some("window_scope_disabled"));
+ assert_eq!(
+ auto_window["result"]["structuredContent"],
+ json!({
+ "session": "scope-auto",
+ "capture_scope": "auto",
+ "effective_scope": "desktop",
+ "desktop_unlocked": true,
+ "escalation_reason": "foreground_ineffective",
+ "escalation_detail": "window ladder exhausted",
+ "code": "window_scope_disabled"
+ })
+ );
+ assert_eq!(
+ auto_window["result"]["content"][0]["text"],
+ "window-scope tool 'get_window_state' is disabled because escalation to desktop scope is permanent for session 'scope-auto'; to recover, call end_session for session 'scope-auto', then call start_session with a new session id"
+ );
let conflict = call(
&mut driver,
diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
index bec6290e08..ff50e94b3b 100644
--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts
@@ -2791,7 +2791,7 @@ export type StartSessionInput = {
*/
session: string,
/**
- * Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.
+ * Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; escalation permanently switches that session to desktop scope. To recover window scope, call end_session, then start_session with a new session id. window and desktop are strict. Immutable for the live session.
*/
captureScope?: CaptureScope,
/**