Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): send value-free task progress so Cua-S1 finishes jev-use native tasks (#4324)
Francesco Bonacci907bb88e2efd
diff --git a/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx b/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdxindex 3ab56af934..6708c5674e 100644--- a/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx+++ b/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx@@ -56,8 +56,10 @@ try background delivery first. If Driver refuses that, the next step offers aseparate foreground click when the task allows it.Native requests use the `cua.jev_choice_request_v2` contract, which adds a-`source` to each candidate (`page`, `ax`, or `visual`). Browser tasks keep-sending `cua.jev_choice_request_v1`.+`source` to each candidate (`page`, `ax`, or `visual`). The built-in tasks+also send `progress`: each required step and how many times this run has+performed it. The runner counts its own successful actions and never reads the+count from the app. Browser tasks keep sending `cua.jev_choice_request_v1`.## Before you startdiff --git a/libs/cua-driver/examples/jev-use/README.md b/libs/cua-driver/examples/jev-use/README.mdindex 9729491e33..a11ab958df 100644--- a/libs/cua-driver/examples/jev-use/README.md+++ b/libs/cua-driver/examples/jev-use/README.md@@ -91,7 +91,11 @@ or closing are excluded unless the task allows that risk. Text comes only fromtask parameters. Actions are element-bound `click`, `set_value`, or`type_text` with background delivery first; a stale token leads to a freshobservation. Native steps send `cua.jev_choice_request_v2`, which adds a-per-candidate `source` and compact value-free `elements`.+per-candidate `source` and compact value-free `elements`. A task that declares+its required steps also sends `progress`: each step with how often this run has+performed it, counted only from the runner's own successful actions and never+read from the app. A step's candidate also says which earlier step must be done+first, for example Save waits for the note to be entered.Visual regions from OmniParser are parsed, from the same `capture_id`, onlywhen the tree offers nothing to act through: Driver reports `ax_tree_empty`, a@@ -170,7 +174,9 @@ The service is not part of this example. Any server that loads the pinnedcheckpoint from [`libs/cua-s1`](../../../cua-s1/README.md), validates therequest with `python/choose_action.py`, and answers with`decision_models.choose(S1DecisionModel(...), request).to_wire()` fits the-contract. For example, to verify the native tasks with it:+contract. Run the service from the same revision as the runners: an older+validator rejects a newer optional field such as `progress`, and the runner+then stops with a logged `S1ServiceError` outcome. For example, to verify the native tasks with it:```bashexport CUA_S1_DECISION_URL=http://127.0.0.1:8791/decidediff --git a/libs/cua-driver/examples/jev-use/fixtures/native/native-progress-requests-v2.json b/libs/cua-driver/examples/jev-use/fixtures/native/native-progress-requests-v2.jsonnew file mode 100644index 0000000000..d65649fe98--- /dev/null+++ b/libs/cua-driver/examples/jev-use/fixtures/native/native-progress-requests-v2.json@@ -0,0 +1,386 @@+{+ "_fixture": "Native v2 requests with task progress (#4313); python/tests/test_native_progress.py and typescript/native_progress.test.ts must both reproduce each request exactly from fixtures/native/appkit-window-state-initial-v1.json and the recorded runner history.",+ "cases": [+ {+ "task": "appkit-save-note",+ "history": [],+ "request": {+ "schema": "cua.jev_choice_request_v2",+ "goal": "Enter the note text into the Note field, then save the note.",+ "capture_id": "cap-appkit-initial",+ "snapshot_id": "s00000086",+ "regions": [],+ "elements": [+ {+ "role_class": "button",+ "label": "Increment",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Reset",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Click target (left / right / double)",+ "state": "enabled"+ },+ {+ "role_class": "checkbox",+ "label": "I agree",+ "state": "unchecked"+ },+ {+ "role_class": "button",+ "label": "Right-click for context menu",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Exit",+ "state": "enabled"+ },+ {+ "role_class": "text_input",+ "label": "Note",+ "state": "empty"+ },+ {+ "role_class": "button",+ "label": "Save note",+ "state": "enabled"+ },+ {+ "role_class": "radio",+ "label": "Small",+ "state": "not_selected"+ },+ {+ "role_class": "radio",+ "label": "Medium",+ "state": "not_selected"+ },+ {+ "role_class": "radio",+ "label": "Large",+ "state": "not_selected"+ }+ ],+ "history": [],+ "candidates": [+ {+ "id": "ax:button:increment",+ "description": "Press the button labeled \"Increment\".",+ "source": "ax"+ },+ {+ "id": "ax:button:click-target-left-right-double",+ "description": "Press the button labeled \"Click target (left / right / double)\".",+ "source": "ax"+ },+ {+ "id": "ax:button:right-click-for-context-menu",+ "description": "Press the button labeled \"Right-click for context menu\".",+ "source": "ax"+ },+ {+ "id": "ax:text_input:note:set:note",+ "description": "Set the text field \"Note\" to the task parameter \"note\", replacing its contents. The field currently reports empty. The task still requires this 1 more time(s).",+ "source": "ax"+ },+ {+ "id": "ax:button:save-note",+ "description": "Press the button labeled \"Save note\". The task requires this only after: Set the text field \"Note\" to the task parameter \"note\" (not done yet).",+ "source": "ax"+ },+ {+ "id": "reobserve",+ "description": "Take no action and obtain a fresh observation of the window, because the current observation looks stale, incomplete, or contradicts the goal's progress."+ },+ {+ "id": "abstain",+ "description": "Stop without acting if none of the proposed actions is safe or moves toward the goal."+ }+ ],+ "progress": [+ {+ "step": "Set the text field \"Note\" to the task parameter \"note\"",+ "done": 0,+ "required": 1+ },+ {+ "step": "Press the button labeled \"Save note\"",+ "done": 0,+ "required": 1+ }+ ]+ }+ },+ {+ "task": "appkit-save-note",+ "history": [+ {+ "step": 1,+ "selected_id": "ax:text_input:note:set:note",+ "outcome": "set \"Note\" to parameter note",+ "performed": true+ }+ ],+ "request": {+ "schema": "cua.jev_choice_request_v2",+ "goal": "Enter the note text into the Note field, then save the note.",+ "capture_id": "cap-appkit-initial",+ "snapshot_id": "s00000086",+ "regions": [],+ "elements": [+ {+ "role_class": "button",+ "label": "Increment",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Reset",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Click target (left / right / double)",+ "state": "enabled"+ },+ {+ "role_class": "checkbox",+ "label": "I agree",+ "state": "unchecked"+ },+ {+ "role_class": "button",+ "label": "Right-click for context menu",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Exit",+ "state": "enabled"+ },+ {+ "role_class": "text_input",+ "label": "Note",+ "state": "empty"+ },+ {+ "role_class": "button",+ "label": "Save note",+ "state": "enabled"+ },+ {+ "role_class": "radio",+ "label": "Small",+ "state": "not_selected"+ },+ {+ "role_class": "radio",+ "label": "Medium",+ "state": "not_selected"+ },+ {+ "role_class": "radio",+ "label": "Large",+ "state": "not_selected"+ }+ ],+ "history": [+ {+ "selected_id": "ax:text_input:note:set:note",+ "outcome": "set \"Note\" to parameter note"+ }+ ],+ "candidates": [+ {+ "id": "ax:button:increment",+ "description": "Press the button labeled \"Increment\".",+ "source": "ax"+ },+ {+ "id": "ax:button:click-target-left-right-double",+ "description": "Press the button labeled \"Click target (left / right / double)\".",+ "source": "ax"+ },+ {+ "id": "ax:button:right-click-for-context-menu",+ "description": "Press the button labeled \"Right-click for context menu\".",+ "source": "ax"+ },+ {+ "id": "ax:text_input:note:set:note",+ "description": "Set the text field \"Note\" to the task parameter \"note\", replacing its contents. The field currently reports empty. This run already did this the 1 time(s) the task requires.",+ "source": "ax"+ },+ {+ "id": "ax:button:save-note",+ "description": "Press the button labeled \"Save note\". The task still requires this 1 more time(s).",+ "source": "ax"+ },+ {+ "id": "reobserve",+ "description": "Take no action and obtain a fresh observation of the window, because the current observation looks stale, incomplete, or contradicts the goal's progress."+ },+ {+ "id": "abstain",+ "description": "Stop without acting if none of the proposed actions is safe or moves toward the goal."+ }+ ],+ "progress": [+ {+ "step": "Set the text field \"Note\" to the task parameter \"note\"",+ "done": 1,+ "required": 1+ },+ {+ "step": "Press the button labeled \"Save note\"",+ "done": 0,+ "required": 1+ }+ ]+ }+ },+ {+ "task": "appkit-counter",+ "history": [+ {+ "step": 1,+ "selected_id": "ax:button:increment",+ "outcome": "pressed \"Increment\"",+ "performed": true+ },+ {+ "step": 2,+ "selected_id": "ax:button:increment",+ "outcome": "pressed \"Increment\"",+ "performed": true+ },+ {+ "step": 3,+ "selected_id": "ax:button:increment",+ "outcome": "Driver refused background delivery (background_unsupported); nothing happened"+ }+ ],+ "request": {+ "schema": "cua.jev_choice_request_v2",+ "goal": "The counter starts at 0. Set it to exactly 3 by pressing Increment once per step, then stop.",+ "capture_id": "cap-appkit-initial",+ "snapshot_id": "s00000086",+ "regions": [],+ "elements": [+ {+ "role_class": "button",+ "label": "Increment",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Reset",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Click target (left / right / double)",+ "state": "enabled"+ },+ {+ "role_class": "checkbox",+ "label": "I agree",+ "state": "unchecked"+ },+ {+ "role_class": "button",+ "label": "Right-click for context menu",+ "state": "enabled"+ },+ {+ "role_class": "button",+ "label": "Exit",+ "state": "enabled"+ },+ {+ "role_class": "text_input",+ "label": "Note",+ "state": "empty"+ },+ {+ "role_class": "button",+ "label": "Save note",+ "state": "enabled"+ },+ {+ "role_class": "radio",+ "label": "Small",+ "state": "not_selected"+ },+ {+ "role_class": "radio",+ "label": "Medium",+ "state": "not_selected"+ },+ {+ "role_class": "radio",+ "label": "Large",+ "state": "not_selected"+ }+ ],+ "history": [+ {+ "selected_id": "ax:button:increment",+ "outcome": "pressed \"Increment\""+ },+ {+ "selected_id": "ax:button:increment",+ "outcome": "pressed \"Increment\""+ },+ {+ "selected_id": "ax:button:increment",+ "outcome": "Driver refused background delivery (background_unsupported); nothing happened"+ }+ ],+ "candidates": [+ {+ "id": "ax:button:increment",+ "description": "Press the button labeled \"Increment\". The task still requires this 1 more time(s).",+ "source": "ax"+ },+ {+ "id": "ax:button:click-target-left-right-double",+ "description": "Press the button labeled \"Click target (left / right / double)\".",+ "source": "ax"+ },+ {+ "id": "ax:button:right-click-for-context-menu",+ "description": "Press the button labeled \"Right-click for context menu\".",+ "source": "ax"+ },+ {+ "id": "ax:button:save-note",+ "description": "Press the button labeled \"Save note\".",+ "source": "ax"+ },+ {+ "id": "reobserve",+ "description": "Take no action and obtain a fresh observation of the window, because the current observation looks stale, incomplete, or contradicts the goal's progress."+ },+ {+ "id": "abstain",+ "description": "Stop without acting if none of the proposed actions is safe or moves toward the goal."+ }+ ],+ "progress": [+ {+ "step": "Press the button labeled \"Increment\"",+ "done": 2,+ "required": 3+ }+ ]+ }+ }+ ]+}diff --git a/libs/cua-driver/examples/jev-use/python/choose_action.py b/libs/cua-driver/examples/jev-use/python/choose_action.pyindex e558cbdfcf..df2cde5e3f 100644--- a/libs/cua-driver/examples/jev-use/python/choose_action.py+++ b/libs/cua-driver/examples/jev-use/python/choose_action.py@@ -11,7 +11,9 @@ from native_roles import ROLE_CLASSESREQUEST_SCHEMA = "cua.jev_choice_request_v1"# Additive over v1 (RFC #4268): per-candidate ``source``, an optional root-# ``snapshot_id``, and an optional compact ``elements`` list of native controls.+# ``snapshot_id``, an optional compact ``elements`` list of native controls, and+# an optional ``progress`` list of task steps counted from the runner's own+# performed actions (#4313).REQUEST_SCHEMA_V2 = "cua.jev_choice_request_v2"REQUEST_SCHEMAS = (REQUEST_SCHEMA, REQUEST_SCHEMA_V2)RESPONSE_SCHEMA = "cua.jev_choice_v1"@@ -20,6 +22,8 @@ MAX_CANDIDATES = 32MAX_REGIONS = 100MAX_HISTORY = 16MAX_ELEMENTS = 64+MAX_PROGRESS = 16+MAX_PROGRESS_COUNT = 64ID_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,63}\Z")RESERVED_IDS = frozenset({"reobserve", "abstain"})CANDIDATE_SOURCES = frozenset({"page", "ax", "visual"})@@ -27,7 +31,7 @@ ELEMENT_STATES = frozenset({"enabled", "checked", "unchecked", "selected", "not_selected", "empty", "has_text"})V1_ROOT_KEYS = frozenset({"schema", "goal", "capture_id", "regions", "history", "candidates"})-V2_OPTIONAL_ROOT_KEYS = frozenset({"snapshot_id", "elements"})+V2_OPTIONAL_ROOT_KEYS = frozenset({"snapshot_id", "elements", "progress"})def _record(value: Any, message: str) -> dict[str, Any]:@@ -71,13 +75,46 @@ def _validate_elements(value: Any) -> list[dict[str, str]]:return elements+def _count(value: Any, name: str, minimum: int) -> int:+ if (+ not isinstance(value, int)+ or isinstance(value, bool)+ or not minimum <= value <= MAX_PROGRESS_COUNT+ ):+ raise ValueError(f"{name} must be an integer from {minimum} to {MAX_PROGRESS_COUNT}")+ return value+++def _validate_progress(value: Any) -> list[dict[str, Any]]:+ """Validate task steps and how often this run has performed each one.++ ``done`` is counted from the runner's own performed actions, never read+ from the application, so ``progress`` carries no application values.+ """+ if not isinstance(value, list) or len(value) > MAX_PROGRESS:+ raise ValueError(f"progress must be an array of at most {MAX_PROGRESS} items")+ progress: list[dict[str, Any]] = []+ for raw_value in value:+ raw = _record(raw_value, "progress item must be an object")+ if set(raw) != {"step", "done", "required"}:+ raise ValueError("progress item may contain only step, done, and required")+ progress.append(+ {+ "step": _bounded_string(raw["step"], "progress step", 200),+ "done": _count(raw["done"], "progress done", 0),+ "required": _count(raw["required"], "progress required", 1),+ }+ )+ return progress++def validate_request(value: Any) -> dict[str, Any]:"""Validate a ``cua.jev_choice_request_v1`` or ``_v2`` request strictly.v1 is unchanged: exactly its six root keys and ``id``/``description``candidates, so any v2 field in a v1 request is rejected. v2 may add a root- ``snapshot_id`` and ``elements`` and a per-candidate ``source``; reserved- candidates never carry a source.+ ``snapshot_id``, ``elements``, and ``progress`` and a per-candidate+ ``source``; reserved candidates never carry a source."""root = _record(value, "request must be a JSON object")schema = root.get("schema")@@ -217,6 +254,7 @@ def validate_request(value: Any) -> dict[str, Any]:None if snapshot_id is None else _bounded_string(snapshot_id, "snapshot_id", 64))validated["elements"] = _validate_elements(root.get("elements", []))+ validated["progress"] = _validate_progress(root.get("progress", []))return validated@@ -224,8 +262,9 @@ def provider_observation(validated: Mapping[str, Any]) -> dict[str, Any]:"""Build the observation a provider sees for a validated request.A v1 request keeps its exact v1 observation. A v2 request adds the native- ``snapshot_id``, the compact ``elements``, and each candidate's ``source``;- none of these contain element tokens, values, or pixels.+ ``snapshot_id``, the compact ``elements``, and each candidate's ``source``,+ plus ``progress`` when the request carries any; none of these contain+ element tokens, values, or pixels."""observation: dict[str, Any] = {"capture_id": validated["capture_id"],@@ -238,6 +277,8 @@ def provider_observation(validated: Mapping[str, Any]) -> dict[str, Any]:observation["candidate_sources"] = {item["id"]: item["source"] for item in validated["candidates"] if "source" in item}+ if validated.get("progress"):+ observation["progress"] = validated["progress"]return observationdiff --git a/libs/cua-driver/examples/jev-use/python/decision_models.py b/libs/cua-driver/examples/jev-use/python/decision_models.pyindex d59069dbaf..8691832477 100644--- a/libs/cua-driver/examples/jev-use/python/decision_models.py+++ b/libs/cua-driver/examples/jev-use/python/decision_models.py@@ -27,10 +27,12 @@ class DecisionRequest:regions: tuple[Mapping[str, Any], ...]history: tuple[Mapping[str, str], ...]candidates: tuple[Mapping[str, str], ...]- # v2 only (RFC #4268): native snapshot and compact accessibility elements.+ # v2 only (RFC #4268): native snapshot and compact accessibility elements,+ # and task steps counted from the runner's own performed actions (#4313).schema: str = REQUEST_SCHEMAsnapshot_id: str | None = Noneelements: tuple[Mapping[str, str], ...] = ()+ progress: tuple[Mapping[str, Any], ...] = ()@classmethoddef from_validated(cls, request: Mapping[str, Any]) -> "DecisionRequest":@@ -43,6 +45,7 @@ class DecisionRequest:schema=request.get("schema", REQUEST_SCHEMA),snapshot_id=request.get("snapshot_id"),elements=tuple(MappingProxyType(dict(item)) for item in request.get("elements", ())),+ progress=tuple(MappingProxyType(dict(item)) for item in request.get("progress", ())),)@property@@ -61,6 +64,7 @@ class DecisionRequest:if self.schema == REQUEST_SCHEMA_V2:validated["snapshot_id"] = self.snapshot_idvalidated["elements"] = [dict(item) for item in self.elements]+ validated["progress"] = [dict(item) for item in self.progress]return provider_observation(validated)@@ -219,7 +223,8 @@ def native_elements_as_text(request: DecisionRequest) -> str:"""Render a v2 request's native elements as an accessibility tree for S1.Each line is one supplied native control: its role class, label, and state.- Visual regions from the same capture, if any, follow as a separate section.+ Visual regions from the same capture, if any, follow as a separate section,+ then the prior decisions and, when supplied, the task progress."""lines = [f"Accessibility tree for snapshot {json.dumps(request.snapshot_id)} "@@ -237,6 +242,20 @@ def native_elements_as_text(request: DecisionRequest) -> str:if request.history:lines.append("Prior bounded decisions:")lines.extend(json.dumps(item, ensure_ascii=False) for item in request.history)+ if request.progress:+ lines.append(progress_as_text(request.progress))+ return "\n".join(lines)+++def progress_as_text(progress: tuple[Mapping[str, Any], ...]) -> str:+ """Render task progress; counts come from the run's actions, not the app."""+ lines = [+ "Task progress (counted from this run's own completed actions, not read from the app):"+ ]+ for item in progress:+ done, required = item["done"], item["required"]+ status = f"{required - done} remaining" if done < required else "complete"+ lines.append(f"- {item['step']}: done {done} of {required} times; {status}")return "\n".join(lines)diff --git a/libs/cua-driver/examples/jev-use/python/native_tasks.py b/libs/cua-driver/examples/jev-use/python/native_tasks.pyindex 2a45dabaab..bcf22d9822 100644--- a/libs/cua-driver/examples/jev-use/python/native_tasks.py+++ b/libs/cua-driver/examples/jev-use/python/native_tasks.py@@ -29,7 +29,13 @@ from dataclasses import dataclass, field, replacefrom pathlib import Pathfrom typing import Any, Callable, Literal, Mapping-from choose_action import MAX_ELEMENTS, MAX_HISTORY, REQUEST_SCHEMA_V2+from choose_action import (+ MAX_ELEMENTS,+ MAX_HISTORY,+ MAX_PROGRESS,+ MAX_PROGRESS_COUNT,+ REQUEST_SCHEMA_V2,+)from native import (PARAMETER_NAME_PATTERN,NativeControl,@@ -180,6 +186,36 @@ def compose(return kept + list(reserved), stats+@dataclass(frozen=True)+class TaskStep:+ """One step a task requires, counted from the runner's own performed actions.++ ``candidate_id`` names the candidate that performs the step; its+ ``:foreground`` variant counts too. The description is task-authored and+ value-free: it may name a parameter, never its value. A step with+ ``after_previous`` must wait for every earlier step to be done.+ """++ description: str+ candidate_id: str+ times: int = 1+ after_previous: bool = True+++def performed_counts(history: list[Mapping[str, Any]]) -> dict[str, int]:+ """Count the actions this run dispatched successfully, by candidate ID.++ Only history entries marked ``performed`` count: a refused, stale, or+ reobserve step did nothing. Nothing here is read from the application.+ """+ counts: dict[str, int] = {}+ for item in history:+ if item.get("performed") is True:+ base = str(item["selected_id"]).removesuffix(":foreground")+ counts[base] = counts.get(base, 0) + 1+ return counts++@dataclass(frozen=True)class NativeStep:"""One step's closed candidate set and its model-safe context."""@@ -214,6 +250,11 @@ class NativeTask:# browser path. Exact-text uniqueness still applies at any bar.visual_min_confidence: float = 0.8mock_preferences: tuple[str, ...] = ()+ # Ordered steps the task requires (#4313). The request reports how often+ # this run has performed each, and a step's candidate names any earlier+ # step that is not done yet, so a model need not infer order or count+ # from history. Empty means the request carries no progress.+ steps: tuple[TaskStep, ...] = ()# The oracle is polled after every action, so no candidate is special.completion_candidate_ids: frozenset[str] = field(default=frozenset(), init=False)@@ -226,6 +267,11 @@ class NativeTask:raise ValueError("parameter names must match [a-z][a-z0-9_]{0,7}")if self.max_steps < 1:raise ValueError("max_steps must be positive")+ if len(self.steps) > MAX_PROGRESS:+ raise ValueError(f"a task declares at most {MAX_PROGRESS} steps")+ for task_step in self.steps:+ if not 1 <= task_step.times <= MAX_PROGRESS_COUNT:+ raise ValueError(f"step times must be from 1 to {MAX_PROGRESS_COUNT}")@propertydef allowed_action_kinds(self) -> frozenset[str]:@@ -403,7 +449,59 @@ class NativeTask:text = "took no action and requested a fresh observation"else:text = outcome or "completed"- return {"step": step, "selected_id": candidate_id, "outcome": self.redact(text)[:128]}+ entry: dict[str, Any] = {+ "step": step,+ "selected_id": candidate_id,+ "outcome": self.redact(text)[:128],+ }+ if not stale and refusal is None and candidate_id not in {"reobserve", "abstain"}:+ entry["performed"] = True # runner-side only; never sent to a provider+ return entry++ # -- progress ---------------------------------------------------------++ def progress(self, history: list[Mapping[str, Any]]) -> list[dict[str, Any]]:+ """Each declared step and how often this run has performed it."""+ counts = performed_counts(history)+ return [+ {+ "step": self.redact(task_step.description),+ "done": min(counts.get(task_step.candidate_id, 0), MAX_PROGRESS_COUNT),+ "required": task_step.times,+ }+ for task_step in self.steps+ ]++ def step_note(self, candidate_id: str, history: list[Mapping[str, Any]]) -> str:+ """A sentence stating a candidate's place in the task's declared steps.++ A step that is already done says so; a step with an earlier step not yet+ done names that step as its precondition; a step that is due says how+ many more times the task requires it. Other candidates get nothing.+ """+ base = candidate_id.removesuffix(":foreground")+ counts = performed_counts(history)+ for index, task_step in enumerate(self.steps):+ if task_step.candidate_id != base:+ continue+ if counts.get(base, 0) >= task_step.times:+ return (+ f" This run already did this the {task_step.times} time(s) the task requires."+ )+ pending = [+ earlier.description+ for earlier in (self.steps[:index] if task_step.after_previous else ())+ if counts.get(earlier.candidate_id, 0) < earlier.times+ ]+ if pending:+ return (+ " The task requires this only after: "+ + "; ".join(pending)+ + " (not done yet)."+ )+ remaining = task_step.times - counts.get(base, 0)+ return f" The task still requires this {remaining} more time(s)."+ return ""def reset(self) -> None:"""The harness starts fresh for every run; there is nothing to reset."""@@ -488,19 +586,21 @@ def native_choice_request("""Build the ``cua.jev_choice_request_v2`` the provider receives.It carries candidate IDs, descriptions and sources, compact value-free- elements, and compact history. Element tokens, values, and pixels stay in- the runner.+ elements, compact history, and, for a task that declares steps, the+ progress counted from this run's performed actions. Element tokens,+ values, and pixels stay in the runner."""if sources.ax is None or sources.ax.observation.capture_id is None:raise ValueError("a native request needs an observation with a capture_id")observation = sources.ax.observationcandidates = []for candidate in step.candidates:- item: dict[str, Any] = {"id": candidate.id, "description": task.redact(candidate.description)}+ description = candidate.description + task.step_note(candidate.id, history)+ item: dict[str, Any] = {"id": candidate.id, "description": task.redact(description)}if candidate.source is not None:item["source"] = candidate.sourcecandidates.append(item)- return {+ request: dict[str, Any] = {"schema": REQUEST_SCHEMA_V2,"goal": task.redact(task.goal),"capture_id": observation.capture_id,@@ -513,6 +613,9 @@ def native_choice_request(],"candidates": candidates,}+ if task.steps:+ request["progress"] = task.progress(history)+ return request# -- Harness tasks ------------------------------------------------------------@@ -686,6 +789,9 @@ def native_task(allow_foreground=allow_foreground,max_steps=COUNTER_TARGET + 3,mock_preferences=("ax:button:increment",),+ steps=(+ TaskStep('Press the button labeled "Increment"', "ax:button:increment", COUNTER_TARGET),+ ),)if kind == "save-note":return NativeTask(@@ -699,6 +805,13 @@ def native_task(allow_foreground=allow_foreground,max_steps=5,mock_preferences=("ax:text_input:note:set:note", "ax:button:save-note"),+ steps=(+ TaskStep(+ 'Set the text field "Note" to the task parameter "note"',+ "ax:text_input:note:set:note",+ ),+ TaskStep('Press the button labeled "Save note"', "ax:button:save-note"),+ ),)return NativeTask(id=task_id,@@ -710,6 +823,11 @@ def native_task(allow_foreground=allow_foreground,max_steps=5,mock_preferences=("ax:radio:large", "ax:checkbox:i-agree"),+ steps=(+ TaskStep('Select the radio option "Large"', "ax:radio:large"),+ # The oracle accepts either order, so neither step waits for the other.+ TaskStep('Toggle the checkbox "I agree"', "ax:checkbox:i-agree", after_previous=False),+ ),)diff --git a/libs/cua-driver/examples/jev-use/python/run_native.py b/libs/cua-driver/examples/jev-use/python/run_native.pyindex 215d0e8a59..11c772dca2 100644--- a/libs/cua-driver/examples/jev-use/python/run_native.py+++ b/libs/cua-driver/examples/jev-use/python/run_native.py@@ -43,7 +43,7 @@ from native_tasks import (native_choice_request,visual_fallback_reason,)-from s1_service import choose_s1_service, s1_service_url+from s1_service import S1ServiceError, choose_s1_service, s1_service_urlfrom run import Driver, DriverToolError, background_refusal_code, supports_capture_bound_clickfrom sources import NativeAccessibilitySource, VisualRegionSourcefrom tasks import TaskSources@@ -284,7 +284,17 @@ async def run_task(args: argparse.Namespace, task: NativeTask) -> str:task, sources, plan.candidates, history)elif args.provider == "s1":- choice, confidence, probabilities = await asyncio.to_thread(choose_s1_service, request)+ try:+ choice, confidence, probabilities = await asyncio.to_thread(+ choose_s1_service, request+ )+ except S1ServiceError as error:+ # A tied or malformed score is not an action; fail closed+ # with a logged outcome instead of a traceback.+ write_event(log_path, {"event": "outcome", "outcome": "unknown",+ "phase": "decide", "step": step,+ "error": "S1ServiceError", "reason": str(error)[:128]})+ return "unknown"else:choice, confidence, probabilities = await asyncio.to_thread(choose_live, request)decide_ms = round((time.perf_counter() - decide_started) * 1000, 2)diff --git a/libs/cua-driver/examples/jev-use/python/tests/test_native_progress.py b/libs/cua-driver/examples/jev-use/python/tests/test_native_progress.pynew file mode 100644index 0000000000..ebdaf7ad76--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/tests/test_native_progress.py@@ -0,0 +1,231 @@+"""Task progress in cua.jev_choice_request_v2 (#4313).++A native task declares the steps it requires. The request reports how often+this run has performed each step, counted only from the runner's own+successful actions, and a step's candidate names any earlier step that is not+done yet. Nothing in ``progress`` is read from the application.+"""++from __future__ import annotations++import copy+import json+import sys+import unittest+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from choose_action import provider_observation, validate_request+from decision_models import DecisionRequest, native_elements_as_text, progress_as_text+from native import NativeObservation+from native_tasks import (+ CANVAS_TASK_ID,+ HARNESSES,+ TaskStep,+ native_choice_request,+ native_task,+ performed_counts,+)+from sources import NativeAccessibilitySource+from tasks import TaskSources++ROOT = Path(__file__).resolve().parents[2]+FIXTURES = ROOT / "fixtures/native"+++def v2_request() -> dict:+ return json.loads((ROOT / "fixtures/jev-choice-request-v2.json").read_text(encoding="utf-8"))+++def v1_request() -> dict:+ return json.loads((ROOT / "fixtures/jev-choice-request-v1.json").read_text(encoding="utf-8"))+++def sources(task, harness: str, name: str = "initial") -> TaskSources:+ payload = json.loads((FIXTURES / f"{harness}-window-state-{name}-v1.json").read_text("utf-8"))+ observation = NativeObservation.from_window_state(+ payload, expected_pid=payload["pid"], expected_window_id=payload["window_id"]+ )+ ax = NativeAccessibilitySource.from_observation(+ observation, HARNESSES[harness].platform, redact=task.redact_text, text_method=task.text_method+ )+ return TaskSources(ax=ax)+++def request_for(task_id: str, history: list[dict], note_text: str = "jev-use native note") -> dict:+ task = native_task(task_id, Path("/tmp/none.json"), note_text=note_text)+ task_sources = sources(task, task_id.split("-")[0])+ return native_choice_request(task, task_sources, task.plan(task_sources), history)+++def performed(task, step: int, candidate_id: str) -> dict:+ return task.history_entry(step, candidate_id, outcome="done")+++class ProgressContractTest(unittest.TestCase):+ def test_v2_accepts_progress_and_the_observation_carries_it(self) -> None:+ request = v2_request()+ request["progress"] = [{"step": 'Press "Increment"', "done": 2, "required": 3}]+ validated = validate_request(request)+ self.assertEqual(validated["progress"], request["progress"])+ self.assertEqual(provider_observation(validated)["progress"], request["progress"])++ def test_v2_without_progress_keeps_its_observation(self) -> None:+ validated = validate_request(v2_request())+ self.assertEqual(validated["progress"], [])+ self.assertNotIn("progress", provider_observation(validated))++ def test_v1_rejects_progress(self) -> None:+ request = v1_request()+ request["progress"] = []+ with self.assertRaises(ValueError):+ validate_request(request)++ def test_v2_rejects_malformed_progress(self) -> None:+ item = {"step": "Press", "done": 0, "required": 1}+ mutations = {+ "not a list": {"step": "Press"},+ "too many": [item] * 17,+ "extra key": [{**item, "value": "secret"}],+ "missing key": [{"step": "Press", "done": 0}],+ "empty step": [{**item, "step": " "}],+ "long step": [{**item, "step": "x" * 201}],+ "negative done": [{**item, "done": -1}],+ "boolean done": [{**item, "done": True}],+ "float done": [{**item, "done": 1.0}],+ "zero required": [{**item, "required": 0}],+ "huge required": [{**item, "required": 65}],+ }+ for name, progress in mutations.items():+ request = copy.deepcopy(v2_request())+ request["progress"] = progress+ with self.subTest(name=name), self.assertRaises(ValueError):+ validate_request(request)++ def test_s1_text_renders_progress_after_history(self) -> None:+ request = v2_request()+ request["progress"] = [+ {"step": 'Press "Increment"', "done": 2, "required": 3},+ {"step": 'Press "Save note"', "done": 1, "required": 1},+ ]+ text = native_elements_as_text(DecisionRequest.from_validated(validate_request(request)))+ self.assertIn('- Press "Increment": done 2 of 3 times; 1 remaining', text)+ self.assertIn('- Press "Save note": done 1 of 1 times; complete', text)+ self.assertLess(text.index("Prior bounded decisions:"), text.index("Task progress"))++ def test_s1_text_without_progress_is_unchanged(self) -> None:+ request = DecisionRequest.from_validated(validate_request(v2_request()))+ self.assertNotIn("Task progress", native_elements_as_text(request))+ self.assertTrue(progress_as_text(()).startswith("Task progress"))++ def test_decision_request_observation_matches_the_validated_one(self) -> None:+ raw = v2_request()+ raw["progress"] = [{"step": "Press", "done": 0, "required": 1}]+ validated = validate_request(raw)+ request = DecisionRequest.from_validated(validated)+ self.assertEqual(request.provider_observation(), provider_observation(validated))+++class TaskProgressTest(unittest.TestCase):+ def test_only_performed_actions_count(self) -> None:+ task = native_task("appkit-counter", Path("/tmp/none.json"))+ history = [+ performed(task, 1, "ax:button:increment"),+ task.history_entry(2, "ax:button:increment", refusal="background_unsupported"),+ task.history_entry(3, "ax:button:increment", stale=True),+ task.history_entry(4, "reobserve"),+ performed(task, 5, "ax:button:increment:foreground"),+ ]+ self.assertEqual(performed_counts(history), {"ax:button:increment": 2})+ self.assertEqual(+ task.progress(history),+ [{"step": 'Press the button labeled "Increment"', "done": 2, "required": 3}],+ )++ def test_counter_request_reports_increments_done(self) -> None:+ task = native_task("appkit-counter", Path("/tmp/none.json"))+ history = [performed(task, 1, "ax:button:increment"), performed(task, 2, "ax:button:increment")]+ request = request_for("appkit-counter", history)+ validate_request(request)+ self.assertEqual(request["progress"][0]["done"], 2)+ increment = {c["id"]: c for c in request["candidates"]}["ax:button:increment"]+ self.assertTrue(increment["description"].endswith("still requires this 1 more time(s)."))+ self.assertEqual(request["progress"][0]["required"], 3)+ # The runner-side marker never reaches the provider.+ self.assertEqual(set(request["history"][0]), {"selected_id", "outcome"})++ def test_save_is_described_as_waiting_for_the_note(self) -> None:+ for harness in HARNESSES:+ with self.subTest(harness=harness):+ request = request_for(f"{harness}-save-note", [], note_text="secret note")+ save = {c["id"]: c for c in request["candidates"]}["ax:button:save-note"]+ self.assertIn(+ 'The task requires this only after: Set the text field "Note" to the task '+ 'parameter "note" (not done yet).',+ save["description"],+ )+ self.assertEqual([item["done"] for item in request["progress"]], [0, 0])+ self.assertNotIn("secret note", json.dumps(request))++ def test_save_precondition_clears_once_the_note_is_set(self) -> None:+ task = native_task("appkit-save-note", Path("/tmp/none.json"))+ request = request_for("appkit-save-note", [performed(task, 1, "ax:text_input:note:set:note")])+ save = {c["id"]: c for c in request["candidates"]}["ax:button:save-note"]+ self.assertEqual(+ save["description"],+ 'Press the button labeled "Save note". The task still requires this 1 more time(s).',+ )+ self.assertEqual([item["done"] for item in request["progress"]], [1, 0])++ def test_a_completed_step_says_so(self) -> None:+ task = native_task("appkit-counter", Path("/tmp/none.json"))+ history = [performed(task, step, "ax:button:increment") for step in (1, 2, 3)]+ request = request_for("appkit-counter", history)+ increment = {c["id"]: c for c in request["candidates"]}["ax:button:increment"]+ self.assertTrue(+ increment["description"].endswith("This run already did this the 3 time(s) the task requires.")+ )++ def test_unordered_steps_have_no_precondition(self) -> None:+ request = request_for("appkit-choose-size", [])+ agree = {c["id"]: c for c in request["candidates"]}["ax:checkbox:i-agree"]+ self.assertNotIn("only after", agree["description"])+ self.assertTrue(agree["description"].endswith("still requires this 1 more time(s)."))++ def test_a_task_without_steps_sends_no_progress(self) -> None:+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))+ self.assertEqual(task.steps, ())+ self.assertEqual(task.progress([]), [])++ def test_task_step_bounds(self) -> None:+ task = native_task("appkit-counter", Path("/tmp/none.json"))+ from dataclasses import replace++ with self.assertRaises(ValueError):+ replace(task, steps=(TaskStep("Press", "ax:button:increment", 0),))+ with self.assertRaises(ValueError):+ replace(task, steps=tuple(TaskStep("Press", f"id{i}") for i in range(17)))++ def test_recorded_requests_are_reproduced_exactly(self) -> None:+ golden = json.loads((FIXTURES / "native-progress-requests-v2.json").read_text("utf-8"))+ for case in golden["cases"]:+ with self.subTest(task=case["task"], steps=len(case["history"])):+ task = native_task(case["task"], Path("/tmp/none.json"))+ task_sources = sources(task, "appkit")+ request = native_choice_request(+ task, task_sources, task.plan(task_sources), case["history"]+ )+ self.assertEqual(request, case["request"])+ validate_request(request)++ def test_same_progress_on_every_harness(self) -> None:+ requests = {+ harness: request_for(f"{harness}-save-note", []) for harness in HARNESSES+ }+ progress = {json.dumps(r["progress"]) for r in requests.values()}+ self.assertEqual(len(progress), 1)+++if __name__ == "__main__":+ unittest.main()diff --git a/libs/cua-driver/examples/jev-use/typescript/choose_action.ts b/libs/cua-driver/examples/jev-use/typescript/choose_action.tsindex e193c783b8..bc352e0341 100644--- a/libs/cua-driver/examples/jev-use/typescript/choose_action.ts+++ b/libs/cua-driver/examples/jev-use/typescript/choose_action.ts@@ -8,7 +8,9 @@ import { ROLE_CLASSES, type RoleClass } from './native_roles.js';export const REQUEST_SCHEMA = 'cua.jev_choice_request_v1';// Additive over v1 (RFC #4268): per-candidate `source`, an optional root-// `snapshot_id`, and an optional compact `elements` list of native controls.+// `snapshot_id`, an optional compact `elements` list of native controls, and+// an optional `progress` list of task steps counted from the runner's own+// performed actions (#4313).export const REQUEST_SCHEMA_V2 = 'cua.jev_choice_request_v2';export const RESPONSE_SCHEMA = 'cua.jev_choice_v1';const MAX_INPUT_BYTES = 65_536;@@ -16,6 +18,8 @@ const MAX_CANDIDATES = 32;const MAX_REGIONS = 100;const MAX_HISTORY = 16;export const MAX_ELEMENTS = 64;+export const MAX_PROGRESS = 16;+export const MAX_PROGRESS_COUNT = 64;const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,63}$/;const RESERVED_IDS = new Set(['reobserve', 'abstain']);export type CandidateSourceKind = 'page' | 'ax' | 'visual';@@ -38,10 +42,11 @@ const ELEMENT_STATES = new Set<string>(['has_text',]);const V1_ROOT_KEYS = ['candidates', 'capture_id', 'goal', 'history', 'regions', 'schema'];-const V2_OPTIONAL_ROOT_KEYS = ['elements', 'snapshot_id'];+const V2_OPTIONAL_ROOT_KEYS = ['elements', 'progress', 'snapshot_id'];type JsonRecord = Record<string, unknown>;export type CompactElement = { role_class: RoleClass; label: string; state: ElementState };+export type ProgressItem = { step: string; done: number; required: number };export type ValidatedCandidate = { id: string; description: string; source?: CandidateSourceKind };export type ValidatedRequest = {schema: typeof REQUEST_SCHEMA | typeof REQUEST_SCHEMA_V2;@@ -52,6 +57,7 @@ export type ValidatedRequest = {candidates: ValidatedCandidate[];snapshot_id?: string | null;elements?: CompactElement[];+ progress?: ProgressItem[];};function record(value: unknown, message: string): JsonRecord {@@ -104,11 +110,40 @@ function validateElements(value: unknown): CompactElement[] {});}+function count(value: unknown, name: string, minimum: number): number {+ if (!Number.isInteger(value) || Number(value) < minimum || Number(value) > MAX_PROGRESS_COUNT) {+ throw new Error(`${name} must be an integer from ${minimum} to ${MAX_PROGRESS_COUNT}`);+ }+ return Number(value);+}++/**+ * Validate task steps and how often this run has performed each one. `done`+ * is counted from the runner's own performed actions, never read from the+ * application, so progress carries no application values.+ */+function validateProgress(value: unknown): ProgressItem[] {+ if (!Array.isArray(value) || value.length > MAX_PROGRESS) {+ throw new Error(`progress must be an array of at most ${MAX_PROGRESS} items`);+ }+ return value.map((item) => {+ const raw = record(item, 'progress item must be an object');+ if (!exactKeys(raw, ['done', 'required', 'step'])) {+ throw new Error('progress item may contain only step, done, and required');+ }+ return {+ step: boundedString(raw.step, 'progress step', 200),+ done: count(raw.done, 'progress done', 0),+ required: count(raw.required, 'progress required', 1),+ };+ });+}+/*** Validate a cua.jev_choice_request_v1 or _v2 request strictly. v1 is* unchanged, so any v2 field in a v1 request is rejected. v2 may add a root- * snapshot_id and elements and a per-candidate source; reserved candidates- * never carry a source.+ * snapshot_id, elements, and progress and a per-candidate source; reserved+ * candidates never carry a source.*/export function validateRequest(value: unknown): ValidatedRequest {const root = record(value, 'request must be a JSON object');@@ -269,13 +304,15 @@ export function validateRequest(value: unknown): ValidatedRequest {? null: boundedString(root.snapshot_id, 'snapshot_id', 64);validated.elements = validateElements(root.elements ?? []);+ validated.progress = validateProgress(root.progress ?? []);}return validated;}/*** The observation a provider sees. A v1 request keeps its exact v1- * observation; v2 adds the snapshot, compact elements, and candidate sources.+ * observation; v2 adds the snapshot, compact elements, and candidate sources,+ * plus progress when the request carries any.*/export function providerObservation(validated: ValidatedRequest): JsonRecord {const observation: JsonRecord = {@@ -289,6 +326,7 @@ export function providerObservation(validated: ValidatedRequest): JsonRecord {observation.candidate_sources = Object.fromEntries(validated.candidates.filter((item) => item.source).map((item) => [item.id, item.source]));+ if (validated.progress?.length) observation.progress = validated.progress;}return observation;}diff --git a/libs/cua-driver/examples/jev-use/typescript/native_progress.test.ts b/libs/cua-driver/examples/jev-use/typescript/native_progress.test.tsnew file mode 100644index 0000000000..8f5aaf71f5--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/native_progress.test.ts@@ -0,0 +1,146 @@+// Task progress in cua.jev_choice_request_v2 (#4313). Mirrors+// python/tests/test_native_progress.py: a native task declares the steps it+// requires, the request reports how often this run has performed each step+// (counted only from the runner's own successful actions), and a step's+// candidate names any earlier step that is not done yet.+import assert from 'node:assert/strict';+import { readFileSync } from 'node:fs';+import test from 'node:test';++import { providerObservation, validateRequest } from './choose_action.js';+import { parseWindowState } from './native.js';+import {+ CANVAS_TASK_ID,+ HARNESSES,+ NativeTask,+ nativeChoiceRequest,+ nativeTask,+ performedCounts,+} from './native_tasks.js';+import { NativeAccessibilitySource } from './sources.js';+import type { HistoryEntry, TaskSources } from './tasks.js';++const fixture = (path: string): any => JSON.parse(readFileSync(new URL(`../fixtures/${path}`, import.meta.url), 'utf8'));+const v2Request = () => fixture('jev-choice-request-v2.json');+const v1Request = () => fixture('jev-choice-request-v1.json');++function sources(task: NativeTask, harness: string): TaskSources {+ const payload = fixture(`native/${harness}-window-state-initial-v1.json`);+ return {+ ax: NativeAccessibilitySource.fromObservation(+ parseWindowState(payload, payload.pid, payload.window_id),+ HARNESSES[harness].platform,+ { redact: task.redactText, textMethod: task.textMethod }+ ),+ visualPath: false,+ foregroundIds: new Set(),+ };+}++function requestFor(taskId: string, history: HistoryEntry[], noteText = 'jev-use native note') {+ const task = nativeTask(taskId, '/tmp/none.json', { noteText });+ const taskSources = sources(task, taskId.split('-')[0]);+ return nativeChoiceRequest(task, taskSources, task.plan(taskSources), history) as any;+}++const performed = (task: NativeTask, step: number, id: string) => task.historyEntry(step, id, undefined, { outcome: 'done' });+const byId = (request: any) => Object.fromEntries(request.candidates.map((c: any) => [c.id, c]));++test('v2 accepts progress and the observation carries it', () => {+ const request = v2Request();+ request.progress = [{ step: 'Press "Increment"', done: 2, required: 3 }];+ const validated = validateRequest(request);+ assert.deepEqual(validated.progress, request.progress);+ assert.deepEqual(providerObservation(validated).progress, request.progress);+});++test('v2 without progress keeps its observation, and v1 rejects progress', () => {+ const validated = validateRequest(v2Request());+ assert.deepEqual(validated.progress, []);+ assert.equal('progress' in providerObservation(validated), false);+ const v1 = v1Request();+ v1.progress = [];+ assert.throws(() => validateRequest(v1));+});++test('v2 rejects malformed progress', () => {+ const item = { step: 'Press', done: 0, required: 1 };+ const mutations: Record<string, unknown> = {+ 'not a list': { step: 'Press' },+ 'too many': Array(17).fill(item),+ 'extra key': [{ ...item, value: 'secret' }],+ 'missing key': [{ step: 'Press', done: 0 }],+ 'empty step': [{ ...item, step: ' ' }],+ 'long step': [{ ...item, step: 'x'.repeat(201) }],+ 'negative done': [{ ...item, done: -1 }],+ 'boolean done': [{ ...item, done: true }],+ 'fractional done': [{ ...item, done: 0.5 }],+ 'zero required': [{ ...item, required: 0 }],+ 'huge required': [{ ...item, required: 65 }],+ };+ for (const [name, progress] of Object.entries(mutations)) {+ const request = v2Request();+ request.progress = progress;+ assert.throws(() => validateRequest(request), Error, name);+ }+});++test('only performed actions count', () => {+ const task = nativeTask('appkit-counter', '/tmp/none.json');+ const history = [+ performed(task, 1, 'ax:button:increment'),+ task.historyEntry(2, 'ax:button:increment', 'background_unsupported'),+ task.historyEntry(3, 'ax:button:increment', undefined, { stale: true }),+ task.historyEntry(4, 'reobserve'),+ performed(task, 5, 'ax:button:increment:foreground'),+ ];+ assert.deepEqual(performedCounts(history), { 'ax:button:increment': 2 });+ assert.deepEqual(task.progress(history), [+ { step: 'Press the button labeled "Increment"', done: 2, required: 3 },+ ]);+ const request = requestFor('appkit-counter', history);+ assert.deepEqual(Object.keys(request.history[0]).sort(), ['outcome', 'selected_id']);+});++test('save is described as waiting for the note until the note is set', () => {+ for (const harness of Object.keys(HARNESSES)) {+ const request = requestFor(`${harness}-save-note`, [], 'secret note');+ assert.match(+ byId(request)['ax:button:save-note'].description,+ /The task requires this only after: Set the text field "Note" to the task parameter "note" \(not done yet\)\.$/+ );+ assert.equal(JSON.stringify(request).includes('secret note'), false);+ }+ const task = nativeTask('appkit-save-note', '/tmp/none.json');+ const after = requestFor('appkit-save-note', [performed(task, 1, 'ax:text_input:note:set:note')]);+ assert.equal(+ byId(after)['ax:button:save-note'].description,+ 'Press the button labeled "Save note". The task still requires this 1 more time(s).'+ );+ assert.deepEqual(after.progress.map((item: any) => item.done), [1, 0]);+});++test('a completed step says so and unordered steps have no precondition', () => {+ const task = nativeTask('appkit-counter', '/tmp/none.json');+ const done = requestFor('appkit-counter', [1, 2, 3].map((step) => performed(task, step, 'ax:button:increment')));+ assert.match(byId(done)['ax:button:increment'].description, /already did this the 3 time\(s\) the task requires\.$/);+ const size = requestFor('appkit-choose-size', []);+ assert.equal(byId(size)['ax:checkbox:i-agree'].description.includes('only after'), false);+ assert.match(byId(size)['ax:checkbox:i-agree'].description, /still requires this 1 more time\(s\)\.$/);+});++test('a task without steps sends no progress', () => {+ const task = nativeTask(CANVAS_TASK_ID, '/tmp/none.json');+ assert.deepEqual(task.steps, []);+ assert.deepEqual(task.progress([]), []);+});++test('recorded Python requests are reproduced exactly', () => {+ for (const item of fixture('native/native-progress-requests-v2.json').cases) {+ const task = nativeTask(item.task, '/tmp/none.json');+ const taskSources = sources(task, 'appkit');+ const request = nativeChoiceRequest(task, taskSources, task.plan(taskSources), item.history);+ assert.deepEqual(request, item.request, `${item.task} after ${item.history.length} steps`);+ validateRequest(request);+ }+});diff --git a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts b/libs/cua-driver/examples/jev-use/typescript/native_tasks.tsindex 955da3923e..4d57189a5c 100644--- a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts+++ b/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts@@ -9,7 +9,13 @@*/import { readFile } from 'node:fs/promises';-import { MAX_ELEMENTS, REQUEST_SCHEMA_V2 } from './choose_action.js';+import {+ MAX_ELEMENTS,+ MAX_PROGRESS,+ MAX_PROGRESS_COUNT,+ REQUEST_SCHEMA_V2,+ type ProgressItem,+} from './choose_action.js';import {PARAMETER_NAME_PATTERN,elementState,@@ -159,6 +165,33 @@ export function compose(export type CompactElement = { role_class: string; label: string; state: string };+/**+ * One step a task requires, counted from the runner's own performed actions.+ * `candidateId` names the candidate that performs the step; its `:foreground`+ * variant counts too. The description is task-authored and value-free. A step+ * with `afterPrevious` (the default) waits for every earlier step to be done.+ */+export type TaskStep = Readonly<{+ description: string;+ candidateId: string;+ times?: number;+ afterPrevious?: boolean;+}>;++/**+ * Count the actions this run dispatched successfully, by candidate ID. Only+ * entries marked `performed` count; nothing is read from the application.+ */+export function performedCounts(history: readonly HistoryEntry[]): Record<string, number> {+ const counts: Record<string, number> = {};+ for (const item of history) {+ if (item.performed !== true) continue;+ const base = item.selected_id.replace(/:foreground$/, '');+ counts[base] = (counts[base] ?? 0) + 1;+ }+ return counts;+}+export type NativeStep = Readonly<{candidates: Candidate[];stats: ComposeStats;@@ -185,6 +218,7 @@ export type NativeTaskSpec = Readonly<{visualTargets?: readonly string[];visualMinConfidence?: number;mockPreferences?: readonly string[];+ steps?: readonly TaskStep[];}>;export class NativeTask implements Task {@@ -203,6 +237,12 @@ export class NativeTask implements Task {/** OCR confidence a visual target must reach; exact-text uniqueness still applies. */readonly visualMinConfidence: number;readonly mockPreferences: readonly string[];+ /**+ * Ordered steps the task requires (#4313). The request reports how often+ * this run has performed each, and a step's candidate names any earlier+ * step that is not done yet. Empty means the request carries no progress.+ */+ readonly steps: readonly TaskStep[];/** The oracle is polled after every action, so no candidate is special. */readonly completionCandidateIds: ReadonlySet<string> = new Set();@@ -221,6 +261,7 @@ export class NativeTask implements Task {this.visualTargets = spec.visualTargets ?? [];this.visualMinConfidence = spec.visualMinConfidence ?? 0.8;this.mockPreferences = spec.mockPreferences ?? [];+ this.steps = spec.steps ?? [];const unknown = [...this.allowedActions].filter((action) => !ACTION_KINDS.has(action));if (unknown.length) throw new Error(`unknown action kinds: ${unknown.sort().join(', ')}`);for (const parameter of this.parameters) {@@ -229,6 +270,13 @@ export class NativeTask implements Task {}}if (!Number.isInteger(this.maxSteps) || this.maxSteps < 1) throw new Error('maxSteps must be positive');+ if (this.steps.length > MAX_PROGRESS) throw new Error(`a task declares at most ${MAX_PROGRESS} steps`);+ for (const taskStep of this.steps) {+ const times = taskStep.times ?? 1;+ if (!Number.isInteger(times) || times < 1 || times > MAX_PROGRESS_COUNT) {+ throw new Error(`step times must be from 1 to ${MAX_PROGRESS_COUNT}`);+ }+ }}get allowedActionKinds(): ReadonlySet<string> {@@ -378,7 +426,46 @@ export class NativeTask implements Task {: candidateId === 'reobserve'? 'took no action and requested a fresh observation': (options.outcome ?? 'completed');- return { step, selected_id: candidateId, outcome: (this.redact(text) as string).slice(0, 128) };+ const performed =+ !options.stale && refusal === undefined && candidateId !== 'reobserve' && candidateId !== 'abstain';+ return {+ step,+ selected_id: candidateId,+ outcome: (this.redact(text) as string).slice(0, 128),+ ...(performed ? { performed: true as const } : {}),+ };+ }++ /** Each declared step and how often this run has performed it. */+ progress(history: readonly HistoryEntry[]): ProgressItem[] {+ const counts = performedCounts(history);+ return this.steps.map((taskStep) => ({+ step: this.redact(taskStep.description) as string,+ done: Math.min(counts[taskStep.candidateId] ?? 0, MAX_PROGRESS_COUNT),+ required: taskStep.times ?? 1,+ }));+ }++ /**+ * A sentence stating a candidate's place in the task's declared steps: a+ * done step says so; a step with an earlier step not yet done names it; a+ * due step says how many more times the task requires it.+ */+ stepNote(candidateId: string, history: readonly HistoryEntry[]): string {+ const base = candidateId.replace(/:foreground$/, '');+ const counts = performedCounts(history);+ const index = this.steps.findIndex((taskStep) => taskStep.candidateId === base);+ if (index < 0) return '';+ const taskStep = this.steps[index];+ const times = taskStep.times ?? 1;+ if ((counts[base] ?? 0) >= times) {+ return ` This run already did this the ${times} time(s) the task requires.`;+ }+ const pending = (taskStep.afterPrevious === false ? [] : this.steps.slice(0, index))+ .filter((earlier) => (counts[earlier.candidateId] ?? 0) < (earlier.times ?? 1))+ .map((earlier) => earlier.description);+ if (pending.length) return ` The task requires this only after: ${pending.join('; ')} (not done yet).`;+ return ` The task still requires this ${times - (counts[base] ?? 0)} more time(s).`;}async reset(): Promise<void> {@@ -419,7 +506,11 @@ export function visualFallbackReason(return undefined;}-/** Build the cua.jev_choice_request_v2 the provider receives; no tokens, values, or pixels. */+/**+ * Build the cua.jev_choice_request_v2 the provider receives; no tokens,+ * values, or pixels. A task that declares steps adds the progress counted+ * from this run's performed actions.+ */export function nativeChoiceRequest(task: NativeTask,sources: TaskSources,@@ -449,9 +540,10 @@ export function nativeChoiceRequest(history: history.slice(-MAX_HISTORY).map((item) => ({ selected_id: item.selected_id, outcome: item.outcome })),candidates: step.candidates.map((candidate) => ({id: candidate.id,- description: task.redact(candidate.description),+ description: task.redact(candidate.description + task.stepNote(candidate.id, history)),...(candidate.source ? { source: candidate.source } : {}),})),+ ...(task.steps.length ? { progress: task.progress(history) } : {}),};}@@ -588,6 +680,9 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTaallowForeground,maxSteps: COUNTER_TARGET + 3,mockPreferences: ['ax:button:increment'],+ steps: [+ { description: 'Press the button labeled "Increment"', candidateId: 'ax:button:increment', times: COUNTER_TARGET },+ ],});}if (kind === 'save-note') {@@ -604,6 +699,13 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTaallowForeground,maxSteps: 5,mockPreferences: ['ax:text_input:note:set:note', 'ax:button:save-note'],+ steps: [+ {+ description: 'Set the text field "Note" to the task parameter "note"',+ candidateId: 'ax:text_input:note:set:note',+ },+ { description: 'Press the button labeled "Save note"', candidateId: 'ax:button:save-note' },+ ],});}return new NativeTask({@@ -616,6 +718,11 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTaallowForeground,maxSteps: 5,mockPreferences: ['ax:radio:large', 'ax:checkbox:i-agree'],+ steps: [+ { description: 'Select the radio option "Large"', candidateId: 'ax:radio:large' },+ // The oracle accepts either order, so neither step waits for the other.+ { description: 'Toggle the checkbox "I agree"', candidateId: 'ax:checkbox:i-agree', afterPrevious: false },+ ],});}diff --git a/libs/cua-driver/examples/jev-use/typescript/run_native.ts b/libs/cua-driver/examples/jev-use/typescript/run_native.tsindex fabf718f87..63551edb16 100644--- a/libs/cua-driver/examples/jev-use/typescript/run_native.ts+++ b/libs/cua-driver/examples/jev-use/typescript/run_native.ts@@ -28,7 +28,7 @@ import {windowStateArguments,type NativeTask,} from './native_tasks.js';-import { chooseS1Service, s1ServiceUrl } from './s1_service.js';+import { chooseS1Service, S1ServiceError, s1ServiceUrl } from './s1_service.js';import { backgroundRefusalCode, Driver, DriverToolError, supportsCaptureBoundClick } from './run.js';import { NativeAccessibilitySource, VisualRegionSource } from './sources.js';import type { HistoryEntry, Outcome, TaskSources } from './tasks.js';@@ -274,12 +274,24 @@ export async function runTask(args: Arguments, task: NativeTask): Promise<OutcomvalidateRequest(request);const decideStarted = performance.now();- const decision =- args.provider === 'mock'- ? chooseMockForTask(task, sources, plan.candidates, history)- : args.provider === 's1'- ? await chooseS1Service(request)- : await chooseLive(request);+ let decision: { choice: string | null; confidence: number; probabilities: Readonly<Record<string, number>> };+ try {+ decision =+ args.provider === 'mock'+ ? chooseMockForTask(task, sources, plan.candidates, history)+ : args.provider === 's1'+ ? await chooseS1Service(request)+ : await chooseLive(request);+ } catch (error) {+ if (!(error instanceof S1ServiceError)) throw error;+ // A tied or malformed score is not an action; fail closed with a+ // logged outcome instead of a stack trace.+ await writeEvent(args.log, {+ event: 'outcome', outcome: 'unknown', phase: 'decide', step,+ error: 'S1ServiceError', reason: error.message.slice(0, 128),+ });+ return 'unknown';+ }const baseEvent: Record<string, unknown> = {event: 'step',step,diff --git a/libs/cua-driver/examples/jev-use/typescript/tasks.ts b/libs/cua-driver/examples/jev-use/typescript/tasks.tsindex 93fac5a78e..8caec8c26f 100644--- a/libs/cua-driver/examples/jev-use/typescript/tasks.ts+++ b/libs/cua-driver/examples/jev-use/typescript/tasks.ts@@ -76,7 +76,11 @@ export function requirePage(sources: TaskSources): BrowserSemanticSource {return sources.page;}-export type HistoryEntry = Readonly<{ step: number; selected_id: string; outcome: string }>;+/**+ * One runner history item. `performed` marks a dispatched, successful native+ * action; it stays in the runner and is never sent to a provider.+ */+export type HistoryEntry = Readonly<{ step: number; selected_id: string; outcome: string; performed?: true }>;/** The interface the runner uses for every task. */export interface Task {diff --git a/rfcs/4268-jev-use-native-candidates.md b/rfcs/4268-jev-use-native-candidates.mdindex 9cf1b433d7..2a02db9400 100644--- a/rfcs/4268-jev-use-native-candidates.md+++ b/rfcs/4268-jev-use-native-candidates.md@@ -3,7 +3,7 @@ title: Native accessibility candidates for jev-use decision loopsauthors:- f-trycuacreated: 2026-09-27-last_updated: 2026-09-28+last_updated: 2026-09-29status: accepteddiscussion: https://github.com/trycua/cua/issues/4268rfc_pr: https://github.com/trycua/cua/pull/4269@@ -16,6 +16,7 @@ implementation:- https://github.com/trycua/cua/pull/4298- https://github.com/trycua/cua/pull/4301- https://github.com/trycua/cua/pull/4299+ - https://github.com/trycua/cua/pull/4324supersedes:superseded_by:---@@ -562,7 +563,13 @@ to v1:- the root may include `snapshot_id` alongside the required `capture_id`; and- the root may include `elements`, a compact, redacted list of at most 64native elements (`role_class`, `label`, `state`). S1's text adapter renders- it as a genuine accessibility tree.+ it as a genuine accessibility tree; and+- the root may include `progress`, at most 16 task steps+ (`step`, `done`, `required`), added by+ [#4324](https://github.com/trycua/cua/pull/4324). `done` counts only the+ runner's own successful actions and is never read from the application, so+ it carries no field values. `provider_observation` adds it only when it is+ present.The v1 schema, validators, and fixtures remain supported unchanged. A v2request that has no `source`, `snapshot_id`, or `elements` is semantically@@ -846,6 +853,14 @@ Implementation:to Cancel.- Phase 4, [#4299](https://github.com/trycua/cua/pull/4299): the nativeapps guide and the `skills/jev-use` update.+- S1 reliability, [#4324](https://github.com/trycua/cua/pull/4324) for+ [#4313](https://github.com/trycua/cua/issues/4313): each built-in task+ declares its ordered steps. The request's `progress` reports how often this+ run has performed each step, and each step's candidate description names an+ unfinished earlier step (Save waits for the note), says the step is done, or+ says how many more times it is due. Choose-size's two steps are unordered,+ as its oracle is. The runners log an S1 service failure as an outcome, not+ a stack trace.Phase 3 live evidence (released Driver 0.30.2, cua-perception 0.2.1,`cua-s1-4b-0.2@16818868`; each run verified by the harness's own state file):@@ -866,7 +881,26 @@ Phase 3 live evidence (released Driver 0.30.2, cua-perception 0.2.1,Windows.- The tasks produce 3–7 candidates. Jev chose correctly in 220/220 decisionsand S1 in 211/216.--Acceptance criteria not yet met, which keep this RFC `accepted`: S1 5/5 in-every native row, WinUI3 coverage, and accuracy at about 12 and 24-candidates.+- Cause of the S1 misses, reproduced by replaying the recorded AppKit states+ with the failing runs' capture IDs: after two increments S1 split 0.45/0.44+ between abstain and Increment; at the first save-note step it split+ 0.39/0.38 between setting the note and Save; and the service error was that+ same split landing on an exact fp16 logit tie, which `choose()` refuses.++S1 reliability evidence for #4324 (released Driver 0.30.4, same checkpoint):++- Offline replay of the AppKit, WPF, and GTK3 fixtures at eight mid-task+ states, eight capture IDs each, plus the three recorded failing captures:+ S1 was correct in 191/195 decisions on `main` (two abstains, one Save+ before the note, and one tie error) and 195/195 with progress. Its lowest confidence after two increments rose+ from 0.45 to 0.81, and at the first save-note step from 0.38 to 0.79. Jev+ was correct in 195/195 either way.+- Live on macOS in a Lume worker, recorded: AppKit counter, save-note, and+ choose-size passed 5/5 for S1 and 5/5 for Jev in both Python and TypeScript+ (60/60 runs, 140 decisions, all correct). S1's lowest confidence per task+ was 0.83, 0.79, and 0.65. The Windows and Linux rows were verified by the+ offline replay only in this round.++Acceptance criteria not yet met, which keep this RFC `accepted`: a live+rerun of the WPF and GTK3 rows with `progress` to confirm S1 5/5 there,+WinUI3 coverage, and accuracy at about 12 and 24 candidates.diff --git a/skills/jev-use/SKILL.md b/skills/jev-use/SKILL.mdindex f848cff89d..b3ef894f98 100644--- a/skills/jev-use/SKILL.md+++ b/skills/jev-use/SKILL.md@@ -22,7 +22,8 @@ probabilities. Invoke the Python interpreter and absolute chooser path directlywithout a shell.For native desktop applications, use `NativeAccessibilitySource` and`cua.jev_choice_request_v2`, which adds a per-candidate `source` (`page`, `ax`,-or `visual`) and compact value-free `elements`. Browser tasks keep sending v1.+or `visual`), compact value-free `elements`, and optional `progress` counted+from the runner's own performed actions. Browser tasks keep sending v1.Prefer browser DOM and semantic evidence. The optional visual adapter consumesthe public `cua.visual_regions_v1` result only when Driver advertises both`parse_visual_regions` and the capture-bound `click.capture_id` input.