Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): send value-free task progress so Cua-S1 finishes jev-use native tasks (#4324)

Francesco Bonacci907bb88e2efd
diff --git a/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx b/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx
index 3ab56af934..6708c5674e 100644
--- a/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx
+++ b/docs/content/docs/how-to-guides/driver/use-jev-use-with-native-apps.mdx
@@ -56,8 +56,10 @@ try background delivery first. If Driver refuses that, the next step offers a
separate foreground click when the task allows it.
Native requests use the `cua.jev_choice_request_v2` contract, which adds a
-`source` to each candidate (`page`, `ax`, or `visual`). Browser tasks keep
-sending `cua.jev_choice_request_v1`.
+`source` to each candidate (`page`, `ax`, or `visual`). The built-in tasks
+also send `progress`: each required step and how many times this run has
+performed it. The runner counts its own successful actions and never reads the
+count from the app. Browser tasks keep sending `cua.jev_choice_request_v1`.
## Before you start
diff --git a/libs/cua-driver/examples/jev-use/README.md b/libs/cua-driver/examples/jev-use/README.md
index 9729491e33..a11ab958df 100644
--- a/libs/cua-driver/examples/jev-use/README.md
+++ b/libs/cua-driver/examples/jev-use/README.md
@@ -91,7 +91,11 @@ or closing are excluded unless the task allows that risk. Text comes only from
task parameters. Actions are element-bound `click`, `set_value`, or
`type_text` with background delivery first; a stale token leads to a fresh
observation. Native steps send `cua.jev_choice_request_v2`, which adds a
-per-candidate `source` and compact value-free `elements`.
+per-candidate `source` and compact value-free `elements`. A task that declares
+its required steps also sends `progress`: each step with how often this run has
+performed it, counted only from the runner's own successful actions and never
+read from the app. A step's candidate also says which earlier step must be done
+first, for example Save waits for the note to be entered.
Visual regions from OmniParser are parsed, from the same `capture_id`, only
when the tree offers nothing to act through: Driver reports `ax_tree_empty`, a
@@ -170,7 +174,9 @@ The service is not part of this example. Any server that loads the pinned
checkpoint from [`libs/cua-s1`](../../../cua-s1/README.md), validates the
request with `python/choose_action.py`, and answers with
`decision_models.choose(S1DecisionModel(...), request).to_wire()` fits the
-contract. For example, to verify the native tasks with it:
+contract. Run the service from the same revision as the runners: an older
+validator rejects a newer optional field such as `progress`, and the runner
+then stops with a logged `S1ServiceError` outcome. For example, to verify the native tasks with it:
```bash
export CUA_S1_DECISION_URL=http://127.0.0.1:8791/decide
diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/native-progress-requests-v2.json b/libs/cua-driver/examples/jev-use/fixtures/native/native-progress-requests-v2.json
new file mode 100644
index 0000000000..d65649fe98
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/fixtures/native/native-progress-requests-v2.json
@@ -0,0 +1,386 @@
+{
+ "_fixture": "Native v2 requests with task progress (#4313); python/tests/test_native_progress.py and typescript/native_progress.test.ts must both reproduce each request exactly from fixtures/native/appkit-window-state-initial-v1.json and the recorded runner history.",
+ "cases": [
+ {
+ "task": "appkit-save-note",
+ "history": [],
+ "request": {
+ "schema": "cua.jev_choice_request_v2",
+ "goal": "Enter the note text into the Note field, then save the note.",
+ "capture_id": "cap-appkit-initial",
+ "snapshot_id": "s00000086",
+ "regions": [],
+ "elements": [
+ {
+ "role_class": "button",
+ "label": "Increment",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Reset",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Click target (left / right / double)",
+ "state": "enabled"
+ },
+ {
+ "role_class": "checkbox",
+ "label": "I agree",
+ "state": "unchecked"
+ },
+ {
+ "role_class": "button",
+ "label": "Right-click for context menu",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Exit",
+ "state": "enabled"
+ },
+ {
+ "role_class": "text_input",
+ "label": "Note",
+ "state": "empty"
+ },
+ {
+ "role_class": "button",
+ "label": "Save note",
+ "state": "enabled"
+ },
+ {
+ "role_class": "radio",
+ "label": "Small",
+ "state": "not_selected"
+ },
+ {
+ "role_class": "radio",
+ "label": "Medium",
+ "state": "not_selected"
+ },
+ {
+ "role_class": "radio",
+ "label": "Large",
+ "state": "not_selected"
+ }
+ ],
+ "history": [],
+ "candidates": [
+ {
+ "id": "ax:button:increment",
+ "description": "Press the button labeled \"Increment\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:click-target-left-right-double",
+ "description": "Press the button labeled \"Click target (left / right / double)\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:right-click-for-context-menu",
+ "description": "Press the button labeled \"Right-click for context menu\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:text_input:note:set:note",
+ "description": "Set the text field \"Note\" to the task parameter \"note\", replacing its contents. The field currently reports empty. The task still requires this 1 more time(s).",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:save-note",
+ "description": "Press the button labeled \"Save note\". The task requires this only after: Set the text field \"Note\" to the task parameter \"note\" (not done yet).",
+ "source": "ax"
+ },
+ {
+ "id": "reobserve",
+ "description": "Take no action and obtain a fresh observation of the window, because the current observation looks stale, incomplete, or contradicts the goal's progress."
+ },
+ {
+ "id": "abstain",
+ "description": "Stop without acting if none of the proposed actions is safe or moves toward the goal."
+ }
+ ],
+ "progress": [
+ {
+ "step": "Set the text field \"Note\" to the task parameter \"note\"",
+ "done": 0,
+ "required": 1
+ },
+ {
+ "step": "Press the button labeled \"Save note\"",
+ "done": 0,
+ "required": 1
+ }
+ ]
+ }
+ },
+ {
+ "task": "appkit-save-note",
+ "history": [
+ {
+ "step": 1,
+ "selected_id": "ax:text_input:note:set:note",
+ "outcome": "set \"Note\" to parameter note",
+ "performed": true
+ }
+ ],
+ "request": {
+ "schema": "cua.jev_choice_request_v2",
+ "goal": "Enter the note text into the Note field, then save the note.",
+ "capture_id": "cap-appkit-initial",
+ "snapshot_id": "s00000086",
+ "regions": [],
+ "elements": [
+ {
+ "role_class": "button",
+ "label": "Increment",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Reset",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Click target (left / right / double)",
+ "state": "enabled"
+ },
+ {
+ "role_class": "checkbox",
+ "label": "I agree",
+ "state": "unchecked"
+ },
+ {
+ "role_class": "button",
+ "label": "Right-click for context menu",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Exit",
+ "state": "enabled"
+ },
+ {
+ "role_class": "text_input",
+ "label": "Note",
+ "state": "empty"
+ },
+ {
+ "role_class": "button",
+ "label": "Save note",
+ "state": "enabled"
+ },
+ {
+ "role_class": "radio",
+ "label": "Small",
+ "state": "not_selected"
+ },
+ {
+ "role_class": "radio",
+ "label": "Medium",
+ "state": "not_selected"
+ },
+ {
+ "role_class": "radio",
+ "label": "Large",
+ "state": "not_selected"
+ }
+ ],
+ "history": [
+ {
+ "selected_id": "ax:text_input:note:set:note",
+ "outcome": "set \"Note\" to parameter note"
+ }
+ ],
+ "candidates": [
+ {
+ "id": "ax:button:increment",
+ "description": "Press the button labeled \"Increment\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:click-target-left-right-double",
+ "description": "Press the button labeled \"Click target (left / right / double)\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:right-click-for-context-menu",
+ "description": "Press the button labeled \"Right-click for context menu\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:text_input:note:set:note",
+ "description": "Set the text field \"Note\" to the task parameter \"note\", replacing its contents. The field currently reports empty. This run already did this the 1 time(s) the task requires.",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:save-note",
+ "description": "Press the button labeled \"Save note\". The task still requires this 1 more time(s).",
+ "source": "ax"
+ },
+ {
+ "id": "reobserve",
+ "description": "Take no action and obtain a fresh observation of the window, because the current observation looks stale, incomplete, or contradicts the goal's progress."
+ },
+ {
+ "id": "abstain",
+ "description": "Stop without acting if none of the proposed actions is safe or moves toward the goal."
+ }
+ ],
+ "progress": [
+ {
+ "step": "Set the text field \"Note\" to the task parameter \"note\"",
+ "done": 1,
+ "required": 1
+ },
+ {
+ "step": "Press the button labeled \"Save note\"",
+ "done": 0,
+ "required": 1
+ }
+ ]
+ }
+ },
+ {
+ "task": "appkit-counter",
+ "history": [
+ {
+ "step": 1,
+ "selected_id": "ax:button:increment",
+ "outcome": "pressed \"Increment\"",
+ "performed": true
+ },
+ {
+ "step": 2,
+ "selected_id": "ax:button:increment",
+ "outcome": "pressed \"Increment\"",
+ "performed": true
+ },
+ {
+ "step": 3,
+ "selected_id": "ax:button:increment",
+ "outcome": "Driver refused background delivery (background_unsupported); nothing happened"
+ }
+ ],
+ "request": {
+ "schema": "cua.jev_choice_request_v2",
+ "goal": "The counter starts at 0. Set it to exactly 3 by pressing Increment once per step, then stop.",
+ "capture_id": "cap-appkit-initial",
+ "snapshot_id": "s00000086",
+ "regions": [],
+ "elements": [
+ {
+ "role_class": "button",
+ "label": "Increment",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Reset",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Click target (left / right / double)",
+ "state": "enabled"
+ },
+ {
+ "role_class": "checkbox",
+ "label": "I agree",
+ "state": "unchecked"
+ },
+ {
+ "role_class": "button",
+ "label": "Right-click for context menu",
+ "state": "enabled"
+ },
+ {
+ "role_class": "button",
+ "label": "Exit",
+ "state": "enabled"
+ },
+ {
+ "role_class": "text_input",
+ "label": "Note",
+ "state": "empty"
+ },
+ {
+ "role_class": "button",
+ "label": "Save note",
+ "state": "enabled"
+ },
+ {
+ "role_class": "radio",
+ "label": "Small",
+ "state": "not_selected"
+ },
+ {
+ "role_class": "radio",
+ "label": "Medium",
+ "state": "not_selected"
+ },
+ {
+ "role_class": "radio",
+ "label": "Large",
+ "state": "not_selected"
+ }
+ ],
+ "history": [
+ {
+ "selected_id": "ax:button:increment",
+ "outcome": "pressed \"Increment\""
+ },
+ {
+ "selected_id": "ax:button:increment",
+ "outcome": "pressed \"Increment\""
+ },
+ {
+ "selected_id": "ax:button:increment",
+ "outcome": "Driver refused background delivery (background_unsupported); nothing happened"
+ }
+ ],
+ "candidates": [
+ {
+ "id": "ax:button:increment",
+ "description": "Press the button labeled \"Increment\". The task still requires this 1 more time(s).",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:click-target-left-right-double",
+ "description": "Press the button labeled \"Click target (left / right / double)\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:right-click-for-context-menu",
+ "description": "Press the button labeled \"Right-click for context menu\".",
+ "source": "ax"
+ },
+ {
+ "id": "ax:button:save-note",
+ "description": "Press the button labeled \"Save note\".",
+ "source": "ax"
+ },
+ {
+ "id": "reobserve",
+ "description": "Take no action and obtain a fresh observation of the window, because the current observation looks stale, incomplete, or contradicts the goal's progress."
+ },
+ {
+ "id": "abstain",
+ "description": "Stop without acting if none of the proposed actions is safe or moves toward the goal."
+ }
+ ],
+ "progress": [
+ {
+ "step": "Press the button labeled \"Increment\"",
+ "done": 2,
+ "required": 3
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/libs/cua-driver/examples/jev-use/python/choose_action.py b/libs/cua-driver/examples/jev-use/python/choose_action.py
index e558cbdfcf..df2cde5e3f 100644
--- a/libs/cua-driver/examples/jev-use/python/choose_action.py
+++ b/libs/cua-driver/examples/jev-use/python/choose_action.py
@@ -11,7 +11,9 @@ from native_roles import ROLE_CLASSES
REQUEST_SCHEMA = "cua.jev_choice_request_v1"
# Additive over v1 (RFC #4268): per-candidate ``source``, an optional root
-# ``snapshot_id``, and an optional compact ``elements`` list of native controls.
+# ``snapshot_id``, an optional compact ``elements`` list of native controls, and
+# an optional ``progress`` list of task steps counted from the runner's own
+# performed actions (#4313).
REQUEST_SCHEMA_V2 = "cua.jev_choice_request_v2"
REQUEST_SCHEMAS = (REQUEST_SCHEMA, REQUEST_SCHEMA_V2)
RESPONSE_SCHEMA = "cua.jev_choice_v1"
@@ -20,6 +22,8 @@ MAX_CANDIDATES = 32
MAX_REGIONS = 100
MAX_HISTORY = 16
MAX_ELEMENTS = 64
+MAX_PROGRESS = 16
+MAX_PROGRESS_COUNT = 64
ID_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,63}\Z")
RESERVED_IDS = frozenset({"reobserve", "abstain"})
CANDIDATE_SOURCES = frozenset({"page", "ax", "visual"})
@@ -27,7 +31,7 @@ ELEMENT_STATES = frozenset(
{"enabled", "checked", "unchecked", "selected", "not_selected", "empty", "has_text"}
)
V1_ROOT_KEYS = frozenset({"schema", "goal", "capture_id", "regions", "history", "candidates"})
-V2_OPTIONAL_ROOT_KEYS = frozenset({"snapshot_id", "elements"})
+V2_OPTIONAL_ROOT_KEYS = frozenset({"snapshot_id", "elements", "progress"})
def _record(value: Any, message: str) -> dict[str, Any]:
@@ -71,13 +75,46 @@ def _validate_elements(value: Any) -> list[dict[str, str]]:
return elements
+def _count(value: Any, name: str, minimum: int) -> int:
+ if (
+ not isinstance(value, int)
+ or isinstance(value, bool)
+ or not minimum <= value <= MAX_PROGRESS_COUNT
+ ):
+ raise ValueError(f"{name} must be an integer from {minimum} to {MAX_PROGRESS_COUNT}")
+ return value
+
+
+def _validate_progress(value: Any) -> list[dict[str, Any]]:
+ """Validate task steps and how often this run has performed each one.
+
+ ``done`` is counted from the runner's own performed actions, never read
+ from the application, so ``progress`` carries no application values.
+ """
+ if not isinstance(value, list) or len(value) > MAX_PROGRESS:
+ raise ValueError(f"progress must be an array of at most {MAX_PROGRESS} items")
+ progress: list[dict[str, Any]] = []
+ for raw_value in value:
+ raw = _record(raw_value, "progress item must be an object")
+ if set(raw) != {"step", "done", "required"}:
+ raise ValueError("progress item may contain only step, done, and required")
+ progress.append(
+ {
+ "step": _bounded_string(raw["step"], "progress step", 200),
+ "done": _count(raw["done"], "progress done", 0),
+ "required": _count(raw["required"], "progress required", 1),
+ }
+ )
+ return progress
+
+
def validate_request(value: Any) -> dict[str, Any]:
"""Validate a ``cua.jev_choice_request_v1`` or ``_v2`` request strictly.
v1 is unchanged: exactly its six root keys and ``id``/``description``
candidates, so any v2 field in a v1 request is rejected. v2 may add a root
- ``snapshot_id`` and ``elements`` and a per-candidate ``source``; reserved
- candidates never carry a source.
+ ``snapshot_id``, ``elements``, and ``progress`` and a per-candidate
+ ``source``; reserved candidates never carry a source.
"""
root = _record(value, "request must be a JSON object")
schema = root.get("schema")
@@ -217,6 +254,7 @@ def validate_request(value: Any) -> dict[str, Any]:
None if snapshot_id is None else _bounded_string(snapshot_id, "snapshot_id", 64)
)
validated["elements"] = _validate_elements(root.get("elements", []))
+ validated["progress"] = _validate_progress(root.get("progress", []))
return validated
@@ -224,8 +262,9 @@ def provider_observation(validated: Mapping[str, Any]) -> dict[str, Any]:
"""Build the observation a provider sees for a validated request.
A v1 request keeps its exact v1 observation. A v2 request adds the native
- ``snapshot_id``, the compact ``elements``, and each candidate's ``source``;
- none of these contain element tokens, values, or pixels.
+ ``snapshot_id``, the compact ``elements``, and each candidate's ``source``,
+ plus ``progress`` when the request carries any; none of these contain
+ element tokens, values, or pixels.
"""
observation: dict[str, Any] = {
"capture_id": validated["capture_id"],
@@ -238,6 +277,8 @@ def provider_observation(validated: Mapping[str, Any]) -> dict[str, Any]:
observation["candidate_sources"] = {
item["id"]: item["source"] for item in validated["candidates"] if "source" in item
}
+ if validated.get("progress"):
+ observation["progress"] = validated["progress"]
return observation
diff --git a/libs/cua-driver/examples/jev-use/python/decision_models.py b/libs/cua-driver/examples/jev-use/python/decision_models.py
index d59069dbaf..8691832477 100644
--- a/libs/cua-driver/examples/jev-use/python/decision_models.py
+++ b/libs/cua-driver/examples/jev-use/python/decision_models.py
@@ -27,10 +27,12 @@ class DecisionRequest:
regions: tuple[Mapping[str, Any], ...]
history: tuple[Mapping[str, str], ...]
candidates: tuple[Mapping[str, str], ...]
- # v2 only (RFC #4268): native snapshot and compact accessibility elements.
+ # v2 only (RFC #4268): native snapshot and compact accessibility elements,
+ # and task steps counted from the runner's own performed actions (#4313).
schema: str = REQUEST_SCHEMA
snapshot_id: str | None = None
elements: tuple[Mapping[str, str], ...] = ()
+ progress: tuple[Mapping[str, Any], ...] = ()
@classmethod
def from_validated(cls, request: Mapping[str, Any]) -> "DecisionRequest":
@@ -43,6 +45,7 @@ class DecisionRequest:
schema=request.get("schema", REQUEST_SCHEMA),
snapshot_id=request.get("snapshot_id"),
elements=tuple(MappingProxyType(dict(item)) for item in request.get("elements", ())),
+ progress=tuple(MappingProxyType(dict(item)) for item in request.get("progress", ())),
)
@property
@@ -61,6 +64,7 @@ class DecisionRequest:
if self.schema == REQUEST_SCHEMA_V2:
validated["snapshot_id"] = self.snapshot_id
validated["elements"] = [dict(item) for item in self.elements]
+ validated["progress"] = [dict(item) for item in self.progress]
return provider_observation(validated)
@@ -219,7 +223,8 @@ def native_elements_as_text(request: DecisionRequest) -> str:
"""Render a v2 request's native elements as an accessibility tree for S1.
Each line is one supplied native control: its role class, label, and state.
- Visual regions from the same capture, if any, follow as a separate section.
+ Visual regions from the same capture, if any, follow as a separate section,
+ then the prior decisions and, when supplied, the task progress.
"""
lines = [
f"Accessibility tree for snapshot {json.dumps(request.snapshot_id)} "
@@ -237,6 +242,20 @@ def native_elements_as_text(request: DecisionRequest) -> str:
if request.history:
lines.append("Prior bounded decisions:")
lines.extend(json.dumps(item, ensure_ascii=False) for item in request.history)
+ if request.progress:
+ lines.append(progress_as_text(request.progress))
+ return "\n".join(lines)
+
+
+def progress_as_text(progress: tuple[Mapping[str, Any], ...]) -> str:
+ """Render task progress; counts come from the run's actions, not the app."""
+ lines = [
+ "Task progress (counted from this run's own completed actions, not read from the app):"
+ ]
+ for item in progress:
+ done, required = item["done"], item["required"]
+ status = f"{required - done} remaining" if done < required else "complete"
+ lines.append(f"- {item['step']}: done {done} of {required} times; {status}")
return "\n".join(lines)
diff --git a/libs/cua-driver/examples/jev-use/python/native_tasks.py b/libs/cua-driver/examples/jev-use/python/native_tasks.py
index 2a45dabaab..bcf22d9822 100644
--- a/libs/cua-driver/examples/jev-use/python/native_tasks.py
+++ b/libs/cua-driver/examples/jev-use/python/native_tasks.py
@@ -29,7 +29,13 @@ from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Any, Callable, Literal, Mapping
-from choose_action import MAX_ELEMENTS, MAX_HISTORY, REQUEST_SCHEMA_V2
+from choose_action import (
+ MAX_ELEMENTS,
+ MAX_HISTORY,
+ MAX_PROGRESS,
+ MAX_PROGRESS_COUNT,
+ REQUEST_SCHEMA_V2,
+)
from native import (
PARAMETER_NAME_PATTERN,
NativeControl,
@@ -180,6 +186,36 @@ def compose(
return kept + list(reserved), stats
+@dataclass(frozen=True)
+class TaskStep:
+ """One step a task requires, counted from the runner's own performed actions.
+
+ ``candidate_id`` names the candidate that performs the step; its
+ ``:foreground`` variant counts too. The description is task-authored and
+ value-free: it may name a parameter, never its value. A step with
+ ``after_previous`` must wait for every earlier step to be done.
+ """
+
+ description: str
+ candidate_id: str
+ times: int = 1
+ after_previous: bool = True
+
+
+def performed_counts(history: list[Mapping[str, Any]]) -> dict[str, int]:
+ """Count the actions this run dispatched successfully, by candidate ID.
+
+ Only history entries marked ``performed`` count: a refused, stale, or
+ reobserve step did nothing. Nothing here is read from the application.
+ """
+ counts: dict[str, int] = {}
+ for item in history:
+ if item.get("performed") is True:
+ base = str(item["selected_id"]).removesuffix(":foreground")
+ counts[base] = counts.get(base, 0) + 1
+ return counts
+
+
@dataclass(frozen=True)
class NativeStep:
"""One step's closed candidate set and its model-safe context."""
@@ -214,6 +250,11 @@ class NativeTask:
# browser path. Exact-text uniqueness still applies at any bar.
visual_min_confidence: float = 0.8
mock_preferences: tuple[str, ...] = ()
+ # Ordered steps the task requires (#4313). The request reports how often
+ # this run has performed each, and a step's candidate names any earlier
+ # step that is not done yet, so a model need not infer order or count
+ # from history. Empty means the request carries no progress.
+ steps: tuple[TaskStep, ...] = ()
# The oracle is polled after every action, so no candidate is special.
completion_candidate_ids: frozenset[str] = field(default=frozenset(), init=False)
@@ -226,6 +267,11 @@ class NativeTask:
raise ValueError("parameter names must match [a-z][a-z0-9_]{0,7}")
if self.max_steps < 1:
raise ValueError("max_steps must be positive")
+ if len(self.steps) > MAX_PROGRESS:
+ raise ValueError(f"a task declares at most {MAX_PROGRESS} steps")
+ for task_step in self.steps:
+ if not 1 <= task_step.times <= MAX_PROGRESS_COUNT:
+ raise ValueError(f"step times must be from 1 to {MAX_PROGRESS_COUNT}")
@property
def allowed_action_kinds(self) -> frozenset[str]:
@@ -403,7 +449,59 @@ class NativeTask:
text = "took no action and requested a fresh observation"
else:
text = outcome or "completed"
- return {"step": step, "selected_id": candidate_id, "outcome": self.redact(text)[:128]}
+ entry: dict[str, Any] = {
+ "step": step,
+ "selected_id": candidate_id,
+ "outcome": self.redact(text)[:128],
+ }
+ if not stale and refusal is None and candidate_id not in {"reobserve", "abstain"}:
+ entry["performed"] = True # runner-side only; never sent to a provider
+ return entry
+
+ # -- progress ---------------------------------------------------------
+
+ def progress(self, history: list[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ """Each declared step and how often this run has performed it."""
+ counts = performed_counts(history)
+ return [
+ {
+ "step": self.redact(task_step.description),
+ "done": min(counts.get(task_step.candidate_id, 0), MAX_PROGRESS_COUNT),
+ "required": task_step.times,
+ }
+ for task_step in self.steps
+ ]
+
+ def step_note(self, candidate_id: str, history: list[Mapping[str, Any]]) -> str:
+ """A sentence stating a candidate's place in the task's declared steps.
+
+ A step that is already done says so; a step with an earlier step not yet
+ done names that step as its precondition; a step that is due says how
+ many more times the task requires it. Other candidates get nothing.
+ """
+ base = candidate_id.removesuffix(":foreground")
+ counts = performed_counts(history)
+ for index, task_step in enumerate(self.steps):
+ if task_step.candidate_id != base:
+ continue
+ if counts.get(base, 0) >= task_step.times:
+ return (
+ f" This run already did this the {task_step.times} time(s) the task requires."
+ )
+ pending = [
+ earlier.description
+ for earlier in (self.steps[:index] if task_step.after_previous else ())
+ if counts.get(earlier.candidate_id, 0) < earlier.times
+ ]
+ if pending:
+ return (
+ " The task requires this only after: "
+ + "; ".join(pending)
+ + " (not done yet)."
+ )
+ remaining = task_step.times - counts.get(base, 0)
+ return f" The task still requires this {remaining} more time(s)."
+ return ""
def reset(self) -> None:
"""The harness starts fresh for every run; there is nothing to reset."""
@@ -488,19 +586,21 @@ def native_choice_request(
"""Build the ``cua.jev_choice_request_v2`` the provider receives.
It carries candidate IDs, descriptions and sources, compact value-free
- elements, and compact history. Element tokens, values, and pixels stay in
- the runner.
+ elements, compact history, and, for a task that declares steps, the
+ progress counted from this run's performed actions. Element tokens,
+ values, and pixels stay in the runner.
"""
if sources.ax is None or sources.ax.observation.capture_id is None:
raise ValueError("a native request needs an observation with a capture_id")
observation = sources.ax.observation
candidates = []
for candidate in step.candidates:
- item: dict[str, Any] = {"id": candidate.id, "description": task.redact(candidate.description)}
+ description = candidate.description + task.step_note(candidate.id, history)
+ item: dict[str, Any] = {"id": candidate.id, "description": task.redact(description)}
if candidate.source is not None:
item["source"] = candidate.source
candidates.append(item)
- return {
+ request: dict[str, Any] = {
"schema": REQUEST_SCHEMA_V2,
"goal": task.redact(task.goal),
"capture_id": observation.capture_id,
@@ -513,6 +613,9 @@ def native_choice_request(
],
"candidates": candidates,
}
+ if task.steps:
+ request["progress"] = task.progress(history)
+ return request
# -- Harness tasks ------------------------------------------------------------
@@ -686,6 +789,9 @@ def native_task(
allow_foreground=allow_foreground,
max_steps=COUNTER_TARGET + 3,
mock_preferences=("ax:button:increment",),
+ steps=(
+ TaskStep('Press the button labeled "Increment"', "ax:button:increment", COUNTER_TARGET),
+ ),
)
if kind == "save-note":
return NativeTask(
@@ -699,6 +805,13 @@ def native_task(
allow_foreground=allow_foreground,
max_steps=5,
mock_preferences=("ax:text_input:note:set:note", "ax:button:save-note"),
+ steps=(
+ TaskStep(
+ 'Set the text field "Note" to the task parameter "note"',
+ "ax:text_input:note:set:note",
+ ),
+ TaskStep('Press the button labeled "Save note"', "ax:button:save-note"),
+ ),
)
return NativeTask(
id=task_id,
@@ -710,6 +823,11 @@ def native_task(
allow_foreground=allow_foreground,
max_steps=5,
mock_preferences=("ax:radio:large", "ax:checkbox:i-agree"),
+ steps=(
+ TaskStep('Select the radio option "Large"', "ax:radio:large"),
+ # The oracle accepts either order, so neither step waits for the other.
+ TaskStep('Toggle the checkbox "I agree"', "ax:checkbox:i-agree", after_previous=False),
+ ),
)
diff --git a/libs/cua-driver/examples/jev-use/python/run_native.py b/libs/cua-driver/examples/jev-use/python/run_native.py
index 215d0e8a59..11c772dca2 100644
--- a/libs/cua-driver/examples/jev-use/python/run_native.py
+++ b/libs/cua-driver/examples/jev-use/python/run_native.py
@@ -43,7 +43,7 @@ from native_tasks import (
native_choice_request,
visual_fallback_reason,
)
-from s1_service import choose_s1_service, s1_service_url
+from s1_service import S1ServiceError, choose_s1_service, s1_service_url
from run import Driver, DriverToolError, background_refusal_code, supports_capture_bound_click
from sources import NativeAccessibilitySource, VisualRegionSource
from tasks import TaskSources
@@ -284,7 +284,17 @@ async def run_task(args: argparse.Namespace, task: NativeTask) -> str:
task, sources, plan.candidates, history
)
elif args.provider == "s1":
- choice, confidence, probabilities = await asyncio.to_thread(choose_s1_service, request)
+ try:
+ choice, confidence, probabilities = await asyncio.to_thread(
+ choose_s1_service, request
+ )
+ except S1ServiceError as error:
+ # A tied or malformed score is not an action; fail closed
+ # with a logged outcome instead of a traceback.
+ write_event(log_path, {"event": "outcome", "outcome": "unknown",
+ "phase": "decide", "step": step,
+ "error": "S1ServiceError", "reason": str(error)[:128]})
+ return "unknown"
else:
choice, confidence, probabilities = await asyncio.to_thread(choose_live, request)
decide_ms = round((time.perf_counter() - decide_started) * 1000, 2)
diff --git a/libs/cua-driver/examples/jev-use/python/tests/test_native_progress.py b/libs/cua-driver/examples/jev-use/python/tests/test_native_progress.py
new file mode 100644
index 0000000000..ebdaf7ad76
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/python/tests/test_native_progress.py
@@ -0,0 +1,231 @@
+"""Task progress in cua.jev_choice_request_v2 (#4313).
+
+A native task declares the steps it requires. The request reports how often
+this run has performed each step, counted only from the runner's own
+successful actions, and a step's candidate names any earlier step that is not
+done yet. Nothing in ``progress`` is read from the application.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from choose_action import provider_observation, validate_request
+from decision_models import DecisionRequest, native_elements_as_text, progress_as_text
+from native import NativeObservation
+from native_tasks import (
+ CANVAS_TASK_ID,
+ HARNESSES,
+ TaskStep,
+ native_choice_request,
+ native_task,
+ performed_counts,
+)
+from sources import NativeAccessibilitySource
+from tasks import TaskSources
+
+ROOT = Path(__file__).resolve().parents[2]
+FIXTURES = ROOT / "fixtures/native"
+
+
+def v2_request() -> dict:
+ return json.loads((ROOT / "fixtures/jev-choice-request-v2.json").read_text(encoding="utf-8"))
+
+
+def v1_request() -> dict:
+ return json.loads((ROOT / "fixtures/jev-choice-request-v1.json").read_text(encoding="utf-8"))
+
+
+def sources(task, harness: str, name: str = "initial") -> TaskSources:
+ payload = json.loads((FIXTURES / f"{harness}-window-state-{name}-v1.json").read_text("utf-8"))
+ observation = NativeObservation.from_window_state(
+ payload, expected_pid=payload["pid"], expected_window_id=payload["window_id"]
+ )
+ ax = NativeAccessibilitySource.from_observation(
+ observation, HARNESSES[harness].platform, redact=task.redact_text, text_method=task.text_method
+ )
+ return TaskSources(ax=ax)
+
+
+def request_for(task_id: str, history: list[dict], note_text: str = "jev-use native note") -> dict:
+ task = native_task(task_id, Path("/tmp/none.json"), note_text=note_text)
+ task_sources = sources(task, task_id.split("-")[0])
+ return native_choice_request(task, task_sources, task.plan(task_sources), history)
+
+
+def performed(task, step: int, candidate_id: str) -> dict:
+ return task.history_entry(step, candidate_id, outcome="done")
+
+
+class ProgressContractTest(unittest.TestCase):
+ def test_v2_accepts_progress_and_the_observation_carries_it(self) -> None:
+ request = v2_request()
+ request["progress"] = [{"step": 'Press "Increment"', "done": 2, "required": 3}]
+ validated = validate_request(request)
+ self.assertEqual(validated["progress"], request["progress"])
+ self.assertEqual(provider_observation(validated)["progress"], request["progress"])
+
+ def test_v2_without_progress_keeps_its_observation(self) -> None:
+ validated = validate_request(v2_request())
+ self.assertEqual(validated["progress"], [])
+ self.assertNotIn("progress", provider_observation(validated))
+
+ def test_v1_rejects_progress(self) -> None:
+ request = v1_request()
+ request["progress"] = []
+ with self.assertRaises(ValueError):
+ validate_request(request)
+
+ def test_v2_rejects_malformed_progress(self) -> None:
+ item = {"step": "Press", "done": 0, "required": 1}
+ mutations = {
+ "not a list": {"step": "Press"},
+ "too many": [item] * 17,
+ "extra key": [{**item, "value": "secret"}],
+ "missing key": [{"step": "Press", "done": 0}],
+ "empty step": [{**item, "step": " "}],
+ "long step": [{**item, "step": "x" * 201}],
+ "negative done": [{**item, "done": -1}],
+ "boolean done": [{**item, "done": True}],
+ "float done": [{**item, "done": 1.0}],
+ "zero required": [{**item, "required": 0}],
+ "huge required": [{**item, "required": 65}],
+ }
+ for name, progress in mutations.items():
+ request = copy.deepcopy(v2_request())
+ request["progress"] = progress
+ with self.subTest(name=name), self.assertRaises(ValueError):
+ validate_request(request)
+
+ def test_s1_text_renders_progress_after_history(self) -> None:
+ request = v2_request()
+ request["progress"] = [
+ {"step": 'Press "Increment"', "done": 2, "required": 3},
+ {"step": 'Press "Save note"', "done": 1, "required": 1},
+ ]
+ text = native_elements_as_text(DecisionRequest.from_validated(validate_request(request)))
+ self.assertIn('- Press "Increment": done 2 of 3 times; 1 remaining', text)
+ self.assertIn('- Press "Save note": done 1 of 1 times; complete', text)
+ self.assertLess(text.index("Prior bounded decisions:"), text.index("Task progress"))
+
+ def test_s1_text_without_progress_is_unchanged(self) -> None:
+ request = DecisionRequest.from_validated(validate_request(v2_request()))
+ self.assertNotIn("Task progress", native_elements_as_text(request))
+ self.assertTrue(progress_as_text(()).startswith("Task progress"))
+
+ def test_decision_request_observation_matches_the_validated_one(self) -> None:
+ raw = v2_request()
+ raw["progress"] = [{"step": "Press", "done": 0, "required": 1}]
+ validated = validate_request(raw)
+ request = DecisionRequest.from_validated(validated)
+ self.assertEqual(request.provider_observation(), provider_observation(validated))
+
+
+class TaskProgressTest(unittest.TestCase):
+ def test_only_performed_actions_count(self) -> None:
+ task = native_task("appkit-counter", Path("/tmp/none.json"))
+ history = [
+ performed(task, 1, "ax:button:increment"),
+ task.history_entry(2, "ax:button:increment", refusal="background_unsupported"),
+ task.history_entry(3, "ax:button:increment", stale=True),
+ task.history_entry(4, "reobserve"),
+ performed(task, 5, "ax:button:increment:foreground"),
+ ]
+ self.assertEqual(performed_counts(history), {"ax:button:increment": 2})
+ self.assertEqual(
+ task.progress(history),
+ [{"step": 'Press the button labeled "Increment"', "done": 2, "required": 3}],
+ )
+
+ def test_counter_request_reports_increments_done(self) -> None:
+ task = native_task("appkit-counter", Path("/tmp/none.json"))
+ history = [performed(task, 1, "ax:button:increment"), performed(task, 2, "ax:button:increment")]
+ request = request_for("appkit-counter", history)
+ validate_request(request)
+ self.assertEqual(request["progress"][0]["done"], 2)
+ increment = {c["id"]: c for c in request["candidates"]}["ax:button:increment"]
+ self.assertTrue(increment["description"].endswith("still requires this 1 more time(s)."))
+ self.assertEqual(request["progress"][0]["required"], 3)
+ # The runner-side marker never reaches the provider.
+ self.assertEqual(set(request["history"][0]), {"selected_id", "outcome"})
+
+ def test_save_is_described_as_waiting_for_the_note(self) -> None:
+ for harness in HARNESSES:
+ with self.subTest(harness=harness):
+ request = request_for(f"{harness}-save-note", [], note_text="secret note")
+ save = {c["id"]: c for c in request["candidates"]}["ax:button:save-note"]
+ self.assertIn(
+ 'The task requires this only after: Set the text field "Note" to the task '
+ 'parameter "note" (not done yet).',
+ save["description"],
+ )
+ self.assertEqual([item["done"] for item in request["progress"]], [0, 0])
+ self.assertNotIn("secret note", json.dumps(request))
+
+ def test_save_precondition_clears_once_the_note_is_set(self) -> None:
+ task = native_task("appkit-save-note", Path("/tmp/none.json"))
+ request = request_for("appkit-save-note", [performed(task, 1, "ax:text_input:note:set:note")])
+ save = {c["id"]: c for c in request["candidates"]}["ax:button:save-note"]
+ self.assertEqual(
+ save["description"],
+ 'Press the button labeled "Save note". The task still requires this 1 more time(s).',
+ )
+ self.assertEqual([item["done"] for item in request["progress"]], [1, 0])
+
+ def test_a_completed_step_says_so(self) -> None:
+ task = native_task("appkit-counter", Path("/tmp/none.json"))
+ history = [performed(task, step, "ax:button:increment") for step in (1, 2, 3)]
+ request = request_for("appkit-counter", history)
+ increment = {c["id"]: c for c in request["candidates"]}["ax:button:increment"]
+ self.assertTrue(
+ increment["description"].endswith("This run already did this the 3 time(s) the task requires.")
+ )
+
+ def test_unordered_steps_have_no_precondition(self) -> None:
+ request = request_for("appkit-choose-size", [])
+ agree = {c["id"]: c for c in request["candidates"]}["ax:checkbox:i-agree"]
+ self.assertNotIn("only after", agree["description"])
+ self.assertTrue(agree["description"].endswith("still requires this 1 more time(s)."))
+
+ def test_a_task_without_steps_sends_no_progress(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ self.assertEqual(task.steps, ())
+ self.assertEqual(task.progress([]), [])
+
+ def test_task_step_bounds(self) -> None:
+ task = native_task("appkit-counter", Path("/tmp/none.json"))
+ from dataclasses import replace
+
+ with self.assertRaises(ValueError):
+ replace(task, steps=(TaskStep("Press", "ax:button:increment", 0),))
+ with self.assertRaises(ValueError):
+ replace(task, steps=tuple(TaskStep("Press", f"id{i}") for i in range(17)))
+
+ def test_recorded_requests_are_reproduced_exactly(self) -> None:
+ golden = json.loads((FIXTURES / "native-progress-requests-v2.json").read_text("utf-8"))
+ for case in golden["cases"]:
+ with self.subTest(task=case["task"], steps=len(case["history"])):
+ task = native_task(case["task"], Path("/tmp/none.json"))
+ task_sources = sources(task, "appkit")
+ request = native_choice_request(
+ task, task_sources, task.plan(task_sources), case["history"]
+ )
+ self.assertEqual(request, case["request"])
+ validate_request(request)
+
+ def test_same_progress_on_every_harness(self) -> None:
+ requests = {
+ harness: request_for(f"{harness}-save-note", []) for harness in HARNESSES
+ }
+ progress = {json.dumps(r["progress"]) for r in requests.values()}
+ self.assertEqual(len(progress), 1)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/libs/cua-driver/examples/jev-use/typescript/choose_action.ts b/libs/cua-driver/examples/jev-use/typescript/choose_action.ts
index e193c783b8..bc352e0341 100644
--- a/libs/cua-driver/examples/jev-use/typescript/choose_action.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/choose_action.ts
@@ -8,7 +8,9 @@ import { ROLE_CLASSES, type RoleClass } from './native_roles.js';
export const REQUEST_SCHEMA = 'cua.jev_choice_request_v1';
// Additive over v1 (RFC #4268): per-candidate `source`, an optional root
-// `snapshot_id`, and an optional compact `elements` list of native controls.
+// `snapshot_id`, an optional compact `elements` list of native controls, and
+// an optional `progress` list of task steps counted from the runner's own
+// performed actions (#4313).
export const REQUEST_SCHEMA_V2 = 'cua.jev_choice_request_v2';
export const RESPONSE_SCHEMA = 'cua.jev_choice_v1';
const MAX_INPUT_BYTES = 65_536;
@@ -16,6 +18,8 @@ const MAX_CANDIDATES = 32;
const MAX_REGIONS = 100;
const MAX_HISTORY = 16;
export const MAX_ELEMENTS = 64;
+export const MAX_PROGRESS = 16;
+export const MAX_PROGRESS_COUNT = 64;
const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,63}$/;
const RESERVED_IDS = new Set(['reobserve', 'abstain']);
export type CandidateSourceKind = 'page' | 'ax' | 'visual';
@@ -38,10 +42,11 @@ const ELEMENT_STATES = new Set<string>([
'has_text',
]);
const V1_ROOT_KEYS = ['candidates', 'capture_id', 'goal', 'history', 'regions', 'schema'];
-const V2_OPTIONAL_ROOT_KEYS = ['elements', 'snapshot_id'];
+const V2_OPTIONAL_ROOT_KEYS = ['elements', 'progress', 'snapshot_id'];
type JsonRecord = Record<string, unknown>;
export type CompactElement = { role_class: RoleClass; label: string; state: ElementState };
+export type ProgressItem = { step: string; done: number; required: number };
export type ValidatedCandidate = { id: string; description: string; source?: CandidateSourceKind };
export type ValidatedRequest = {
schema: typeof REQUEST_SCHEMA | typeof REQUEST_SCHEMA_V2;
@@ -52,6 +57,7 @@ export type ValidatedRequest = {
candidates: ValidatedCandidate[];
snapshot_id?: string | null;
elements?: CompactElement[];
+ progress?: ProgressItem[];
};
function record(value: unknown, message: string): JsonRecord {
@@ -104,11 +110,40 @@ function validateElements(value: unknown): CompactElement[] {
});
}
+function count(value: unknown, name: string, minimum: number): number {
+ if (!Number.isInteger(value) || Number(value) < minimum || Number(value) > MAX_PROGRESS_COUNT) {
+ throw new Error(`${name} must be an integer from ${minimum} to ${MAX_PROGRESS_COUNT}`);
+ }
+ return Number(value);
+}
+
+/**
+ * Validate task steps and how often this run has performed each one. `done`
+ * is counted from the runner's own performed actions, never read from the
+ * application, so progress carries no application values.
+ */
+function validateProgress(value: unknown): ProgressItem[] {
+ if (!Array.isArray(value) || value.length > MAX_PROGRESS) {
+ throw new Error(`progress must be an array of at most ${MAX_PROGRESS} items`);
+ }
+ return value.map((item) => {
+ const raw = record(item, 'progress item must be an object');
+ if (!exactKeys(raw, ['done', 'required', 'step'])) {
+ throw new Error('progress item may contain only step, done, and required');
+ }
+ return {
+ step: boundedString(raw.step, 'progress step', 200),
+ done: count(raw.done, 'progress done', 0),
+ required: count(raw.required, 'progress required', 1),
+ };
+ });
+}
+
/**
* Validate a cua.jev_choice_request_v1 or _v2 request strictly. v1 is
* unchanged, so any v2 field in a v1 request is rejected. v2 may add a root
- * snapshot_id and elements and a per-candidate source; reserved candidates
- * never carry a source.
+ * snapshot_id, elements, and progress and a per-candidate source; reserved
+ * candidates never carry a source.
*/
export function validateRequest(value: unknown): ValidatedRequest {
const root = record(value, 'request must be a JSON object');
@@ -269,13 +304,15 @@ export function validateRequest(value: unknown): ValidatedRequest {
? null
: boundedString(root.snapshot_id, 'snapshot_id', 64);
validated.elements = validateElements(root.elements ?? []);
+ validated.progress = validateProgress(root.progress ?? []);
}
return validated;
}
/**
* The observation a provider sees. A v1 request keeps its exact v1
- * observation; v2 adds the snapshot, compact elements, and candidate sources.
+ * observation; v2 adds the snapshot, compact elements, and candidate sources,
+ * plus progress when the request carries any.
*/
export function providerObservation(validated: ValidatedRequest): JsonRecord {
const observation: JsonRecord = {
@@ -289,6 +326,7 @@ export function providerObservation(validated: ValidatedRequest): JsonRecord {
observation.candidate_sources = Object.fromEntries(
validated.candidates.filter((item) => item.source).map((item) => [item.id, item.source])
);
+ if (validated.progress?.length) observation.progress = validated.progress;
}
return observation;
}
diff --git a/libs/cua-driver/examples/jev-use/typescript/native_progress.test.ts b/libs/cua-driver/examples/jev-use/typescript/native_progress.test.ts
new file mode 100644
index 0000000000..8f5aaf71f5
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/typescript/native_progress.test.ts
@@ -0,0 +1,146 @@
+// Task progress in cua.jev_choice_request_v2 (#4313). Mirrors
+// python/tests/test_native_progress.py: a native task declares the steps it
+// requires, the request reports how often this run has performed each step
+// (counted only from the runner's own successful actions), and a step's
+// candidate names any earlier step that is not done yet.
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import test from 'node:test';
+
+import { providerObservation, validateRequest } from './choose_action.js';
+import { parseWindowState } from './native.js';
+import {
+ CANVAS_TASK_ID,
+ HARNESSES,
+ NativeTask,
+ nativeChoiceRequest,
+ nativeTask,
+ performedCounts,
+} from './native_tasks.js';
+import { NativeAccessibilitySource } from './sources.js';
+import type { HistoryEntry, TaskSources } from './tasks.js';
+
+const fixture = (path: string): any => JSON.parse(readFileSync(new URL(`../fixtures/${path}`, import.meta.url), 'utf8'));
+const v2Request = () => fixture('jev-choice-request-v2.json');
+const v1Request = () => fixture('jev-choice-request-v1.json');
+
+function sources(task: NativeTask, harness: string): TaskSources {
+ const payload = fixture(`native/${harness}-window-state-initial-v1.json`);
+ return {
+ ax: NativeAccessibilitySource.fromObservation(
+ parseWindowState(payload, payload.pid, payload.window_id),
+ HARNESSES[harness].platform,
+ { redact: task.redactText, textMethod: task.textMethod }
+ ),
+ visualPath: false,
+ foregroundIds: new Set(),
+ };
+}
+
+function requestFor(taskId: string, history: HistoryEntry[], noteText = 'jev-use native note') {
+ const task = nativeTask(taskId, '/tmp/none.json', { noteText });
+ const taskSources = sources(task, taskId.split('-')[0]);
+ return nativeChoiceRequest(task, taskSources, task.plan(taskSources), history) as any;
+}
+
+const performed = (task: NativeTask, step: number, id: string) => task.historyEntry(step, id, undefined, { outcome: 'done' });
+const byId = (request: any) => Object.fromEntries(request.candidates.map((c: any) => [c.id, c]));
+
+test('v2 accepts progress and the observation carries it', () => {
+ const request = v2Request();
+ request.progress = [{ step: 'Press "Increment"', done: 2, required: 3 }];
+ const validated = validateRequest(request);
+ assert.deepEqual(validated.progress, request.progress);
+ assert.deepEqual(providerObservation(validated).progress, request.progress);
+});
+
+test('v2 without progress keeps its observation, and v1 rejects progress', () => {
+ const validated = validateRequest(v2Request());
+ assert.deepEqual(validated.progress, []);
+ assert.equal('progress' in providerObservation(validated), false);
+ const v1 = v1Request();
+ v1.progress = [];
+ assert.throws(() => validateRequest(v1));
+});
+
+test('v2 rejects malformed progress', () => {
+ const item = { step: 'Press', done: 0, required: 1 };
+ const mutations: Record<string, unknown> = {
+ 'not a list': { step: 'Press' },
+ 'too many': Array(17).fill(item),
+ 'extra key': [{ ...item, value: 'secret' }],
+ 'missing key': [{ step: 'Press', done: 0 }],
+ 'empty step': [{ ...item, step: ' ' }],
+ 'long step': [{ ...item, step: 'x'.repeat(201) }],
+ 'negative done': [{ ...item, done: -1 }],
+ 'boolean done': [{ ...item, done: true }],
+ 'fractional done': [{ ...item, done: 0.5 }],
+ 'zero required': [{ ...item, required: 0 }],
+ 'huge required': [{ ...item, required: 65 }],
+ };
+ for (const [name, progress] of Object.entries(mutations)) {
+ const request = v2Request();
+ request.progress = progress;
+ assert.throws(() => validateRequest(request), Error, name);
+ }
+});
+
+test('only performed actions count', () => {
+ const task = nativeTask('appkit-counter', '/tmp/none.json');
+ const history = [
+ performed(task, 1, 'ax:button:increment'),
+ task.historyEntry(2, 'ax:button:increment', 'background_unsupported'),
+ task.historyEntry(3, 'ax:button:increment', undefined, { stale: true }),
+ task.historyEntry(4, 'reobserve'),
+ performed(task, 5, 'ax:button:increment:foreground'),
+ ];
+ assert.deepEqual(performedCounts(history), { 'ax:button:increment': 2 });
+ assert.deepEqual(task.progress(history), [
+ { step: 'Press the button labeled "Increment"', done: 2, required: 3 },
+ ]);
+ const request = requestFor('appkit-counter', history);
+ assert.deepEqual(Object.keys(request.history[0]).sort(), ['outcome', 'selected_id']);
+});
+
+test('save is described as waiting for the note until the note is set', () => {
+ for (const harness of Object.keys(HARNESSES)) {
+ const request = requestFor(`${harness}-save-note`, [], 'secret note');
+ assert.match(
+ byId(request)['ax:button:save-note'].description,
+ /The task requires this only after: Set the text field "Note" to the task parameter "note" \(not done yet\)\.$/
+ );
+ assert.equal(JSON.stringify(request).includes('secret note'), false);
+ }
+ const task = nativeTask('appkit-save-note', '/tmp/none.json');
+ const after = requestFor('appkit-save-note', [performed(task, 1, 'ax:text_input:note:set:note')]);
+ assert.equal(
+ byId(after)['ax:button:save-note'].description,
+ 'Press the button labeled "Save note". The task still requires this 1 more time(s).'
+ );
+ assert.deepEqual(after.progress.map((item: any) => item.done), [1, 0]);
+});
+
+test('a completed step says so and unordered steps have no precondition', () => {
+ const task = nativeTask('appkit-counter', '/tmp/none.json');
+ const done = requestFor('appkit-counter', [1, 2, 3].map((step) => performed(task, step, 'ax:button:increment')));
+ assert.match(byId(done)['ax:button:increment'].description, /already did this the 3 time\(s\) the task requires\.$/);
+ const size = requestFor('appkit-choose-size', []);
+ assert.equal(byId(size)['ax:checkbox:i-agree'].description.includes('only after'), false);
+ assert.match(byId(size)['ax:checkbox:i-agree'].description, /still requires this 1 more time\(s\)\.$/);
+});
+
+test('a task without steps sends no progress', () => {
+ const task = nativeTask(CANVAS_TASK_ID, '/tmp/none.json');
+ assert.deepEqual(task.steps, []);
+ assert.deepEqual(task.progress([]), []);
+});
+
+test('recorded Python requests are reproduced exactly', () => {
+ for (const item of fixture('native/native-progress-requests-v2.json').cases) {
+ const task = nativeTask(item.task, '/tmp/none.json');
+ const taskSources = sources(task, 'appkit');
+ const request = nativeChoiceRequest(task, taskSources, task.plan(taskSources), item.history);
+ assert.deepEqual(request, item.request, `${item.task} after ${item.history.length} steps`);
+ validateRequest(request);
+ }
+});
diff --git a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts b/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts
index 955da3923e..4d57189a5c 100644
--- a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts
@@ -9,7 +9,13 @@
*/
import { readFile } from 'node:fs/promises';
-import { MAX_ELEMENTS, REQUEST_SCHEMA_V2 } from './choose_action.js';
+import {
+ MAX_ELEMENTS,
+ MAX_PROGRESS,
+ MAX_PROGRESS_COUNT,
+ REQUEST_SCHEMA_V2,
+ type ProgressItem,
+} from './choose_action.js';
import {
PARAMETER_NAME_PATTERN,
elementState,
@@ -159,6 +165,33 @@ export function compose(
export type CompactElement = { role_class: string; label: string; state: string };
+/**
+ * One step a task requires, counted from the runner's own performed actions.
+ * `candidateId` names the candidate that performs the step; its `:foreground`
+ * variant counts too. The description is task-authored and value-free. A step
+ * with `afterPrevious` (the default) waits for every earlier step to be done.
+ */
+export type TaskStep = Readonly<{
+ description: string;
+ candidateId: string;
+ times?: number;
+ afterPrevious?: boolean;
+}>;
+
+/**
+ * Count the actions this run dispatched successfully, by candidate ID. Only
+ * entries marked `performed` count; nothing is read from the application.
+ */
+export function performedCounts(history: readonly HistoryEntry[]): Record<string, number> {
+ const counts: Record<string, number> = {};
+ for (const item of history) {
+ if (item.performed !== true) continue;
+ const base = item.selected_id.replace(/:foreground$/, '');
+ counts[base] = (counts[base] ?? 0) + 1;
+ }
+ return counts;
+}
+
export type NativeStep = Readonly<{
candidates: Candidate[];
stats: ComposeStats;
@@ -185,6 +218,7 @@ export type NativeTaskSpec = Readonly<{
visualTargets?: readonly string[];
visualMinConfidence?: number;
mockPreferences?: readonly string[];
+ steps?: readonly TaskStep[];
}>;
export class NativeTask implements Task {
@@ -203,6 +237,12 @@ export class NativeTask implements Task {
/** OCR confidence a visual target must reach; exact-text uniqueness still applies. */
readonly visualMinConfidence: number;
readonly mockPreferences: readonly string[];
+ /**
+ * Ordered steps the task requires (#4313). The request reports how often
+ * this run has performed each, and a step's candidate names any earlier
+ * step that is not done yet. Empty means the request carries no progress.
+ */
+ readonly steps: readonly TaskStep[];
/** The oracle is polled after every action, so no candidate is special. */
readonly completionCandidateIds: ReadonlySet<string> = new Set();
@@ -221,6 +261,7 @@ export class NativeTask implements Task {
this.visualTargets = spec.visualTargets ?? [];
this.visualMinConfidence = spec.visualMinConfidence ?? 0.8;
this.mockPreferences = spec.mockPreferences ?? [];
+ this.steps = spec.steps ?? [];
const unknown = [...this.allowedActions].filter((action) => !ACTION_KINDS.has(action));
if (unknown.length) throw new Error(`unknown action kinds: ${unknown.sort().join(', ')}`);
for (const parameter of this.parameters) {
@@ -229,6 +270,13 @@ export class NativeTask implements Task {
}
}
if (!Number.isInteger(this.maxSteps) || this.maxSteps < 1) throw new Error('maxSteps must be positive');
+ if (this.steps.length > MAX_PROGRESS) throw new Error(`a task declares at most ${MAX_PROGRESS} steps`);
+ for (const taskStep of this.steps) {
+ const times = taskStep.times ?? 1;
+ if (!Number.isInteger(times) || times < 1 || times > MAX_PROGRESS_COUNT) {
+ throw new Error(`step times must be from 1 to ${MAX_PROGRESS_COUNT}`);
+ }
+ }
}
get allowedActionKinds(): ReadonlySet<string> {
@@ -378,7 +426,46 @@ export class NativeTask implements Task {
: candidateId === 'reobserve'
? 'took no action and requested a fresh observation'
: (options.outcome ?? 'completed');
- return { step, selected_id: candidateId, outcome: (this.redact(text) as string).slice(0, 128) };
+ const performed =
+ !options.stale && refusal === undefined && candidateId !== 'reobserve' && candidateId !== 'abstain';
+ return {
+ step,
+ selected_id: candidateId,
+ outcome: (this.redact(text) as string).slice(0, 128),
+ ...(performed ? { performed: true as const } : {}),
+ };
+ }
+
+ /** Each declared step and how often this run has performed it. */
+ progress(history: readonly HistoryEntry[]): ProgressItem[] {
+ const counts = performedCounts(history);
+ return this.steps.map((taskStep) => ({
+ step: this.redact(taskStep.description) as string,
+ done: Math.min(counts[taskStep.candidateId] ?? 0, MAX_PROGRESS_COUNT),
+ required: taskStep.times ?? 1,
+ }));
+ }
+
+ /**
+ * A sentence stating a candidate's place in the task's declared steps: a
+ * done step says so; a step with an earlier step not yet done names it; a
+ * due step says how many more times the task requires it.
+ */
+ stepNote(candidateId: string, history: readonly HistoryEntry[]): string {
+ const base = candidateId.replace(/:foreground$/, '');
+ const counts = performedCounts(history);
+ const index = this.steps.findIndex((taskStep) => taskStep.candidateId === base);
+ if (index < 0) return '';
+ const taskStep = this.steps[index];
+ const times = taskStep.times ?? 1;
+ if ((counts[base] ?? 0) >= times) {
+ return ` This run already did this the ${times} time(s) the task requires.`;
+ }
+ const pending = (taskStep.afterPrevious === false ? [] : this.steps.slice(0, index))
+ .filter((earlier) => (counts[earlier.candidateId] ?? 0) < (earlier.times ?? 1))
+ .map((earlier) => earlier.description);
+ if (pending.length) return ` The task requires this only after: ${pending.join('; ')} (not done yet).`;
+ return ` The task still requires this ${times - (counts[base] ?? 0)} more time(s).`;
}
async reset(): Promise<void> {
@@ -419,7 +506,11 @@ export function visualFallbackReason(
return undefined;
}
-/** Build the cua.jev_choice_request_v2 the provider receives; no tokens, values, or pixels. */
+/**
+ * Build the cua.jev_choice_request_v2 the provider receives; no tokens,
+ * values, or pixels. A task that declares steps adds the progress counted
+ * from this run's performed actions.
+ */
export function nativeChoiceRequest(
task: NativeTask,
sources: TaskSources,
@@ -449,9 +540,10 @@ export function nativeChoiceRequest(
history: history.slice(-MAX_HISTORY).map((item) => ({ selected_id: item.selected_id, outcome: item.outcome })),
candidates: step.candidates.map((candidate) => ({
id: candidate.id,
- description: task.redact(candidate.description),
+ description: task.redact(candidate.description + task.stepNote(candidate.id, history)),
...(candidate.source ? { source: candidate.source } : {}),
})),
+ ...(task.steps.length ? { progress: task.progress(history) } : {}),
};
}
@@ -588,6 +680,9 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTa
allowForeground,
maxSteps: COUNTER_TARGET + 3,
mockPreferences: ['ax:button:increment'],
+ steps: [
+ { description: 'Press the button labeled "Increment"', candidateId: 'ax:button:increment', times: COUNTER_TARGET },
+ ],
});
}
if (kind === 'save-note') {
@@ -604,6 +699,13 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTa
allowForeground,
maxSteps: 5,
mockPreferences: ['ax:text_input:note:set:note', 'ax:button:save-note'],
+ steps: [
+ {
+ description: 'Set the text field "Note" to the task parameter "note"',
+ candidateId: 'ax:text_input:note:set:note',
+ },
+ { description: 'Press the button labeled "Save note"', candidateId: 'ax:button:save-note' },
+ ],
});
}
return new NativeTask({
@@ -616,6 +718,11 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTa
allowForeground,
maxSteps: 5,
mockPreferences: ['ax:radio:large', 'ax:checkbox:i-agree'],
+ steps: [
+ { description: 'Select the radio option "Large"', candidateId: 'ax:radio:large' },
+ // The oracle accepts either order, so neither step waits for the other.
+ { description: 'Toggle the checkbox "I agree"', candidateId: 'ax:checkbox:i-agree', afterPrevious: false },
+ ],
});
}
diff --git a/libs/cua-driver/examples/jev-use/typescript/run_native.ts b/libs/cua-driver/examples/jev-use/typescript/run_native.ts
index fabf718f87..63551edb16 100644
--- a/libs/cua-driver/examples/jev-use/typescript/run_native.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/run_native.ts
@@ -28,7 +28,7 @@ import {
windowStateArguments,
type NativeTask,
} from './native_tasks.js';
-import { chooseS1Service, s1ServiceUrl } from './s1_service.js';
+import { chooseS1Service, S1ServiceError, s1ServiceUrl } from './s1_service.js';
import { backgroundRefusalCode, Driver, DriverToolError, supportsCaptureBoundClick } from './run.js';
import { NativeAccessibilitySource, VisualRegionSource } from './sources.js';
import type { HistoryEntry, Outcome, TaskSources } from './tasks.js';
@@ -274,12 +274,24 @@ export async function runTask(args: Arguments, task: NativeTask): Promise<Outcom
validateRequest(request);
const decideStarted = performance.now();
- const decision =
- args.provider === 'mock'
- ? chooseMockForTask(task, sources, plan.candidates, history)
- : args.provider === 's1'
- ? await chooseS1Service(request)
- : await chooseLive(request);
+ let decision: { choice: string | null; confidence: number; probabilities: Readonly<Record<string, number>> };
+ try {
+ decision =
+ args.provider === 'mock'
+ ? chooseMockForTask(task, sources, plan.candidates, history)
+ : args.provider === 's1'
+ ? await chooseS1Service(request)
+ : await chooseLive(request);
+ } catch (error) {
+ if (!(error instanceof S1ServiceError)) throw error;
+ // A tied or malformed score is not an action; fail closed with a
+ // logged outcome instead of a stack trace.
+ await writeEvent(args.log, {
+ event: 'outcome', outcome: 'unknown', phase: 'decide', step,
+ error: 'S1ServiceError', reason: error.message.slice(0, 128),
+ });
+ return 'unknown';
+ }
const baseEvent: Record<string, unknown> = {
event: 'step',
step,
diff --git a/libs/cua-driver/examples/jev-use/typescript/tasks.ts b/libs/cua-driver/examples/jev-use/typescript/tasks.ts
index 93fac5a78e..8caec8c26f 100644
--- a/libs/cua-driver/examples/jev-use/typescript/tasks.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/tasks.ts
@@ -76,7 +76,11 @@ export function requirePage(sources: TaskSources): BrowserSemanticSource {
return sources.page;
}
-export type HistoryEntry = Readonly<{ step: number; selected_id: string; outcome: string }>;
+/**
+ * One runner history item. `performed` marks a dispatched, successful native
+ * action; it stays in the runner and is never sent to a provider.
+ */
+export type HistoryEntry = Readonly<{ step: number; selected_id: string; outcome: string; performed?: true }>;
/** The interface the runner uses for every task. */
export interface Task {
diff --git a/rfcs/4268-jev-use-native-candidates.md b/rfcs/4268-jev-use-native-candidates.md
index 9cf1b433d7..2a02db9400 100644
--- a/rfcs/4268-jev-use-native-candidates.md
+++ b/rfcs/4268-jev-use-native-candidates.md
@@ -3,7 +3,7 @@ title: Native accessibility candidates for jev-use decision loops
authors:
- f-trycua
created: 2026-09-27
-last_updated: 2026-09-28
+last_updated: 2026-09-29
status: accepted
discussion: https://github.com/trycua/cua/issues/4268
rfc_pr: https://github.com/trycua/cua/pull/4269
@@ -16,6 +16,7 @@ implementation:
- https://github.com/trycua/cua/pull/4298
- https://github.com/trycua/cua/pull/4301
- https://github.com/trycua/cua/pull/4299
+ - https://github.com/trycua/cua/pull/4324
supersedes:
superseded_by:
---
@@ -562,7 +563,13 @@ to v1:
- the root may include `snapshot_id` alongside the required `capture_id`; and
- the root may include `elements`, a compact, redacted list of at most 64
native elements (`role_class`, `label`, `state`). S1's text adapter renders
- it as a genuine accessibility tree.
+ it as a genuine accessibility tree; and
+- the root may include `progress`, at most 16 task steps
+ (`step`, `done`, `required`), added by
+ [#4324](https://github.com/trycua/cua/pull/4324). `done` counts only the
+ runner's own successful actions and is never read from the application, so
+ it carries no field values. `provider_observation` adds it only when it is
+ present.
The v1 schema, validators, and fixtures remain supported unchanged. A v2
request that has no `source`, `snapshot_id`, or `elements` is semantically
@@ -846,6 +853,14 @@ Implementation:
to Cancel.
- Phase 4, [#4299](https://github.com/trycua/cua/pull/4299): the native
apps guide and the `skills/jev-use` update.
+- S1 reliability, [#4324](https://github.com/trycua/cua/pull/4324) for
+ [#4313](https://github.com/trycua/cua/issues/4313): each built-in task
+ declares its ordered steps. The request's `progress` reports how often this
+ run has performed each step, and each step's candidate description names an
+ unfinished earlier step (Save waits for the note), says the step is done, or
+ says how many more times it is due. Choose-size's two steps are unordered,
+ as its oracle is. The runners log an S1 service failure as an outcome, not
+ a stack trace.
Phase 3 live evidence (released Driver 0.30.2, cua-perception 0.2.1,
`cua-s1-4b-0.2@16818868`; each run verified by the harness's own state file):
@@ -866,7 +881,26 @@ Phase 3 live evidence (released Driver 0.30.2, cua-perception 0.2.1,
Windows.
- The tasks produce 3–7 candidates. Jev chose correctly in 220/220 decisions
and S1 in 211/216.
-
-Acceptance criteria not yet met, which keep this RFC `accepted`: S1 5/5 in
-every native row, WinUI3 coverage, and accuracy at about 12 and 24
-candidates.
+- Cause of the S1 misses, reproduced by replaying the recorded AppKit states
+ with the failing runs' capture IDs: after two increments S1 split 0.45/0.44
+ between abstain and Increment; at the first save-note step it split
+ 0.39/0.38 between setting the note and Save; and the service error was that
+ same split landing on an exact fp16 logit tie, which `choose()` refuses.
+
+S1 reliability evidence for #4324 (released Driver 0.30.4, same checkpoint):
+
+- Offline replay of the AppKit, WPF, and GTK3 fixtures at eight mid-task
+ states, eight capture IDs each, plus the three recorded failing captures:
+ S1 was correct in 191/195 decisions on `main` (two abstains, one Save
+ before the note, and one tie error) and 195/195 with progress. Its lowest confidence after two increments rose
+ from 0.45 to 0.81, and at the first save-note step from 0.38 to 0.79. Jev
+ was correct in 195/195 either way.
+- Live on macOS in a Lume worker, recorded: AppKit counter, save-note, and
+ choose-size passed 5/5 for S1 and 5/5 for Jev in both Python and TypeScript
+ (60/60 runs, 140 decisions, all correct). S1's lowest confidence per task
+ was 0.83, 0.79, and 0.65. The Windows and Linux rows were verified by the
+ offline replay only in this round.
+
+Acceptance criteria not yet met, which keep this RFC `accepted`: a live
+rerun of the WPF and GTK3 rows with `progress` to confirm S1 5/5 there,
+WinUI3 coverage, and accuracy at about 12 and 24 candidates.
diff --git a/skills/jev-use/SKILL.md b/skills/jev-use/SKILL.md
index f848cff89d..b3ef894f98 100644
--- a/skills/jev-use/SKILL.md
+++ b/skills/jev-use/SKILL.md
@@ -22,7 +22,8 @@ probabilities. Invoke the Python interpreter and absolute chooser path directly
without a shell.
For native desktop applications, use `NativeAccessibilitySource` and
`cua.jev_choice_request_v2`, which adds a per-candidate `source` (`page`, `ax`,
-or `visual`) and compact value-free `elements`. Browser tasks keep sending v1.
+or `visual`), compact value-free `elements`, and optional `progress` counted
+from the runner's own performed actions. Browser tasks keep sending v1.
Prefer browser DOM and semantic evidence. The optional visual adapter consumes
the public `cua.visual_regions_v1` result only when Driver advertises both
`parse_visual_regions` and the capture-bound `click.capture_id` input.