Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
Send a refused combobox back to the rows the tool named (SKY-16261) (#8543)
pedrohsdbbb33e95930ec
diff --git a/skyvern/forge/taskv3/engine.py b/skyvern/forge/taskv3/engine.pyindex dd7161800..e3824e2eb 100644--- a/skyvern/forge/taskv3/engine.py+++ b/skyvern/forge/taskv3/engine.py@@ -96,7 +96,7 @@ How to work:- Act by ref: pass the printed `ref=N` exactly as printed as the `selector` argument of `type`, `select_option`, `select_combobox`, `click`, `hover`, `press_key`, `scroll`, `wait`, `file_upload`, `get_html`. A real CSS selector is still accepted, for the rare element only `get_html` revealed.- Be efficient — this is the whole point of the engine. After observing a form once, fill every field you can before doing anything that reloads the page. Minimize tool calls and turns.- Batch aggressively: in ONE turn you can `type` into many fields AND `click` many radio/checkbox options AND `select_option` on several dropdowns. Answer a whole form section in a single turn — never spend a separate turn on each click.-- Autocomplete / typeahead / combobox fields (location, school, employer lookups) render suggestions only AFTER you type, and the raw text you type is NOT accepted until you pick a suggestion. Use the `select_combobox` tool (selector + value) for these — it types, waits for the suggestions to render, selects the best-matching one, and verifies the field committed. Do NOT `type` into them or press keys yourself. If `select_combobox` returns an error, the field is genuinely unfilled — try a fuller value or report it; never treat it as done.+- Autocomplete / typeahead / combobox fields (location, school, employer lookups) render suggestions only AFTER you type, and the raw text you type is NOT accepted until you pick a suggestion. Use the `select_combobox` tool (selector + value) for these — it types, waits for the suggestions to render, selects the best-matching one, and verifies the field committed. Do NOT `type` into them or press keys on your own initiative. If `select_combobox` returns an error, the field is genuinely unfilled — never treat it as done. Act on what that error tells you rather than substituting a value of your own: this field commits only the suggestions the page itself offers, and those are often coarser than the value you hold.- `observe` already gives you everything you need to fill a field (ref, label, type, current value, options, and the surrounding question text) — act on it directly. `get_html` markup is a rare last resort for ONE specific element `observe` failed to describe: NEVER read a whole page/form/section's markup, NEVER call it twice for the same element, and NEVER inspect more than once before acting. Its text format (the page's visible text) is the one whole-page read that is cheap and honest — use it when the goal is about what the page shows, not where a control is.- `look` is a separate last resort for when the TEXT tools are not enough: you can't tell what the page looks like, a control you expect isn't in `observe` (custom or shadow-DOM widgets), or an action isn't taking and you can't tell why. It returns ONE screenshot with every visible control boxed and numbered; then act on a number with `click(mark=N)` or `type(mark=N, text=...)`. Do NOT call `look` to double-check what `observe` already told you, and do not call it every turn — it is for when you are genuinely stuck on something visual.- Inspecting the page does NOT progress the task — only `type`/`select_option`/`click` do. If your recent turns were mostly `observe`/`get_html` with little typing or clicking, you are stuck inspecting: stop, and fill every field you can from the latest `observe` snapshot using its refs before doing anything else.diff --git a/tests/unit/test_taskv3_engine.py b/tests/unit/test_taskv3_engine.pyindex d0cd44c17..33728ee45 100644--- a/tests/unit/test_taskv3_engine.py+++ b/tests/unit/test_taskv3_engine.py@@ -27,6 +27,7 @@ from skyvern.forge.taskv3.engine import (DEFAULT_MAX_TURNS,MAX_TOOL_CALLS_PER_ACTION_STEP,MAX_TURNS_PER_ACTION_STEP,+ SYSTEM_PROMPT,coerce_v3_parameters,run_task_v3_agent_loop,taskv3_runaway_backstops,@@ -1691,3 +1692,46 @@ async def test_a_run_with_no_identity_is_not_given_a_code_tool(monkeypatch: pyteassert not askedassert _advertised(caller) == _SURFACE_OFF_TOOL_NAMES | {"finish"}+++# The combobox bullet, pinned verbatim. This bullet produced FIVE defects in one PR, every one of them+# a claim that some `select_combobox` error class falsifies -- only 4 of its 18 classes are row-related,+# so guidance quantifying over them is wrong for most. A phrase blacklist cannot fire on wording nobody+# anticipated ("every refusal provides a recovery action" would pass and is false), so the enforcing+# mechanism is a snapshot: it fires on ALL change, which is the point for this text and not a cost.+_COMBOBOX_BULLET = (+ "- Autocomplete / typeahead / combobox fields (location, school, employer lookups) render suggestions only "+ "AFTER you type, and the raw text you type is NOT accepted until you pick a suggestion. Use the "+ "`select_combobox` tool (selector + value) for these — it types, waits for the suggestions to render, selects "+ "the best-matching one, and verifies the field committed. Do NOT `type` into them or press keys on your own "+ "initiative. If `select_combobox` returns an error, the field is genuinely unfilled — never treat it as done. "+ "Act on what that error tells you rather than substituting a value of your own: this field commits only the "+ "suggestions the page itself offers, and those are often coarser than the value you hold."+)+++def test_combobox_bullet_is_pinned_so_every_edit_is_re_derived_against_the_error_taxonomy() -> None:+ """The property, which the snapshot enforces rather than expresses: the bullet may claim what the+ MODEL should do, never what an ERROR CONTAINS, and any prohibition is scoped by PROVENANCE (a value+ or keystroke the model originates) rather than by shape. Four wordings broke the first rule -- "try+ a fuller value", "pass a listed row's text back" (identical_rows wants a click), "the error states a+ step" (row-less commit failures state none), "do not retype a longer value" (ambiguous_rows' own+ next_step IS longer) -- and the unconditional typing ban broke the second, contradicting+ identical_rows' "type the value to reopen the list" (tools.py:666).++ WHAT THIS IS: a change-detector, not a correctness test. It cannot tell a semantic defect from a+ rewording -- editing this constant alongside a broken prompt restores green. All it does is force a+ human to look, which is the most any test here can do.++ WHY EXACT PROSE, given CLAUDE.md:129 ("do not assert exact prompt prose ... WHEN A BEHAVIOR/CONTRACT+ ASSERTION EXISTS"): that precondition is not met, and the claim is checkable. Every browser e2e+ substitutes `ScriptedLLMCaller` for the model -- test_taskv3_fixture_parity.py says so in its own+ docstring -- so no test in this repo can assert prompt-driven behaviour. The structural fix (a+ mechanical guidance-vs-next_step consistency check) is tracked as SKY-16299, not built here.++ Updating this snapshot is not a formality: re-derive the new text against every error class in+ tools.py first, and against the WHOLE bullet -- a new clause can falsify an older one, which is how+ the pre-existing unconditional typing ban became a live contradiction."""+ bullet = next(line for line in SYSTEM_PROMPT.splitlines() if "select_combobox` tool" in line)++ assert bullet == _COMBOBOX_BULLET