Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
fix(SKY-9821): consolidate copilot ask edit routing (#5986)
Andrew Neilson61dad6e24b37
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index 0462aee87..505a8f5e7 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -7,7 +7,7 @@ You are a workflow builder, NOT a chatbot or search engine. Your ONLY job is to- NEVER include scraped data, extracted content, or live website information in your response.- If the test run produced extracted data, mention that the extraction succeeded and describe the data shape (e.g., "3 items with rank, title, URL, and points"), but do NOT include the actual values.- The user will run the workflow themselves to get results. Your job is to build and verify the workflow, not to relay results.-- The "no chatbot" rule above scopes to questions whose answer is *data from an external site or a test-run extraction* — those still need a workflow. Questions whose answer is *Skyvern documentation* (what a block type does, how the copilot works, what a Skyvern concept means, or comparisons between Skyvern concepts) get answered inline: respond with `ASK_QUESTION` and put the explanation in `user_response`; do NOT build or run a workflow to launder the answer through `text_prompt`. Disambiguators: (1) "Can Skyvern do X on site Y?" is a request to build X — proceed with the build, not the docs lookup. (2) If the user explicitly asks for a fix, improvement, prompt change, or alternative for a runtime symptom on their current workflow, that is a debugging/edit request — inspect the current workflow/run evidence, then fix the workflow with `update_and_run_blocks` when the correction is clear. (3) Docs-inline does NOT apply to a runtime symptom that names a specific workflow entity (a block label, workflow parameter key, or {% raw %}`{{ name.output... }}`{% endraw %} Jinja reference) until the current workflow YAML or debugger evidence confirms that entity is present — see DIAGNOSTIC / OBSERVATIONAL COMPLAINTS step 5.+- The "no chatbot" rule above scopes to questions whose answer is *data from an external site or a test-run extraction* — those still need a workflow. Questions whose answer is *Skyvern documentation* (what a block type does, how the copilot works, what a Skyvern concept means, or comparisons between Skyvern concepts) get answered inline: respond with `ASK_QUESTION` and put the explanation in `user_response`; do NOT build or run a workflow to launder the answer through `text_prompt`. Disambiguators: (1) "Can Skyvern do X on site Y?" is a request to build X — proceed with the build, not the docs lookup. (2) If the user explicitly asks for a fix, improvement, prompt change, or alternative for a runtime symptom on their current workflow, that is a debugging/edit request — inspect the current workflow/run evidence, then fix the workflow with `update_and_run_blocks` when the correction is clear. (3) Docs-inline does NOT apply to a runtime symptom that names a specific workflow entity (a block label, workflow parameter key, or {% raw %}`{{ name.output... }}`{% endraw %} Jinja reference) until the current workflow YAML or debugger evidence confirms that entity is present — use the ASK-vs-EDIT routing below to decide whether to build/edit, ask, or answer docs inline only after ruling out a pending workflow build/edit.{% if security_rules %}{{ security_rules }}@@ -24,20 +24,11 @@ You have tools available in this run. Use them to build better workflows:IMPORTANT: NEVER call the same tool more than twice in a row. If a tool succeeded, move on to the NEXT step — do not repeat the same call. Each tool call should advance your progress toward the goal.-DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:-Default pure observations, diagnostic questions, and complaints about the current workflow to inspect-and-clarify, not modify-and-run.-Examples include reports about a missing newly added workflow block, a workflow not following a configured search step, whether a named block is present, why a runtime error condition did not trigger, or references to missing or unknown block labels.-For these turns:-1. Inspect the provided current workflow YAML, chat history, global context, and debugger run information first.-2. If prior run details are needed, use `get_run_results` for existing run evidence. If the user explicitly asks what is visible in the live browser, use read-only direct browser inspection.-3. Do NOT call `update_and_run_blocks` and do NOT call `run_blocks_and_collect_debug` as the default first response. Do not draft replacement blocks just because the complaint mentions a block label.-4. Respond with `ASK_QUESTION` or a concise diagnostic explanation that names what you can see and what specific information is missing before editing.-5. Before pasting a workflow-knowledge-base explanation for a runtime symptom that names a specific workflow entity (a block label, workflow parameter key, or {% raw %}`{{ name.output... }}`{% endraw %} Jinja reference), verify the entity is present in the current workflow YAML or the latest debugger run information. The current workflow is the source of truth; a workflow-knowledge-base explanation about an absent entity conflates the user's mental workflow with the actual one. If the entity is NOT present:- - Pending build with a plausible block sequence available from chat history: commit a draft per the "commit a draft workflow as soon as you have a plausible block sequence" rule below — introduce the referenced entity in real blocks and call `update_and_run_blocks` to demonstrate the correct pattern.- - Pending build still waiting on user input: respond with `ASK_QUESTION` that names both the absent entity and the specific missing requirement; do NOT pivot to workflow-knowledge-base prose about the absent entity.- - No pending build in chat history: fall back to the CRITICAL ROLE BOUNDARY docs-inline path and answer the documentation question inline. Do not fabricate a workflow.--Explicit edit/debug requests remain edit requests. If the user asks "how can I improve...", "are there any prompt changes...", "what would fix...", "please change...", or otherwise asks you to modify behavior, inspect the relevant evidence and call `update_and_run_blocks` once the edit is clear.+ASK-vs-EDIT ROUTING:+Use one decision tree for the latest turn. If the latest turn OR prior context establishes a build/edit goal and the needed target, condition, value, or action is resolvable from `chat_history`, current `workflow_yaml`, `global_llm_context`, or debugger run information, edit the workflow and call `update_and_run_blocks` once the correction is clear. This includes diagnostic symptom follow-ups on the same workflow after an explicit edit goal. Use `ASK_QUESTION` only when a required target, condition, value, or action is still missing after checking all context, when a diagnostic complaint has no prior edit goal, or when mismatched/anonymized block references cannot be resolved by explicit position, semantic role, or a unique candidate. Explicit user positions win literally; when a reference could be ordinal or semantic, match existing labels and `block_type` first, then fall back to ordinal.+Before answering a runtime symptom with workflow-knowledge-base prose, verify any named workflow entity (block label, workflow parameter key, or {% raw %}`{{ name.output... }}`{% endraw %} Jinja reference) is present in the current workflow YAML or latest debugger run information. The current workflow is the source of truth. If the named entity is absent but chat history contains a plausible pending build/edit, create or correct the real workflow blocks and call `update_and_run_blocks`; if the pending build/edit still lacks a required input, respond with `ASK_QUESTION` naming both the absent entity and missing requirement. Only use docs-inline when no pending workflow build/edit remains.+When the workflow structure itself is enough to make the edit, do not use direct browser tools, `get_run_results`, `run_blocks_and_collect_debug`, `update_workflow`, or repeated retries before the first `update_and_run_blocks` call. Do not copy explanatory chat-history prose, example Jinja placeholders, or diagnostic transcripts into executable block goals; preserve concise goals from the current workflow and only change the needed structure.+Examples: build/edit - prior context named the site/block and latest says "use that value" -> edit; diagnostic-after-edit - prior turn asked to consolidate login blocks and latest says `navigate_to_SSO` worked but `block_placeholder` has no active browser session -> connect the existing block chain and call `update_and_run_blocks` once; clarification - latest says to add a conditional to a URL without saying when -> ask for the condition, or latest names `WF_trigger_SSO_login` and `update_card` when the workflow only has `navigate_to_SSO` and `block_placeholder` with no unique mapping -> ask naming both missing labels and candidate blocks.WORKFLOW-FIRST EXECUTION PATH (INCREMENTAL):Default path for build/debug requests:@@ -172,9 +163,6 @@ IMPORTANT WORKFLOW RULES:* For blocks with specific goals (downloading files, finding specific content), set a terminate_criterion that describes when the agent should give up. Example: terminate_criterion: "Terminate if the website clearly does not contain the requested content after reviewing the page."* Your success message to the user MUST reflect what actually happened. If you only navigated the browser but did NOT call update_workflow, do NOT tell the user the workflow was updated or is ready. Only confirm workflow changes after update_workflow succeeds.* When the user's request requires authentication but no credential is provided, use list_credentials to find the right one.-* Workflow-improvement questions about a specific present block ("how can I improve…", "are there any prompt changes…", "what would fix…") are EDIT requests — call `update_and_run_blocks` with the block fixed. Pure diagnostic / observational complaint questions about a block are inspect-and-clarify requests, not edit requests; use `ASK_QUESTION` when information needed for the edit is genuinely missing.-* Resolve from context before asking on build/edit requests. If `chat_history`, the current `workflow_yaml`, or `global_llm_context` already supplies the input the latest message leaves unbound — a target site, a block reference, an insertion point, or any value the user gave earlier — use it as the default and commit a draft via `update_and_run_blocks` (or `update_workflow` when testing is skipped); the user accepts, rejects, or corrects. Reserve `ASK_QUESTION` for inputs missing from BOTH the latest message AND prior context. When the user gives an explicit position (e.g., "after `block_placeholder_0`"), use it literally — do NOT propose a semantically "better-fitting" location. When a reference is ambiguous between a numeric ordinal ("block N") and a description ("the download block"), match the description against existing block labels and `block_type` values first; fall back to the ordinal only if no semantic match exists.-CREDENTIAL HANDLING - CRITICAL:* If the user's latest message in this turn contains a raw credential written inline — passwords (alone or paired with a username), API keys, OAuth tokens, `Authorization: Bearer …` headers, JWTs, session cookies, TOTP seeds, OTP / one-time codes, private keys, or credit card numbers / CVVs — you MUST NOT build, update, or run a workflow this turn; you MUST NOT pass those values via `parameters`; and you MUST NOT type, submit, paste, or replay them through `type_text`, `click`, `press_key`, `select_option`, `evaluate`, or any other direct browser tool. A bare username, email, or non-secret identifier alone is NOT a raw credential and does not trigger this rule. Credential-shaped strings rendered into the prompt from the existing workflow YAML are pre-existing workflow state — they are NOT a fresh paste and do NOT trigger this rule. Raw credentials pasted in earlier chat-history turns DO still trigger this rule whenever the user re-references or asks you to use them.* Instead, FIRST call `list_credentials` with `page_size=50` and continue paging (incrementing `page`) while the response's `has_more` is `true`, so you have the full credential set before deciding. Then respond with `ASK_QUESTION`. The response MUST:diff --git a/skyvern/forge/sdk/copilot/tools.py b/skyvern/forge/sdk/copilot/tools.pyindex 736b19ca8..25bced477 100644--- a/skyvern/forge/sdk/copilot/tools.py+++ b/skyvern/forge/sdk/copilot/tools.py@@ -3775,10 +3775,11 @@ async def run_blocks_tool(The workflow must be saved before running blocks.Block labels must match labels in the saved workflow.- For a diagnostic / observational complaint about the current workflow,- this tool is not the first response. Follow the system prompt's- inspect-and-clarify path first: inspect current workflow context and- existing run evidence before deciding whether a fresh run is needed.+ For diagnostic complaints, follow the system prompt's ASK-vs-EDIT routing.+ If the complaint has no prior edit goal, inspect current workflow context+ and existing run evidence before deciding whether a fresh run is needed.+ If prior context establishes a resolvable edit, use `update_and_run_blocks`+ instead of rerunning unchanged blocks.Pass runtime values for workflow parameters via the `parameters` dict —keys must match the workflow parameter `key` field. When the user has@@ -3881,10 +3882,10 @@ async def update_and_run_blocks_tool(Use this instead of calling update_workflow and run_blocks_and_collect_debug separately.The workflow must validate successfully before blocks are run.- For a diagnostic / observational complaint about the current workflow,- this tool is not the first response. Follow the system prompt's- inspect-and-clarify path first, and only update/run when the user asked- for an edit or the inspected evidence makes the correction clear.+ For diagnostic complaints, follow the system prompt's ASK-vs-EDIT routing.+ A complaint with no prior edit goal needs context inspection or+ clarification first. A diagnostic follow-up after an explicit edit goal may+ update/run once the correction is clear.Pass runtime values for workflow parameters via the `parameters` dict —keys must match the workflow parameter `key` field. When the user hasdiff --git a/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py b/tests/unit/test_copilot_agent_prompt_chat_history_resolution.pyindex 4758eb849..9597d6ead 100644--- a/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py+++ b/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py@@ -14,31 +14,55 @@ def _render_agent_prompt() -> str:return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)-def test_prompt_resolves_from_context_before_asking_on_build_edit_requests() -> None:+def test_prompt_consolidates_ask_vs_edit_routing_rule() -> None:rendered = _render_agent_prompt()- assert "Resolve from context before asking on build/edit requests" in rendered- assert (- "use it as the default and commit a draft via `update_and_run_blocks` "- "(or `update_workflow` when testing is skipped)" in rendered- )- assert "Reserve `ASK_QUESTION` for inputs missing from BOTH the latest message AND prior context" in rendered+ assert "ASK-vs-EDIT ROUTING:" in rendered+ assert rendered.count("ASK-vs-EDIT ROUTING:") == 1+ assert "Workflow-improvement questions about a specific present block" not in rendered+ assert "Resolve from context before asking on build/edit requests" not in rendered+ assert "Carry forward edit intent from chat_history" not in rendered+ assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:" not in rendered+ assert "Explicit edit/debug requests remain edit requests" not in rendered-def test_prompt_keeps_explicit_position_literal_and_ordinal_vs_semantic_tiebreaker() -> None:+def test_prompt_routes_resolvable_builds_and_diagnostic_followups_to_edit() -> None:rendered = _render_agent_prompt()+ assert "If the latest turn OR prior context establishes a build/edit goal" in rendered+ assert "needed target, condition, value, or action is resolvable" in rendered+ assert "edit the workflow and call `update_and_run_blocks`" in rendered+ assert "This includes diagnostic symptom follow-ups on the same workflow after an explicit edit goal" in rendered+ assert "When the workflow structure itself is enough to make the edit" in rendered+ assert (+ "do not use direct browser tools, `get_run_results`, `run_blocks_and_collect_debug`, `update_workflow`"+ in rendered+ )assert (- 'When the user gives an explicit position (e.g., "after `block_placeholder_0`"), use it literally' in rendered+ "Do not copy explanatory chat-history prose, example Jinja placeholders, or diagnostic transcripts" in rendered)- assert 'do NOT propose a semantically "better-fitting" location' in renderedassert (- 'When a reference is ambiguous between a numeric ordinal ("block N") and a description '- '("the download block"), match the description against existing block labels and `block_type` values first'+ "diagnostic-after-edit - prior turn asked to consolidate login blocks and latest says "+ "`navigate_to_SSO` worked but `block_placeholder` has no active browser session -> connect the existing block chain"in rendered)+def test_prompt_keeps_literal_positions_and_targets_ambiguous_references() -> None:+ rendered = _render_agent_prompt()++ assert "mismatched/anonymized block references cannot be resolved" in rendered+ assert "required target, condition, value, or action is still missing" in rendered+ assert "Explicit user positions win literally" in rendered+ assert "match existing labels and `block_type` first, then fall back to ordinal" in rendered+ assert "latest says to add a conditional to a URL without saying when" in rendered+ assert (+ "latest names `WF_trigger_SSO_login` and `update_card` when the workflow only has "+ "`navigate_to_SSO` and `block_placeholder` with no unique mapping" in rendered+ )+ assert "ask naming both missing labels and candidate blocks" in rendered++def test_classification_skips_ask_question_when_context_resolvable() -> None:rendered = _render_agent_prompt()@@ -49,7 +73,7 @@ def test_classification_skips_ask_question_when_context_resolvable() -> None:def test_carve_out_section_headers_present_in_prompt() -> None:rendered = _render_agent_prompt()- assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:" in rendered+ assert "ASK-vs-EDIT ROUTING:" in renderedassert "CREDENTIAL HANDLING - CRITICAL:" in renderedassert "PARAMETERIZED REQUESTS WITHOUT A SAMPLE VALUE:" in rendereddiff --git a/tests/unit/test_workflow_copilot_diagnostic_intent.py b/tests/unit/test_workflow_copilot_diagnostic_intent.pyindex cedfb7daf..5a16ec652 100644--- a/tests/unit/test_workflow_copilot_diagnostic_intent.py+++ b/tests/unit/test_workflow_copilot_diagnostic_intent.py@@ -1,7 +1,5 @@"""Prompt and tool-description guards for diagnostic copilot turns."""-import re-from skyvern.forge.prompts import prompt_enginefrom skyvern.forge.sdk.copilot.tools import run_blocks_tool, update_and_run_blocks_tool@@ -18,88 +16,94 @@ def _render_agent_prompt() -> str:class TestDiagnosticObservationIntent:- """Pin SKY-9651: observational complaints inspect first instead of edit/run by default."""+ """Pin diagnostic complaint routing to the consolidated ASK-vs-EDIT rule."""- def test_agent_prompt_has_diagnostic_observation_section(self) -> None:+ def test_agent_prompt_uses_consolidated_ask_vs_edit_rule(self) -> None:rendered = _render_agent_prompt()- assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS" in rendered- assert "inspect-and-clarify" in rendered- assert "Do NOT call `update_and_run_blocks`" in rendered- assert "do NOT call `run_blocks_and_collect_debug`" in rendered+ assert "ASK-vs-EDIT ROUTING:" in rendered+ assert "Use one decision tree for the latest turn" in rendered+ assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS" not in rendered+ assert "Explicit edit/debug requests remain edit requests" not in rendered- def test_agent_prompt_names_failed_repro_shapes(self) -> None:+ def test_agent_prompt_routes_diagnostic_followups_to_edit_when_context_resolves(self) -> None:rendered = _render_agent_prompt()- for phrase in (- "missing newly added workflow block",- "not following a configured search step",- "whether a named block is present",- "runtime error condition did not trigger",- "missing or unknown block labels",- ):- assert phrase in rendered+ assert (+ "This includes diagnostic symptom follow-ups on the same workflow after an explicit edit goal" in rendered+ )+ assert "diagnostic-after-edit - prior turn asked to consolidate login blocks" in rendered+ assert "connect the existing block chain and call `update_and_run_blocks` once" in rendered- def test_agent_prompt_preserves_explicit_edit_requests(self) -> None:+ def test_agent_prompt_keeps_unresolved_diagnostics_as_clarification(self) -> None:rendered = _render_agent_prompt()- assert "Explicit edit/debug requests remain edit requests" in rendered- assert "how can I improve" in rendered- assert "what would fix" in rendered- assert "call `update_and_run_blocks`" in rendered+ assert "when a diagnostic complaint has no prior edit goal" in rendered+ assert "mismatched/anonymized block references cannot be resolved" in rendered+ assert "latest says to add a conditional to a URL without saying when -> ask for the condition" in rendered+ assert "ask naming both missing labels and candidate blocks" in rendered- def test_block_running_tools_defer_diagnostic_complaints(self) -> None:+ def test_agent_prompt_avoids_debug_tools_before_clear_structural_edit(self) -> None:+ rendered = _render_agent_prompt()++ assert "When the workflow structure itself is enough to make the edit" in rendered+ assert (+ "do not use direct browser tools, `get_run_results`, `run_blocks_and_collect_debug`, `update_workflow`"+ in rendered+ )+ assert (+ "Do not copy explanatory chat-history prose, example Jinja placeholders, or diagnostic transcripts"+ in rendered+ )++ def test_block_running_tools_share_consolidated_diagnostic_routing(self) -> None:for tool in (run_blocks_tool, update_and_run_blocks_tool):desc = tool.description # type: ignore[attr-defined]- assert "diagnostic / observational complaint" in desc- assert "inspect-and-clarify" in desc- assert "not the first response" in desc+ assert "diagnostic complaints" in desc+ assert "ASK-vs-EDIT routing" in desc++ run_desc = run_blocks_tool.description # type: ignore[attr-defined]+ assert "no prior edit goal" in run_desc+ assert "use `update_and_run_blocks`" in run_desc+ assert "instead of rerunning unchanged blocks" in run_desc++ update_desc = update_and_run_blocks_tool.description # type: ignore[attr-defined]+ assert "diagnostic follow-up after an explicit edit goal" in update_desc+ assert "update/run once the correction is clear" in update_descclass TestDiagnosticAbsentEntity:- """Pin SKY-9756: diagnostic complaints naming an absent workflow entity must not get a workflow-knowledge-base explanation in isolation."""+ """Pin SKY-9756 absent-entity symptoms to workflow state before docs prose."""- def test_diagnostic_section_requires_entity_presence_check(self) -> None:+ def test_ask_vs_edit_routing_requires_entity_presence_check(self) -> None:rendered = _render_agent_prompt()for phrase in ("block label, workflow parameter key, or `{{ name.output... }}` Jinja reference",- "verify the entity is present in the current workflow YAML",+ "is present in the current workflow YAML","The current workflow is the source of truth",- "Pending build with a plausible block sequence",- "Pending build still waiting on user input",- "No pending build in chat history",- "fall back to the CRITICAL ROLE BOUNDARY docs-inline path",- "Do not fabricate a workflow",+ "chat history contains a plausible pending build/edit",+ "create or correct the real workflow blocks and call `update_and_run_blocks`",+ "respond with `ASK_QUESTION` naming both the absent entity and missing requirement",+ "Only use docs-inline when no pending workflow build/edit remains",):- assert phrase in rendered, f"missing diagnostic absent-entity phrase: {phrase!r}"+ assert phrase in rendered, f"missing absent-entity routing phrase: {phrase!r}"- def test_step_5_spells_out_knowledge_base_does_not_use_kb_acronym(self) -> None:- # The rest of the prompt spells out "WORKFLOW KNOWLEDGE BASE"; the new- # step 5 must match that convention so the LLM has no ambiguity about- # what "KB" would have meant.+ def test_ask_vs_edit_spells_out_knowledge_base_does_not_use_kb_acronym(self) -> None:+ # The rest of the prompt spells out "WORKFLOW KNOWLEDGE BASE"; the+ # routing rule should not introduce a new "KB" abbreviation.rendered = _render_agent_prompt()- section_marker = "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:"- end_marker = "Explicit edit/debug requests remain edit requests"+ section_marker = "ASK-vs-EDIT ROUTING:"+ end_marker = "WORKFLOW-FIRST EXECUTION PATH"start = rendered.index(section_marker)end = rendered.index(end_marker, start)section = rendered[start:end]- assert "workflow-knowledge-base explanation" in sectionassert "workflow-knowledge-base prose" in section- assert not re.search(r"\bKB\b", section), "step 5 must spell out 'knowledge base', not abbreviate to 'KB'"+ assert "KB" not in section- def test_docs_inline_disambiguator_defers_to_diagnostic_step_5(self) -> None:+ def test_docs_inline_disambiguator_defers_to_ask_vs_edit_routing(self) -> None:rendered = _render_agent_prompt()assert "Docs-inline does NOT apply to a runtime symptom" in renderedassert "a block label, workflow parameter key, or `{{ name.output... }}` Jinja reference" in rendered- assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS step 5" in rendered-- def test_diagnostic_step_5_is_actually_numbered_in_section(self) -> None:- rendered = _render_agent_prompt()- section_marker = "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:"- end_marker = "Explicit edit/debug requests remain edit requests"- start = rendered.index(section_marker)- end = rendered.index(end_marker, start)- section = rendered[start:end]- assert "\n5. " in section, "step 5 missing from DIAGNOSTIC / OBSERVATIONAL COMPLAINTS numbered list"+ assert "use the ASK-vs-EDIT routing below" in rendered