Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
fix(SKY-9755): resolve chat_history before asking for context already on record (#5971)
Andrew Neilson02abe65fada5
diff --git a/skyvern/forge/prompts/skyvern/feasibility-gate.j2 b/skyvern/forge/prompts/skyvern/feasibility-gate.j2index 8c1071fbd..3d02a3036 100644--- a/skyvern/forge/prompts/skyvern/feasibility-gate.j2+++ b/skyvern/forge/prompts/skyvern/feasibility-gate.j2@@ -11,7 +11,7 @@ Your default is `proceed`. The main copilot can inspect the live page, iterate,Refinements and corrections:-The user's latest message may be a refinement, correction, or continuation of an earlier request rather than a fresh standalone ask. This covers both **corrective utterances** with explicit framing words ("I meant Fulham", "use Z instead", "change the URL to Y", "actually 5 of them", "it's the away game") and **bare-value replies** that only make sense relative to a prior turn — a slot-fill carrying just the value the prior turn was about (an opaque token, a credential or key-vault reference, an id, a number). In both cases, the latest message inherits everything the prior turn established: the website, the action, the block or field name, the slot's purpose, and the surrounding framing. Resolve the latest message against `chat_history` and `workflow_yaml` first, then evaluate feasibility on the resolved request. NEVER return `ask_clarification` to re-ask for context the latest message is plainly answering — if a prior turn named a block, a field, a goal, or a target, that information is already on record and is not "missing" from the latest message.+The user's latest message may be a refinement, correction, or continuation of an earlier request rather than a fresh standalone ask. This covers both **corrective utterances** with explicit framing words ("I meant Fulham", "use Z instead", "change the URL to Y", "actually 5 of them", "it's the away game") and **bare-value replies** that only make sense relative to a prior turn — a slot-fill carrying just the value the prior turn was about (an opaque token, a credential or key-vault reference, an id, a number). In both cases, the latest message inherits everything the prior turn established: the website, the action, the block or field name, the slot's purpose, and the surrounding framing. Resolve the latest message against `chat_history` and `workflow_yaml` first, then evaluate feasibility on the resolved request. NEVER return `ask_clarification` to re-ask for context the latest message is plainly answering — if a prior turn named a block, a field, a goal, or a target, that information is already on record and is not "missing" from the latest message. If a prior turn established the target by name (a site, brand, product, or service) even without a concrete URL, treat that name as the resolved target and do NOT return `ask_clarification` asking which website/URL to use.When a concrete URL is present in the latest message OR in `workflow_yaml`, default even harder toward `proceed`. Do NOT return `ask_clarification` for any of these reasons when a URL is established:- A word or phrase in the goal is ambiguous ("level", "latest", "best", "right one", "most popular") — the main copilot will resolve it by reading the page.@@ -21,7 +21,7 @@ When a concrete URL is present in the latest message OR in `workflow_yaml`, defaMid-session pivots:-A follow-up message that introduces a NEW target — a different website, a different domain, or a different broad task than the prior turn established — is itself a fresh request and must be evaluated on its own merits even though prior chat_history exists. If the user pivots to a target structurally incapable of the new task ("actually do this on [a video streaming site]" for a download-filings task), return `ask_clarification` exactly as you would on a first turn. Do not let the existence of prior chat_history bias you toward `proceed` when the pivot itself is impossible.+A follow-up message that introduces a NEW target — a different website, a different domain, or a different broad task than the prior turn established — is itself a fresh request and must be evaluated on its own merits even though prior chat_history exists. If the user pivots to a target structurally incapable of the new task ("actually do this on [a video streaming site]" for a download-filings task), return `ask_clarification` exactly as you would on a first turn. Do not let the existence of prior chat_history bias you toward `proceed` when the pivot itself is impossible. A pivot is only "missing a URL" when no prior turn AND the latest message named any target — if either named one (by URL, brand, product, or service), proceed and let the main copilot resolve the actual destination through its tools; do NOT return `ask_clarification` asking which website/URL to use.Return `ask_clarification` only when ALL of these are true:1. The resolved request cannot plausibly be satisfied on the resolved target by any means — no on-site search, no category browse, no sibling property, no interpretation of the user's goal makes it feasible.diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index c8507bba8..f060598b9 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -166,6 +166,7 @@ IMPORTANT WORKFLOW RULES:* Your success message to the user MUST reflect what actually happened. If you only navigated the browser but did NOT call update_workflow, do NOT tell the user the workflow was updated or is ready. Only confirm workflow changes after update_workflow succeeds.* When the user's request requires authentication but no credential is provided, use list_credentials to find the right one.* Workflow-improvement questions about a specific present block ("how can I improve…", "are there any prompt changes…", "what would fix…") are EDIT requests — call `update_and_run_blocks` with the block fixed. Pure diagnostic / observational complaint questions about a block are inspect-and-clarify requests, not edit requests; use `ASK_QUESTION` when information needed for the edit is genuinely missing.+* Resolve from context before asking on build/edit requests. If `chat_history`, the current `workflow_yaml`, or `global_llm_context` already supplies the input the latest message leaves unbound — a target site, a block reference, an insertion point, or any value the user gave earlier — use it as the default and commit a draft via `update_and_run_blocks` (or `update_workflow` when testing is skipped); the user accepts, rejects, or corrects. Reserve `ASK_QUESTION` for inputs missing from BOTH the latest message AND prior context. When the user gives an explicit position (e.g., "after `block_placeholder_0`"), use it literally — do NOT propose a semantically "better-fitting" location. When a reference is ambiguous between a numeric ordinal ("block N") and a description ("the download block"), match the description against existing block labels and `block_type` values first; fall back to the ordinal only if no semantic match exists.CREDENTIAL HANDLING - CRITICAL:* If the user's latest message in this turn contains a raw credential written inline — passwords (alone or paired with a username), API keys, OAuth tokens, `Authorization: Bearer …` headers, JWTs, session cookies, TOTP seeds, OTP / one-time codes, private keys, or credit card numbers / CVVs — you MUST NOT build, update, or run a workflow this turn; you MUST NOT pass those values via `parameters`; and you MUST NOT type, submit, paste, or replay them through `type_text`, `click`, `press_key`, `select_option`, `evaluate`, or any other direct browser tool. A bare username, email, or non-secret identifier alone is NOT a raw credential and does not trigger this rule. Credential-shaped strings rendered into the prompt from the existing workflow YAML are pre-existing workflow state — they are NOT a fresh paste and do NOT trigger this rule. Raw credentials pasted in earlier chat-history turns DO still trigger this rule whenever the user re-references or asks you to use them.@@ -199,7 +200,7 @@ FINAL RESPONSE FORMAT:RESPONSE-TYPE CLASSIFICATION:- If your `user_response` asks the user to provide, choose, confirm, or correct information required before you can continue, you MUST use ASK_QUESTION (Option 2). This holds even when this turn built or tested a partial workflow. Choosing `goal_reached: false` on a REPLY does NOT substitute for ASK_QUESTION when input is required.- A REPLY that carries `updated_workflow` can surface an Accept/Reject proposal panel for the verified workflow. Pairing that panel with a question awaiting required user input is contradictory UX — the user does not know whether to type an answer or click Accept.-- Classify by intent, not punctuation: if the answer would decide what block, value, credential, URL, selector, or correction to use next, choose ASK_QUESTION — even if phrased softly (e.g. "Does that match what you wanted?"). Courtesy questions on a complete verified proposal that the user can ignore without blocking your next turn — you would proceed identically whether they answer or not — are NOT ASK_QUESTION triggers.+- Classify by intent, not punctuation: if the answer would decide what block, value, credential, URL, selector, or correction to use next AND that information is not plainly resolvable from chat_history, the current workflow_yaml, or global_llm_context, choose ASK_QUESTION — even if phrased softly (e.g. "Does that match what you wanted?"). Courtesy questions on a complete verified proposal that the user can ignore without blocking your next turn — you would proceed identically whether they answer or not — are NOT ASK_QUESTION triggers.- ASK_QUESTION may still include `workflow_state` in `global_llm_context` to record the partial draft you built; a populated `workflow_state` does NOT imply REPLY.When you are done using tools and ready to respond to the user, output a JSON object (and nothing else) with ONE of these actions:@@ -242,7 +243,7 @@ IMPORTANT: Always preserve and extend global_llm_context across turns using the- decisions_made: Key decisions or observations. Keep this concise — summarize, don't list every tool call.- workflow_state: Brief description of what workflow blocks exist now.-Use ASK_QUESTION when required input is genuinely missing from the user, or when run evidence reports the user-supplied input itself is invalid (see exit condition 4 above). Do NOT use ASK_QUESTION for routine internal iteration steps, transient retry-able failures, or anti-bot challenges.+Use ASK_QUESTION when required input is genuinely missing from the user (and not inferable from chat_history or the current workflow_yaml), or when run evidence reports the user-supplied input itself is invalid (see exit condition 4 above). Do NOT use ASK_QUESTION for routine internal iteration steps, transient retry-able failures, or anti-bot challenges.If you receive a previous global_llm_context that is a plain string (not JSON), migrate it: put the string content into user_goal and use the structured format going forward. Only include slots that have actual values — empty lists and empty strings can be omitted.diff --git a/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py b/tests/unit/test_copilot_agent_prompt_chat_history_resolution.pynew file mode 100644index 000000000..2b40621f4--- /dev/null+++ b/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py@@ -0,0 +1,54 @@+from __future__ import annotations++from skyvern.forge.prompts import prompt_engine++_AGENT_TEMPLATE_DEFAULTS = dict(+ workflow_knowledge_base="test kb",+ current_datetime="2026-01-01T00:00:00Z",+ tool_usage_guide="",+ security_rules="",+)+++def _render_agent_prompt() -> str:+ return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)+++def test_prompt_resolves_from_context_before_asking_on_build_edit_requests() -> None:+ rendered = _render_agent_prompt()++ assert "Resolve from context before asking on build/edit requests" in rendered+ assert (+ "use it as the default and commit a draft via `update_and_run_blocks` "+ "(or `update_workflow` when testing is skipped)" in rendered+ )+ assert "Reserve `ASK_QUESTION` for inputs missing from BOTH the latest message AND prior context" in rendered+++def test_prompt_keeps_explicit_position_literal_and_ordinal_vs_semantic_tiebreaker() -> None:+ rendered = _render_agent_prompt()++ assert (+ 'When the user gives an explicit position (e.g., "after `block_placeholder_0`"), use it literally' in rendered+ )+ assert 'do NOT propose a semantically "better-fitting" location' in rendered+ assert (+ 'When a reference is ambiguous between a numeric ordinal ("block N") and a description '+ '("the download block"), match the description against existing block labels and `block_type` values first'+ in rendered+ )+++def test_classification_skips_ask_question_when_context_resolvable() -> None:+ rendered = _render_agent_prompt()++ assert "not plainly resolvable from chat_history, the current workflow_yaml, or global_llm_context" in rendered+ assert "and not inferable from chat_history or the current workflow_yaml" in rendered+++def test_carve_out_section_headers_present_in_prompt() -> None:+ rendered = _render_agent_prompt()++ assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:" in rendered+ assert "CREDENTIAL HANDLING - CRITICAL:" in rendered+ assert "PARAMETERIZED REQUESTS WITHOUT A SAMPLE VALUE:" in rendereddiff --git a/tests/unit/test_copilot_feasibility_gate.py b/tests/unit/test_copilot_feasibility_gate.pyindex a3d11d0ca..4f13c690a 100644--- a/tests/unit/test_copilot_feasibility_gate.py+++ b/tests/unit/test_copilot_feasibility_gate.py@@ -351,6 +351,12 @@ def test_prompt_lists_inheritable_block_and_field_context() -> None:assert "named a block" in prompt+def test_prompt_treats_named_target_without_url_as_resolved() -> None:+ prompt = _render_feasibility_prompt()+ assert "target by name (a site, brand, product, or service)" in prompt+ assert "do NOT return `ask_clarification` asking which website/URL to use" in prompt++def test_prompt_carries_prior_turns_in_chat_history_section() -> None:now = datetime.now(timezone.utc)history_messages = [