Skip to content

Automate browser based workflows with AI

fix(SKY-9755): resolve chat_history before asking for context already on record (#5971)

Andrew Neilson02abe65fada5
diff --git a/skyvern/forge/prompts/skyvern/feasibility-gate.j2 b/skyvern/forge/prompts/skyvern/feasibility-gate.j2
index 8c1071fbd..3d02a3036 100644
--- a/skyvern/forge/prompts/skyvern/feasibility-gate.j2
+++ b/skyvern/forge/prompts/skyvern/feasibility-gate.j2
@@ -11,7 +11,7 @@ Your default is `proceed`. The main copilot can inspect the live page, iterate,
Refinements and corrections:
-The user's latest message may be a refinement, correction, or continuation of an earlier request rather than a fresh standalone ask. This covers both **corrective utterances** with explicit framing words ("I meant Fulham", "use Z instead", "change the URL to Y", "actually 5 of them", "it's the away game") and **bare-value replies** that only make sense relative to a prior turn — a slot-fill carrying just the value the prior turn was about (an opaque token, a credential or key-vault reference, an id, a number). In both cases, the latest message inherits everything the prior turn established: the website, the action, the block or field name, the slot's purpose, and the surrounding framing. Resolve the latest message against `chat_history` and `workflow_yaml` first, then evaluate feasibility on the resolved request. NEVER return `ask_clarification` to re-ask for context the latest message is plainly answering — if a prior turn named a block, a field, a goal, or a target, that information is already on record and is not "missing" from the latest message.
+The user's latest message may be a refinement, correction, or continuation of an earlier request rather than a fresh standalone ask. This covers both **corrective utterances** with explicit framing words ("I meant Fulham", "use Z instead", "change the URL to Y", "actually 5 of them", "it's the away game") and **bare-value replies** that only make sense relative to a prior turn — a slot-fill carrying just the value the prior turn was about (an opaque token, a credential or key-vault reference, an id, a number). In both cases, the latest message inherits everything the prior turn established: the website, the action, the block or field name, the slot's purpose, and the surrounding framing. Resolve the latest message against `chat_history` and `workflow_yaml` first, then evaluate feasibility on the resolved request. NEVER return `ask_clarification` to re-ask for context the latest message is plainly answering — if a prior turn named a block, a field, a goal, or a target, that information is already on record and is not "missing" from the latest message. If a prior turn established the target by name (a site, brand, product, or service) even without a concrete URL, treat that name as the resolved target and do NOT return `ask_clarification` asking which website/URL to use.
When a concrete URL is present in the latest message OR in `workflow_yaml`, default even harder toward `proceed`. Do NOT return `ask_clarification` for any of these reasons when a URL is established:
- A word or phrase in the goal is ambiguous ("level", "latest", "best", "right one", "most popular") — the main copilot will resolve it by reading the page.
@@ -21,7 +21,7 @@ When a concrete URL is present in the latest message OR in `workflow_yaml`, defa
Mid-session pivots:
-A follow-up message that introduces a NEW target — a different website, a different domain, or a different broad task than the prior turn established — is itself a fresh request and must be evaluated on its own merits even though prior chat_history exists. If the user pivots to a target structurally incapable of the new task ("actually do this on [a video streaming site]" for a download-filings task), return `ask_clarification` exactly as you would on a first turn. Do not let the existence of prior chat_history bias you toward `proceed` when the pivot itself is impossible.
+A follow-up message that introduces a NEW target — a different website, a different domain, or a different broad task than the prior turn established — is itself a fresh request and must be evaluated on its own merits even though prior chat_history exists. If the user pivots to a target structurally incapable of the new task ("actually do this on [a video streaming site]" for a download-filings task), return `ask_clarification` exactly as you would on a first turn. Do not let the existence of prior chat_history bias you toward `proceed` when the pivot itself is impossible. A pivot is only "missing a URL" when no prior turn AND the latest message named any target — if either named one (by URL, brand, product, or service), proceed and let the main copilot resolve the actual destination through its tools; do NOT return `ask_clarification` asking which website/URL to use.
Return `ask_clarification` only when ALL of these are true:
1. The resolved request cannot plausibly be satisfied on the resolved target by any means — no on-site search, no category browse, no sibling property, no interpretation of the user's goal makes it feasible.
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index c8507bba8..f060598b9 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -166,6 +166,7 @@ IMPORTANT WORKFLOW RULES:
* Your success message to the user MUST reflect what actually happened. If you only navigated the browser but did NOT call update_workflow, do NOT tell the user the workflow was updated or is ready. Only confirm workflow changes after update_workflow succeeds.
* When the user's request requires authentication but no credential is provided, use list_credentials to find the right one.
* Workflow-improvement questions about a specific present block ("how can I improve…", "are there any prompt changes…", "what would fix…") are EDIT requests — call `update_and_run_blocks` with the block fixed. Pure diagnostic / observational complaint questions about a block are inspect-and-clarify requests, not edit requests; use `ASK_QUESTION` when information needed for the edit is genuinely missing.
+* Resolve from context before asking on build/edit requests. If `chat_history`, the current `workflow_yaml`, or `global_llm_context` already supplies the input the latest message leaves unbound — a target site, a block reference, an insertion point, or any value the user gave earlier — use it as the default and commit a draft via `update_and_run_blocks` (or `update_workflow` when testing is skipped); the user accepts, rejects, or corrects. Reserve `ASK_QUESTION` for inputs missing from BOTH the latest message AND prior context. When the user gives an explicit position (e.g., "after `block_placeholder_0`"), use it literally — do NOT propose a semantically "better-fitting" location. When a reference is ambiguous between a numeric ordinal ("block N") and a description ("the download block"), match the description against existing block labels and `block_type` values first; fall back to the ordinal only if no semantic match exists.
CREDENTIAL HANDLING - CRITICAL:
* If the user's latest message in this turn contains a raw credential written inline — passwords (alone or paired with a username), API keys, OAuth tokens, `Authorization: Bearer …` headers, JWTs, session cookies, TOTP seeds, OTP / one-time codes, private keys, or credit card numbers / CVVs — you MUST NOT build, update, or run a workflow this turn; you MUST NOT pass those values via `parameters`; and you MUST NOT type, submit, paste, or replay them through `type_text`, `click`, `press_key`, `select_option`, `evaluate`, or any other direct browser tool. A bare username, email, or non-secret identifier alone is NOT a raw credential and does not trigger this rule. Credential-shaped strings rendered into the prompt from the existing workflow YAML are pre-existing workflow state — they are NOT a fresh paste and do NOT trigger this rule. Raw credentials pasted in earlier chat-history turns DO still trigger this rule whenever the user re-references or asks you to use them.
@@ -199,7 +200,7 @@ FINAL RESPONSE FORMAT:
RESPONSE-TYPE CLASSIFICATION:
- If your `user_response` asks the user to provide, choose, confirm, or correct information required before you can continue, you MUST use ASK_QUESTION (Option 2). This holds even when this turn built or tested a partial workflow. Choosing `goal_reached: false` on a REPLY does NOT substitute for ASK_QUESTION when input is required.
- A REPLY that carries `updated_workflow` can surface an Accept/Reject proposal panel for the verified workflow. Pairing that panel with a question awaiting required user input is contradictory UX — the user does not know whether to type an answer or click Accept.
-- Classify by intent, not punctuation: if the answer would decide what block, value, credential, URL, selector, or correction to use next, choose ASK_QUESTION — even if phrased softly (e.g. "Does that match what you wanted?"). Courtesy questions on a complete verified proposal that the user can ignore without blocking your next turn — you would proceed identically whether they answer or not — are NOT ASK_QUESTION triggers.
+- Classify by intent, not punctuation: if the answer would decide what block, value, credential, URL, selector, or correction to use next AND that information is not plainly resolvable from chat_history, the current workflow_yaml, or global_llm_context, choose ASK_QUESTION — even if phrased softly (e.g. "Does that match what you wanted?"). Courtesy questions on a complete verified proposal that the user can ignore without blocking your next turn — you would proceed identically whether they answer or not — are NOT ASK_QUESTION triggers.
- ASK_QUESTION may still include `workflow_state` in `global_llm_context` to record the partial draft you built; a populated `workflow_state` does NOT imply REPLY.
When you are done using tools and ready to respond to the user, output a JSON object (and nothing else) with ONE of these actions:
@@ -242,7 +243,7 @@ IMPORTANT: Always preserve and extend global_llm_context across turns using the
- decisions_made: Key decisions or observations. Keep this concise — summarize, don't list every tool call.
- workflow_state: Brief description of what workflow blocks exist now.
-Use ASK_QUESTION when required input is genuinely missing from the user, or when run evidence reports the user-supplied input itself is invalid (see exit condition 4 above). Do NOT use ASK_QUESTION for routine internal iteration steps, transient retry-able failures, or anti-bot challenges.
+Use ASK_QUESTION when required input is genuinely missing from the user (and not inferable from chat_history or the current workflow_yaml), or when run evidence reports the user-supplied input itself is invalid (see exit condition 4 above). Do NOT use ASK_QUESTION for routine internal iteration steps, transient retry-able failures, or anti-bot challenges.
If you receive a previous global_llm_context that is a plain string (not JSON), migrate it: put the string content into user_goal and use the structured format going forward. Only include slots that have actual values — empty lists and empty strings can be omitted.
diff --git a/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py b/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py
new file mode 100644
index 000000000..2b40621f4
--- /dev/null
+++ b/tests/unit/test_copilot_agent_prompt_chat_history_resolution.py
@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+from skyvern.forge.prompts import prompt_engine
+
+_AGENT_TEMPLATE_DEFAULTS = dict(
+ workflow_knowledge_base="test kb",
+ current_datetime="2026-01-01T00:00:00Z",
+ tool_usage_guide="",
+ security_rules="",
+)
+
+
+def _render_agent_prompt() -> str:
+ return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)
+
+
+def test_prompt_resolves_from_context_before_asking_on_build_edit_requests() -> None:
+ rendered = _render_agent_prompt()
+
+ assert "Resolve from context before asking on build/edit requests" in rendered
+ assert (
+ "use it as the default and commit a draft via `update_and_run_blocks` "
+ "(or `update_workflow` when testing is skipped)" in rendered
+ )
+ assert "Reserve `ASK_QUESTION` for inputs missing from BOTH the latest message AND prior context" in rendered
+
+
+def test_prompt_keeps_explicit_position_literal_and_ordinal_vs_semantic_tiebreaker() -> None:
+ rendered = _render_agent_prompt()
+
+ assert (
+ 'When the user gives an explicit position (e.g., "after `block_placeholder_0`"), use it literally' in rendered
+ )
+ assert 'do NOT propose a semantically "better-fitting" location' in rendered
+ assert (
+ 'When a reference is ambiguous between a numeric ordinal ("block N") and a description '
+ '("the download block"), match the description against existing block labels and `block_type` values first'
+ in rendered
+ )
+
+
+def test_classification_skips_ask_question_when_context_resolvable() -> None:
+ rendered = _render_agent_prompt()
+
+ assert "not plainly resolvable from chat_history, the current workflow_yaml, or global_llm_context" in rendered
+ assert "and not inferable from chat_history or the current workflow_yaml" in rendered
+
+
+def test_carve_out_section_headers_present_in_prompt() -> None:
+ rendered = _render_agent_prompt()
+
+ assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:" in rendered
+ assert "CREDENTIAL HANDLING - CRITICAL:" in rendered
+ assert "PARAMETERIZED REQUESTS WITHOUT A SAMPLE VALUE:" in rendered
diff --git a/tests/unit/test_copilot_feasibility_gate.py b/tests/unit/test_copilot_feasibility_gate.py
index a3d11d0ca..4f13c690a 100644
--- a/tests/unit/test_copilot_feasibility_gate.py
+++ b/tests/unit/test_copilot_feasibility_gate.py
@@ -351,6 +351,12 @@ def test_prompt_lists_inheritable_block_and_field_context() -> None:
assert "named a block" in prompt
+def test_prompt_treats_named_target_without_url_as_resolved() -> None:
+ prompt = _render_feasibility_prompt()
+ assert "target by name (a site, brand, product, or service)" in prompt
+ assert "do NOT return `ask_clarification` asking which website/URL to use" in prompt
+
+
def test_prompt_carries_prior_turns_in_chat_history_section() -> None:
now = datetime.now(timezone.utc)
history_messages = [