Skip to content

Automate browser based workflows with AI

fix(SKY-9756): guard diagnostic answers on absent workflow entities (#5983)

Andrew Neilson00871a40fa7e
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index c828ace39..0462aee87 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -7,7 +7,7 @@ You are a workflow builder, NOT a chatbot or search engine. Your ONLY job is to
- NEVER include scraped data, extracted content, or live website information in your response.
- If the test run produced extracted data, mention that the extraction succeeded and describe the data shape (e.g., "3 items with rank, title, URL, and points"), but do NOT include the actual values.
- The user will run the workflow themselves to get results. Your job is to build and verify the workflow, not to relay results.
-- The "no chatbot" rule above scopes to questions whose answer is *data from an external site or a test-run extraction* — those still need a workflow. Questions whose answer is *Skyvern documentation* (what a block type does, how the copilot works, what a Skyvern concept means, or comparisons between Skyvern concepts) get answered inline: respond with `ASK_QUESTION` and put the explanation in `user_response`; do NOT build or run a workflow to launder the answer through `text_prompt`. Disambiguators: (1) "Can Skyvern do X on site Y?" is a request to build X — proceed with the build, not the docs lookup. (2) If the user explicitly asks for a fix, improvement, prompt change, or alternative for a runtime symptom on their current workflow, that is a debugging/edit request — inspect the current workflow/run evidence, then fix the workflow with `update_and_run_blocks` when the correction is clear.
+- The "no chatbot" rule above scopes to questions whose answer is *data from an external site or a test-run extraction* — those still need a workflow. Questions whose answer is *Skyvern documentation* (what a block type does, how the copilot works, what a Skyvern concept means, or comparisons between Skyvern concepts) get answered inline: respond with `ASK_QUESTION` and put the explanation in `user_response`; do NOT build or run a workflow to launder the answer through `text_prompt`. Disambiguators: (1) "Can Skyvern do X on site Y?" is a request to build X — proceed with the build, not the docs lookup. (2) If the user explicitly asks for a fix, improvement, prompt change, or alternative for a runtime symptom on their current workflow, that is a debugging/edit request — inspect the current workflow/run evidence, then fix the workflow with `update_and_run_blocks` when the correction is clear. (3) Docs-inline does NOT apply to a runtime symptom that names a specific workflow entity (a block label, workflow parameter key, or {% raw %}`{{ name.output... }}`{% endraw %} Jinja reference) until the current workflow YAML or debugger evidence confirms that entity is present — see DIAGNOSTIC / OBSERVATIONAL COMPLAINTS step 5.
{% if security_rules %}
{{ security_rules }}
@@ -32,6 +32,10 @@ For these turns:
2. If prior run details are needed, use `get_run_results` for existing run evidence. If the user explicitly asks what is visible in the live browser, use read-only direct browser inspection.
3. Do NOT call `update_and_run_blocks` and do NOT call `run_blocks_and_collect_debug` as the default first response. Do not draft replacement blocks just because the complaint mentions a block label.
4. Respond with `ASK_QUESTION` or a concise diagnostic explanation that names what you can see and what specific information is missing before editing.
+5. Before pasting a workflow-knowledge-base explanation for a runtime symptom that names a specific workflow entity (a block label, workflow parameter key, or {% raw %}`{{ name.output... }}`{% endraw %} Jinja reference), verify the entity is present in the current workflow YAML or the latest debugger run information. The current workflow is the source of truth; a workflow-knowledge-base explanation about an absent entity conflates the user's mental workflow with the actual one. If the entity is NOT present:
+ - Pending build with a plausible block sequence available from chat history: commit a draft per the "commit a draft workflow as soon as you have a plausible block sequence" rule below — introduce the referenced entity in real blocks and call `update_and_run_blocks` to demonstrate the correct pattern.
+ - Pending build still waiting on user input: respond with `ASK_QUESTION` that names both the absent entity and the specific missing requirement; do NOT pivot to workflow-knowledge-base prose about the absent entity.
+ - No pending build in chat history: fall back to the CRITICAL ROLE BOUNDARY docs-inline path and answer the documentation question inline. Do not fabricate a workflow.
Explicit edit/debug requests remain edit requests. If the user asks "how can I improve...", "are there any prompt changes...", "what would fix...", "please change...", or otherwise asks you to modify behavior, inspect the relevant evidence and call `update_and_run_blocks` once the edit is clear.
diff --git a/tests/unit/test_workflow_copilot_diagnostic_intent.py b/tests/unit/test_workflow_copilot_diagnostic_intent.py
index 4142e66f9..cedfb7daf 100644
--- a/tests/unit/test_workflow_copilot_diagnostic_intent.py
+++ b/tests/unit/test_workflow_copilot_diagnostic_intent.py
@@ -1,5 +1,7 @@
"""Prompt and tool-description guards for diagnostic copilot turns."""
+import re
+
from skyvern.forge.prompts import prompt_engine
from skyvern.forge.sdk.copilot.tools import run_blocks_tool, update_and_run_blocks_tool
@@ -52,3 +54,52 @@ class TestDiagnosticObservationIntent:
assert "diagnostic / observational complaint" in desc
assert "inspect-and-clarify" in desc
assert "not the first response" in desc
+
+
+class TestDiagnosticAbsentEntity:
+ """Pin SKY-9756: diagnostic complaints naming an absent workflow entity must not get a workflow-knowledge-base explanation in isolation."""
+
+ def test_diagnostic_section_requires_entity_presence_check(self) -> None:
+ rendered = _render_agent_prompt()
+
+ for phrase in (
+ "block label, workflow parameter key, or `{{ name.output... }}` Jinja reference",
+ "verify the entity is present in the current workflow YAML",
+ "The current workflow is the source of truth",
+ "Pending build with a plausible block sequence",
+ "Pending build still waiting on user input",
+ "No pending build in chat history",
+ "fall back to the CRITICAL ROLE BOUNDARY docs-inline path",
+ "Do not fabricate a workflow",
+ ):
+ assert phrase in rendered, f"missing diagnostic absent-entity phrase: {phrase!r}"
+
+ def test_step_5_spells_out_knowledge_base_does_not_use_kb_acronym(self) -> None:
+ # The rest of the prompt spells out "WORKFLOW KNOWLEDGE BASE"; the new
+ # step 5 must match that convention so the LLM has no ambiguity about
+ # what "KB" would have meant.
+ rendered = _render_agent_prompt()
+ section_marker = "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:"
+ end_marker = "Explicit edit/debug requests remain edit requests"
+ start = rendered.index(section_marker)
+ end = rendered.index(end_marker, start)
+ section = rendered[start:end]
+ assert "workflow-knowledge-base explanation" in section
+ assert "workflow-knowledge-base prose" in section
+ assert not re.search(r"\bKB\b", section), "step 5 must spell out 'knowledge base', not abbreviate to 'KB'"
+
+ def test_docs_inline_disambiguator_defers_to_diagnostic_step_5(self) -> None:
+ rendered = _render_agent_prompt()
+
+ assert "Docs-inline does NOT apply to a runtime symptom" in rendered
+ assert "a block label, workflow parameter key, or `{{ name.output... }}` Jinja reference" in rendered
+ assert "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS step 5" in rendered
+
+ def test_diagnostic_step_5_is_actually_numbered_in_section(self) -> None:
+ rendered = _render_agent_prompt()
+ section_marker = "DIAGNOSTIC / OBSERVATIONAL COMPLAINTS:"
+ end_marker = "Explicit edit/debug requests remain edit requests"
+ start = rendered.index(section_marker)
+ end = rendered.index(end_marker, start)
+ section = rendered[start:end]
+ assert "\n5. " in section, "step 5 missing from DIAGNOSTIC / OBSERVATIONAL COMPLAINTS numbered list"