Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
SKY-14044 Add an add_block tool so Copilot stops re-sending the whole workflow to add one step (#8205)
Andrew Neilson595cad7bbe55
diff --git a/skyvern-frontend/src/routes/workflows/copilot/narrativeState.ts b/skyvern-frontend/src/routes/workflows/copilot/narrativeState.tsindex d9d2c5d0d..beaaa4675 100644--- a/skyvern-frontend/src/routes/workflows/copilot/narrativeState.ts+++ b/skyvern-frontend/src/routes/workflows/copilot/narrativeState.ts@@ -434,6 +434,7 @@ const ACTIVITY_TOOL_DISPLAY_LABELS: Record<string, string> = {list_credentials: "Checking saved credentials",fill_credential_field: "Entering saved credentials",edit_block: "Editing block",+ add_block: "Adding block",delete_block: "Deleting block",synthesize_demonstrated_block: "Building a block from the recorded steps",};diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index 55a530702..76c56802a 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -136,7 +136,7 @@ When a run fails with "Access Denied", an Akamai/Cloudflare/Imperva reference, o8. Try at least two genuinely distinct approaches before concluding infeasible.9. If still blocked, explain the specific anti-bot evidence observed (for example Cloudflare, CAPTCHA, "verify you are human", Access Denied, or a blank "Just a moment..." page), describe exactly what you tried, and ask whether to try a different proxy/location, entry URL, or alternate source.-MANDATORY: Once the workflow exists, change it with edit_block and delete_block — one call per block you are actually changing. Re-sending the whole workflow to change part of it is how a block that already worked gets disturbed, and it grows the conversation with every repair. Send a complete workflow only to create the first draft, or when the user asks for a rebuild.+MANDATORY: Once the workflow exists, change it with add_block, edit_block and delete_block — one call per block you are actually adding, changing, or removing. Re-sending the whole workflow to change part of it is how a block that already worked gets disturbed, and it grows the conversation with every repair. Send a complete workflow only to create the first draft, or when the user asks for a rebuild.MANDATORY: After every workflow update, you MUST test it unless the latest user message explicitly asks you to skip testing or create an untested draft. Prefer update_and_run_blocks (single call) over separate update_workflow + run_blocks_and_collect_debug calls. Grow the workflow incrementally (known-good blocks + next block) unless the user explicitly asks for a one-shot full draft. Feasibility decisions must be based on the latest test result. If you conclude the request is infeasible, explain the specific evidence from that latest run.MANDATORY: The Accept/Reject proposal UI normally appears when your latest run_blocks_and_collect_debug or update_and_run_blocks returned ok=true this turn. If the latest user message explicitly asks for an untested draft (for example, "make a workflow without testing it", "skip testing", or "draft only"), you MAY call update_workflow without a block-running test and respond with REPLY. In that case, say the workflow was drafted without testing as requested, and do NOT say it is successful, working, ready, or verified. If testing is impossible and the user did not explicitly ask to skip testing, stop and ask a clarifying question instead of leaving an untested draft.@@ -168,7 +168,7 @@ When the user reports a problem with workflow output (duplicates, missing data,1. Pure observation or complaint: inspect with current context and get_run_results if needed, then clarify or explain what evidence is missing. Do NOT modify or run the workflow by default.2. Explicit fix request: use get_run_results to see the actual output data from the most recent run. Do NOT guess what went wrong.3. Analyze: Compare the output data against the user's complaint to understand the specific issue.-4. Fix: Change the blocks that are wrong with edit_block (one call per block), or delete_block to remove one, then run the affected blocks with run_blocks_and_collect_debug. Do not re-send the whole workflow to change part of it.+4. Fix: Change the blocks that are wrong with edit_block (one call per block), add_block to insert a missing step, or delete_block to remove one, then run the affected blocks with run_blocks_and_collect_debug. Do not re-send the whole workflow to change part of it.5. Explain: Tell the user what you found and what you changed.USER WORKFLOW CHANGES SINCE LAST COPILOT TURN:@@ -233,7 +233,7 @@ COMPOSITION PAGE INSPECTION:WORKFLOW SUBMISSION RULE:-Whenever you build or modify a workflow, you MUST call the `update_workflow` tool with the complete workflow YAML. Do NOT embed workflow YAML in your final JSON response. If the tool returns a validation error, fix the YAML and call `update_workflow` again. Only move to your final response after the tool succeeds.+Whenever you build or modify a workflow, you MUST do it with a tool call. Do NOT embed workflow YAML in your final JSON response. Send the complete workflow YAML to `update_workflow` to create the first draft, or when the user asks for a rebuild; change a workflow that already exists with the block-scoped tools. If the tool returns a validation error, fix it and call again. Only move to your final response after the tool succeeds.FINAL RESPONSE FORMAT:diff --git a/skyvern/forge/sdk/copilot/narration.py b/skyvern/forge/sdk/copilot/narration.pyindex 9922b1021..107d249a9 100644--- a/skyvern/forge/sdk/copilot/narration.py+++ b/skyvern/forge/sdk/copilot/narration.py@@ -108,6 +108,7 @@ _TOOL_ACTIVITY_DISPLAY_LABELS = {"wait_for_either_state": "Waiting for the page","fill_credential_field": "Entering saved credentials","edit_block": "Editing block",+ "add_block": "Adding block","delete_block": "Deleting block",}diff --git a/skyvern/forge/sdk/copilot/output_utils.py b/skyvern/forge/sdk/copilot/output_utils.pyindex 68fbccb0f..640fd3b47 100644--- a/skyvern/forge/sdk/copilot/output_utils.py+++ b/skyvern/forge/sdk/copilot/output_utils.py@@ -698,6 +698,7 @@ _USER_FACING_EMPTY_SUCCESS_TOOLS: frozenset[str] = frozenset(# The server-authored display label already names the operation and its# target block; a bare "OK" summary would render instead of it."edit_block",+ "add_block","delete_block",})diff --git a/skyvern/forge/sdk/copilot/tools/__init__.py b/skyvern/forge/sdk/copilot/tools/__init__.pyindex 89e0e873b..fc172c960 100644--- a/skyvern/forge/sdk/copilot/tools/__init__.py+++ b/skyvern/forge/sdk/copilot/tools/__init__.py@@ -33,9 +33,16 @@ from skyvern.forge.sdk.copilot.output_utils import (from skyvern.forge.sdk.copilot.screenshot_utils import enqueue_screenshot_from_resultfrom skyvern.forge.sdk.copilot.secret_scrub import scrub_secrets_from_structurefrom skyvern.forge.sdk.copilot.tracing_setup import copilot_span-from skyvern.forge.sdk.copilot.workflow_yaml import BlockEditError+from skyvern.forge.sdk.copilot.workflow_yaml import (+ BlockEditError,+)from skyvern.forge.sdk.copilot.workflow_yaml import _process_workflow_yaml as _process_workflow_yaml-from skyvern.forge.sdk.copilot.workflow_yaml import apply_block_edit, delete_block_from_workflow, stored_workflow_yaml+from skyvern.forge.sdk.copilot.workflow_yaml import (+ add_block_to_workflow,+ apply_block_edit,+ delete_block_from_workflow,+ stored_workflow_yaml,+)from ._shared import _COMPOSITION_STRIPPED_HTML_MAX_CHARS as _COMPOSITION_STRIPPED_HTML_MAX_CHARSfrom ._shared import _DISCOVERY_PER_CALL_TIMEOUT_SECONDS as _DISCOVERY_PER_CALL_TIMEOUT_SECONDS@@ -295,15 +302,30 @@ async def update_workflow_tool(return json.dumps(sanitized)-async def _persist_block_scoped_edit(copilot_ctx: Any, tool_name: str, workflow_yaml: str, arguments: dict) -> str:+async def _persist_block_scoped_edit(+ copilot_ctx: Any,+ tool_name: str,+ workflow_yaml: str,+ arguments: dict,+ *,+ code_artifact_metadata: list[CodeArtifactMetadata] | None = None,+ block_observation_refs: list[BlockObservationRef] | None = None,+) -> str:"""Send a server-composed workflow through the normal persistence path.The model never sends the whole workflow, but the saved result still goes through everyauthor-time check, so a block edit cannot slip past what a full submission must satisfy."""prior_definition = await _get_prior_workflow_definition(copilot_ctx)+ params: dict[str, Any] = {"workflow_yaml": workflow_yaml}+ if code_artifact_metadata is not None:+ params["code_artifact_metadata"] = _code_artifact_metadata_as_tool_argument(code_artifact_metadata)+ params["raw_code_artifact_metadata"] = code_artifact_metadata+ if block_observation_refs is not None:+ params["block_observation_refs"] = normalize_block_observation_refs(block_observation_refs)+ params["raw_block_observation_refs"] = block_observation_refswith copilot_span(tool_name, data={"yaml_length": len(workflow_yaml)}):- result = await _update_workflow({"workflow_yaml": workflow_yaml}, copilot_ctx)+ result = await _update_workflow(params, copilot_ctx)_record_workflow_update_result(copilot_ctx, result, prior_definition)record_tool_step_result_for_ctx(copilot_ctx, tool_name, arguments, result)if result.get("ok") is False:@@ -357,6 +379,55 @@ async def edit_block_tool(return await _persist_block_scoped_edit(copilot_ctx, "edit_block", workflow_yaml, arguments)+@function_tool(name_override="add_block", strict_mode=False)+async def add_block_tool(+ ctx: RunContextWrapper,+ after_label: str,+ block_yaml: str,+ parameters: list[dict[str, Any]] | None = None,+ code_artifact_metadata: list[CodeArtifactMetadata] | None = None,+ block_observation_refs: list[BlockObservationRef] | None = None,+) -> str:+ """Add one new block after an existing one, leaving every other block exactly as it is.++ Prefer this over update_and_run_blocks whenever you are adding to a workflow that already exists:+ you send only the new block, so the blocks that already work cannot be disturbed and the workflow+ is not retyped. `after_label` must name a block that exists; the new block is linked in directly+ after it and inherits what that block pointed at.++ Pass `block_yaml` as a single block mapping including its `label`. Declare any new top-level+ workflow parameters the block reads in `parameters` — a new block and the parameter it consumes+ have to land in the same call, or the workflow is briefly saved in a state that cannot run. For a+ code block pass its `code_artifact_metadata` row here too, since a brand-new block has none yet.++ To change a block that already exists use edit_block; to remove one use delete_block.+ """+ copilot_ctx = ctx.context+ arguments = {"after_label": after_label, "parameters": parameters}+ authority_error = _authority_tool_error(copilot_ctx, "add_block")+ if authority_error:+ return json.dumps({"ok": False, "error": authority_error})+ try:+ workflow_yaml = add_block_to_workflow(+ _stored_workflow_yaml(copilot_ctx),+ after_label,+ block_yaml,+ parameters=parameters,+ )+ except BlockEditError as exc:+ result = {"ok": False, "error": str(exc)}+ record_tool_step_result_for_ctx(copilot_ctx, "add_block", arguments, result)+ return json.dumps(result)+ return await _persist_block_scoped_edit(+ copilot_ctx,+ "add_block",+ workflow_yaml,+ arguments,+ code_artifact_metadata=code_artifact_metadata,+ block_observation_refs=block_observation_refs,+ )++@function_tool(name_override="delete_block")async def delete_block_tool(ctx: RunContextWrapper, label: str) -> str:"""Remove one block from the workflow by label.@@ -959,6 +1030,7 @@ async def fill_credential_field_tool(NATIVE_TOOLS = [update_workflow_tool,edit_block_tool,+ add_block_tool,delete_block_tool,list_credentials_tool,list_integrations_tool,diff --git a/skyvern/forge/sdk/copilot/workflow_yaml.py b/skyvern/forge/sdk/copilot/workflow_yaml.pyindex bb2fb0145..90be80136 100644--- a/skyvern/forge/sdk/copilot/workflow_yaml.py+++ b/skyvern/forge/sdk/copilot/workflow_yaml.py@@ -749,6 +749,72 @@ def apply_block_edit(return yaml.safe_dump(parsed, sort_keys=False)+def _merge_new_workflow_parameters(parsed: dict[str, Any], parameters: list[Any]) -> None:+ definition = parsed["workflow_definition"]+ existing = definition.get("parameters")+ if not isinstance(existing, list):+ existing = []+ declared = {str(p.get("key") or "") for p in existing if isinstance(p, dict)}+ for parameter in parameters:+ if not isinstance(parameter, dict):+ raise BlockEditError("Each entry in parameters must be a mapping with a key.")+ key = str(parameter.get("key") or "")+ if not key:+ raise BlockEditError("Each entry in parameters needs a key.")+ # Declaring is additive only: a key the workflow already has keeps its current definition.+ if key in declared:+ continue+ existing.append(parameter)+ declared.add(key)+ definition["parameters"] = existing+++def add_block_to_workflow(+ stored_yaml: str,+ after_label: str,+ block_yaml: str,+ *,+ parameters: list[Any] | None = None,+) -> str:+ """Splice one new block in after ``after_label`` and return the whole workflow.++ The predecessor hands its ``next_block_label`` to the new block and points at it instead, so the+ chain stays intact without the model retyping the blocks it is not changing.+ """+ try:+ parsed = safe_load_no_dates(stored_yaml)+ except Exception as exc:+ raise BlockEditError(f"The stored workflow is not parseable: {exc}") from exc+ blocks = _workflow_blocks(parsed)+ if blocks is None:+ raise BlockEditError("The stored workflow has no workflow_definition.blocks to edit.")+ predecessor = _block_by_label(blocks, after_label)++ try:+ new_block = safe_load_no_dates(block_yaml)+ except Exception as exc:+ raise BlockEditError(f"block_yaml is not parseable: {exc}") from exc+ if not isinstance(new_block, dict):+ raise BlockEditError("block_yaml must be a single block mapping.")+ new_label = str(new_block.get("label") or "")+ if not new_label:+ raise BlockEditError("The new block needs a label.")+ known = sorted(str(b.get("label") or "") for b in blocks if isinstance(b, dict))+ if new_label in known:+ raise BlockEditError(+ f"A block labelled {new_label!r} already exists. The workflow has: {', '.join(known)}. "+ "Use edit_block to change it, or give the new block a different label."+ )++ new_block["next_block_label"] = predecessor.get("next_block_label")+ predecessor["next_block_label"] = new_label+ blocks.insert(blocks.index(predecessor) + 1, new_block)++ if parameters:+ _merge_new_workflow_parameters(parsed, parameters)+ return yaml.safe_dump(parsed, sort_keys=False)++def delete_block_from_workflow(stored_yaml: str, label: str) -> str:"""Remove one block by label and return the whole workflow.diff --git a/tests/unit/test_copilot_block_scoped_edits.py b/tests/unit/test_copilot_block_scoped_edits.pyindex 7fe486079..51b9bdc71 100644--- a/tests/unit/test_copilot_block_scoped_edits.py+++ b/tests/unit/test_copilot_block_scoped_edits.py@@ -8,9 +8,12 @@ from __future__ import annotationsfrom types import SimpleNamespaceimport pytest+import yaml+from skyvern.forge.sdk.copilot.tools.workflow_update import _code_block_safety_errorsfrom skyvern.forge.sdk.copilot.workflow_yaml import (BlockEditError,+ add_block_to_workflow,apply_block_edit,delete_block_from_workflow,stored_block_code,@@ -120,6 +123,124 @@ class TestFieldEdit:assert "read_total" in out+def _blocks(workflow: str) -> dict[str, dict]:+ parsed = yaml.safe_load(workflow)+ return {b["label"]: b for b in parsed["workflow_definition"]["blocks"]}+++_NEW_BLOCK = """block_type: code+label: check_pages+code: |+ for path in page_paths:+ await page.goto(path)+"""+++class TestAddBlock:+ def test_splices_after_the_named_block_and_relinks_the_chain(self) -> None:+ out = add_block_to_workflow(_WORKFLOW, "open_portal", _NEW_BLOCK)+ blocks = _blocks(out)+ assert blocks["open_portal"]["next_block_label"] == "check_pages"+ assert blocks["check_pages"]["next_block_label"] == "read_total", (+ "the new block must inherit what its predecessor pointed at, or the chain is cut"+ )++ def test_leaves_every_other_block_byte_identical(self) -> None:+ """The property the whole design turns on: adding must not re-decide a block that already works."""+ before, after = _blocks(_WORKFLOW), _blocks(add_block_to_workflow(_WORKFLOW, "open_portal", _NEW_BLOCK))+ assert after["read_total"] == before["read_total"]+ predecessor = dict(after["open_portal"])+ assert predecessor.pop("next_block_label") == "check_pages"+ assert predecessor == {k: v for k, v in before["open_portal"].items() if k != "next_block_label"}++ def test_appending_after_the_last_block_leaves_the_new_block_terminal(self) -> None:+ blocks = _blocks(add_block_to_workflow(_WORKFLOW, "read_total", _NEW_BLOCK))+ assert blocks["read_total"]["next_block_label"] == "check_pages"+ assert blocks["check_pages"]["next_block_label"] is None++ def test_a_duplicate_label_is_refused_and_names_what_exists(self) -> None:+ with pytest.raises(BlockEditError) as exc:+ add_block_to_workflow(_WORKFLOW, "open_portal", "block_type: code\nlabel: read_total\ncode: x\n")+ assert "already exists" in str(exc.value)+ assert "open_portal" in str(exc.value) and "read_total" in str(exc.value)++ def test_an_unknown_after_label_names_what_exists(self) -> None:+ with pytest.raises(BlockEditError) as exc:+ add_block_to_workflow(_WORKFLOW, "ghost", _NEW_BLOCK)+ assert "open_portal" in str(exc.value) and "read_total" in str(exc.value)++ def test_a_block_without_a_label_is_refused(self) -> None:+ with pytest.raises(BlockEditError):+ add_block_to_workflow(_WORKFLOW, "open_portal", "block_type: code\ncode: x\n")++ @pytest.mark.parametrize("block_yaml", ["{{ not yaml", "- a\n- b\n"])+ def test_block_yaml_must_be_one_block_mapping(self, block_yaml: str) -> None:+ with pytest.raises(BlockEditError):+ add_block_to_workflow(_WORKFLOW, "open_portal", block_yaml)+++class TestAddBlockParameters:+ """A new block and the parameter it reads have to land in one write, or the saved workflow cannot run."""++ def test_declares_the_new_parameter_alongside_the_block(self) -> None:+ out = add_block_to_workflow(+ _WORKFLOW,+ "open_portal",+ _NEW_BLOCK,+ parameters=[{"key": "page_paths", "parameter_type": "workflow", "workflow_parameter_type": "json"}],+ )+ parsed = yaml.safe_load(out)+ assert [p["key"] for p in parsed["workflow_definition"]["parameters"]] == ["page_paths"]+ assert "check_pages" in _blocks(out)++ def test_an_already_declared_key_keeps_its_current_definition(self) -> None:+ """Editing existing parameters is out of scope, so a repeated key must not be redefined."""+ workflow = _WORKFLOW.replace(+ "workflow_definition:\n",+ "workflow_definition:\n parameters:\n - key: page_paths\n parameter_type: workflow\n",+ )+ out = add_block_to_workflow(+ workflow, "open_portal", _NEW_BLOCK, parameters=[{"key": "page_paths", "parameter_type": "credential"}]+ )+ parameters = yaml.safe_load(out)["workflow_definition"]["parameters"]+ assert parameters == [{"key": "page_paths", "parameter_type": "workflow"}]++ def test_a_parameter_without_a_key_is_refused(self) -> None:+ with pytest.raises(BlockEditError):+ add_block_to_workflow(_WORKFLOW, "open_portal", _NEW_BLOCK, parameters=[{"parameter_type": "workflow"}])+++class TestAddBlockRunsTheSameAuthorTimeChecks:+ """add_block composes a whole workflow server-side and persists it through the shared path, so it+ cannot be a way around a check a whole-document write must satisfy."""++ _UNSAFE = 'block_type: code\nlabel: exfiltrate\ncode: |\n await page.request.get("https://example.test/x")\n'++ def test_unsafe_code_added_this_way_still_trips_the_code_safety_reject(self) -> None:+ spliced = add_block_to_workflow(_WORKFLOW, "read_total", self._UNSAFE)++ errors = _code_block_safety_errors(spliced, _WORKFLOW)++ assert [e.reason_code for e in errors] == ["AUTHOR_PAGE_REQUEST"]++ def test_it_trips_exactly_what_the_whole_document_write_trips(self) -> None:+ spliced = add_block_to_workflow(_WORKFLOW, "read_total", self._UNSAFE)+ retyped = yaml.safe_load(_WORKFLOW)+ retyped["workflow_definition"]["blocks"].append(yaml.safe_load(self._UNSAFE))+ whole_document = yaml.safe_dump(retyped, sort_keys=False)++ assert [e.reason_code for e in _code_block_safety_errors(spliced, _WORKFLOW)] == [+ e.reason_code for e in _code_block_safety_errors(whole_document, _WORKFLOW)+ ]++ def test_the_untouched_blocks_are_not_re_checked(self) -> None:+ """The gate is label-scoped against the prior workflow, so a purely additive splice presents+ exactly one changed block."""+ spliced = add_block_to_workflow(_WORKFLOW, "read_total", _NEW_BLOCK)++ assert _code_block_safety_errors(spliced, _WORKFLOW) == []++class TestDelete:def test_removes_the_block_and_unlinks_what_pointed_at_it(self) -> None:out = delete_block_from_workflow(_WORKFLOW, "read_total")diff --git a/tests/unit/test_copilot_output_utils.py b/tests/unit/test_copilot_output_utils.pyindex 62400b539..7e163caa2 100644--- a/tests/unit/test_copilot_output_utils.py+++ b/tests/unit/test_copilot_output_utils.py@@ -1309,7 +1309,7 @@ def test_summarize_tool_result_for_credentials_is_unchanged_for_agent_state() ->def test_surgical_edit_success_summary_is_empty_so_the_row_shows_its_label() -> None:- for tool_name in ("edit_block", "delete_block"):+ for tool_name in ("edit_block", "add_block", "delete_block"):assert format_tool_result_for_user(tool_name, {"ok": True, "data": {"label": "login_form"}}) == ""