Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
fix(SKY-9189): restore raw-credential refusal in Workflow Copilot v2 (#5598)
Andrew Neilson0dad5ea50bc3
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index cec6bae4f..3c4c6bb38 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -127,17 +127,20 @@ IMPORTANT WORKFLOW RULES:* Your success message to the user MUST reflect what actually happened. If you only navigated the browser but did NOT call update_workflow, do NOT tell the user the workflow was updated or is ready. Only confirm workflow changes after update_workflow succeeds.* When the user's request requires authentication but no credential is provided, use list_credentials to find the right one.-CREDENTIAL HANDLING:-* When the user needs authentication, call list_credentials first.-* If credentials are found, pick the most appropriate one and use its credential_id.-* Create a credential parameter with a descriptive key (e.g., "login_credentials"), put the credential_id in default_value.-* Reference the descriptive key in parameter_keys, NOT the credential ID directly.-* NEVER expose passwords or secrets.+CREDENTIAL HANDLING - CRITICAL:+* If the user's message contains a raw credential written inline — passwords (alone or paired with a username), API keys, OAuth tokens, `Authorization: Bearer …` headers, JWTs, session cookies, TOTP seeds, OTP / one-time codes, private keys, or credit card numbers / CVVs — you MUST NOT build, update, or run a workflow this turn; you MUST NOT pass those values via `parameters`; and you MUST NOT type, submit, paste, or replay them through `type_text`, `click`, `press_key`, `select_option`, `evaluate`, or any other direct browser tool. A bare username, email, or non-secret identifier alone is NOT a raw credential and does not trigger this rule.+* Instead, FIRST call `list_credentials` with `page_size=50` and continue paging (incrementing `page`) while the response's `has_more` is `true`, so you have the full credential set before deciding. Then respond with `ASK_QUESTION`. The response MUST:+ - Tell the user not to paste raw credentials/secrets in chat, and briefly explain why (model telemetry, execution traces).+ - If the paged `list_credentials` results contain exactly one credential whose `name` or `username` clearly matches the user's target (normalized substring match), offer that credential by its `credential_id` (e.g., `cred_123`). Otherwise, enumerate the candidate credentials as `credential_id` + `name` + `username` tuples and ask the user to pick one. `list_credentials` has no site/domain field, so matching is on `name` / `username` only, never on URL.+ - Only conclude "no stored credential exists" after you have exhausted pagination (final page returned `has_more=false`). In that case, direct the user to create a credential in the Skyvern Credentials UI and reply with its id.+ - Include the literal string `DO NOT PROVIDE RAW LOGIN/PASSWORD` in all-caps (preserved from the prior copilot for recognizability).+* Do NOT echo the raw secret back in `user_response` or `global_llm_context`.+* When the user DOES provide a `credential_id`, pick the most appropriate credential and use its credential_id. Create a credential parameter with a descriptive key (e.g., "login_credentials"), put the credential_id in default_value, and reference the descriptive key in parameter_keys — never the credential ID directly. NEVER expose passwords or secrets.PASSING RUNTIME VALUES WHEN TESTING:-* When the user's message contains concrete values that correspond to workflow parameters (names, emails, IDs, plain JSON), pass them via the `parameters` argument on your first `update_and_run_blocks` or `run_blocks_and_collect_debug` call. Do NOT wait for a failed test to learn the values — the user already gave them to you.-* For sensitive values (password, secret, token, api_key, credential, totp/otp/one_time_code, private_key, auth): FIRST call `list_credentials` to see if a stored credential matches. If so, use its credential_id via a credential parameter (see CREDENTIAL HANDLING). If no matching credential exists and the user supplied an inline value, pass it via `parameters` — values under those specific key names are redacted from the outbound client stream and the tool-execution trace. Prefer credentials when available: they avoid any exposure of raw secrets in model telemetry.-* Creating new credentials from inline values is not currently available to you. If the user wants a reusable credential, tell them to create one in the UI.+* When the user's message contains concrete non-secret values that correspond to workflow parameters (names, emails, IDs, plain JSON), pass them via the `parameters` argument on your first `update_and_run_blocks` or `run_blocks_and_collect_debug` call. Do NOT wait for a failed test to learn the values — the user already gave them to you.+* For credential-shaped values (password, secret, token, api_key, credential, totp/otp/one_time_code, private_key, auth), follow the CREDENTIAL HANDLING refusal rule above instead of routing them through `parameters`.+* Creating new credentials from inline values is not available to you. If the user wants a reusable credential, tell them to create one in the UI.WORKFLOW SUBMISSION RULE:diff --git a/skyvern/forge/sdk/copilot/output_utils.py b/skyvern/forge/sdk/copilot/output_utils.pyindex 150c7f7ba..2a9d0ee7e 100644--- a/skyvern/forge/sdk/copilot/output_utils.py+++ b/skyvern/forge/sdk/copilot/output_utils.py@@ -38,7 +38,14 @@ def extract_final_text(result: RunResultStreaming) -> str:def parse_final_response(text: str) -> dict[str, Any]:- """Parse the agent's final JSON envelope, stripping markdown code fences."""+ """Parse the agent's final JSON envelope, stripping markdown code fences.++ Tolerant of literal control characters (newlines/tabs/CR) inside string+ values. Some model outputs wrap long `user_response` prose across real+ newlines instead of `\\n` escapes; strict `json.loads` rejects those, and+ without this fallback the frontend renders the full raw JSON object as+ the user-visible reply (see SKY-9189 test-2 regression).+ """cleaned = text.strip()for prefix in ("```json", "```"):if cleaned.startswith(prefix):@@ -48,12 +55,13 @@ def parse_final_response(text: str) -> dict[str, Any]:cleaned = cleaned[:-3]cleaned = cleaned.strip()- try:- parsed = json.loads(cleaned)- if isinstance(parsed, dict):- return parsed- except json.JSONDecodeError:- pass+ for strict in (True, False):+ try:+ parsed = json.loads(cleaned, strict=strict)+ if isinstance(parsed, dict):+ return parsed+ except json.JSONDecodeError:+ continuereturn {"type": "REPLY", "user_response": text}diff --git a/skyvern/forge/sdk/copilot/tools.py b/skyvern/forge/sdk/copilot/tools.pyindex 287747b51..ddda69433 100644--- a/skyvern/forge/sdk/copilot/tools.py+++ b/skyvern/forge/sdk/copilot/tools.py@@ -1483,7 +1483,11 @@ def _build_skyvern_mcp_overlays() -> dict[str, SchemaOverlay]:description=("Type text into an input element. Use a CSS selector, an AI intent ""description, or both to target the field. "- "Optionally clear the field first. Use this for form filling."+ "Optionally clear the field first. Use this for form filling. "+ "NEVER type inline passwords, API keys, tokens, cookies, TOTP/OTP "+ "codes, private keys, or other raw credentials/secrets received in "+ "chat — stop and follow the CREDENTIAL HANDLING refusal rule in the "+ "system prompt instead."),hide_params=frozenset({"session_id", "cdp_url", "delay"}),required_overrides=["text"],@@ -1718,6 +1722,11 @@ async def list_credentials_tool() -> str:"""List stored credentials (metadata only — never passwords or secrets).Use this to find credential IDs for login blocks.++ Paginated. `page_size` caps at 50. The response includes `has_more`;+ before concluding no credential exists, keep incrementing `page` until+ `has_more` is `false` — otherwise you risk telling the user to create+ a credential they have already stored on a later page."""copilot_ctx = ctx.contextloop_error = _tool_loop_error(copilot_ctx, "list_credentials")@@ -1771,15 +1780,14 @@ async def run_blocks_tool(Pass runtime values for workflow parameters via the `parameters` dict —keys must match the workflow parameter `key` field. When the user has- supplied concrete values in their message (names, emails, IDs), pass them- on the first call rather than letting the workflow fall back to+ supplied concrete non-secret values in their message (names, emails, IDs),+ pass them on the first call rather than letting the workflow fall back toplaceholders. For sensitive values (password, secret, token, api_key,- credential, totp, otp, one_time_code, private_key, auth), prefer calling- list_credentials first; if a stored credential matches, reference its- credential_id via a credential parameter. If no matching credential- exists and the user supplied an inline value, you may pass it via- `parameters` — values under those sensitive key names are redacted from- the outbound client stream and the tool-execution trace.+ credential, totp, otp, one_time_code, private_key, auth) — call+ `list_credentials` and use a credential parameter whose default_value is+ the stored `credential_id`. If no stored credential matches, do NOT pass+ the inline value via `parameters`; stop and follow the CREDENTIAL+ HANDLING refusal rule in the system prompt."""copilot_ctx = ctx.contextloop_error = _tool_loop_error(copilot_ctx, "run_blocks_and_collect_debug")@@ -1858,15 +1866,14 @@ async def update_and_run_blocks_tool(Pass runtime values for workflow parameters via the `parameters` dict —keys must match the workflow parameter `key` field. When the user has- supplied concrete values in their message (names, emails, IDs), pass them- on the first call rather than letting the workflow fall back to+ supplied concrete non-secret values in their message (names, emails, IDs),+ pass them on the first call rather than letting the workflow fall back toplaceholders. For sensitive values (password, secret, token, api_key,- credential, totp, otp, one_time_code, private_key, auth), prefer calling- list_credentials first; if a stored credential matches, reference its- credential_id via a credential parameter. If no matching credential- exists and the user supplied an inline value, you may pass it via- `parameters` — values under those sensitive key names are redacted from- the outbound client stream and the tool-execution trace.+ credential, totp, otp, one_time_code, private_key, auth) — call+ `list_credentials` and use a credential parameter whose default_value is+ the stored `credential_id`. If no stored credential matches, do NOT pass+ the inline value via `parameters`; stop and follow the CREDENTIAL+ HANDLING refusal rule in the system prompt."""copilot_ctx = ctx.contextloop_error = _tool_loop_error(copilot_ctx, "update_and_run_blocks")diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.pyindex 75b466c9f..a50e8d611 100644--- a/tests/unit/test_copilot_agent_helpers.py+++ b/tests/unit/test_copilot_agent_helpers.py@@ -314,3 +314,42 @@ class TestTranslateToAgentResultGating:assert "test failed" in agent_result.user_response.lower()assert "All done" not in agent_result.user_responseassert agent_result.updated_workflow is None+++class TestCredentialRefusalReachesAgent:+ """Prove the SKY-9189 refusal rule is actually delivered to the agent.++ `run_copilot_agent` constructs the openai-agents SDK `Agent(...)` with+ `instructions=_build_system_prompt(...)` and `tools=list(NATIVE_TOOLS)`.+ A behavior test would require patching the agent loop and is fragile; a+ construction test (rule text flows through the exact helpers the route+ uses) is deterministic and catches both prompt and tool-surface drift.+ """++ def test_build_system_prompt_carries_refusal_clause(self) -> None:+ from skyvern.forge.sdk.copilot.agent import _build_system_prompt++ prompt = _build_system_prompt(tool_usage_guide="", security_rules="")++ assert "CREDENTIAL HANDLING - CRITICAL" in prompt+ assert "DO NOT PROVIDE RAW LOGIN/PASSWORD" in prompt+ assert "MUST NOT build, update, or run a workflow" in prompt+ assert "redacted from the outbound client stream" not in prompt++ def test_native_tools_carry_refusal_reference(self) -> None:+ import re++ from skyvern.forge.sdk.copilot.tools import NATIVE_TOOLS++ targets = {"run_blocks_and_collect_debug", "update_and_run_blocks"}+ matched = {tool.name for tool in NATIVE_TOOLS if tool.name in targets}+ assert matched == targets, f"missing tools in NATIVE_TOOLS: {targets - matched}"++ cross_ref = re.compile(r"CREDENTIAL\s+HANDLING refusal rule")+ for tool in NATIVE_TOOLS:+ if tool.name not in targets:+ continue+ desc = tool.description+ assert "redacted from" not in desc, f"{tool.name} still claims redaction"+ assert "you may pass it via" not in desc, f"{tool.name} still permits inline secrets"+ assert cross_ref.search(desc), f"{tool.name} missing refusal cross-reference"diff --git a/tests/unit/test_copilot_output_utils.py b/tests/unit/test_copilot_output_utils.pyindex 52e1c6a8b..c001b561c 100644--- a/tests/unit/test_copilot_output_utils.py+++ b/tests/unit/test_copilot_output_utils.py@@ -5,6 +5,7 @@ from __future__ import annotationsfrom unittest.mock import MagicMockfrom skyvern.forge.sdk.copilot.output_utils import (+ parse_final_response,sanitize_tool_result_for_llm,summarize_tool_result,truncate_output,@@ -323,3 +324,43 @@ class TestSummarizeToolResult:def test_unknown_tool_returns_ok(self) -> None:summary = self._summarize("unknown_tool", {"ok": True})assert summary == "OK"+++class TestParseFinalResponse:+ """parse_final_response is the last mile between model output and the frontend.++ A parse failure falls back to `{"type": "REPLY", "user_response": text}`,+ which means the raw JSON object is rendered in the chat bubble. Real model+ outputs sometimes embed literal newlines inside string values (strict+ `json.loads` rejects those) — seen in SKY-9189 test-2 where the full+ refusal envelope landed in the user bubble instead of just user_response.+ """++ def test_parses_clean_json_envelope(self) -> None:+ envelope = '{"type": "ASK_QUESTION", "user_response": "hi"}'+ parsed = parse_final_response(envelope)+ assert parsed == {"type": "ASK_QUESTION", "user_response": "hi"}++ def test_strips_json_code_fence(self) -> None:+ envelope = '```json\n{"type": "REPLY", "user_response": "ok"}\n```'+ assert parse_final_response(envelope)["type"] == "REPLY"++ def test_tolerates_literal_newline_inside_string_value(self) -> None:+ # Real model output shape: a multi-line user_response split across+ # actual newlines instead of \n escapes. strict=True rejects this,+ # strict=False accepts it. Without the fallback, the whole JSON blob+ # gets shown to the user.+ envelope = '{"type": "ASK_QUESTION", "user_response": "line one\nline two"}'+ parsed = parse_final_response(envelope)+ assert parsed["type"] == "ASK_QUESTION"+ assert parsed["user_response"] == "line one\nline two"++ def test_unparseable_text_falls_back_to_reply(self) -> None:+ # Genuinely broken output still degrades gracefully.+ parsed = parse_final_response("not json at all")+ assert parsed == {"type": "REPLY", "user_response": "not json at all"}++ def test_non_dict_json_falls_back_to_reply(self) -> None:+ # A JSON array at top level is valid JSON but not a valid envelope.+ parsed = parse_final_response("[1, 2, 3]")+ assert parsed == {"type": "REPLY", "user_response": "[1, 2, 3]"}diff --git a/tests/unit/test_workflow_copilot_credential_refusal.py b/tests/unit/test_workflow_copilot_credential_refusal.pynew file mode 100644index 000000000..b7652af1a--- /dev/null+++ b/tests/unit/test_workflow_copilot_credential_refusal.py@@ -0,0 +1,177 @@+"""Deterministic prompt + tool-docstring tests for the v2 raw-credential refusal.++Locks in the fix for SKY-9189. The rule lives in two places the agent reads+as operating instructions: the system prompt template+`workflow-copilot-agent.j2`, and the tool docstrings for+`run_blocks_and_collect_debug` and `update_and_run_blocks` (exposed to the+agents SDK via FunctionTool.description). Both must state the same policy,+or the agent follows whichever it weights higher.+"""++from skyvern.forge.prompts import prompt_engine+from skyvern.forge.sdk.copilot.tools import (+ run_blocks_tool,+ update_and_run_blocks_tool,+)++_AGENT_TEMPLATE_DEFAULTS = dict(+ workflow_knowledge_base="test kb",+ current_datetime="2026-01-01T00:00:00Z",+ tool_usage_guide="",+ security_rules="",+)+++def _render_agent_prompt() -> str:+ return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)+++class TestAgentPromptRefusalClause:+ """The rewritten system prompt must carry the hard refusal rule."""++ def test_uppercase_marker_present(self) -> None:+ """The recognizability marker preserved from v1 must render verbatim."""+ rendered = _render_agent_prompt()+ assert "DO NOT PROVIDE RAW LOGIN/PASSWORD" in rendered++ def test_critical_section_header_present(self) -> None:+ """The section is labelled CRITICAL so prompt scanners can find it."""+ rendered = _render_agent_prompt()+ assert "CREDENTIAL HANDLING - CRITICAL" in rendered++ def test_ask_question_response_required(self) -> None:+ """On refusal, the agent must use ASK_QUESTION, not a workflow build."""+ rendered = _render_agent_prompt()+ assert "ASK_QUESTION" in rendered+ assert "MUST NOT build, update, or run a workflow" in rendered++ def test_generalized_secret_surface_enumerated(self) -> None:+ """Refusal covers more than username/password — all the common leakage shapes."""+ rendered = _render_agent_prompt()+ for term in (+ "API keys",+ "OAuth tokens",+ "Authorization: Bearer",+ "JWTs",+ "session cookies",+ "TOTP seeds",+ "OTP",+ "private keys",+ "credit card numbers",+ "CVVs",+ ):+ assert term in rendered, f"missing secret-surface term: {term}"++ def test_browser_tools_also_forbidden(self) -> None:+ """The refusal must block not just workflow build/run but direct browser-tool paths too.++ Without this, an agent can 'comply' by skipping workflow tools and still+ type_text the pasted password into a live login form. Explicit names of+ the browser tools keep the rule concrete for the LLM.+ """+ rendered = _render_agent_prompt()+ assert "type_text" in rendered+ assert "click" in rendered+ assert "press_key" in rendered+ assert "any other direct browser tool" in rendered++ def test_list_credentials_pagination_instruction(self) -> None:+ """Prompt must tell the agent to page list_credentials before concluding 'no match'.++ list_credentials defaults to page_size=10 and caps at 50. Without this+ instruction the agent can falsely tell the user 'no stored credential+ exists' and send them to the UI to create one they already have.+ """+ rendered = _render_agent_prompt()+ assert "page_size=50" in rendered+ assert "has_more" in rendered++ def test_negative_bare_identifier_is_not_a_credential(self) -> None:+ """A bare username/email alone must not trigger the refusal rule.++ The section explicitly calls out that bare non-secret identifiers don't+ count, so the prompt doesn't over-refuse ('my username is alice').+ """+ rendered = _render_agent_prompt()+ assert "bare username" in rendered+ assert "NOT a raw credential" in rendered+ assert "does not trigger this rule" in rendered++ def test_list_credentials_called_first(self) -> None:+ """The refusal flow routes through list_credentials for discoverable matches."""+ rendered = _render_agent_prompt()+ assert "FIRST call `list_credentials`" in rendered++ def test_no_domain_matching(self) -> None:+ """list_credentials has no domain field, so the rule must match on name/username only."""+ rendered = _render_agent_prompt()+ assert "no site/domain field" in rendered+ assert "on `name` / `username` only" in rendered++ def test_no_raw_secret_echoed(self) -> None:+ """The agent must not echo the raw secret into its reply or persistent context."""+ rendered = _render_agent_prompt()+ assert "Do NOT echo the raw secret back" in rendered++ def test_old_permissive_clause_is_gone(self) -> None:+ """The v2 template used to authorize inline secrets via `parameters` — must be removed."""+ rendered = _render_agent_prompt()+ assert "redacted from the outbound client stream" not in rendered+ assert "you may pass it via `parameters`" not in rendered+++class TestToolDocstringsRefusalClause:+ """Tool docstrings reach the agent via FunctionTool.description — they must agree with the prompt."""++ def _tools(self) -> list[object]:+ return [run_blocks_tool, update_and_run_blocks_tool]++ def test_old_permissive_clause_gone_from_tools(self) -> None:+ """The clause that told the agent inline secrets were fine via `parameters` is removed."""+ for tool in self._tools():+ desc = tool.description # type: ignore[attr-defined]+ assert "redacted from" not in desc, f"{tool.name} still claims redaction" # type: ignore[attr-defined]+ assert "you may pass it via" not in desc, f"{tool.name} still permits inline secrets" # type: ignore[attr-defined]++ def test_new_refusal_reference_in_tools(self) -> None:+ """Docstrings point back at the CREDENTIAL HANDLING refusal rule in the system prompt."""+ import re++ cross_ref = re.compile(r"CREDENTIAL\s+HANDLING refusal rule")+ for tool in self._tools():+ desc = tool.description # type: ignore[attr-defined]+ assert "do NOT pass" in desc, f"{tool.name} does not forbid inline secret pass-through" # type: ignore[attr-defined]+ assert cross_ref.search(desc), f"{tool.name} does not cross-reference the refusal rule" # type: ignore[attr-defined]++ def test_non_secret_parameters_guidance_preserved(self) -> None:+ """The `parameters` dict is still the right channel for non-secret runtime values."""+ for tool in self._tools():+ desc = tool.description # type: ignore[attr-defined]+ assert "non-secret values" in desc, f"{tool.name} missing non-secret guidance" # type: ignore[attr-defined]++ def test_list_credentials_tool_describes_pagination(self) -> None:+ """list_credentials docstring must warn about paging before concluding no match."""+ from skyvern.forge.sdk.copilot.tools import list_credentials_tool++ desc = list_credentials_tool.description # type: ignore[attr-defined]+ assert "has_more" in desc+ assert "already stored on a later page" in desc+++class TestBrowserToolOverlayRefusalCaveat:+ """The MCP browser-tool overlays are also operating instructions for the agent.++ type_text is the primary leakage vector (typing a pasted password into a+ live login form). Its overlay description must tell the agent to refuse+ there too — otherwise prompt-level refusal is bypassable by the agent+ following the overlay's own (previously silent) description.+ """++ def test_type_text_overlay_forbids_inline_secrets(self) -> None:+ from skyvern.forge.sdk.copilot.tools import _build_skyvern_mcp_overlays++ overlays = _build_skyvern_mcp_overlays()+ assert "type_text" in overlays+ desc = overlays["type_text"].description or ""+ assert "NEVER type inline passwords" in desc+ assert "CREDENTIAL HANDLING refusal rule" in desc