Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
feat(SKY-9720) (1/3): Structure request policy scope (#5942)
Andrew Neilsone6505e4aed83
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2index f08b5e979..a6a3bf732 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2@@ -1,5 +1,5 @@Classify the latest Workflow Copilot request. Return only JSON with keys:-testing_intent=require_test|skip_test|unspecified; credential_input_kind=none|raw_secret|credential_id|credential_name|website_stored_credential|placeholder; credential_refs; login_page_urls; requires_user_clarification; completion_contract.+testing_intent=require_test|skip_test|unspecified; credential_input_kind=none|raw_secret|credential_id|credential_name|website_stored_credential|placeholder; credential_refs; login_page_urls; requires_user_clarification; clarification_reason; completion_contract.testing_intent:- skip_test: the user asks for an untested draft, says they will test it themselves, says draft only, just sketch it, no validation run, or skip testing.@@ -10,7 +10,7 @@ testing_intent:credential_input_kind:- raw_secret: the user pasted inline login credentials or secrets, including passwords, API keys, tokens, bearer headers, cookies, TOTP/OTP codes, private keys, card numbers, or CVVs.- credential_id: the user provided one or more exact credential IDs beginning with `cred_`. Put every ID in credential_refs.-- credential_name: the user gave an exact saved credential name, or vaguely says to use "my credentials", "my saved credentials", "credentials saved already", or an entry in the Credentials UI without enough website context. Put exact names in credential_refs; leave credential_refs empty when the reference is vague.+- credential_name: the user gave an exact saved credential name, or vaguely says to use "my credentials", "my saved credentials", "credentials saved already", or an entry in the Credentials UI without enough website context. Treat quoted names and bare identifier-like names tied to credential verbs as exact names, for example `using azure_credentials`, `get the code from mfa_email`, or `use Login Prod`. Put exact names in credential_refs; leave credential_refs empty when the reference is vague.- website_stored_credential: the user asks to use stored/saved credentials for a specific website or login page URL. Put URLs in login_page_urls.- placeholder: the user asks to add credential/password/username fields but explicitly says there are no real credentials yet.- none: the request mentions a login workflow but does not provide, request, or refer to any credential material.@@ -24,6 +24,21 @@ completion_contract:- The value must be an exact span copied from the latest user message. Do not paraphrase, summarize, or infer a completion condition.- Set to null when the user does not provide an explicit completion condition.+clarification_reason:+- none: no clarification is needed.+- raw_secret: the latest user message includes raw credential material.+- credential_name_unresolved: the user asks for a saved login/credential without an exact saved credential name, credential ID, or resolvable login page URL.+- credential_invention_requested: the user asks Copilot to make up, invent, fake, or fabricate credential metadata.+- ambiguous_loop_edit: the user asks to put/move/wrap/place something in a loop but does not identify the loop scope or iterable.+- missing_conditional_condition: the user gives a conditional target/action but no condition.+- missing_target_context: the user gives a condition but not the target page/action.++Rules:+- Do not use skip_test to bypass missing or ambiguous credential scope; skip_test only means a workflow draft should not be executed after the credential scope is explicit.+- Put exact saved credential names and credential IDs in credential_refs; do not include ordinary workflow parameters.+- Use recent chat history when classifying a follow-up. For example, if the latest message says to draft without testing and the prior turn gave exact saved credential names, carry those names forward in credential_refs.+- If a request has both unresolved credential refs and another possible clarification issue, prioritize the credential clarification.+- Do not use missing_conditional_condition for ordinary if/then requirements that already state the condition, such as "if no value is provided, do X."Latest user message:```{{ user_message }}```Recent chat history:diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.pyindex 3ed9cb100..446d1dc93 100644--- a/skyvern/forge/sdk/copilot/agent.py+++ b/skyvern/forge/sdk/copilot/agent.py@@ -438,12 +438,26 @@ _PROPOSAL_ACCEPT_UI_ACTION_RE = re.compile(r"\b(?:accept|always\s+accept)\b", re_PROPOSAL_REJECT_UI_ACTION_RE = re.compile(r"\b(?:reject|discard)\b", re.IGNORECASE)_UNVALIDATED_PROPOSAL_AFFORDANCE = ("I have a draft workflow proposal. Use Review to inspect it, Accept to save it, or Reject to discard it. "- "It has not been verified end-to-end."+ "It has not been tested or verified end-to-end."+)+_UNVALIDATED_DISCLOSURE_PHRASES = (+ "not tested",+ "not been tested",+ "not verified",+ "not been verified",+ "hasn't been tested",+ "hasn't been verified",+ "unvalidated",)def _ensure_unvalidated_proposal_affordance(user_response: str) -> str:- if _PROPOSAL_ACCEPT_UI_ACTION_RE.search(user_response) and _PROPOSAL_REJECT_UI_ACTION_RE.search(user_response):+ lower = user_response.lower()+ has_ui_affordance = bool(+ _PROPOSAL_ACCEPT_UI_ACTION_RE.search(user_response) and _PROPOSAL_REJECT_UI_ACTION_RE.search(user_response)+ )+ has_unvalidated_disclosure = any(phrase in lower for phrase in _UNVALIDATED_DISCLOSURE_PHRASES)+ if has_ui_affordance and has_unvalidated_disclosure:return user_responseif user_response.strip():return f"{user_response}\n\n{_UNVALIDATED_PROPOSAL_AFFORDANCE}"diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.pyindex 65c6c1da3..17b78c844 100644--- a/skyvern/forge/sdk/copilot/request_policy.py+++ b/skyvern/forge/sdk/copilot/request_policy.py@@ -3,7 +3,7 @@ from __future__ import annotationsimport asyncioimport refrom dataclasses import dataclass, field-from typing import Any+from typing import Any, Literal, cast, get_argsfrom urllib.parse import urlparseimport structlog@@ -11,6 +11,7 @@ import structlogfrom skyvern.config import settingsfrom skyvern.forge import appfrom skyvern.forge.prompts import prompt_engine+from skyvern.forge.sdk.copilot.context import StructuredContextfrom skyvern.forge.sdk.copilot.output_utils import parse_final_responsefrom skyvern.forge.sdk.copilot.tracing_setup import copilot_spanfrom skyvern.forge.sdk.schemas.credentials import Credential@@ -20,6 +21,33 @@ LOG = structlog.get_logger()PROMPT_NAME = "workflow-copilot-request-policy"_TESTING_INTENTS = {"require_test", "skip_test", "unspecified"}_KINDS = {"none", "raw_secret", "credential_id", "credential_name", "website_stored_credential", "placeholder"}+ClarificationReason = Literal[+ "none",+ "raw_secret",+ "credential_name_unresolved",+ "credential_invention_requested",+ "ambiguous_loop_edit",+ "missing_conditional_condition",+ "missing_target_context",+]+_VALID_CLARIFICATION_REASONS: frozenset[ClarificationReason] = frozenset(get_args(ClarificationReason))+_PRE_RESOLUTION_CLARIFICATION_REASONS = {+ "credential_invention_requested",+ "ambiguous_loop_edit",+ "missing_conditional_condition",+ "missing_target_context",+}+_REASONS_OVERRIDDEN_BY_CREDENTIAL_REFS = {+ "ambiguous_loop_edit",+ "missing_conditional_condition",+ "missing_target_context",+}+_RAW_SECRET_QUESTION = (+ "Please do not paste raw login credentials or secrets in chat because they can enter model telemetry and execution traces. "+ "Store the credential in the Skyvern Credentials UI and reply with its exact saved credential name or a credential ID beginning with cred_. DO NOT PROVIDE RAW LOGIN/PASSWORD."+)+_SAVED_CREDENTIAL_NAME_QUESTION = "Which saved credential should I use? Please provide the exact credential name or a credential ID beginning with cred_."+_STORED_CREDENTIAL_URL_QUESTION = "Which website or login page should I use to look up the stored credential?"_CREDENTIAL_ID_RE = re.compile(r"\bcred_[A-Za-z0-9][A-Za-z0-9_-]*\b")_RAW_SECRET_PATTERNS = (re.compile(r"\b(?:password|passcode|api[_ -]?key|secret|token|bearer|authorization)\s*[:=]\s*\S+", re.I),@@ -31,6 +59,8 @@ _RAW_SECRET_PATTERNS = (re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),)+# Reused by output-policy guardrails as syntactic leak backstops.+RAW_SECRET_PATTERNS = _RAW_SECRET_PATTERNS@dataclass@@ -49,11 +79,13 @@ class RequestPolicy:invalid_credential_ids: list[str] = field(default_factory=list)clarification_question: str | None = Noneraw_secret_detected: bool = False+ clarification_reason: ClarificationReason = "none"def to_trace_data(self) -> dict[str, Any]:return {"testing_intent": self.testing_intent,"credential_input_kind": self.credential_input_kind,+ "clarification_reason": self.clarification_reason,"allow_update_workflow": self.allow_update_workflow,"allow_run_blocks": self.allow_run_blocks,"allow_missing_credentials_in_draft": self.allow_missing_credentials_in_draft,@@ -66,6 +98,7 @@ class RequestPolicy:lines = [f"testing_intent: {self.testing_intent}",f"credential_input_kind: {self.credential_input_kind}",+ f"clarification_reason: {self.clarification_reason}",f"allow_update_workflow: {self.allow_update_workflow}",f"allow_run_blocks: {self.allow_run_blocks}",f"allow_missing_credentials_in_draft: {self.allow_missing_credentials_in_draft}",@@ -94,6 +127,12 @@ def _raw_secret_detected(text: str) -> bool:return any(pattern.search(text or "") for pattern in _RAW_SECRET_PATTERNS)+def _coerce_clarification_reason(value: Any) -> ClarificationReason:+ if value in _VALID_CLARIFICATION_REASONS:+ return cast(ClarificationReason, value)+ return "none"++def redact_raw_secrets_for_prompt(text: str) -> str:redacted = text or ""for pattern in _RAW_SECRET_PATTERNS:@@ -117,7 +156,10 @@ def _classification_from_raw(raw: Any) -> RequestPolicy:login_page_urls=_clean_list(raw.get("login_page_urls") or []),requires_user_clarification=bool(raw.get("requires_user_clarification")),completion_contract=completion_contract or None,+ clarification_reason=_coerce_clarification_reason(raw.get("clarification_reason")),)+ if policy.credential_input_kind == "raw_secret":+ policy.clarification_reason = "raw_secret"return policy@@ -138,7 +180,12 @@ async def _classify_request() -> RequestPolicy:ids = _credential_ids(user_message)if _raw_secret_detected(user_message):- return RequestPolicy(credential_input_kind="raw_secret", credential_refs=ids, raw_secret_detected=True)+ return RequestPolicy(+ credential_input_kind="raw_secret",+ credential_refs=ids,+ raw_secret_detected=True,+ clarification_reason="raw_secret",+ )if handler is None:return RequestPolicy(credential_input_kind="credential_id" if ids else "none", credential_refs=ids)@@ -188,10 +235,18 @@ def _safe_label(credential: Credential) -> str:return " - ".join(parts)-def _block(policy: RequestPolicy, question: str, candidates: list[Credential] | None = None) -> None:+def _block(+ policy: RequestPolicy,+ question: str,+ candidates: list[Credential] | None = None,+ *,+ reason: ClarificationReason | None = None,+) -> None:policy.requires_user_clarification = Truepolicy.user_response_policy = "ask_clarification"policy.allow_update_workflow = policy.allow_run_blocks = False+ if reason is not None:+ policy.clarification_reason = reasonif candidates:question += "\n\nSafe matches:\n" + "\n".join(f"- {_safe_label(candidate)}" for candidate in candidates)policy.clarification_question = question@@ -223,13 +278,74 @@ def _match_by_url(credentials: list[Credential], urls: list[str]) -> list[Credendef _clarification_question(policy: RequestPolicy) -> str:+ if policy.clarification_reason == "raw_secret":+ return _RAW_SECRET_QUESTION+ if policy.clarification_reason == "credential_name_unresolved":+ if policy.credential_input_kind == "website_stored_credential":+ return _STORED_CREDENTIAL_URL_QUESTION+ return _SAVED_CREDENTIAL_NAME_QUESTION+ if policy.clarification_reason == "credential_invention_requested":+ return (+ "I cannot invent a credential ID. Please provide a valid saved credential ID, "+ "select an existing credential, or create one in the Credentials UI."+ )+ if policy.clarification_reason == "ambiguous_loop_edit":+ return "Which block or blocks should go inside the loop, and what should the loop iterate over or stop on?"+ if policy.clarification_reason == "missing_conditional_condition":+ return "What condition should trigger this conditional route?"+ if policy.clarification_reason == "missing_target_context":+ if policy.credential_input_kind == "website_stored_credential":+ return _STORED_CREDENTIAL_URL_QUESTION+ return "Which page or URL should the workflow go to?"if policy.credential_input_kind == "credential_name":- return "Which saved credential name should I use? Please provide the exact credential name or a credential ID beginning with cred_."+ return _SAVED_CREDENTIAL_NAME_QUESTIONif policy.credential_input_kind == "website_stored_credential":- return "Which website or login page should I use to look up stored credentials?"+ return _STORED_CREDENTIAL_URL_QUESTIONreturn "I need one more detail before I can build and test this workflow safely."+def _has_resolvable_credential_scope(policy: RequestPolicy) -> bool:+ if policy.credential_input_kind == "credential_id":+ return any(ref.startswith("cred_") for ref in policy.credential_refs)+ if policy.credential_input_kind == "credential_name":+ return bool(policy.credential_refs)+ if policy.credential_input_kind == "website_stored_credential":+ return bool(policy.login_page_urls)+ return False+++def _prioritize_credential_clarification(policy: RequestPolicy) -> None:+ if policy.credential_input_kind not in ("credential_id", "credential_name"):+ return+ if not policy.credential_refs:+ return+ if policy.clarification_reason not in _REASONS_OVERRIDDEN_BY_CREDENTIAL_REFS:+ return+ policy.clarification_reason = "credential_name_unresolved"+++def _previous_credential_clarification_was_asked(global_llm_context: str) -> bool:+ structured = StructuredContext.from_json_str(global_llm_context)+ return any(+ decision.startswith("request-policy clarification required:") and "/credential_name_unresolved" in decision+ for decision in structured.decisions_made+ )+++def _can_defer_unresolved_credential_name_for_draft(+ policy: RequestPolicy,+ *,+ global_llm_context: str,+) -> bool:+ if policy.clarification_reason != "credential_name_unresolved":+ return False+ if _has_resolvable_credential_scope(policy):+ return True+ if _previous_credential_clarification_was_asked(global_llm_context):+ return True+ return False++async def _resolve_credentials(policy: RequestPolicy, organization_id: str) -> None:if policy.credential_input_kind == "credential_id":ids = _clean_list([ref for ref in policy.credential_refs if ref.startswith("cred_")])@@ -244,6 +360,7 @@ async def _resolve_credentials(policy: RequestPolicy, organization_id: str) -> N_block(policy,f"The credential ID(s) {formatted} were not found in this organization. Please provide a valid saved credential ID or explicitly ask for an unvalidated draft that will not be run yet.",+ reason="credential_name_unresolved",)elif policy.invalid_credential_ids:policy.allow_run_blocks = False@@ -251,13 +368,21 @@ async def _resolve_credentials(policy: RequestPolicy, organization_id: str) -> Nreturnif policy.credential_input_kind == "credential_name" and not policy.credential_refs:+ if policy.testing_intent == "skip_test" and policy.allow_missing_credentials_in_draft:+ policy.allow_run_blocks = False+ return_block(policy,- "Which saved credential name should I use? Please provide the exact credential name or a credential ID beginning with cred_.",+ _SAVED_CREDENTIAL_NAME_QUESTION,+ reason="credential_name_unresolved",)returnif policy.credential_input_kind == "website_stored_credential" and not policy.login_page_urls:- _block(policy, "Which website or login page should I use to look up stored credentials?")+ _block(+ policy,+ _STORED_CREDENTIAL_URL_QUESTION,+ reason="missing_target_context",+ )returnif policy.credential_input_kind not in ("credential_name", "website_stored_credential"):return@@ -270,7 +395,10 @@ async def _resolve_credentials(policy: RequestPolicy, organization_id: str) -> Npolicy.resolved_credentials.append(matches[0])elif matches:_block(- policy, "I found multiple stored credentials with that exact name. Which one should I use?", matches+ policy,+ "I found multiple stored credentials with that exact name. Which one should I use?",+ matches,+ reason="credential_name_unresolved",)returnelif policy.testing_intent == "skip_test":@@ -279,6 +407,7 @@ async def _resolve_credentials(policy: RequestPolicy, organization_id: str) -> N_block(policy,f"I could not find a stored credential named `{ref}`. Please choose an existing credential by exact name or a credential ID beginning with cred_.",+ reason="credential_name_unresolved",)returnreturn@@ -287,11 +416,17 @@ async def _resolve_credentials(policy: RequestPolicy, organization_id: str) -> Nif len(matches) == 1:policy.resolved_credentials = matcheselif matches:- _block(policy, "I found multiple stored credentials for that login page. Which one should I use?", matches)+ _block(+ policy,+ "I found multiple stored credentials for that login page. Which one should I use?",+ matches,+ reason="credential_name_unresolved",+ )else:_block(policy,"I could not find a stored credential for that login page. Please select a saved credential by exact name or a credential ID beginning with cred_, or create one in the Credentials UI.",+ reason="credential_name_unresolved",)@@ -306,21 +441,42 @@ async def build_request_policy() -> RequestPolicy:policy = await _classify_request(user_message, workflow_yaml, chat_history, global_llm_context, handler)policy.raw_secret_detected = policy.raw_secret_detected or policy.credential_input_kind == "raw_secret"+ _prioritize_credential_clarification(policy)if policy.testing_intent == "skip_test":policy.allow_run_blocks = False- policy.allow_missing_credentials_in_draft = True- if policy.credential_input_kind != "raw_secret":- policy.requires_user_clarification = False+ if (+ policy.credential_input_kind != "raw_secret"+ and policy.clarification_reason not in _PRE_RESOLUTION_CLARIFICATION_REASONS+ ):+ if (+ policy.clarification_reason == "credential_name_unresolved"+ and not _can_defer_unresolved_credential_name_for_draft(+ policy,+ global_llm_context=global_llm_context,+ )+ ):+ policy.requires_user_clarification = True+ policy.allow_update_workflow = False+ else:+ policy.requires_user_clarification = False+ policy.allow_missing_credentials_in_draft = Trueif policy.raw_secret_detected:_block(policy,- "Please do not paste raw login credentials or secrets in chat because they can enter model telemetry and execution traces. Store the credential in the Skyvern Credentials UI and reply with its exact saved credential name or a credential ID beginning with cred_. DO NOT PROVIDE RAW LOGIN/PASSWORD.",+ _RAW_SECRET_QUESTION,+ reason="raw_secret",)- elif policy.requires_user_clarification:+ elif policy.requires_user_clarification and policy.clarification_reason in _PRE_RESOLUTION_CLARIFICATION_REASONS:+ _block(policy, _clarification_question(policy))+ elif policy.requires_user_clarification and not _has_resolvable_credential_scope(policy):_block(policy, _clarification_question(policy))else:try:+ # A resolvable credential scope can override the classifier's+ # conservative clarification flag; _resolve_credentials will block+ # again if the lookup is missing or ambiguous.+ policy.requires_user_clarification = Falseawait _resolve_credentials(policy, organization_id)except Exception:LOG.warning(diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.pyindex ef24a8dcb..28c7620ca 100644--- a/tests/unit/test_copilot_agent_helpers.py+++ b/tests/unit/test_copilot_agent_helpers.py@@ -615,7 +615,8 @@ workflow_definition:result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1")- assert agent_result.user_response == response+ assert response in agent_result.user_response+ assert "not been tested or verified" in agent_result.user_responseassert agent_result.updated_workflow is wfdef test_goal_reached_false_flips_validated_proposal_to_unvalidated(self) -> None:@@ -1040,6 +1041,7 @@ class TestRequestPolicyCredentialResolution:assert policy.allow_update_workflow is Falseassert policy.allow_run_blocks is Falseassert "DO NOT PROVIDE RAW LOGIN/PASSWORD" in (policy.clarification_question or "")+ assert "hunter2" not in (policy.clarification_question or "")handler.assert_not_awaited()get_credentials_by_ids.assert_not_called()@@ -1048,7 +1050,7 @@ class TestRequestPolicyCredentialResolution:from skyvern.forge.sdk.copilot import request_policy as policy_modulefrom skyvern.forge.sdk.copilot.request_policy import build_request_policy- credential = SimpleNamespace(credential_id="cred_bank", name="Bank", tested_url="https://bank.co.uk/login")+ credential = SimpleNamespace(credential_id="cred_bank", name="Bank", tested_url="https://bank.example/login")credentials = SimpleNamespace(get_credentials=AsyncMock(return_value=[credential]))monkeypatch.setattr(policy_module.app, "DATABASE", SimpleNamespace(credentials=credentials))@@ -1066,7 +1068,7 @@ class TestRequestPolicyCredentialResolution:handler.response = {"credential_input_kind": "website_stored_credential",- "login_page_urls": ["https://bank.co.uk/login"],+ "login_page_urls": ["https://bank.example/login"],}site_policy = await build_request_policy(user_message="use the saved login", chat_history="", **args)assert site_policy.user_response_policy == "proceed"@@ -1074,11 +1076,190 @@ class TestRequestPolicyCredentialResolution:handler.response = {"credential_input_kind": "website_stored_credential",- "login_page_urls": ["https://evil.co.uk/login"],+ "login_page_urls": ["https://evil.example/login"],}url_policy = await build_request_policy(user_message="use the saved login", chat_history="", **args)assert url_policy.user_response_policy == "ask_clarification" and not url_policy.resolved_credentials+ handler.response = {+ "testing_intent": "require_test",+ "credential_input_kind": "website_stored_credential",+ "login_page_urls": ["https://bank.example/login"],+ "requires_user_clarification": False,+ "clarification_reason": "none",+ }+ exact_url_policy = await build_request_policy(+ user_message=(+ "Build and test a workflow that logs into https://bank.example/login "+ "using the saved credential for that site."+ ),+ chat_history="",+ **args,+ )+ assert exact_url_policy.user_response_policy == "proceed"+ assert exact_url_policy.testing_intent == "require_test"+ assert exact_url_policy.credential_input_kind == "website_stored_credential"+ assert exact_url_policy.resolved_credentials == [credential]+ assert exact_url_policy.allow_update_workflow and exact_url_policy.allow_run_blocks++ handler.response = {+ "credential_input_kind": "website_stored_credential",+ "login_page_urls": ["https://evil.example/login"],+ }+ no_suffix_policy = await build_request_policy(+ user_message="Use the stored credential for https://evil.example/login.",+ chat_history="",+ **args,+ )+ assert no_suffix_policy.user_response_policy == "ask_clarification"+ assert not no_suffix_policy.resolved_credentials+ assert "could not find a stored credential" in (no_suffix_policy.clarification_question or "")++ handler.response = {+ "credential_input_kind": "website_stored_credential",+ "requires_user_clarification": True,+ "clarification_reason": "missing_target_context",+ }+ missing_url_policy = await build_request_policy(+ user_message="use my saved login for this site",+ chat_history="",+ **args,+ )+ assert missing_url_policy.user_response_policy == "ask_clarification"+ assert "stored credential" in (missing_url_policy.clarification_question or "")++ handler.response = {+ "testing_intent": "skip_test",+ "credential_input_kind": "none",+ "requires_user_clarification": True,+ "clarification_reason": "credential_name_unresolved",+ }+ vague_skip_policy = await build_request_policy(+ user_message="use my saved login for this site and finish the workflow",+ chat_history="user: create a login workflow",+ **args,+ )+ assert vague_skip_policy.user_response_policy == "ask_clarification"+ assert not vague_skip_policy.allow_update_workflow+ assert not vague_skip_policy.allow_run_blocks+ assert "saved credential" in (vague_skip_policy.clarification_question or "")++ prior_clarification_context = (+ '{"decisions_made":["request-policy clarification required: credential_name/credential_name_unresolved"]}'+ )+ history_refs_from_context = await build_request_policy(+ user_message="Just draft a workflow without testing it.",+ workflow_yaml="",+ chat_history="user: login using the 'azure_credentials' and get the code from the 'mfa_email'",+ global_llm_context=prior_clarification_context,+ organization_id="org-1",+ handler=handler,+ )+ assert history_refs_from_context.user_response_policy == "proceed"+ assert history_refs_from_context.allow_update_workflow and not history_refs_from_context.allow_run_blocks+ assert history_refs_from_context.allow_missing_credentials_in_draft++ handler.response = {+ "testing_intent": "skip_test",+ "credential_input_kind": "credential_name",+ "requires_user_clarification": True,+ "clarification_reason": "credential_name_unresolved",+ }+ first_turn_missing_name_policy = await build_request_policy(+ user_message="Draft but do not test using my saved credential.",+ workflow_yaml="",+ chat_history="",+ global_llm_context="",+ organization_id="org-1",+ handler=handler,+ )+ assert first_turn_missing_name_policy.user_response_policy == "ask_clarification"+ assert not first_turn_missing_name_policy.allow_update_workflow++ follow_up_missing_name_policy = await build_request_policy(+ user_message="Just draft a workflow without testing it.",+ workflow_yaml="",+ chat_history="user: login using azure_credentials",+ global_llm_context="",+ organization_id="org-1",+ handler=handler,+ )+ assert follow_up_missing_name_policy.user_response_policy == "ask_clarification"+ assert not follow_up_missing_name_policy.allow_update_workflow+ assert not follow_up_missing_name_policy.allow_run_blocks++ handler.response = {+ "testing_intent": "skip_test",+ "credential_input_kind": "credential_name",+ "credential_refs": ["azure_credentials", "mfa_email"],+ "requires_user_clarification": True,+ "clarification_reason": "credential_name_unresolved",+ }+ history_refs = await build_request_policy(+ user_message="Just draft a workflow without testing it.",+ chat_history="user: login using the 'azure_credentials' and get the code from the 'mfa_email'",+ **args,+ )+ assert history_refs.user_response_policy == "proceed"+ assert history_refs.credential_input_kind == "credential_name"+ assert history_refs.credential_refs == ["azure_credentials", "mfa_email"]+ assert history_refs.allow_update_workflow and not history_refs.allow_run_blocks+ assert history_refs.allow_missing_credentials_in_draft++ handler.response = {+ "testing_intent": "unspecified",+ "credential_input_kind": "credential_name",+ "credential_refs": ["azure_credentials", "mfa_email"],+ "requires_user_clarification": True,+ "clarification_reason": "missing_conditional_condition",+ }+ credential_priority_policy = await build_request_policy(+ user_message="Log in using the 'azure_credentials' and use 'mfa_email' for MFA. If no account is provided, search by account number.",+ chat_history="",+ **args,+ )+ assert credential_priority_policy.user_response_policy == "ask_clarification"+ assert credential_priority_policy.clarification_reason == "credential_name_unresolved"+ assert "azure_credentials" in (credential_priority_policy.clarification_question or "")+ assert "condition" not in (credential_priority_policy.clarification_question or "").lower()++ handler.response = {+ "testing_intent": "skip_test",+ "credential_input_kind": "credential_name",+ "credential_refs": ["azure_credentials", "mfa_email"],+ "requires_user_clarification": True,+ "clarification_reason": "missing_conditional_condition",+ }+ history_refs_with_noncredential_reason = await build_request_policy(+ user_message="Just draft a workflow without testing it.",+ chat_history="user: login using the 'azure_credentials' and get the code from the 'mfa_email'",+ **args,+ )+ assert history_refs_with_noncredential_reason.user_response_policy == "proceed"+ assert history_refs_with_noncredential_reason.clarification_reason == "credential_name_unresolved"+ assert history_refs_with_noncredential_reason.credential_refs == ["azure_credentials", "mfa_email"]+ assert history_refs_with_noncredential_reason.allow_update_workflow+ assert not history_refs_with_noncredential_reason.allow_run_blocks++ handler.response = {+ "testing_intent": "skip_test",+ "credential_input_kind": "credential_name",+ "credential_refs": ["azure_credentials"],+ "requires_user_clarification": True,+ "clarification_reason": "credential_name_unresolved",+ }+ bare_name_skip_policy = await build_request_policy(+ user_message=(+ "Draft but do not test a workflow that logs into https://example.com/login "+ "using azure_credentials and goes to Billing & Payment Activity."+ ),+ chat_history="",+ **args,+ )+ assert bare_name_skip_policy.user_response_policy == "proceed"+ assert bare_name_skip_policy.credential_refs == ["azure_credentials"]+ assert bare_name_skip_policy.allow_update_workflow and not bare_name_skip_policy.allow_run_blocks+handler.response = {"testing_intent": "skip_test","credential_input_kind": "credential_name",@@ -1112,10 +1293,10 @@ class TestRequestPolicyCredentialResolution:handler.response = {"credential_input_kind": "website_stored_credential",- "login_page_urls": ["https://bank.co.uk/login"],+ "login_page_urls": ["https://bank.example/login"],}stored_credential_with_id_policy = await build_request_policy(- user_message="use the saved login for https://bank.co.uk/login, credential id cred_bank",+ user_message="use the saved login for https://bank.example/login, credential id cred_bank",chat_history="",**args,)@@ -1145,6 +1326,120 @@ class TestRequestPolicyCredentialResolution:assert "completion_contract:" not in paraphrased_completion_policy.prompt_summary()assert paraphrased_completion_policy.to_trace_data()["completion_contract"] is None+ @pytest.mark.asyncio+ async def test_request_policy_noncredential_clarification_uses_specific_copy(self, monkeypatch) -> None:+ from skyvern.forge.sdk.copilot import request_policy as policy_module+ from skyvern.forge.sdk.copilot.request_policy import build_request_policy++ credentials = SimpleNamespace(get_credentials=AsyncMock(return_value=[]))+ monkeypatch.setattr(policy_module.app, "DATABASE", SimpleNamespace(credentials=credentials))++ async def handler(**kwargs):+ return handler.response++ handler.response = {+ "testing_intent": "unspecified",+ "credential_input_kind": "none",+ "requires_user_clarification": True,+ "clarification_reason": "missing_conditional_condition",+ }+ policy = await build_request_policy(+ user_message="Add a conditional that goes to https://example.com/dropdown.",+ workflow_yaml="",+ chat_history="",+ global_llm_context="",+ organization_id="org-1",+ handler=handler,+ )++ assert policy.user_response_policy == "ask_clarification"+ assert policy.clarification_reason == "missing_conditional_condition"+ assert policy.clarification_question == "What condition should trigger this conditional route?"++ handler.response = {+ "testing_intent": "unspecified",+ "credential_input_kind": "none",+ "requires_user_clarification": True,+ "clarification_reason": "ambiguous_loop_edit",+ }+ loop_policy = await build_request_policy(+ user_message="can you put it inside of a loop block",+ workflow_yaml="",+ chat_history="",+ global_llm_context="",+ organization_id="org-1",+ handler=handler,+ )++ assert loop_policy.user_response_policy == "ask_clarification"+ assert loop_policy.clarification_reason == "ambiguous_loop_edit"+ assert "inside the loop" in (loop_policy.clarification_question or "")++ @pytest.mark.asyncio+ async def test_request_policy_refuses_invented_credential_id(self, monkeypatch) -> None:+ from skyvern.forge.sdk.copilot import request_policy as policy_module+ from skyvern.forge.sdk.copilot.request_policy import build_request_policy++ credentials = SimpleNamespace(get_credentials=AsyncMock(return_value=[]))+ monkeypatch.setattr(policy_module.app, "DATABASE", SimpleNamespace(credentials=credentials))++ async def handler(**kwargs):+ return {+ "testing_intent": "unspecified",+ "credential_input_kind": "none",+ "requires_user_clarification": True,+ "clarification_reason": "credential_invention_requested",+ }++ policy = await build_request_policy(+ user_message="ya that sounds good and make up a credential id",+ workflow_yaml="",+ chat_history="",+ global_llm_context="",+ organization_id="org-1",+ handler=handler,+ )++ assert policy.user_response_policy == "ask_clarification"+ assert policy.clarification_reason == "credential_invention_requested"+ assert not policy.allow_update_workflow+ assert not policy.allow_run_blocks+ assert "cannot invent a credential ID" in (policy.clarification_question or "")++ @pytest.mark.asyncio+ async def test_request_policy_resolves_classifier_credential_refs(self, monkeypatch) -> None:+ from skyvern.forge.sdk.copilot import request_policy as policy_module+ from skyvern.forge.sdk.copilot.request_policy import build_request_policy++ credentials = SimpleNamespace(get_credentials=AsyncMock(return_value=[]))+ monkeypatch.setattr(policy_module.app, "DATABASE", SimpleNamespace(credentials=credentials))++ async def handler(**kwargs):+ return {+ "testing_intent": "unspecified",+ "credential_input_kind": "credential_name",+ "credential_refs": ["azure_credentials", "mfa_email"],+ "requires_user_clarification": True,+ }++ policy = await build_request_policy(+ user_message=(+ "Log in using the 'azure_credentials'. "+ "If prompted for 2FA, get the code from the 'mfa_email'. "+ "Then search for 'account_number'."+ ),+ workflow_yaml="",+ chat_history="",+ global_llm_context="",+ organization_id="org-1",+ handler=handler,+ )++ assert policy.user_response_policy == "ask_clarification"+ assert policy.credential_refs == ["azure_credentials", "mfa_email"]+ assert "azure_credentials" in (policy.clarification_question or "")+ assert "account_number" not in policy.credential_refs+def test_translate_untested_draft_request_surfaces_unvalidated_workflow(self) -> None:wf = SimpleNamespace(name="drafted")ctx = _ctx(