Skip to content

Automate browser based workflows with AI

test(SKY-9716): add request policy classifier eval (#5921)

Andrew Neilson492fbd2006df
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
index a396450d1..4bf2133bc 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
@@ -1,6 +1,23 @@
Classify the latest Workflow Copilot request. Return only JSON with keys:
testing_intent=require_test|skip_test|unspecified; credential_input_kind=none|raw_secret|credential_id|credential_name|website_stored_credential|placeholder; credential_refs; login_page_urls; requires_user_clarification.
-Use skip_test for untested draft / I'll test it myself / just sketch it / draft only. Use raw_secret for inline login credentials, especially passwords, plus API keys, tokens, bearer headers, cookies, TOTP/OTP codes, private keys, card numbers, or CVVs.
+
+testing_intent:
+- skip_test: the user asks for an untested draft, says they will test it themselves, says draft only, just sketch it, no validation run, or skip testing.
+- require_test: the user asks to build/create and test/run/verify/end-to-end it, or asks for a complete workflow that should be validated.
+- unspecified: the user does not say whether Skyvern should test the workflow.
+
+credential_input_kind:
+- raw_secret: the user pasted inline login credentials or secrets, including passwords, API keys, tokens, bearer headers, cookies, TOTP/OTP codes, private keys, card numbers, or CVVs.
+- credential_id: the user provided one or more exact credential IDs beginning with `cred_`. Put every ID in credential_refs.
+- credential_name: the user gave an exact saved credential name, or vaguely says to use "my credentials", "my saved credentials", "credentials saved already", or an entry in the Credentials UI without enough website context. Put exact names in credential_refs; leave credential_refs empty when the reference is vague.
+- website_stored_credential: the user asks to use stored/saved credentials for a specific website or login page URL. Put URLs in login_page_urls.
+- placeholder: the user asks to add credential/password/username fields but explicitly says there are no real credentials yet.
+- none: the request mentions a login workflow but does not provide, request, or refer to any credential material.
+
+requires_user_clarification:
+- true only when the user refers to credentials but does not provide enough information to identify a saved credential or login page.
+- false for normal build/test/verify requests with no credential reference, for explicit credential IDs/names/URLs, for placeholders, and for skip_test requests.
+
Latest user message:
```{{ user_message }}```
Recent chat history:
diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.py
index 85d3f45ed..5f4160b1f 100644
--- a/skyvern/forge/sdk/copilot/request_policy.py
+++ b/skyvern/forge/sdk/copilot/request_policy.py
@@ -23,6 +23,10 @@ _KINDS = {"none", "raw_secret", "credential_id", "credential_name", "website_sto
_CREDENTIAL_ID_RE = re.compile(r"\bcred_[A-Za-z0-9][A-Za-z0-9_-]*\b")
_RAW_SECRET_PATTERNS = (
re.compile(r"\b(?:password|passcode|api[_ -]?key|secret|token|bearer|authorization)\s*[:=]\s*\S+", re.I),
+ re.compile(
+ r"\b(?:otp|totp|mfa|2fa|verification|auth(?:entication)? code)(?:\s+code)?\s*(?:is|[:=])?\s*\d{6,8}\b",
+ re.I,
+ ),
re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b"),
re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),
re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
@@ -140,7 +144,7 @@ async def _classify_request(
policy = _classification_from_raw(raw)
policy.credential_refs = _clean_list(policy.credential_refs + ids)
- if ids and policy.credential_input_kind in ("none", "placeholder"):
+ if ids and policy.credential_input_kind != "raw_secret":
policy.credential_input_kind = "credential_id"
return policy