Skip to content

Automate browser based workflows with AI

Align MCP/CLI browser guidance and harden eval baseline (#5503)

Marc Kelechavaec17530f43e4
diff --git a/skyvern/cli/mcp_tools/__init__.py b/skyvern/cli/mcp_tools/__init__.py
index 203708e37..9159add22 100644
--- a/skyvern/cli/mcp_tools/__init__.py
+++ b/skyvern/cli/mcp_tools/__init__.py
@@ -120,7 +120,12 @@ Skyvern is the complete browser MCP for AI agents. Use Skyvern for ALL browser i
DO NOT use Skyvern for: REST API calls (use curl/requests), downloading raw files (use wget/curl), \
fetching static JSON/XML endpoints (use WebFetch), or general web search (use WebSearch).
-## Task Classification — ALWAYS classify before choosing a tool
+## ALWAYS Start Here: Session + Classification
+
+**If a browser session is already open, keep using it. Otherwise start with:** skyvern_browser_session_create -> skyvern_navigate(url="...") -> [work] -> skyvern_browser_session_close()
+**Passwords:** NEVER type passwords via skyvern_type or skyvern_act. ALWAYS use skyvern_login with stored credentials.
+
+## Task Classification — classify before choosing a tool
| Classification | Signal | Tool | Cost | What Happens |
|---|---|---|---|---|
@@ -129,8 +134,18 @@ fetching static JSON/XML endpoints (use WebFetch), or general web search (use We
| Single action (known target) | "click #submit" | skyvern_click / skyvern_type | 0 LLM | Deterministic Playwright. No AI. Fastest. |
| Single action (unknown target) | "click the submit button" | skyvern_act | 2-3 LLM, no screenshots | No screenshots in reasoning. Economy a11y tree. For visual targets, use observe first. |
| Multi-step (simple, fast) | "fill the form and submit" | skyvern_observe + skyvern_execute | 0 Skyvern LLM | A11y tree + YOUR LLM plans from refs + batched primitives. Fast, cheap. |
-| Multi-step (complex) | "navigate a multi-page wizard" | skyvern_run_task | N LLM + screenshots | Full ForgeAgent: screenshot + DOM + Skyvern LLM + execute + verify + loop. Reliable. |
-| Repeated/production | "automate this weekly" | skyvern_workflow_create | Varies | Caching converts AI runs into deterministic scripts over time. |
+| Throwaway autonomous trial | "try this once", "see if this works" | skyvern_run_task | Higher | One-off autonomous agent for exploratory work. Do not use for reusable or multi-page production automations. |
+| Multi-step (complex) | "navigate a multi-page wizard" | skyvern_workflow_create (multi-block) | N LLM + screenshots | Build a workflow with one navigation block per step. Each block gets visual reasoning + verification. |
+| Repeated/production | "automate this weekly", "run every Monday", "schedule", "recurring" | skyvern_workflow_create + run | Varies | Caching converts AI runs into deterministic scripts over time (10-100x faster on repeat). |
+
+## Decision Rules (highest precedence)
+
+1. If the user gives a selector, id, XPath, or exact field target, use browser primitives -- not skyvern_act.
+2. If you only need a yes/no answer, use skyvern_validate -- not skyvern_extract or skyvern_act.
+3. If the work stays on one page and the UI is standard, prefer skyvern_observe + skyvern_execute.
+4. If the user says "try this once", "see if this works", or clearly wants a one-off exploratory trial, use skyvern_run_task.
+5. If the task spans multiple pages and is meant to be reusable, scheduled, repeatable, or explicitly "set up" as automation, use skyvern_workflow_create.
+6. Never type passwords. Always use skyvern_login with stored credentials.
## Quick Start by Classification
@@ -147,11 +162,14 @@ fetching static JSON/XML endpoints (use WebFetch), or general web search (use We
2. Your LLM plans which refs to interact with
3. skyvern_execute(steps=[{"tool":"click","params":{"ref":"e0"}}, {"tool":"type","params":{"ref":"e1","text":"hello"}}])
-**Multi-step (complex) — for visual complexity, multi-page flows:**
-skyvern_run_task(prompt="Complete the checkout flow", url="https://example.com/cart")
+**Throwaway autonomous trial:**
+skyvern_run_task(prompt="Try the checkout flow once and tell me whether it succeeds")
-**Repeated/production:**
-skyvern_workflow_create(...) -> skyvern_workflow_run(workflow_id="wpid_...") -> skyvern_workflow_status(run_id="wr_...")
+**Multi-step (complex) — build a workflow with one block per step:**
+skyvern_workflow_create(
+ definition='{"title":"Checkout","workflow_definition":{"blocks":[{"block_type":"navigation","label":"shipping","navigation_goal":"Fill shipping info"},{"block_type":"navigation","label":"payment","navigation_goal":"Select payment and submit"},{"block_type":"extraction","label":"confirm","data_extraction_goal":"Extract order number"}]}}',
+ format="json"
+) -> skyvern_workflow_run(workflow_id="wpid_...") -> skyvern_workflow_status(run_id="wr_...")
**QA testing:** Use the qa_test prompt to test frontend changes — reads git diff, generates + runs browser tests.
@@ -159,8 +177,8 @@ skyvern_workflow_create(...) -> skyvern_workflow_run(workflow_id="wpid_...") ->
1. **act has NO screenshots** in its LLM reasoning. It uses an economy accessibility tree. \
Fine for well-labeled elements. For visually complex targets, use skyvern_observe then skyvern_click with the ref.
-2. **observe+execute is DIFFERENT from run_task.** observe+execute uses the accessibility tree and YOUR LLM plans \
-(no Skyvern LLM calls, fast/cheap). run_task uses the full ForgeAgent with screenshots + visual reasoning (slow/reliable).
+2. **observe+execute is DIFFERENT from workflows.** observe+execute uses the accessibility tree and YOUR LLM plans \
+(no Skyvern LLM calls, fast/cheap). Workflows use the full ForgeAgent per block with screenshots + visual reasoning (reliable).
3. **validate is the cheapest AI option** for yes/no — prefer it over extract or act for boolean checks.
4. **extract DOES use screenshots** (unlike act) with a dedicated extraction LLM. Better than screenshot then read.
5. **NEVER type passwords** through skyvern_type or skyvern_act. Use skyvern_login with stored credentials.
@@ -174,7 +192,7 @@ These use AI and cost tokens, but cached results enable free re-execution.
- **skyvern_validate**(prompt) — boolean check WITH screenshots. Cheapest AI.
- **skyvern_observe**() — a11y tree snapshot with element refs (e0, e1...). No LLM calls.
- **skyvern_execute**(steps) — batch primitives using refs from observe. No Skyvern LLM calls.
-- **skyvern_run_task**(prompt) — full autonomous agent. Screenshots + visual reasoning + verification loop.
+- **skyvern_run_task**(prompt) — one-off exploratory agent. Do not use when the user is asking to set up, automate, schedule, or reuse the flow.
- **skyvern_login**(credential_id) — AI-powered login with stored credentials.
### Tier 2: Browser Primitives (use when you know the exact selector)
@@ -250,13 +268,41 @@ navigation (most common), extraction, for_loop, conditional, code, text_prompt,
wait, login, validation, http_request, send_email, file_download, file_upload. \
Call skyvern_block_schema() for full schemas.
+## Common Patterns
+
+**Logging in securely:**
+1. skyvern_credential_list -- find the credential
+2. skyvern_browser_session_create -- start session
+3. skyvern_navigate(url="https://login.example.com") -- go to login page
+4. skyvern_login(credential_id="cred_...") -- AI handles the full login flow
+5. skyvern_screenshot -- verify login succeeded
+
+**Debugging browser issues:**
+skyvern_browser_session_create -> skyvern_navigate -> perform actions -> \
+skyvern_console_messages(level="error") for JS errors, skyvern_network_requests for API calls
+
+**Testing feasibility before building a workflow:**
+Walk through the site interactively -- use skyvern_act on each page and skyvern_screenshot to verify. \
+Once confirmed, compose steps into a workflow with skyvern_workflow_create.
+
+## Writing Scripts (ONLY when user explicitly asks)
+
+Use the Skyvern Python SDK: from skyvern import Skyvern. \
+NEVER import from skyvern.cli.mcp_tools -- those are internal server modules.
+In verbose mode (--verbose), every tool response includes an sdk_equivalent field for script conversion.
+
+**Hybrid xpath+prompt pattern** -- the recommended approach for production scripts:
+ await page.click("xpath=//button[@id='submit']", prompt="the Submit button")
+ await page.fill("xpath=//input[@name='email']", "[email protected]", prompt="email input field")
+This tries the xpath first (fast, deterministic) and falls back to AI if the selector breaks. \
+To get xpaths, use skyvern_click -- its resolved_selector response field gives you the xpath.
+
## Critical Rules
1. Create a session (skyvern_browser_session_create) before any browser tool.
2. NEVER scrape by guessing API endpoints — use skyvern_navigate + skyvern_extract.
3. After page-changing actions, use skyvern_screenshot to verify.
4. NEVER type passwords — use skyvern_login with stored credentials.
5. NEVER create single-block workflows with long prompts — split into one block per step.
-6. For SDK scripts: from skyvern import Skyvern. NEVER import from skyvern.cli.mcp_tools.
""",
)
mcp.add_middleware(MCPTelemetryMiddleware())
diff --git a/skyvern/cli/skills/skyvern/SKILL.md b/skyvern/cli/skills/skyvern/SKILL.md
index b4dcc67ea..84e655d50 100644
--- a/skyvern/cli/skills/skyvern/SKILL.md
+++ b/skyvern/cli/skills/skyvern/SKILL.md
@@ -16,11 +16,22 @@ Skyvern uses AI to navigate and interact with websites. Every command below is a
| Quick inspection | "what does the page show?" | `skyvern browser extract` | 1 LLM + screenshots | Dedicated extraction LLM + schema validation + caching. |
| Single action (known target) | "click #submit" | `skyvern browser click/type` | 0 LLM | Deterministic Playwright. No AI. Fastest. |
| Single action (unknown target) | "click the submit button" | `skyvern browser act` | 2-3 LLM, no screenshots | No screenshots in reasoning. Economy a11y tree. For visual targets, use hybrid mode (selector + intent). |
-| Multi-step (simple, fast) | "fill the form and submit" | MCP: `observe + execute` | 0 Skyvern LLM | A11y tree + YOUR LLM plans from refs. Not available as CLI commands -- use MCP or `act`. |
-| Multi-step (complex) | "navigate a multi-page wizard" | `skyvern browser run-task` | N LLM + screenshots | Full ForgeAgent: screenshot + DOM + Skyvern LLM + execute + verify + loop. Reliable. |
-| Repeated/production | "automate this weekly" | `skyvern workflow create` | Varies | Caching converts AI runs into deterministic scripts over time. |
+| Same-page multi-step | "fill the form and submit" | `skyvern browser act` or primitive chain | 2-3 LLM or 0 LLM | Use `act` when labels are clear. Use click/type/select directly when you know selectors. |
+| Throwaway autonomous trial | "try this once", "see if this works" | `skyvern browser run-task` | Higher | One-off autonomous agent for exploration. Do not use for recurring or multi-page production automations. |
+| Multi-page or reusable automation | "navigate a multi-page wizard", "set this up", "automate this weekly" | `skyvern workflow create` + `run` | N LLM + screenshots | Build a workflow with one block per step. Each block gets visual reasoning, verification, and reusable run history. |
-## Step 2: Create a Session
+**MCP note:** if you are using the Skyvern MCP instead of the CLI, prefer `observe + execute` for same-page multi-step UI work. The CLI does not expose that pair directly.
+
+## Step 2: Apply These Decision Rules
+
+1. If the prompt includes a selector, id, XPath, or exact field target, use browser primitives -- not `act`.
+2. If you only need a yes/no answer, use `validate` -- not `extract` or `act`.
+3. If the work stays on one page and labels are clear, use `act` or a primitive chain.
+4. If the user says `try this once`, `see if this works`, or clearly wants a one-off exploratory trial, use `run-task`.
+5. If the task spans multiple pages and is meant to be reusable, scheduled, repeatable, or explicitly `set up` as automation, use `workflow create`.
+6. Never type passwords. Always use stored credentials with `skyvern browser login`.
+
+## Step 3: Create a Session
Every browser command needs a session. Create one first:
@@ -38,7 +49,7 @@ skyvern browser session connect --cdp "ws://localhost:9222"
Session state persists between commands. After `session create`, subsequent commands auto-attach.
Override with `--session pbs_...`. Close when done: `skyvern browser session close`.
-## Step 3: Execute by Classification
+## Step 4: Execute by Classification
### Quick check (yes/no)
@@ -81,23 +92,36 @@ skyvern browser act --prompt "Close the cookie banner, then click Sign In"
**Warning:** act has NO screenshots in its LLM reasoning. It uses an economy accessibility tree.
Fine for well-labeled elements. For visually complex targets, use MCP observe+click or hybrid mode.
-### Multi-step (simple, fast) -- MCP only
+### Same-page multi-step
+
+```bash
+skyvern browser act --prompt "Fill the shipping form and click Continue"
+```
-The `observe + execute` pattern uses the accessibility tree and YOUR LLM to plan, with no Skyvern
-LLM calls. This is MCP-only (not available as CLI commands). For CLI, use `act` with clear prompts
-or break into multiple click/type commands.
+Use `act` when the fields and buttons are clearly labeled and the flow stays on one page.
+If you need tighter control, break the work into `click`, `type`, `select`, `press-key`, and `wait`.
-### Multi-step (complex, reliable)
+### Throwaway autonomous trial
```bash
skyvern browser run-task \
- --prompt "Go to the pricing page and extract all plan names and prices" \
--url "https://example.com" \
- --schema '{"type":"object","properties":{"plans":{"type":"array"}}}'
+ --prompt "Check whether the checkout flow works end to end and extract the confirmation number"
+```
+
+Use `run-task` to prove feasibility or do one-off exploration. If the task becomes important enough
+to rerun, debug, or share, convert it to a workflow.
+
+### Multi-page or reusable automation — build a workflow with one block per step
+
+```bash
+skyvern workflow create --definition @checkout-workflow.yaml
+skyvern workflow run --id wpid_123 --wait
+skyvern workflow status --run-id wr_789
```
-Full ForgeAgent: screenshots + visual reasoning + verification + retry loop. Always uses engine 2.0.
-Flags: `--prompt` (required), `--url`, `--schema`, `--max-steps`, `--timeout` (default 180s).
+Each navigation block runs with visual reasoning + verification. Split complex flows into
+multiple blocks (one per page/step). First run uses AI; subsequent runs replay cached scripts.
### Repeated/production
@@ -111,7 +135,7 @@ Split into one block per step. Use **navigation** blocks for actions, **extracti
First run uses AI; subsequent runs replay a cached script (10-100x faster).
Set `--run-with agent` to force AI mode for debugging.
-## Step 4: Verify
+## Step 5: Verify
Always verify after page-changing actions:
@@ -121,7 +145,7 @@ skyvern browser validate --prompt "Was the form submitted successfully?" # bool
skyvern browser evaluate --expression "document.title" # JS state check
```
-## Step 5: Error Recovery
+## Step 6: Error Recovery
| Problem | Fix |
|---------|-----|
@@ -157,6 +181,34 @@ skyvern block validate --block-json @block.json # validate before creating
Engine: known path = 1.0 (default). Dynamic planning = 2.0. Split into multiple 1.0 blocks when in doubt.
Status lifecycle: `created -> queued -> running -> completed | failed | canceled | terminated | timed_out`
+## Common Patterns
+
+**Login flow:**
+```bash
+skyvern credential list # find credential ID
+skyvern browser session create
+skyvern browser navigate --url "https://login.example.com"
+skyvern browser login --url "https://login.example.com" --credential-id cred_123
+skyvern browser validate --prompt "Is the user logged in?"
+skyvern browser screenshot
+```
+
+**Pagination loop:**
+```bash
+skyvern browser extract --prompt "Extract all rows"
+skyvern browser validate --prompt "Is there a Next button that is not disabled?"
+# If true:
+skyvern browser act --prompt "Click the Next page button"
+# Repeat extraction. Stop when: no next button, duplicate first row, or max page limit.
+```
+
+**Debugging:**
+```bash
+skyvern browser screenshot # visual state
+skyvern browser evaluate --expression "document.title"
+skyvern browser evaluate --expression "document.querySelectorAll('table tr').length"
+```
+
## Agent Mode
All commands accept `--json` for structured output. Set `SKYVERN_NON_INTERACTIVE=1` to prevent prompts.
diff --git a/skyvern/cli/skills/skyvern/references/engines.md b/skyvern/cli/skills/skyvern/references/engines.md
index d8490f46a..636ae200e 100644
--- a/skyvern/cli/skills/skyvern/references/engines.md
+++ b/skyvern/cli/skills/skyvern/references/engines.md
@@ -1,17 +1,20 @@
-# Engine Choice for Quick Automation
+# Choosing Between `run_task` and Workflows
-Use one-off tools by default for short tasks.
+Default to workflows for real automation work. Reach for `skyvern_run_task` only when you are
+doing a throwaway trial and do not want to keep the result.
-## Prefer `skyvern_run_task`
+## Prefer `skyvern_workflow_create`
-- You need a throwaway automation now.
-- The task can complete in a small number of steps.
-- Reusability is not required.
+- The task spans multiple pages.
+- The user says `set this up`, `automate`, `workflow`, `reusable`, `repeat`, or `schedule`.
+- You want block-level observability, reruns, parameters, or cached scripts.
+- You expect to debug or hand the automation to someone else later.
-## Prefer a workflow instead
+## Prefer `skyvern_run_task`
-- The task will be rerun with different parameters.
-- You need branching, loops, or explicit block-level observability.
-- You need reproducible runs for operations teams.
+- You need a one-off exploratory trial right now.
+- The result is disposable and not worth saving.
+- You are checking feasibility before deciding whether to build a workflow.
-Rule of thumb: if you need to run the same automation twice with different inputs, move to `building-workflows`.
+Rule of thumb: if the task crosses page boundaries or sounds like real automation instead of a trial,
+build a workflow first.
diff --git a/skyvern/cli/skills/skyvern/references/tool-map.md b/skyvern/cli/skills/skyvern/references/tool-map.md
index 59899bd00..f56e8d0ce 100644
--- a/skyvern/cli/skills/skyvern/references/tool-map.md
+++ b/skyvern/cli/skills/skyvern/references/tool-map.md
@@ -1,10 +1,26 @@
# Tool Map by Outcome
-## Run a one-off task
+## Quick check or extraction
| Tool | Purpose |
|------|---------|
-| `skyvern_run_task` | Execute a single automation with a prompt and URL |
+| `skyvern_validate` | Answer a yes/no question about the current page |
+| `skyvern_extract` | Pull structured data from the current page |
+
+## Single action
+
+| Tool | Purpose |
+|------|---------|
+| `skyvern_click` | Click an element when you know the target |
+| `skyvern_type` | Type text into an element when you know the target |
+| `skyvern_select_option` | Select a dropdown option |
+| `skyvern_act` | Perform a natural-language action when you do not know exact selectors |
+
+## Throwaway autonomous trial
+
+| Tool | Purpose |
+|------|---------|
+| `skyvern_run_task` | Execute a one-off exploratory automation with a prompt and URL |
## Open and operate a website
@@ -25,16 +41,13 @@
| Tool | Purpose |
|------|---------|
-| `skyvern_click` | Click an element |
-| `skyvern_type` | Type text into an element |
-| `skyvern_select_option` | Select a dropdown option |
| `skyvern_hover` | Hover over an element |
| `skyvern_scroll` | Scroll the page |
| `skyvern_press_key` | Press a keyboard key |
| `skyvern_wait` | Wait for a condition or duration |
| `skyvern_evaluate` | Execute JavaScript in the page |
-## Build reusable automation
+## Build reusable or multi-page automation
| Tool | Purpose |
|------|---------|
diff --git a/tests/unit/test_mcp_workflow_guidance.py b/tests/unit/test_mcp_workflow_guidance.py
index d3a089b0c..d8d82f03a 100644
--- a/tests/unit/test_mcp_workflow_guidance.py
+++ b/tests/unit/test_mcp_workflow_guidance.py
@@ -16,7 +16,12 @@ def test_build_workflow_prompt_guides_text_prompt_defaults() -> None:
def test_mcp_instructions_guide_text_prompt_defaults() -> None:
# text_prompt guidance moved to build_workflow prompt; instructions now focus on task classification.
+ # Verify the classification table (the core of the rewritten instructions) is present.
assert "Task Classification" in mcp.instructions
+ assert "If a browser session is already open, keep using it." in mcp.instructions
+ assert '"try this once"' in mcp.instructions
+ assert "skyvern_run_task" in mcp.instructions
+ assert "skyvern_workflow_create" in mcp.instructions
assert "skyvern_validate" in mcp.instructions
assert "skyvern_extract" in mcp.instructions
assert "skyvern_act" in mcp.instructions