Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
Bundle QA skill in pip package and auto-install during setup (#4985)
Marc Kelechava58d9b980d2a7
diff --git a/fern/integrations/cli.mdx b/fern/integrations/cli.mdxindex eb06d19b8..76a71da84 100644--- a/fern/integrations/cli.mdx+++ b/fern/integrations/cli.mdx@@ -159,14 +159,17 @@ skyvern block validate --file block.json # Validate a block definitionRegister Skyvern's MCP server with your AI coding tool in one command:```bash-skyvern setup claude-code # Register with Claude Code (global)+skyvern setup claude-code # Register with Claude Code + install skills (/qa, etc.)skyvern setup claude-code --project # Register with Claude Code (project-level)+skyvern setup claude-code --skip-skills # MCP only, no skillsskyvern setup claude-desktop # Register with Claude Desktopskyvern setup cursor # Register with Cursorskyvern setup windsurf # Register with Windsurfskyvern setup codex # Register with Codex```+`skyvern setup claude-code` writes the MCP config **and** copies bundled skills (including `/qa`) into your project's `.claude/skills/` directory. Use `--skip-skills` to opt out.+### Other```bash@@ -190,30 +193,34 @@ The CLI and MCP server share the same underlying logic. The CLI is for humans an## Skills-Skills are bundled reference markdown files that teach AI coding tools how to use Skyvern. They are **not** the same as MCP tools — they are documentation that an AI agent can load to learn the CLI and API.+Skills are bundled markdown files that teach AI coding tools how to use Skyvern. They ship with `pip install skyvern` and are **automatically installed** when you run `skyvern setup claude-code`.++| Skill | What it does |+|-------|-------------|+| **qa** | Reads your `git diff`, generates browser tests, runs them against your dev server, reports pass/fail with screenshots. Invoke with `/qa` in Claude Code. |+| **skyvern** | CLI reference covering all browser automation, workflow, and credential commands. |+| **testing** | Smoke tests for verifying Skyvern deployments. |```bashskyvern skill list # List available skills-skyvern skill show skyvern # Render a skill in the terminal-skyvern skill path skyvern # Print the absolute path to a skill file-skyvern skill path # Print the skills directory-skyvern skill copy --output ./docs # Copy all skills to a local directory-skyvern skill copy skyvern -o . # Copy a single skill+skyvern skill show qa # Render a skill in the terminal+skyvern skill copy --output .claude/skills # Copy all skills manually+skyvern skill copy qa -o .claude/skills # Copy a single skill```<Accordion title="Loading skills into AI tools">-Skills are plain markdown files. You can load them into any AI coding tool that supports custom instructions:+Skills are plain markdown files. Load them into any AI coding tool that supports custom instructions:-**Claude Code** — add the skill path as a custom instructions file or use `skyvern setup claude-code` which configures MCP (the richer integration path).+**Claude Code** — run `skyvern setup claude-code`, which registers the MCP server **and** installs skills into `.claude/skills/`. The `/qa` skill is immediately available.-**Codex** — copy the skill into your project's `.codex/skills/` directory:+**Codex** — copy skills into your project's `.codex/skills/` directory:```bash-skyvern skill copy skyvern -o .codex/skills/+skyvern skill copy -o .codex/skills/```-**Any tool** — point your tool at the file path returned by `skyvern skill path skyvern`.+**Any tool** — point your tool at the file path returned by `skyvern skill path qa`.</Accordion>diff --git a/fern/integrations/mcp.mdx b/fern/integrations/mcp.mdxindex b4a99cd7d..ec81ee8f6 100644--- a/fern/integrations/mcp.mdx+++ b/fern/integrations/mcp.mdx@@ -267,6 +267,33 @@ This is especially powerful during development: make a code change, then ask youAlways use `--api-key` when exposing your browser via tunnel. Without it, anyone with the ngrok URL has full browser control.</Warning>+## QA your frontend changes++If you install the Skyvern CLI, `skyvern setup claude-code` registers the MCP server **and** installs Claude Code skills — including `/qa`, which reads your git diff, generates browser tests, and runs them against your dev server.++```bash+pip install skyvern+skyvern setup claude-code+```++Then make a frontend change and type `/qa` in Claude Code. It will:++1. Read your `git diff` and the full source of each changed file+2. Generate targeted test cases (plus regression tests for adjacent behavior)+3. Open a browser against your running dev server+4. Run the tests — navigate, click, fill forms, check the DOM+5. Report pass/fail with screenshots++For localhost testing, start a local browser first:++```bash+skyvern browser serve --tunnel+```++<Tip>+The `/qa` skill uses `skyvern_evaluate` for fast DOM assertions (~10ms each) and `skyvern_act` for natural-language interactions. Tests are generated fresh from each diff — no test files to maintain.+</Tip>+## Troubleshooting<Accordion title="Invalid API key or 401 errors">diff --git a/skyvern/cli/mcp_tools/prompts.py b/skyvern/cli/mcp_tools/prompts.pyindex a1518cf86..4d76a1dff 100644--- a/skyvern/cli/mcp_tools/prompts.py+++ b/skyvern/cli/mcp_tools/prompts.py@@ -458,8 +458,10 @@ def extract_data(# qa_test# ----------------------------------------------------------------------------# NOTE: This content is also maintained in .claude/skills/qa/SKILL.md-# for the /qa Claude Code skill. Keep both in sync when updating.+# NOTE: This content is maintained in three places — keep all in sync:+# 1. skyvern/cli/skills/qa/SKILL.md (bundled with pip package — canonical)+# 2. .claude/skills/qa/SKILL.md (project-local copy for this repo)+# 3. skyvern/cli/mcp_tools/prompts.py (QA_TEST_CONTENT — this file)QA_TEST_CONTENT = """\# QA — Test Frontend Changes in a Real Browserdiff --git a/skyvern/cli/setup_commands.py b/skyvern/cli/setup_commands.pyindex c239ad88d..bd2c61b14 100644--- a/skyvern/cli/setup_commands.py+++ b/skyvern/cli/setup_commands.py@@ -5,6 +5,7 @@ from __future__ import annotationsimport jsonimport osimport platform+import shutilimport sysfrom pathlib import Pathfrom urllib.parse import urlparse@@ -14,6 +15,7 @@ from dotenv import load_dotenvfrom rich.syntax import Syntaxfrom skyvern.cli.console import console+from skyvern.cli.skill_commands import get_skill_dirsfrom skyvern.utils.env_paths import resolve_backend_env_path# NOTE: skyvern/cli/mcp.py has older setup_*_config() helpers called from@@ -251,6 +253,47 @@ def _run_setup(_upsert_mcp_config(config_path, tool_name, entry, dry_run=dry_run, yes=yes)+def _install_skills(project_dir: Path, dry_run: bool = False) -> None:+ """Install bundled skills into a project's .claude/skills/ directory.++ Skips skills that already exist at the destination (non-destructive).+ """+ skills_dst = project_dir / ".claude" / "skills"+ dirs = get_skill_dirs()+ if not dirs:+ return++ installed: list[str] = []+ skipped: list[str] = []+ failed: list[str] = []+ ignore = shutil.ignore_patterns("__pycache__", "*.pyc")+ for d in dirs:+ target = skills_dst / d.name+ if target.exists():+ skipped.append(d.name)+ continue+ if not dry_run:+ try:+ target.parent.mkdir(parents=True, exist_ok=True)+ shutil.copytree(d, target, ignore=ignore)+ except OSError as e:+ console.print(f"[yellow]Warning: failed to install skill '{d.name}': {e}[/yellow]")+ failed.append(d.name)+ continue+ installed.append(d.name)++ if installed:+ names = ", ".join(installed)+ if dry_run:+ console.print(f"\n[yellow]Dry run — would install skills: {names}[/yellow]")+ else:+ console.print(f"\n[green]Installed skills to {skills_dst}: {names}[/green]")+ if "qa" in installed:+ console.print("[bold]Tip:[/bold] Make a frontend change and type /qa to test it in a real browser.")+ if skipped:+ console.print(f"[dim]Skills already installed: {', '.join(skipped)}[/dim]")++# ---------------------------------------------------------------------------# Commands# ---------------------------------------------------------------------------@@ -288,11 +331,15 @@ def setup_claude_code(use_python_path: bool = _python_path_opt,url: str | None = _url_opt,project: bool = typer.Option(False, "--project", help="Write to .mcp.json in current dir instead of global config"),+ skip_skills: bool = typer.Option(False, "--skip-skills", help="Don't install Claude Code skills (e.g. /qa)"),) -> None:- """Register Skyvern MCP with Claude Code (remote by default)."""+ """Register Skyvern MCP with Claude Code and install skills (remote by default)."""config_path = Path.cwd() / ".mcp.json" if project else _claude_code_global_config_path()_run_setup("Claude Code", config_path, api_key, dry_run, yes, local, use_python_path, url)+ if not skip_skills:+ _install_skills(Path.cwd(), dry_run=dry_run)+@setup_app.command("cursor")def setup_cursor(diff --git a/skyvern/cli/skill_commands.py b/skyvern/cli/skill_commands.pyindex a52011f20..666a3706a 100644--- a/skyvern/cli/skill_commands.py+++ b/skyvern/cli/skill_commands.py@@ -19,7 +19,7 @@ SKILLS_DIR = Path(__file__).parent / "skills"_FRONTMATTER_RE = re.compile(r"^---\n(.*?)\n---", re.DOTALL)-def _get_skill_dirs() -> list[Path]:+def get_skill_dirs() -> list[Path]:"""Return sorted list of skill directories (those containing SKILL.md)."""if not SKILLS_DIR.exists():return []@@ -60,7 +60,7 @@ def _extract_description(skill_md: Path) -> str:@skill_app.command("list")def skill_list() -> None:"""List all bundled skills."""- dirs = _get_skill_dirs()+ dirs = get_skill_dirs()if not dirs:console.print("[red]No skills found in package. Re-install skyvern.[/red]")raise typer.Exit(code=1)@@ -120,7 +120,7 @@ def skill_copy(shutil.copytree(src, target, dirs_exist_ok=overwrite, ignore=_ignore)console.print(f"[green]Copied skill '{name}' to {target.resolve()}[/green]")else:- dirs = _get_skill_dirs()+ dirs = get_skill_dirs()if not dirs:console.print("[red]No skills found in package. Re-install skyvern.[/red]")raise typer.Exit(code=1)diff --git a/skyvern/cli/skills/README.md b/skyvern/cli/skills/README.mdindex 52726b185..38179798b 100644--- a/skyvern/cli/skills/README.md+++ b/skyvern/cli/skills/README.md@@ -1,21 +1,29 @@# Skyvern Skills Package-AI-powered browser automation skill for coding agents. Bundled with `pip install skyvern`.+AI-powered browser automation skills for coding agents. Bundled with `pip install skyvern`.## Quick Start```bashpip install skyvernexport SKYVERN_API_KEY="YOUR_KEY" # get one at https://app.skyvern.com++# Set up MCP + install skills in one step:+skyvern setup claude-code```-The skill teaches CLI commands via `skyvern <command>` invocations. For richer-AI-coding-tool integration, run `skyvern setup claude-code --project` to enable-MCP (Model Context Protocol) with auto-tool-calling.+`skyvern setup claude-code` registers the Skyvern MCP server and installs these+skills into your project's `.claude/skills/` directory automatically.## What's Included-A single `skyvern` skill covering all browser automation capabilities:+### qa+QA test your frontend changes in a real browser. Reads your `git diff`, generates+targeted browser tests, runs them against your local dev server, and reports+pass/fail with screenshots. Invoke with `/qa` in Claude Code.++### skyvern+CLI reference covering all browser automation capabilities:- Browser session lifecycle (create, navigate, close)- AI actions: act, extract, validate, screenshot@@ -26,19 +34,27 @@ A single `skyvern` skill covering all browser automation capabilities:- Block schema discovery and validation- Debugging with screenshot + validate loops+### testing+Smoke-test skill for verifying Skyvern deployments.+## Structure```+qa/+ SKILL.md Diff-driven frontend QA testingskyvern/SKILL.md Main skill file (CLI-first, all capabilities)references/ 17 deep-dive reference filesexamples/ Workflow JSON examples+testing/+ SKILL.md Deployment smoke testing```-## Install to a Project+## Manual Install++If you prefer to install skills without running setup:```bash-# Copy skill files to your projectskyvern skill copy --output .claude/skillsskyvern skill copy --output .agents/skills```diff --git a/skyvern/cli/skills/qa/SKILL.md b/skyvern/cli/skills/qa/SKILL.mdnew file mode 100644index 000000000..9d3d19431--- /dev/null+++ b/skyvern/cli/skills/qa/SKILL.md@@ -0,0 +1,270 @@+---+name: qa+description: "QA test your frontend changes in a real browser. Reads your git diff, generates targeted browser tests, runs them against your local dev server, and reports pass/fail with screenshots."+---++# QA — Test Your Frontend Changes in a Real Browser++<!-- NOTE: This content is maintained in three places — keep all in sync:+ 1. skyvern/cli/skills/qa/SKILL.md (bundled with pip package — canonical)+ 2. .claude/skills/qa/SKILL.md (project-local copy for this repo)+ 3. skyvern/cli/mcp_tools/prompts.py (QA_TEST_CONTENT for the MCP prompt) -->++You changed some code. This skill reads your diff, understands what UI was affected, opens+a real browser against your running dev server, and tests that your changes actually work.++## Quick Start++```+/qa # Diff-based: test what you changed+/qa http://localhost:3000 # Same, explicit URL+/qa -- test the checkout flow # Targeted: test specific behavior+```++## How It Works++1. **Read your code changes** (`git diff`) to understand what you modified+2. **Read the changed files** to understand the UI: routes, components, props, text+3. **Generate targeted test cases** based on what the code actually does+4. **Open a browser** against your running dev server+5. **Run the tests** — navigate, interact, assert, screenshot+6. **Report** pass/fail with evidence++This is NOT a generic website crawler. It tests YOUR changes specifically.++---++## Step 1: Understand the Changes++### Get the diff++```bash+# What files changed?+git diff --name-only HEAD~1 # vs last commit (if changes are committed)+git diff --name-only # vs working tree (if uncommitted)++# Full diff for context+git diff HEAD~1 # or git diff for uncommitted+```++Pick whichever diff has content. If both are empty, there's nothing to QA.++### Read the changed files++For every changed frontend file (`.tsx`, `.jsx`, `.ts`, `.js`, `.css`, `.html`):+- Read the FULL file (not just the diff) to understand the component+- Look for: route paths, component names, text labels, form fields, button labels,+ API endpoints called, conditional rendering, error states++### Classify the changes++| Change Type | What to Test |+|-------------|-------------|+| New component/page | Navigate to it, verify it renders, interact with its elements |+| Modified component | Navigate to it, verify the specific change works (new button, new text, new behavior) |+| Styling changes | Navigate, screenshot, verify layout isn't broken |+| API integration | Navigate, trigger the action, verify the API call works (check network, verify UI updates) |+| Form changes | Fill the form, submit, verify validation and success states |+| Route changes | Navigate to old and new routes, verify routing works |+| Shared component (used in many places) | Test 2-3 pages that use it |+| Bug fix | Reproduce the original bug scenario, verify it's fixed |++### Generate test cases++For each changed file, write specific test cases. Example:++If the diff shows changes to `LoginForm.tsx` adding a "Forgot password" link:+```+Test 1: Login page renders the new "Forgot password" link+ - Navigate to /login+ - Assert: link with text "Forgot password" exists+ - Click it+ - Assert: navigated to /forgot-password (or modal appeared)++Test 2: Login form still works (regression)+ - Navigate to /login+ - Verify email and password inputs exist+ - Submit empty form, verify validation errors appear+```++**Be specific.** Don't write "verify the page works." Write "verify the 'Forgot password'+link navigates to /forgot-password."++---++## Step 2: Find the Dev Server++If the user provided a URL, use it. Otherwise, auto-detect:++```+# Try common dev server ports+# 5173 (Vite), 3000 (Next/CRA), 3001, 8080, 8000, 4200 (Angular)+```++Navigate to each until one responds. If none respond, tell the user:+"Start your dev server first, then run `/qa` again."++---++## Step 3: Connect to a Browser++```+skyvern_browser_session_create(local=true, headless=false, timeout=15)+```++Use `local=true` so it can reach `localhost`. Use `headless=false` so the user can watch.++If local fails, fall back to cloud (warn that URL must be publicly accessible):+```+skyvern_browser_session_create(timeout=15)+```++---++## Step 4: Run the Tests++For each test case generated in Step 1:++### Navigate+```+skyvern_navigate(url="http://localhost:<port>/<route>")+```++### Health gate (after every navigate, ~10ms)+```+skyvern_evaluate(expression="(() => {+ const errors = [];+ const body = document.body?.innerText || '';+ if (body.includes('Something went wrong')) errors.push('error_message');+ if (body.includes('Cannot read properties')) errors.push('js_error_in_ui');+ if (/\bundefined\b/.test(body) && !/\bif\b|\btypeof\b|\bdocument|tutorial|example/i.test(body) && body.length < 5000) errors.push('undefined_text');+ if (body.includes('connection refused')) errors.push('connection_refused');+ if (/sign.?in|log.?in|auth/i.test(window.location.pathname)) errors.push('auth_redirect');+ if (document.querySelector('[role=\"alert\"]')) errors.push('alert_element');+ if (!document.querySelector('main, [role=\"main\"], nav, header, h1, h2, [class*=\"layout\" i], [class*=\"page\" i], [class*=\"app\" i]'))+ errors.push('blank_page');+ return JSON.stringify({ pass: errors.length === 0, errors });+})()")+```++### Assert with DOM queries (prefer `skyvern_evaluate` — fast, deterministic)+```+# Element exists+skyvern_evaluate(expression="!!document.querySelector('a[href=\"/forgot-password\"]')")++# Text content+skyvern_evaluate(expression="document.querySelector('h1')?.textContent?.trim()")++# Element count+skyvern_evaluate(expression="document.querySelectorAll('.card').length")++# URL after navigation+skyvern_evaluate(expression="window.location.pathname")+```++### Interact (use `skyvern_act` for natural language actions)+```+skyvern_act(prompt="Click the 'Forgot password' link")+skyvern_act(prompt="Fill the email field with '[email protected]' and click Submit")+skyvern_act(prompt="Open the dropdown menu and select 'Settings'")+```++### Visual checks (use `skyvern_validate` only when DOM queries aren't enough)+```+skyvern_validate(prompt="The login form shows email and password fields with a blue Submit button")+```++### Screenshot (after every significant action)+```+skyvern_screenshot()+```++### Failed network requests (once per page)+```+skyvern_evaluate(expression="(() => {+ const entries = performance.getEntriesByType('resource').filter(e => e.responseStatus >= 400);+ return JSON.stringify({ failed: entries.map(e => ({ url: e.name, status: e.responseStatus })).slice(0, 5) });+})()")+```++---++## Step 5: Report Results++```markdown+## QA Report++### Changes Tested+Files: `LoginForm.tsx`, `ForgotPassword.tsx`+Diff summary: Added "Forgot password" link to login form, new /forgot-password page++### Results+| # | Test | Result | Screenshot |+|---|------|--------|------------|+| 1 | Login page renders "Forgot password" link | PASS | screenshot_1 |+| 2 | Clicking link navigates to /forgot-password | PASS | screenshot_2 |+| 3 | Forgot password page renders form | PASS | screenshot_3 |+| 4 | Login form still works (regression) | PASS | screenshot_4 |+| 5 | Empty form shows validation errors | FAIL | screenshot_5 |++### Issues Found+1. **Empty login form submits without validation** — Submitting with no email/password+ doesn't show error messages. The form submits and the page reloads.+ Expected: validation errors. Screenshot: screenshot_5++### Network+- No failed requests detected++### Verdict+4/5 tests passed. 1 issue found: missing form validation on empty submit.+```++---++## Tool Selection++| What you need | Tool | Speed |+|---------------|------|-------|+| Check element exists, text, count, URL | `skyvern_evaluate` | ~10ms |+| Click, type, fill forms, multi-step interaction | `skyvern_act` | 5-30s |+| "Does this look right?" visual check | `skyvern_validate` | 15-50s |+| Get structured data from a page | `skyvern_extract` | 15-50s |+| Screenshot | `skyvern_screenshot` | ~1s |+| Wait for async content | `skyvern_wait` | varies |++**Default to `skyvern_evaluate` for assertions.** Only use `skyvern_validate` when you+can't express the check as a DOM query (visual layout, "does this look like a dashboard").++---++## Error Handling++| Problem | Action |+|---------|--------|+| No git diff found | Ask user what they want to test, fall back to explore mode |+| Dev server not running | Tell user to start it. Suggest common commands (npm run dev, etc.) |+| Auth redirect on page | Report it. Ask if they want to provide credentials or skip that route. |+| Component doesn't render | Screenshot + check console. Report with the specific error. |+| Session create fails | Try cloud fallback. Warn about URL accessibility. |++## Session Cleanup++ALWAYS close the session when done, even if errors occurred:+```+skyvern_browser_session_close()+```++---++## Fallback: Explore Mode++If there's no git diff (user just wants a general QA pass), fall back to exploring:++1. Navigate to the root URL+2. Extract nav links and page structure with `skyvern_extract`+3. Visit each major route, health gate + screenshot+4. Test interactive elements (forms, buttons, links)+5. Report findings++But the primary mode is **diff-driven**. The agent should always try to read the code+changes first.