Skip to content

Automate browser based workflows with AI

docs: update smoke-test skill with Chrome DevTools MCP and screenshot evidence (#6193)

Aaron Perez234ef86d5bbf
diff --git a/skyvern/cli/skills/smoke-test/SKILL.md b/skyvern/cli/skills/smoke-test/SKILL.md
index 9e89f772c..feb48357a 100644
--- a/skyvern/cli/skills/smoke-test/SKILL.md
+++ b/skyvern/cli/skills/smoke-test/SKILL.md
@@ -1,11 +1,11 @@
---
name: smoke-test
-description: "Run smoke tests against a deployed or local app based on your git diff. Each test uses Skyvern browser tools (navigate, act, validate, screenshot) and reports a pass/fail table as a PR comment."
+description: "Run smoke tests against a deployed or local app based on your git diff. Each test uses Skyvern browser tools (navigate, act, validate, screenshot) with Chrome DevTools MCP as fallback. Posts screenshot evidence as PR comments."
---
# Smoke Test — CI-Oriented Validation via Skyvern Browser Tools
-Read the diff, classify what changed, start the app, and run targeted smoke tests via Skyvern browser tools (`skyvern_navigate`, `skyvern_act`, `skyvern_validate`, `skyvern_screenshot`) — the same tools /qa uses, formatted for CI and PR comments.
+Read the diff, classify changes, start the app, and run targeted smoke tests via Skyvern browser tools with Chrome DevTools MCP as fallback.
<!-- NOTE: This content is maintained in two places — keep in sync:
1. skyvern/cli/skills/smoke-test/SKILL.md (bundled with pip — canonical)
@@ -15,9 +15,11 @@ Read the diff, classify what changed, start the app, and run targeted smoke test
mirror those fixes here. -->
You changed code. This skill reads the diff, generates targeted smoke tests, and runs
-each one via Skyvern browser tools — navigate, act, validate, screenshot. It is /qa's
-CI companion: same diff-reading, same classification, same app startup, same browser
-tools, formatted for CI output and PR comments.
+each one via Skyvern browser tools - navigate, act, validate, screenshot. If Skyvern
+tools are unavailable (no Skyvern MCP), fall back to Chrome DevTools MCP, which connects
+to a real Chrome instance and supports authenticated sessions. It is /qa's CI companion:
+same diff-reading, same classification, same app startup, formatted for CI output and
+PR comments with screenshot evidence on every test.
## Quick Start
@@ -25,6 +27,7 @@ tools, formatted for CI output and PR comments.
/smoke-test # Diff-driven, auto-detect everything
/smoke-test https://staging.example.com # Explicit app URL
/smoke-test -- focus on the settings page
+/smoke-test --pr 11488 # Target a specific PR
```
## How It Works
@@ -32,12 +35,14 @@ tools, formatted for CI output and PR comments.
1. Read git diff (reused from /qa)
2. Classify changes → identify testable surfaces (reused from /qa)
3. Choose validation strategy (reused from /qa)
-4. Start the app if needed (reused from /qa)
-5. Generate 3-8 smoke test cases as action sequences (happy paths only)
-6. Run each test via browser tools: navigate → act → validate → screenshot
-7. Collect results
-8. Report | Flow | Result | Evidence | table
-9. Post to PR if GITHUB_TOKEN available
+4. Pick browser backend (Chrome DevTools MCP or Skyvern)
+5. Handle auth (connect to authenticated session or bypass)
+6. Start the app if needed (reused from /qa)
+7. Generate 3-8 smoke test cases as action sequences (happy paths only)
+8. Run each test via browser tools: navigate → act → validate → screenshot
+9. Collect results with screenshot evidence
+10. Report with embedded screenshots
+11. Post to PR with images
## Step 1: Understand the Changes
@@ -109,6 +114,48 @@ for the changed logic. Only run smoke tests if the change affects exposed behavi
Validate the backend first, then run frontend smoke tests against the flow that depends on it.
If the backend contract is broken, frontend results are not trustworthy.
+## Step 3b: Pick Browser Backend
+
+Try Skyvern browser tools first. Fall back to Chrome DevTools MCP if Skyvern MCP is
+unavailable.
+
+### Skyvern browser tools (primary)
+
+Use `skyvern_browser_session_create`, `skyvern_navigate`, `skyvern_act`,
+`skyvern_validate`, `skyvern_screenshot`. Works in CI and locally. Create a `local=true`
+session to reach localhost.
+
+### Chrome DevTools MCP (fallback)
+
+If Skyvern MCP tools are not available, invoke the `/chrome-devtools` skill to learn the
+tool API. Key tools:
+
+- `list_pages` / `new_page` / `navigate_page` - page management
+- `take_screenshot` - capture viewport (save to a path within workspace roots)
+- `take_snapshot` - get page structure with element `uid`s for interaction
+- `click` / `fill` / `press_key` - interact with elements by `uid`
+- `evaluate_script` - run JS for health gates and data extraction
+- `wait_for` - wait for content to load
+
+Chrome DevTools MCP connects to a real Chrome instance with the user's profile, so
+**authenticated sessions carry over** - no separate auth step needed if the user is
+already logged in. Useful when the app requires auth that Skyvern sessions can't provide.
+
+## Step 3c: Handle Auth
+
+Check whether the app requires authentication:
+
+1. **Chrome DevTools MCP with existing session**: Call `list_pages`. If the user already
+ has the app open and authenticated, `navigate_page` inherits their session. No auth
+ step needed.
+2. **User needs to log in**: Call `new_page` or `navigate_page` to the app URL. If a
+ login page appears, tell the user: "Chrome is open to the login page. Please
+ authenticate, then tell me when you're done." Wait for confirmation before proceeding.
+3. **Auth bypass**: If the repo has a local auth-bypass mechanism (env var or dev
+ branch), use it. Check the project's dev setup docs or CLAUDE.md for details.
+4. **CI / headless**: Use Skyvern browser tools with `local=true` against an
+ unauthenticated dev server, or test only public-facing pages.
+
## Step 4: Start the App
If the user provided a URL argument, skip startup and use that URL directly.
@@ -164,7 +211,7 @@ Test: Login redirect works
## Step 6: Run Tests via Browser Tools
-### Set up a browser session first
+### Option A: Skyvern browser tools (primary)
For localhost URLs, create a local browser session:
@@ -178,17 +225,14 @@ For publicly reachable URLs, create a cloud session instead:
skyvern_browser_session_create(timeout=15)
```
-### Execute each test
-
For each test case, run its action sequence. Every test starts with `skyvern_navigate`:
```text
skyvern_navigate(url="http://localhost:5173/settings")
```
-**CRITICAL: Always include the `url` parameter in every `skyvern_navigate` call.** Never
-omit it and rely on the current page — this prevents test-to-test state bleed. Each test
-must navigate fresh.
+**CRITICAL: Always include the `url` parameter in every `skyvern_navigate` call.**
+Never omit it and rely on the current page - this prevents test-to-test state bleed.
After navigation, run the health gate to catch broken pages early:
@@ -208,8 +252,8 @@ skyvern_evaluate(expression="(() => {
})()")
```
-If the health gate fails, mark the test as FAIL and move on. Otherwise, continue with the
-test's action steps:
+If the health gate fails, mark the test as FAIL and move on. Otherwise, continue with
+the test's action steps:
```text
skyvern_act(prompt="Fill Company Name with 'Test Corp', click Save")
@@ -217,15 +261,79 @@ skyvern_validate(prompt="Success toast visible, no error state")
skyvern_screenshot()
```
+### Option B: Chrome DevTools MCP (fallback)
+
+If Skyvern MCP tools are unavailable, use Chrome DevTools MCP. Invoke the
+`/chrome-devtools` skill first.
+
+No session setup needed - the browser launches automatically on first tool call.
+For each test:
+
+```text
+1. navigate_page(url="http://localhost:8080/workflows")
+2. wait_for(text="Workflows", timeout=5000)
+3. take_snapshot() # get element UIDs
+4. click(uid="<target-element>") # interact
+5. take_screenshot(filePath="/path/within/workspace/test-name.png")
+```
+
+Run the health gate via `evaluate_script`:
+
+```text
+evaluate_script(function="() => {
+ const errors = [];
+ const body = document.body?.innerText || '';
+ if (body.includes('Something went wrong')) errors.push('error_message');
+ if (body.includes('Cannot read properties')) errors.push('js_error_in_ui');
+ if (/\\bundefined\\b/.test(body) && !/\\bif\\b|\\btypeof\\b|\\bdocument|tutorial|example/i.test(body) && body.length < 5000) errors.push('undefined_text');
+ if (body.includes('connection refused')) errors.push('connection_refused');
+ if (/sign.?in|log.?in|auth/i.test(window.location.pathname)) errors.push('auth_redirect');
+ if (document.querySelector('[role=\"alert\"]')) errors.push('alert_element');
+ if (!document.querySelector('main, [role=\"main\"], nav, header, h1, h2, [class*=\"layout\" i], [class*=\"page\" i], [class*=\"app\" i]'))
+ errors.push('blank_page');
+ return JSON.stringify({ pass: errors.length === 0, errors });
+}")
+```
+
+**Screenshot paths** must be within the workspace roots reported by the MCP server.
+
+Chrome DevTools MCP connects to a real Chrome instance with the user's profile, so
+authenticated sessions carry over. If the user is already logged in, no auth step needed.
+If a login page appears, tell the user to authenticate in the browser window and wait.
+
### Collect results
For each test, record:
-- **result** — PASS or FAIL based on `skyvern_validate` outcome and health gate
-- **evidence** — one-line description of what was observed (from validate result or health gate errors)
+- **result** - PASS or FAIL based on validation outcome and health gate
+- **evidence** - one-line description of what was observed
+- **screenshot** - file path to the captured screenshot (required for every test)
## Step 7: Report Results
+The report must include embedded screenshots. Upload screenshot files to GitHub
+and reference them as markdown images.
+
+### Upload screenshots to GitHub
+
+Look up the PR number and short SHA first so filenames are stable:
+
+```bash
+PR_NUMBER=$(gh pr view --json number -q '.number' 2>/dev/null || echo "draft")
+SHORT_SHA=$(git rev-parse --short HEAD)
+
+# Create a release for screenshot assets (once per repo)
+gh release create qa-screenshots --repo OWNER/REPO --title "QA Screenshots" --notes "Asset store for smoke test screenshots" 2>/dev/null || true
+
+# Upload each screenshot (namespaced to avoid clobbering prior runs)
+gh release upload qa-screenshots "/path/to/screenshot.png#pr${PR_NUMBER}-${SHORT_SHA}-screenshot.png" --repo OWNER/REPO --clobber
+```
+
+The download URL follows the pattern:
+`https://github.com/OWNER/REPO/releases/download/qa-screenshots/pr123-abc1234-screenshot.png`
+
+### Report format
+
```markdown
## Smoke Test Report
@@ -233,17 +341,17 @@ For each test, record:
- <summary of what changed, from the diff>
### Results
-| Flow | Result | Evidence |
-|------|--------|----------|
-| Settings save | PASS | Form submitted, success toast shown |
-| Login redirect | PASS | Redirected to /dashboard after sign-in |
-| Dashboard nav | FAIL | Sidebar link to /reports returned 404 |
+| Flow | Result | Screenshot |
+|------|--------|------------|
+| Workflows page | PASS | ![workflows](https://github.com/OWNER/REPO/releases/download/qa-screenshots/pr123-abc1234-workflows.png) |
+| Task detail | PASS | ![task-detail](https://github.com/OWNER/REPO/releases/download/qa-screenshots/pr123-abc1234-task-detail.png) |
+| Settings save | FAIL | ![settings-error](https://github.com/OWNER/REPO/releases/download/qa-screenshots/pr123-abc1234-settings.png) |
### Verdict
2/3 tests passed. 1 issue found.
```
-The Evidence column contains a one-line summary of what was observed during the test.
+Every test row must have a screenshot. The screenshot is the primary evidence.
## Step 8: Post to PR
@@ -252,11 +360,7 @@ evidence survives beyond the conversation.
### Check for an open PR
-```bash
-PR_NUMBER=$(gh pr view --json number -q '.number' 2>/dev/null)
-```
-
-If no PR exists for the current branch:
+`PR_NUMBER` was set in Step 7. If it is `"draft"` (no PR found):
1. Save the full report markdown to `.qa/latest-smoke-report.md` in the project root (create the directory if needed).
2. Tell the user: "No open PR found for this branch. Smoke test report saved to `.qa/latest-smoke-report.md`. Run /smoke-test again after creating a PR to post it."
3. Stop here — do not attempt to create a PR.
@@ -308,10 +412,12 @@ rm -f "$COMMENT_FILE"
|---------|--------|
| No git diff found | Ask what behavior to validate, then fall back to explore mode |
| App not running and no startup command found | Start the most direct repo-documented local command; only ask user if no command exists or startup fails |
-| Skyvern browser tools unavailable (no Skyvern MCP) | Report "Skyvern MCP tools not available. Install Skyvern to enable /smoke-test." |
-| Health gate fails on navigation | Mark the test as FAIL with the health gate errors as evidence, continue to next test |
-| `skyvern_validate` reports failure | Mark the test as FAIL with the validation result as evidence |
-| No testable changes (docs-only, config-only) | Report "Changes are non-behavioral — no smoke tests generated." |
+| Chrome DevTools MCP browser profile locked | Kill stale Chrome processes, retry. If still locked, fall back to Skyvern |
+| Auth redirect detected | Tell user to authenticate in the Chrome DevTools browser, wait for confirmation |
+| Skyvern browser tools also unavailable | Use Playwright directly (`npx playwright`) as last resort |
+| Health gate fails on navigation | Mark the test as FAIL, take a screenshot anyway for evidence, continue |
+| Screenshot upload fails | Save locally to `.qa/screenshots/`, list paths in the PR comment, and tell the user to upload manually |
+| No testable changes (docs-only, config-only) | Report "Changes are non-behavioral - no smoke tests generated." |
## CI Setup
diff --git a/tests/unit/test_smoke_test_skill_content.py b/tests/unit/test_smoke_test_skill_content.py
index 69d188375..9125fe77a 100644
--- a/tests/unit/test_smoke_test_skill_content.py
+++ b/tests/unit/test_smoke_test_skill_content.py
@@ -36,7 +36,7 @@ def test_smoke_test_contains_browser_tool_execution_model() -> None:
def test_smoke_test_output_table_format() -> None:
skill_text = BUNDLED_SMOKE_TEST_SKILL.read_text(encoding="utf-8")
- assert "| Flow | Result | Evidence |" in skill_text
+ assert "| Flow | Result | Screenshot |" in skill_text
def test_smoke_test_pr_evidence_markers_present() -> None: