Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
chore: add playwright-devops skill for CI failure analysis (#40291)
Pavel Feldmanb86038bd105b
diff --git a/.claude/skills/playwright-devops/SKILL.md b/.claude/skills/playwright-devops/SKILL.mdnew file mode 100644index 0000000000..b4a9deccc6--- /dev/null+++ b/.claude/skills/playwright-devops/SKILL.md@@ -0,0 +1,12 @@+---+name: playwright-devops+description: DevOps workflows for Playwright - CI failure analysis, workflow debugging, and release operations.+user_invocable: true+---++# Playwright DevOps++## Guides++- [CI Commit Failure Report](commit-failures.md) — analyze GitHub Actions failures for the last commit on main+- [fetch-commit-logs.sh](fetch-commit-logs.sh) — script to download failed job logs into `~/tmp/commit-<sha>/`diff --git a/.claude/skills/playwright-devops/commit-failures.md b/.claude/skills/playwright-devops/commit-failures.mdnew file mode 100644index 0000000000..6fc3e2ffd6--- /dev/null+++ b/.claude/skills/playwright-devops/commit-failures.md@@ -0,0 +1,69 @@+# CI Health Report++Generate a CI health report for the last commit on the `main` branch of `microsoft/playwright`.+This is an overall tree health report — not a commit regression analysis. The goal is to show+the full picture of what's failing, grouping by root cause. If any failures appear to be+regressions introduced by this specific commit, call that out, but most failures will be+pre-existing flakes, infrastructure issues, or platform-specific problems.++## Phase 1 — Fetch logs++Run the fetch script to download all failed job logs into a local folder:++```+bash .claude/skills/playwright-devops/fetch-commit-logs.sh [<sha>]+```++- If no SHA is provided, it fetches the last commit on `main`.+- Creates `~/tmp/commit-<short-sha>/` with:+ - `summary.json` — commit info, failed workflows, and failed job metadata+ - `<workflow-name>/<job-name>.log` — failed log output for each failed job++**Note:** The script fetches failed jobs from both failed AND in-progress workflows.+Workflows may still be running while some of their jobs have already failed — these+must be included in the report. If any workflows are still in progress, note this in+the report summary.++## Phase 2 — Analyze++1. **Read `summary.json`** to get the commit message and the list of failed workflows/jobs.++2. **Read each `.log` file** and extract failing test names and error messages.++3. **Compile the report leading with a summary**, then detailed tables:++ ```markdown+ # CI Health Report — <short-sha>++ Commit: `<commit message>`++ ## Summary++ Brief overview: N workflows, N failed jobs, N total test failures.+ Note if any workflows are still in progress.++ ### Possible regressions (may be related to this commit)+ - **N failures** in `test/file.spec.ts` across <browsers/platforms> — <brief description>++ ### Pre-existing / flaky+ - **N failures** in `test/file.spec.ts` — <brief description> (timeouts, infrastructure, platform-specific)++ ### Infrastructure issues+ - <description of non-test failures>++ ---++ ## Detailed Failures++ ### Workflow: <name> (run <id>)++ #### <job name> (job <id>) -- N failures+ | Test | Error |+ |------|-------|+ | `path/to/test.spec.ts:line` -- test title | error message |+ ```++ The summary should appear first so readers immediately see what matters. Group related failures+ (e.g. same test failing across browsers) into single summary bullet points rather than listing each individually.++4. **Save the report** to `ci-failures-<short-sha>.md` in the repo root.diff --git a/.claude/skills/playwright-devops/fetch-commit-logs.sh b/.claude/skills/playwright-devops/fetch-commit-logs.shnew file mode 100755index 0000000000..fb5bca862f--- /dev/null+++ b/.claude/skills/playwright-devops/fetch-commit-logs.sh@@ -0,0 +1,107 @@+#!/bin/bash+set -euo pipefail++# Usage: fetch-commit-logs.sh [<sha>]+# Fetches CI failure logs for a commit into ~/tmp/commit-<short-sha>/++REPO="microsoft/playwright"+REF="${1:-main}"++# Resolve commit+COMMIT_JSON=$(gh api "repos/$REPO/commits/$REF" --jq '{sha: .sha, message: .commit.message}')+SHA=$(echo "$COMMIT_JSON" | jq -r '.sha')+SHORT_SHA="${SHA:0:7}"+MESSAGE=$(echo "$COMMIT_JSON" | jq -r '.message' | head -1)++OUTDIR="$HOME/tmp/commit-$SHORT_SHA"+mkdir -p "$OUTDIR"++echo "Commit: $SHORT_SHA — $MESSAGE"+echo "Output: $OUTDIR"++# Get all workflow runs for this commit+RUNS_JSON=$(gh api "repos/$REPO/actions/runs?head_sha=$SHA&per_page=50" \+ --jq '[.workflow_runs[] | {id, name, conclusion, workflow_id}]')++# Filter to failed workflows+FAILED_RUNS=$(echo "$RUNS_JSON" | jq -c '[.[] | select(.conclusion == "failure" or .conclusion == null)]')+FAILED_COUNT=$(echo "$FAILED_RUNS" | jq 'length')++if [ "$FAILED_COUNT" -eq 0 ]; then+ echo "No failed workflows."+ echo '{"sha":"'"$SHA"'","short_sha":"'"$SHORT_SHA"'","message":"'"$MESSAGE"'","workflows":[]}' | jq . > "$OUTDIR/summary.json"+ echo "$OUTDIR"+ exit 0+fi++echo "Found $FAILED_COUNT failed workflow(s). Fetching failed jobs..."++# Build summary and collect jobs to fetch+SUMMARY='{"sha":"'"$SHA"'","short_sha":"'"$SHORT_SHA"'","message":'"$(echo "$MESSAGE" | jq -Rs .)"',"workflows":[]}'++sanitize() {+ echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9._-]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//'+}++PIDS=()++for i in $(seq 0 $((FAILED_COUNT - 1))); do+ RUN=$(echo "$FAILED_RUNS" | jq -c ".[$i]")+ RUN_ID=$(echo "$RUN" | jq -r '.id')+ RUN_NAME=$(echo "$RUN" | jq -r '.name')+ WORKFLOW_DIR=$(sanitize "$RUN_NAME")++ # Get failed jobs for this run+ JOBS_JSON=$(gh run view "$RUN_ID" --json jobs \+ --jq '[.jobs[] | select(.conclusion == "failure") | {name: .name, id: .databaseId}]')+ JOB_COUNT=$(echo "$JOBS_JSON" | jq 'length')++ if [ "$JOB_COUNT" -eq 0 ]; then+ continue+ fi++ mkdir -p "$OUTDIR/$WORKFLOW_DIR"++ # Add to summary+ WORKFLOW_ENTRY=$(jq -n \+ --arg name "$RUN_NAME" \+ --argjson id "$RUN_ID" \+ --argjson jobs "$JOBS_JSON" \+ '{name: $name, id: $id, failed_jobs: $jobs}')+ SUMMARY=$(echo "$SUMMARY" | jq --argjson w "$WORKFLOW_ENTRY" '.workflows += [$w]')++ # Fetch logs in parallel+ for j in $(seq 0 $((JOB_COUNT - 1))); do+ JOB=$(echo "$JOBS_JSON" | jq -c ".[$j]")+ JOB_ID=$(echo "$JOB" | jq -r '.id')+ JOB_NAME=$(echo "$JOB" | jq -r '.name')+ JOB_FILE=$(sanitize "$JOB_NAME")+ LOG_PATH="$OUTDIR/$WORKFLOW_DIR/$JOB_FILE.log"++ (+ echo "# $JOB_NAME" > "$LOG_PATH"+ echo "" >> "$LOG_PATH"+ gh run view --job "$JOB_ID" --log-failed 2>&1 | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' | sed $'s/^[^\t]*\t[^\t]*\t\xef\xbb\xbf\{0,1\}[0-9T:.Z-]* //' >> "$LOG_PATH" || true+ # If only header (e.g. workflow still in progress), fetch via jobs API+ if [ "$(wc -l < "$LOG_PATH")" -le 3 ]; then+ echo "# $JOB_NAME" > "$LOG_PATH"+ echo "" >> "$LOG_PATH"+ gh api "repos/$REPO/actions/jobs/$JOB_ID/logs" 2>&1 | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' | sed $'s/\xef\xbb\xbf//g' | sed 's/^[0-9T:.Z-]* //' >> "$LOG_PATH" || true+ fi+ echo " Fetched: $WORKFLOW_DIR/$JOB_FILE.log"+ ) &+ PIDS+=($!)+ done+done++# Wait for all parallel fetches+for pid in "${PIDS[@]}"; do+ wait "$pid" 2>/dev/null || true+done++# Write summary+echo "$SUMMARY" | jq . > "$OUTDIR/summary.json"++echo ""+echo "Done. Logs saved to: $OUTDIR"+echo "$OUTDIR"