Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
chore: remove fix flakes bot - it is noise (#42432)
Pavel Feldman6f65bbb5695a
diff --git a/.github/workflows/fix-flakes-prompt.md b/.github/workflows/fix-flakes-prompt.mddeleted file mode 100644index 03eb4b328c..0000000000--- a/.github/workflows/fix-flakes-prompt.md+++ /dev/null@@ -1,143 +0,0 @@-# Playwright: Fix a Flaky or Red Test--Turn CI test-results data into **one** concrete fix: pick a high-impact flaky-or-red test that is reproducible on your OS,-confirm nobody's on it, fix the root cause *or* scope a skip, pick a reviewer, and hand off a-single commit that becomes the PR. Fully autonomous — no approval stops.--The GitHub CLI (`gh`) is not authenticated in this job. Do not use it for GitHub API operations;-use GitHub MCP tools instead.--## 1. Pick one target--Query the DB following the patterns in `.claude/skills/playwright-test-results/SKILL.md`.-Two families:--- **Cross-run flake** — the *final* verdict (after retries) flips between runs.-- **Consistent red** — `expected_status = 'passed'` yet failing in ~every run.--Rank by **impact**: fail %, run count (floor `runs >= 10` so it isn't a one-off), and how many-bots/PRs it disrupts — **not** by "has a tidy error message".-Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.-Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.-Once you have the target, use the "Generate a linked emoji run history" recipe in-`.claude/skills/playwright-test-results/SKILL.md`. Include the complete output in the PR body:-never trim it or remove green runs, and check the square count matches the per-run row count.--## 2. Check nobody's on it — and that it isn't already fixed--Two dead ends to rule out before touching a candidate. If either fires, drop it and go-down your step-1 ranking, re-checking each; only stop once the whole shortlist is covered.--- **Already being worked on** — search PRs/issues (open *and* recently merged/closed) for- the test title words and file path, plus any issue linked in the test's `annotation`.-- **Already fixed** — the DB is a rolling window that still holds the failing runs from- before a fix landed, so a test fixed mid-window looks maximally bimodal (old fails + new- passes), which is exactly what floats it up step 1. Check whether a fix has landed since it- last flaked: find the commit its most recent failing run ran on, and look at what's changed- in that test's file since. Only move on if enough subsequent runs show the failures stopped;- a browser roll alone is not evidence of a fix.--Compare the test across every bot that runs it, not only the failing bot. A clean boundary-between stable, beta, and dev browser channels often points to a browser-version regression.-Read the exact versions from the job logs and record the failing and passing versions; channel-names alone do not identify the affected behaviour.--## 3. Reproduce on this OS--Read the test and its `error_message`.--Build first (`npm run build`; watch is **not** running — if you touch generated-code files see-`CLAUDE.md`). Then reproduce, scoped to the failing target.--The DB `project_name` **is** the Playwright `--project`; `bot_name` encodes the OS. **Always-pass a `<file>:<line>` filter** — a bare run launches the whole suite. The browser test scripts-are project-locked, so use the one matching the failing bot:--| Failing target | Command |-|---|---|-| chromium | `npm run ctest -- <file>:<line>` |-| firefox | `npm run ftest -- <file>:<line>` |-| webkit | `npm run wtest -- <file>:<line>` |-| `tests/playwright-test/**` | `npm run ttest -- <file>:<line>` |-| `tests/mcp/**` | `npm run test-mcp -- --project=<name> <file>:<line>` |--Other suites (electron `etest`, etc.): see `package.json` scripts.-Add `--repeat-each=N` to force a flake's flip.--**You can only reproduce what this OS and architecture reaches.** Match both from `bot_name`;-non-reproduction under lighter local load or on another architecture is not itself a skip candidate.-Keep any OS/architecture handling keyed on the *current* environment, never hardcoded.-Broader OS coverage comes from different agent runs on other OSes, not from you.--## 4. Fix — root cause or scoped skip--- **Tractable → fix the source.** Usually a test-side race: a missing `await`, waiting on the- wrong signal, an under-specified locator, leaked state. Fix the test (or the product bug).-- **Timeout → check the budget before skipping.** Compare passing durations and neighboring tests;- if working tests share an expensive operation near the timeout, prefer a scoped `it.slow()`.-- **Engine/OS-specific, unreachable here, or genuinely hard → scope a skip**, narrowed to the- exact failing condition — never a whole file or browser:-- ```ts- it.fixme(browserName === 'webkit' && isLinux, 'https://github.com/microsoft/playwright/issues/NNNNN');- ```-- For browser-specific failures, prefer the narrowest observed version predicate over a channel- predicate. Channels roll forward, so `channel === 'chrome'` can keep skipping the test after- the browser fixes the bug. If stable fails while beta/dev passes, use `browserVersion` or- `browserMajorVersion` and gate only the affected version:-- ```ts- it.fixme(browserName === 'chromium' && browserMajorVersion === 150, 'https://github.com/microsoft/playwright/issues/NNNNN');- ```-- Do not invent a wider threshold for untested versions. Keep a channel predicate only when the- same browser version behaves differently between channels.-- Link an issue if one exists or explain why the test is skipped. **Default to `fixme`** — it parks- the debt and stays greppable. Use `skip` only if reproduction shows the test is genuinely- mis-scoped for that config (`skip` claims "this failure is expected and correct," which a- flake-fighter rarely is). Never `skip` just to turn a red green.--**Verify locally — your PR gets no CI (step 6), so this is the only proof:** flake → re-run with-`--repeat-each` and confirm it's stable; slow → confirm it still runs and passes; skip → confirm-it's skipped on the target config and **still runs elsewhere**. Then run `npm run flint`. Record-exactly what you ran — it goes in the PR body.--## 5. Pick a reviewer--No CODEOWNERS. Derive one from the touched file(s) — recent authors and frequent committers:--```bash-git log --format='%an %ae' -n 20 -- <file>-git log --format='%an' -n 200 -- <file> | sort | uniq -c | sort -rn-```--Also skim recent merged PRs on those paths for who reviewed them. Pick someone with real recency-+ ownership. Record it as a git trailer so it survives the handoff:--```-Suggested-reviewer: dgozman-```--## 6. Commit and hand off (you never open the PR in CI)--You **never push or open a PR** in CI or locally. The harness does that for you, after you commit.--- **Exactly one commit** on the current branch (`CLAUDE.md` conventions; no- co-author / "generated with" trailers; never amend).-- **The commit message _is_ the PR** — the harness runs `gh pr create --fill`, mapping- subject→title and body→description. Write both in the Playwright bot voice- (`.github/workflows/bot-voice.md`) — verdict first, short, no slop. The- body must carry: the **DB evidence** (fail %, runs, bots) + any related issue; the- **linked emoji run history** for the selected test and bot; **what you verified locally**- and on which OS.-- **Nothing actionable?** Make no commit — the harness then skips the PR.--Report what you committed, the reviewer, and why.--## Guardrails--- **One target, one commit.** Don't batch.-- **Scoped skips only** — narrow to the failing condition, never a whole file or browser matrix.-- **No unverified PRs** — if you cannot validate a scoped mitigation, make no commit.diff --git a/.github/workflows/fix-flakes.yml b/.github/workflows/fix-flakes.ymldeleted file mode 100644index cb1d435a71..0000000000--- a/.github/workflows/fix-flakes.yml+++ /dev/null@@ -1,298 +0,0 @@-name: "Fix flaky tests"--on:- workflow_dispatch:- inputs:- hint:- description: "Optional focus hint passed to the triage and fix agents (e.g. 'webkit flakes')"- type: string- schedule:- - cron: "17 6 * * 1-5"--permissions: {}--jobs:- triage:- runs-on: ubuntu-latest- if: github.repository == 'microsoft/playwright'- outputs:- runner: ${{ steps.validate.outputs.runner }}- has_target: ${{ steps.validate.outputs.has_target }}- permissions:- actions: read- copilot-requests: write- concurrency:- group: fix-flakes-triage- cancel-in-progress: false- steps:- - name: Checkout- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1-- - name: Set up Node.js- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0- with:- node-version: lts/*-- - name: Install dependencies- run: npm ci-- - name: Download and refresh test-results DB- env:- GITHUB_TOKEN: ${{ github.token }}- run: |- node utils/test-results-db/cli.ts download- node utils/test-results-db/cli.ts update --lookback-days 3-- - name: Install Copilot CLI- run: npm install -g @github/copilot-- - name: Pick an eligible OS with Copilot CLI- shell: bash- env:- COPILOT_GITHUB_TOKEN: ${{ github.token }}- HINT: ${{ inputs.hint }}- run: |- mkdir -p output- cat > output/triage-prompt.md <<EOF- You are triage for the fix-flakes workflow. Pick the ONE operating system where booting- an expensive runner to fix a flaky or red test is most worthwhile right now, and hand- off only that runner label. Do NOT fix anything.-- The GitHub CLI is not authenticated in this job. Do not run gh; use GitHub MCP tools- for all GitHub reads.-- Optimise for the OS with the highest-impact actionable flakiness/reds that isn't- already being worked on OR already fixed. A candidate is dead if a fix PR touches it- (open, or recently merged/closed) — the DB window still holds the failing runs from- before a fix landed, so a just-fixed test keeps showing old fails and looks bimodal.- For a top candidate, check open and merged/closed PRs plus any linked annotation- issue (GitHub MCP tools) for a plausible fix before betting an OS on it. All else- equal, favour variety so Linux, Windows and macOS each get attention over time.-- The DB is already downloaded at utils/test-results-db/test-results.duckdb — query it- via the playwright-test-results skill. \`bot_name\` encodes the OS.-- Write your answer to output/runner.txt, and nothing else there: exactly one of these- seven runner labels, or the single word \`none\` if no platform is worth a run.- ubuntu-22.04 ubuntu-22.04-arm ubuntu-24.04- macos-14-xlarge macos-15-large macos-15-xlarge windows-latest- Map \`*-macos-15-large\` to \`macos-15-large\` and \`*-macos-15-xlarge\` to- \`macos-15-xlarge\`; never substitute one architecture for another. Explain your- reasoning briefly in your final message.-- ${HINT:+Focus hint from the operator: $HINT}- EOF- copilot \- --allow-all-tools \- --allow-all-paths \- --no-ask-user \- --enable-all-github-mcp-tools \- --share=output/copilot-session.md \- --model claude-opus-4.8 \- --max-ai-credits 200 \- -p "$(cat output/triage-prompt.md)"-- - name: Validate chosen runner- id: validate- shell: bash- run: |- runner="$(tr -d '[:space:]' < output/runner.txt 2>/dev/null || true)"- if [ -z "$runner" ] || [ "$runner" = "none" ]; then- echo "No eligible platform this run — skipping the fix job."- echo "has_target=false" >> "$GITHUB_OUTPUT"- exit 0- fi- case "$runner" in- ubuntu-22.04|ubuntu-22.04-arm|ubuntu-24.04|macos-14-xlarge|macos-15-large|macos-15-xlarge|windows-latest) ;;- *) echo "::error::Triage chose a non-allowlisted runner: '$runner'." >&2; exit 1 ;;- esac- echo "Triage chose runner: $runner"- echo "runner=$runner" >> "$GITHUB_OUTPUT"- echo "has_target=true" >> "$GITHUB_OUTPUT"-- - name: Add triage transcript to job summary- if: always()- shell: bash- run: |- {- echo "## Fix-flakes triage transcript"- echo ''- cat "output/copilot-session.md" 2>/dev/null || echo "(no transcript)"- } >> "$GITHUB_STEP_SUMMARY"-- fix:- needs: triage- if: needs.triage.outputs.has_target == 'true'- runs-on: ${{ needs.triage.outputs.runner }}- timeout-minutes: 60- outputs:- has_fix: ${{ steps.export.outputs.has_fix }}- permissions:- actions: read- copilot-requests: write- steps:- - name: Checkout- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1- with:- ref: ${{ github.sha }}- fetch-depth: 0-- - name: Set up Node.js- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0- with:- node-version: lts/*-- - name: Install dependencies- run: npm ci-- - name: Build- run: npm run build-- - name: Install browsers- shell: bash- run: |- if [ "$RUNNER_OS" = "Linux" ]; then- npx playwright install --with-deps- else- npx playwright install- fi-- - name: Download and refresh test-results DB- env:- GITHUB_TOKEN: ${{ github.token }}- run: |- node utils/test-results-db/cli.ts download- node utils/test-results-db/cli.ts update --lookback-days 3-- - name: Install Copilot CLI- run: npm install -g @github/copilot-- - name: Configure git identity- shell: bash- run: |- git config user.name "Copilot"- git config user.email "[email protected]"-- - name: Record base commit- id: base- shell: bash- run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"-- - name: Fix a flaky/red test with Copilot CLI- shell: bash- env:- COPILOT_GITHUB_TOKEN: ${{ github.token }}- RUNNER: ${{ needs.triage.outputs.runner }}- HINT: ${{ inputs.hint }}- WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}- run: |- mkdir -p output- cat > output/fix-prompt.md <<EOF- Read .github/workflows/fix-flakes-prompt.md and follow it.-- You are running on $RUNNER.- Include the link to this run in your commit message: $WORKFLOW_URL.-- ${HINT:+Operator focus hint: $HINT}- EOF- copilot \- --allow-all-tools \- --allow-all-paths \- --no-ask-user \- --enable-all-github-mcp-tools \- --share=output/copilot-session.md \- --model claude-opus-4.8 \- --max-ai-credits 1000 \- -p "$(cat output/fix-prompt.md)"-- - name: Export the fix commit- id: export- shell: bash- run: |- base="${{ steps.base.outputs.sha }}"- count=$(git rev-list --count "$base"..HEAD)- if [ "$count" -eq 0 ]; then- echo "Agent produced no commit — nothing to open."- echo "has_fix=false" >> "$GITHUB_OUTPUT"- exit 0- fi- if [ "$count" -ne 1 ]; then- echo "::error::Expected exactly one commit, got $count. Refusing to hand off." >&2- exit 1- fi- mkdir -p handoff- git format-patch -1 HEAD --stdout > handoff/fix.patch- echo "has_fix=true" >> "$GITHUB_OUTPUT"-- - name: Upload handoff (fix patch)- if: steps.export.outputs.has_fix == 'true'- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1- with:- name: fix-flakes-handoff-${{ needs.triage.outputs.runner }}- path: handoff/- if-no-files-found: error-- - name: Add session transcript to job summary- if: always()- shell: bash- run: |- {- echo "## Fix-flakes session transcript (${{ needs.triage.outputs.runner }})"- echo ''- cat "output/copilot-session.md" 2>/dev/null || echo "(no transcript)"- } >> "$GITHUB_STEP_SUMMARY"-- - name: Upload session transcript- if: always()- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1- with:- name: fix-flakes-session-${{ needs.triage.outputs.runner }}- path: output/copilot-session.md- if-no-files-found: warn-- open_pr:- needs: [triage, fix]- if: needs.fix.outputs.has_fix == 'true'- runs-on: ubuntu-latest- permissions:- contents: write- pull-requests: write- steps:- - name: Checkout base commit- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1- with:- ref: ${{ github.sha }}- fetch-depth: 0-- - name: Download handoff- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1- with:- name: fix-flakes-handoff-${{ needs.triage.outputs.runner }}- path: handoff-- - uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0- id: app-token- with:- client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }}- private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }}-- - name: Apply commit and open PR- env:- GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}- RUNNER: ${{ needs.triage.outputs.runner }}- run: |- git config user.name "github-actions[bot]"- git config user.email "41898282+github-actions[bot]@users.noreply.github.com"- git fetch --no-tags origin main:refs/remotes/origin/main- branch="fix-flakes/${RUNNER}-${{ github.run_id }}"- git checkout -b "$branch"- git am handoff/fix.patch- git push origin "$branch"- reviewer=$(git log -1 --format='%(trailers:key=Suggested-reviewer,valueonly)' | head -n1 | tr -d '[:space:]@')- args=(--repo "${{ github.repository }}" --base main --head "$branch" --fill)- if [ -n "$reviewer" ]; then- args+=(--reviewer "$reviewer")- else- echo "::warning::No Suggested-reviewer trailer on the fix commit."- fi- gh pr create "${args[@]}"