Skip to content

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

chore: remove fix flakes bot - it is noise (#42432)

Pavel Feldman6f65bbb5695a
diff --git a/.github/workflows/fix-flakes-prompt.md b/.github/workflows/fix-flakes-prompt.md
deleted file mode 100644
index 03eb4b328c..0000000000
--- a/.github/workflows/fix-flakes-prompt.md
+++ /dev/null
@@ -1,143 +0,0 @@
-# Playwright: Fix a Flaky or Red Test
-
-Turn CI test-results data into **one** concrete fix: pick a high-impact flaky-or-red test that is reproducible on your OS,
-confirm nobody's on it, fix the root cause *or* scope a skip, pick a reviewer, and hand off a
-single commit that becomes the PR. Fully autonomous — no approval stops.
-
-The GitHub CLI (`gh`) is not authenticated in this job. Do not use it for GitHub API operations;
-use GitHub MCP tools instead.
-
-## 1. Pick one target
-
-Query the DB following the patterns in `.claude/skills/playwright-test-results/SKILL.md`.
-Two families:
-
-- **Cross-run flake** — the *final* verdict (after retries) flips between runs.
-- **Consistent red** — `expected_status = 'passed'` yet failing in ~every run.
-
-Rank by **impact**: fail %, run count (floor `runs >= 10` so it isn't a one-off), and how many
-bots/PRs it disrupts — **not** by "has a tidy error message".
-Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.
-Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.
-Once you have the target, use the "Generate a linked emoji run history" recipe in
-`.claude/skills/playwright-test-results/SKILL.md`. Include the complete output in the PR body:
-never trim it or remove green runs, and check the square count matches the per-run row count.
-
-## 2. Check nobody's on it — and that it isn't already fixed
-
-Two dead ends to rule out before touching a candidate. If either fires, drop it and go
-down your step-1 ranking, re-checking each; only stop once the whole shortlist is covered.
-
-- **Already being worked on** — search PRs/issues (open *and* recently merged/closed) for
- the test title words and file path, plus any issue linked in the test's `annotation`.
-- **Already fixed** — the DB is a rolling window that still holds the failing runs from
- before a fix landed, so a test fixed mid-window looks maximally bimodal (old fails + new
- passes), which is exactly what floats it up step 1. Check whether a fix has landed since it
- last flaked: find the commit its most recent failing run ran on, and look at what's changed
- in that test's file since. Only move on if enough subsequent runs show the failures stopped;
- a browser roll alone is not evidence of a fix.
-
-Compare the test across every bot that runs it, not only the failing bot. A clean boundary
-between stable, beta, and dev browser channels often points to a browser-version regression.
-Read the exact versions from the job logs and record the failing and passing versions; channel
-names alone do not identify the affected behaviour.
-
-## 3. Reproduce on this OS
-
-Read the test and its `error_message`.
-
-Build first (`npm run build`; watch is **not** running — if you touch generated-code files see
-`CLAUDE.md`). Then reproduce, scoped to the failing target.
-
-The DB `project_name` **is** the Playwright `--project`; `bot_name` encodes the OS. **Always
-pass a `<file>:<line>` filter** — a bare run launches the whole suite. The browser test scripts
-are project-locked, so use the one matching the failing bot:
-
-| Failing target | Command |
-|---|---|
-| chromium | `npm run ctest -- <file>:<line>` |
-| firefox | `npm run ftest -- <file>:<line>` |
-| webkit | `npm run wtest -- <file>:<line>` |
-| `tests/playwright-test/**` | `npm run ttest -- <file>:<line>` |
-| `tests/mcp/**` | `npm run test-mcp -- --project=<name> <file>:<line>` |
-
-Other suites (electron `etest`, etc.): see `package.json` scripts.
-Add `--repeat-each=N` to force a flake's flip.
-
-**You can only reproduce what this OS and architecture reaches.** Match both from `bot_name`;
-non-reproduction under lighter local load or on another architecture is not itself a skip candidate.
-Keep any OS/architecture handling keyed on the *current* environment, never hardcoded.
-Broader OS coverage comes from different agent runs on other OSes, not from you.
-
-## 4. Fix — root cause or scoped skip
-
-- **Tractable → fix the source.** Usually a test-side race: a missing `await`, waiting on the
- wrong signal, an under-specified locator, leaked state. Fix the test (or the product bug).
-- **Timeout → check the budget before skipping.** Compare passing durations and neighboring tests;
- if working tests share an expensive operation near the timeout, prefer a scoped `it.slow()`.
-- **Engine/OS-specific, unreachable here, or genuinely hard → scope a skip**, narrowed to the
- exact failing condition — never a whole file or browser:
-
- ```ts
- it.fixme(browserName === 'webkit' && isLinux, 'https://github.com/microsoft/playwright/issues/NNNNN');
- ```
-
- For browser-specific failures, prefer the narrowest observed version predicate over a channel
- predicate. Channels roll forward, so `channel === 'chrome'` can keep skipping the test after
- the browser fixes the bug. If stable fails while beta/dev passes, use `browserVersion` or
- `browserMajorVersion` and gate only the affected version:
-
- ```ts
- it.fixme(browserName === 'chromium' && browserMajorVersion === 150, 'https://github.com/microsoft/playwright/issues/NNNNN');
- ```
-
- Do not invent a wider threshold for untested versions. Keep a channel predicate only when the
- same browser version behaves differently between channels.
-
- Link an issue if one exists or explain why the test is skipped. **Default to `fixme`** — it parks
- the debt and stays greppable. Use `skip` only if reproduction shows the test is genuinely
- mis-scoped for that config (`skip` claims "this failure is expected and correct," which a
- flake-fighter rarely is). Never `skip` just to turn a red green.
-
-**Verify locally — your PR gets no CI (step 6), so this is the only proof:** flake → re-run with
-`--repeat-each` and confirm it's stable; slow → confirm it still runs and passes; skip → confirm
-it's skipped on the target config and **still runs elsewhere**. Then run `npm run flint`. Record
-exactly what you ran — it goes in the PR body.
-
-## 5. Pick a reviewer
-
-No CODEOWNERS. Derive one from the touched file(s) — recent authors and frequent committers:
-
-```bash
-git log --format='%an %ae' -n 20 -- <file>
-git log --format='%an' -n 200 -- <file> | sort | uniq -c | sort -rn
-```
-
-Also skim recent merged PRs on those paths for who reviewed them. Pick someone with real recency
-+ ownership. Record it as a git trailer so it survives the handoff:
-
-```
-Suggested-reviewer: dgozman
-```
-
-## 6. Commit and hand off (you never open the PR in CI)
-
-You **never push or open a PR** in CI or locally. The harness does that for you, after you commit.
-
-- **Exactly one commit** on the current branch (`CLAUDE.md` conventions; no
- co-author / "generated with" trailers; never amend).
-- **The commit message _is_ the PR** — the harness runs `gh pr create --fill`, mapping
- subject→title and body→description. Write both in the Playwright bot voice
- (`.github/workflows/bot-voice.md`) — verdict first, short, no slop. The
- body must carry: the **DB evidence** (fail %, runs, bots) + any related issue; the
- **linked emoji run history** for the selected test and bot; **what you verified locally**
- and on which OS.
-- **Nothing actionable?** Make no commit — the harness then skips the PR.
-
-Report what you committed, the reviewer, and why.
-
-## Guardrails
-
-- **One target, one commit.** Don't batch.
-- **Scoped skips only** — narrow to the failing condition, never a whole file or browser matrix.
-- **No unverified PRs** — if you cannot validate a scoped mitigation, make no commit.
diff --git a/.github/workflows/fix-flakes.yml b/.github/workflows/fix-flakes.yml
deleted file mode 100644
index cb1d435a71..0000000000
--- a/.github/workflows/fix-flakes.yml
+++ /dev/null
@@ -1,298 +0,0 @@
-name: "Fix flaky tests"
-
-on:
- workflow_dispatch:
- inputs:
- hint:
- description: "Optional focus hint passed to the triage and fix agents (e.g. 'webkit flakes')"
- type: string
- schedule:
- - cron: "17 6 * * 1-5"
-
-permissions: {}
-
-jobs:
- triage:
- runs-on: ubuntu-latest
- if: github.repository == 'microsoft/playwright'
- outputs:
- runner: ${{ steps.validate.outputs.runner }}
- has_target: ${{ steps.validate.outputs.has_target }}
- permissions:
- actions: read
- copilot-requests: write
- concurrency:
- group: fix-flakes-triage
- cancel-in-progress: false
- steps:
- - name: Checkout
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-
- - name: Set up Node.js
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
- with:
- node-version: lts/*
-
- - name: Install dependencies
- run: npm ci
-
- - name: Download and refresh test-results DB
- env:
- GITHUB_TOKEN: ${{ github.token }}
- run: |
- node utils/test-results-db/cli.ts download
- node utils/test-results-db/cli.ts update --lookback-days 3
-
- - name: Install Copilot CLI
- run: npm install -g @github/copilot
-
- - name: Pick an eligible OS with Copilot CLI
- shell: bash
- env:
- COPILOT_GITHUB_TOKEN: ${{ github.token }}
- HINT: ${{ inputs.hint }}
- run: |
- mkdir -p output
- cat > output/triage-prompt.md <<EOF
- You are triage for the fix-flakes workflow. Pick the ONE operating system where booting
- an expensive runner to fix a flaky or red test is most worthwhile right now, and hand
- off only that runner label. Do NOT fix anything.
-
- The GitHub CLI is not authenticated in this job. Do not run gh; use GitHub MCP tools
- for all GitHub reads.
-
- Optimise for the OS with the highest-impact actionable flakiness/reds that isn't
- already being worked on OR already fixed. A candidate is dead if a fix PR touches it
- (open, or recently merged/closed) — the DB window still holds the failing runs from
- before a fix landed, so a just-fixed test keeps showing old fails and looks bimodal.
- For a top candidate, check open and merged/closed PRs plus any linked annotation
- issue (GitHub MCP tools) for a plausible fix before betting an OS on it. All else
- equal, favour variety so Linux, Windows and macOS each get attention over time.
-
- The DB is already downloaded at utils/test-results-db/test-results.duckdb — query it
- via the playwright-test-results skill. \`bot_name\` encodes the OS.
-
- Write your answer to output/runner.txt, and nothing else there: exactly one of these
- seven runner labels, or the single word \`none\` if no platform is worth a run.
- ubuntu-22.04 ubuntu-22.04-arm ubuntu-24.04
- macos-14-xlarge macos-15-large macos-15-xlarge windows-latest
- Map \`*-macos-15-large\` to \`macos-15-large\` and \`*-macos-15-xlarge\` to
- \`macos-15-xlarge\`; never substitute one architecture for another. Explain your
- reasoning briefly in your final message.
-
- ${HINT:+Focus hint from the operator: $HINT}
- EOF
- copilot \
- --allow-all-tools \
- --allow-all-paths \
- --no-ask-user \
- --enable-all-github-mcp-tools \
- --share=output/copilot-session.md \
- --model claude-opus-4.8 \
- --max-ai-credits 200 \
- -p "$(cat output/triage-prompt.md)"
-
- - name: Validate chosen runner
- id: validate
- shell: bash
- run: |
- runner="$(tr -d '[:space:]' < output/runner.txt 2>/dev/null || true)"
- if [ -z "$runner" ] || [ "$runner" = "none" ]; then
- echo "No eligible platform this run — skipping the fix job."
- echo "has_target=false" >> "$GITHUB_OUTPUT"
- exit 0
- fi
- case "$runner" in
- ubuntu-22.04|ubuntu-22.04-arm|ubuntu-24.04|macos-14-xlarge|macos-15-large|macos-15-xlarge|windows-latest) ;;
- *) echo "::error::Triage chose a non-allowlisted runner: '$runner'." >&2; exit 1 ;;
- esac
- echo "Triage chose runner: $runner"
- echo "runner=$runner" >> "$GITHUB_OUTPUT"
- echo "has_target=true" >> "$GITHUB_OUTPUT"
-
- - name: Add triage transcript to job summary
- if: always()
- shell: bash
- run: |
- {
- echo "## Fix-flakes triage transcript"
- echo ''
- cat "output/copilot-session.md" 2>/dev/null || echo "(no transcript)"
- } >> "$GITHUB_STEP_SUMMARY"
-
- fix:
- needs: triage
- if: needs.triage.outputs.has_target == 'true'
- runs-on: ${{ needs.triage.outputs.runner }}
- timeout-minutes: 60
- outputs:
- has_fix: ${{ steps.export.outputs.has_fix }}
- permissions:
- actions: read
- copilot-requests: write
- steps:
- - name: Checkout
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- with:
- ref: ${{ github.sha }}
- fetch-depth: 0
-
- - name: Set up Node.js
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
- with:
- node-version: lts/*
-
- - name: Install dependencies
- run: npm ci
-
- - name: Build
- run: npm run build
-
- - name: Install browsers
- shell: bash
- run: |
- if [ "$RUNNER_OS" = "Linux" ]; then
- npx playwright install --with-deps
- else
- npx playwright install
- fi
-
- - name: Download and refresh test-results DB
- env:
- GITHUB_TOKEN: ${{ github.token }}
- run: |
- node utils/test-results-db/cli.ts download
- node utils/test-results-db/cli.ts update --lookback-days 3
-
- - name: Install Copilot CLI
- run: npm install -g @github/copilot
-
- - name: Configure git identity
- shell: bash
- run: |
- git config user.name "Copilot"
- git config user.email "[email protected]"
-
- - name: Record base commit
- id: base
- shell: bash
- run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
-
- - name: Fix a flaky/red test with Copilot CLI
- shell: bash
- env:
- COPILOT_GITHUB_TOKEN: ${{ github.token }}
- RUNNER: ${{ needs.triage.outputs.runner }}
- HINT: ${{ inputs.hint }}
- WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
- run: |
- mkdir -p output
- cat > output/fix-prompt.md <<EOF
- Read .github/workflows/fix-flakes-prompt.md and follow it.
-
- You are running on $RUNNER.
- Include the link to this run in your commit message: $WORKFLOW_URL.
-
- ${HINT:+Operator focus hint: $HINT}
- EOF
- copilot \
- --allow-all-tools \
- --allow-all-paths \
- --no-ask-user \
- --enable-all-github-mcp-tools \
- --share=output/copilot-session.md \
- --model claude-opus-4.8 \
- --max-ai-credits 1000 \
- -p "$(cat output/fix-prompt.md)"
-
- - name: Export the fix commit
- id: export
- shell: bash
- run: |
- base="${{ steps.base.outputs.sha }}"
- count=$(git rev-list --count "$base"..HEAD)
- if [ "$count" -eq 0 ]; then
- echo "Agent produced no commit — nothing to open."
- echo "has_fix=false" >> "$GITHUB_OUTPUT"
- exit 0
- fi
- if [ "$count" -ne 1 ]; then
- echo "::error::Expected exactly one commit, got $count. Refusing to hand off." >&2
- exit 1
- fi
- mkdir -p handoff
- git format-patch -1 HEAD --stdout > handoff/fix.patch
- echo "has_fix=true" >> "$GITHUB_OUTPUT"
-
- - name: Upload handoff (fix patch)
- if: steps.export.outputs.has_fix == 'true'
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
- with:
- name: fix-flakes-handoff-${{ needs.triage.outputs.runner }}
- path: handoff/
- if-no-files-found: error
-
- - name: Add session transcript to job summary
- if: always()
- shell: bash
- run: |
- {
- echo "## Fix-flakes session transcript (${{ needs.triage.outputs.runner }})"
- echo ''
- cat "output/copilot-session.md" 2>/dev/null || echo "(no transcript)"
- } >> "$GITHUB_STEP_SUMMARY"
-
- - name: Upload session transcript
- if: always()
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
- with:
- name: fix-flakes-session-${{ needs.triage.outputs.runner }}
- path: output/copilot-session.md
- if-no-files-found: warn
-
- open_pr:
- needs: [triage, fix]
- if: needs.fix.outputs.has_fix == 'true'
- runs-on: ubuntu-latest
- permissions:
- contents: write
- pull-requests: write
- steps:
- - name: Checkout base commit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- with:
- ref: ${{ github.sha }}
- fetch-depth: 0
-
- - name: Download handoff
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
- with:
- name: fix-flakes-handoff-${{ needs.triage.outputs.runner }}
- path: handoff
-
- - uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
- id: app-token
- with:
- client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }}
- private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }}
-
- - name: Apply commit and open PR
- env:
- GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}
- RUNNER: ${{ needs.triage.outputs.runner }}
- run: |
- git config user.name "github-actions[bot]"
- git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
- git fetch --no-tags origin main:refs/remotes/origin/main
- branch="fix-flakes/${RUNNER}-${{ github.run_id }}"
- git checkout -b "$branch"
- git am handoff/fix.patch
- git push origin "$branch"
- reviewer=$(git log -1 --format='%(trailers:key=Suggested-reviewer,valueonly)' | head -n1 | tr -d '[:space:]@')
- args=(--repo "${{ github.repository }}" --base main --head "$branch" --fill)
- if [ -n "$reviewer" ]; then
- args+=(--reviewer "$reviewer")
- else
- echo "::warning::No Suggested-reviewer trailer on the fix commit."
- fi
- gh pr create "${args[@]}"