Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
chore(devops): don't pick already-fixed flakes in fix-flakes (#41704)
Simon Knott11a83314d072
diff --git a/.github/workflows/fix-flakes-prompt.md b/.github/workflows/fix-flakes-prompt.mdindex bf30bdc66c..a0c0c1178e 100644--- a/.github/workflows/fix-flakes-prompt.md+++ b/.github/workflows/fix-flakes-prompt.md@@ -17,13 +17,20 @@ bots/PRs it disrupts — **not** by "has a tidy error message".Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.-## 2. Check nobody's already on it--Search open PRs/issues for the test title words and file path before touching anything.-Also check any issue linked in the-test's `annotation`.-If an open PR already touches the test, **it's taken — drop it and go down your step-1 ranking**, re-checking each.-Only stop and report if the whole shortlist is covered.+## 2. Check nobody's on it — and that it isn't already fixed++Two dead ends to rule out before touching a candidate. If either fires, drop it and go+down your step-1 ranking, re-checking each; only stop once the whole shortlist is covered.++- **Already being worked on** — search PRs/issues (open *and* recently merged/closed) for+ the test title words and file path, plus any issue linked in the test's `annotation`.+- **Already fixed** — the DB is a rolling window that still holds the failing runs from+ before a fix landed, so a test fixed mid-window looks maximally bimodal (old fails + new+ passes), which is exactly what floats it up step 1. Check whether a fix has landed since it+ last flaked: find the commit its most recent failing run ran on, and look at what's changed+ in that test's file since. A commit that plausibly targets the test or its feature — a+ browser roll, or a fix to the test or the code it exercises — means the remaining DB fails+ are from before the fix, so move on.## 3. Reproduce on this OSdiff --git a/.github/workflows/fix-flakes.yml b/.github/workflows/fix-flakes.ymlindex 3ea6440f52..98fec1e5ce 100644--- a/.github/workflows/fix-flakes.yml+++ b/.github/workflows/fix-flakes.yml@@ -36,10 +36,12 @@ jobs:- name: Install dependenciesrun: npm ci- - name: Download test-results DB+ - name: Download and refresh test-results DBenv:GITHUB_TOKEN: ${{ github.token }}- run: node utils/test-results-db/cli.ts download+ run: |+ node utils/test-results-db/cli.ts download+ node utils/test-results-db/cli.ts update --lookback-days 3- name: Install Copilot CLIrun: npm install -g @github/copilot@@ -57,9 +59,12 @@ jobs:off only that runner label. Do NOT fix anything.Optimise for the OS with the highest-impact actionable flakiness/reds that isn't- already being worked on (its top candidates aren't covered by an open fix PR — check- with the GitHub MCP tools). All else equal, favour variety so Linux, Windows and macOS- each get attention over time.+ already being worked on OR already fixed. A candidate is dead if a fix PR touches it+ (open, or recently merged/closed) — the DB window still holds the failing runs from+ before a fix landed, so a just-fixed test keeps showing old fails and looks bimodal.+ For a top candidate, check open and merged/closed PRs plus any linked annotation+ issue (GitHub MCP tools) for a plausible fix before betting an OS on it. All else+ equal, favour variety so Linux, Windows and macOS each get attention over time.The DB is already downloaded at utils/test-results-db/test-results.duckdb — query itvia the playwright-test-results skill. \`bot_name\` encodes the OS.@@ -148,10 +153,12 @@ jobs:npx playwright installfi- - name: Download test-results DB+ - name: Download and refresh test-results DBenv:GITHUB_TOKEN: ${{ github.token }}- run: node utils/test-results-db/cli.ts download+ run: |+ node utils/test-results-db/cli.ts download+ node utils/test-results-db/cli.ts update --lookback-days 3- name: Install Copilot CLIrun: npm install -g @github/copilot