Skip to content

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

chore(devops): don't pick already-fixed flakes in fix-flakes (#41704)

Simon Knott11a83314d072
diff --git a/.github/workflows/fix-flakes-prompt.md b/.github/workflows/fix-flakes-prompt.md
index bf30bdc66c..a0c0c1178e 100644
--- a/.github/workflows/fix-flakes-prompt.md
+++ b/.github/workflows/fix-flakes-prompt.md
@@ -17,13 +17,20 @@ bots/PRs it disrupts — **not** by "has a tidy error message".
Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.
Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.
-## 2. Check nobody's already on it
-
-Search open PRs/issues for the test title words and file path before touching anything.
-Also check any issue linked in the
-test's `annotation`.
-If an open PR already touches the test, **it's taken — drop it and go down your step-1 ranking**, re-checking each.
-Only stop and report if the whole shortlist is covered.
+## 2. Check nobody's on it — and that it isn't already fixed
+
+Two dead ends to rule out before touching a candidate. If either fires, drop it and go
+down your step-1 ranking, re-checking each; only stop once the whole shortlist is covered.
+
+- **Already being worked on** — search PRs/issues (open *and* recently merged/closed) for
+ the test title words and file path, plus any issue linked in the test's `annotation`.
+- **Already fixed** — the DB is a rolling window that still holds the failing runs from
+ before a fix landed, so a test fixed mid-window looks maximally bimodal (old fails + new
+ passes), which is exactly what floats it up step 1. Check whether a fix has landed since it
+ last flaked: find the commit its most recent failing run ran on, and look at what's changed
+ in that test's file since. A commit that plausibly targets the test or its feature — a
+ browser roll, or a fix to the test or the code it exercises — means the remaining DB fails
+ are from before the fix, so move on.
## 3. Reproduce on this OS
diff --git a/.github/workflows/fix-flakes.yml b/.github/workflows/fix-flakes.yml
index 3ea6440f52..98fec1e5ce 100644
--- a/.github/workflows/fix-flakes.yml
+++ b/.github/workflows/fix-flakes.yml
@@ -36,10 +36,12 @@ jobs:
- name: Install dependencies
run: npm ci
- - name: Download test-results DB
+ - name: Download and refresh test-results DB
env:
GITHUB_TOKEN: ${{ github.token }}
- run: node utils/test-results-db/cli.ts download
+ run: |
+ node utils/test-results-db/cli.ts download
+ node utils/test-results-db/cli.ts update --lookback-days 3
- name: Install Copilot CLI
run: npm install -g @github/copilot
@@ -57,9 +59,12 @@ jobs:
off only that runner label. Do NOT fix anything.
Optimise for the OS with the highest-impact actionable flakiness/reds that isn't
- already being worked on (its top candidates aren't covered by an open fix PR — check
- with the GitHub MCP tools). All else equal, favour variety so Linux, Windows and macOS
- each get attention over time.
+ already being worked on OR already fixed. A candidate is dead if a fix PR touches it
+ (open, or recently merged/closed) — the DB window still holds the failing runs from
+ before a fix landed, so a just-fixed test keeps showing old fails and looks bimodal.
+ For a top candidate, check open and merged/closed PRs plus any linked annotation
+ issue (GitHub MCP tools) for a plausible fix before betting an OS on it. All else
+ equal, favour variety so Linux, Windows and macOS each get attention over time.
The DB is already downloaded at utils/test-results-db/test-results.duckdb — query it
via the playwright-test-results skill. \`bot_name\` encodes the OS.
@@ -148,10 +153,12 @@ jobs:
npx playwright install
fi
- - name: Download test-results DB
+ - name: Download and refresh test-results DB
env:
GITHUB_TOKEN: ${{ github.token }}
- run: node utils/test-results-db/cli.ts download
+ run: |
+ node utils/test-results-db/cli.ts download
+ node utils/test-results-db/cli.ts update --lookback-days 3
- name: Install Copilot CLI
run: npm install -g @github/copilot