Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
devops(fix-flakes): add linked run history (#41774)
Simon Knott60ecab05aa21
diff --git a/.claude/skills/playwright-test-results/SKILL.md b/.claude/skills/playwright-test-results/SKILL.mdindex 39af23be61..0e08d94779 100644--- a/.claude/skills/playwright-test-results/SKILL.md+++ b/.claude/skills/playwright-test-results/SKILL.md@@ -123,6 +123,73 @@ ORDER BY runs DESCLIMIT 20;```+## Generate a linked emoji run history++For a compact result that drops straight into a GitHub comment, render each+final run verdict as a linked square. Edit the four test identity fields, then+run:++```bash+node --input-type=module <<'EOF'+import { DuckDBInstance } from "@duckdb/node-api";++const repository = "microsoft/playwright";+const test = {+ projectName: "firefox-library",+ file: "library/proxy.spec.ts",+ testTitle: "should exclude patterns",+ botName: "firefox-macos-15-large",+};++const conn = await (await DuckDBInstance.create(+ "utils/test-results-db/test-results.duckdb"+)).connect();+const result = await conn.runAndReadAll(`+ WITH per_run AS (+ SELECT run_id, run_attempt,+ any_value(run_started_at) AS run_started_at,+ arg_max(status, retry) AS final_status,+ arg_max(expected_status, retry) AS expected_status,+ list(status ORDER BY retry) AS attempt_statuses+ FROM test_results+ WHERE project_name = $projectName+ AND file = $file+ AND test_title = $testTitle+ AND bot_name = $botName+ GROUP BY run_id, run_attempt+ )+ SELECT run_id, run_attempt, final_status, attempt_statuses+ FROM per_run+ WHERE expected_status = 'passed'+ AND final_status IN ('passed', 'failed', 'timedOut')+ ORDER BY run_started_at, run_id, run_attempt+`, test);++const markdown = result.getRowObjectsJson().map(row => {+ const rescued = row.final_status === "passed" &&+ row.attempt_statuses.some(status => status === "failed" || status === "timedOut");+ const emoji = rescued ? "🟧" : row.final_status === "passed" ? "🟩" : "🟥";+ const url = `https://github.com/${repository}/actions/runs/${row.run_id}/attempts/${row.run_attempt}`;+ return `[${emoji}](${url})`;+}).join("");++console.log(markdown);+EOF+```++The output is Markdown:++```markdown+[🟩](https://github.com/microsoft/playwright/actions/runs/123/attempts/1)[🟧](https://github.com/microsoft/playwright/actions/runs/456/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/789/attempts/1)+```++Each square is one workflow run attempt, oldest first. Green means passed,+orange means a retry rescued an earlier failure, and red means failed or timed+out. `arg_max(status, retry)` picks the final verdict after retries, while+grouping by `(run_id, run_attempt)` keeps retries from turning into extra+squares. The `/attempts/<n>` URL links to the exact rerun that produced the+result.+## Fetching the full detailThe db stores per-result summaries. For the full step tree / attachments / stdio,diff --git a/.github/workflows/fix-flakes-prompt.md b/.github/workflows/fix-flakes-prompt.mdindex a0c0c1178e..c56b9dfa02 100644--- a/.github/workflows/fix-flakes-prompt.md+++ b/.github/workflows/fix-flakes-prompt.md@@ -16,6 +16,8 @@ Rank by **impact**: fail %, run count (floor `runs >= 10` so it isn't a one-off)bots/PRs it disrupts — **not** by "has a tidy error message".Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.+Once you have the target, use the "Generate a linked emoji run history" recipe in+`.claude/skills/playwright-test-results/SKILL.md`. Keep the Markdown output for the PR body.## 2. Check nobody's on it — and that it isn't already fixed@@ -105,8 +107,9 @@ You **never push or open a PR** in CI or locally. The harness does that for you,- **The commit message _is_ the PR** — the harness runs `gh pr create --fill`, mappingsubject→title and body→description. Write both in the Playwright bot voice(`.github/workflows/bot-voice.md`) — verdict first, short, no slop. The- body must carry: the **DB evidence** (fail %, runs, bots) + any related issue; **what you- verified locally** and on which OS.+ body must carry: the **DB evidence** (fail %, runs, bots) + any related issue; the+ **linked emoji run history** for the selected test and bot; **what you verified locally**+ and on which OS.- **Nothing actionable?** Make no commit — the harness then skips the PR.Report what you committed, the reviewer, and why.