The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Design the autonomous e2e flake fixer (#75909)
Nemanja Glumacc8350535e75c
diff --git a/.claude/commands/fix-flaky-test.md b/.claude/commands/fix-flaky-test.mdindex be132f94cca..dd1edee06ce 100644--- a/.claude/commands/fix-flaky-test.md+++ b/.claude/commands/fix-flaky-test.md@@ -11,9 +11,14 @@ The Linear issue is the input. Everything else is reasoning over files.**Hard rules — do not deviate:**- **Max 3 attempts.** One attempt = diagnose → fix → push → 2 stress runs → interpret.After 3 failed attempts, stop and hand back.-- **One upfront approval authorizes the whole run's pushes.** Ask once (Phase 3), before- the first push. That single OK covers every push/trigger for the rest of this run on- this branch. Never push before that OK.+- **Branch existence is the go/no-go gate.** Before starting, check whether the remote branch+ `automated-e2e-flake-fix-<TEAM_SLUG>-<ISSUE_NUMBER>` already exists (Phase 0). If it does, a run+ is already in flight (or already landed a fix) → **stop, do nothing**. If it doesn't,+ proceed through the whole loop and push without pausing for confirmation. This runs+ identically whether a human or automation triggered the command.+- **Linear access:** prefer the Linear MCP (`mcp__linear__*`); if it's unavailable (e.g.+ headless CI), fall back to the Linear REST/GraphQL API with a token — for both reads and+ comment writes.- **Each attempt triggers TWO stress runs in parallel** — `enable_network_throttling=true`and `=false` — on the same branch/commit. The fix is verified only if **both** pass.- **CI is the source of truth.** No local Cypress runs, no `bin/` stress scripts.@@ -32,15 +37,28 @@ The Linear issue is the input. Everything else is reasoning over files.body for failure detail / screenshots / stack, and capture the **ranked failurereasons** (Trunk lists them "most common" first) — #1 is what you fix first (Phase 1).- Otherwise treat it as a spec path or fuzzy test name (fallback; no Linear context).-2. **Resolve the spec file** from the exact test name:+2. **Compute the branch name + dedup gate (this is the go decision).** The branch name is+ derived **only** from the Linear ref, never the test name:+ `automated-e2e-flake-fix-<TEAM_SLUG>-<ISSUE_NUMBER>` (e.g. `DEV-2181` →+ `automated-e2e-flake-fix-DEV-2181`). Check whether it already exists on the remote:+ ```bash+ git ls-remote --exit-code --heads origin automated-e2e-flake-fix-<TEAM_SLUG>-<ISSUE_NUMBER>+ ```+ - **Exists (exit 0)** → a run is already in flight or already landed a fix → **stop, do+ nothing.** This is the whole dedup mechanism.+ - **Doesn't exist (exit 2)** → that's the **go** signal. Proceed without pausing for+ confirmation.+ (On the no-Linear fallback path there's no ref to build the name from; that path stays+ interactive and is for local debugging only.)+3. **Resolve the spec file** from the exact test name:```bashgrep -rl "<exact test name>" e2e/test/scenarios```Require **exactly one** match. If 0 or >1, stop and ask the user to disambiguate. (Trya shorter unique substring of the test name if the literal has special characters.)-3. **Decide `qa_db`:** scan the spec / its `describe` for external-DB helpers+4. **Decide `qa_db`:** scan the spec / its `describe` for external-DB helpers(postgres/mysql/mongo-writable) or an `@external` tag → `true`, else `false`.-4. **Write the state file** `local/claude/flake-fix/<DEV-id-or-slug>.md`:+5. **Write the state file** `local/claude/e2e-flake-fix/<TEAM_SLUG>-<ISSUE_NUMBER>.md`:```markdown# Flake fix: <test name>- linear: <DEV-#### or n/a>@@ -48,9 +66,8 @@ The Linear issue is the input. Everything else is reasoning over files.- test_name (grep): <exact test name>- qa_db: <true|false>- edition: ee- - branch: <fix-flake-...>+ - branch: automated-e2e-flake-fix-<TEAM_SLUG>-<ISSUE_NUMBER>- attempt: 0 / 3- - pushes_authorized: no## Failure reasons (Trunk, ranked — fix #1 first)1. <dominant / most common>@@ -86,11 +103,11 @@ The Linear issue is the input. Everything else is reasoning over files.## Phase 2 — Fix on a branch-- On **attempt 1 only**, create the branch (dashes only, no `/`):+- On **attempt 1 only**, create the branch computed in Phase 0 (dashes only, no `/`):```bash- git switch -c fix-flake-<DEV-id-or-test-slug>+ git switch -c automated-e2e-flake-fix-<TEAM_SLUG>-<ISSUE_NUMBER>```- Record it in the state file. On later attempts, keep committing to the same branch.+ On later attempts, keep committing to the same branch.- Apply a **minimal** fix:- **Default:** test-scoped (deterministic waits via `cy.intercept` + `cy.wait("@alias")`before asserting; positive anchor before a negative assertion; scoped selectors).@@ -117,11 +134,9 @@ and stops immediately (fast red + artifacts) — no need for a separate cheap saA good fix runs all 50 to green (full confirmation). So every attempt is a single pair ofruns at `burn_in=50` with `fail_fast=true` (throttle on + off).-- **Approval gate (once per run).** If `pushes_authorized: no`, ask the user now, showing:- branch name, spec, test name (grep), `qa_db`, edition, `build_jar`, and the plan (≤3 attempts; each =- a pair of `burn_in=50, fail_fast=true` runs, throttle on + off). On approval, set- `pushes_authorized: yes` in the state file. This OK covers all later pushes/triggers this- run — do not re-ask.+- **Push without pausing for confirmation.** The branch-existence check in Phase 0 was the+ go decision; if you reached this phase you're clear to commit, push, and trigger (same for+ a human or automated trigger).- Commit and push (first attempt only creates the branch; later attempts push the revisedfix), then trigger.@@ -172,10 +187,10 @@ done## Phase 4 — Wait + interpret (gh only)-- **Auto-poll** with `ScheduleWakeup` (~1200s cadence; setup alone is ~15–20 min) until- **both** runs leave `queued`/`in_progress`. Pass the same `/fix-flaky-test $ARGUMENTS`+- **Auto-poll** with `ScheduleWakeup` (~900s / 15 min cadence) until **both** runs leave+ `queued`/`in_progress`. Pass the same `/fix-flaky-test $ARGUMENTS`back so the loop resumes; re-read the state file on each wake (a manual wake must also- work). Don't poll faster than ~1200s.+ work). Don't poll faster than ~900s.- When both finish, read each conclusion:```bashgh run view <id> --json status,conclusion@@ -187,7 +202,7 @@ done- **Either `failure` / `timed_out`** → fix is not holding (`fail_fast` stopped at thefirst bad iteration, so the artifact is fresh and minimal):```bash- gh run download <failed-id> --dir local/claude/flake-fix/<id>-artifacts+ gh run download <failed-id> --dir local/claude/e2e-flake-fix/<failed-id>-artifacts```The Cypress error lives in the GitHub step log (`gh run view <id> --log-failed`) and thescreenshots/video — **not** in `logs/test.log` (that's only the backend health log).@@ -235,10 +250,15 @@ done- **Always — attach the run log to Linear (regardless of outcome).** As the final step onboth the fixed and exhausted paths, post the full state file- `local/claude/flake-fix/<id>.md` to the Linear issue as a comment titled+ `local/claude/e2e-flake-fix/<TEAM_SLUG>-<ISSUE_NUMBER>.md` to the Linear issue as a comment titled**"🤖 Flake-fix run log (audit trail)"** (fence it so it renders verbatim). This is theend-to-end audit trail — every hypothesis, fix, commit, run URL, and verdict — kept for ahuman reviewer and as a learning artifact even when the fix didn't land. Skip only on the- no-Linear fallback path. (Posting the markdown as a comment is sufficient; a file+ no-Linear fallback path.+ - **Lead the comment with a run-cost line** (above the fenced log):+ **total wall-clock time spent** on the run and **total tokens used** — e.g.+ `⏱ 2h 14m · 🪙 1.8M tokens (N attempts)`. Compute time from the first state-file+ timestamp to now; report tokens for the whole session (best available figure). This makes+ the cost of each automated run visible at a glance on the issue. (Posting the markdown as a comment is sufficient; a fileattachment via `mcp__linear__create_attachment_from_upload` is optional if the user wantsa downloadable file.)diff --git a/.github/workflows/claude-e2e-flake-fix.yml b/.github/workflows/claude-e2e-flake-fix.ymlnew file mode 100644index 00000000000..253a65430b6--- /dev/null+++ b/.github/workflows/claude-e2e-flake-fix.yml@@ -0,0 +1,174 @@+name: "Claude E2E Flake Fix"++# Runs the /fix-flaky-test command headlessly to fix a flaky e2e test tracked by a Linear+# issue. Triggered manually or by automation (e.g. ci-conductor) via workflow_dispatch /+# repository_dispatch. ci-conductor only dispatches this workflow — it never runs the agent+# itself. The agent opens a green, review-ready PR; a human always reviews and merges.++on:+ workflow_dispatch:+ inputs:+ issue:+ description: "Linear issue ref for the flaky test (e.g. DEV-2181)"+ required: true+ type: string+ repository_dispatch:+ types: [e2e-flake-fix]++concurrency:+ # One in-flight run per Linear issue. Belt-and-suspenders alongside the branch-existence+ # dedup the command itself performs.+ group: e2e-flake-fix-${{ github.event.inputs.issue || github.event.client_payload.issue }}+ cancel-in-progress: false++jobs:+ flake-fix:+ runs-on: ${{ vars.DEFAULT_RUNNER_KEY }}+ # 3 attempts x 2 stress runs (burn_in=50) can run for hours. If this bumps the runner's+ # job ceiling, decompose via workflow_run completion events + --resume (see DEV-2182).+ timeout-minutes: 360+ permissions:+ contents: write+ pull-requests: write+ id-token: write+ steps:+ - name: Resolve Linear issue ref+ id: ref+ env:+ DISPATCH_ISSUE: ${{ github.event.inputs.issue }}+ PAYLOAD_ISSUE: ${{ github.event.client_payload.issue }}+ run: |+ if [ "${{ github.event_name }}" = "repository_dispatch" ]; then+ REF="$PAYLOAD_ISSUE"+ else+ REF="$DISPATCH_ISSUE"+ fi+ if [ -z "$REF" ]; then+ echo "::error::No Linear issue ref supplied"+ exit 1+ fi+ echo "ref=$REF" >> "$GITHUB_OUTPUT"+ echo "branch=automated-e2e-flake-fix-$REF" >> "$GITHUB_OUTPUT"+ echo "Flake fix for Linear issue: $REF"++ # Check out metabase itself with the Metabase Automation token so the agent's+ # git push / gh dispatch run as that user (a PAT, so it triggers the stress workflow).+ - uses: actions/checkout@v6+ with:+ token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}+ fetch-depth: 0++ - name: Dedup — skip if the fix branch already exists+ id: dedup+ env:+ BRANCH: ${{ steps.ref.outputs.branch }}+ run: |+ if git ls-remote --exit-code --heads origin "$BRANCH" >/dev/null 2>&1; then+ echo "::notice::$BRANCH already exists — a run is in flight or already landed. Skipping."+ echo "exists=true" >> "$GITHUB_OUTPUT"+ else+ echo "exists=false" >> "$GITHUB_OUTPUT"+ fi++ - name: Configure git identity (Metabase Automation)+ if: steps.dedup.outputs.exists == 'false'+ run: |+ git config --global user.name "Metabase Automation"+ git config --global user.email "[email protected]"++ # Make the Linear MCP available to the agent (it prefers MCP, falls back to the+ # REST/GraphQL API with the same key when MCP is absent).+ - name: Register Linear MCP server+ if: steps.dedup.outputs.exists == 'false'+ env:+ LINEAR_API_KEY: ${{ secrets.LINEAR_API_KEY }}+ run: |+ jq -n \+ --arg key "$LINEAR_API_KEY" \+ '{+ mcpServers: {+ "linear-server": {+ url: "https://mcp.linear.app/mcp",+ type: "http",+ headers: { Authorization: ("Bearer " + $key) }+ }+ }+ }' > ~/.claude.json++ - name: Run /fix-flaky-test with Claude Code+ id: claude+ if: steps.dedup.outputs.exists == 'false'+ continue-on-error: true+ uses: anthropics/claude-code-action@787c5a0ce96a9a6cfb050ea0c8f4c05f2447c251 # v1.0.133+ env:+ GH_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}+ GITHUB_TOKEN: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}+ LINEAR_API_KEY: ${{ secrets.LINEAR_API_KEY }}+ GITHUB_ACTIONS_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}+ timeout-minutes: 350+ with:+ anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}+ github_token: ${{ secrets.METABASE_AUTOMATION_USER_TOKEN }}+ allowed_bots: linear+ prompt: |+ Read and follow the instructions in .claude/commands/fix-flaky-test.md to fix the+ flaky e2e test tracked by Linear issue ${{ steps.ref.outputs.ref }}. Treat+ "${{ steps.ref.outputs.ref }}" as the command's $ARGUMENTS.++ ## CI Environment Notes+ - You are running headlessly in GitHub Actions as the "Metabase Automation" user.+ git, gh, and the Linear MCP are pre-configured — no human is watching, so never+ ask for approval. The branch-existence dedup gate in Phase 0 is the only go/no-go.+ - The fix branch is `${{ steps.ref.outputs.branch }}`.+ - Polling: `ScheduleWakeup` is not available here. Instead, block on the stress runs+ with `gh run watch <run-id> --exit-status` (the job is long-lived by design).+ - NEVER merge: do not run `gh pr merge` or enable auto-merge. Stop at a green,+ review-ready PR. A human owns the merge.+ - If a phase fails catastrophically, post the failure (with GITHUB_ACTIONS_RUN_URL)+ to the Linear issue before exiting, per the command's audit-trail step.+ claude_args: >-+ --allowedTools+ "Glob,+ Grep,+ Read,+ ToolSearch,+ Skill(*),+ WebFetch(domain:github.com),+ Edit(//./**),+ Edit(/tmp/**),+ Write(//./**),+ Write(/tmp/**),+ Bash(cat *),+ Bash(cd *),+ Bash(diff *),+ Bash(echo *),+ Bash(find *),+ Bash(gh *),+ Bash(git *),+ Bash(grep *),+ Bash(head *),+ Bash(jq *),+ Bash(ls *),+ Bash(mkdir *),+ Bash(printf *),+ Bash(sed *),+ Bash(sort *),+ Bash(tail *),+ Bash(tee *),+ Bash(wc *),+ Bash(which *),+ mcp__linear-server__get_issue,+ mcp__linear-server__list_comments,+ mcp__linear-server__list_issues,+ mcp__linear-server__save_comment,+ mcp__linear-server__create_attachment_from_upload"+ --disallowedTools "Bash(gh pr merge*),Bash(git push origin master*),Bash(rm -rf /*)"++ - name: Verify the run did not crash+ if: steps.dedup.outputs.exists == 'false'+ run: |+ if [ "${{ steps.claude.outcome }}" = "failure" ]; then+ echo "::error::Claude Code crashed or exited with an error"+ exit 1+ fi+ echo "Flake-fix run completed (see the Linear issue for the audit trail and PR)."