Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Add cypress claude setup (#70715)

Aleksandr Lesnenko5537f0f258b0
diff --git a/.claude/skills/e2e-test-create/SKILL.md b/.claude/skills/e2e-test-create/SKILL.md
new file mode 100644
index 00000000000..8a9435f03db
--- /dev/null
+++ b/.claude/skills/e2e-test-create/SKILL.md
@@ -0,0 +1,260 @@
+---
+name: e2e-test-create
+description: >
+ Analyze React component source code to understand UI structure,
+ then generate idiomatic Cypress E2E tests following Metabase conventions.
+ Falls back to Playwright MCP browser exploration only when code reading
+ and screenshot debugging are insufficient.
+disable-model-invocation: true
+allowed-tools:
+ - Bash
+ - Read
+ - Write
+ - Grep
+ - Glob
+ - Skill
+ - "mcp__playwright__*"
+---
+
+# Code-Reading-First → Generate Cypress Tests (Metabase)
+
+You are writing Cypress E2E tests for the **Metabase** codebase.
+Before generating ANY test code, you MUST analyze React component source code
+to understand DOM structure, selectors, and user flows.
+
+## Phase 0 — Research
+
+1. Read existing helpers before writing anything:
+ - `e2e/support/helpers/` — all shared helpers (restore, signInAs, openOrdersTable, etc.)
+ - `e2e/support/cypress_sample_database.ts` — table/field schema constants (ORDERS, PRODUCTS, etc.)
+ - `e2e/support/cypress_sample_instance_data.ts` — instance-specific IDs (ORDERS_DASHBOARD_ID, NORMAL_USER_ID, etc.)
+2. Glob `e2e/test/scenarios/` to find the closest existing spec to the area under test.
+ Study its patterns — match them exactly.
+3. Glob `frontend/src/metabase/` to find React components for the feature area.
+
+## Phase 1 — Code Analysis
+
+Read React component source to understand DOM structure. No browser needed — source code has everything.
+
+1. **Find relevant components**: Glob and grep `frontend/src/metabase/` for the feature area.
+2. **Extract selectors**: Grep for `data-testid` in relevant components.
+3. **Note visible text**: Read component JSX for button labels, headings, placeholders.
+4. **Note aria attributes**: Grep for `aria-label` in relevant components.
+5. **Understand user flows**: Read event handlers (onClick, onSubmit, onChange) to understand interactions.
+6. **Find API calls**: Grep for `Api.use`, `fetch`, `useQuery`, endpoint definitions to identify API calls to intercept.
+7. **Cross-reference with existing specs**: Find specs in the same area and reuse their proven selectors and `cy.intercept` patterns.
+
+## Phase 2 — Start Backend
+
+Use `MB_EDITION=oss` by default. Only use `MB_EDITION=ee` when the user explicitly asks to write an enterprise test.
+
+Start the backend using `run_in_background: true` (NOT `&`).
+`bin/e2e-backend` automatically detects if a backend is already running and reuses it.
+```bash
+MB_EDITION=oss bin/e2e-backend
+```
+
+Do NOT manually generate snapshots by running unrelated test specs.
+The `bun test-cypress` runner has `GENERATE_SNAPSHOTS: true` by default and automatically
+generates snapshots before running any spec. When running tests in Phase 4 via the `/e2e-test` skill,
+snapshots will be generated on the first run if they don't already exist.
+
+Restore clean test data:
+```bash
+curl -sf -X POST http://localhost:4000/api/testing/restore/default
+```
+
+## Phase 3 — Generate Cypress Spec
+
+### File location & naming
+
+Place specs in `e2e/test/scenarios/<area>/` mirroring the URL structure.
+Name: `<feature>.cy.spec.js` (NOT `.cy.spec.ts`).
+
+### Metabase conventions (mandatory)
+
+- **Helpers via `cy.H`**: All helpers are accessed via `const { H } = cy;` — NOT via direct imports from `e2e/support/helpers`.
+
+```js
+const { H } = cy;
+
+describe("feature name", () => {
+ beforeEach(() => {
+ H.restore();
+ cy.signInAsAdmin();
+ });
+});
+```
+
+- **Sample Database schema** (table/field definitions): Import from `cypress_sample_database`.
+
+```js
+import { ORDERS, ORDERS_ID, PRODUCTS } from "e2e/support/cypress_sample_database";
+```
+
+- **Instance data** (dashboard IDs, user IDs, etc.): Import from `cypress_sample_instance_data`.
+
+```js
+import { ORDERS_DASHBOARD_ID } from "e2e/support/cypress_sample_instance_data";
+```
+
+- **Selectors** (priority order):
+ 1. `cy.findByText()` / `cy.findByLabelText()` / `cy.findByRole()` — from `@testing-library/cypress`
+ 2. `cy.findByTestId()` — for `data-testid` attributes
+ 3. `cy.get("[data-testid='...']")` — fallback
+ 4. NEVER use positional selectors, CSS class names, or XPaths.
+
+- **Navigation helpers**: Use existing helpers like `H.openOrdersTable()`, `H.openNativeEditor()`,
+ `H.visitDashboard(id)`, `H.visitQuestion(id)` instead of raw `cy.visit()` chains.
+ Grep `e2e/support/helpers/` to discover what's available for your area.
+
+- **API setup over UI setup**: Use `cy.request()` or existing API helpers to set up state.
+ Only use the UI for the flow you're actually testing.
+
+- **Assertions**: Assert on visible text, URL, aria state — not DOM structure.
+
+- **Waits**: Never use `cy.wait(ms)`. Use `cy.intercept()` + `cy.wait("@alias")` for API calls,
+ or `cy.findByText().should("be.visible")` for DOM readiness.
+
+- **Isolation**: Each `it()` block must be independently runnable.
+ Don't depend on state from a previous `it()`.
+
+### Spec structure template
+
+```js
+const { H } = cy;
+import { ORDERS_DASHBOARD_ID } from "e2e/support/cypress_sample_instance_data";
+import { ORDERS, ORDERS_ID } from "e2e/support/cypress_sample_database";
+
+describe("area > sub-area > feature (#issue-number)", () => {
+ beforeEach(() => {
+ H.restore();
+ cy.signInAsAdmin();
+ });
+
+ it("should do the primary happy-path thing", () => {
+ // test
+ });
+
+ it("should handle the edge case", () => {
+ // test
+ });
+});
+```
+
+### Intercepts
+
+When you identified API calls during code analysis, stub or wait on them:
+
+```js
+cy.intercept("POST", "/api/dataset").as("dataset");
+// ... trigger action ...
+cy.wait("@dataset");
+```
+
+## Phase 4 — Validate
+
+After generating specs:
+1. Check that all imported helpers exist (Grep `e2e/support/helpers/`).
+2. **You MUST use the `/e2e-test` skill** to run tests — do NOT run `bun test-cypress` directly.
+ The `/e2e-test` skill handles edition selection, snapshot management, and correct env vars.
+ `/e2e-test GREP="should do the thing" --spec e2e/test/scenarios/<path>`
+ If you created multiple `it()` blocks, run each one individually to isolate failures.
+
+## Phase 5 — Fix Failures (up to 2 attempts)
+
+When a test fails, **try to fix it from Cypress output first**:
+1. Read the failure screenshot (path printed under `(Screenshots)`).
+2. Read the error message and code frame from the console output.
+3. Fix the test and re-run (back to Phase 4, step 2).
+
+If you cannot diagnose the issue after 2 attempts, proceed to Phase 6.
+
+## Phase 6 — Playwright Fallback
+
+Only reach this phase after 2 failed fix attempts from Phase 5. The backend is already running.
+
+Restore clean test data:
+```bash
+curl -sf -X POST http://localhost:4000/api/testing/restore/default
+```
+
+**Bypass CSP** headers before navigating (Metabase serves strict CSP that blocks dev server scripts).
+Use `browser_run_code` to set this up:
+
+```js
+async (page) => {
+ // Strip CSP headers so the page loads (mirrors Cypress chromeWebSecurity: false)
+ await page.context().route('**/*', async (route) => {
+ const response = await route.fetch();
+ const headers = { ...response.headers() };
+ delete headers['content-security-policy'];
+ delete headers['content-security-policy-report-only'];
+ await route.fulfill({ response, headers });
+ });
+
+ // Sign in via API
+ const response = await page.request.post('http://localhost:4000/api/session', {
+ data: { username: '[email protected]', password: '12341234' }
+ });
+ const session = await response.json();
+ await page.context().addCookies([{
+ name: 'metabase.DEVICE',
+ value: session.id,
+ domain: 'localhost',
+ path: '/'
+ }]);
+
+ await page.goto('http://localhost:4000');
+ await page.waitForLoadState('networkidle');
+ return 'signed in';
+}
+```
+
+**Maintain an observation log incrementally.** After EVERY significant Playwright interaction,
+IMMEDIATELY append what you observed to the scratch file BEFORE performing the next interaction:
+
+```bash
+cat >> /tmp/e2e-observations.md << 'OBSERVATION'
+## [Page/Flow name]
+- URL: /question/notebook#...
+- Clicked: "Box plot" button → visible text "Box plot", role: radio
+- Selectors: data-testid="viz-type-button", findByText("Box plot")
+- API call: POST /api/dataset (triggered on viz change)
+- Key state: after selecting viz type, summary sidebar shows metric picker
+OBSERVATION
+```
+
+For each page/flow:
+- Take an accessibility snapshot (`browser_snapshot`).
+- Click through interactive elements, fill forms, trigger modals.
+- **Append to the observation log immediately after each step**: URLs, visible text, aria labels, `data-testid` attrs, API calls.
+- Screenshot key states.
+
+After exploration:
+1. Read back your observation log: `cat /tmp/e2e-observations.md`
+2. Fix the test using observed selectors and behavior.
+3. Re-run the test (back to Phase 4).
+4. Clean up: `rm -f /tmp/e2e-observations.md`
+
+## Phase 7 — Cleanup
+
+After all tests pass (or after giving up on fixing failures), **always kill the backend on port 4000**:
+```bash
+lsof -ti:4000 | xargs kill 2>/dev/null || true
+```
+
+Do NOT use broad `pkill` patterns — there may be other Metabase instances on different ports.
+The backend process started in Phase 2 will NOT be killed automatically when the Claude session ends.
+Leaving it running wastes resources and can interfere with future sessions. Always clean up.
+
+## What NOT to do
+
+- Do NOT use Playwright as the first step — always analyze source code first.
+- Do NOT kill the backend between phases — it stays running throughout.
+- Do NOT invent selectors you didn't find in source code or observe in the browser.
+- Do NOT hardcode database IDs — import from `cypress_sample_database` or `cypress_sample_instance_data`.
+- Do NOT use `cy.wait(1000)` or any numeric wait.
+- Do NOT create setup flows through the UI when an API helper exists.
+- Do NOT put tests in `cypress/e2e/` — Metabase uses `e2e/test/scenarios/`.
+- Do NOT import helpers directly from `e2e/support/helpers` — use `const { H } = cy;`.
diff --git a/.claude/skills/e2e-test/SKILL.md b/.claude/skills/e2e-test/SKILL.md
new file mode 100644
index 00000000000..c4b78145ace
--- /dev/null
+++ b/.claude/skills/e2e-test/SKILL.md
@@ -0,0 +1,78 @@
+---
+name: e2e-test
+description: Run Cypress E2E tests, analyze failures including screenshots, and stress test for flakiness
+disable-model-invocation: false
+---
+
+Run Cypress E2E test: $ARGUMENTS
+
+## Edition selection
+
+**Default: `MB_EDITION=oss`** — runs without enterprise tokens, simpler and faster.
+
+Only use `MB_EDITION=ee` when the user explicitly asks to write or run an enterprise test.
+
+**For `MB_EDITION=ee`:**
+These tokens must be exported in the shell before starting Claude Code:
+- `CYPRESS_MB_ALL_FEATURES_TOKEN` — all features (most EE tests need this)
+- `CYPRESS_MB_PRO_SELF_HOSTED_TOKEN` — pro self-hosted (permissions, SSO, sandboxing)
+- `CYPRESS_MB_STARTER_CLOUD_TOKEN` — starter plan (rarely needed)
+- `CYPRESS_MB_PRO_CLOUD_TOKEN` — pro cloud (rarely needed)
+
+**NEVER echo, print, or log token values. Only check if they are set:**
+```bash
+echo "ALL_FEATURES: ${CYPRESS_MB_ALL_FEATURES_TOKEN:+set}" && echo "PRO_SELF_HOSTED: ${CYPRESS_MB_PRO_SELF_HOSTED_TOKEN:+set}" && echo "STARTER: ${CYPRESS_MB_STARTER_CLOUD_TOKEN:+set}" && echo "PRO_CLOUD: ${CYPRESS_MB_PRO_CLOUD_TOKEN:+set}"
+```
+
+If tokens are missing, tell the user: "EE tokens are not set. Either export them and restart Claude Code, or add `MB_EDITION=oss` to run OSS-only tests."
+
+## Running
+
+First check if snapshots exist:
+```bash
+ls e2e/snapshots/default.sql 2>/dev/null && echo "snapshots exist" || echo "snapshots missing"
+```
+
+If snapshots exist, skip regeneration for speed:
+```bash
+MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false GENERATE_SNAPSHOTS=false bun test-cypress $ARGUMENTS
+```
+
+If snapshots are missing (first run), let the runner generate them:
+```bash
+MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false bun test-cypress $ARGUMENTS
+```
+
+When running enterprise tests, replace `MB_EDITION=oss` with `MB_EDITION=ee` in the commands above.
+
+If `$ARGUMENTS` is empty, ask the user which spec to run.
+
+If the user provides a test name or fuzzy name instead of a spec path, find the spec file:
+```bash
+find e2e/test/scenarios -name "*FUZZY_NAME*.cy.spec.*" -type f
+```
+Then run with `--spec "path/to/matched.cy.spec.js"`.
+
+To filter by test name, use the `GREP` env var (not `--env grep=`, which breaks on commas):
+```bash
+GREP="test name here" MB_EDITION=oss bun test-cypress --spec path/to/spec.js
+```
+
+## Stress test (flaky detection)
+
+When the user asks to verify a test is not flaky:
+
+```bash
+MB_EDITION=oss bin/e2e-stress-test $ARGUMENTS
+```
+
+Set `E2E_STRESS_RUNS=N` for more iterations (default: 5). Stops on first failure and prints screenshot paths. Read those screenshots to analyze the failure.
+
+## Analyzing failures
+
+When tests fail:
+
+1. Read the **screenshot** for each failure — paths are printed in Cypress output under `(Screenshots)`, typically at `cypress/screenshots/<spec-name>/<test-name> (failed).png`
+2. Read the error message and code frame from the console output
+3. Identify the root cause: is it a test bug, a product bug, or a timing issue?
+4. Suggest a fix with specific code changes
diff --git a/bin/e2e-backend b/bin/e2e-backend
new file mode 100755
index 00000000000..f53e4d7310c
--- /dev/null
+++ b/bin/e2e-backend
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+PORT="${MB_JETTY_PORT:-4000}"
+
+if curl -sf "http://localhost:$PORT/api/health" > /dev/null 2>&1; then
+ echo "⚠️ Backend already running on :$PORT — reusing it. If you need a different MB_EDITION, kill it first: lsof -ti:$PORT | xargs kill"
+ exit 0
+fi
+
+exec node e2e/runner/start-backend.js
diff --git a/bin/e2e-stress-test b/bin/e2e-stress-test
new file mode 100755
index 00000000000..ca1ae29ce48
--- /dev/null
+++ b/bin/e2e-stress-test
@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+RUNS="${E2E_STRESS_RUNS:-5}"
+
+if [ $# -eq 0 ]; then
+ echo "Usage: bin/e2e-stress-test [cypress args...]"
+ echo " e.g. bin/e2e-stress-test --spec e2e/test/scenarios/organization/bookmarks-dashboard.cy.spec.js"
+ echo ""
+ echo "Set E2E_STRESS_RUNS=N to change iteration count (default: 5)"
+ exit 1
+fi
+
+rm -rf cypress/screenshots cypress/videos
+
+for i in $(seq 1 "$RUNS"); do
+ echo ""
+ echo "=========================================="
+ echo " Run $i / $RUNS"
+ echo "=========================================="
+ echo ""
+
+ if ! CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false GENERATE_SNAPSHOTS=false bun test-cypress "$@"; then
+ echo ""
+ echo "=========================================="
+ echo " FAILED on run $i / $RUNS"
+ echo "=========================================="
+
+ if ls cypress/screenshots/**/*.png 1>/dev/null 2>&1; then
+ echo ""
+ echo "Screenshots:"
+ find cypress/screenshots -name "*.png" -print
+ fi
+
+ exit 1
+ fi
+done
+
+echo ""
+echo "=========================================="
+echo " All $RUNS runs passed"
+echo "=========================================="
diff --git a/e2e/runner/start-backend.js b/e2e/runner/start-backend.js
new file mode 100644
index 00000000000..0e6ab8c68a7
--- /dev/null
+++ b/e2e/runner/start-backend.js
@@ -0,0 +1,27 @@
+#!/usr/bin/env node
+
+const backend = require("./cypress-runner-backend");
+
+async function main() {
+ const jarPath = process.env.JAR_PATH;
+
+ if (jarPath) {
+ await backend.runFromJar(jarPath);
+ } else {
+ await backend.runFromSource();
+ }
+
+ console.log(`Backend ready on :${process.env.MB_JETTY_PORT}`);
+
+ const cleanup = () => {
+ backend.stop();
+ process.exit();
+ };
+ process.on("SIGTERM", cleanup);
+ process.on("SIGINT", cleanup);
+}
+
+main().catch((error) => {
+ console.error(error);
+ process.exit(1);
+});
diff --git a/e2e/support/config.js b/e2e/support/config.js
index d8d712bb13b..98d99f9f383 100644
--- a/e2e/support/config.js
+++ b/e2e/support/config.js
@@ -181,7 +181,7 @@ const defaultConfig = {
viewportHeight: 800,
viewportWidth: 1280,
// enable video recording in run mode
- video: true,
+ video: process.env["CYPRESS_VIDEO"] !== "false",
videoCompression: true,
};
@@ -213,7 +213,10 @@ const mainConfig = {
},
},
retries: {
- runMode: 2,
+ runMode:
+ process.env["CYPRESS_RETRIES"] != null
+ ? parseInt(process.env["CYPRESS_RETRIES"], 10)
+ : 2,
openMode: 0,
},
};