The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Add cypress claude setup (#70715)
Aleksandr Lesnenko5537f0f258b0
diff --git a/.claude/skills/e2e-test-create/SKILL.md b/.claude/skills/e2e-test-create/SKILL.mdnew file mode 100644index 00000000000..8a9435f03db--- /dev/null+++ b/.claude/skills/e2e-test-create/SKILL.md@@ -0,0 +1,260 @@+---+name: e2e-test-create+description: >+ Analyze React component source code to understand UI structure,+ then generate idiomatic Cypress E2E tests following Metabase conventions.+ Falls back to Playwright MCP browser exploration only when code reading+ and screenshot debugging are insufficient.+disable-model-invocation: true+allowed-tools:+ - Bash+ - Read+ - Write+ - Grep+ - Glob+ - Skill+ - "mcp__playwright__*"+---++# Code-Reading-First → Generate Cypress Tests (Metabase)++You are writing Cypress E2E tests for the **Metabase** codebase.+Before generating ANY test code, you MUST analyze React component source code+to understand DOM structure, selectors, and user flows.++## Phase 0 — Research++1. Read existing helpers before writing anything:+ - `e2e/support/helpers/` — all shared helpers (restore, signInAs, openOrdersTable, etc.)+ - `e2e/support/cypress_sample_database.ts` — table/field schema constants (ORDERS, PRODUCTS, etc.)+ - `e2e/support/cypress_sample_instance_data.ts` — instance-specific IDs (ORDERS_DASHBOARD_ID, NORMAL_USER_ID, etc.)+2. Glob `e2e/test/scenarios/` to find the closest existing spec to the area under test.+ Study its patterns — match them exactly.+3. Glob `frontend/src/metabase/` to find React components for the feature area.++## Phase 1 — Code Analysis++Read React component source to understand DOM structure. No browser needed — source code has everything.++1. **Find relevant components**: Glob and grep `frontend/src/metabase/` for the feature area.+2. **Extract selectors**: Grep for `data-testid` in relevant components.+3. **Note visible text**: Read component JSX for button labels, headings, placeholders.+4. **Note aria attributes**: Grep for `aria-label` in relevant components.+5. **Understand user flows**: Read event handlers (onClick, onSubmit, onChange) to understand interactions.+6. **Find API calls**: Grep for `Api.use`, `fetch`, `useQuery`, endpoint definitions to identify API calls to intercept.+7. **Cross-reference with existing specs**: Find specs in the same area and reuse their proven selectors and `cy.intercept` patterns.++## Phase 2 — Start Backend++Use `MB_EDITION=oss` by default. Only use `MB_EDITION=ee` when the user explicitly asks to write an enterprise test.++Start the backend using `run_in_background: true` (NOT `&`).+`bin/e2e-backend` automatically detects if a backend is already running and reuses it.+```bash+MB_EDITION=oss bin/e2e-backend+```++Do NOT manually generate snapshots by running unrelated test specs.+The `bun test-cypress` runner has `GENERATE_SNAPSHOTS: true` by default and automatically+generates snapshots before running any spec. When running tests in Phase 4 via the `/e2e-test` skill,+snapshots will be generated on the first run if they don't already exist.++Restore clean test data:+```bash+curl -sf -X POST http://localhost:4000/api/testing/restore/default+```++## Phase 3 — Generate Cypress Spec++### File location & naming++Place specs in `e2e/test/scenarios/<area>/` mirroring the URL structure.+Name: `<feature>.cy.spec.js` (NOT `.cy.spec.ts`).++### Metabase conventions (mandatory)++- **Helpers via `cy.H`**: All helpers are accessed via `const { H } = cy;` — NOT via direct imports from `e2e/support/helpers`.++```js+const { H } = cy;++describe("feature name", () => {+ beforeEach(() => {+ H.restore();+ cy.signInAsAdmin();+ });+});+```++- **Sample Database schema** (table/field definitions): Import from `cypress_sample_database`.++```js+import { ORDERS, ORDERS_ID, PRODUCTS } from "e2e/support/cypress_sample_database";+```++- **Instance data** (dashboard IDs, user IDs, etc.): Import from `cypress_sample_instance_data`.++```js+import { ORDERS_DASHBOARD_ID } from "e2e/support/cypress_sample_instance_data";+```++- **Selectors** (priority order):+ 1. `cy.findByText()` / `cy.findByLabelText()` / `cy.findByRole()` — from `@testing-library/cypress`+ 2. `cy.findByTestId()` — for `data-testid` attributes+ 3. `cy.get("[data-testid='...']")` — fallback+ 4. NEVER use positional selectors, CSS class names, or XPaths.++- **Navigation helpers**: Use existing helpers like `H.openOrdersTable()`, `H.openNativeEditor()`,+ `H.visitDashboard(id)`, `H.visitQuestion(id)` instead of raw `cy.visit()` chains.+ Grep `e2e/support/helpers/` to discover what's available for your area.++- **API setup over UI setup**: Use `cy.request()` or existing API helpers to set up state.+ Only use the UI for the flow you're actually testing.++- **Assertions**: Assert on visible text, URL, aria state — not DOM structure.++- **Waits**: Never use `cy.wait(ms)`. Use `cy.intercept()` + `cy.wait("@alias")` for API calls,+ or `cy.findByText().should("be.visible")` for DOM readiness.++- **Isolation**: Each `it()` block must be independently runnable.+ Don't depend on state from a previous `it()`.++### Spec structure template++```js+const { H } = cy;+import { ORDERS_DASHBOARD_ID } from "e2e/support/cypress_sample_instance_data";+import { ORDERS, ORDERS_ID } from "e2e/support/cypress_sample_database";++describe("area > sub-area > feature (#issue-number)", () => {+ beforeEach(() => {+ H.restore();+ cy.signInAsAdmin();+ });++ it("should do the primary happy-path thing", () => {+ // test+ });++ it("should handle the edge case", () => {+ // test+ });+});+```++### Intercepts++When you identified API calls during code analysis, stub or wait on them:++```js+cy.intercept("POST", "/api/dataset").as("dataset");+// ... trigger action ...+cy.wait("@dataset");+```++## Phase 4 — Validate++After generating specs:+1. Check that all imported helpers exist (Grep `e2e/support/helpers/`).+2. **You MUST use the `/e2e-test` skill** to run tests — do NOT run `bun test-cypress` directly.+ The `/e2e-test` skill handles edition selection, snapshot management, and correct env vars.+ `/e2e-test GREP="should do the thing" --spec e2e/test/scenarios/<path>`+ If you created multiple `it()` blocks, run each one individually to isolate failures.++## Phase 5 — Fix Failures (up to 2 attempts)++When a test fails, **try to fix it from Cypress output first**:+1. Read the failure screenshot (path printed under `(Screenshots)`).+2. Read the error message and code frame from the console output.+3. Fix the test and re-run (back to Phase 4, step 2).++If you cannot diagnose the issue after 2 attempts, proceed to Phase 6.++## Phase 6 — Playwright Fallback++Only reach this phase after 2 failed fix attempts from Phase 5. The backend is already running.++Restore clean test data:+```bash+curl -sf -X POST http://localhost:4000/api/testing/restore/default+```++**Bypass CSP** headers before navigating (Metabase serves strict CSP that blocks dev server scripts).+Use `browser_run_code` to set this up:++```js+async (page) => {+ // Strip CSP headers so the page loads (mirrors Cypress chromeWebSecurity: false)+ await page.context().route('**/*', async (route) => {+ const response = await route.fetch();+ const headers = { ...response.headers() };+ delete headers['content-security-policy'];+ delete headers['content-security-policy-report-only'];+ await route.fulfill({ response, headers });+ });++ // Sign in via API+ const response = await page.request.post('http://localhost:4000/api/session', {+ data: { username: '[email protected]', password: '12341234' }+ });+ const session = await response.json();+ await page.context().addCookies([{+ name: 'metabase.DEVICE',+ value: session.id,+ domain: 'localhost',+ path: '/'+ }]);++ await page.goto('http://localhost:4000');+ await page.waitForLoadState('networkidle');+ return 'signed in';+}+```++**Maintain an observation log incrementally.** After EVERY significant Playwright interaction,+IMMEDIATELY append what you observed to the scratch file BEFORE performing the next interaction:++```bash+cat >> /tmp/e2e-observations.md << 'OBSERVATION'+## [Page/Flow name]+- URL: /question/notebook#...+- Clicked: "Box plot" button → visible text "Box plot", role: radio+- Selectors: data-testid="viz-type-button", findByText("Box plot")+- API call: POST /api/dataset (triggered on viz change)+- Key state: after selecting viz type, summary sidebar shows metric picker+OBSERVATION+```++For each page/flow:+- Take an accessibility snapshot (`browser_snapshot`).+- Click through interactive elements, fill forms, trigger modals.+- **Append to the observation log immediately after each step**: URLs, visible text, aria labels, `data-testid` attrs, API calls.+- Screenshot key states.++After exploration:+1. Read back your observation log: `cat /tmp/e2e-observations.md`+2. Fix the test using observed selectors and behavior.+3. Re-run the test (back to Phase 4).+4. Clean up: `rm -f /tmp/e2e-observations.md`++## Phase 7 — Cleanup++After all tests pass (or after giving up on fixing failures), **always kill the backend on port 4000**:+```bash+lsof -ti:4000 | xargs kill 2>/dev/null || true+```++Do NOT use broad `pkill` patterns — there may be other Metabase instances on different ports.+The backend process started in Phase 2 will NOT be killed automatically when the Claude session ends.+Leaving it running wastes resources and can interfere with future sessions. Always clean up.++## What NOT to do++- Do NOT use Playwright as the first step — always analyze source code first.+- Do NOT kill the backend between phases — it stays running throughout.+- Do NOT invent selectors you didn't find in source code or observe in the browser.+- Do NOT hardcode database IDs — import from `cypress_sample_database` or `cypress_sample_instance_data`.+- Do NOT use `cy.wait(1000)` or any numeric wait.+- Do NOT create setup flows through the UI when an API helper exists.+- Do NOT put tests in `cypress/e2e/` — Metabase uses `e2e/test/scenarios/`.+- Do NOT import helpers directly from `e2e/support/helpers` — use `const { H } = cy;`.diff --git a/.claude/skills/e2e-test/SKILL.md b/.claude/skills/e2e-test/SKILL.mdnew file mode 100644index 00000000000..c4b78145ace--- /dev/null+++ b/.claude/skills/e2e-test/SKILL.md@@ -0,0 +1,78 @@+---+name: e2e-test+description: Run Cypress E2E tests, analyze failures including screenshots, and stress test for flakiness+disable-model-invocation: false+---++Run Cypress E2E test: $ARGUMENTS++## Edition selection++**Default: `MB_EDITION=oss`** — runs without enterprise tokens, simpler and faster.++Only use `MB_EDITION=ee` when the user explicitly asks to write or run an enterprise test.++**For `MB_EDITION=ee`:**+These tokens must be exported in the shell before starting Claude Code:+- `CYPRESS_MB_ALL_FEATURES_TOKEN` — all features (most EE tests need this)+- `CYPRESS_MB_PRO_SELF_HOSTED_TOKEN` — pro self-hosted (permissions, SSO, sandboxing)+- `CYPRESS_MB_STARTER_CLOUD_TOKEN` — starter plan (rarely needed)+- `CYPRESS_MB_PRO_CLOUD_TOKEN` — pro cloud (rarely needed)++**NEVER echo, print, or log token values. Only check if they are set:**+```bash+echo "ALL_FEATURES: ${CYPRESS_MB_ALL_FEATURES_TOKEN:+set}" && echo "PRO_SELF_HOSTED: ${CYPRESS_MB_PRO_SELF_HOSTED_TOKEN:+set}" && echo "STARTER: ${CYPRESS_MB_STARTER_CLOUD_TOKEN:+set}" && echo "PRO_CLOUD: ${CYPRESS_MB_PRO_CLOUD_TOKEN:+set}"+```++If tokens are missing, tell the user: "EE tokens are not set. Either export them and restart Claude Code, or add `MB_EDITION=oss` to run OSS-only tests."++## Running++First check if snapshots exist:+```bash+ls e2e/snapshots/default.sql 2>/dev/null && echo "snapshots exist" || echo "snapshots missing"+```++If snapshots exist, skip regeneration for speed:+```bash+MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false GENERATE_SNAPSHOTS=false bun test-cypress $ARGUMENTS+```++If snapshots are missing (first run), let the runner generate them:+```bash+MB_EDITION=oss CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false bun test-cypress $ARGUMENTS+```++When running enterprise tests, replace `MB_EDITION=oss` with `MB_EDITION=ee` in the commands above.++If `$ARGUMENTS` is empty, ask the user which spec to run.++If the user provides a test name or fuzzy name instead of a spec path, find the spec file:+```bash+find e2e/test/scenarios -name "*FUZZY_NAME*.cy.spec.*" -type f+```+Then run with `--spec "path/to/matched.cy.spec.js"`.++To filter by test name, use the `GREP` env var (not `--env grep=`, which breaks on commas):+```bash+GREP="test name here" MB_EDITION=oss bun test-cypress --spec path/to/spec.js+```++## Stress test (flaky detection)++When the user asks to verify a test is not flaky:++```bash+MB_EDITION=oss bin/e2e-stress-test $ARGUMENTS+```++Set `E2E_STRESS_RUNS=N` for more iterations (default: 5). Stops on first failure and prints screenshot paths. Read those screenshots to analyze the failure.++## Analyzing failures++When tests fail:++1. Read the **screenshot** for each failure — paths are printed in Cypress output under `(Screenshots)`, typically at `cypress/screenshots/<spec-name>/<test-name> (failed).png`+2. Read the error message and code frame from the console output+3. Identify the root cause: is it a test bug, a product bug, or a timing issue?+4. Suggest a fix with specific code changesdiff --git a/bin/e2e-backend b/bin/e2e-backendnew file mode 100755index 00000000000..f53e4d7310c--- /dev/null+++ b/bin/e2e-backend@@ -0,0 +1,11 @@+#!/usr/bin/env bash+set -euo pipefail++PORT="${MB_JETTY_PORT:-4000}"++if curl -sf "http://localhost:$PORT/api/health" > /dev/null 2>&1; then+ echo "⚠️ Backend already running on :$PORT — reusing it. If you need a different MB_EDITION, kill it first: lsof -ti:$PORT | xargs kill"+ exit 0+fi++exec node e2e/runner/start-backend.jsdiff --git a/bin/e2e-stress-test b/bin/e2e-stress-testnew file mode 100755index 00000000000..ca1ae29ce48--- /dev/null+++ b/bin/e2e-stress-test@@ -0,0 +1,42 @@+#!/usr/bin/env bash+set -euo pipefail++RUNS="${E2E_STRESS_RUNS:-5}"++if [ $# -eq 0 ]; then+ echo "Usage: bin/e2e-stress-test [cypress args...]"+ echo " e.g. bin/e2e-stress-test --spec e2e/test/scenarios/organization/bookmarks-dashboard.cy.spec.js"+ echo ""+ echo "Set E2E_STRESS_RUNS=N to change iteration count (default: 5)"+ exit 1+fi++rm -rf cypress/screenshots cypress/videos++for i in $(seq 1 "$RUNS"); do+ echo ""+ echo "=========================================="+ echo " Run $i / $RUNS"+ echo "=========================================="+ echo ""++ if ! CYPRESS_VIDEO=false CYPRESS_RETRIES=0 CYPRESS_GUI=false GENERATE_SNAPSHOTS=false bun test-cypress "$@"; then+ echo ""+ echo "=========================================="+ echo " FAILED on run $i / $RUNS"+ echo "=========================================="++ if ls cypress/screenshots/**/*.png 1>/dev/null 2>&1; then+ echo ""+ echo "Screenshots:"+ find cypress/screenshots -name "*.png" -print+ fi++ exit 1+ fi+done++echo ""+echo "=========================================="+echo " All $RUNS runs passed"+echo "=========================================="diff --git a/e2e/runner/start-backend.js b/e2e/runner/start-backend.jsnew file mode 100644index 00000000000..0e6ab8c68a7--- /dev/null+++ b/e2e/runner/start-backend.js@@ -0,0 +1,27 @@+#!/usr/bin/env node++const backend = require("./cypress-runner-backend");++async function main() {+ const jarPath = process.env.JAR_PATH;++ if (jarPath) {+ await backend.runFromJar(jarPath);+ } else {+ await backend.runFromSource();+ }++ console.log(`Backend ready on :${process.env.MB_JETTY_PORT}`);++ const cleanup = () => {+ backend.stop();+ process.exit();+ };+ process.on("SIGTERM", cleanup);+ process.on("SIGINT", cleanup);+}++main().catch((error) => {+ console.error(error);+ process.exit(1);+});diff --git a/e2e/support/config.js b/e2e/support/config.jsindex d8d712bb13b..98d99f9f383 100644--- a/e2e/support/config.js+++ b/e2e/support/config.js@@ -181,7 +181,7 @@ const defaultConfig = {viewportHeight: 800,viewportWidth: 1280,// enable video recording in run mode- video: true,+ video: process.env["CYPRESS_VIDEO"] !== "false",videoCompression: true,};@@ -213,7 +213,10 @@ const mainConfig = {},},retries: {- runMode: 2,+ runMode:+ process.env["CYPRESS_RETRIES"] != null+ ? parseInt(process.env["CYPRESS_RETRIES"], 10)+ : 2,openMode: 0,},};