Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

ci: run the e2e matrix from the ready-for-e2e label (#16899)

Othman Abu Ajamiehd5a364540120
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index 72e406f6cca..30437130a89 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -1,144 +1,23 @@
name: E2E Tests
+# Runs the E2E matrix on a PR that carries the ready-for-e2e label.
on:
- pull_request_target:
- types: [opened, synchronize, reopened, labeled]
+ pull_request:
+ types: [opened, reopened, synchronize, labeled]
branches:
- main
- - gh-actions-test-branch
- workflow_dispatch:
permissions:
- actions: write
contents: read
- issues: write
- pull-requests: write
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
- cancel-in-progress: true
+ packages: read
jobs:
- check-label:
- runs-on: ubuntu-latest
- outputs:
- should-run-e2e: ${{ steps.check-if-pr-has-label.outputs.run-e2e }}
- steps:
- - name: Check if PR exists with ready-for-e2e label for this SHA
- id: check-if-pr-has-label
- uses: actions/github-script@v8
- with:
- script: |
- // Always run on manual workflow dispatch
- if (context.eventName === 'workflow_dispatch') {
- core.setOutput('run-e2e', true);
- return;
- }
-
- let labels = [];
- if (context.payload.pull_request) {
- labels = context.payload.pull_request.labels;
- } else {
- try {
- const sha = context.sha;
- const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
- owner: context.repo.owner,
- repo: context.repo.repo,
- commit_sha: sha
- });
- if (prs.length === 0) {
- core.setOutput('run-e2e', false);
- return;
- }
- const pr = prs[0];
- labels = pr.labels;
- } catch (e) {
- core.setOutput('run-e2e', false);
- return;
- }
- }
-
- const labelFound = labels.map(l => l.name).includes('ready-for-e2e');
- core.setOutput('run-e2e', labelFound);
-
- test-e2e-ee:
- needs: check-label
- if: needs.check-label.outputs.should-run-e2e == 'true'
- uses: ./.github/workflows/tests-e2e-ee.yml
-
- test-e2e-ce:
- needs: check-label
- if: needs.check-label.outputs.should-run-e2e == 'true'
- uses: ./.github/workflows/tests-e2e-ce.yml
-
- notify-on-completion:
- needs: [check-label, test-e2e-ce, test-e2e-ee]
- if: always() && needs.check-label.outputs.should-run-e2e == 'true'
- runs-on: ubuntu-latest
- steps:
- - name: Download all artifacts
- uses: actions/download-artifact@v7
- with:
- path: ./artifacts
- pattern: "*"
- merge-multiple: false
-
- - name: Re-upload consolidated artifacts
- uses: actions/upload-artifact@v6
- if: always()
- with:
- name: e2e-test-artifacts-all
- path: ./artifacts/
- retention-days: 30
-
- - name: Comment on PR with test results
- if: always() && github.event_name == 'pull_request_target'
- uses: actions/github-script@v8
- with:
- script: |
- // Determine which tests failed
- const results = {
- ee: '${{ needs.test-e2e-ee.result }}',
- ce: '${{ needs.test-e2e-ce.result }}'
- };
-
- const failed = Object.entries(results).filter(([_, status]) => status === 'failure');
- const passed = Object.entries(results).filter(([_, status]) => status === 'success');
- const skipped = Object.entries(results).filter(([_, status]) => status === 'skipped');
-
- // Build status summary
- let summary = "## 🧪 E2E Test Results\n\n";
-
- if (failed.length > 0) {
- summary += "### ❌ Failed Tests\n";
- failed.forEach(([edition, _]) => {
- summary += `- **${edition.toUpperCase()} Edition**\n`;
- });
- summary += "\n";
- }
-
- if (passed.length > 0) {
- summary += "### ✅ Passed Tests\n";
- passed.forEach(([edition, _]) => {
- summary += `- **${edition.toUpperCase()} Edition**\n`;
- });
- summary += "\n";
- }
-
- if (skipped.length > 0) {
- summary += "### ⏭️ Skipped Tests\n";
- skipped.forEach(([edition, _]) => {
- summary += `- **${edition.toUpperCase()} Edition**\n`;
- });
- summary += "\n";
- }
-
- summary += `🤖 Automated E2E test results from [workflow run ${{ github.run_id }}](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})`;
-
- // Post comment
- await github.rest.issues.createComment({
- issue_number: context.issue.number,
- owner: context.repo.owner,
- repo: context.repo.repo,
- body: summary
- });
\ No newline at end of file
+ e2e:
+ if: >
+ (github.event.action == 'labeled' && github.event.label.name == 'ready-for-e2e') ||
+ (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'ready-for-e2e'))
+ concurrency:
+ group: e2e-${{ github.event.pull_request.number }}
+ cancel-in-progress: true
+ uses: ./.github/workflows/e2e-matrix.yml
+ secrets: inherit
diff --git a/.github/workflows/tests-e2e-ce.yml b/.github/workflows/tests-e2e-ce.yml
deleted file mode 100644
index e84f491234e..00000000000
--- a/.github/workflows/tests-e2e-ce.yml
+++ /dev/null
@@ -1,109 +0,0 @@
-name: E2E Tests - Community Edition
-
-on:
- workflow_call:
- workflow_dispatch:
-
-env:
- AP_DEV_PIECES: store,webhook
- AP_PIECES_SYNC_MODE: NONE
- AP_PIECES_SOURCE: FILE
- AP_API_KEY: api-key
- AP_DB_TYPE: POSTGRES
- AP_POSTGRES_HOST: localhost
- AP_POSTGRES_PORT: 5432
- AP_POSTGRES_USERNAME: postgres
- AP_POSTGRES_PASSWORD: postgres
- AP_POSTGRES_DATABASE: activepieces
- AP_POSTGRES_USE_SSL: "false"
- AP_REDIS_URL: redis://localhost:6379
- AP_QUEUE_MODE: REDIS
- AP_FRONTEND_URL: http://localhost:4200
- AP_WEBHOOK_TIMEOUT_SECONDS: "30"
- AP_TRIGGER_DEFAULT_POLL_INTERVAL: "1"
- AP_EXECUTION_MODE: UNSANDBOXED
- AP_TELEMETRY_ENABLED: "false"
- AP_ENVIRONMENT: test
- AP_JWT_SECRET: "secret"
- AP_ENCRYPTION_KEY: "fdbe51af3678563560972e4a912aee6d"
-
-jobs:
- test-e2e-ce:
- runs-on: depot-ubuntu-24.04-4
- timeout-minutes: 30
-
- services:
- postgres:
- image: postgres:17
- env:
- POSTGRES_USER: postgres
- POSTGRES_PASSWORD: postgres
- POSTGRES_DB: activepieces
- options: >-
- --health-cmd pg_isready
- --health-interval 10s
- --health-timeout 5s
- --health-retries 5
- ports:
- - 5432:5432
- redis:
- image: redis:latest
- ports:
- - 6379:6379
- options: >-
- --health-cmd "redis-cli ping"
- --health-interval 10s
- --health-timeout 5s
- --health-retries 5
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v5
- with:
- ref: ${{ github.event.pull_request.head.sha }}
- fetch-depth: 2
-
- - name: Cache dependencies
- uses: actions/cache@v5
- with:
- path: ~/.bun/install/cache
- key: bun-${{ hashFiles('bun.lock', 'package.json') }}
- restore-keys: bun-
-
- - name: Setup nodejs
- uses: actions/setup-node@v6
- with:
- node-version: 24
-
- - name: Setup Bun
- uses: oven-sh/setup-bun@v2
- with:
- bun-version: latest
-
- - name: Install dependencies
- run: bun install
-
- - name: Install Playwright browsers
- run: npx playwright install --with-deps chromium
-
- - name: Run E2E tests
- run: |
- npm run test:e2e
- env:
- CI: true
-
- - name: Upload Playwright Report
- uses: actions/upload-artifact@v6
- if: always()
- with:
- name: playwright-report-ce
- path: packages/tests-e2e/playwright-report/
- retention-days: 10
-
- - name: Upload Test Results
- uses: actions/upload-artifact@v6
- if: always()
- with:
- name: test-results-ce
- path: packages/tests-e2e/test-results/
- retention-days: 10
diff --git a/.github/workflows/tests-e2e-ee.yml b/.github/workflows/tests-e2e-ee.yml
deleted file mode 100644
index 0f666fcca66..00000000000
--- a/.github/workflows/tests-e2e-ee.yml
+++ /dev/null
@@ -1,114 +0,0 @@
-name: E2E Tests - Enterprise Edition
-
-on:
- workflow_call:
- workflow_dispatch:
-
-env:
- AP_EDITION: ee
- AP_DEV_PIECES: store,webhook
- AP_PIECES_SYNC_MODE: NONE
- AP_PIECES_SOURCE: FILE
- AP_API_KEY: api-key
- AP_LICENSE_KEY: ${{ secrets.EE_LICENSE_KEY }}
- AP_DB_TYPE: POSTGRES
- AP_POSTGRES_HOST: localhost
- AP_POSTGRES_PORT: 5432
- AP_POSTGRES_USERNAME: postgres
- AP_POSTGRES_PASSWORD: postgres
- AP_POSTGRES_DATABASE: activepieces
- AP_POSTGRES_USE_SSL: "false"
- AP_JWT_SECRET: "secret"
- AP_ENCRYPTION_KEY: "fdbe51af3678563560972e4a912aee6d"
- AP_FRONTEND_URL: http://localhost:4200
- AP_REDIS_URL: redis://localhost:6379
- AP_QUEUE_MODE: REDIS
- AP_EXECUTION_MODE: SANDBOX_CODE_ONLY
- AP_TELEMETRY_ENABLED: "false"
- NODE_ENV: test
-
-jobs:
- test-e2e-ee:
- runs-on: depot-ubuntu-24.04
- timeout-minutes: 30
-
- services:
- postgres:
- image: postgres:17
- env:
- POSTGRES_USER: postgres
- POSTGRES_PASSWORD: postgres
- POSTGRES_DB: activepieces
- options: >-
- --health-cmd pg_isready
- --health-interval 10s
- --health-timeout 5s
- --health-retries 5
- ports:
- - 5432:5432
- redis:
- image: redis:latest
- ports:
- - 6379:6379
- options: >-
- --health-cmd "redis-cli ping"
- --health-interval 10s
- --health-timeout 5s
- --health-retries 5
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v5
- with:
- ref: ${{ github.event.pull_request.head.sha }}
- fetch-depth: 2
-
- - name: Cache dependencies
- uses: actions/cache@v5
- with:
- path: ~/.bun/install/cache
- key: bun-${{ hashFiles('bun.lock', 'package.json') }}
- restore-keys: bun-
-
- - name: Setup nodejs
- uses: actions/setup-node@v6
- with:
- node-version: 24
-
- - name: Setup Bun
- uses: oven-sh/setup-bun@v2
- with:
- bun-version: latest
-
- - name: Install dependencies
- run: bun install
-
- - name: Install Playwright browsers
- run: npx playwright install --with-deps chromium
-
- - name: Run E2E tests
- run: |
- bun run test:e2e
- env:
- CI: true
-
- - name: Upload Playwright Report
- uses: actions/upload-artifact@v6
- if: always()
- with:
- name: playwright-report-ee
- path: packages/tests-e2e/playwright-report/
- retention-days: 30
-
- - name: Upload Test Results
- uses: actions/upload-artifact@v6
- if: always()
- with:
- name: test-results-ee
- path: packages/tests-e2e/test-results/
- retention-days: 30
-
- - name: Notify on failure
- if: failure()
- run: |
- echo "::error::Enterprise Edition E2E tests failed. This may indicate licensing or advanced feature issues."
diff --git a/brain/knowledge/engineering/e2e-tests-and-monitors.md b/brain/knowledge/engineering/e2e-tests-and-monitors.md
index a3ada483194..cb8e81a1188 100644
--- a/brain/knowledge/engineering/e2e-tests-and-monitors.md
+++ b/brain/knowledge/engineering/e2e-tests-and-monitors.md
@@ -16,7 +16,8 @@ One Playwright suite in `packages/tests-e2e` feeds three consumers that fail ind
- **BetterStack does not read the repo — the repo pushes to it.** `.github/workflows/sync-betterstack-playwright.yml` fires on push to `main` and `PATCH`es the file's contents into the hardcoded monitor `4211060` as `playwright_script`. One-way and `main`-only: the monitor updates at *merge*, never on the PR, and any edit made in the BetterStack UI is silently overwritten by the next push.
- **The BetterStack file is deliberately flat and duplicated.** BetterStack executes one self-contained script, so it cannot `require` the `pages/` objects — its sign-in is a copy. Fix the page object and you have *not* fixed the monitor; both files need the change.
- **The `.flat.spec.js` runs nowhere else.** Playwright matches `*.spec.ts` and Checkly matches `**/scenarios/**/*.spec.ts`, so a broken flat file is invisible locally and in CI until it fails in BetterStack.
-- **CI only runs on the `ready-for-e2e` label** (`e2e.yml` gates both edition workflows on it), which is why the suite can rot for weeks without anyone noticing.
+- **PRs do not run E2E unless they touch the suite or carry `ready-for-e2e`.** `e2e-matrix.yml` runs on its own only when `packages/tests-e2e/**` or its compose files change, so an unlabeled server change merges with no E2E check and the first signal is staging. A missing check is not a pass. The label makes `e2e.yml` call the same matrix.
+- **The label runs on `pull_request`, not `pull_request_target`.** A fork PR gets no `EE_LICENSE_KEY`, so its two EE configs fail; read the CE ones. Do not move the trigger back to `pull_request_target` to "fix" that: it hands PR code the secret.
- **Turbo strict env mode silently strips most of `.env.e2e`.** `globalPassThroughEnv` in `turbo.json` is an allow-list, so vars not named there never reach the `serve` tasks — verify with `tr '\0' '\n' < /proc/<api-pid>/environ`. `AP_ENVIRONMENT` is among the casualties, so CI falls back to the `prod` default. Widening it to `AP_*` does forward them, but that alone broke worker→API Socket.IO auth (jobs queue up unconsumed), so the passthrough and the worker's `AP_WORKER_TOKEN` have to be sorted out together.
- **`AP_DEV_PIECES` loads from `packages/pieces/**/dist`, which `npm run dev` does not build.** Only pieces that happen to be build dependencies of api/worker have a `dist`, so a default dev instance serves **0 pieces** and every spec that picks a trigger times out on the piece search. Build them explicitly: `npx turbo run build --filter=@activepieces/piece-webhook --filter=@activepieces/piece-store`.
- **CE sign-up is invitation-only once a platform exists** (`INVITATION_ONLY_SIGN_UP`), so the suite's sign-up path only works on a genuinely fresh instance. Against a dev-seeded database, set `E2E_EMAIL` / `E2E_PASSWORD` instead — `global-setup.ts` prefers them and signs in rather than signing up.
@@ -27,5 +28,6 @@ One Playwright suite in `packages/tests-e2e` feeds three consumers that fail ind
## Key files
- `packages/tests-e2e` — `playwright.config.ts` (local/CI), `checkly.config.ts` (Cloud monitors), `global-setup.ts` (provisions or signs in the seed account), `pages/` (shared page objects), `scenarios/betterstack/` (the standalone monitor script)
-- `.github/workflows/e2e.yml` — the `ready-for-e2e` gate that calls the per-edition workflows
+- `.github/workflows/e2e-matrix.yml` — the only E2E suite that gates a release: the built image in four CE/EE configurations
+- `.github/workflows/e2e.yml` — the `ready-for-e2e` label trigger for that matrix
- `.github/workflows/sync-betterstack-playwright.yml` — the push-to-`main` upload
diff --git a/packages/tests-e2e/README.md b/packages/tests-e2e/README.md
index b2e28ec43ce..9f94156638f 100644
--- a/packages/tests-e2e/README.md
+++ b/packages/tests-e2e/README.md
@@ -38,10 +38,11 @@ Required environment variables:
## CI
-Triggered by:
+PRs do not run E2E unless they change the suite or carry the `ready-for-e2e` label. Staging and the release candidate run it.
-- `.github/workflows/tests-e2e-ce.yml` — CE Playwright suite
-- `.github/workflows/tests-e2e-ee.yml` — EE Playwright suite
+- `.github/workflows/e2e-matrix.yml` — the only E2E suite that gates a release: builds the image and runs the suite in four CE/EE configurations. Runs on PRs that touch `packages/tests-e2e/**`, and from the Cloud gate.
+- `.github/workflows/e2e.yml` — runs `e2e-matrix.yml` on a PR with the `ready-for-e2e` label, and again on each push while the label stays
+- `.github/workflows/continuous-delivery-stg.yml` — runs the CE specs against staging after each deploy
- `.github/workflows/e2e-tests-checkly.yml` — scheduled Checkly run (optional)
## Related documentation