AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
icon: 🎭
E2E Tests & Monitors
One Playwright suite in packages/tests-e2e feeds three consumers that fail independently: CI on a fresh throwaway instance, Checkly monitors against production Cloud, and a single BetterStack monitor. A change that only breaks one of them looks green everywhere else, so it is worth knowing which reads what.
How to write a spec lives in one place: "Writing a spec" in packages/tests-e2e/README.md. This page holds the gotchas only.
Local / CI suite — playwright.config.ts runs every scenarios/**/*.spec.ts; AP_EDITION=ce skips scenarios/ee. Boots the whole stack itself via the webServer block unless E2E_INSTANCE_URL is set. Global setup signs in once and saves the session to .auth/user.json, so specs start signed in.
Checkly monitors — checkly.config.ts picks up the same **/scenarios/**/*.spec.ts files and runs them every 10 minutes with baseURL: https://cloud.activepieces.com, signing in with E2E_EMAIL / E2E_PASSWORD. It runs the specs from its last deploy, not the specs on main.
BetterStack monitor — one standalone file, scenarios/betterstack/*.flat.spec.js.
Gotchas
- Checkly runs the shared specs against production Cloud, so anything the page objects assume about the login screen has to hold on Cloud too — not just on the SMTP-less instance CI boots. This is the usual reason a UI change breaks the monitors but not CI. See the auth-card gotchas on CE Authentication.
- Checkly deploys only by hand. The
Deploy-Checklyjob inemergency-cloud-deploy.ymlis the only path; nothing has callede2e-tests-checkly.ymlsince September 2025, and thetest-checkly/deploy-checklyscripts are gone. A spec merged tomainis not monitored until someone runs that workflow. - BetterStack does not read the repo — the repo pushes to it.
.github/workflows/sync-betterstack-playwright.ymlfires on push tomainandPATCHes the file's contents into the hardcoded monitor4211060asplaywright_script. One-way andmain-only: the monitor updates at merge, never on the PR, and any edit made in the BetterStack UI is silently overwritten by the next push. - The BetterStack file is deliberately flat and duplicated. BetterStack executes one self-contained script, so it cannot
requirethepages/objects — its sign-in is a copy. Fix the page object and you have not fixed the monitor; both files need the change. - The
.flat.spec.jsruns nowhere else. Playwright matches*.spec.tsand Checkly matches**/scenarios/**/*.spec.ts, so a broken flat file is invisible locally and in CI until it fails in BetterStack. - PRs do not run E2E unless they touch the suite or carry
ready-for-e2e.e2e-matrix.ymlruns on its own only whenpackages/tests-e2e/**or its compose files change, so an unlabeled server change merges with no E2E check and the first signal is staging. A missing check is not a pass. The label makese2e.ymlcall the same matrix. - The label runs on
pull_request, notpull_request_target. A fork PR gets noEE_LICENSE_KEY, so its two EE configs fail; read the CE ones. Do not move the trigger back topull_request_targetto "fix" that: it hands PR code the secret. - Turbo strict env mode silently strips most of
.env.e2e.globalPassThroughEnvinturbo.jsonis an allow-list, so vars not named there never reach theservetasks — verify withtr '\0' '\n' < /proc/<api-pid>/environ.AP_ENVIRONMENTis among the casualties, so CI falls back to theproddefault. Widening it toAP_*does forward them, but that alone broke worker→API Socket.IO auth (jobs queue up unconsumed), so the passthrough and the worker'sAP_WORKER_TOKENhave to be sorted out together. AP_DEV_PIECESloads frompackages/pieces/**/dist, whichnpm run devdoes not build. Only pieces that happen to be build dependencies of api/worker have adist, so a default dev instance serves 0 pieces and every spec that picks a trigger times out on the piece search. Build them explicitly:npx turbo run build --filter=@activepieces/piece-webhook --filter=@activepieces/piece-store.- CE sign-up is invitation-only once a platform exists (
INVITATION_ONLY_SIGN_UP), so the suite's sign-up path only works on a genuinely fresh instance. Against a dev-seeded database, setE2E_EMAIL/E2E_PASSWORDinstead —global-setup.tsprefers them and signs in rather than signing up. - Every workspace declares its own deps.
@faker-js/fakerwas imported by the page objects for months while onlyserver/apideclared it; under Bun's isolated linker that means the suite cannot import its own page objects at all. npx playwright testcan resolve a globally installed Playwright instead of the repo's, and the two copies fail withPlaywright Test did not expect test.describe() to be called hereon every spec, including ones that have not been touched. The message points at your file and the cause is elsewhere. Run./node_modules/.bin/playwright test --config=packages/tests-e2e/playwright.config.tsinstead.- Specs own their data. Each flow and connection a spec makes is named
E2E …and deleted by theflowCleanup/connectionCleanupfixtures, also on failure; global setup only sweepsE2Eflows and connections older than an hour. The old setup deleted every flow, which put staging in the empty state on every run and once wiped a shared account. - Automations renders "Create New" while it loads, then may swap to the empty state. Clicking it during the load opens a menu that disappears.
AutomationsPage.createFlowwaits forautomations-loadingto detach before it picks an entry. - "Test Flow" does not flush pending edits. Step edits save through a 1 s debounce, and a test run started inside that window runs the old draft. The page object waits for the
POST /api/v1/flows/<id>save after each edit; Publish flushes on its own. - Playwright sees every button in the builder breadcrumb as disabled.
BreadcrumbPagerendersrole="link" aria-disabled="true", which Playwright inherits, so the flow menu button never clicks. Click the flow name text to rename instead. - Checkly runs no global setup. It matches the same
*.spec.tsfiles, so a spec that relies on the saved session fails there. - A spec that needs a live third-party connection cannot live in
scenarios/. Checkly and CI both pick up every*.spec.tsthere, and neither has the connection, so it fails for everyone but its author. Keep piece-connection specs outside the repo and pin the regression in the piece's own unit tests instead. To cover the connection path itself,connection-step.spec.tspoints the Activepieces Platform piece's Base URL at a webhook flow of its own, then reads theAuthorizationheader in that flow's run. - A
FormLabelnames its input only throughFormControl.FormControlputs the label's id on its child, so an input outside it, or a wrapper that dropsid, leavesgetByLabelwith nothing. Fix the markup rather than locate the field by nearby text. - A step must not call a
/syncwebhook on the same instance. The step test holds a worker slot while the called flow waits for one, so with one slot the call times out (408). The matrix split configs run one slot:benchmark/docker-compose.ymlpassesAP_WORKER_CONCURRENCYas an empty string, which the worker reads as invalid and replaces with 1, not the default 5. Call the async webhook and check the called flow's run instead. - Custom API Call leaves its URL empty in the builder, although the action's default is the connection's base URL. "Test Step" stays disabled until the spec fills the URL.
@quarantinespecs are skipped only in CI. The config'sgrepInvertapplies whenCIis set, and a CLI--grep-invertreplaces it, soe2e-matrix.ymlrepeats@quarantinein its own pattern.
Key files
packages/tests-e2e—playwright.config.ts(local/CI),checkly.config.ts(Cloud monitors),global-setup.ts(signs in or signs up, saves the session, sweeps staleE2Eflows and connections),fixtures/(flowCleanup,connectionCleanup,e2eFlows),pages/(shared page objects),scenarios/betterstack/(the standalone monitor script).github/workflows/e2e-matrix.yml— the only E2E suite that gates a release: the built image in four CE/EE configurations.github/workflows/e2e.yml— theready-for-e2elabel trigger for that matrix.github/workflows/sync-betterstack-playwright.yml— the push-to-mainupload