AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
icon: 🚀
Cloud Deployment Paths
How code reaches cloud.activepieces.com. continuous-delivery-cloud.yml is the Cloud gate (ENG-1021): cut → e2e-matrix + upgrade-test → report + tickets → approve → deploy (migration check, canary, fleet) → linear-release. It runs by hand only; the Sunday-to-Thursday 08:00 UTC schedule returns once E2E coverage is broader.
The normal path
cut takes the newest main commit whose e2e-staging check run passed (a manual run can name a sha, which must also have passed), and stops a scheduled run when the fleet already runs it. Every later job uses the version.sha.beta image that staging built from that commit; nothing is built again. The approve job holds the only approval (production environment). The deploy job then runs the breaking-migration check and deploys the canary and the fleet, a few minutes apart.
Hotfixes
A hotfix merges into main first. A manual gate run with hotfix_prs (ENG-1098) cherry-picks only those PRs onto the commit Cloud runs, pushes hotfix/cloud/<run id>-<attempt>, builds <version>.<sha>.pick in build-pick, and sends it through the same checks, approval and deploy. Picks stack until the next normal run, which must contain every picked PR. emergency-cloud-deploy.yml and continuous-delivery-rollback.yml stay outside the gate.
Staging (upstream of the gate)
continuous-delivery-stg.yml builds every push to main and deploys it with Kamal, not Kubernetes: it SSHes to the devops box and runs kamal deploy --config-file=config/{app,worker}.yml from /root/mrsk/stg. App containers live on one host, workers on another; kubectl's stg context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in config/app.yml under env.clear (secrets are name-listed under env.secret and read from .kamal/secrets). After both deploys, e2e-staging runs the Playwright suite against staging; its check run on the commit is what the gate's cut reads.
Gotchas
- Kamal reads its config at deploy time, so editing
config/app.ymlduring an in-flight CD run silently misses. The build takes ~5 min and the deploy job reads the file when it starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-runkamal deploy --version <same tag> --config-file=config/app.yml --skip-pushafter editing, and verify withdocker inspect <container> --format '{{range .Config.Env}}...'on every container, not one. The command needs a TTY (ssh -tt); without it Kamal exits 0 having done nothing ("the input device is not a TTY"). - The breaking-migration check runs in the
deployjob, after the approval. It fails when any pending migration carriesbreaking = true(rollback safety — seetools/scripts/check-manifest-migrations.ts), so the fleet never deploys. To ship a breaking migration on purpose, run the gate by hand withskip_migration_check. deployre-reads Cloud under the lock and refuses to drop anything Cloud runs. It fails when the canary and the fleet run different images, when the fleet moved aftercut, or when a commit inmerge-base(cloud, candidate)..cloudhas no(#N)subject or its PR number is missing from the candidate's history. That one rule blocks a downgrade and a silent revert of a fix shipped through another path (an emergency deploy, later hotfix mode); comparing commits alone misses the second case, because cherry-picked SHAs never matchmain. Afteremergency-cloud-deploy.yml(fleet only), deploy the fleet's tag to the canary by hand or the gate stays blocked.- A hotfix run has its own approval group (
cloud-approval-hotfix-<run id>). The full gate run's group cancels a waiting older run, which would otherwise cancel a hotfix waiting for approval. The re-read under the lock indeploycatches a hotfix and a full gate run that were cut from the same Cloud commit. - The hotfix push uses the release GitHub App token, not
GITHUB_TOKEN. A picked commit can change a workflow file, andGITHUB_TOKENcannot push those.RELEASE_APP_IDandRELEASE_APP_PRIVATE_KEYmust be repo secrets:cuthas no environment, so the copies in thereleaseenvironment are empty there. - Every job below
build-pickneeds an explicit status function in itsif. A normal run skipsbuild-pick, and anifwith no status function gets an implicitsuccess(), which is false when any earlier job in the chain was skipped. The first rehearsal (2026-10-29) passed every check, then skippedapproveanddeployand still showed green. A job that must not run after a failed check (approve,deploy,linear-release) uses!cancelled()plusneeds.<job>.result == 'success', notalways(), so a cancelled run never deploys. A job that reports failures (report) keepsalways(). - Tag readers accept
.betaand.pick. The gate andcontinuous-delivery-rollback.ymlboth parse the running tag; a reader that matches only.betareads nothing after a hotfix deploy. - The approval sits before the canary on purpose. The canary runs migrations on the shared database at boot, so a canary deploy that waits days for the fleet's approval brings back the ENG-1062 problem.
- A reusable workflow checks out
github.sha, not the candidate.e2e-matrix.ymlandupgrade-test.ymltake arefinput, and the gate passes the candidate SHA; without it the specs come from the newestmain. - The canary and the fleet deploy in one job on purpose. GitHub keeps only one pending job per concurrency group, even with
cancel-in-progress: false, so with a lock per job a third run could replace the second run's pending fleet deploy after its canary had migrated the shared database. Thedeployjob's group ispromote-production, the same ascontinuous-delivery-rollback.yml, so a rollback cannot interleave with a gate deploy either. /flagscannot prove the upgrade test swapped images. After the release sync PR,maincarries the samepackage.jsonversion as the last release, so the upgrade test compares the running containers' image ID with the candidate's instead. Itsswappedoutput tellsreportwhether a red result is the candidate or the setup before the upgrade.- The canary has no daily schedule and no manual dispatch since ENG-1062. It shares the prod database and Redis, so a daily build of
mainran newer migrations under the older fleet (2026-10-06: days of old-shape writes). The canary app runs migrations at boot, so it now gets only the image of a Cloud deploy (since ENG-1021 it has no workflow of its own), andcontinuous-delivery-rollback-canary.yml(which randown()on the shared database) is deleted. The Cloud rollback workflow does not touch the canary hosts, so after a rollback they keep the bad image until the next Cloud deploy. docker pull … error from registry: deniedon the ops host can be transient. Seen 2026-08-12 pulling a.betatag to the canary host; re-running the same job with no other change pulled and deployed fine. The tag existed in GHCR the whole time. Re-run once before suspecting the host's registry credential — GHCR does answerGET /v2/for a token that cannot pull, so login-succeeds/pull-denied is consistent with an expired credential, but it is not evidence of one.- Never put a BuildKit cache mount on
/var/cache/aptor/var/lib/apt. Thenode:*-bullseye-slimbase ships/etc/apt/apt.conf.d/docker-clean, which wipes downloaded.debs and setsKeep-Downloaded-Packages "false"— so the mount caches nothing, but it does persist staleaptlists andpartial/leftovers across Depot builds. Whenbullseye-securityrepublishes and old.debs rotate out, the next build dies onHash Sum mismatch/Unable to fetch some archives→exit code: 100, which reads like a missing package but isn't. Removed from both Dockerfiles on 2026-08-12; a plainapt-get update && apt-get installis what works. Also: bullseyemainhas been frozen since Aug 2025 and Debian 11 LTS ends Aug 2026, so the base image needs a bookworm bump. - Building the image locally from a branch can fail before your code is even copied. Two independent causes seen 2026-09-10 on arm64: (1)
snapshot.debian.orgserved corrupt.debs (Hash Sum mismatch,File has unexpected size … Mirror sync in progress?,Bad header line) across three retries, with no cache mount involved — workaround is a throwaway Dockerfile whose base stage isFROM ghcr.io/activepieces/activepieces:<root package.json version> AS base(same bun/npm/isolated-vm tooling, no apt), plusENV NODE_ENV=developmentandRUN rm -rf /usr/src/appat the top of the build stage (the published image setsNODE_ENV=production, which makesbun installskip devDependencies). (2)bun install --frozen-lockfilefails whenever any workspacepackage.jsonversion differs from itsbun.lockentry; check withbun install --lockfile-onlyin a copy anddiffthe result before blaming your change. breaking = trueon a migration and the⛓️💥 breaking-changePR label are different axes. The migration flag is about rollback safety and is what stops deploys; the label is about self-hoster upgrade impact and is enforced bybreaking-change-check.ymlon PRs. Neither implies the other.- The Dockerfile's prune step deletes
packages/pieces/core(and every community piece but the api's 5), then re-runsbun installwithout a lockfile. Any other kept workspace that declares one of the removed pieces asworkspace:*breaks the image build withWorkspace dependency "@activepieces/piece-…" not found. #15842 hit this by adding core pieces to the engine's devDependencies for CI tests; the prune step now strips the engine's@activepieces/piece-*devDependencies. Check any new piece dependency outsidepackages/piecesagainst that step. - The run stage installs with
bun install --production, so anything a kept workspace's compileddistrequires must be in itsdependencies, notdevDependencies. The run stage copiespackages/from the build stage, workspacenode_moduleslinks from the full install included, and bun 1.4.0 left those stale devDependency links in place, which hid thattslibwas dev-only inpieces/framework,pieces/common,core/ai-providersand the api's 5 pieces, and thatserver/apilisted those 5 pieces as devDependencies while importing them. bun 1.4.4 removes stale links (oven-sh/bun#45307), and the image died at boot withCannot find module 'tslib'(seen 2026-10-27, also reported by a self-hosted customer building their own image).bun installin the repo stays green; only the built image shows it, and a new package that compiles withimportHelpersreintroduces it (ai-providers did within days). After a bun bump, build the image and resolve every barerequireinpackages/**/distfrom its own directory.