Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

11.0 KB
Raw Blame

icon: 🚀

Cloud Deployment Paths

How code reaches cloud.activepieces.com. continuous-delivery-cloud.yml is the Cloud gate (ENG-1021): cut → e2e-matrix + upgrade-test → report + tickets → approve → deploy (migration check, canary, fleet) → linear-release. It runs by hand only; the Sunday-to-Thursday 08:00 UTC schedule returns once E2E coverage is broader.

The normal path

cut takes the newest main commit whose e2e-staging check run passed (a manual run can name a sha, which must also have passed), and stops a scheduled run when the fleet already runs it. Every later job uses the version.sha.beta image that staging built from that commit; nothing is built again. The approve job holds the only approval (production environment). The deploy job then runs the breaking-migration check and deploys the canary and the fleet, a few minutes apart.

Hotfixes

A hotfix merges into main first. A manual gate run with hotfix_prs (ENG-1098) cherry-picks only those PRs onto the commit Cloud runs, pushes hotfix/cloud/<run id>-<attempt>, builds <version>.<sha>.pick in build-pick, and sends it through the same checks, approval and deploy. Picks stack until the next normal run, which must contain every picked PR. emergency-cloud-deploy.yml and continuous-delivery-rollback.yml stay outside the gate.

Staging (upstream of the gate)

continuous-delivery-stg.yml builds every push to main and deploys it with Kamal, not Kubernetes: it SSHes to the devops box and runs kamal deploy --config-file=config/{app,worker}.yml from /root/mrsk/stg. App containers live on one host, workers on another; kubectl's stg context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in config/app.yml under env.clear (secrets are name-listed under env.secret and read from .kamal/secrets). After both deploys, e2e-staging runs the Playwright suite against staging; its check run on the commit is what the gate's cut reads.

Gotchas

  • Kamal reads its config at deploy time, so editing config/app.yml during an in-flight CD run silently misses. The build takes ~5 min and the deploy job reads the file when it starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-run kamal deploy --version <same tag> --config-file=config/app.yml --skip-push after editing, and verify with docker inspect <container> --format '{{range .Config.Env}}...' on every container, not one. The command needs a TTY (ssh -tt); without it Kamal exits 0 having done nothing ("the input device is not a TTY").
  • The breaking-migration check runs in the deploy job, after the approval. It fails when any pending migration carries breaking = true (rollback safety — see tools/scripts/check-manifest-migrations.ts), so the fleet never deploys. To ship a breaking migration on purpose, run the gate by hand with skip_migration_check.
  • deploy re-reads Cloud under the lock and refuses to drop anything Cloud runs. It fails when the canary and the fleet run different images, when the fleet moved after cut, or when a commit in merge-base(cloud, candidate)..cloud has no (#N) subject or its PR number is missing from the candidate's history. That one rule blocks a downgrade and a silent revert of a fix shipped through another path (an emergency deploy, later hotfix mode); comparing commits alone misses the second case, because cherry-picked SHAs never match main. After emergency-cloud-deploy.yml (fleet only), deploy the fleet's tag to the canary by hand or the gate stays blocked.
  • A hotfix run has its own approval group (cloud-approval-hotfix-<run id>). The full gate run's group cancels a waiting older run, which would otherwise cancel a hotfix waiting for approval. The re-read under the lock in deploy catches a hotfix and a full gate run that were cut from the same Cloud commit.
  • The hotfix push uses the release GitHub App token, not GITHUB_TOKEN. A picked commit can change a workflow file, and GITHUB_TOKEN cannot push those. RELEASE_APP_ID and RELEASE_APP_PRIVATE_KEY must be repo secrets: cut has no environment, so the copies in the release environment are empty there.
  • Every job below build-pick needs an explicit status function in its if. A normal run skips build-pick, and an if with no status function gets an implicit success(), which is false when any earlier job in the chain was skipped. The first rehearsal (2026-10-29) passed every check, then skipped approve and deploy and still showed green. A job that must not run after a failed check (approve, deploy, linear-release) uses !cancelled() plus needs.<job>.result == 'success', not always(), so a cancelled run never deploys. A job that reports failures (report) keeps always().
  • Tag readers accept .beta and .pick. The gate and continuous-delivery-rollback.yml both parse the running tag; a reader that matches only .beta reads nothing after a hotfix deploy.
  • The approval sits before the canary on purpose. The canary runs migrations on the shared database at boot, so a canary deploy that waits days for the fleet's approval brings back the ENG-1062 problem.
  • A reusable workflow checks out github.sha, not the candidate. e2e-matrix.yml and upgrade-test.yml take a ref input, and the gate passes the candidate SHA; without it the specs come from the newest main.
  • The canary and the fleet deploy in one job on purpose. GitHub keeps only one pending job per concurrency group, even with cancel-in-progress: false, so with a lock per job a third run could replace the second run's pending fleet deploy after its canary had migrated the shared database. The deploy job's group is promote-production, the same as continuous-delivery-rollback.yml, so a rollback cannot interleave with a gate deploy either.
  • /flags cannot prove the upgrade test swapped images. After the release sync PR, main carries the same package.json version as the last release, so the upgrade test compares the running containers' image ID with the candidate's instead. Its swapped output tells report whether a red result is the candidate or the setup before the upgrade.
  • The canary has no daily schedule and no manual dispatch since ENG-1062. It shares the prod database and Redis, so a daily build of main ran newer migrations under the older fleet (2026-10-06: days of old-shape writes). The canary app runs migrations at boot, so it now gets only the image of a Cloud deploy (since ENG-1021 it has no workflow of its own), and continuous-delivery-rollback-canary.yml (which ran down() on the shared database) is deleted. The Cloud rollback workflow does not touch the canary hosts, so after a rollback they keep the bad image until the next Cloud deploy.
  • docker pull … error from registry: denied on the ops host can be transient. Seen 2026-08-12 pulling a .beta tag to the canary host; re-running the same job with no other change pulled and deployed fine. The tag existed in GHCR the whole time. Re-run once before suspecting the host's registry credential — GHCR does answer GET /v2/ for a token that cannot pull, so login-succeeds/pull-denied is consistent with an expired credential, but it is not evidence of one.
  • Never put a BuildKit cache mount on /var/cache/apt or /var/lib/apt. The node:*-bullseye-slim base ships /etc/apt/apt.conf.d/docker-clean, which wipes downloaded .debs and sets Keep-Downloaded-Packages "false" — so the mount caches nothing, but it does persist stale apt lists and partial/ leftovers across Depot builds. When bullseye-security republishes and old .debs rotate out, the next build dies on Hash Sum mismatch / Unable to fetch some archives → exit code: 100, which reads like a missing package but isn't. Removed from both Dockerfiles on 2026-08-12; a plain apt-get update && apt-get install is what works. Also: bullseye main has been frozen since Aug 2025 and Debian 11 LTS ends Aug 2026, so the base image needs a bookworm bump.
  • Building the image locally from a branch can fail before your code is even copied. Two independent causes seen 2026-09-10 on arm64: (1) snapshot.debian.org served corrupt .debs (Hash Sum mismatch, File has unexpected size … Mirror sync in progress?, Bad header line) across three retries, with no cache mount involved — workaround is a throwaway Dockerfile whose base stage is FROM ghcr.io/activepieces/activepieces:<root package.json version> AS base (same bun/npm/isolated-vm tooling, no apt), plus ENV NODE_ENV=development and RUN rm -rf /usr/src/app at the top of the build stage (the published image sets NODE_ENV=production, which makes bun install skip devDependencies). (2) bun install --frozen-lockfile fails whenever any workspace package.json version differs from its bun.lock entry; check with bun install --lockfile-only in a copy and diff the result before blaming your change.
  • breaking = true on a migration and the ⛓️‍💥 breaking-change PR label are different axes. The migration flag is about rollback safety and is what stops deploys; the label is about self-hoster upgrade impact and is enforced by breaking-change-check.yml on PRs. Neither implies the other.
  • The Dockerfile's prune step deletes packages/pieces/core (and every community piece but the api's 5), then re-runs bun install without a lockfile. Any other kept workspace that declares one of the removed pieces as workspace:* breaks the image build with Workspace dependency "@activepieces/piece-…" not found. #15842 hit this by adding core pieces to the engine's devDependencies for CI tests; the prune step now strips the engine's @activepieces/piece-* devDependencies. Check any new piece dependency outside packages/pieces against that step.
  • The run stage installs with bun install --production, so anything a kept workspace's compiled dist requires must be in its dependencies, not devDependencies. The run stage copies packages/ from the build stage, workspace node_modules links from the full install included, and bun 1.4.0 left those stale devDependency links in place, which hid that tslib was dev-only in pieces/framework, pieces/common, core/ai-providers and the api's 5 pieces, and that server/api listed those 5 pieces as devDependencies while importing them. bun 1.4.4 removes stale links (oven-sh/bun#45307), and the image died at boot with Cannot find module 'tslib' (seen 2026-10-27, also reported by a self-hosted customer building their own image). bun install in the repo stays green; only the built image shows it, and a new package that compiles with importHelpers reintroduces it (ai-providers did within days). After a bun bump, build the image and resolve every bare require in packages/**/dist from its own directory.