AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
ci(cloud): turn off the daily canary deploy (#16992)
Othman Abu Ajamieh8bfe8b390f7a
diff --git a/.github/workflows/continuous-delivery-canary.yml b/.github/workflows/continuous-delivery-canary.ymlindex 6deea1fe854..be9fdfa08f5 100644--- a/.github/workflows/continuous-delivery-canary.yml+++ b/.github/workflows/continuous-delivery-canary.yml@@ -1,7 +1,6 @@name: Continuous Delivery — Canaryon:- workflow_dispatch:workflow_call:inputs:image_tag:@@ -14,8 +13,6 @@ on:type: booleanrequired: falsedefault: false- schedule:- - cron: '0 9 * * *' # Daily 9 AM UTC — scheduled runs always use the default branchjobs:build-image:diff --git a/.github/workflows/continuous-delivery-cloud.yml b/.github/workflows/continuous-delivery-cloud.ymlindex 30f71c1afa5..183c9d94645 100644--- a/.github/workflows/continuous-delivery-cloud.yml+++ b/.github/workflows/continuous-delivery-cloud.yml@@ -12,6 +12,10 @@ on:type: choiceoptions:- cloud-hotfix # Build from current branch and deploy directly to production, bypassing staging+ skip_migration_check:+ description: 'Deploy even when a pending migration is marked breaking'+ type: boolean+ default: falsejobs:guard:@@ -81,7 +85,7 @@ jobs:uses: ./.github/workflows/continuous-delivery-canary.ymlwith:image_tag: ${{ needs.build-image.outputs.image_tag }}- skip_migration_check: ${{ github.event_name == 'workflow_dispatch' }}+ skip_migration_check: ${{ inputs.skip_migration_check == true }}secrets: inheritpromote-to-production:diff --git a/.github/workflows/continuous-delivery-rollback-canary.yml b/.github/workflows/continuous-delivery-rollback-canary.ymldeleted file mode 100644index 613a291af8b..00000000000--- a/.github/workflows/continuous-delivery-rollback-canary.yml+++ /dev/null@@ -1,93 +0,0 @@-name: Continuous Delivery — Rollback Canary--on:- workflow_dispatch:- inputs:- rollback_to_image_tag:- description: 'Image tag to roll back TO (e.g. 0.51.0.abc1234.canary). Leave blank to roll back to the current cloud production image.'- required: false- default: ''- force:- description: 'Force rollback even if breaking migrations exist'- type: boolean- default: false--jobs:- rollback:- runs-on: ubuntu-latest- environment:- name: canary- steps:- - name: Configure SSH- run: |- mkdir -p ~/.ssh/- echo "$SSH_KEY" > ~/.ssh/ops.key- chmod 600 ~/.ssh/ops.key- cat >>~/.ssh/config <<END- Host ops- HostName $SSH_HOST- User $SSH_USER- IdentityFile ~/.ssh/ops.key- StrictHostKeyChecking no- END- env:- SSH_USER: ${{ secrets.DEV_OPS_USERNAME }}- SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}- SSH_HOST: ${{ secrets.DEV_OPS_HOST }}-- - name: Resolve rollback image tag- id: target- run: |- TAG="${{ inputs.rollback_to_image_tag }}"- if [ -z "$TAG" ]; then- TAG=$(ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal app version --config-file=config/app.yml 2>&1 | grep -oE \"[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]+\.beta\" | head -1 | tr -d \"[:space:]\"; exit"' | tr -d '\r' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]+\.beta' | head -1)- echo "No tag provided — resolved current cloud production tag: $TAG"- fi- if [ -z "$TAG" ]; then- echo "ERROR: Could not resolve a rollback image tag."- exit 1- fi- echo "image_tag=$TAG" >> $GITHUB_OUTPUT-- - name: Login to GitHub Container Registry- uses: docker/login-action@v4- with:- registry: ghcr.io- username: ${{ github.actor }}- password: ${{ secrets.GITHUB_TOKEN }}-- - name: Extract migration manifest from target image- id: manifest- run: |- IMAGE="ghcr.io/activepieces/activepieces-cloud:${{ steps.target.outputs.image_tag }}"- docker pull "$IMAGE"- MANIFEST=$(docker run --rm --entrypoint cat "$IMAGE" \- /usr/src/app/packages/server/api/dist/src/migration-manifest.json)- echo "json=$MANIFEST" >> $GITHUB_OUTPUT-- - name: Rollback DB migrations- run: |- FORCE_FLAG=${{ inputs.force && '"--force"' || '""' }}- MANIFEST='${{ steps.manifest.outputs.json }}'-- printf '%s' "$MANIFEST" | ssh ops 'cat > /tmp/rollback-manifest.json'-- ssh ops -t -t 'bash -ic "$(cat <<'"'"'EOF'"'"'- cd mrsk/prod- MANIFEST=$(cat /tmp/rollback-manifest.json | jq -c '"'"'[.[] | select(. != null)]'"'"')- MANIFEST_ESCAPED="${MANIFEST//\"/\\\"}"- kamal app exec --reuse --config-file=config/app-canary.yml -- node /usr/src/app/packages/server/api/dist/src/rollback.js --manifest "$MANIFEST_ESCAPED"- EOF- )"'-- - name: Redeploy canary app to target image- run: |- ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy \- --version ${{ steps.target.outputs.image_tag }} \- --config-file=config/app-canary.yml --skip-push; exit"'-- - name: Redeploy canary workers to target image- run: |- ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal deploy \- --version ${{ steps.target.outputs.image_tag }} \- --config-file=config/worker-canary.yml --skip-push; exit"'diff --git a/brain/knowledge/engineering/cloud-deployment-paths.md b/brain/knowledge/engineering/cloud-deployment-paths.mdindex 56a9360505f..eb4f7130c67 100644--- a/brain/knowledge/engineering/cloud-deployment-paths.md+++ b/brain/knowledge/engineering/cloud-deployment-paths.md@@ -10,15 +10,16 @@ How code reaches `cloud.activepieces.com`. Two workflows in `.github/workflows/`Cloud's `workflow_call`/scheduled run skips `build-image`, so `deploy-canary` gets an empty `image_tag` and the canary workflow builds its own `.canary` image and runs `check-migrations`. Prod then deploys the `release-candidate` tag, not that image.## The override path-`Continuous Delivery — Cloud` → **Run workflow** → `cloud-hotfix` builds one `.beta` image from the current branch and hands its tag to the canary workflow via `image_tag`, so canary deploys the exact artifact prod is about to get instead of rebuilding it. It also passes `skip_migration_check: true`. A `guard` job refuses the hotfix if the scheduled promotion is under an hour away, and a rejected guard skips canary too.+`Continuous Delivery — Cloud` → **Run workflow** → `cloud-hotfix` builds one `.beta` image from the current branch and hands its tag to the canary workflow via `image_tag`, so canary deploys the exact artifact prod is about to get instead of rebuilding it. It passes the run's `skip_migration_check` input (default `false`), so the breaking-migration check runs unless you tick it. A `guard` job refuses the hotfix if the scheduled promotion is under an hour away, and a rejected guard skips canary too.## Staging (upstream of both)`continuous-delivery-stg.yml` builds every push to `main` and deploys it with **Kamal**, not Kubernetes: it SSHes to the devops box and runs `kamal deploy --config-file=config/{app,worker}.yml` from `/root/mrsk/stg`. App containers live on one host, workers on another; `kubectl`'s `stg` context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in `config/app.yml` under `env.clear` (secrets are name-listed under `env.secret` and read from `.kamal/secrets`). The Thursday job retags whatever staging is running as `release-candidate`, which is what cloud promotes.## Gotchas- **Kamal reads its config at deploy time, so editing `config/app.yml` during an in-flight CD run silently misses.** The build takes ~5 min and the deploy job reads the file when *it* starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-run `kamal deploy --version <same tag> --config-file=config/app.yml --skip-push` after editing, and verify with `docker inspect <container> --format '{{range .Config.Env}}...'` on **every** container, not one. The command needs a TTY (`ssh -tt`); without it Kamal exits 0 having done nothing ("the input device is not a TTY").-- **`check-migrations` gates canary *and* the cloud promotion that calls it.** The canary workflow fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`), and the scheduled cloud run inherits that gate. `cloud-hotfix` passes `skip_migration_check: true` to bypass it: a deliberate cloud release ships the breaking migration to prod anyway, so blocking canary on it only leaves canary behind prod. There is still no canary-only override — dispatch the cloud workflow.+- **`check-migrations` gates canary *and* the cloud promotion that calls it.** The canary workflow fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`), and the scheduled cloud run inherits that gate. To ship a breaking migration on purpose, dispatch `cloud-hotfix` with `skip_migration_check`. Until ENG-1062 every dispatch skipped the check, so only the daily canary cron ran it.- **Skipping a `needs` job reads the same as a rejected one.** `build-image` is skipped both when the run is scheduled *and* when `guard` fails a hotfix, so `needs.build-image.result == 'skipped'` alone would deploy canary for a hotfix the guard just refused. `deploy-canary` therefore also needs `guard` and checks `needs.guard.result != 'failure'`.+- **The canary has no daily schedule and no manual dispatch since ENG-1062.** It shares the prod database and Redis, so a daily build of `main` ran newer migrations under the older fleet (2026-10-06: days of old-shape writes). The canary app runs migrations at boot, so it now gets only the image of a Cloud deploy, and `continuous-delivery-rollback-canary.yml` (which ran `down()` on the shared database) is deleted. The Cloud rollback workflow does not touch the canary hosts, so after a rollback they keep the bad image until the next Cloud deploy.- **`docker pull … error from registry: denied` on the ops host can be transient.** Seen 2026-08-12 pulling a `.beta` tag to the canary host; re-running the same job with no other change pulled and deployed fine. The tag existed in GHCR the whole time. Re-run once before suspecting the host's registry credential — GHCR does answer `GET /v2/` for a token that cannot pull, so login-succeeds/pull-denied is *consistent* with an expired credential, but it is not evidence of one.- **Never put a BuildKit cache mount on `/var/cache/apt` or `/var/lib/apt`.** The `node:*-bullseye-slim` base ships `/etc/apt/apt.conf.d/docker-clean`, which wipes downloaded `.deb`s and sets `Keep-Downloaded-Packages "false"` — so the mount caches nothing, but it does persist stale `apt` lists and `partial/` leftovers across Depot builds. When `bullseye-security` republishes and old `.deb`s rotate out, the next build dies on `Hash Sum mismatch` / `Unable to fetch some archives` → `exit code: 100`, which reads like a missing package but isn't. Removed from both Dockerfiles on 2026-08-12; a plain `apt-get update && apt-get install` is what works. Also: bullseye `main` has been frozen since Aug 2025 and Debian 11 LTS ends Aug 2026, so the base image needs a bookworm bump.- **Building the image locally from a branch can fail before your code is even copied.** Two independent causes seen 2026-09-10 on arm64: (1) `snapshot.debian.org` served corrupt `.deb`s (`Hash Sum mismatch`, `File has unexpected size … Mirror sync in progress?`, `Bad header line`) across three retries, with no cache mount involved — workaround is a throwaway Dockerfile whose base stage is `FROM ghcr.io/activepieces/activepieces:<root package.json version> AS base` (same bun/npm/isolated-vm tooling, no apt), plus `ENV NODE_ENV=development` and `RUN rm -rf /usr/src/app` at the top of the build stage (the published image sets `NODE_ENV=production`, which makes `bun install` skip devDependencies). (2) `bun install --frozen-lockfile` fails whenever any workspace `package.json` version differs from its `bun.lock` entry; check with `bun install --lockfile-only` in a copy and `diff` the result before blaming your change.diff --git a/docs/handbook/engineering/onboarding/release-cycle.mdx b/docs/handbook/engineering/onboarding/release-cycle.mdxindex aea2d94c83f..bc9717dd1c1 100644--- a/docs/handbook/engineering/onboarding/release-cycle.mdx+++ b/docs/handbook/engineering/onboarding/release-cycle.mdx@@ -12,7 +12,7 @@ Merge to main ────────────→ Deploy to staging│5 PM UTC Thu: RC tagged│- Fri–Sat: canary runs daily from main+ Fri–Sat: cloud is quiet│Sun 9 AM UTC: canary rebuilt from main│ → release-candidate promoted to production@@ -48,7 +48,7 @@ On Monday at 9 AM UTC, the self-hosted release is published: git tag, GitHub rel|---|---|| Any time | Merge to `main` → auto-deployed to staging || **Thu 5 PM UTC** | Staging image + commit tagged as `release-candidate` |-| Fri–Sat | Canary deploys daily from `main`; cloud is quiet |+| Fri–Sat | Cloud is quiet || **Sun 9 AM UTC** | Canary rebuilt from latest `main`; `release-candidate` promoted to cloud production; `deploy/cloud/YYYY-MM-DD` branch created || **Mon 9 AM UTC** | Self-hosted release published (re-tags cloud RC image) |diff --git a/docs/handbook/engineering/playbooks/canary-deployment.mdx b/docs/handbook/engineering/playbooks/canary-deployment.mdxindex 823755e11b8..58e42bc59c7 100644--- a/docs/handbook/engineering/playbooks/canary-deployment.mdx+++ b/docs/handbook/engineering/playbooks/canary-deployment.mdx@@ -85,27 +85,18 @@ Jobs for canary platforms are enqueued to `canaryWorkerJobs`, which only canary## Deploying a New Canary Build-The **Continuous Delivery — Canary** workflow (`continuous-delivery-canary.yml`) runs automatically every day at 9 AM UTC, building from the latest `main`. It is also invoked as a reusable workflow from `continuous-delivery-cloud.yml` on the Sunday promotion, so the canary is always refreshed from the latest `main` immediately before production is updated.+The **Continuous Delivery — Canary** workflow (`continuous-delivery-canary.yml`) runs as a reusable workflow from `continuous-delivery-cloud.yml`, with the image that Cloud is about to deploy, a few minutes before the rest of production. It has no daily schedule and no manual dispatch: the canary shares the production database and runs migrations at boot, so a daily build of `main` ran newer migrations under the older fleet.The workflow:1. Builds and pushes a new image tagged `<version>.<sha>.canary`-2. Checks if any new migrations are breaking — fails the workflow if they are (no override)+2. Checks if any new migrations are breaking — fails the workflow if they are, unless the Cloud run sets `skip_migration_check`3. Deploys the canary app (`config/app-canary.yml`) and canary workersRequired GitHub secrets: `AP_API_KEY` (the primary app's `AP_API_KEY` value).## Rolling Back a Canary Deployment-Trigger the **Continuous Delivery — Rollback Canary** workflow (`continuous-delivery-rollback-canary.yml`) with the image tag you want to roll back to. The workflow:--1. Extracts the migration manifest from the target image-2. Rolls back DB migrations not present in that manifest-3. Redeploys the canary app and workers to the target image--| Input | Description |-|---|---|-| `rollback_to_image_tag` | Image tag to roll back to (e.g. `0.51.0.abc1234.canary`) |-| `force` | Force rollback even if breaking migrations exist. Default: `false`. |+The next Cloud deploy sets the canary image. The Cloud rollback workflow does not touch the canary hosts, so after a Cloud rollback the canary keeps the newer image until the next Cloud deploy. There is no separate canary rollback, because running migrations down on the shared database can drop data the newer code wrote.## Promoting canary build to production@@ -116,7 +107,7 @@ Canary is a validation environment, not a promotion path. Full promotion happens### Enable canary routing for a platform```bash-curl -X POST https://cloud.activepieces.com/v1/admin/platforms/canary \+curl -X POST https://cloud.activepieces.com/api/v1/admin/platforms/canary \-H "api-key: <AP_API_KEY>" \-H "Content-Type: application/json" \-d '{"platformId": "<platformId>", "canary": true}'@@ -125,7 +116,7 @@ curl -X POST https://cloud.activepieces.com/v1/admin/platforms/canary \### Disable canary routing for a platform```bash-curl -X POST https://cloud.activepieces.com/v1/admin/platforms/canary \+curl -X POST https://cloud.activepieces.com/api/v1/admin/platforms/canary \-H "api-key: <AP_API_KEY>" \-H "Content-Type: application/json" \-d '{"platformId": "<platformId>", "canary": false}'diff --git a/docs/handbook/engineering/playbooks/releases.mdx b/docs/handbook/engineering/playbooks/releases.mdxindex 2a36f97e80d..b26239411bc 100644--- a/docs/handbook/engineering/playbooks/releases.mdx+++ b/docs/handbook/engineering/playbooks/releases.mdx@@ -10,10 +10,9 @@ All changes flow through a staging environment before reaching production. ThereFive separate workflows cover the full delivery lifecycle:1. **Merge to `main`** — `continuous-delivery-stg.yml` automatically builds a Docker image and deploys to **staging** (`stg.activepieces.com`).-2. **Daily 9 AM UTC** — `continuous-delivery-canary.yml` builds a `version.sha.canary` image from the latest `main` and deploys it to the **canary** environment. Breaking migrations block the deployment. See the [Canary Deployment playbook](/handbook/engineering/playbooks/canary-deployment) for details.-3. **Thursday 5 PM UTC** — `tag-release-candidate` job tags the current staging image and commit as `release-candidate`.-4. **Sunday 9 AM UTC** — `continuous-delivery-cloud.yml` runs a fresh canary build from `main` (by calling `continuous-delivery-canary.yml` as a reusable workflow), then promotes `release-candidate` to production and creates a `deploy/cloud/YYYY-MM-DD` branch.-5. **Monday 9 AM UTC** — `continuous-delivery-release.yml` re-tags the `release-candidate` image and publishes the self-hosted release.+2. **Thursday 5 PM UTC** — `tag-release-candidate` job tags the current staging image and commit as `release-candidate`.+3. **Sunday 9 AM UTC** — `continuous-delivery-cloud.yml` runs a fresh canary build from `main` (by calling `continuous-delivery-canary.yml` as a reusable workflow), then promotes `release-candidate` to production and creates a `deploy/cloud/YYYY-MM-DD` branch.+4. **Monday 9 AM UTC** — `continuous-delivery-release.yml` re-tags the `release-candidate` image and publishes the self-hosted release.## Environments@@ -21,7 +20,7 @@ Five separate workflows cover the full delivery lifecycle:|---|---|---|| Preview | `branch-name.activepieces.com` | Per-PR ephemeral environment for review. *Requires the `preview` label on the PR.* || Staging | `stg.activepieces.com` | Internal testing |-| Canary | `canary.activepieces.com` | Daily cut from `main`; catches issues before they reach production |+| Canary | `canary.activepieces.com` | Gets each Cloud deploy a few minutes before the rest of production || Production | `cloud.activepieces.com` | Live customers |## Hotfix Workflow@@ -71,20 +70,11 @@ To run any workflow manually, go to the repo's **Actions** tab, select the workf| Trigger | What happens ||---|---|-| **Scheduled (daily 9 AM UTC)** | Builds a `version.sha.canary` image from the latest `main`, checks migrations, deploys the canary app and workers. Breaking migrations always block the deployment. |-| **`workflow_call`** | Same as scheduled, triggered by `continuous-delivery-cloud.yml` on the Sunday promotion so the canary always receives a fresh build just before production is updated. |-| **Manual dispatch** | Same as scheduled, on demand. |+| **`workflow_call`** | Called by `continuous-delivery-cloud.yml` before production. Deploys the image Cloud is about to ship to the canary app and workers. Breaking migrations block the deployment unless the Cloud run sets `skip_migration_check`. |-See the [Canary Deployment playbook](/handbook/engineering/playbooks/canary-deployment) for full details.--### `continuous-delivery-rollback-canary.yml` — Canary Rollback--Manual dispatch only. Rolls back the canary environment to a previous image tag. Reverses DB migrations not present in the target image's manifest, then redeploys the canary app and workers to that tag.+There is no daily schedule and no manual dispatch since ENG-1062: the canary app runs its migrations on the shared production database at boot, so only an image that Cloud is about to deploy may reach it.-| Input | Description |-|---|---|-| `rollback_to_image_tag` | Target image tag to roll back to (e.g. `0.51.0.abc1234.canary`). Leave blank to roll back to the current cloud production image. |-| `force` | Force rollback even if breaking migrations exist. Default: `false`. |+See the [Canary Deployment playbook](/handbook/engineering/playbooks/canary-deployment) for full details.### `release-self-hosted.yml`