AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
ci(cloud): deploy only chosen hotfix PRs on top of what Cloud runs (#16997)
Othman Abu Ajamieh48dee01cca5b
diff --git a/.github/workflows/continuous-delivery-cloud.yml b/.github/workflows/continuous-delivery-cloud.ymlindex 68f3bf7606d..728fb959818 100644--- a/.github/workflows/continuous-delivery-cloud.yml+++ b/.github/workflows/continuous-delivery-cloud.yml@@ -2,6 +2,8 @@ name: Continuous Delivery — Cloud# The Cloud gate. Picks the newest main commit that passed E2E on staging, tests the image staging built# from it, waits for one approval, then deploys that image to the canary and to the fleet.+# With hotfix_prs, it instead cherry-picks only those merged PRs onto the commit Cloud runs, builds that,+# and sends it through the same checks, approval and deploy.on:workflow_dispatch:inputs:@@ -9,6 +11,14 @@ on:description: 'Commit on main to deploy. Empty takes the newest one that passed E2E on staging.'type: stringdefault: ''+ hotfix_prs:+ description: 'Comma-separated PR numbers already merged into main. Deploys only these on top of what Cloud runs.'+ type: string+ default: ''+ allow_migrations:+ description: 'Let a hotfix PR that changes a migration through'+ type: boolean+ default: falseskip_migration_check:description: 'Deploy even when a pending migration is marked breaking'type: boolean@@ -23,21 +33,64 @@ jobs:cut:runs-on: ubuntu-latestoutputs:- sha: ${{ steps.pick.outputs.sha }}- image_tag: ${{ steps.pick.outputs.image_tag }}- from_tag: ${{ steps.pick.outputs.from_tag }}+ sha: ${{ steps.pick.outputs.sha || steps.hotfix.outputs.sha }}+ image_tag: ${{ steps.pick.outputs.image_tag || steps.hotfix.outputs.image_tag }}+ from_tag: ${{ steps.from.outputs.tag }}cloud_sha: ${{ steps.cloud.outputs.sha }}- skip: ${{ steps.cloud.outputs.skip }}+ skip: ${{ steps.pick.outputs.skip }}steps:+ # The App token can push a pick branch whose commits change workflow files; GITHUB_TOKEN cannot.+ - name: Create GitHub App token+ id: app-token+ if: inputs.hotfix_prs+ uses: actions/create-github-app-token@v2+ with:+ app-id: ${{ secrets.RELEASE_APP_ID }}+ private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}+- uses: actions/checkout@v5with:fetch-depth: 0+ token: ${{ steps.app-token.outputs.token || github.token }}++ - name: Configure SSH+ run: |+ mkdir -p ~/.ssh/+ echo "$SSH_KEY" > ~/.ssh/ops.key+ chmod 600 ~/.ssh/ops.key+ cat >>~/.ssh/config <<END+ Host ops+ HostName $SSH_HOST+ User $SSH_USER+ IdentityFile ~/.ssh/ops.key+ StrictHostKeyChecking no+ END+ env:+ SSH_USER: ${{ secrets.DEV_OPS_USERNAME }}+ SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}+ SSH_HOST: ${{ secrets.DEV_OPS_HOST }}++ - name: Read the commit Cloud runs+ id: cloud+ run: |+ CLOUD_SHA=$(ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal app version --config-file=config/app.yml 2>&1; exit"' \+ | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)$' || true; } | head -1 | cut -d. -f4)+ if [ -z "$CLOUD_SHA" ]; then+ echo "::error::Could not read the version the Cloud fleet runs"+ exit 1+ fi+ echo "sha=$CLOUD_SHA" >> "$GITHUB_OUTPUT"+ echo "Cloud runs \`$CLOUD_SHA\`" >> "$GITHUB_STEP_SUMMARY"+ # A scheduled run stops when every app and worker host, canary included, already runs the candidate,+ # so a quiet day asks for no approval and a deploy that stopped half way is retried.- name: Pick the candidateid: pick+ if: ${{ !inputs.hotfix_prs }}env:GH_TOKEN: ${{ github.token }}INPUT_SHA: ${{ inputs.sha }}+ CLOUD_SHA: ${{ steps.cloud.outputs.sha }}run: |e2e_passed() {[ -n "$(gh api "repos/$GITHUB_REPOSITORY/commits/$1/check-runs?check_name=e2e-staging" \@@ -67,50 +120,14 @@ jobs:fifiIMAGE_TAG="$(git show "$SHA:package.json" | jq -r .version).$SHA.beta"- FROM_TAG=$(gh release list --repo "$GITHUB_REPOSITORY" --exclude-drafts --exclude-pre-releases -L 50 --json tagName --jq '.[].tagName' \- | grep -E '^[0-9]+\.[0-9]+\.[0-9]+(-hotfix\.[0-9]+)?$' | sort -V | tail -1)echo "sha=$SHA" >> "$GITHUB_OUTPUT"echo "image_tag=$IMAGE_TAG" >> "$GITHUB_OUTPUT"- echo "from_tag=$FROM_TAG" >> "$GITHUB_OUTPUT"- echo "Candidate \`$SHA\`, image \`$IMAGE_TAG\`, upgrade test from \`$FROM_TAG\`" >> "$GITHUB_STEP_SUMMARY"-- - name: Configure SSH- run: |- mkdir -p ~/.ssh/- echo "$SSH_KEY" > ~/.ssh/ops.key- chmod 600 ~/.ssh/ops.key- cat >>~/.ssh/config <<END- Host ops- HostName $SSH_HOST- User $SSH_USER- IdentityFile ~/.ssh/ops.key- StrictHostKeyChecking no- END- env:- SSH_USER: ${{ secrets.DEV_OPS_USERNAME }}- SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}- SSH_HOST: ${{ secrets.DEV_OPS_HOST }}-- # A scheduled run stops when every app and worker host, canary included, already runs the candidate,- # so a quiet day asks for no approval and a deploy that stopped half way is retried.- - name: Read the commit Cloud runs- id: cloud- env:- SHA: ${{ steps.pick.outputs.sha }}- run: |- CLOUD_SHA=$(ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal app version --config-file=config/app.yml 2>&1; exit"' \- | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta$' || true; } | head -1 | cut -d. -f4)- if [ -z "$CLOUD_SHA" ]; then- echo "::error::Could not read the version the Cloud fleet runs"- exit 1- fi- echo "sha=$CLOUD_SHA" >> "$GITHUB_OUTPUT"- echo "Cloud runs \`$CLOUD_SHA\`" >> "$GITHUB_STEP_SUMMARY"+ echo "Candidate \`$SHA\`, image \`$IMAGE_TAG\`" >> "$GITHUB_STEP_SUMMARY"if [ "$GITHUB_EVENT_NAME" = schedule ] && [ "$CLOUD_SHA" = "$SHA" ]; thenSKIP=truefor CONFIG in app worker app-canary worker-canary; doCONFIG_SHA=$(ssh ops -t -t "bash -ic \"cd mrsk/prod && kamal app version --config-file=config/$CONFIG.yml 2>&1; exit\"" \- | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta$' || true; } | sort -u | cut -d. -f4)+ | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)$' || true; } | sort -u | cut -d. -f4)if [ "$CONFIG_SHA" != "$SHA" ]; thenSKIP=falseecho "\`config/$CONFIG.yml\` does not run the candidate on every host" >> "$GITHUB_STEP_SUMMARY"@@ -122,9 +139,110 @@ jobs:fifi- e2e-matrix:+ # Each named PR's squash commit is cherry-picked from main, oldest first, onto the commit Cloud runs.+ # The branch is pushed so the commit Cloud will run stays in the repository.+ - name: Cherry-pick the hotfix PRs onto what Cloud runs+ id: hotfix+ if: inputs.hotfix_prs+ env:+ GH_TOKEN: ${{ github.token }}+ HOTFIX_PRS: ${{ inputs.hotfix_prs }}+ INPUT_SHA: ${{ inputs.sha }}+ ALLOW_MIGRATIONS: ${{ inputs.allow_migrations }}+ CLOUD_SHA: ${{ steps.cloud.outputs.sha }}+ run: |+ if [ -n "$INPUT_SHA" ]; then+ echo "::error::Set either sha or hotfix_prs, not both"+ exit 1+ fi+ COMMITS=""+ for PR in $(echo "$HOTFIX_PRS" | tr ',#' ' '); do+ read -r STATE BASE MERGE <<< "$(gh pr view "$PR" --repo "$GITHUB_REPOSITORY" --json state,baseRefName,mergeCommit \+ --jq '"\(.state) \(.baseRefName) \(.mergeCommit.oid // "")"')"+ if [ "$STATE" != MERGED ] || [ "$BASE" != main ] || [ -z "$MERGE" ] || ! git merge-base --is-ancestor "$MERGE" origin/main; then+ echo "::error::#$PR is not merged into main"+ exit 1+ fi+ if git log --format=%s "$CLOUD_SHA" | grep -qE "\(#$PR\)$"; then+ echo "::error::#$PR is already on Cloud"+ exit 1+ fi+ MIGRATIONS=$(git diff --name-only "$MERGE^" "$MERGE" -- packages/server/api/src/app/database/migration/ packages/server/api/src/app/ee/database/migrations/)+ if [ -n "$MIGRATIONS" ] && [ "$ALLOW_MIGRATIONS" != true ]; then+ echo "::error::#$PR changes a migration ($MIGRATIONS). It may need schema changes Cloud does not have yet. Set allow_migrations to pick it anyway."+ exit 1+ fi+ COMMITS="$COMMITS$MERGE"$'\n'+ done+ BRANCH="hotfix/cloud/$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"+ git config user.name "github-actions[bot]"+ git config user.email "github-actions[bot]@users.noreply.github.com"+ git checkout -q -b "$BRANCH" "$CLOUD_SHA"+ for COMMIT in $(git rev-list --first-parent --reverse origin/main | grep -Fx -f <(printf '%s' "$COMMITS")); do+ if ! git cherry-pick -x "$COMMIT"; then+ git cherry-pick --abort+ echo "::error::$(git log -1 --format=%s "$COMMIT") does not apply cleanly on what Cloud runs"+ exit 1+ fi+ done+ git push origin "$BRANCH"+ SHA=$(git rev-parse HEAD)+ IMAGE_TAG="$(git show "$SHA:package.json" | jq -r .version).$SHA.pick"+ EARLIER=$(git log --first-parent --format=%s "$(git merge-base "$CLOUD_SHA" origin/main)..$CLOUD_SHA" | grep -oE '#[0-9]+\)$' | tr -d ')' | paste -sd ' ' || true)+ ADDED=$(git log --first-parent --reverse --format=%s "$CLOUD_SHA..$SHA" | grep -oE '#[0-9]+\)$' | tr -d ')' | paste -sd ' ')+ echo "sha=$SHA" >> "$GITHUB_OUTPUT"+ echo "image_tag=$IMAGE_TAG" >> "$GITHUB_OUTPUT"+ {+ echo "**Hotfix:** Cloud runs \`${CLOUD_SHA:0:10}\`${EARLIER:+ (main plus $EARLIER)}. This run adds $ADDED."+ echo+ echo "Branch \`$BRANCH\`, image \`$IMAGE_TAG\`"+ } >> "$GITHUB_STEP_SUMMARY"++ - name: Pick the release the upgrade test starts from+ id: from+ if: steps.pick.outputs.skip != 'true'+ env:+ GH_TOKEN: ${{ github.token }}+ run: |+ FROM_TAG=$(gh release list --repo "$GITHUB_REPOSITORY" --exclude-drafts --exclude-pre-releases -L 50 --json tagName --jq '.[].tagName' \+ | grep -E '^[0-9]+\.[0-9]+\.[0-9]+(-hotfix\.[0-9]+)?$' | sort -V | tail -1)+ echo "tag=$FROM_TAG" >> "$GITHUB_OUTPUT"+ echo "Upgrade test from \`$FROM_TAG\`" >> "$GITHUB_STEP_SUMMARY"++ # Staging never built a hotfix combination, so the gate builds it. The .pick tag never replaces a .beta image.+ build-pick:needs: cut- if: needs.cut.outputs.skip != 'true'+ if: inputs.hotfix_prs+ runs-on: ubuntu-24.04+ permissions:+ contents: read+ packages: write+ steps:+ - uses: actions/checkout@v5+ with:+ ref: ${{ needs.cut.outputs.sha }}+ - uses: docker/login-action@v4+ with:+ registry: ghcr.io+ username: ${{ github.actor }}+ password: ${{ secrets.GITHUB_TOKEN }}+ - uses: depot/setup-action@v1+ - uses: depot/build-push-action@v1+ with:+ project: du7O4b0e8P+ token: ${{ secrets.DEPOT_PROJECT_TOKEN }}+ context: .+ file: ./Dockerfile+ platforms: |+ linux/amd64+ linux/arm64+ push: true+ no-cache: true+ tags: ghcr.io/activepieces/activepieces-cloud:${{ needs.cut.outputs.image_tag }}++ e2e-matrix:+ needs: [cut, build-pick]+ if: always() && needs.cut.result == 'success' && needs.cut.outputs.skip != 'true' && (needs.build-pick.result == 'success' || needs.build-pick.result == 'skipped')uses: ./.github/workflows/e2e-matrix.ymlwith:image: ghcr.io/activepieces/activepieces-cloud:${{ needs.cut.outputs.image_tag }}@@ -132,8 +250,8 @@ jobs:secrets: inheritupgrade-test:- needs: cut- if: needs.cut.outputs.skip != 'true'+ needs: [cut, build-pick]+ if: always() && needs.cut.result == 'success' && needs.cut.outputs.skip != 'true' && (needs.build-pick.result == 'success' || needs.build-pick.result == 'skipped')uses: ./.github/workflows/upgrade-test.ymlwith:image: ghcr.io/activepieces/activepieces-cloud:${{ needs.cut.outputs.image_tag }}@@ -216,7 +334,7 @@ jobs:name: productionurl: https://cloud.activepieces.comconcurrency:- group: cloud-approval+ group: ${{ inputs.hotfix_prs && format('cloud-approval-hotfix-{0}', github.run_id) || 'cloud-approval' }}cancel-in-progress: truesteps:- run: echo "Approved ${{ needs.cut.outputs.image_tag }}"@@ -281,7 +399,7 @@ jobs:run: |read_version() {ssh ops -t -t "bash -ic \"cd mrsk/prod && kamal app version --config-file=$1 2>&1; exit\"" \- | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta$' || true; } | sort -u+ | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)$' || true; } | sort -u}FLEET=$(read_version config/app.yml)CANARY=$(read_version config/app-canary.yml)@@ -363,7 +481,7 @@ jobs:env:WEBHOOK: ${{ secrets.DISCORD_ON_CALL_WEBHOOK }}IMAGE_TAG: ${{ needs.cut.outputs.image_tag }}- DEPLOY_KIND: ${{ github.event_name == 'schedule' && 'daily candidate' || 'manual run' }}+ DEPLOY_KIND: ${{ inputs.hotfix_prs && format('hotfix {0}', inputs.hotfix_prs) || github.event_name == 'schedule' && 'daily candidate' || 'manual run' }}run: |curl -H "Content-Type: application/json" \-X POST \@@ -383,9 +501,10 @@ jobs:"$WEBHOOK"# Records the deploy in the Linear "Cloud" release pipeline. Never fails the workflow.- # A rollback or emergency deploy does not sync.+ # A rollback, an emergency deploy or a hotfix deploy does not sync; the next normal deploy records it.linear-release:needs: [cut, deploy]+ if: ${{ !inputs.hotfix_prs }}runs-on: ubuntu-latesttimeout-minutes: 5continue-on-error: truediff --git a/.github/workflows/continuous-delivery-rollback.yml b/.github/workflows/continuous-delivery-rollback.ymlindex 83b536c42ab..fb0b89cf845 100644--- a/.github/workflows/continuous-delivery-rollback.yml+++ b/.github/workflows/continuous-delivery-rollback.yml@@ -48,7 +48,7 @@ jobs:id: currentrun: |OUTPUT=$(ssh ops -t -t 'bash -ic "cd mrsk/${{ inputs.target }} && kamal app version --primary --config-file=config/app.yml 2>&1; exit"' | tr -d '\r')- CURRENT=$(echo "$OUTPUT" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta' | head -1)+ CURRENT=$(echo "$OUTPUT" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)' | head -1)SHA=$(echo "$CURRENT" | cut -d. -f4)if [ -z "$SHA" ]; thenecho "$OUTPUT"diff --git a/brain/knowledge/engineering/cloud-deployment-paths.md b/brain/knowledge/engineering/cloud-deployment-paths.mdindex 68546ccf05c..3f071574be4 100644--- a/brain/knowledge/engineering/cloud-deployment-paths.md+++ b/brain/knowledge/engineering/cloud-deployment-paths.md@@ -10,7 +10,7 @@ How code reaches `cloud.activepieces.com`. `continuous-delivery-cloud.yml` is th`cut` takes the newest `main` commit whose `e2e-staging` check run passed (a manual run can name a `sha`, which must also have passed), and stops a scheduled run when the fleet already runs it. Every later job uses the `version.sha.beta` image that staging built from that commit; nothing is built again. The `approve` job holds the only approval (`production` environment). The `deploy` job then runs the breaking-migration check and deploys the canary and the fleet, a few minutes apart.## Hotfixes-A hotfix merges into `main` like any other change, and a manual run deploys it once `e2e-staging` passes. There is no path that builds from a branch. `emergency-cloud-deploy.yml` and `continuous-delivery-rollback.yml` stay outside the gate.+A hotfix merges into `main` first. A manual gate run with `hotfix_prs` (ENG-1098) cherry-picks only those PRs onto the commit Cloud runs, pushes `hotfix/cloud/<run id>-<attempt>`, builds `<version>.<sha>.pick` in `build-pick`, and sends it through the same checks, approval and `deploy`. Picks stack until the next normal run, which must contain every picked PR. `emergency-cloud-deploy.yml` and `continuous-delivery-rollback.yml` stay outside the gate.## Staging (upstream of the gate)`continuous-delivery-stg.yml` builds every push to `main` and deploys it with **Kamal**, not Kubernetes: it SSHes to the devops box and runs `kamal deploy --config-file=config/{app,worker}.yml` from `/root/mrsk/stg`. App containers live on one host, workers on another; `kubectl`'s `stg` context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in `config/app.yml` under `env.clear` (secrets are name-listed under `env.secret` and read from `.kamal/secrets`). After both deploys, `e2e-staging` runs the Playwright suite against staging; its check run on the commit is what the gate's `cut` reads.@@ -19,6 +19,9 @@ A hotfix merges into `main` like any other change, and a manual run deploys it o- **Kamal reads its config at deploy time, so editing `config/app.yml` during an in-flight CD run silently misses.** The build takes ~5 min and the deploy job reads the file when *it* starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-run `kamal deploy --version <same tag> --config-file=config/app.yml --skip-push` after editing, and verify with `docker inspect <container> --format '{{range .Config.Env}}...'` on **every** container, not one. The command needs a TTY (`ssh -tt`); without it Kamal exits 0 having done nothing ("the input device is not a TTY").- **The breaking-migration check runs in the `deploy` job, after the approval.** It fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`), so the fleet never deploys. To ship a breaking migration on purpose, run the gate by hand with `skip_migration_check`.- **`deploy` re-reads Cloud under the lock and refuses to drop anything Cloud runs.** It fails when the canary and the fleet run different images, when the fleet moved after `cut`, or when a commit in `merge-base(cloud, candidate)..cloud` has no `(#N)` subject or its PR number is missing from the candidate's history. That one rule blocks a downgrade and a silent revert of a fix shipped through another path (an emergency deploy, later hotfix mode); comparing commits alone misses the second case, because cherry-picked SHAs never match `main`. After `emergency-cloud-deploy.yml` (fleet only), deploy the fleet's tag to the canary by hand or the gate stays blocked.+- **A hotfix run has its own approval group (`cloud-approval-hotfix-<run id>`).** The full gate run's group cancels a waiting older run, which would otherwise cancel a hotfix waiting for approval. The re-read under the lock in `deploy` catches a hotfix and a full gate run that were cut from the same Cloud commit.+- **The hotfix push uses the release GitHub App token, not `GITHUB_TOKEN`.** A picked commit can change a workflow file, and `GITHUB_TOKEN` cannot push those.+- **Tag readers accept `.beta` and `.pick`.** The gate and `continuous-delivery-rollback.yml` both parse the running tag; a reader that matches only `.beta` reads nothing after a hotfix deploy.- **The approval sits before the canary on purpose.** The canary runs migrations on the shared database at boot, so a canary deploy that waits days for the fleet's approval brings back the ENG-1062 problem.- **A reusable workflow checks out `github.sha`, not the candidate.** `e2e-matrix.yml` and `upgrade-test.yml` take a `ref` input, and the gate passes the candidate SHA; without it the specs come from the newest `main`.- **The canary and the fleet deploy in one job on purpose.** GitHub keeps only one pending job per concurrency group, even with `cancel-in-progress: false`, so with a lock per job a third run could replace the second run's pending fleet deploy after its canary had migrated the shared database. The `deploy` job's group is `promote-production`, the same as `continuous-delivery-rollback.yml`, so a rollback cannot interleave with a gate deploy either.diff --git a/docs/handbook/engineering/onboarding/release-cycle.mdx b/docs/handbook/engineering/onboarding/release-cycle.mdxindex 66d36031194..0277c844e91 100644--- a/docs/handbook/engineering/onboarding/release-cycle.mdx+++ b/docs/handbook/engineering/onboarding/release-cycle.mdx@@ -44,7 +44,7 @@ Self-hosted releases use `release-self-hosted.yml`. See [Releases](/handbook/eng## Overrides-- **Hotfix**: Merge the fix into `main`. When `e2e-staging` passes, run `continuous-delivery-cloud.yml` by hand. It runs the same checks and the same approval, like any other Cloud deploy.+- **Hotfix**: Merge the fix into `main`, then run `continuous-delivery-cloud.yml` by hand with `hotfix_prs`. Cloud gets only those PRs on top of what it runs, after the same checks and approval. The rest of `main` waits for the next full gate run.- **Emergency**: Use `emergency-cloud-deploy.yml` to deploy directly to production, bypassing staging entirely. Use sparingly.## FAQdiff --git a/docs/handbook/engineering/playbooks/releases.mdx b/docs/handbook/engineering/playbooks/releases.mdxindex 94f11d7aac9..cf1d01ff3d1 100644--- a/docs/handbook/engineering/playbooks/releases.mdx+++ b/docs/handbook/engineering/playbooks/releases.mdx@@ -23,9 +23,13 @@ All changes flow through a staging environment before reaching production. There## Hotfix Workflow- 1. Merge the fix into `main`. Staging deploys it and runs `e2e-staging`.- 2. When `e2e-staging` passes, run `continuous-delivery-cloud.yml` by hand. Leave `sha` empty to take the newest passed commit, or name the fix commit.- 3. The gate runs the same checks and asks for the same approval as any other Cloud deploy. There is no path that builds from a branch.+ 1. Merge the fix into `main`.+ 2. Run `continuous-delivery-cloud.yml` by hand with `hotfix_prs` set to the PR numbers, for example `17001,17005`. The gate cherry-picks only those PRs, oldest first, onto the commit Cloud runs (branch `hotfix/cloud/<run id>-<attempt>`), builds a `version.sha.pick` image, and runs the E2E matrix and the upgrade test on it. The rest of `main` waits for the next full gate run.+ 3. The run summary names what Cloud runs and what this run adds. The approval and the deploy are the same as any other Cloud deploy.++ The run stops when a PR is not merged into `main`, is already on Cloud, does not apply cleanly, or changes a migration (set `allow_migrations` only when the migration needs nothing Cloud lacks). A hotfix deploy records no Linear release; the next full gate run does. That run deploys `main` only if `main` contains every picked PR.++ To ship all of `main` at once instead, leave `hotfix_prs` empty: the gate takes the newest commit that passed `e2e-staging`, or the commit named in `sha`.If Cloud is down and the gate is too slow, use `emergency-cloud-deploy.yml`. To go back to an earlier image, use `continuous-delivery-rollback.yml`.