Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

ci(cloud): deploy only chosen hotfix PRs on top of what Cloud runs (#16997)

Othman Abu Ajamieh48dee01cca5b
diff --git a/.github/workflows/continuous-delivery-cloud.yml b/.github/workflows/continuous-delivery-cloud.yml
index 68f3bf7606d..728fb959818 100644
--- a/.github/workflows/continuous-delivery-cloud.yml
+++ b/.github/workflows/continuous-delivery-cloud.yml
@@ -2,6 +2,8 @@ name: Continuous Delivery — Cloud
# The Cloud gate. Picks the newest main commit that passed E2E on staging, tests the image staging built
# from it, waits for one approval, then deploys that image to the canary and to the fleet.
+# With hotfix_prs, it instead cherry-picks only those merged PRs onto the commit Cloud runs, builds that,
+# and sends it through the same checks, approval and deploy.
on:
workflow_dispatch:
inputs:
@@ -9,6 +11,14 @@ on:
description: 'Commit on main to deploy. Empty takes the newest one that passed E2E on staging.'
type: string
default: ''
+ hotfix_prs:
+ description: 'Comma-separated PR numbers already merged into main. Deploys only these on top of what Cloud runs.'
+ type: string
+ default: ''
+ allow_migrations:
+ description: 'Let a hotfix PR that changes a migration through'
+ type: boolean
+ default: false
skip_migration_check:
description: 'Deploy even when a pending migration is marked breaking'
type: boolean
@@ -23,21 +33,64 @@ jobs:
cut:
runs-on: ubuntu-latest
outputs:
- sha: ${{ steps.pick.outputs.sha }}
- image_tag: ${{ steps.pick.outputs.image_tag }}
- from_tag: ${{ steps.pick.outputs.from_tag }}
+ sha: ${{ steps.pick.outputs.sha || steps.hotfix.outputs.sha }}
+ image_tag: ${{ steps.pick.outputs.image_tag || steps.hotfix.outputs.image_tag }}
+ from_tag: ${{ steps.from.outputs.tag }}
cloud_sha: ${{ steps.cloud.outputs.sha }}
- skip: ${{ steps.cloud.outputs.skip }}
+ skip: ${{ steps.pick.outputs.skip }}
steps:
+ # The App token can push a pick branch whose commits change workflow files; GITHUB_TOKEN cannot.
+ - name: Create GitHub App token
+ id: app-token
+ if: inputs.hotfix_prs
+ uses: actions/create-github-app-token@v2
+ with:
+ app-id: ${{ secrets.RELEASE_APP_ID }}
+ private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}
+
- uses: actions/checkout@v5
with:
fetch-depth: 0
+ token: ${{ steps.app-token.outputs.token || github.token }}
+
+ - name: Configure SSH
+ run: |
+ mkdir -p ~/.ssh/
+ echo "$SSH_KEY" > ~/.ssh/ops.key
+ chmod 600 ~/.ssh/ops.key
+ cat >>~/.ssh/config <<END
+ Host ops
+ HostName $SSH_HOST
+ User $SSH_USER
+ IdentityFile ~/.ssh/ops.key
+ StrictHostKeyChecking no
+ END
+ env:
+ SSH_USER: ${{ secrets.DEV_OPS_USERNAME }}
+ SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
+ SSH_HOST: ${{ secrets.DEV_OPS_HOST }}
+
+ - name: Read the commit Cloud runs
+ id: cloud
+ run: |
+ CLOUD_SHA=$(ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal app version --config-file=config/app.yml 2>&1; exit"' \
+ | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)$' || true; } | head -1 | cut -d. -f4)
+ if [ -z "$CLOUD_SHA" ]; then
+ echo "::error::Could not read the version the Cloud fleet runs"
+ exit 1
+ fi
+ echo "sha=$CLOUD_SHA" >> "$GITHUB_OUTPUT"
+ echo "Cloud runs \`$CLOUD_SHA\`" >> "$GITHUB_STEP_SUMMARY"
+ # A scheduled run stops when every app and worker host, canary included, already runs the candidate,
+ # so a quiet day asks for no approval and a deploy that stopped half way is retried.
- name: Pick the candidate
id: pick
+ if: ${{ !inputs.hotfix_prs }}
env:
GH_TOKEN: ${{ github.token }}
INPUT_SHA: ${{ inputs.sha }}
+ CLOUD_SHA: ${{ steps.cloud.outputs.sha }}
run: |
e2e_passed() {
[ -n "$(gh api "repos/$GITHUB_REPOSITORY/commits/$1/check-runs?check_name=e2e-staging" \
@@ -67,50 +120,14 @@ jobs:
fi
fi
IMAGE_TAG="$(git show "$SHA:package.json" | jq -r .version).$SHA.beta"
- FROM_TAG=$(gh release list --repo "$GITHUB_REPOSITORY" --exclude-drafts --exclude-pre-releases -L 50 --json tagName --jq '.[].tagName' \
- | grep -E '^[0-9]+\.[0-9]+\.[0-9]+(-hotfix\.[0-9]+)?$' | sort -V | tail -1)
echo "sha=$SHA" >> "$GITHUB_OUTPUT"
echo "image_tag=$IMAGE_TAG" >> "$GITHUB_OUTPUT"
- echo "from_tag=$FROM_TAG" >> "$GITHUB_OUTPUT"
- echo "Candidate \`$SHA\`, image \`$IMAGE_TAG\`, upgrade test from \`$FROM_TAG\`" >> "$GITHUB_STEP_SUMMARY"
-
- - name: Configure SSH
- run: |
- mkdir -p ~/.ssh/
- echo "$SSH_KEY" > ~/.ssh/ops.key
- chmod 600 ~/.ssh/ops.key
- cat >>~/.ssh/config <<END
- Host ops
- HostName $SSH_HOST
- User $SSH_USER
- IdentityFile ~/.ssh/ops.key
- StrictHostKeyChecking no
- END
- env:
- SSH_USER: ${{ secrets.DEV_OPS_USERNAME }}
- SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
- SSH_HOST: ${{ secrets.DEV_OPS_HOST }}
-
- # A scheduled run stops when every app and worker host, canary included, already runs the candidate,
- # so a quiet day asks for no approval and a deploy that stopped half way is retried.
- - name: Read the commit Cloud runs
- id: cloud
- env:
- SHA: ${{ steps.pick.outputs.sha }}
- run: |
- CLOUD_SHA=$(ssh ops -t -t 'bash -ic "cd mrsk/prod && kamal app version --config-file=config/app.yml 2>&1; exit"' \
- | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta$' || true; } | head -1 | cut -d. -f4)
- if [ -z "$CLOUD_SHA" ]; then
- echo "::error::Could not read the version the Cloud fleet runs"
- exit 1
- fi
- echo "sha=$CLOUD_SHA" >> "$GITHUB_OUTPUT"
- echo "Cloud runs \`$CLOUD_SHA\`" >> "$GITHUB_STEP_SUMMARY"
+ echo "Candidate \`$SHA\`, image \`$IMAGE_TAG\`" >> "$GITHUB_STEP_SUMMARY"
if [ "$GITHUB_EVENT_NAME" = schedule ] && [ "$CLOUD_SHA" = "$SHA" ]; then
SKIP=true
for CONFIG in app worker app-canary worker-canary; do
CONFIG_SHA=$(ssh ops -t -t "bash -ic \"cd mrsk/prod && kamal app version --config-file=config/$CONFIG.yml 2>&1; exit\"" \
- | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta$' || true; } | sort -u | cut -d. -f4)
+ | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)$' || true; } | sort -u | cut -d. -f4)
if [ "$CONFIG_SHA" != "$SHA" ]; then
SKIP=false
echo "\`config/$CONFIG.yml\` does not run the candidate on every host" >> "$GITHUB_STEP_SUMMARY"
@@ -122,9 +139,110 @@ jobs:
fi
fi
- e2e-matrix:
+ # Each named PR's squash commit is cherry-picked from main, oldest first, onto the commit Cloud runs.
+ # The branch is pushed so the commit Cloud will run stays in the repository.
+ - name: Cherry-pick the hotfix PRs onto what Cloud runs
+ id: hotfix
+ if: inputs.hotfix_prs
+ env:
+ GH_TOKEN: ${{ github.token }}
+ HOTFIX_PRS: ${{ inputs.hotfix_prs }}
+ INPUT_SHA: ${{ inputs.sha }}
+ ALLOW_MIGRATIONS: ${{ inputs.allow_migrations }}
+ CLOUD_SHA: ${{ steps.cloud.outputs.sha }}
+ run: |
+ if [ -n "$INPUT_SHA" ]; then
+ echo "::error::Set either sha or hotfix_prs, not both"
+ exit 1
+ fi
+ COMMITS=""
+ for PR in $(echo "$HOTFIX_PRS" | tr ',#' ' '); do
+ read -r STATE BASE MERGE <<< "$(gh pr view "$PR" --repo "$GITHUB_REPOSITORY" --json state,baseRefName,mergeCommit \
+ --jq '"\(.state) \(.baseRefName) \(.mergeCommit.oid // "")"')"
+ if [ "$STATE" != MERGED ] || [ "$BASE" != main ] || [ -z "$MERGE" ] || ! git merge-base --is-ancestor "$MERGE" origin/main; then
+ echo "::error::#$PR is not merged into main"
+ exit 1
+ fi
+ if git log --format=%s "$CLOUD_SHA" | grep -qE "\(#$PR\)$"; then
+ echo "::error::#$PR is already on Cloud"
+ exit 1
+ fi
+ MIGRATIONS=$(git diff --name-only "$MERGE^" "$MERGE" -- packages/server/api/src/app/database/migration/ packages/server/api/src/app/ee/database/migrations/)
+ if [ -n "$MIGRATIONS" ] && [ "$ALLOW_MIGRATIONS" != true ]; then
+ echo "::error::#$PR changes a migration ($MIGRATIONS). It may need schema changes Cloud does not have yet. Set allow_migrations to pick it anyway."
+ exit 1
+ fi
+ COMMITS="$COMMITS$MERGE"$'\n'
+ done
+ BRANCH="hotfix/cloud/$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT"
+ git config user.name "github-actions[bot]"
+ git config user.email "github-actions[bot]@users.noreply.github.com"
+ git checkout -q -b "$BRANCH" "$CLOUD_SHA"
+ for COMMIT in $(git rev-list --first-parent --reverse origin/main | grep -Fx -f <(printf '%s' "$COMMITS")); do
+ if ! git cherry-pick -x "$COMMIT"; then
+ git cherry-pick --abort
+ echo "::error::$(git log -1 --format=%s "$COMMIT") does not apply cleanly on what Cloud runs"
+ exit 1
+ fi
+ done
+ git push origin "$BRANCH"
+ SHA=$(git rev-parse HEAD)
+ IMAGE_TAG="$(git show "$SHA:package.json" | jq -r .version).$SHA.pick"
+ EARLIER=$(git log --first-parent --format=%s "$(git merge-base "$CLOUD_SHA" origin/main)..$CLOUD_SHA" | grep -oE '#[0-9]+\)$' | tr -d ')' | paste -sd ' ' || true)
+ ADDED=$(git log --first-parent --reverse --format=%s "$CLOUD_SHA..$SHA" | grep -oE '#[0-9]+\)$' | tr -d ')' | paste -sd ' ')
+ echo "sha=$SHA" >> "$GITHUB_OUTPUT"
+ echo "image_tag=$IMAGE_TAG" >> "$GITHUB_OUTPUT"
+ {
+ echo "**Hotfix:** Cloud runs \`${CLOUD_SHA:0:10}\`${EARLIER:+ (main plus $EARLIER)}. This run adds $ADDED."
+ echo
+ echo "Branch \`$BRANCH\`, image \`$IMAGE_TAG\`"
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ - name: Pick the release the upgrade test starts from
+ id: from
+ if: steps.pick.outputs.skip != 'true'
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ FROM_TAG=$(gh release list --repo "$GITHUB_REPOSITORY" --exclude-drafts --exclude-pre-releases -L 50 --json tagName --jq '.[].tagName' \
+ | grep -E '^[0-9]+\.[0-9]+\.[0-9]+(-hotfix\.[0-9]+)?$' | sort -V | tail -1)
+ echo "tag=$FROM_TAG" >> "$GITHUB_OUTPUT"
+ echo "Upgrade test from \`$FROM_TAG\`" >> "$GITHUB_STEP_SUMMARY"
+
+ # Staging never built a hotfix combination, so the gate builds it. The .pick tag never replaces a .beta image.
+ build-pick:
needs: cut
- if: needs.cut.outputs.skip != 'true'
+ if: inputs.hotfix_prs
+ runs-on: ubuntu-24.04
+ permissions:
+ contents: read
+ packages: write
+ steps:
+ - uses: actions/checkout@v5
+ with:
+ ref: ${{ needs.cut.outputs.sha }}
+ - uses: docker/login-action@v4
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - uses: depot/setup-action@v1
+ - uses: depot/build-push-action@v1
+ with:
+ project: du7O4b0e8P
+ token: ${{ secrets.DEPOT_PROJECT_TOKEN }}
+ context: .
+ file: ./Dockerfile
+ platforms: |
+ linux/amd64
+ linux/arm64
+ push: true
+ no-cache: true
+ tags: ghcr.io/activepieces/activepieces-cloud:${{ needs.cut.outputs.image_tag }}
+
+ e2e-matrix:
+ needs: [cut, build-pick]
+ if: always() && needs.cut.result == 'success' && needs.cut.outputs.skip != 'true' && (needs.build-pick.result == 'success' || needs.build-pick.result == 'skipped')
uses: ./.github/workflows/e2e-matrix.yml
with:
image: ghcr.io/activepieces/activepieces-cloud:${{ needs.cut.outputs.image_tag }}
@@ -132,8 +250,8 @@ jobs:
secrets: inherit
upgrade-test:
- needs: cut
- if: needs.cut.outputs.skip != 'true'
+ needs: [cut, build-pick]
+ if: always() && needs.cut.result == 'success' && needs.cut.outputs.skip != 'true' && (needs.build-pick.result == 'success' || needs.build-pick.result == 'skipped')
uses: ./.github/workflows/upgrade-test.yml
with:
image: ghcr.io/activepieces/activepieces-cloud:${{ needs.cut.outputs.image_tag }}
@@ -216,7 +334,7 @@ jobs:
name: production
url: https://cloud.activepieces.com
concurrency:
- group: cloud-approval
+ group: ${{ inputs.hotfix_prs && format('cloud-approval-hotfix-{0}', github.run_id) || 'cloud-approval' }}
cancel-in-progress: true
steps:
- run: echo "Approved ${{ needs.cut.outputs.image_tag }}"
@@ -281,7 +399,7 @@ jobs:
run: |
read_version() {
ssh ops -t -t "bash -ic \"cd mrsk/prod && kamal app version --config-file=$1 2>&1; exit\"" \
- | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta$' || true; } | sort -u
+ | tr -d '\r' | { grep -oE '^[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)$' || true; } | sort -u
}
FLEET=$(read_version config/app.yml)
CANARY=$(read_version config/app-canary.yml)
@@ -363,7 +481,7 @@ jobs:
env:
WEBHOOK: ${{ secrets.DISCORD_ON_CALL_WEBHOOK }}
IMAGE_TAG: ${{ needs.cut.outputs.image_tag }}
- DEPLOY_KIND: ${{ github.event_name == 'schedule' && 'daily candidate' || 'manual run' }}
+ DEPLOY_KIND: ${{ inputs.hotfix_prs && format('hotfix {0}', inputs.hotfix_prs) || github.event_name == 'schedule' && 'daily candidate' || 'manual run' }}
run: |
curl -H "Content-Type: application/json" \
-X POST \
@@ -383,9 +501,10 @@ jobs:
"$WEBHOOK"
# Records the deploy in the Linear "Cloud" release pipeline. Never fails the workflow.
- # A rollback or emergency deploy does not sync.
+ # A rollback, an emergency deploy or a hotfix deploy does not sync; the next normal deploy records it.
linear-release:
needs: [cut, deploy]
+ if: ${{ !inputs.hotfix_prs }}
runs-on: ubuntu-latest
timeout-minutes: 5
continue-on-error: true
diff --git a/.github/workflows/continuous-delivery-rollback.yml b/.github/workflows/continuous-delivery-rollback.yml
index 83b536c42ab..fb0b89cf845 100644
--- a/.github/workflows/continuous-delivery-rollback.yml
+++ b/.github/workflows/continuous-delivery-rollback.yml
@@ -48,7 +48,7 @@ jobs:
id: current
run: |
OUTPUT=$(ssh ops -t -t 'bash -ic "cd mrsk/${{ inputs.target }} && kamal app version --primary --config-file=config/app.yml 2>&1; exit"' | tr -d '\r')
- CURRENT=$(echo "$OUTPUT" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.beta' | head -1)
+ CURRENT=$(echo "$OUTPUT" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[a-f0-9]{40}\.(beta|pick)' | head -1)
SHA=$(echo "$CURRENT" | cut -d. -f4)
if [ -z "$SHA" ]; then
echo "$OUTPUT"
diff --git a/brain/knowledge/engineering/cloud-deployment-paths.md b/brain/knowledge/engineering/cloud-deployment-paths.md
index 68546ccf05c..3f071574be4 100644
--- a/brain/knowledge/engineering/cloud-deployment-paths.md
+++ b/brain/knowledge/engineering/cloud-deployment-paths.md
@@ -10,7 +10,7 @@ How code reaches `cloud.activepieces.com`. `continuous-delivery-cloud.yml` is th
`cut` takes the newest `main` commit whose `e2e-staging` check run passed (a manual run can name a `sha`, which must also have passed), and stops a scheduled run when the fleet already runs it. Every later job uses the `version.sha.beta` image that staging built from that commit; nothing is built again. The `approve` job holds the only approval (`production` environment). The `deploy` job then runs the breaking-migration check and deploys the canary and the fleet, a few minutes apart.
## Hotfixes
-A hotfix merges into `main` like any other change, and a manual run deploys it once `e2e-staging` passes. There is no path that builds from a branch. `emergency-cloud-deploy.yml` and `continuous-delivery-rollback.yml` stay outside the gate.
+A hotfix merges into `main` first. A manual gate run with `hotfix_prs` (ENG-1098) cherry-picks only those PRs onto the commit Cloud runs, pushes `hotfix/cloud/<run id>-<attempt>`, builds `<version>.<sha>.pick` in `build-pick`, and sends it through the same checks, approval and `deploy`. Picks stack until the next normal run, which must contain every picked PR. `emergency-cloud-deploy.yml` and `continuous-delivery-rollback.yml` stay outside the gate.
## Staging (upstream of the gate)
`continuous-delivery-stg.yml` builds every push to `main` and deploys it with **Kamal**, not Kubernetes: it SSHes to the devops box and runs `kamal deploy --config-file=config/{app,worker}.yml` from `/root/mrsk/stg`. App containers live on one host, workers on another; `kubectl`'s `stg` context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in `config/app.yml` under `env.clear` (secrets are name-listed under `env.secret` and read from `.kamal/secrets`). After both deploys, `e2e-staging` runs the Playwright suite against staging; its check run on the commit is what the gate's `cut` reads.
@@ -19,6 +19,9 @@ A hotfix merges into `main` like any other change, and a manual run deploys it o
- **Kamal reads its config at deploy time, so editing `config/app.yml` during an in-flight CD run silently misses.** The build takes ~5 min and the deploy job reads the file when *it* starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-run `kamal deploy --version <same tag> --config-file=config/app.yml --skip-push` after editing, and verify with `docker inspect <container> --format '{{range .Config.Env}}...'` on **every** container, not one. The command needs a TTY (`ssh -tt`); without it Kamal exits 0 having done nothing ("the input device is not a TTY").
- **The breaking-migration check runs in the `deploy` job, after the approval.** It fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`), so the fleet never deploys. To ship a breaking migration on purpose, run the gate by hand with `skip_migration_check`.
- **`deploy` re-reads Cloud under the lock and refuses to drop anything Cloud runs.** It fails when the canary and the fleet run different images, when the fleet moved after `cut`, or when a commit in `merge-base(cloud, candidate)..cloud` has no `(#N)` subject or its PR number is missing from the candidate's history. That one rule blocks a downgrade and a silent revert of a fix shipped through another path (an emergency deploy, later hotfix mode); comparing commits alone misses the second case, because cherry-picked SHAs never match `main`. After `emergency-cloud-deploy.yml` (fleet only), deploy the fleet's tag to the canary by hand or the gate stays blocked.
+- **A hotfix run has its own approval group (`cloud-approval-hotfix-<run id>`).** The full gate run's group cancels a waiting older run, which would otherwise cancel a hotfix waiting for approval. The re-read under the lock in `deploy` catches a hotfix and a full gate run that were cut from the same Cloud commit.
+- **The hotfix push uses the release GitHub App token, not `GITHUB_TOKEN`.** A picked commit can change a workflow file, and `GITHUB_TOKEN` cannot push those.
+- **Tag readers accept `.beta` and `.pick`.** The gate and `continuous-delivery-rollback.yml` both parse the running tag; a reader that matches only `.beta` reads nothing after a hotfix deploy.
- **The approval sits before the canary on purpose.** The canary runs migrations on the shared database at boot, so a canary deploy that waits days for the fleet's approval brings back the ENG-1062 problem.
- **A reusable workflow checks out `github.sha`, not the candidate.** `e2e-matrix.yml` and `upgrade-test.yml` take a `ref` input, and the gate passes the candidate SHA; without it the specs come from the newest `main`.
- **The canary and the fleet deploy in one job on purpose.** GitHub keeps only one pending job per concurrency group, even with `cancel-in-progress: false`, so with a lock per job a third run could replace the second run's pending fleet deploy after its canary had migrated the shared database. The `deploy` job's group is `promote-production`, the same as `continuous-delivery-rollback.yml`, so a rollback cannot interleave with a gate deploy either.
diff --git a/docs/handbook/engineering/onboarding/release-cycle.mdx b/docs/handbook/engineering/onboarding/release-cycle.mdx
index 66d36031194..0277c844e91 100644
--- a/docs/handbook/engineering/onboarding/release-cycle.mdx
+++ b/docs/handbook/engineering/onboarding/release-cycle.mdx
@@ -44,7 +44,7 @@ Self-hosted releases use `release-self-hosted.yml`. See [Releases](/handbook/eng
## Overrides
-- **Hotfix**: Merge the fix into `main`. When `e2e-staging` passes, run `continuous-delivery-cloud.yml` by hand. It runs the same checks and the same approval, like any other Cloud deploy.
+- **Hotfix**: Merge the fix into `main`, then run `continuous-delivery-cloud.yml` by hand with `hotfix_prs`. Cloud gets only those PRs on top of what it runs, after the same checks and approval. The rest of `main` waits for the next full gate run.
- **Emergency**: Use `emergency-cloud-deploy.yml` to deploy directly to production, bypassing staging entirely. Use sparingly.
## FAQ
diff --git a/docs/handbook/engineering/playbooks/releases.mdx b/docs/handbook/engineering/playbooks/releases.mdx
index 94f11d7aac9..cf1d01ff3d1 100644
--- a/docs/handbook/engineering/playbooks/releases.mdx
+++ b/docs/handbook/engineering/playbooks/releases.mdx
@@ -23,9 +23,13 @@ All changes flow through a staging environment before reaching production. There
## Hotfix Workflow
- 1. Merge the fix into `main`. Staging deploys it and runs `e2e-staging`.
- 2. When `e2e-staging` passes, run `continuous-delivery-cloud.yml` by hand. Leave `sha` empty to take the newest passed commit, or name the fix commit.
- 3. The gate runs the same checks and asks for the same approval as any other Cloud deploy. There is no path that builds from a branch.
+ 1. Merge the fix into `main`.
+ 2. Run `continuous-delivery-cloud.yml` by hand with `hotfix_prs` set to the PR numbers, for example `17001,17005`. The gate cherry-picks only those PRs, oldest first, onto the commit Cloud runs (branch `hotfix/cloud/<run id>-<attempt>`), builds a `version.sha.pick` image, and runs the E2E matrix and the upgrade test on it. The rest of `main` waits for the next full gate run.
+ 3. The run summary names what Cloud runs and what this run adds. The approval and the deploy are the same as any other Cloud deploy.
+
+ The run stops when a PR is not merged into `main`, is already on Cloud, does not apply cleanly, or changes a migration (set `allow_migrations` only when the migration needs nothing Cloud lacks). A hotfix deploy records no Linear release; the next full gate run does. That run deploys `main` only if `main` contains every picked PR.
+
+ To ship all of `main` at once instead, leave `hotfix_prs` empty: the gate takes the newest commit that passed `e2e-staging`, or the commit named in `sha`.
If Cloud is down and the gate is too slow, use `emergency-cloud-deploy.yml`. To go back to an earlier image, use `continuous-delivery-rollback.yml`.