Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(migrations): rollback coordination lock for background migrations (re-land #16469) (#16493)

Othman Abu Ajamieh96770bb26838
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.md
index d89bcb351d8..1bfaa9f0458 100644
--- a/brain/knowledge/data-storage-observability/database.md
+++ b/brain/knowledge/data-storage-observability/database.md
@@ -238,6 +238,38 @@ public async up(q: QueryRunner): Promise<void> {
- Blocking migrations: `down()` required unless `breaking = true`. `check-migration-rollback.ts` enforces the `breaking` flag on destructive DDL.
- Background migrations: forward-only. `down()` is a no-op body; the rollback tooling skips the `background/` folder entirely. To undo a bad backfill, ship a forward-fix migration.
+### Rollback choreography when a blocking migration pairs with a background one
+
+`withBackgroundMigrationLock` acquires the per-migration lock without checking the completion row. `down()` uses this instead of `assertBackgroundMigrationComplete` because rollback must proceed even when the backfill never ran, failed, or was quiesced mid-run — otherwise a release with an incomplete background migration becomes un-rollbackable.
+
+**When `down()` needs the lock — only if its DDL touches objects a background migration also touches:**
+- Same-release reshape pair (blocking adds columns, background CIC on same table): **lock**.
+- `DROP COLUMN foo` while a background `BackfillFoo` could be in-flight: **lock**.
+- Dependency-style pair (M1's up calls `assertBackgroundMigrationComplete(B)` + does `SET NOT NULL`; down drops NOT NULL): **no lock** — B is already complete and won't re-run.
+- Any blocking migration whose paired background touches disjoint objects: **no lock**.
+
+```ts
+public async down(q: QueryRunner): Promise<void> {
+ await migrationHelpers.withBackgroundMigrationLock({
+ log: system.globalLogger(),
+ migration: ReshapeFooIndexes,
+ fn: async () => {
+ await q.query(`...restore old indexes, drop new ones...`)
+ await q.query(`DELETE FROM background_migrations WHERE name = $1`, [new ReshapeFooIndexes().name])
+ await q.query(`ALTER TABLE foo DROP COLUMN bar`)
+ },
+ })
+}
+```
+
+Deleting the `background_migrations` row is a per-migration choice — tells re-upgrade to re-run the reshape. Omit if `down()` leaves the background work's effects intact.
+
+**Caveats:**
+- Lock wait is bounded at ~4 min (`distributedLockFactory` passes `retryCount * retryDelay = (timeout / 200) * 200ms` into RedLock). If `down()` races a long in-flight backfill, the helper throws rather than waits for the full duration. Fix: scale API pods to 0 (quiesces the runner), re-run `rollback-migrations.ts`, redeploy.
+- The helper opens a Redis connection inside the rollback CLI, which keeps the event loop alive after the last `down()`. `src/rollback.ts` therefore ends with `process.exit(0)`; without it the CLI prints "Rollback complete" and never exits.
+- Destructive data changes a background migration already made (uuid migration, type change) cannot be rolled back by the framework. Ship a reverse-fix background migration or accept the residue.
+- Rolling back release N with its background migration B already completed: B's data and row persist; re-deploy of N skips B. If wrong for your case, delete B's row in `down()`.
+
### Recovery — permanently failing background migrations
A background migration that fails every retry stops the app from booting. The guard refuses the dependent blocking DDL; boot catchup exhausts its 10 attempts and throws with the migration name. Operator options, in order of preference:
@@ -268,7 +300,7 @@ A background migration that fails every retry stops the app from booting. The gu
- `packages/server/api/src/app/database/postgres-connection.ts` — `getMigrations()`, `getBackgroundMigrations()`.
- `packages/server/api/src/app/database/migration.ts` — the `Migration` interface.
- `packages/server/api/src/app/database/background-migration.ts` — the `BackgroundMigration` interface.
-- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `createIndexConcurrently`.
+- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `withBackgroundMigrationLock`, `createIndexConcurrently`, `dropIndexConcurrently`.
- `packages/server/api/src/app/database/background-migration-runner.ts` — the BullMQ handler.
- `packages/server/api/src/migrate.ts` — both tracks in one process, for `check-migrations` and `db-migration`.
- `tools/scripts/check-migration-rollback.ts` — the CI check.
diff --git a/packages/server/api/src/app/database/background-migration-runner.ts b/packages/server/api/src/app/database/background-migration-runner.ts
index b5e57e7a5a1..feab3761d04 100644
--- a/packages/server/api/src/app/database/background-migration-runner.ts
+++ b/packages/server/api/src/app/database/background-migration-runner.ts
@@ -105,8 +105,8 @@ async function runOne({
dataSource: DataSource
}): Promise<void> {
await distributedLock(log).runExclusive({
- key: `background_migration:${migration.name}`,
- timeoutInSeconds: RUN_LOCK_TIMEOUT_SECONDS,
+ key: backgroundMigrationLockKey(migration.name),
+ timeoutInSeconds: BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS,
fn: async () => {
const completed = await getCompletedNames(dataSource)
if (completed.has(migration.name)) {
@@ -212,7 +212,9 @@ async function getLatestFailure(dataSource: DataSource, sourceNames: string[]):
}
const MAX_CATCHUP_ATTEMPTS = 10
-const RUN_LOCK_TIMEOUT_SECONDS = 240
+
+export const backgroundMigrationLockKey = (migrationName: string): string => `background_migration:${migrationName}`
+export const BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS = 240
export const BACKGROUND_MIGRATIONS_TABLE = 'background_migrations'
diff --git a/packages/server/api/src/app/database/migration-helpers.ts b/packages/server/api/src/app/database/migration-helpers.ts
index 7aca94641c1..8379d8be134 100644
--- a/packages/server/api/src/app/database/migration-helpers.ts
+++ b/packages/server/api/src/app/database/migration-helpers.ts
@@ -1,14 +1,17 @@
+import { FastifyBaseLogger } from 'fastify'
import { QueryRunner } from 'typeorm'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
import { BackgroundMigration } from './background-migration'
-import { BACKGROUND_MIGRATIONS_TABLE } from './background-migration-runner'
+import { BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS, BACKGROUND_MIGRATIONS_TABLE, backgroundMigrationLockKey } from './background-migration-runner'
import { DatabaseType } from './database-type'
+import { distributedLock } from './redis-connections'
export const migrationHelpers = {
assertBackgroundMigrationComplete,
createIndexConcurrently,
dropIndexConcurrently,
+ withBackgroundMigrationLock,
}
async function assertBackgroundMigrationComplete({
@@ -28,6 +31,23 @@ async function assertBackgroundMigrationComplete({
}
}
+async function withBackgroundMigrationLock<T>({
+ log,
+ migration,
+ fn,
+}: {
+ log: FastifyBaseLogger
+ migration: new () => BackgroundMigration
+ fn: () => Promise<T>
+}): Promise<T> {
+ const name = new migration().name
+ return distributedLock(log).runExclusive({
+ key: backgroundMigrationLockKey(name),
+ timeoutInSeconds: BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS,
+ fn,
+ })
+}
+
async function createIndexConcurrently({
queryRunner,
name,
@@ -85,6 +105,7 @@ async function dropIndexConcurrently({
await queryRunner.query(`DROP INDEX ${concurrently} IF EXISTS "${name}"`)
}
+
const isPGlite = (): boolean => system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITE
function extractIndisvalid(rows: unknown): boolean | undefined {
diff --git a/packages/server/api/src/rollback.ts b/packages/server/api/src/rollback.ts
index 75c1d0b418f..ac801d24642 100644
--- a/packages/server/api/src/rollback.ts
+++ b/packages/server/api/src/rollback.ts
@@ -83,7 +83,9 @@ async function main(): Promise<void> {
await dataSource.destroy()
}
-main().catch((e) => {
- console.error('Rollback failed:', e)
- process.exit(1)
-})
+main()
+ .then(() => process.exit(0))
+ .catch((e) => {
+ console.error('Rollback failed:', e)
+ process.exit(1)
+ })
diff --git a/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts b/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts
index 1aba692ee40..fb304011a0f 100644
--- a/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts
+++ b/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts
@@ -123,6 +123,17 @@ describe('backgroundMigrationRunner', () => {
}
})
+ it('withBackgroundMigrationLock does NOT require completion — safe for down()', async () => {
+ const neverRanBackfill = makeTestMigration({ name: `NeverRan${Date.now()}` })
+ let fnRan = false
+ await migrationHelpers.withBackgroundMigrationLock({
+ log: app!.log,
+ migration: neverRanBackfill,
+ fn: async () => { fnRan = true },
+ })
+ expect(fnRan).toBe(true)
+ })
+
it('reports pending count via getStatus', async () => {
const done = makeTestMigration({ name: `StatusDone${Date.now()}` })
const pending = makeTestMigration({ name: `StatusPending${Date.now()}` })
@@ -187,6 +198,55 @@ describe('runMigrationsWithCatchup', () => {
})
})
+describe('withBackgroundMigrationLock', () => {
+ it('serializes a concurrent runOne against the lock', async () => {
+ let migrationStarted = false
+ let releaseMigration: () => void = () => undefined
+ const migrationGate = new Promise<void>(resolve => { releaseMigration = resolve })
+ const SlowMigration = makeTestMigration({
+ name: `SlowLockMigration${Date.now()}`,
+ up: async () => {
+ migrationStarted = true
+ await migrationGate
+ },
+ })
+ const runnerPromise = backgroundMigrationRunner.run({ log: app!.log, migrations: [SlowMigration] })
+ let gotLock = false
+ let lockWaiter: Promise<void> | null = null
+ try {
+ await waitFor(() => migrationStarted, 2_000)
+ lockWaiter = migrationHelpers.withBackgroundMigrationLock({
+ log: app!.log,
+ migration: SlowMigration,
+ fn: async () => { gotLock = true },
+ })
+ await new Promise(resolve => setTimeout(resolve, 100))
+ expect(gotLock).toBe(false)
+
+ releaseMigration()
+ await runnerPromise
+ await lockWaiter
+ expect(gotLock).toBe(true)
+ }
+ finally {
+ releaseMigration()
+ await Promise.allSettled([runnerPromise, lockWaiter ?? Promise.resolve()])
+ }
+ })
+
+ it('acquires immediately when no migration is in-flight', async () => {
+ const Any = makeTestMigration({ name: `IdleLock${Date.now()}` })
+ const start = Date.now()
+ await migrationHelpers.withBackgroundMigrationLock({
+ log: app!.log,
+ migration: Any,
+ fn: async () => { /* no-op */ },
+ })
+ const elapsed = Date.now() - start
+ expect(elapsed).toBeLessThan(500)
+ })
+})
+
function makeTestMigration({
name: migrationName,
up: upFn = async () => { /* no-op */ },
@@ -203,6 +263,14 @@ function makeTestMigration({
}
}
+async function waitFor(predicate: () => boolean, timeoutMs: number): Promise<void> {
+ const started = Date.now()
+ while (!predicate()) {
+ if (Date.now() - started > timeoutMs) throw new Error(`waitFor timed out after ${timeoutMs}ms`)
+ await new Promise(resolve => setTimeout(resolve, 25))
+ }
+}
+
async function selectByName(name: string): Promise<{ name: string }[]> {
const rows = await databaseConnection().query(
`SELECT "name" FROM "${BACKGROUND_MIGRATIONS_TABLE}" WHERE "executed_at" IS NOT NULL`,