AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
feat(migrations): rollback coordination lock for background migrations (re-land #16469) (#16493)
Othman Abu Ajamieh96770bb26838
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.mdindex d89bcb351d8..1bfaa9f0458 100644--- a/brain/knowledge/data-storage-observability/database.md+++ b/brain/knowledge/data-storage-observability/database.md@@ -238,6 +238,38 @@ public async up(q: QueryRunner): Promise<void> {- Blocking migrations: `down()` required unless `breaking = true`. `check-migration-rollback.ts` enforces the `breaking` flag on destructive DDL.- Background migrations: forward-only. `down()` is a no-op body; the rollback tooling skips the `background/` folder entirely. To undo a bad backfill, ship a forward-fix migration.+### Rollback choreography when a blocking migration pairs with a background one++`withBackgroundMigrationLock` acquires the per-migration lock without checking the completion row. `down()` uses this instead of `assertBackgroundMigrationComplete` because rollback must proceed even when the backfill never ran, failed, or was quiesced mid-run — otherwise a release with an incomplete background migration becomes un-rollbackable.++**When `down()` needs the lock — only if its DDL touches objects a background migration also touches:**+- Same-release reshape pair (blocking adds columns, background CIC on same table): **lock**.+- `DROP COLUMN foo` while a background `BackfillFoo` could be in-flight: **lock**.+- Dependency-style pair (M1's up calls `assertBackgroundMigrationComplete(B)` + does `SET NOT NULL`; down drops NOT NULL): **no lock** — B is already complete and won't re-run.+- Any blocking migration whose paired background touches disjoint objects: **no lock**.++```ts+public async down(q: QueryRunner): Promise<void> {+ await migrationHelpers.withBackgroundMigrationLock({+ log: system.globalLogger(),+ migration: ReshapeFooIndexes,+ fn: async () => {+ await q.query(`...restore old indexes, drop new ones...`)+ await q.query(`DELETE FROM background_migrations WHERE name = $1`, [new ReshapeFooIndexes().name])+ await q.query(`ALTER TABLE foo DROP COLUMN bar`)+ },+ })+}+```++Deleting the `background_migrations` row is a per-migration choice — tells re-upgrade to re-run the reshape. Omit if `down()` leaves the background work's effects intact.++**Caveats:**+- Lock wait is bounded at ~4 min (`distributedLockFactory` passes `retryCount * retryDelay = (timeout / 200) * 200ms` into RedLock). If `down()` races a long in-flight backfill, the helper throws rather than waits for the full duration. Fix: scale API pods to 0 (quiesces the runner), re-run `rollback-migrations.ts`, redeploy.+- The helper opens a Redis connection inside the rollback CLI, which keeps the event loop alive after the last `down()`. `src/rollback.ts` therefore ends with `process.exit(0)`; without it the CLI prints "Rollback complete" and never exits.+- Destructive data changes a background migration already made (uuid migration, type change) cannot be rolled back by the framework. Ship a reverse-fix background migration or accept the residue.+- Rolling back release N with its background migration B already completed: B's data and row persist; re-deploy of N skips B. If wrong for your case, delete B's row in `down()`.+### Recovery — permanently failing background migrationsA background migration that fails every retry stops the app from booting. The guard refuses the dependent blocking DDL; boot catchup exhausts its 10 attempts and throws with the migration name. Operator options, in order of preference:@@ -268,7 +300,7 @@ A background migration that fails every retry stops the app from booting. The gu- `packages/server/api/src/app/database/postgres-connection.ts` — `getMigrations()`, `getBackgroundMigrations()`.- `packages/server/api/src/app/database/migration.ts` — the `Migration` interface.- `packages/server/api/src/app/database/background-migration.ts` — the `BackgroundMigration` interface.-- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `createIndexConcurrently`.+- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `withBackgroundMigrationLock`, `createIndexConcurrently`, `dropIndexConcurrently`.- `packages/server/api/src/app/database/background-migration-runner.ts` — the BullMQ handler.- `packages/server/api/src/migrate.ts` — both tracks in one process, for `check-migrations` and `db-migration`.- `tools/scripts/check-migration-rollback.ts` — the CI check.diff --git a/packages/server/api/src/app/database/background-migration-runner.ts b/packages/server/api/src/app/database/background-migration-runner.tsindex b5e57e7a5a1..feab3761d04 100644--- a/packages/server/api/src/app/database/background-migration-runner.ts+++ b/packages/server/api/src/app/database/background-migration-runner.ts@@ -105,8 +105,8 @@ async function runOne({dataSource: DataSource}): Promise<void> {await distributedLock(log).runExclusive({- key: `background_migration:${migration.name}`,- timeoutInSeconds: RUN_LOCK_TIMEOUT_SECONDS,+ key: backgroundMigrationLockKey(migration.name),+ timeoutInSeconds: BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS,fn: async () => {const completed = await getCompletedNames(dataSource)if (completed.has(migration.name)) {@@ -212,7 +212,9 @@ async function getLatestFailure(dataSource: DataSource, sourceNames: string[]):}const MAX_CATCHUP_ATTEMPTS = 10-const RUN_LOCK_TIMEOUT_SECONDS = 240++export const backgroundMigrationLockKey = (migrationName: string): string => `background_migration:${migrationName}`+export const BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS = 240export const BACKGROUND_MIGRATIONS_TABLE = 'background_migrations'diff --git a/packages/server/api/src/app/database/migration-helpers.ts b/packages/server/api/src/app/database/migration-helpers.tsindex 7aca94641c1..8379d8be134 100644--- a/packages/server/api/src/app/database/migration-helpers.ts+++ b/packages/server/api/src/app/database/migration-helpers.ts@@ -1,14 +1,17 @@+import { FastifyBaseLogger } from 'fastify'import { QueryRunner } from 'typeorm'import { system } from '../helper/system/system'import { AppSystemProp } from '../helper/system/system-props'import { BackgroundMigration } from './background-migration'-import { BACKGROUND_MIGRATIONS_TABLE } from './background-migration-runner'+import { BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS, BACKGROUND_MIGRATIONS_TABLE, backgroundMigrationLockKey } from './background-migration-runner'import { DatabaseType } from './database-type'+import { distributedLock } from './redis-connections'export const migrationHelpers = {assertBackgroundMigrationComplete,createIndexConcurrently,dropIndexConcurrently,+ withBackgroundMigrationLock,}async function assertBackgroundMigrationComplete({@@ -28,6 +31,23 @@ async function assertBackgroundMigrationComplete({}}+async function withBackgroundMigrationLock<T>({+ log,+ migration,+ fn,+}: {+ log: FastifyBaseLogger+ migration: new () => BackgroundMigration+ fn: () => Promise<T>+}): Promise<T> {+ const name = new migration().name+ return distributedLock(log).runExclusive({+ key: backgroundMigrationLockKey(name),+ timeoutInSeconds: BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS,+ fn,+ })+}+async function createIndexConcurrently({queryRunner,name,@@ -85,6 +105,7 @@ async function dropIndexConcurrently({await queryRunner.query(`DROP INDEX ${concurrently} IF EXISTS "${name}"`)}+const isPGlite = (): boolean => system.get(AppSystemProp.DB_TYPE) === DatabaseType.PGLITEfunction extractIndisvalid(rows: unknown): boolean | undefined {diff --git a/packages/server/api/src/rollback.ts b/packages/server/api/src/rollback.tsindex 75c1d0b418f..ac801d24642 100644--- a/packages/server/api/src/rollback.ts+++ b/packages/server/api/src/rollback.ts@@ -83,7 +83,9 @@ async function main(): Promise<void> {await dataSource.destroy()}-main().catch((e) => {- console.error('Rollback failed:', e)- process.exit(1)-})+main()+ .then(() => process.exit(0))+ .catch((e) => {+ console.error('Rollback failed:', e)+ process.exit(1)+ })diff --git a/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts b/packages/server/api/test/integration/ce/database/background-migration-runner.test.tsindex 1aba692ee40..fb304011a0f 100644--- a/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts+++ b/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts@@ -123,6 +123,17 @@ describe('backgroundMigrationRunner', () => {}})+ it('withBackgroundMigrationLock does NOT require completion — safe for down()', async () => {+ const neverRanBackfill = makeTestMigration({ name: `NeverRan${Date.now()}` })+ let fnRan = false+ await migrationHelpers.withBackgroundMigrationLock({+ log: app!.log,+ migration: neverRanBackfill,+ fn: async () => { fnRan = true },+ })+ expect(fnRan).toBe(true)+ })+it('reports pending count via getStatus', async () => {const done = makeTestMigration({ name: `StatusDone${Date.now()}` })const pending = makeTestMigration({ name: `StatusPending${Date.now()}` })@@ -187,6 +198,55 @@ describe('runMigrationsWithCatchup', () => {})})+describe('withBackgroundMigrationLock', () => {+ it('serializes a concurrent runOne against the lock', async () => {+ let migrationStarted = false+ let releaseMigration: () => void = () => undefined+ const migrationGate = new Promise<void>(resolve => { releaseMigration = resolve })+ const SlowMigration = makeTestMigration({+ name: `SlowLockMigration${Date.now()}`,+ up: async () => {+ migrationStarted = true+ await migrationGate+ },+ })+ const runnerPromise = backgroundMigrationRunner.run({ log: app!.log, migrations: [SlowMigration] })+ let gotLock = false+ let lockWaiter: Promise<void> | null = null+ try {+ await waitFor(() => migrationStarted, 2_000)+ lockWaiter = migrationHelpers.withBackgroundMigrationLock({+ log: app!.log,+ migration: SlowMigration,+ fn: async () => { gotLock = true },+ })+ await new Promise(resolve => setTimeout(resolve, 100))+ expect(gotLock).toBe(false)++ releaseMigration()+ await runnerPromise+ await lockWaiter+ expect(gotLock).toBe(true)+ }+ finally {+ releaseMigration()+ await Promise.allSettled([runnerPromise, lockWaiter ?? Promise.resolve()])+ }+ })++ it('acquires immediately when no migration is in-flight', async () => {+ const Any = makeTestMigration({ name: `IdleLock${Date.now()}` })+ const start = Date.now()+ await migrationHelpers.withBackgroundMigrationLock({+ log: app!.log,+ migration: Any,+ fn: async () => { /* no-op */ },+ })+ const elapsed = Date.now() - start+ expect(elapsed).toBeLessThan(500)+ })+})+function makeTestMigration({name: migrationName,up: upFn = async () => { /* no-op */ },@@ -203,6 +263,14 @@ function makeTestMigration({}}+async function waitFor(predicate: () => boolean, timeoutMs: number): Promise<void> {+ const started = Date.now()+ while (!predicate()) {+ if (Date.now() - started > timeoutMs) throw new Error(`waitFor timed out after ${timeoutMs}ms`)+ await new Promise(resolve => setTimeout(resolve, 25))+ }+}+async function selectByName(name: string): Promise<{ name: string }[]> {const rows = await databaseConnection().query(`SELECT "name" FROM "${BACKGROUND_MIGRATIONS_TABLE}" WHERE "executed_at" IS NOT NULL`,