Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(migrations): build the background track before the schema drift check (#16439)

Othman Abu Ajamiehbd9dbc602fb5
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.md
index 236dffb1cc9..d89bcb351d8 100644
--- a/brain/knowledge/data-storage-observability/database.md
+++ b/brain/knowledge/data-storage-observability/database.md
@@ -260,6 +260,7 @@ A background migration that fails every retry stops the app from booting. The gu
- **A brand-new hot-path index in the same release as the code that needs it can leave the app on seq scans while background CIC builds.** Only applies when there's *no* prior covering index for the query. Fixes: (a) ship the CIC in a prior release so it's built by the time the code lands (verify via `pendingCount` on `/v1/health/system`), or (b) put the CIC in the blocking track so the index is present when the app starts serving. Reshapes (replacing existing indexes) don't hit this — the old index keeps queries fast while new one builds.
- **Same-name migration in both folders would run twice.** The two history tables are independent — a rename or folder move without a seed-row migration double-executes on customers that already ran it in the old track.
- **Background migrations run on API pods only.** Workers don't register system-job handlers. Deployments with zero API pods will never finish backfills; not a supported topology, worth flagging if it comes up.
+- **`check-migrations` and `db-migration` build their database with `src/migrate.ts`, not `typeorm migration:run`.** The CLI runs only the blocking track, so it reports every entity index that a background migration builds as drift. `migrate.ts` runs `runMigrationsWithCatchup` and then the background track, the boot order, through the same `distributedLock` as boot. It ends with `process.exit`: on memory Redis the lock starts redis-memory-server, whose open connection would keep the step from exiting, and the library's killer process stops `redis-server` once the parent is gone.
## Key files
@@ -269,6 +270,7 @@ A background migration that fails every retry stops the app from booting. The gu
- `packages/server/api/src/app/database/background-migration.ts` — the `BackgroundMigration` interface.
- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `createIndexConcurrently`.
- `packages/server/api/src/app/database/background-migration-runner.ts` — the BullMQ handler.
+- `packages/server/api/src/migrate.ts` — both tracks in one process, for `check-migrations` and `db-migration`.
- `tools/scripts/check-migration-rollback.ts` — the CI check.
## Related pages
diff --git a/packages/server/api/package.json b/packages/server/api/package.json
index 0c11b3ede75..66a7ad1222e 100644
--- a/packages/server/api/package.json
+++ b/packages/server/api/package.json
@@ -150,9 +150,9 @@
"test-ce": "export $(cat .env.tests | xargs) && AP_EDITION=ce npm run test-ce-command",
"test-ee": "export $(cat .env.tests | xargs) && AP_EDITION=ee npm run test-ee-command",
"test-cloud": "export $(cat .env.tests | xargs) && AP_EDITION=cloud npm run test-cloud-command",
- "check-migrations": "echo '🔍 Checking for schema changes...' && export $(cat .env.tests | xargs) && export AP_DEV_PIECES='' && export AP_ENVIRONMENT=dev && AP_EDITION=ce && ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json node_modules/typeorm/cli.js migration:run -d src/app/database/migration-data-source.ts > /dev/null 2>&1 && echo 'Checking for schema drift...' && ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json node_modules/typeorm/cli.js migration:generate -p -d src/app/database/migration-data-source.ts src/app/database/migration/postgres/check --dryrun --check || (echo '❌ ERROR: Schema changes detected without migration!' && exit 1) && echo '✅ No missing migrations detected'",
+ "check-migrations": "echo '🔍 Checking for schema changes...' && export $(cat .env.tests | xargs) && export AP_DEV_PIECES='' && export AP_ENVIRONMENT=dev && AP_EDITION=ce && ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json src/migrate.ts && echo 'Checking for schema drift...' && (ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json node_modules/typeorm/cli.js migration:generate -p -d src/app/database/migration-data-source.ts src/app/database/migration/postgres/check --dryrun --check || (echo '❌ ERROR: Schema changes detected without migration!' && exit 1)) && echo '✅ No missing migrations detected'",
"db": "ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json node_modules/typeorm/cli.js",
- "db-migration": "export $(cat .env.tests | xargs) && export AP_DEV_PIECES='' && export AP_ENVIRONMENT=dev && export AP_EDITION=ce && ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json node_modules/typeorm/cli.js migration:generate -p -d src/app/database/migration-data-source.ts",
+ "db-migration": "export $(cat .env.tests | xargs) && export AP_DEV_PIECES='' && export AP_ENVIRONMENT=dev && export AP_EDITION=ce && ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json src/migrate.ts && ts-node --transpile-only -r tsconfig-paths/register -P tsconfig.app.json node_modules/typeorm/cli.js migration:generate -p -d src/app/database/migration-data-source.ts",
"test:docker": "UID=\"$(id -u)\" GID=\"$(id -g)\" docker compose --profile full -f docker-compose.test.yml up --exit-code-from app --attach app",
"rollback": "node dist/src/rollback.js"
}
diff --git a/packages/server/api/src/migrate.ts b/packages/server/api/src/migrate.ts
new file mode 100644
index 00000000000..41f81af553d
--- /dev/null
+++ b/packages/server/api/src/migrate.ts
@@ -0,0 +1,20 @@
+import { backgroundMigrationRunner } from './app/database/background-migration-runner'
+import { databaseConnection } from './app/database/database-connection'
+import { system } from './app/helper/system/system'
+
+async function main(): Promise<void> {
+ process.env.TZ = 'UTC'
+ const log = system.globalLogger()
+ const dataSource = databaseConnection()
+ await dataSource.initialize()
+ await backgroundMigrationRunner.runMigrationsWithCatchup({ dataSource, log })
+ await backgroundMigrationRunner.run({ dataSource, log })
+ await dataSource.destroy()
+}
+
+main()
+ .then(() => process.exit(0))
+ .catch((error: unknown) => {
+ system.globalLogger().error({ error }, '[migrate] Failed to apply blocking and background migrations')
+ process.exit(1)
+ })