Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(migrations): dedicated POSTGRES_MIGRATION_URL for background migrations (#16531)

Amr Elmohamady048bee74dde3
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.md
index 1bfaa9f0458..e06262897e9 100644
--- a/brain/knowledge/data-storage-observability/database.md
+++ b/brain/knowledge/data-storage-observability/database.md
@@ -293,11 +293,15 @@ A background migration that fails every retry stops the app from booting. The gu
- **Same-name migration in both folders would run twice.** The two history tables are independent — a rename or folder move without a seed-row migration double-executes on customers that already ran it in the old track.
- **Background migrations run on API pods only.** Workers don't register system-job handlers. Deployments with zero API pods will never finish backfills; not a supported topology, worth flagging if it comes up.
- **`check-migrations` and `db-migration` build their database with `src/migrate.ts`, not `typeorm migration:run`.** The CLI runs only the blocking track, so it reports every entity index that a background migration builds as drift. `migrate.ts` runs `runMigrationsWithCatchup` and then the background track, the boot order, through the same `distributedLock` as boot. It ends with `process.exit`: on memory Redis the lock starts redis-memory-server, whose open connection would keep the step from exiting, and the library's killer process stops `redis-server` once the parent is gone.
+- **A role-level `statement_timeout` on the main app role kills background migrations, and runtime `SET` can't substitute through a pooler.** Managed Postgres (DO, Supabase) caps the app role, which cancels every long `CREATE/DROP/REINDEX CONCURRENTLY` and chunked backfill. The retry loop leaves 0-byte INVALID index artifacts behind that `IF NOT EXISTS` on the next deploy silently skips (that's how `idx_run_project_id_environment_created_status_archived_at` and `idx_todo_run_id` sat on Cloud `flow_run`/`todo` as orphans). Runtime `SET statement_timeout = 0` is unsound on a transaction-mode pooler: the `SET` lands on one backend, the DDL on another that still has the cap, and the `SET` leaks to normal app traffic on the first backend (empirically verified against DO's pool). Fix: dedicated role (`ALTER ROLE ap_migrator SET statement_timeout = 0`) + `AP_POSTGRES_MIGRATION_URL` pointing the runner at a pool for that role. `migrationDatabaseConnection()` resolves this; unset or non-Postgres falls back to the main DS. Full SQL in `docs/install/reference/breaking-changes-unreleased.mdx`.
+- **The dedicated migration pool uses SESSION pooling, not transaction.** Transaction mode releases the backend between statements and silently breaks `pg_advisory_lock` (session-scoped) and leaves temp tables stranded on a backend serving other clients. Session mode's cost is negligible at pool size 3; apply it to the migration pool only, the main app pool stays transaction.
+- **Every new `AppSystemProp` needs an entry in `systemPropValidators`.** The mapped type is exhaustive — a missing entry fails the api build with TS2741 at `system-validator.ts:95`. Mirror the sibling (`stringValidator` for URL/host, `numberValidator` for counts/ms, `booleanValidator` for flags).
## Key files
- `packages/server/api/src/app/database/` — connection, migrations, helpers.
-- `packages/server/api/src/app/database/postgres-connection.ts` — `getMigrations()`, `getBackgroundMigrations()`.
+- `packages/server/api/src/app/database/postgres-connection.ts` — `getMigrations()`, `getBackgroundMigrations()`, `createPostgresMigrationDataSource()`.
+- `packages/server/api/src/app/database/database-connection.ts` — `databaseConnection()`, `migrationDatabaseConnection()`.
- `packages/server/api/src/app/database/migration.ts` — the `Migration` interface.
- `packages/server/api/src/app/database/background-migration.ts` — the `BackgroundMigration` interface.
- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `withBackgroundMigrationLock`, `createIndexConcurrently`, `dropIndexConcurrently`.
diff --git a/docs/install/reference/breaking-changes-unreleased.mdx b/docs/install/reference/breaking-changes-unreleased.mdx
index 616121ee9eb..93dff07bb9c 100644
--- a/docs/install/reference/breaking-changes-unreleased.mdx
+++ b/docs/install/reference/breaking-changes-unreleased.mdx
@@ -8,6 +8,34 @@ Entries here move to [Breaking Changes](/install/reference/breaking-changes) aut
## What has changed?
+#### Background migrations need a dedicated Postgres role when a `statement_timeout` applies
+
+The app's background migration track (slow data backfills, `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`) runs through whichever database connection the app itself uses. When a `statement_timeout` applies to that connection, every long DDL statement a background migration issues is cancelled at that cap and never completes. The retry loop leaves `INVALID` index artifacts behind that an `IF NOT EXISTS` on the next deploy silently skips.
+
+The app now reads `AP_POSTGRES_MIGRATION_URL` and, when set, runs background migrations through that connection instead of the main one. The connection string should point at a Postgres role whose `statement_timeout` is `0`.
+
+A `SET statement_timeout = 0` at runtime from the app can't substitute: in transaction-pooling mode (the default on managed poolers) the `SET` lands on one backend and the actual DDL on another, so the cap stays in force on the DDL and the `SET` leaks to normal app traffic on the first backend. The role-level default stays with the backend from its startup and never leaks.
+
+#### What you need to do
+
+Nothing if no `statement_timeout` applies to the app's connection (`SHOW statement_timeout;` as the app returns `0` — the Postgres default and the common self-host case). Background migrations keep running on the main connection.
+
+If a `statement_timeout` applies, provision a dedicated migration role and tell the app about it:
+
+```sql
+CREATE ROLE ap_migrator WITH LOGIN PASSWORD '<generated>';
+ALTER ROLE ap_migrator SET statement_timeout = 0;
+
+-- Make ap_migrator a member of the role that owns the app's tables (usually
+-- the same role your AP_POSTGRES_URL connects as — call it <app_role>).
+-- Membership gives ap_migrator ownership privileges on every table <app_role>
+-- owns, including CREATE INDEX / DROP INDEX / REINDEX, which GRANT ALL does
+-- NOT. Covers future tables too, since they inherit the same ownership.
+GRANT <app_role> TO ap_migrator;
+```
+
+Then set `AP_POSTGRES_MIGRATION_URL=postgres://ap_migrator:<password>@<host>:<port>/<database>` on the API deployment. On managed Postgres with a connection pooler, create a small pool (size 3) scoped to this role and use that pool's URL.
+
#### Code steps run on Deno by default
A code step without an explicit `useDeno` flag now executes in the Deno runtime instead of the legacy Node sandbox — effectively migrating every existing flow automatically, with no data rewrite. The legacy executors are still shipped: setting `useDeno: false` on a step keeps it on the previous Node/v8-isolate path, and the admin `revert-from-deno` endpoint writes exactly that flag, so individual flows can still be opted out. Failed code steps are now also logged as structured `Code step failed` warnings (flow run, flow version, project, step name, runtime, and the error message) so they can be found and analyzed in centralized logs.
diff --git a/docs/install/reference/environment-variables.mdx b/docs/install/reference/environment-variables.mdx
index 3494d724274..11ae75de54f 100644
--- a/docs/install/reference/environment-variables.mdx
+++ b/docs/install/reference/environment-variables.mdx
@@ -169,6 +169,7 @@ embedded PGLite database. Set the connection with discrete fields or a single
| `AP_POSTGRES_SSL_CA` | SSL certificate (CA) used to connect to PostgreSQL. | `None` |
| `AP_POSTGRES_POOL_SIZE` | Maximum number of clients in the connection pool. | `None` |
| `AP_POSTGRES_IDLE_TIMEOUT_MS` | Idle timeout (ms) for pooled PostgreSQL connections. | `300000` |
+| `AP_POSTGRES_MIGRATION_URL` | Separate connection string for background migrations only (slow backfills, `CREATE INDEX CONCURRENTLY`). Set this when the app role has `statement_timeout` configured — the migration role on the other end should have `statement_timeout = 0`. If unset, background migrations use the main connection. | `None` |
---
diff --git a/packages/server/api/src/app/database/background-migration-runner.ts b/packages/server/api/src/app/database/background-migration-runner.ts
index feab3761d04..e27365eb89c 100644
--- a/packages/server/api/src/app/database/background-migration-runner.ts
+++ b/packages/server/api/src/app/database/background-migration-runner.ts
@@ -3,7 +3,7 @@ import { wideEvent } from '@activepieces/server-utils'
import { FastifyBaseLogger } from 'fastify'
import { DataSource } from 'typeorm'
import { BackgroundMigration } from './background-migration'
-import { databaseConnection } from './database-connection'
+import { databaseConnection, migrationDatabaseConnection } from './database-connection'
import { BackgroundMigrationNotCompleteError } from './migration-helpers'
import { getBackgroundMigrations } from './postgres-connection'
import { distributedLock } from './redis-connections'
@@ -132,8 +132,9 @@ async function executeMigrationBody({
log: FastifyBaseLogger
dataSource: DataSource
}): Promise<void> {
- const queryRunner = dataSource.createQueryRunner()
try {
+ const executionDs = await migrationDatabaseConnection()
+ const queryRunner = executionDs.createQueryRunner()
try {
await migration.up(queryRunner)
}
diff --git a/packages/server/api/src/app/database/database-connection.ts b/packages/server/api/src/app/database/database-connection.ts
index 3b8beb5f410..00be5e986cb 100644
--- a/packages/server/api/src/app/database/database-connection.ts
+++ b/packages/server/api/src/app/database/database-connection.ts
@@ -70,7 +70,7 @@ import { WaitpointEntity } from '../waitpoints/waitpoint-entity'
import { WaitpointSignalEntity } from '../waitpoints/waitpoint-signal-entity'
import { DatabaseType } from './database-type'
import { createPGliteDataSource } from './pglite-connection'
-import { createPostgresDataSource } from './postgres-connection'
+import { createPostgresDataSource, createPostgresMigrationDataSource } from './postgres-connection'
const databaseType = system.get(AppSystemProp.DB_TYPE)
@@ -180,6 +180,32 @@ export const databaseConnection = (): DataSource => {
return ds
}
+export const migrationDatabaseConnection = async (): Promise<DataSource> => {
+ const persisted = getPersistedMigrationConnection()
+ if (!isNil(persisted)) {
+ return persisted
+ }
+ if (databaseType !== DatabaseType.POSTGRES) {
+ return databaseConnection()
+ }
+ const dedicated = createPostgresMigrationDataSource()
+ if (isNil(dedicated)) {
+ return databaseConnection()
+ }
+ await dedicated.initialize()
+ setPersistedMigrationConnection(dedicated)
+ return dedicated
+}
+
export function resetDatabaseConnection(): void {
setPersistedConnection(null)
+ setPersistedMigrationConnection(null)
+}
+
+const MIGRATION_DB_GLOBAL_KEY = '__AP_DB_MIGRATION_CONNECTION__'
+function getPersistedMigrationConnection(): DataSource | null {
+ return ((globalThis as Record<string, unknown>)[MIGRATION_DB_GLOBAL_KEY] as DataSource) ?? null
+}
+function setPersistedMigrationConnection(ds: DataSource | null): void {
+ (globalThis as Record<string, unknown>)[MIGRATION_DB_GLOBAL_KEY] = ds
}
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index 77b768b7049..ec14f497383 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -988,6 +988,20 @@ export const createPostgresDataSource = (): DataSource => {
})
}
+export const createPostgresMigrationDataSource = (): DataSource | null => {
+ const url = system.get(AppSystemProp.POSTGRES_MIGRATION_URL)
+ if (isNil(url)) return null
+ return new DataSource({
+ type: 'postgres',
+ url,
+ ssl: getSslConfig(),
+ poolSize: 2,
+ migrationsRun: false,
+ synchronize: false,
+ ...commonProperties,
+ })
+}
+
type MigrationConfig = {
migrationsRun?: boolean
migrationsTransactionMode?: 'all' | 'none' | 'each'
diff --git a/packages/server/api/src/app/helper/system-validator.ts b/packages/server/api/src/app/helper/system-validator.ts
index cf257d55e4c..36a719a97e9 100644
--- a/packages/server/api/src/app/helper/system-validator.ts
+++ b/packages/server/api/src/app/helper/system-validator.ts
@@ -165,6 +165,7 @@ const systemPropValidators: {
[AppSystemProp.POSTGRES_PORT]: numberValidator,
[AppSystemProp.POSTGRES_SSL_CA]: stringValidator,
[AppSystemProp.POSTGRES_URL]: stringValidator,
+ [AppSystemProp.POSTGRES_MIGRATION_URL]: stringValidator,
[AppSystemProp.POSTGRES_USERNAME]: stringValidator,
[AppSystemProp.POSTGRES_USE_SSL]: booleanValidator,
[AppSystemProp.POSTGRES_POOL_SIZE]: numberValidator,
diff --git a/packages/server/api/src/app/helper/system/system-props.ts b/packages/server/api/src/app/helper/system/system-props.ts
index 284a132dc9e..dc4eed956b1 100644
--- a/packages/server/api/src/app/helper/system/system-props.ts
+++ b/packages/server/api/src/app/helper/system/system-props.ts
@@ -77,6 +77,7 @@ export enum AppSystemProp {
POSTGRES_DATABASE = 'POSTGRES_DATABASE',
POSTGRES_HOST = 'POSTGRES_HOST',
POSTGRES_IDLE_TIMEOUT_MS = 'POSTGRES_IDLE_TIMEOUT_MS',
+ POSTGRES_MIGRATION_URL = 'POSTGRES_MIGRATION_URL',
POSTGRES_PASSWORD = 'POSTGRES_PASSWORD',
POSTGRES_POOL_SIZE = 'POSTGRES_POOL_SIZE',
POSTGRES_PORT = 'POSTGRES_PORT',