AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(migrations): dedicated POSTGRES_MIGRATION_URL for background migrations (#16531)
Amr Elmohamady048bee74dde3
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.mdindex 1bfaa9f0458..e06262897e9 100644--- a/brain/knowledge/data-storage-observability/database.md+++ b/brain/knowledge/data-storage-observability/database.md@@ -293,11 +293,15 @@ A background migration that fails every retry stops the app from booting. The gu- **Same-name migration in both folders would run twice.** The two history tables are independent — a rename or folder move without a seed-row migration double-executes on customers that already ran it in the old track.- **Background migrations run on API pods only.** Workers don't register system-job handlers. Deployments with zero API pods will never finish backfills; not a supported topology, worth flagging if it comes up.- **`check-migrations` and `db-migration` build their database with `src/migrate.ts`, not `typeorm migration:run`.** The CLI runs only the blocking track, so it reports every entity index that a background migration builds as drift. `migrate.ts` runs `runMigrationsWithCatchup` and then the background track, the boot order, through the same `distributedLock` as boot. It ends with `process.exit`: on memory Redis the lock starts redis-memory-server, whose open connection would keep the step from exiting, and the library's killer process stops `redis-server` once the parent is gone.+- **A role-level `statement_timeout` on the main app role kills background migrations, and runtime `SET` can't substitute through a pooler.** Managed Postgres (DO, Supabase) caps the app role, which cancels every long `CREATE/DROP/REINDEX CONCURRENTLY` and chunked backfill. The retry loop leaves 0-byte INVALID index artifacts behind that `IF NOT EXISTS` on the next deploy silently skips (that's how `idx_run_project_id_environment_created_status_archived_at` and `idx_todo_run_id` sat on Cloud `flow_run`/`todo` as orphans). Runtime `SET statement_timeout = 0` is unsound on a transaction-mode pooler: the `SET` lands on one backend, the DDL on another that still has the cap, and the `SET` leaks to normal app traffic on the first backend (empirically verified against DO's pool). Fix: dedicated role (`ALTER ROLE ap_migrator SET statement_timeout = 0`) + `AP_POSTGRES_MIGRATION_URL` pointing the runner at a pool for that role. `migrationDatabaseConnection()` resolves this; unset or non-Postgres falls back to the main DS. Full SQL in `docs/install/reference/breaking-changes-unreleased.mdx`.+- **The dedicated migration pool uses SESSION pooling, not transaction.** Transaction mode releases the backend between statements and silently breaks `pg_advisory_lock` (session-scoped) and leaves temp tables stranded on a backend serving other clients. Session mode's cost is negligible at pool size 3; apply it to the migration pool only, the main app pool stays transaction.+- **Every new `AppSystemProp` needs an entry in `systemPropValidators`.** The mapped type is exhaustive — a missing entry fails the api build with TS2741 at `system-validator.ts:95`. Mirror the sibling (`stringValidator` for URL/host, `numberValidator` for counts/ms, `booleanValidator` for flags).## Key files- `packages/server/api/src/app/database/` — connection, migrations, helpers.-- `packages/server/api/src/app/database/postgres-connection.ts` — `getMigrations()`, `getBackgroundMigrations()`.+- `packages/server/api/src/app/database/postgres-connection.ts` — `getMigrations()`, `getBackgroundMigrations()`, `createPostgresMigrationDataSource()`.+- `packages/server/api/src/app/database/database-connection.ts` — `databaseConnection()`, `migrationDatabaseConnection()`.- `packages/server/api/src/app/database/migration.ts` — the `Migration` interface.- `packages/server/api/src/app/database/background-migration.ts` — the `BackgroundMigration` interface.- `packages/server/api/src/app/database/migration-helpers.ts` — `assertBackgroundMigrationComplete`, `withBackgroundMigrationLock`, `createIndexConcurrently`, `dropIndexConcurrently`.diff --git a/docs/install/reference/breaking-changes-unreleased.mdx b/docs/install/reference/breaking-changes-unreleased.mdxindex 616121ee9eb..93dff07bb9c 100644--- a/docs/install/reference/breaking-changes-unreleased.mdx+++ b/docs/install/reference/breaking-changes-unreleased.mdx@@ -8,6 +8,34 @@ Entries here move to [Breaking Changes](/install/reference/breaking-changes) aut## What has changed?+#### Background migrations need a dedicated Postgres role when a `statement_timeout` applies++The app's background migration track (slow data backfills, `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`) runs through whichever database connection the app itself uses. When a `statement_timeout` applies to that connection, every long DDL statement a background migration issues is cancelled at that cap and never completes. The retry loop leaves `INVALID` index artifacts behind that an `IF NOT EXISTS` on the next deploy silently skips.++The app now reads `AP_POSTGRES_MIGRATION_URL` and, when set, runs background migrations through that connection instead of the main one. The connection string should point at a Postgres role whose `statement_timeout` is `0`.++A `SET statement_timeout = 0` at runtime from the app can't substitute: in transaction-pooling mode (the default on managed poolers) the `SET` lands on one backend and the actual DDL on another, so the cap stays in force on the DDL and the `SET` leaks to normal app traffic on the first backend. The role-level default stays with the backend from its startup and never leaks.++#### What you need to do++Nothing if no `statement_timeout` applies to the app's connection (`SHOW statement_timeout;` as the app returns `0` — the Postgres default and the common self-host case). Background migrations keep running on the main connection.++If a `statement_timeout` applies, provision a dedicated migration role and tell the app about it:++```sql+CREATE ROLE ap_migrator WITH LOGIN PASSWORD '<generated>';+ALTER ROLE ap_migrator SET statement_timeout = 0;++-- Make ap_migrator a member of the role that owns the app's tables (usually+-- the same role your AP_POSTGRES_URL connects as — call it <app_role>).+-- Membership gives ap_migrator ownership privileges on every table <app_role>+-- owns, including CREATE INDEX / DROP INDEX / REINDEX, which GRANT ALL does+-- NOT. Covers future tables too, since they inherit the same ownership.+GRANT <app_role> TO ap_migrator;+```++Then set `AP_POSTGRES_MIGRATION_URL=postgres://ap_migrator:<password>@<host>:<port>/<database>` on the API deployment. On managed Postgres with a connection pooler, create a small pool (size 3) scoped to this role and use that pool's URL.+#### Code steps run on Deno by defaultA code step without an explicit `useDeno` flag now executes in the Deno runtime instead of the legacy Node sandbox — effectively migrating every existing flow automatically, with no data rewrite. The legacy executors are still shipped: setting `useDeno: false` on a step keeps it on the previous Node/v8-isolate path, and the admin `revert-from-deno` endpoint writes exactly that flag, so individual flows can still be opted out. Failed code steps are now also logged as structured `Code step failed` warnings (flow run, flow version, project, step name, runtime, and the error message) so they can be found and analyzed in centralized logs.diff --git a/docs/install/reference/environment-variables.mdx b/docs/install/reference/environment-variables.mdxindex 3494d724274..11ae75de54f 100644--- a/docs/install/reference/environment-variables.mdx+++ b/docs/install/reference/environment-variables.mdx@@ -169,6 +169,7 @@ embedded PGLite database. Set the connection with discrete fields or a single| `AP_POSTGRES_SSL_CA` | SSL certificate (CA) used to connect to PostgreSQL. | `None` || `AP_POSTGRES_POOL_SIZE` | Maximum number of clients in the connection pool. | `None` || `AP_POSTGRES_IDLE_TIMEOUT_MS` | Idle timeout (ms) for pooled PostgreSQL connections. | `300000` |+| `AP_POSTGRES_MIGRATION_URL` | Separate connection string for background migrations only (slow backfills, `CREATE INDEX CONCURRENTLY`). Set this when the app role has `statement_timeout` configured — the migration role on the other end should have `statement_timeout = 0`. If unset, background migrations use the main connection. | `None` |---diff --git a/packages/server/api/src/app/database/background-migration-runner.ts b/packages/server/api/src/app/database/background-migration-runner.tsindex feab3761d04..e27365eb89c 100644--- a/packages/server/api/src/app/database/background-migration-runner.ts+++ b/packages/server/api/src/app/database/background-migration-runner.ts@@ -3,7 +3,7 @@ import { wideEvent } from '@activepieces/server-utils'import { FastifyBaseLogger } from 'fastify'import { DataSource } from 'typeorm'import { BackgroundMigration } from './background-migration'-import { databaseConnection } from './database-connection'+import { databaseConnection, migrationDatabaseConnection } from './database-connection'import { BackgroundMigrationNotCompleteError } from './migration-helpers'import { getBackgroundMigrations } from './postgres-connection'import { distributedLock } from './redis-connections'@@ -132,8 +132,9 @@ async function executeMigrationBody({log: FastifyBaseLoggerdataSource: DataSource}): Promise<void> {- const queryRunner = dataSource.createQueryRunner()try {+ const executionDs = await migrationDatabaseConnection()+ const queryRunner = executionDs.createQueryRunner()try {await migration.up(queryRunner)}diff --git a/packages/server/api/src/app/database/database-connection.ts b/packages/server/api/src/app/database/database-connection.tsindex 3b8beb5f410..00be5e986cb 100644--- a/packages/server/api/src/app/database/database-connection.ts+++ b/packages/server/api/src/app/database/database-connection.ts@@ -70,7 +70,7 @@ import { WaitpointEntity } from '../waitpoints/waitpoint-entity'import { WaitpointSignalEntity } from '../waitpoints/waitpoint-signal-entity'import { DatabaseType } from './database-type'import { createPGliteDataSource } from './pglite-connection'-import { createPostgresDataSource } from './postgres-connection'+import { createPostgresDataSource, createPostgresMigrationDataSource } from './postgres-connection'const databaseType = system.get(AppSystemProp.DB_TYPE)@@ -180,6 +180,32 @@ export const databaseConnection = (): DataSource => {return ds}+export const migrationDatabaseConnection = async (): Promise<DataSource> => {+ const persisted = getPersistedMigrationConnection()+ if (!isNil(persisted)) {+ return persisted+ }+ if (databaseType !== DatabaseType.POSTGRES) {+ return databaseConnection()+ }+ const dedicated = createPostgresMigrationDataSource()+ if (isNil(dedicated)) {+ return databaseConnection()+ }+ await dedicated.initialize()+ setPersistedMigrationConnection(dedicated)+ return dedicated+}+export function resetDatabaseConnection(): void {setPersistedConnection(null)+ setPersistedMigrationConnection(null)+}++const MIGRATION_DB_GLOBAL_KEY = '__AP_DB_MIGRATION_CONNECTION__'+function getPersistedMigrationConnection(): DataSource | null {+ return ((globalThis as Record<string, unknown>)[MIGRATION_DB_GLOBAL_KEY] as DataSource) ?? null+}+function setPersistedMigrationConnection(ds: DataSource | null): void {+ (globalThis as Record<string, unknown>)[MIGRATION_DB_GLOBAL_KEY] = ds}diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.tsindex 77b768b7049..ec14f497383 100644--- a/packages/server/api/src/app/database/postgres-connection.ts+++ b/packages/server/api/src/app/database/postgres-connection.ts@@ -988,6 +988,20 @@ export const createPostgresDataSource = (): DataSource => {})}+export const createPostgresMigrationDataSource = (): DataSource | null => {+ const url = system.get(AppSystemProp.POSTGRES_MIGRATION_URL)+ if (isNil(url)) return null+ return new DataSource({+ type: 'postgres',+ url,+ ssl: getSslConfig(),+ poolSize: 2,+ migrationsRun: false,+ synchronize: false,+ ...commonProperties,+ })+}+type MigrationConfig = {migrationsRun?: booleanmigrationsTransactionMode?: 'all' | 'none' | 'each'diff --git a/packages/server/api/src/app/helper/system-validator.ts b/packages/server/api/src/app/helper/system-validator.tsindex cf257d55e4c..36a719a97e9 100644--- a/packages/server/api/src/app/helper/system-validator.ts+++ b/packages/server/api/src/app/helper/system-validator.ts@@ -165,6 +165,7 @@ const systemPropValidators: {[AppSystemProp.POSTGRES_PORT]: numberValidator,[AppSystemProp.POSTGRES_SSL_CA]: stringValidator,[AppSystemProp.POSTGRES_URL]: stringValidator,+ [AppSystemProp.POSTGRES_MIGRATION_URL]: stringValidator,[AppSystemProp.POSTGRES_USERNAME]: stringValidator,[AppSystemProp.POSTGRES_USE_SSL]: booleanValidator,[AppSystemProp.POSTGRES_POOL_SIZE]: numberValidator,diff --git a/packages/server/api/src/app/helper/system/system-props.ts b/packages/server/api/src/app/helper/system/system-props.tsindex 284a132dc9e..dc4eed956b1 100644--- a/packages/server/api/src/app/helper/system/system-props.ts+++ b/packages/server/api/src/app/helper/system/system-props.ts@@ -77,6 +77,7 @@ export enum AppSystemProp {POSTGRES_DATABASE = 'POSTGRES_DATABASE',POSTGRES_HOST = 'POSTGRES_HOST',POSTGRES_IDLE_TIMEOUT_MS = 'POSTGRES_IDLE_TIMEOUT_MS',+ POSTGRES_MIGRATION_URL = 'POSTGRES_MIGRATION_URL',POSTGRES_PASSWORD = 'POSTGRES_PASSWORD',POSTGRES_POOL_SIZE = 'POSTGRES_POOL_SIZE',POSTGRES_PORT = 'POSTGRES_PORT',