Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(migrations): background migrations report progress live and page on-call when they give up (#17016)

Amr Elmohamadyba5dfa085b2c
diff --git a/brain/knowledge/data-storage-observability/database.md b/brain/knowledge/data-storage-observability/database.md
index 473e349d735..4991f83e90a 100644
--- a/brain/knowledge/data-storage-observability/database.md
+++ b/brain/knowledge/data-storage-observability/database.md
@@ -73,6 +73,8 @@ The guard also prevents the silently dangerous cases (`DROP COLUMN`, `DELETE FRO
Contract DDL and its backfill ship in **different releases**. Same-release attempts fail boot, catchup runs the backfill, retries — this is the intended path.
+**Temporary code a backfill leaves behind (fallback reads, dual writes, legacy columns) gets a Linear ticket for its owner when the migration merges.** The code goes in release N+4 or later (N is the migration's `release` minor) and the schema drop in N+5 or later, always at least one release after the code. Steps in the [migrations playbook](../../../docs/handbook/engineering/playbooks/database-migration.mdx).
+
**Release-notes convention:** when a release ships a background migration whose dependent contract DDL lands in a later release, that earlier release's notes must say something like "self-hosters should let backfill X complete before upgrading further; the app catches up at boot but it may take ~N minutes." Boot catchup handles skip-release correctly, but operators deserve the heads-up so they can plan the upgrade window.
### Interfaces
@@ -296,6 +298,11 @@ A background migration that fails every retry stops the app from booting. The gu
- **Background migrations run on API pods only.** Workers don't register system-job handlers. Deployments with zero API pods will never finish backfills; not a supported topology, worth flagging if it comes up.
- **`check-migrations` and `db-migration` build their database with `src/migrate.ts`, not `typeorm migration:run`.** The CLI runs only the blocking track, so it reports every entity index that a background migration builds as drift. `migrate.ts` runs `runMigrationsWithCatchup` and then the background track, the boot order, through the same `distributedLock` as boot. It ends with `process.exit`: on memory Redis the lock starts redis-memory-server, whose open connection would keep the step from exiting, and the library's killer process stops `redis-server` once the parent is gone.
- **A role-level `statement_timeout` on the main app role kills background migrations, and runtime `SET` can't substitute through a pooler.** Managed Postgres (DO, Supabase) caps the app role, which cancels every long `CREATE/DROP/REINDEX CONCURRENTLY` and chunked backfill. The retry loop leaves 0-byte INVALID index artifacts behind that `IF NOT EXISTS` on the next deploy silently skips (that's how `idx_run_project_id_environment_created_status_archived_at` and `idx_todo_run_id` sat on Cloud `flow_run`/`todo` as orphans). Runtime `SET statement_timeout = 0` is unsound on a transaction-mode pooler: the `SET` lands on one backend, the DDL on another that still has the cap, and the `SET` leaks to normal app traffic on the first backend (empirically verified against DO's pool). Fix: dedicated role (`ALTER ROLE ap_migrator SET statement_timeout = 0`) + `AP_POSTGRES_MIGRATION_URL` pointing the runner at a pool for that role. `migrationDatabaseConnection()` resolves this; unset or non-Postgres falls back to the main DS. Full SQL in `docs/install/reference/breaking-changes-unreleased.mdx`.
+- **On Kamal, `AP_POSTGRES_MIGRATION_URL` must also be listed under `env.secret` in the API role config.** A key in `.kamal/secrets` alone never reaches the container. Cloud had it in secrets from Oct 18 2026 but not in `app.yml`/`app-canary.yml` until Oct 28, so every run went through `ap` and was cancelled at exactly 60s. Verify with `printenv` inside a running API container, not by reading the secrets file.
+- **Each failed `REINDEX CONCURRENTLY` leaves another INVALID `<index>_ccnew<N>`.** `createIndexConcurrently` retries an INVALID index with `REINDEX CONCURRENTLY`, so every failed retry adds one (Cloud `flow_run` collected 27). Once the real index is valid, drop them with `DROP INDEX CONCURRENTLY`.
+- **A deploy kills a running background migration.** Postgres cancels the statement when the pod goes, and BullMQ fails the job with "job stalled more than allowable limit", which does not retry. The first new pod re-queues it while old pods are still alive, so a migration that takes hours needs a deploy freeze.
+- **Background migrations report progress as their own `background-migration.progress` events.** `phase` is `started`, `running` (a heartbeat every 5 min that carries `pg_stat_progress_create_index`), `completed` or `failed`. The job's `system-job.execute` wide event lands only when the whole run ends. When the job gives up (attempts exhausted or stalled) it pages on-call with `SYSTEM_JOB_FAILED` through `AP_PAGE_ONCALL_WEBHOOK`.
+- **The read-only `ap_readonly` connection is a replica that stops replaying during the nightly `pg_dump`.** On Oct 28 2026 it was over 6h behind. It also cannot see the primary's locks or other roles' sessions. Check `pg_is_in_recovery()` and `now() - pg_last_xact_replay_timestamp()` first, and look at a live migration on the primary as `ap_migrator`: `pg_stat_activity` plus `pg_stat_progress_create_index`.
- **The dedicated migration pool uses SESSION pooling, not transaction.** Transaction mode releases the backend between statements and silently breaks `pg_advisory_lock` (session-scoped) and leaves temp tables stranded on a backend serving other clients. Session mode's cost is negligible at pool size 3; apply it to the migration pool only, the main app pool stays transaction.
- **Every new `AppSystemProp` needs an entry in `systemPropValidators`.** The mapped type is exhaustive — a missing entry fails the api build with TS2741 at `system-validator.ts:95`. Mirror the sibling (`stringValidator` for URL/host, `numberValidator` for counts/ms, `booleanValidator` for flags).
diff --git a/brain/knowledge/pieces-engine/pieces.md b/brain/knowledge/pieces-engine/pieces.md
index 702b97837c4..163344cdc08 100644
--- a/brain/knowledge/pieces-engine/pieces.md
+++ b/brain/knowledge/pieces-engine/pieces.md
@@ -30,7 +30,7 @@ The metadata catalog of automation integrations ("pieces") — each a named inte
- **A piece's `displayName` is translated only for `PieceCategory.CORE` pieces.** Never add it to `pathsToValuesToTranslate`: PR 9124 removed it after brand names got translated (Salesforce became "Force de ventes"). `pieceTranslation.pathsToTranslate(categories)` is the one rule, used by `translatePiece` and the CLI generator. Search runs on the translated catalogue, so `fetchLatestPieces` also returns `untranslatedNames` (English name of every piece whose name was translated) and search matches it in both the Fuse piece match and `searchForSuggestion`. Miss the second and the picker gets the piece with no actions or triggers, which hides it (GIT-1995).
- **The single-piece routes return every locale's `i18n` when called without `locale`: that is the cloud registry contract self-hosted sync depends on.** `pieceSyncService` fetches `GET /v1/pieces/:name?version=X` with no locale and stores whatever `i18n` comes back. #16620 stripped it, so every piece version synced from Oct 20 2026 until the fix landed got stored English-only (GIT-2050). The controller maps a missing locale to `locale: 'all'`, which makes `get()` attach every locale via `pieceTranslationService.getAllLocales`. Internal callers pass no locale, which still means English with `i18n` stripped, so they don't pay for the extra query. Never strip `i18n` from a locale-less single-piece response.
-- **`piece_metadata.i18n` is still written but only read as a fallback, until it is dropped.** `pieceMetadataService.create` writes both the column and `piece_translation` in one transaction; `BackfillPieceTranslation` (background) copies existing rows. While the backfill runs, a piece with no `piece_translation` rows falls back to `i18n -> :locale`, so nobody sees English mid-backfill. The follow-up drops the column behind `assertBackgroundMigrationComplete` and deletes that fallback in `pieceTranslationService.getForLocale`. Synced pieces keep the cloud's `id`, so translation rows must use the id that was actually saved, never a freshly generated one.
+- **`piece_metadata.i18n` is still written but only read as a fallback, until it is dropped.** `pieceMetadataService.create` writes both the column and `piece_translation` in one transaction; `BackfillPieceTranslation` (background) copies existing rows. While the backfill runs, a piece with no `piece_translation` rows falls back to `i18n -> :locale`, so nobody sees English mid-backfill. Removal is two releases, per the [migrations playbook](../../../docs/handbook/engineering/playbooks/database-migration.mdx): 0.97.0 or later removes the fallbacks in `pieceTranslationService` and the column write in `create()`, and 0.98.0 or later (always a release after that) drops the column behind `assertBackgroundMigrationComplete`. Cloud finished the backfill on Oct 28 2026. Synced pieces keep the cloud's `id`, so translation rows must use the id that was actually saved, never a freshly generated one.
- **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.
- `DynamicPropertiesContext` tracks loading by property name only, so two in-flight requests for the same property let the first completion clear the flag for both — briefly re-enabling Test Step while the value is still cleared.
- **`POST /v1/pieces/options` cost scales with the whole flow's sample data, not with the property being resolved.** The handler builds an `EXECUTE_PROPERTY` job whose payload carries `sampleData` — and `getSampleDataForFlow` reads *every* step's sample-data `file` row. That payload then crosses Postgres → API → Redis → worker → engine socket, so a big flow pays megabytes to resolve a trivial resolver like the HTTP piece's `authFields`: measured 7.3 MB on a 25-step flow, and a panel open fires three of these in parallel (GIT-1822). The engine only uses `sampleData` to seed step outputs so `{{step.x}}` inside the property's `input` resolves, so callers must use `getSampleDataReferencedBy` and pass the request `input` — which narrows the read to the steps that input names. When narrowing, a referenced `LOOP_ON_ITEMS` step also needs whatever its `items` expression references: `stateFromFlowVersion` derives a loop's `item` by resolving its settings against the context built so far, never from `sampleData[loopName]`. Matching step names as substrings of the serialized input over-approximates, which is the safe direction — a step reference always spells the name literally, in both the property-path and script branches of `props-resolver`.
diff --git a/docs/handbook/engineering/playbooks/database-migration.mdx b/docs/handbook/engineering/playbooks/database-migration.mdx
index 938e11b4d68..b372b29102b 100644
--- a/docs/handbook/engineering/playbooks/database-migration.mdx
+++ b/docs/handbook/engineering/playbooks/database-migration.mdx
@@ -128,3 +128,29 @@ export class AddMyIndex1234567890 implements Migration {
<Tip>
Always test your migrations by running them both up and down to ensure they work as expected.
</Tip>
+
+## Cleaning Up After a Background Migration
+
+A background migration often leaves temporary code behind: a fallback read from the old shape, a dual write, a legacy column or index, special handling until the backfill finishes. Plan its removal when the migration ships, not later.
+
+<Steps>
+ <Step title="Open a Linear ticket when the migration merges">
+ Create a ticket assigned to the migration's owner (the author of the PR that adds it). Name the migration and the target releases in the title, for example "Remove the `piece_metadata.i18n` fallback after `BackfillPieceTranslation` (code 0.97.0, column 0.98.0)".
+
+ In the body, list every piece of code and schema that becomes removable (symbol and file), link the migration PR, and copy in the two steps below. Link the ticket from the migration PR's description.
+ </Step>
+
+ <Step title="Remove the code in release N+4 or later">
+ N is the minor version in the migration's `release` field, so `release = '0.93.0'` means the code goes in `0.97.0` or later. We support the latest five releases, so once N+4 is the newest, every supported release (N to N+4) already ships the backfill. N+3 is too early, because N-1 is still supported then.
+
+ Ship code only. Remove the fallbacks and dual writes, take the field off the entity, and add a blocking migration whose `up()` only calls `migrationHelpers.assertBackgroundMigrationComplete(...)`. That guard makes an older install that skips releases run the backfill at boot before it serves traffic.
+ </Step>
+
+ <Step title="Drop the schema in release N+5 or later">
+ Drop the column, table or index in release N+5 or later, and always at least one release after the code removal, in a blocking migration guarded the same way, with `breaking = true`. For `release = '0.93.0'` that is `0.98.0` or later. A dropped column is a technical breaking change: it needs the `⛓️‍💥 breaking-change` label and an entry in `docs/install/reference/breaking-changes-unreleased.mdx`.
+ </Step>
+</Steps>
+
+<Warning>
+Never remove the code and drop the schema in the same release. A Cloud rollback runs the previous image on the new schema and does not run `down()`, so the previous release must already work without the dropped object. If it still reads the old column, every request that touches it fails after the rollback.
+</Warning>
diff --git a/packages/server/api/src/app/database/background-migration-runner.ts b/packages/server/api/src/app/database/background-migration-runner.ts
index e27365eb89c..bd1b676c80f 100644
--- a/packages/server/api/src/app/database/background-migration-runner.ts
+++ b/packages/server/api/src/app/database/background-migration-runner.ts
@@ -1,7 +1,7 @@
-import { tryCatch } from '@activepieces/core-utils'
-import { wideEvent } from '@activepieces/server-utils'
+import { isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'
+import { createLogger, wideEvent } from '@activepieces/server-utils'
import { FastifyBaseLogger } from 'fastify'
-import { DataSource } from 'typeorm'
+import { DataSource, QueryRunner } from 'typeorm'
import { BackgroundMigration } from './background-migration'
import { databaseConnection, migrationDatabaseConnection } from './database-connection'
import { BackgroundMigrationNotCompleteError } from './migration-helpers'
@@ -76,10 +76,8 @@ export const backgroundMigrationRunner = {
const { data, error } = await tryCatch(async () => {
const ds = dataSource ?? databaseConnection()
const source = migrations ?? getBackgroundMigrations()
- await ensureBackgroundMigrationsTable(ds)
- const completed = await getCompletedNames(ds)
const sourceNames = source.map(MigrationClass => new MigrationClass().name)
- const pendingCount = sourceNames.filter(name => !completed.has(name)).length
+ const pendingCount = (await backgroundMigrationRunner.listPending({ migrations: source, dataSource: ds })).length
const failedMigration = await getLatestFailure(ds, sourceNames)
return {
pendingCount,
@@ -93,6 +91,20 @@ export const backgroundMigrationRunner = {
}
return data
},
+
+ async listPending({
+ migrations,
+ dataSource,
+ }: {
+ migrations?: (new () => BackgroundMigration)[]
+ dataSource?: DataSource
+ } = {}): Promise<string[]> {
+ const ds = dataSource ?? databaseConnection()
+ const source = migrations ?? getBackgroundMigrations()
+ await ensureBackgroundMigrationsTable(ds)
+ const completed = await getCompletedNames(ds)
+ return source.map(MigrationClass => new MigrationClass().name).filter(name => !completed.has(name))
+ },
}
async function runOne({
@@ -113,7 +125,6 @@ async function runOne({
log.info({ migration: { name: migration.name } }, '[backgroundMigrationRunner] Migration already completed under another lock holder, skipping')
return
}
- log.info({ migration: { name: migration.name } }, '[backgroundMigrationRunner] Running migration')
wideEvent.set({ migration: { name: migration.name } })
await wideEvent.timed({
name: 'backgroundMigration',
@@ -132,10 +143,18 @@ async function executeMigrationBody({
log: FastifyBaseLogger
dataSource: DataSource
}): Promise<void> {
+ const startedAt = Date.now()
+ let finished = false
+ let backendPid: number | undefined
+ emitProgress({ migrationName: migration.name, phase: 'started' })
+ const heartbeat = setInterval(() => {
+ emitHeartbeat({ migrationName: migration.name, startedAt, backendPid, isFinished: () => finished }).catch(() => undefined)
+ }, BACKGROUND_MIGRATION_HEARTBEAT_INTERVAL_MS)
try {
const executionDs = await migrationDatabaseConnection()
const queryRunner = executionDs.createQueryRunner()
try {
+ backendPid = await readBackendPid({ queryRunner })
await migration.up(queryRunner)
}
finally {
@@ -152,16 +171,85 @@ async function executeMigrationBody({
if (recordError) {
log.error({ migration: { name: migration.name }, error: recordError instanceof Error ? recordError.message : String(recordError) }, '[backgroundMigrationRunner] Failed to persist migration failure; original error follows')
}
- log.error({ migration: { name: migration.name }, error: message }, '[backgroundMigrationRunner] Migration failed')
+ emitProgress({ migrationName: migration.name, phase: 'failed', elapsedMs: Date.now() - startedAt, error })
throw error
}
+ finally {
+ finished = true
+ clearInterval(heartbeat)
+ }
await dataSource.query(
`INSERT INTO "${BACKGROUND_MIGRATIONS_TABLE}" ("name", "executed_at") VALUES ($1, NOW())
ON CONFLICT ("name") DO UPDATE SET "executed_at" = NOW(), "failed_at" = NULL, "last_error" = NULL`,
[migration.name],
)
- log.info({ migration: { name: migration.name } }, '[backgroundMigrationRunner] Migration completed')
+ emitProgress({ migrationName: migration.name, phase: 'completed', elapsedMs: Date.now() - startedAt })
+}
+
+function emitProgress({ migrationName, phase, elapsedMs, error, details }: EmitProgressParams): void {
+ const logger = createLogger({
+ event: 'background-migration.progress',
+ migration: { name: migrationName, phase, ...spreadIfDefined('elapsedMs', elapsedMs), ...details },
+ })
+ if (!isNil(error)) {
+ logger.error(error instanceof Error ? error : new Error(String(error)))
+ }
+ logger.emit({ _forceKeep: true })
+}
+
+async function emitHeartbeat({ migrationName, startedAt, backendPid, isFinished }: EmitHeartbeatParams): Promise<void> {
+ const createIndexProgress = isNil(backendPid) ? undefined : await readCreateIndexProgress({ backendPid })
+ if (isFinished()) {
+ return
+ }
+ emitProgress({
+ migrationName,
+ phase: 'running',
+ elapsedMs: Date.now() - startedAt,
+ details: isNil(createIndexProgress) ? undefined : { createIndexProgress },
+ })
+}
+
+async function readBackendPid({ queryRunner }: { queryRunner: QueryRunner }): Promise<number | undefined> {
+ const { data } = await tryCatch(async () => {
+ const rows: unknown = await queryRunner.query('SELECT pg_backend_pid() AS "pid"')
+ if (!Array.isArray(rows) || rows.length === 0) return undefined
+ const [row] = rows
+ if (row === null || typeof row !== 'object' || !('pid' in row) || typeof row.pid !== 'number') return undefined
+ return row.pid
+ })
+ return data ?? undefined
+}
+
+async function readCreateIndexProgress({ backendPid }: { backendPid: number }): Promise<unknown[] | undefined> {
+ const { data } = await tryCatch(async () => Promise.race([
+ queryCreateIndexProgress({ backendPid }),
+ rejectAfter({ ms: BACKGROUND_MIGRATION_PROGRESS_READ_TIMEOUT_MS }),
+ ]))
+ return data ?? undefined
+}
+
+async function queryCreateIndexProgress({ backendPid }: { backendPid: number }): Promise<unknown[] | undefined> {
+ const executionDs = await migrationDatabaseConnection()
+ const rows: unknown = await executionDs.query(
+ `SELECT p."phase",
+ p."index_relid"::regclass::text AS "indexName",
+ p."blocks_done" AS "blocksDone",
+ p."blocks_total" AS "blocksTotal",
+ p."tuples_done" AS "tuplesDone",
+ p."tuples_total" AS "tuplesTotal"
+ FROM pg_stat_progress_create_index p
+ WHERE p."pid" = $1`,
+ [backendPid],
+ )
+ return Array.isArray(rows) && rows.length > 0 ? rows : undefined
+}
+
+function rejectAfter({ ms }: { ms: number }): Promise<never> {
+ return new Promise((_resolve, reject) => {
+ setTimeout(() => reject(new Error(`timed out after ${ms}ms`)), ms).unref()
+ })
}
async function ensureBackgroundMigrationsTable(dataSource: DataSource): Promise<void> {
@@ -216,6 +304,8 @@ const MAX_CATCHUP_ATTEMPTS = 10
export const backgroundMigrationLockKey = (migrationName: string): string => `background_migration:${migrationName}`
export const BACKGROUND_MIGRATION_LOCK_TIMEOUT_SECONDS = 240
+export const BACKGROUND_MIGRATION_HEARTBEAT_INTERVAL_MS = 5 * 60 * 1000
+export const BACKGROUND_MIGRATION_PROGRESS_READ_TIMEOUT_MS = 10_000
export const BACKGROUND_MIGRATIONS_TABLE = 'background_migrations'
@@ -231,3 +321,20 @@ export type BackgroundMigrationStatus = {
error: string | null
failedMigration: FailedBackgroundMigration | null
}
+
+type BackgroundMigrationPhase = 'started' | 'running' | 'completed' | 'failed'
+
+type EmitProgressParams = {
+ migrationName: string
+ phase: BackgroundMigrationPhase
+ elapsedMs?: number
+ error?: unknown
+ details?: Record<string, unknown>
+}
+
+type EmitHeartbeatParams = {
+ migrationName: string
+ startedAt: number
+ backendPid: number | undefined
+ isFinished: () => boolean
+}
diff --git a/packages/server/api/src/app/helper/system-jobs/system-job-failure-pager.ts b/packages/server/api/src/app/helper/system-jobs/system-job-failure-pager.ts
new file mode 100644
index 00000000000..c8e2a665cda
--- /dev/null
+++ b/packages/server/api/src/app/helper/system-jobs/system-job-failure-pager.ts
@@ -0,0 +1,64 @@
+import { tryCatch } from '@activepieces/core-utils'
+import { onCallService } from '@activepieces/server-utils'
+import { UnrecoverableError } from 'bullmq'
+import { FastifyBaseLogger } from 'fastify'
+import { backgroundMigrationRunner } from '../../database/background-migration-runner'
+import { system } from '../system/system'
+import { AppSystemProp } from '../system/system-props'
+import { SystemJobName } from './common'
+
+export const systemJobFailurePager = {
+ gaveUp({ attemptsMade, attempts, error }: GaveUpParams): boolean {
+ return error instanceof UnrecoverableError || attemptsMade >= (attempts ?? 1)
+ },
+
+ pageOnFinalFailure({ jobName, jobId, attemptsMade, error, log }: PageOnFinalFailureParams): void {
+ if (!PAGED_SYSTEM_JOBS.has(jobName)) {
+ return
+ }
+ listPendingMigrations({ log })
+ .then((pendingMigrations) => onCallService(log, system.get(AppSystemProp.PAGE_ONCALL_WEBHOOK)).page({
+ code: 'SYSTEM_JOB_FAILED',
+ message: `Background migrations stopped: "${jobName}" gave up after ${attemptsMade} attempts (${error.message}). Pending: ${pendingMigrations?.join(', ') ?? 'unknown, background_migrations could not be read'}. Restart an API pod or deploy to retry them.`,
+ params: { jobName, jobId, attemptsMade, error: error.message, pendingMigrations: pendingMigrations ?? null },
+ }))
+ .catch((pageError) => {
+ log.error({ pageError }, '[systemJob#worker] Failed to send on-call page for a failed system job')
+ })
+ },
+}
+
+async function listPendingMigrations({ log }: { log: FastifyBaseLogger }): Promise<string[] | undefined> {
+ const { data, error } = await tryCatch(async () => Promise.race([
+ backgroundMigrationRunner.listPending(),
+ rejectAfter({ ms: PENDING_LOOKUP_TIMEOUT_MS }),
+ ]))
+ if (error) {
+ log.error({ error: error.message }, '[systemJob#worker] Could not list pending background migrations for the on-call page')
+ return undefined
+ }
+ return data
+}
+
+function rejectAfter({ ms }: { ms: number }): Promise<never> {
+ return new Promise((_resolve, reject) => {
+ setTimeout(() => reject(new Error(`timed out after ${ms}ms`)), ms).unref()
+ })
+}
+
+const PAGED_SYSTEM_JOBS = new Set<string>([SystemJobName.RUN_BACKGROUND_MIGRATIONS])
+const PENDING_LOOKUP_TIMEOUT_MS = 5_000
+
+type GaveUpParams = {
+ attemptsMade: number
+ attempts: number | undefined
+ error: Error
+}
+
+type PageOnFinalFailureParams = {
+ jobName: string
+ jobId: string | undefined
+ attemptsMade: number
+ error: Error
+ log: FastifyBaseLogger
+}
diff --git a/packages/server/api/src/app/helper/system-jobs/system-job.ts b/packages/server/api/src/app/helper/system-jobs/system-job.ts
index b8617a91a2a..204c0795f86 100644
--- a/packages/server/api/src/app/helper/system-jobs/system-job.ts
+++ b/packages/server/api/src/app/helper/system-jobs/system-job.ts
@@ -8,6 +8,7 @@ import { jobFailureLogger } from '../../workers/job-queue/job-failure-logger'
import { exceptionHandler } from '../exception-handler'
import { SystemJobData, SystemJobName, SystemJobSchedule, UpsertJobResult } from './common'
import { systemJobHandlers } from './job-handlers'
+import { systemJobFailurePager } from './system-job-failure-pager'
const FIFTEEN_MINUTES = apDayjsDuration(15, 'minute').asMilliseconds()
const ONE_MONTH = apDayjsDuration(1, 'month').asSeconds()
@@ -87,8 +88,7 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
systemJobWorker.on('failed', (job, err) => {
const attemptsUsed = job?.attemptsMade ?? 0
- const maxAttempts = job?.opts?.attempts ?? 1
- if (attemptsUsed >= maxAttempts) {
+ if (systemJobFailurePager.gaveUp({ attemptsMade: attemptsUsed, attempts: job?.opts?.attempts, error: err })) {
jobFailureLogger.logJobFailed({
queueName: SYSTEM_JOB_QUEUE,
jobId: job?.id,
@@ -97,6 +97,13 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
log,
})
exceptionHandler.handle(err, log)
+ systemJobFailurePager.pageOnFinalFailure({
+ jobName: job?.name ?? 'unknown',
+ jobId: job?.id,
+ attemptsMade: attemptsUsed,
+ error: err,
+ log,
+ })
}
})
diff --git a/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts b/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts
index d3a7e89a9f1..d56b7488d14 100644
--- a/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts
+++ b/packages/server/api/test/integration/ce/database/background-migration-runner.test.ts
@@ -3,6 +3,8 @@ import { QueryRunner } from 'typeorm'
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from 'vitest'
import { BackgroundMigration } from '../../../../src/app/database/background-migration'
import {
+ BACKGROUND_MIGRATION_HEARTBEAT_INTERVAL_MS,
+ BACKGROUND_MIGRATION_PROGRESS_READ_TIMEOUT_MS,
BACKGROUND_MIGRATIONS_TABLE,
backgroundMigrationRunner,
} from '../../../../src/app/database/background-migration-runner'
@@ -14,6 +16,24 @@ import { RefetchMissingPieceTranslations1879000000000 } from '../../../../src/ap
import { getBackgroundMigrations } from '../../../../src/app/database/postgres-connection'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+const { emittedSpy } = vi.hoisted(() => ({ emittedSpy: vi.fn() }))
+
+vi.mock('@activepieces/server-utils', async (importOriginal) => {
+ const original = await importOriginal<typeof import('@activepieces/server-utils')>()
+ return {
+ ...original,
+ createLogger: (...args: Parameters<typeof original.createLogger>) => {
+ const logger = original.createLogger(...args)
+ const emit = logger.emit.bind(logger)
+ logger.emit = (overrides) => {
+ emittedSpy(logger.getContext())
+ return emit(overrides)
+ }
+ return logger
+ },
+ }
+})
+
let app: FastifyInstance | null = null
beforeAll(async () => {
@@ -148,6 +168,23 @@ describe('backgroundMigrationRunner', () => {
expect(status.completedCount).toBeGreaterThanOrEqual(1)
})
+ it('listPending names every unfinished migration, including one whose pod died without recording a failure', async () => {
+ const done = makeTestMigration({ name: `PendingDone${Date.now()}` })
+ const failed = makeTestMigration({
+ name: `PendingFailed${Date.now()}`,
+ up: async () => {
+ throw new Error('canceling statement due to statement timeout')
+ },
+ })
+ const killedMidRun = makeTestMigration({ name: `PendingKilled${Date.now()}` })
+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [done] })
+ await expect(backgroundMigrationRunner.run({ log: app!.log, migrations: [failed] })).rejects.toThrow(/statement timeout/)
+
+ const pending = await backgroundMigrationRunner.listPending({ migrations: [done, failed, killedMidRun] })
+
+ expect(pending).toEqual([new failed().name, new killedMidRun().name])
+ })
+
it('getStatus returns error-shape instead of throwing on DB failure', async () => {
const ds = databaseConnection()
const spy = vi.spyOn(ds, 'query').mockRejectedValueOnce(new Error('db down'))
@@ -160,6 +197,97 @@ describe('backgroundMigrationRunner', () => {
})
})
+describe('background migration progress events', () => {
+ afterEach(() => {
+ vi.useRealTimers()
+ emittedSpy.mockClear()
+ })
+
+ it('emits started and completed as their own events, not only inside the job wide event', async () => {
+ const MigrationClass = makeTestMigration({ name: `ProgressHappy${Date.now()}` })
+ const name = new MigrationClass().name
+
+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [MigrationClass] })
+
+ expect(progressPhasesFor(name)).toEqual(['started', 'completed'])
+ expect(progressEventsFor(name).at(-1)?.migration.elapsedMs).toEqual(expect.any(Number))
+ })
+
+ it('emits failed with the elapsed time when the migration throws', async () => {
+ const MigrationClass = makeTestMigration({
+ name: `ProgressFails${Date.now()}`,
+ up: async () => {
+ throw new Error('canceling statement due to statement timeout')
+ },
+ })
+ const name = new MigrationClass().name
+
+ await expect(
+ backgroundMigrationRunner.run({ log: app!.log, migrations: [MigrationClass] }),
+ ).rejects.toThrow(/statement timeout/)
+
+ expect(progressPhasesFor(name)).toEqual(['started', 'failed'])
+ expect(progressEventsFor(name).at(-1)?.error?.message).toMatch(/statement timeout/)
+ })
+
+ it('emits a running heartbeat while a long migration is still in progress', async () => {
+ vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] })
+ const name = `ProgressHeartbeat${Date.now()}`
+ const MigrationClass = makeTestMigration({
+ name,
+ up: async () => {
+ vi.advanceTimersByTime(BACKGROUND_MIGRATION_HEARTBEAT_INTERVAL_MS)
+ await vi.waitFor(() => expect(progressPhasesFor(name)).toContain('running'))
+ },
+ })
+
+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [MigrationClass] })
+
+ expect(progressPhasesFor(name)).toEqual(['started', 'running', 'completed'])
+ const heartbeat = progressEventsFor(name).find((event) => event.migration.phase === 'running')
+ expect(heartbeat?.migration.elapsedMs).toEqual(expect.any(Number))
+ })
+
+ it('still emits the heartbeat, without index details, when the progress read waits on a busy pool', { timeout: 30_000 }, async () => {
+ vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] })
+ const ds = databaseConnection()
+ const query = ds.query.bind(ds)
+ const spy = vi.spyOn(ds, 'query').mockImplementation(async (sql: string, parameters?: unknown[]) =>
+ sql.includes('pg_stat_progress_create_index') ? new Promise<never>(() => undefined) : query(sql, parameters))
+ const name = `ProgressBusyPool${Date.now()}`
+ const MigrationClass = makeTestMigration({
+ name,
+ up: async () => {
+ vi.advanceTimersByTime(BACKGROUND_MIGRATION_HEARTBEAT_INTERVAL_MS)
+ await vi.waitFor(() => expect(progressPhasesFor(name)).toContain('running'), { timeout: BACKGROUND_MIGRATION_PROGRESS_READ_TIMEOUT_MS + 5_000 })
+ },
+ })
+
+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [MigrationClass] })
+ spy.mockRestore()
+
+ const heartbeat = progressEventsFor(name).find((event) => event.migration.phase === 'running')
+ expect(heartbeat?.migration.elapsedMs).toEqual(expect.any(Number))
+ expect(progressPhasesFor(name)).toEqual(['started', 'running', 'completed'])
+ })
+
+ it('drops a heartbeat that is still reading progress when the migration finishes', async () => {
+ vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] })
+ const name = `ProgressLateHeartbeat${Date.now()}`
+ const MigrationClass = makeTestMigration({
+ name,
+ up: async () => {
+ vi.advanceTimersByTime(BACKGROUND_MIGRATION_HEARTBEAT_INTERVAL_MS)
+ },
+ })
+
+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [MigrationClass] })
+ await new Promise((resolve) => setTimeout(resolve, 200))
+
+ expect(progressPhasesFor(name)).toEqual(['started', 'completed'])
+ })
+})
+
describe('getBackgroundMigrations', () => {
it('runs the quick piece-translation backfill before the hours-long flow_run index reshape', () => {
const order = getBackgroundMigrations()
@@ -292,6 +420,20 @@ async function waitFor(predicate: () => boolean, timeoutMs: number): Promise<voi
}
}
+function progressEventsFor(name: string): ProgressEvent[] {
+ return emittedSpy.mock.calls
+ .map(([context]) => context)
+ .filter((context): context is ProgressEvent => isProgressEvent(context) && context.migration.name === name)
+}
+
+function progressPhasesFor(name: string): string[] {
+ return progressEventsFor(name).map((event) => event.migration.phase)
+}
+
+function isProgressEvent(context: unknown): context is ProgressEvent {
+ return typeof context === 'object' && context !== null && 'event' in context && context.event === 'background-migration.progress'
+}
+
async function selectByName(name: string): Promise<{ name: string }[]> {
const rows = await databaseConnection().query(
`SELECT "name" FROM "${BACKGROUND_MIGRATIONS_TABLE}" WHERE "executed_at" IS NOT NULL`,
@@ -302,3 +444,9 @@ async function selectByName(name: string): Promise<{ name: string }[]> {
async function selectCompletedByName(name: string): Promise<{ name: string }[]> {
return selectByName(name)
}
+
+type ProgressEvent = {
+ event: 'background-migration.progress'
+ migration: { name: string, phase: string, elapsedMs?: number }
+ error?: { message: string }
+}
diff --git a/packages/server/api/test/unit/app/helper/system-jobs/system-job-failure-pager.test.ts b/packages/server/api/test/unit/app/helper/system-jobs/system-job-failure-pager.test.ts
new file mode 100644
index 00000000000..8b8b79d602d
--- /dev/null
+++ b/packages/server/api/test/unit/app/helper/system-jobs/system-job-failure-pager.test.ts
@@ -0,0 +1,143 @@
+import { UnrecoverableError } from 'bullmq'
+import { FastifyBaseLogger } from 'fastify'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { SystemJobName } from '../../../../../src/app/helper/system-jobs/common'
+import { systemJobFailurePager } from '../../../../../src/app/helper/system-jobs/system-job-failure-pager'
+
+const { pageMock, onCallServiceMock, listPendingMock } = vi.hoisted(() => {
+ const pageMock = vi.fn()
+ return { pageMock, onCallServiceMock: vi.fn(() => ({ page: pageMock })), listPendingMock: vi.fn() }
+})
+
+vi.mock('@activepieces/server-utils', async (importOriginal) => {
+ const original = await importOriginal<typeof import('@activepieces/server-utils')>()
+ return {
+ ...original,
+ onCallService: onCallServiceMock,
+ }
+})
+
+vi.mock('../../../../../src/app/database/background-migration-runner', () => ({
+ backgroundMigrationRunner: { listPending: listPendingMock },
+}))
+
+const log = { error: vi.fn() } as unknown as FastifyBaseLogger
+const WEBHOOK_URL = 'https://oncall.example.com/hook'
+const PENDING = ['ReshapeFlowRunListIndexesForBarrierChildren1871000000000', 'BackfillPieceTranslation1874000000000']
+
+describe('systemJobFailurePager.pageOnFinalFailure', () => {
+ beforeEach(() => {
+ process.env.AP_PAGE_ONCALL_WEBHOOK = WEBHOOK_URL
+ pageMock.mockResolvedValue(undefined)
+ listPendingMock.mockResolvedValue(PENDING)
+ })
+
+ afterEach(() => {
+ delete process.env.AP_PAGE_ONCALL_WEBHOOK
+ vi.clearAllMocks()
+ })
+
+ it('pages on-call with the pending migrations and the next step when the job gives up, including a stalled worker', async () => {
+ systemJobFailurePager.pageOnFinalFailure({
+ jobName: SystemJobName.RUN_BACKGROUND_MIGRATIONS,
+ jobId: 'run-background-migrations',
+ attemptsMade: 28,
+ error: new Error('job stalled more than allowable limit'),
+ log,
+ })
+
+ await vi.waitFor(() => expect(pageMock).toHaveBeenCalled())
+ expect(onCallServiceMock).toHaveBeenCalledWith(log, WEBHOOK_URL)
+ const [payload] = pageMock.mock.calls[0]
+ expect(payload.code).toBe('SYSTEM_JOB_FAILED')
+ expect(payload.message).toContain('job stalled more than allowable limit')
+ expect(payload.message).toContain(PENDING.join(', '))
+ expect(payload.message).toContain('Restart an API pod or deploy to retry them')
+ expect(payload.params).toEqual({
+ jobName: SystemJobName.RUN_BACKGROUND_MIGRATIONS,
+ jobId: 'run-background-migrations',
+ attemptsMade: 28,
+ error: 'job stalled more than allowable limit',
+ pendingMigrations: PENDING,
+ })
+ })
+
+ it('still pages, saying the pending list is unknown, when background_migrations cannot be read', async () => {
+ listPendingMock.mockRejectedValue(new Error('connection terminated'))
+
+ systemJobFailurePager.pageOnFinalFailure({
+ jobName: SystemJobName.RUN_BACKGROUND_MIGRATIONS,
+ jobId: 'run-background-migrations',
+ attemptsMade: 10,
+ error: new Error('canceling statement due to statement timeout'),
+ log,
+ })
+
+ await vi.waitFor(() => expect(pageMock).toHaveBeenCalled())
+ const [payload] = pageMock.mock.calls[0]
+ expect(payload.message).toContain('Pending: unknown')
+ expect(payload.params.pendingMigrations).toBeNull()
+ })
+
+ it('still pages, saying the pending list is unknown, when the lookup hangs on an exhausted database pool', async () => {
+ vi.useFakeTimers({ toFake: ['setTimeout'] })
+ listPendingMock.mockReturnValue(new Promise(() => undefined))
+
+ systemJobFailurePager.pageOnFinalFailure({
+ jobName: SystemJobName.RUN_BACKGROUND_MIGRATIONS,
+ jobId: 'run-background-migrations',
+ attemptsMade: 10,
+ error: new Error('job stalled more than allowable limit'),
+ log,
+ })
+ await vi.advanceTimersByTimeAsync(5_000)
+ vi.useRealTimers()
+
+ await vi.waitFor(() => expect(pageMock).toHaveBeenCalled())
+ const [payload] = pageMock.mock.calls[0]
+ expect(payload.message).toContain('Pending: unknown')
+ expect(payload.params.pendingMigrations).toBeNull()
+ })
+
+ it('does not page for system jobs outside the paged set', async () => {
+ systemJobFailurePager.pageOnFinalFailure({
+ jobName: SystemJobName.FILE_CLEANUP_TRIGGER,
+ jobId: 'file-cleanup-trigger',
+ attemptsMade: 3,
+ error: new Error('boom'),
+ log,
+ })
+
+ await new Promise((resolve) => setTimeout(resolve, 20))
+ expect(listPendingMock).not.toHaveBeenCalled()
+ expect(pageMock).not.toHaveBeenCalled()
+ })
+
+ it('logs instead of throwing when the page itself fails', async () => {
+ pageMock.mockRejectedValue(new Error('webhook down'))
+
+ systemJobFailurePager.pageOnFinalFailure({
+ jobName: SystemJobName.RUN_BACKGROUND_MIGRATIONS,
+ jobId: 'run-background-migrations',
+ attemptsMade: 10,
+ error: new Error('canceling statement due to statement timeout'),
+ log,
+ })
+
+ await vi.waitFor(() => expect(log.error).toHaveBeenCalled())
+ })
+})
+
+describe('systemJobFailurePager.gaveUp', () => {
+ it('is true for a stalled job on its first attempt, because BullMQ fails it as unrecoverable', () => {
+ expect(systemJobFailurePager.gaveUp({ attemptsMade: 1, attempts: 10, error: new UnrecoverableError('job stalled more than allowable limit') })).toBe(true)
+ })
+
+ it('is true once the attempts are used up', () => {
+ expect(systemJobFailurePager.gaveUp({ attemptsMade: 10, attempts: 10, error: new Error('canceling statement due to statement timeout') })).toBe(true)
+ })
+
+ it('is false while BullMQ will still retry', () => {
+ expect(systemJobFailurePager.gaveUp({ attemptsMade: 3, attempts: 10, error: new Error('canceling statement due to statement timeout') })).toBe(false)
+ })
+})