The open alternative to Salesforce, designed for AI.
TypeScript crmcrm-systemcustomergood-first-issuegraphqlhacktoberfestjavascriptmarketingmonoreponestjsopen-sourcepostgresqlreactreactjssalestypescriptweb
Recover and retry deferred workspace migration actions (#26371)
Charles Bochet5bea6ed63805
diff --git a/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md b/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.mdindex 180e082250d..b4672ddd134 100644--- a/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md+++ b/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md@@ -1,6 +1,6 @@# Deferred workspace migration actions-Implementation: [twentyhq/twenty#26222](https://github.com/twentyhq/twenty/pull/26222) (PR 1 of 6, behind a feature flag). In-repo copy: `packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md`.+Implementation: [twentyhq/twenty#26222](https://github.com/twentyhq/twenty/pull/26222) (PR 1 of 6, merged), [twentyhq/twenty#26371](https://github.com/twentyhq/twenty/pull/26371) (PR 2, recovery), behind a feature flag. In-repo copy: `packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md`.## Problem@@ -33,7 +33,6 @@ Non-goals (this doc and PR 1):- Unique indexes, enum migrations, index updates (drop and rebuild), search vector rebuilds. Each can become a deferrable action later with the same contract.- Serializing concurrent metadata migrations, blocking changes while work is pending, and surfacing that state to users (PRs 3 and 4 below).-- Recovery of stranded rows and retry of failed ones (PR 2).## Design@@ -115,16 +114,27 @@ Index on `(workspaceId, status)`. A row is deleted once its action succeeds. The### Execution-`RunDeferredWorkspaceMigrationActionsJob` runs on `workspaceQueue`, deduplicated per workspace, 3 attempts retried immediately. `DeferredWorkspaceMigrationActionRunnerService.runPendingActions`:+`RunDeferredWorkspaceMigrationActionsJob` runs on `workspaceQueue` with 3 attempts and an exponential backoff starting at 30 seconds. It is deduplicated per workspace with BullMQ `deduplication` and `keepLastIfActive`: a job enqueued while another one runs for the same workspace waits for it, since two concurrent runs would each skip the rows the other claimed and break the ordering. `DeferredWorkspaceMigrationActionRunnerService.runPendingActions`:1. loads the workspace's `PENDING` rows once, ordered by creation then `position` (actions created later come with their own job);2. claims each row with a conditional update (`PENDING` to `IN_PROGRESS`, `attempts + 1`);3. resolves the handler with `actionHandlerKey` (`executeDeferredActionHandler`, registry `:118`) and calls `executeDeferredAction` with the flat entity maps loaded once per run through `WorkspaceManyOrAllFlatEntityMapsCacheService`, for the same metadata names the migration runner would load (`getMetadataNamesToLoadForWorkspaceMigration`), on a dedicated connection without the client `query_timeout` and with a server-side `statement_timeout` of one hour, so Postgres cancels a stuck statement itself;-4. deletes the row on success, or sets it back to `PENDING` (`FAILED` after the last attempt) with `lastError`;+4. deletes the row on success, or sets it back to `PENDING` (`FAILED` after the last attempt) with `lastError`, both scoped to the claim it took (`IN_PROGRESS` with its own attempt number) so a worker resuming after its row was recovered cannot overwrite the newer run;5. stops at the first failure and fails the job, so the queue retries it and the remaining actions keep running in order.`create_index` reads the index from the maps it receives and completes as obsolete if it is absent. The maps are fresh because the runner invalidates the cache after commit, before enqueueing the job. On a retry it runs `DROP INDEX CONCURRENTLY IF EXISTS` first, since a failed concurrent build leaves an invalid index behind.+Each execution records `deferred-workspace-migration-action/duration-ms` with `actionHandlerKey` and `status`, and `twenty_deferred_workspace_migration_actions` gauges the rows by status across workspaces.++### Recovery++`DeferredWorkspaceMigrationActionRecoveryService`, driven by `DeferredWorkspaceMigrationActionRecoveryCronJob`, runs every 10 minutes (`cron:deferred-workspace-migration-action-recovery`, registered by `cron:register:all`):++1. an `IN_PROGRESS` row started more than the statement timeout plus 15 minutes ago belongs to a worker that died mid-run, since Postgres has cancelled its statement by then; it goes back to `PENDING`, or to `FAILED` if it used its last attempt;+2. every workspace with `PENDING` rows is enqueued, which covers lost enqueues and the actions left behind a failure once the job ran out of attempts.++`workspace:retry-failed-deferred-migration-actions [-w <workspaceId>]` resets `FAILED` rows to `PENDING` with no attempts, for one workspace or all of them, and enqueues them.+## Alternatives considered- **Keep deferred work on `afterCommitSideEffects`.** It is in-process and fire-and-forget: a crash after commit loses the work and leaves metadata describing an index that does not exist. Lost because the work must be durable.@@ -140,9 +150,9 @@ Delivery, each PR merged on its own behind the flag:| PR | Scope || --- | --- || 1 | Table, `DEFERRABLE_WORKSPACE_MIGRATION_ACTIONS`, handler contract, runner persistence, deferred action runner and job with retries and timeout; `create_index` and `delete_logicFunction`; `afterCommitSideEffects` removed. |-| 2 | Recovery: sweeper cron resetting `IN_PROGRESS` rows past the timeout and re-enqueueing workspaces with `PENDING` rows; retry of `FAILED` rows (admin mutation, CLI); metrics on pending, failed and duration. |+| 2 | Recovery: cron resetting `IN_PROGRESS` rows past the timeout and enqueueing workspaces with `PENDING` rows; CLI retry of `FAILED` rows; job deduplication per workspace and retry backoff; metrics on duration and on rows by status. || 3 | Workspace migration lock: `schemaMigrationStatus` (`IDLE`, `MIGRATING`) and `schemaMigrationStartedAt` on `core.workspace`, taken with a conditional update before migrations touching objects, fields or indexes, released in `finally` with a takeover timeout. While taken, or while schema-affecting deferred actions are pending (`create_index`, not `delete_logicFunction`), such migrations fail with `SCHEMA_MIGRATION_IN_PROGRESS` or `DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS`. `FAILED` rows do not block. |-| 4 | Error surfacing: runner codes sent as `subCode` (today `extensions.code` is overwritten by the GraphQL error code), object and field exception handlers map the new codes to `ConflictError`, dedicated front message with the pending count, SDK CLI reads `subCode` and `userFriendlyMessage`. |+| 4 | Error surfacing and admin retry of `FAILED` rows from the admin panel: runner codes sent as `subCode` (today `extensions.code` is overwritten by the GraphQL error code), object and field exception handlers map the new codes to `ConflictError`, dedicated front message with the pending count, SDK CLI reads `subCode` and `userFriendlyMessage`. || 5 | Foreign keys: `ADD CONSTRAINT ... NOT VALID` for join columns created in the same action, plus a deferrable `create_fieldMetadata` action running `VALIDATE CONSTRAINT` with the constraint name in its payload. || 6 | Enable the flag for the affected self-hosted workspace, then cloud, then default on and remove the flag. |@@ -154,7 +164,9 @@ Tested in PR 1:- integration, through the public APIs only: with the flag on, creating an object adds one join column index to each system relation object, records of the new object get timeline activities, and deleting a logic function succeeds; existing object, field, index and logic function suites pass;- manual, 3M `timelineActivity` rows: flag off logic function deletion removes files right after commit; flag on persists actions with the worker stopped and processes them once started (`timelineActivity` index built in 2.4s outside the transaction); an object deleted before the worker ran completes its actions as obsolete; a broken column fails three times then `FAILED` while the other indexes build immediately; restoring it before a retry drops and rebuilds the index.-Known limitations until PRs 2 and 3: a worker crash leaves a row `IN_PROGRESS`; `FAILED` rows need a manual reset and no longer hold back the actions after them; metadata changes are not blocked while actions are pending, so a deletion racing an in-flight build can leave a physical index without metadata.+Tested in PR 2, manually on the same data: two object creations with the worker stopped queue a single job; with the enqueue removed and rows left `IN_PROGRESS` two hours ago, nothing runs until the cron resets them (one to `PENDING`, the one at its last attempt to `FAILED`) and enqueues the workspace; the retry command builds the `FAILED` index; a broken column is retried after 30 then 60 seconds, and the actions behind it run at the next cron once it is `FAILED`.++Known limitation until PR 3: metadata changes are not blocked while actions are pending. A deletion racing an in-flight build can leave a physical index without metadata, and a migration altering a table while its index builds concurrently can deadlock with it; Postgres then cancels one of them (observed locally: the index build was cancelled and retried).## Open questionsdiff --git a/packages/twenty-server/src/database/commands/cron-register-all.command.ts b/packages/twenty-server/src/database/commands/cron-register-all.command.tsindex 39db714e6ea..5be371110a4 100644--- a/packages/twenty-server/src/database/commands/cron-register-all.command.ts+++ b/packages/twenty-server/src/database/commands/cron-register-all.command.ts@@ -21,6 +21,7 @@ import { WebhookSubscriptionRenewalCronCommand } from 'src/modules/connected-accimport { EmailingOngoingStaleCronCommand } from 'src/modules/emailing/crons/commands/emailing-ongoing-stale.cron.command';import { ReconcileCampaignStatsCronCommand } from 'src/modules/emailing/crons/commands/reconcile-campaign-stats.cron.command';import { TrashCleanupCronCommand } from 'src/engine/trash-cleanup/commands/trash-cleanup.cron.command';+import { DeferredWorkspaceMigrationActionRecoveryCronCommand } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/commands/deferred-workspace-migration-action-recovery.cron.command';import { CleanOnboardingWorkspacesCronCommand } from 'src/engine/workspace-manager/workspace-cleaner/commands/clean-onboarding-workspaces.cron.command';import { CleanSuspendedWorkspacesCronCommand } from 'src/engine/workspace-manager/workspace-cleaner/commands/clean-suspended-workspaces.cron.command';import { CalendarEventListFetchCronCommand } from 'src/modules/calendar/calendar-event-import-manager/crons/commands/calendar-event-list-fetch.cron.command';@@ -79,6 +80,7 @@ export class CronRegisterAllCommand extends CommandRunner {private readonly marketplaceCatalogSyncCronCommand: MarketplaceCatalogSyncCronCommand,private readonly staleRegistrationCleanupCronCommand: StaleRegistrationCleanupCronCommand,private readonly pendingFileCleanupCronCommand: PendingFileCleanupCronCommand,+ private readonly deferredWorkspaceMigrationActionRecoveryCronCommand: DeferredWorkspaceMigrationActionRecoveryCronCommand,private readonly billingReminderCronCommand: BillingReminderCronCommand,private readonly applicationRecurringChargeCronCommand: ApplicationRecurringChargeCronCommand,private readonly userSessionCleanupCronCommand: UserSessionCleanupCronCommand,@@ -219,6 +221,10 @@ export class CronRegisterAllCommand extends CommandRunner {name: 'PendingFileCleanup',command: this.pendingFileCleanupCronCommand,},+ {+ name: 'DeferredWorkspaceMigrationActionRecovery',+ command: this.deferredWorkspaceMigrationActionRecoveryCronCommand,+ },{name: 'BillingReminder',command: this.billingReminderCronCommand,diff --git a/packages/twenty-server/src/database/commands/database-command.module.ts b/packages/twenty-server/src/database/commands/database-command.module.tsindex d019e3036b0..5754556c147 100644--- a/packages/twenty-server/src/database/commands/database-command.module.ts+++ b/packages/twenty-server/src/database/commands/database-command.module.ts@@ -52,6 +52,7 @@ import { DevSeederModule } from 'src/engine/workspace-manager/dev-seeder/dev-seeimport { WorkspaceCleanerModule } from 'src/engine/workspace-manager/workspace-cleaner/workspace-cleaner.module';import { WorkspaceManagerModule } from 'src/engine/workspace-manager/workspace-manager.module';import { WorkspaceMigrationModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration.module';+import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module';import { WorkspaceVersionModule } from 'src/engine/workspace-manager/workspace-version/workspace-version.module';import { WebhookSubscriptionModule } from 'src/modules/connected-account/webhook-subscription-manager/webhook-subscription.module';import { CalendarEventImportManagerModule } from 'src/modules/calendar/calendar-event-import-manager/calendar-event-import-manager.module';@@ -86,6 +87,7 @@ import { WorkflowCoreConsistencyModule } from 'src/modules/workflow/workflow-corFeatureFlagModule,WorkspaceCleanerModule,WorkspaceMigrationModule,+ WorkspaceMigrationRunnerModule,TrashCleanupModule,BillingReminderModule,ApplicationRecurringChargeModule,diff --git a/packages/twenty-server/src/engine/core-modules/message-queue/jobs.module.ts b/packages/twenty-server/src/engine/core-modules/message-queue/jobs.module.tsindex 110f62dabd3..144f18dad29 100644--- a/packages/twenty-server/src/engine/core-modules/message-queue/jobs.module.ts+++ b/packages/twenty-server/src/engine/core-modules/message-queue/jobs.module.ts@@ -58,6 +58,7 @@ import { LogicFunctionModule } from 'src/engine/metadata-modules/logic-function/import { NavigationMenuItemModule } from 'src/engine/metadata-modules/navigation-menu-item/navigation-menu-item.module';import { ObjectMetadataModule } from 'src/engine/metadata-modules/object-metadata/object-metadata.module';import { WebhookJobModule } from 'src/engine/metadata-modules/webhook/jobs/webhook-job.module';+import { DeferredWorkspaceMigrationActionRecoveryCronJob } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/jobs/deferred-workspace-migration-action-recovery.cron.job';import { RunDeferredWorkspaceMigrationActionsJob } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/jobs/run-deferred-workspace-migration-actions.job';import { WorkspaceMigrationRunnerModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module';import { SubscriptionsModule } from 'src/engine/subscriptions/subscriptions.module';@@ -149,6 +150,7 @@ import { WorkflowModule } from 'src/modules/workflow/workflow.module';TriggerUninstallApplicationJob,WarmUpApplicationLogicFunctionsJob,RunDeferredWorkspaceMigrationActionsJob,+ DeferredWorkspaceMigrationActionRecoveryCronJob,],})export class JobsModule {diff --git a/packages/twenty-server/src/engine/core-modules/metrics/constants/deferred-workspace-migration-action-duration-ms-bucket-boundaries.constant.ts b/packages/twenty-server/src/engine/core-modules/metrics/constants/deferred-workspace-migration-action-duration-ms-bucket-boundaries.constant.tsnew file mode 100644index 00000000000..2f4005b49b5--- /dev/null+++ b/packages/twenty-server/src/engine/core-modules/metrics/constants/deferred-workspace-migration-action-duration-ms-bucket-boundaries.constant.ts@@ -0,0 +1,2 @@+export const DEFERRED_WORKSPACE_MIGRATION_ACTION_DURATION_MS_BUCKET_BOUNDARIES =+ [100, 1000, 10000, 60000, 300000, 1800000] as const;diff --git a/packages/twenty-server/src/engine/core-modules/metrics/types/metrics-keys.type.ts b/packages/twenty-server/src/engine/core-modules/metrics/types/metrics-keys.type.tsindex a4693a53d35..a326af881e0 100644--- a/packages/twenty-server/src/engine/core-modules/metrics/types/metrics-keys.type.ts+++ b/packages/twenty-server/src/engine/core-modules/metrics/types/metrics-keys.type.ts@@ -94,6 +94,7 @@ export enum MetricsKeys {WorkspaceMigrationBuildEntityPhaseDurationMs = 'workspace-migration/build-entity-phase-duration-ms',WorkspaceMigrationRunDurationMs = 'workspace-migration/run-duration-ms',WorkspaceMigrationRunPhaseDurationMs = 'workspace-migration/run-phase-duration-ms',+ DeferredWorkspaceMigrationActionDurationMs = 'deferred-workspace-migration-action/duration-ms',WorkspaceMigrationActionDurationMs = 'workspace-migration/action-duration-ms',WorkspaceMigrationActionCount = 'workspace-migration/action-count',OrmV2ReadPathUsed = 'orm-v2/read-path-used',diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/commands/retry-failed-deferred-workspace-migration-actions.command.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/commands/retry-failed-deferred-workspace-migration-actions.command.tsnew file mode 100644index 00000000000..86d63c29b06--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/commands/retry-failed-deferred-workspace-migration-actions.command.ts@@ -0,0 +1,49 @@+import { Logger } from '@nestjs/common';++import { Command, CommandRunner, Option } from 'nest-commander';++import { DeferredWorkspaceMigrationActionRecoveryService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service';++type RetryFailedDeferredWorkspaceMigrationActionsCommandOptions = {+ workspaceId?: string;+};++@Command({+ name: 'workspace:retry-failed-deferred-migration-actions',+ description:+ 'Reset FAILED deferred workspace migration actions to PENDING and enqueue them, for one workspace or all of them',+})+export class RetryFailedDeferredWorkspaceMigrationActionsCommand extends CommandRunner {+ private readonly logger = new Logger(+ RetryFailedDeferredWorkspaceMigrationActionsCommand.name,+ );++ constructor(+ private readonly deferredWorkspaceMigrationActionRecoveryService: DeferredWorkspaceMigrationActionRecoveryService,+ ) {+ super();+ }++ @Option({+ flags: '-w, --workspace-id <workspaceId>',+ description: 'Only retry the failed actions of this workspace',+ required: false,+ })+ parseWorkspaceId(value: string): string {+ return value;+ }++ async run(+ _passedParams: string[],+ options: RetryFailedDeferredWorkspaceMigrationActionsCommandOptions,+ ): Promise<void> {+ const retriedActionCount =+ await this.deferredWorkspaceMigrationActionRecoveryService.retryFailedActions(+ options.workspaceId,+ );++ this.logger.log(+ `Retried ${retriedActionCount} failed deferred workspace migration action(s)`,+ );+ }+}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-recovery-cron-pattern.constant.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-recovery-cron-pattern.constant.tsnew file mode 100644index 00000000000..48cd3f6e9aa--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-recovery-cron-pattern.constant.ts@@ -0,0 +1,2 @@+export const DEFERRED_WORKSPACE_MIGRATION_ACTION_RECOVERY_CRON_PATTERN =+ '*/10 * * * *';diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-retry-backoff.constant.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-retry-backoff.constant.tsnew file mode 100644index 00000000000..b14fb7ceb45--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-retry-backoff.constant.ts@@ -0,0 +1,7 @@+import { type QueueJobBackoffOptions } from 'src/engine/core-modules/message-queue/drivers/interfaces/job-options.interface';++export const DEFERRED_WORKSPACE_MIGRATION_ACTION_RETRY_BACKOFF: QueueJobBackoffOptions =+ {+ strategy: 'exponential',+ initialDelayMilliseconds: 30_000,+ };diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-stale-threshold-ms.constant.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-stale-threshold-ms.constant.tsnew file mode 100644index 00000000000..45927df3241--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-stale-threshold-ms.constant.ts@@ -0,0 +1,4 @@+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_STATEMENT_TIMEOUT_MS } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-statement-timeout-ms.constant';++export const DEFERRED_WORKSPACE_MIGRATION_ACTION_STALE_THRESHOLD_MS =+ DEFERRED_WORKSPACE_MIGRATION_ACTION_STATEMENT_TIMEOUT_MS + 15 * 60 * 1000;diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/commands/deferred-workspace-migration-action-recovery.cron.command.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/commands/deferred-workspace-migration-action-recovery.cron.command.tsnew file mode 100644index 00000000000..c178e841088--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/commands/deferred-workspace-migration-action-recovery.cron.command.ts@@ -0,0 +1,33 @@+import { Command, CommandRunner } from 'nest-commander';++import { InjectMessageQueue } from 'src/engine/core-modules/message-queue/decorators/message-queue.decorator';+import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';+import { MessageQueueService } from 'src/engine/core-modules/message-queue/services/message-queue.service';+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_RECOVERY_CRON_PATTERN } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-recovery-cron-pattern.constant';+import { DeferredWorkspaceMigrationActionRecoveryCronJob } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/jobs/deferred-workspace-migration-action-recovery.cron.job';++@Command({+ name: 'cron:deferred-workspace-migration-action-recovery',+ description:+ 'Starts a cron job to recover stale deferred workspace migration actions and run pending ones',+})+export class DeferredWorkspaceMigrationActionRecoveryCronCommand extends CommandRunner {+ constructor(+ @InjectMessageQueue(MessageQueue.cronQueue)+ private readonly messageQueueService: MessageQueueService,+ ) {+ super();+ }++ async run(): Promise<void> {+ await this.messageQueueService.addCron<undefined>({+ jobName: DeferredWorkspaceMigrationActionRecoveryCronJob.name,+ data: undefined,+ options: {+ repeat: {+ pattern: DEFERRED_WORKSPACE_MIGRATION_ACTION_RECOVERY_CRON_PATTERN,+ },+ },+ });+ }+}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/jobs/deferred-workspace-migration-action-recovery.cron.job.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/jobs/deferred-workspace-migration-action-recovery.cron.job.tsnew file mode 100644index 00000000000..267c4f75190--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/jobs/deferred-workspace-migration-action-recovery.cron.job.ts@@ -0,0 +1,46 @@+import { Injectable, Logger } from '@nestjs/common';++import { SentryCronMonitor } from 'src/engine/core-modules/cron/sentry-cron-monitor.decorator';+import { ExceptionHandlerService } from 'src/engine/core-modules/exception-handler/exception-handler.service';+import { Process } from 'src/engine/core-modules/message-queue/decorators/process.decorator';+import { Processor } from 'src/engine/core-modules/message-queue/decorators/processor.decorator';+import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_RECOVERY_CRON_PATTERN } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-recovery-cron-pattern.constant';+import { DeferredWorkspaceMigrationActionRecoveryService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service';++@Injectable()+@Processor(MessageQueue.cronQueue)+export class DeferredWorkspaceMigrationActionRecoveryCronJob {+ private readonly logger = new Logger(+ DeferredWorkspaceMigrationActionRecoveryCronJob.name,+ );++ constructor(+ private readonly deferredWorkspaceMigrationActionRecoveryService: DeferredWorkspaceMigrationActionRecoveryService,+ private readonly exceptionHandlerService: ExceptionHandlerService,+ ) {}++ @Process(DeferredWorkspaceMigrationActionRecoveryCronJob.name)+ @SentryCronMonitor(+ DeferredWorkspaceMigrationActionRecoveryCronJob.name,+ DEFERRED_WORKSPACE_MIGRATION_ACTION_RECOVERY_CRON_PATTERN,+ )+ async handle(): Promise<void> {+ try {+ const resetActionCount =+ await this.deferredWorkspaceMigrationActionRecoveryService.resetStaleInProgressActions();++ const enqueuedWorkspaceCount =+ await this.deferredWorkspaceMigrationActionRecoveryService.enqueueWorkspacesWithPendingActions();++ if (resetActionCount > 0 || enqueuedWorkspaceCount > 0) {+ this.logger.log(+ `Reset ${resetActionCount} stale in-progress deferred action(s), enqueued ${enqueuedWorkspaceCount} workspace(s) with pending actions`,+ );+ }+ } catch (error) {+ this.exceptionHandlerService.captureExceptions([error]);+ throw error;+ }+ }+}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-gauge.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-gauge.service.tsnew file mode 100644index 00000000000..4f079f40b4c--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-gauge.service.ts@@ -0,0 +1,54 @@+import { Injectable, type OnModuleInit } from '@nestjs/common';+import { InjectDataSource } from '@nestjs/typeorm';++import { DataSource } from 'typeorm';++import { MetricsService } from 'src/engine/core-modules/metrics/metrics.service';+import { DeferredWorkspaceMigrationActionEntity } from 'src/engine/metadata-modules/deferred-workspace-migration-action/deferred-workspace-migration-action.entity';+import { type DeferredWorkspaceMigrationActionStatus } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action-status.type';++const DEFERRED_WORKSPACE_MIGRATION_ACTIONS_METRIC_NAME =+ 'twenty_deferred_workspace_migration_actions';++const DEFERRED_WORKSPACE_MIGRATION_ACTION_STATUSES: DeferredWorkspaceMigrationActionStatus[] =+ ['PENDING', 'IN_PROGRESS', 'FAILED'];++@Injectable()+export class DeferredWorkspaceMigrationActionGaugeService implements OnModuleInit {+ constructor(+ @InjectDataSource()+ private readonly coreDataSource: DataSource,+ private readonly metricsService: MetricsService,+ ) {}++ onModuleInit() {+ this.metricsService.createMultiObservableGauge({+ metricName: DEFERRED_WORKSPACE_MIGRATION_ACTIONS_METRIC_NAME,+ options: {+ description:+ 'Number of deferred workspace migration actions, labelled by status.',+ },+ callback: async () => {+ const actionCountByStatus = await this.getActionCountByStatus();++ return DEFERRED_WORKSPACE_MIGRATION_ACTION_STATUSES.map((status) => ({+ value: actionCountByStatus.get(status) ?? 0,+ attributes: { status },+ }));+ },+ cacheValue: true,+ });+ }++ private async getActionCountByStatus(): Promise<Map<string, number>> {+ const rows = await this.coreDataSource+ .getRepository(DeferredWorkspaceMigrationActionEntity)+ .createQueryBuilder('deferredWorkspaceMigrationAction')+ .select('deferredWorkspaceMigrationAction.status', 'status')+ .addSelect('COUNT(*)', 'actionCount')+ .groupBy('deferredWorkspaceMigrationAction.status')+ .getRawMany<{ status: string; actionCount: string }>();++ return new Map(rows.map((row) => [row.status, Number(row.actionCount)]));+ }+}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service.tsnew file mode 100644index 00000000000..fd1869358a7--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service.ts@@ -0,0 +1,104 @@+import { Injectable } from '@nestjs/common';++import { isDefined } from 'twenty-shared/utils';+import { LessThan, MoreThanOrEqual } from 'typeorm';++import { DeferredWorkspaceMigrationActionEntity } from 'src/engine/metadata-modules/deferred-workspace-migration-action/deferred-workspace-migration-action.entity';+import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';+import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_MAX_ATTEMPTS } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-max-attempts.constant';+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_STALE_THRESHOLD_MS } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-stale-threshold-ms.constant';+import { DeferredWorkspaceMigrationActionRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service';+import { type DeferredWorkspaceMigrationActionStatus } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action-status.type';++@Injectable()+export class DeferredWorkspaceMigrationActionRecoveryService {+ constructor(+ @InjectWorkspaceScopedRepository(DeferredWorkspaceMigrationActionEntity)+ private readonly deferredWorkspaceMigrationActionRepository: WorkspaceScopedRepository<DeferredWorkspaceMigrationActionEntity>,+ private readonly deferredWorkspaceMigrationActionRunnerService: DeferredWorkspaceMigrationActionRunnerService,+ ) {}++ async resetStaleInProgressActions(): Promise<number> {+ const staleBefore = new Date(+ Date.now() - DEFERRED_WORKSPACE_MIGRATION_ACTION_STALE_THRESHOLD_MS,+ );+ const lastError = 'Worker stopped while the action was in progress';++ const failedResult = await this.deferredWorkspaceMigrationActionRepository+ .createQueryBuilder()+ .update()+ .set({ status: 'FAILED', lastError })+ .where({+ status: 'IN_PROGRESS',+ startedAt: LessThan(staleBefore),+ attempts: MoreThanOrEqual(+ DEFERRED_WORKSPACE_MIGRATION_ACTION_MAX_ATTEMPTS,+ ),+ })+ .execute();++ const pendingResult = await this.deferredWorkspaceMigrationActionRepository+ .createQueryBuilder()+ .update()+ .set({ status: 'PENDING', lastError })+ .where({+ status: 'IN_PROGRESS',+ startedAt: LessThan(staleBefore),+ })+ .execute();++ return (failedResult.affected ?? 0) + (pendingResult.affected ?? 0);+ }++ async enqueueWorkspacesWithPendingActions(): Promise<number> {+ const workspaceIds = await this.findWorkspaceIdsByStatus('PENDING');++ for (const workspaceId of workspaceIds) {+ await this.deferredWorkspaceMigrationActionRunnerService.enqueue(+ workspaceId,+ );+ }++ return workspaceIds.length;+ }++ async retryFailedActions(workspaceId?: string): Promise<number> {+ const workspaceIds = isDefined(workspaceId)+ ? [workspaceId]+ : await this.findWorkspaceIdsByStatus('FAILED');++ let retriedActionCount = 0;++ for (const workspaceIdToRetry of workspaceIds) {+ const { affected } =+ await this.deferredWorkspaceMigrationActionRepository.update(+ workspaceIdToRetry,+ { status: 'FAILED' },+ { status: 'PENDING', attempts: 0, startedAt: null },+ );++ if ((affected ?? 0) > 0) {+ retriedActionCount += affected ?? 0;+ await this.deferredWorkspaceMigrationActionRunnerService.enqueue(+ workspaceIdToRetry,+ );+ }+ }++ return retriedActionCount;+ }++ private async findWorkspaceIdsByStatus(+ status: DeferredWorkspaceMigrationActionStatus,+ ): Promise<string[]> {+ const rows = await this.deferredWorkspaceMigrationActionRepository+ .createQueryBuilder('deferredWorkspaceMigrationAction')+ .select('deferredWorkspaceMigrationAction.workspaceId', 'workspaceId')+ .distinct(true)+ .where('deferredWorkspaceMigrationAction.status = :status', { status })+ .getRawMany<{ workspaceId: string }>();++ return rows.map(({ workspaceId }) => workspaceId);+ }+}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service.tsindex 8ab38b98d5b..352d7518140 100644--- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service.ts+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service.ts@@ -6,6 +6,9 @@ import { DataSource, type QueryRunner } from 'typeorm';import { InjectMessageQueue } from 'src/engine/core-modules/message-queue/decorators/message-queue.decorator';import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queue.constants';import { MessageQueueService } from 'src/engine/core-modules/message-queue/services/message-queue.service';+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_DURATION_MS_BUCKET_BOUNDARIES } from 'src/engine/core-modules/metrics/constants/deferred-workspace-migration-action-duration-ms-bucket-boundaries.constant';+import { MetricsService } from 'src/engine/core-modules/metrics/metrics.service';+import { MetricsKeys } from 'src/engine/core-modules/metrics/types/metrics-keys.type';import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';import { DeferredWorkspaceMigrationActionEntity } from 'src/engine/metadata-modules/deferred-workspace-migration-action/deferred-workspace-migration-action.entity';import { WorkspaceManyOrAllFlatEntityMapsCacheService } from 'src/engine/metadata-modules/flat-entity/services/workspace-many-or-all-flat-entity-maps-cache.service';@@ -14,6 +17,7 @@ import { getMetadataFlatEntityMapsKey } from 'src/engine/metadata-modules/flat-eimport { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';import { DEFERRED_WORKSPACE_MIGRATION_ACTION_MAX_ATTEMPTS } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-max-attempts.constant';+import { DEFERRED_WORKSPACE_MIGRATION_ACTION_RETRY_BACKOFF } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-retry-backoff.constant';import { DEFERRED_WORKSPACE_MIGRATION_ACTION_STATEMENT_TIMEOUT_MS } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/deferred-workspace-migration-action-statement-timeout-ms.constant';import { RUN_DEFERRED_WORKSPACE_MIGRATION_ACTIONS_JOB_NAME } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/run-deferred-workspace-migration-actions-job-name.constant';import {@@ -44,6 +48,7 @@ export class DeferredWorkspaceMigrationActionRunnerService {private readonly flatEntityMapsCacheService: WorkspaceManyOrAllFlatEntityMapsCacheService,private readonly workspaceMigrationRunnerActionHandlerRegistry: WorkspaceMigrationRunnerActionHandlerRegistryService,private readonly twentyConfigService: TwentyConfigService,+ private readonly metricsService: MetricsService,) {}async persist({@@ -145,14 +150,18 @@ export class DeferredWorkspaceMigrationActionRunnerService {}}- private async enqueue(workspaceId: string): Promise<void> {+ async enqueue(workspaceId: string): Promise<void> {try {await this.messageQueueService.add<RunDeferredWorkspaceMigrationActionsJobData>(RUN_DEFERRED_WORKSPACE_MIGRATION_ACTIONS_JOB_NAME,{ workspaceId },{- id: `deferred-workspace-migration-actions.${workspaceId}`,+ deduplication: {+ id: `deferred-workspace-migration-actions:${workspaceId}`,+ keepLastIfActive: true,+ },retryLimit: DEFERRED_WORKSPACE_MIGRATION_ACTION_MAX_ATTEMPTS - 1,+ backoff: DEFERRED_WORKSPACE_MIGRATION_ACTION_RETRY_BACKOFF,},);} catch (error) {@@ -272,18 +281,28 @@ export class DeferredWorkspaceMigrationActionRunnerService {await this.deferredWorkspaceMigrationActionRepository.delete(workspaceId,- { id },+ { id, status: 'IN_PROGRESS', attempts: attempt },);+ this.recordExecutionDuration({+ status: 'success',+ durationMs: performance.now() - executionStart,+ });+this.logger.log(`Deferred action ${pendingAction.actionHandlerKey} ${id} completed for workspace ${workspaceId} in ${(performance.now() - executionStart).toFixed(0)}ms`,);return true;} catch (error) {+ this.recordExecutionDuration({+ status: 'fail',+ durationMs: performance.now() - executionStart,+ });+await this.deferredWorkspaceMigrationActionRepository.update(workspaceId,- { id },+ { id, status: 'IN_PROGRESS', attempts: attempt },{status:attempt >= DEFERRED_WORKSPACE_MIGRATION_ACTION_MAX_ATTEMPTS@@ -304,6 +323,23 @@ export class DeferredWorkspaceMigrationActionRunnerService {}}+ private recordExecutionDuration({+ status,+ durationMs,+ }: {+ status: 'success' | 'fail';+ durationMs: number;+ }): void {+ this.metricsService.recordHistogram({+ key: MetricsKeys.DeferredWorkspaceMigrationActionDurationMs,+ value: durationMs,+ unit: 'ms',+ attributes: { status },+ bucketBoundaries:+ DEFERRED_WORKSPACE_MIGRATION_ACTION_DURATION_MS_BUCKET_BOUNDARIES,+ });+ }+private async createDeferredActionDataSource(): Promise<DataSource> {const coreDataSourceOptions = this.coreDataSource.options;diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action-execution-args.type.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action-execution-args.type.tsindex 976b3ebf5e5..4b0e11ed2cd 100644--- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action-execution-args.type.ts+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action-execution-args.type.ts@@ -1,12 +1,11 @@-import { type QueryRunner } from 'typeorm';+import { type WorkspaceMigrationActionRunnerArgs } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/workspace-migration-action-runner-args.type';+import { type AllUniversalWorkspaceMigrationAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-action-common';-import { type AllFlatEntityMaps } from 'src/engine/metadata-modules/flat-entity/types/all-flat-entity-maps.type';--export type DeferredWorkspaceMigrationActionExecutionArgs<TPayload> = {- workspaceId: string;+export type DeferredWorkspaceMigrationActionExecutionArgs<TPayload> = Pick<+ WorkspaceMigrationActionRunnerArgs<AllUniversalWorkspaceMigrationAction>,+ 'queryRunner' | 'allFlatEntityMaps' | 'workspaceId'+> & {applicationUniversalIdentifier: string;payload: TPayload;- allFlatEntityMaps: AllFlatEntityMaps;attempt: number;- queryRunner: QueryRunner;};diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.tsindex 9bb078b59a1..e86d73d928d 100644--- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.ts+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.ts@@ -15,7 +15,11 @@ import { WorkspaceCacheStorageModule } from 'src/engine/workspace-cache-storage/import { WorkspaceCacheModule } from 'src/engine/workspace-cache/workspace-cache.module';import { WorkspaceSchemaMigrationRunnerActionHandlersModule } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/action-handlers/workspace-schema-migration-runner-action-handlers.module';import { FlatCacheInvalidateCommand } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/commands/flat-cache-invalidate.command';+import { RetryFailedDeferredWorkspaceMigrationActionsCommand } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/commands/retry-failed-deferred-workspace-migration-actions.command';+import { DeferredWorkspaceMigrationActionRecoveryCronCommand } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/crons/commands/deferred-workspace-migration-action-recovery.cron.command';import { WorkspaceMigrationRunnerActionHandlerRegistryService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/registry/workspace-migration-runner-action-handler-registry.service';+import { DeferredWorkspaceMigrationActionRecoveryService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service';+import { DeferredWorkspaceMigrationActionGaugeService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-gauge.service';import { DeferredWorkspaceMigrationActionRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service';import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';@@ -40,12 +44,18 @@ import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/woWorkspaceMigrationRunnerService,WorkspaceMigrationRunnerActionHandlerRegistryService,DeferredWorkspaceMigrationActionRunnerService,+ DeferredWorkspaceMigrationActionRecoveryService,provideWorkspaceScopedRepository(DeferredWorkspaceMigrationActionEntity),+ DeferredWorkspaceMigrationActionGaugeService,FlatCacheInvalidateCommand,+ RetryFailedDeferredWorkspaceMigrationActionsCommand,+ DeferredWorkspaceMigrationActionRecoveryCronCommand,],exports: [WorkspaceMigrationRunnerService,DeferredWorkspaceMigrationActionRunnerService,+ DeferredWorkspaceMigrationActionRecoveryService,+ DeferredWorkspaceMigrationActionRecoveryCronCommand,],})export class WorkspaceMigrationRunnerModule {}