The open alternative to Salesforce, designed for AI.
TypeScript crmcrm-systemcustomergood-first-issuegraphqlhacktoberfestjavascriptmarketingmonoreponestjsopen-sourcepostgresqlreactreactjssalestypescriptweb
Refuse schema migrations while deferred index builds are in flight (#26446)
Charles Bochet7c4e3603b746
diff --git a/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md b/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.mdindex b4672ddd134..b2768168508 100644--- a/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md+++ b/packages/twenty-server/docs/DEFERRED_WORKSPACE_MIGRATION_ACTIONS.md@@ -32,7 +32,7 @@ Goals:Non-goals (this doc and PR 1):- Unique indexes, enum migrations, index updates (drop and rebuild), search vector rebuilds. Each can become a deferrable action later with the same contract.-- Serializing concurrent metadata migrations, blocking changes while work is pending, and surfacing that state to users (PRs 3 and 4 below).+- Surfacing the blocked state to users through the API and the front end (PR 4 below).## Design@@ -135,6 +135,16 @@ Each execution records `deferred-workspace-migration-action/duration-ms` with `a`workspace:retry-failed-deferred-migration-actions [-w <workspaceId>]` resets `FAILED` rows to `PENDING` with no attempts, for one workspace or all of them, and enqueues them.+### Guarding migrations against in-flight builds++A migration and a concurrent index build on the same table deadlock: Postgres cancels one of them, and the loser can be the migration a user is waiting on. `InFlightDeferredWorkspaceMigrationActionsService`, called from `WorkspaceMigrationRunnerService.run` for workspaces with the flag on and migrations carrying at least one `objectMetadata`, `fieldMetadata` or `index` action (`SCHEMA_AFFECTING_WORKSPACE_MIGRATION_METADATA_NAMES`), refuses the migration with `DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS` while the workspace has `PENDING` or `IN_PROGRESS` rows for a deferred action that builds workspace schema objects.++`SCHEMA_AFFECTING_DEFERRED_WORKSPACE_MIGRATION_ACTIONS` lists which ones those are, today `create_index` and not `delete_logicFunction`. `FAILED` rows do not block, since they need a manual retry and would otherwise freeze the data model.++The deferred action table is the whole state: an in-flight build is exactly a row in it, so nothing else has to be stored. Migrations that leave the workspace schema alone (views, dashboards, logic functions, page layouts, workflow sync) are not refused, which matters for the ones that write a workspace record first and sync it afterwards: a refusal there would leave the two copies out of sync with nothing retrying.++Two metadata migrations started at the same moment are not serialized by this, which is unchanged from today: they meet on the `ACCESS EXCLUSIVE` locks their DDL takes, and one of them hits the runner's 8 second `lock_timeout`.+## Alternatives considered- **Keep deferred work on `afterCommitSideEffects`.** It is in-process and fire-and-forget: a crash after commit loses the work and leaves metadata describing an index that does not exist. Lost because the work must be durable.@@ -151,7 +161,7 @@ Delivery, each PR merged on its own behind the flag:| --- | --- || 1 | Table, `DEFERRABLE_WORKSPACE_MIGRATION_ACTIONS`, handler contract, runner persistence, deferred action runner and job with retries and timeout; `create_index` and `delete_logicFunction`; `afterCommitSideEffects` removed. || 2 | Recovery: cron resetting `IN_PROGRESS` rows past the timeout and enqueueing workspaces with `PENDING` rows; CLI retry of `FAILED` rows; job deduplication per workspace and retry backoff; metrics on duration and on rows by status. |-| 3 | Workspace migration lock: `schemaMigrationStatus` (`IDLE`, `MIGRATING`) and `schemaMigrationStartedAt` on `core.workspace`, taken with a conditional update before migrations touching objects, fields or indexes, released in `finally` with a takeover timeout. While taken, or while schema-affecting deferred actions are pending (`create_index`, not `delete_logicFunction`), such migrations fail with `SCHEMA_MIGRATION_IN_PROGRESS` or `DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS`. `FAILED` rows do not block. |+| 3 | Refuse schema-affecting migrations while the workspace has in-flight deferred builds: see above. || 4 | Error surfacing and admin retry of `FAILED` rows from the admin panel: runner codes sent as `subCode` (today `extensions.code` is overwritten by the GraphQL error code), object and field exception handlers map the new codes to `ConflictError`, dedicated front message with the pending count, SDK CLI reads `subCode` and `userFriendlyMessage`. || 5 | Foreign keys: `ADD CONSTRAINT ... NOT VALID` for join columns created in the same action, plus a deferrable `create_fieldMetadata` action running `VALIDATE CONSTRAINT` with the constraint name in its payload. || 6 | Enable the flag for the affected self-hosted workspace, then cloud, then default on and remove the flag. |@@ -166,7 +176,9 @@ Tested in PR 1:Tested in PR 2, manually on the same data: two object creations with the worker stopped queue a single job; with the enqueue removed and rows left `IN_PROGRESS` two hours ago, nothing runs until the cron resets them (one to `PENDING`, the one at its last attempt to `FAILED`) and enqueues the workspace; the retry command builds the `FAILED` index; a broken column is retried after 30 then 60 seconds, and the actions behind it run at the next cron once it is `FAILED`.-Known limitation until PR 3: metadata changes are not blocked while actions are pending. A deletion racing an in-flight build can leave a physical index without metadata, and a migration altering a table while its index builds concurrently can deadlock with it; Postgres then cancels one of them (observed locally: the index build was cancelled and retried).+Tested in PR 3, manually on the same data: a second object creation is refused while the first one's index builds are pending, and accepted once they drain; a view is created while builds are pending; a pending logic function cleanup on its own refuses nothing; with the flag off nothing is refused. Integration: the deferred, index and object metadata suites pass.++Known limitation until PR 4: the refusal reaches the client as `INTERNAL_SERVER_ERROR` with the right `userFriendlyMessage`, because the GraphQL error code overwrites `extensions.code`.## Open questionsdiff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-deferred-workspace-migration-actions.constant.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-deferred-workspace-migration-actions.constant.tsnew file mode 100644index 00000000000..78f1dba6a2b--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-deferred-workspace-migration-actions.constant.ts@@ -0,0 +1,5 @@+import { type DeferrableWorkspaceMigrationActionHandlerKey } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action.type';++export const SCHEMA_AFFECTING_DEFERRED_WORKSPACE_MIGRATION_ACTIONS = [+ 'create_index',+] as const satisfies readonly DeferrableWorkspaceMigrationActionHandlerKey[];diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-workspace-migration-metadata-names.constant.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-workspace-migration-metadata-names.constant.tsnew file mode 100644index 00000000000..f0627395256--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-workspace-migration-metadata-names.constant.ts@@ -0,0 +1,7 @@+import { type AllMetadataName } from 'twenty-shared/metadata';++export const SCHEMA_AFFECTING_WORKSPACE_MIGRATION_METADATA_NAMES = [+ 'objectMetadata',+ 'fieldMetadata',+ 'index',+] as const satisfies readonly AllMetadataName[];diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/exceptions/workspace-migration-runner.exception.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/exceptions/workspace-migration-runner.exception.tsindex b2cf9a3d79e..7ca666b16e9 100644--- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/exceptions/workspace-migration-runner.exception.ts+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/exceptions/workspace-migration-runner.exception.ts@@ -11,6 +11,8 @@ export const WorkspaceMigrationRunnerExceptionCode = {EXECUTION_FAILED: 'EXECUTION_FAILED',APPLICATION_NOT_FOUND: 'APPLICATION_NOT_FOUND',DDL_LOCKED: 'DDL_LOCKED',+ DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS:+ 'DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS',} as const;const getWorkspaceMigrationRunnerExceptionUserFriendlyMessage = (@@ -25,6 +27,8 @@ const getWorkspaceMigrationRunnerExceptionUserFriendlyMessage = (return msg`Application not found.`;case WorkspaceMigrationRunnerExceptionCode.DDL_LOCKED:return msg`Workspace schema changes are temporarily locked.`;+ case WorkspaceMigrationRunnerExceptionCode.DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS:+ return msg`A previous data model change is still being applied in the background. Please try again in a moment.`;default:assertUnreachable(code);}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/in-flight-deferred-workspace-migration-actions.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/in-flight-deferred-workspace-migration-actions.service.tsnew file mode 100644index 00000000000..b1e35d72f0a--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/in-flight-deferred-workspace-migration-actions.service.ts@@ -0,0 +1,41 @@+import { Injectable } from '@nestjs/common';++import { In } from 'typeorm';++import { DeferredWorkspaceMigrationActionEntity } from 'src/engine/metadata-modules/deferred-workspace-migration-action/deferred-workspace-migration-action.entity';+import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';+import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';+import { SCHEMA_AFFECTING_DEFERRED_WORKSPACE_MIGRATION_ACTIONS } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-deferred-workspace-migration-actions.constant';+import {+ WorkspaceMigrationRunnerException,+ WorkspaceMigrationRunnerExceptionCode,+} from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/exceptions/workspace-migration-runner.exception';++@Injectable()+export class InFlightDeferredWorkspaceMigrationActionsService {+ constructor(+ @InjectWorkspaceScopedRepository(DeferredWorkspaceMigrationActionEntity)+ private readonly deferredWorkspaceMigrationActionRepository: WorkspaceScopedRepository<DeferredWorkspaceMigrationActionEntity>,+ ) {}++ async throwIfSchemaAffectingActionsAreInProgress(+ workspaceId: string,+ ): Promise<void> {+ const inProgressActionCount =+ await this.deferredWorkspaceMigrationActionRepository.count(workspaceId, {+ where: {+ status: In(['PENDING', 'IN_PROGRESS']),+ actionHandlerKey: In([+ ...SCHEMA_AFFECTING_DEFERRED_WORKSPACE_MIGRATION_ACTIONS,+ ]),+ },+ });++ if (inProgressActionCount > 0) {+ throw new WorkspaceMigrationRunnerException({+ message: `${inProgressActionCount} deferred schema action(s) are still running on workspace ${workspaceId}`,+ code: WorkspaceMigrationRunnerExceptionCode.DEFERRED_WORKSPACE_MIGRATION_ACTIONS_IN_PROGRESS,+ });+ }+ }+}diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.tsindex 04cd9d81f3f..977bbf07a2e 100644--- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service.ts@@ -26,6 +26,8 @@ import {WorkspaceMigrationRunnerException,WorkspaceMigrationRunnerExceptionCode,} from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/exceptions/workspace-migration-runner.exception';+import { InFlightDeferredWorkspaceMigrationActionsService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/in-flight-deferred-workspace-migration-actions.service';+import { isSchemaAffectingWorkspaceMigration } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/is-schema-affecting-workspace-migration.util';import { WorkspaceMigrationRunnerActionHandlerRegistryService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/registry/workspace-migration-runner-action-handler-registry.service';import { DeferredWorkspaceMigrationActionRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service';import { type DeferredWorkspaceMigrationAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/types/deferred-workspace-migration-action.type';@@ -47,6 +49,7 @@ export class WorkspaceMigrationRunnerService {private readonly twentyConfigService: TwentyConfigService,private readonly featureFlagService: FeatureFlagService,private readonly deferredWorkspaceMigrationActionRunnerService: DeferredWorkspaceMigrationActionRunnerService,+ private readonly inFlightDeferredWorkspaceMigrationActionsService: InFlightDeferredWorkspaceMigrationActionsService,) {}private getLegacyCacheInvalidation(@@ -196,6 +199,8 @@ export class WorkspaceMigrationRunnerService {}> => {const runStart = performance.now();+ await this.throwIfSchemaAffectingDeferredActionsAreInProgress(args);+try {const result = await this.executeRun(args);@@ -221,6 +226,30 @@ export class WorkspaceMigrationRunnerService {}};+ private throwIfSchemaAffectingDeferredActionsAreInProgress = async ({+ workspaceMigration: { actions },+ workspaceId,+ }: {+ workspaceMigration: WorkspaceMigration;+ workspaceId: string;+ }): Promise<void> => {+ const featureFlagsMap =+ await this.featureFlagService.getWorkspaceFeatureFlagsMap(workspaceId);++ if (+ !featureFlagsMap[+ FeatureFlagKey.IS_DEFERRED_WORKSPACE_MIGRATION_ACTIONS_ENABLED+ ] ||+ !isSchemaAffectingWorkspaceMigration(actions)+ ) {+ return;+ }++ await this.inFlightDeferredWorkspaceMigrationActionsService.throwIfSchemaAffectingActionsAreInProgress(+ workspaceId,+ );+ };+private executeRun = async ({workspaceMigration: { actions, applicationUniversalIdentifier },workspaceId,diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/__tests__/is-schema-affecting-workspace-migration.util.spec.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/__tests__/is-schema-affecting-workspace-migration.util.spec.tsnew file mode 100644index 00000000000..1f85f143271--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/__tests__/is-schema-affecting-workspace-migration.util.spec.ts@@ -0,0 +1,33 @@+import { type AllMetadataName } from 'twenty-shared/metadata';++import { isSchemaAffectingWorkspaceMigration } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/is-schema-affecting-workspace-migration.util';++const buildAction = (metadataName: AllMetadataName) => ({ metadataName });++describe('isSchemaAffectingWorkspaceMigration', () => {+ it.each<AllMetadataName>(['objectMetadata', 'fieldMetadata', 'index'])(+ 'should return true for a migration touching %s',+ (metadataName) => {+ expect(+ isSchemaAffectingWorkspaceMigration([+ buildAction('view'),+ buildAction(metadataName),+ ]),+ ).toBe(true);+ },+ );++ it('should return false for a migration leaving the workspace schema alone', () => {+ expect(+ isSchemaAffectingWorkspaceMigration([+ buildAction('view'),+ buildAction('viewField'),+ buildAction('workflow'),+ ]),+ ).toBe(false);+ });++ it('should return false for an empty migration', () => {+ expect(isSchemaAffectingWorkspaceMigration([])).toBe(false);+ });+});diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/is-schema-affecting-workspace-migration.util.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/is-schema-affecting-workspace-migration.util.tsnew file mode 100644index 00000000000..b120e86b0f9--- /dev/null+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/utils/is-schema-affecting-workspace-migration.util.ts@@ -0,0 +1,12 @@+import { SCHEMA_AFFECTING_WORKSPACE_MIGRATION_METADATA_NAMES } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/constants/schema-affecting-workspace-migration-metadata-names.constant';+import { type AllUniversalWorkspaceMigrationAction } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-builder/types/workspace-migration-action-common';++export const isSchemaAffectingWorkspaceMigration = (+ actions: Pick<AllUniversalWorkspaceMigrationAction, 'metadataName'>[],+): boolean =>+ actions.some(({ metadataName }) =>+ SCHEMA_AFFECTING_WORKSPACE_MIGRATION_METADATA_NAMES.some(+ (schemaAffectingMetadataName) =>+ schemaAffectingMetadataName === metadataName,+ ),+ );diff --git a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.ts b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.tsindex e86d73d928d..f038b0779e8 100644--- a/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.ts+++ b/packages/twenty-server/src/engine/workspace-manager/workspace-migration/workspace-migration-runner/workspace-migration-runner.module.ts@@ -21,6 +21,7 @@ import { WorkspaceMigrationRunnerActionHandlerRegistryService } from 'src/engineimport { DeferredWorkspaceMigrationActionRecoveryService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-recovery.service';import { DeferredWorkspaceMigrationActionGaugeService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-gauge.service';import { DeferredWorkspaceMigrationActionRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/deferred-workspace-migration-action-runner.service';+import { InFlightDeferredWorkspaceMigrationActionsService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/in-flight-deferred-workspace-migration-actions.service';import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/workspace-migration/workspace-migration-runner/services/workspace-migration-runner.service';@Module({@@ -45,6 +46,7 @@ import { WorkspaceMigrationRunnerService } from 'src/engine/workspace-manager/woWorkspaceMigrationRunnerActionHandlerRegistryService,DeferredWorkspaceMigrationActionRunnerService,DeferredWorkspaceMigrationActionRecoveryService,+ InFlightDeferredWorkspaceMigrationActionsService,provideWorkspaceScopedRepository(DeferredWorkspaceMigrationActionEntity),DeferredWorkspaceMigrationActionGaugeService,FlatCacheInvalidateCommand,diff --git a/packages/twenty-server/test/integration/metadata/suites/deferred-workspace-migration-action/deferred-workspace-migration-actions.integration-spec.ts b/packages/twenty-server/test/integration/metadata/suites/deferred-workspace-migration-action/deferred-workspace-migration-actions.integration-spec.tsindex f5d6544aced..ae2ef6920d5 100644--- a/packages/twenty-server/test/integration/metadata/suites/deferred-workspace-migration-action/deferred-workspace-migration-actions.integration-spec.ts+++ b/packages/twenty-server/test/integration/metadata/suites/deferred-workspace-migration-action/deferred-workspace-migration-actions.integration-spec.ts@@ -73,16 +73,20 @@ describe('Deferred workspace migration actions', () => {});afterAll(async () => {- await updateOneObjectMetadata({- expectToFail: false,- input: {- idToUpdate: objectMetadataId,- updatePayload: { isActive: false },- },+ await expectEventually(async () => {+ await updateOneObjectMetadata({+ expectToFail: false,+ input: {+ idToUpdate: objectMetadataId,+ updatePayload: { isActive: false },+ },+ });});- await deleteOneObjectMetadata({- expectToFail: false,- input: { idToDelete: objectMetadataId },+ await expectEventually(async () => {+ await deleteOneObjectMetadata({+ expectToFail: false,+ input: { idToDelete: objectMetadataId },+ });});await updateFeatureFlag({featureFlag: