Skip to content

The open alternative to Salesforce, designed for AI.

15.0 KB
Raw Blame

Upgrade Commands

The upgrade process relies on two types of commands:

  • Instance commands — schema and data migrations that run once at the instance level (replacing raw TypeORM migrations).
  • Workspace commands — commands that iterate over all active or suspended workspaces to apply per-workspace changes.

Both are registered via decorators and automatically discovered by the upgrade pipeline.

Instance Commands

Generating an instance command

npx nx run twenty-server:database:migrate:generate --name <name> --type <fast|slow>

This generates a timestamped file and auto-registers it in instance-commands.constant.ts — do not edit that file manually.

Fast instance commands

Fast commands run immediately during the upgrade. They are used for schema changes that could introduce breaking inconsistencies between the database and the server if delayed.

A fast command implements FastInstanceCommand and provides up / down methods:

@RegisteredInstanceCommand('1.22.0', 1775758621017)
export class AddWorkspaceIdToTotoFastInstanceCommand
  implements FastInstanceCommand
{
  public async up(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `ALTER TABLE "core"."toto" ADD "workspaceId" uuid`,
    );
  }

  public async down(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `ALTER TABLE "core"."toto" DROP COLUMN "workspaceId"`,
    );
  }
}

Slow instance commands

Slow commands are used when a potentially long-running data migration must happen before the schema change. They only run when the --include-slow flag is passed.

A slow command implements SlowInstanceCommand, which extends FastInstanceCommand with an additional runDataMigration method that executes before up:

@RegisteredInstanceCommand('1.22.0', 1775758621018, { type: 'slow' })
export class BackfillWorkspaceIdSlowInstanceCommand
  implements SlowInstanceCommand
{
  async runDataMigration(dataSource: DataSource): Promise<void> {
    // Backfill logic (can be slow — e.g. iterating over workspaces, cache recomputation)
  }

  public async up(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `ALTER TABLE "core"."toto" ALTER COLUMN "workspaceId" SET NOT NULL`,
    );
  }

  public async down(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `ALTER TABLE "core"."toto" ALTER COLUMN "workspaceId" DROP NOT NULL`,
    );
  }
}

A common pattern is to pair a fast command (add a nullable column) with a slow command (backfill existing rows, then set NOT NULL).

Workspace Commands

Workspace commands run per-workspace logic across all active or suspended workspaces. They are registered with the @RegisteredWorkspaceCommand decorator alongside nest-commander's @Command decorator:

@RegisteredWorkspaceCommand('1.22.0', 1780000002000)
@Command({
  name: 'upgrade:1-22:backfill-standard-skills',
  description:
    'Backfill standard skills for existing workspaces',
})
export class BackfillStandardSkillsCommand
  extends ActiveOrSuspendedWorkspaceCommandRunner
{
  constructor(
    protected readonly workspaceIteratorService: WorkspaceIteratorService,
    // inject any services you need
  ) {
    super(workspaceIteratorService);
  }

  override async runOnWorkspace({
    workspaceId,
    options,
  }: RunOnWorkspaceArgs): Promise<void> {
    // Per-workspace logic goes here
    // options.dryRun, options.verbose are available for free
  }
}

The base class ActiveOrSuspendedWorkspaceCommandRunner handles workspace iteration and provides --dry-run, --verbose, and workspace filter options automatically.

Applying a migration matrix: side-effect vs legacy path

Commands that build a metadata migration go through WorkspaceMigrationValidateBuildAndRunService. Two entry points exist:

  • validateBuildAndRunWorkspaceMigration (default): runs the operation matrix through the metadata side-effect engine (expandWithSideEffects) before building. The engine injects and cascades engine-owned companions (system fields and relations, the searchVector field and its GIN index, searchFieldMetadata rows, unique backing indexes). This is what the live API and application manifests rely on, so commands that create or mutate custom or application-owned metadata should use it.
  • validateBuildAndRunLegacyWorkspaceMigration: skips side-effect expansion and applies the matrix literally, exactly as it was authored.

The side-effect engine landed in v2.19. Commands authored before then declared their companions explicitly and were never designed to flow through the engine. Running them through it retroactively changes their behavior: it can hard-fail on reserved-identifier collisions (RESERVED_SYSTEM_UNIVERSAL_IDENTIFIER) and silently create rows the command never intended (for example, the deterministic searchFieldMetadata rows that the standalone upgrade:2-16:backfill-search-field-metadata backfill then re-inserts, hitting IDX_SEARCH_FIELD_METADATA_OBJECT_FIELD_UNIQUE).

twenty standard metadata never flows through the engine either: twenty-standard declares its companions itself and syncs through the FromTo path. An upgrade command that mutates twenty standard entities (isSystemBuild: true with the twenty standard application universal identifier, for example updating a standard commandMenuItem) must apply its matrix literally with the legacy method whatever its target version, as upgrade:2-41:move-message-campaign-commands-to-campaign-flag and upgrade:2-42:unpin-creation-commands-on-record-selection do.

Rule of thumb:

  • Operations on twenty standard metadata, any version → use the legacy method.
  • Target version < 2.19 → use the legacy method.
  • Target version >= 2.19 on custom or application-owned metadata → use the default side-effect method.

All pre-2.19 commands follow this rule, including upgrade:2-10:sync-call-recording-standard-objects: it builds its create-set from the static twenty-standard definition (which declares all of callRecording's fields, including the searchVector system field) and runs it through the legacy path so nothing is injected on top. Its matrix contains no searchFieldMetadata operations; the deterministic rows are created later in the same upgrade pipeline by upgrade:2-16:backfill-search-field-metadata, which derives them from the standard definition.

Known gap: the static definition does not yet declare callRecording's searchVector GIN index (every other searchable standard object declares its GIN index statically), so workspaces upgrading through 2-10 on the legacy path create the searchVector column unindexed. The static declaration plus a backfill for already-upgraded workspaces land in a follow-up (twentyhq/core-team-issues#2672), which must ship in the same release as this legacy path.

Keeping command code in the command

An upgrade command is frozen once released: self-hosters can jump several versions in one upgrade, so the code that runs for them is whatever the current release ships for that old command. Two rules follow.

  • Command logic lives in its version folder. Helpers, SQL builders, constants and legacy formats that only a command needs go under upgrade-version-command/<version>/ (or its utils/), never in src/engine or src/modules. Generic primitives (flat-entity utils, schema managers, repositories) are fine to call. The twenty/no-runtime-import-from-upgrade-command lint rule rejects runtime imports from upgrade-version-command/, except the *-upgrade-command-name.constant files that upgrade-aware entities use.
  • Runtime code never branches on migration state. Do not teach the runtime to serve both the old and the new shape. When a runtime must not run against a workspace mid-migration, fence it with one explicit check and remove that check once the command leaves the cross-upgrade window.
  • A workspace command the new runtime depends on needs a fence, or must be safe without one. On cloud the new version serves traffic before the command reaches every workspace. Instance commands get this from the upgrade-aware decorators; workspace commands do not, so decide which one applies when you write the command.

A command should also not reach into runtime services to do its work. Their behavior changes with every release; the command's must not.

Execution Order

Within a given version of Twenty, the upgrade pipeline runs commands in this order, sorted by timestamp within each group:

  1. Instance fast commands
  2. Instance slow commands
  3. Workspace commands

Workspace commands are executed sequentially across all active/suspended workspaces.

Dry run

upgrade --dry-run stops as soon as it reaches an instance command, logging Dry run stopped before instance step "<name>". It does not execute that command or any later step, and records nothing in upgradeMigration.

If the run starts within a workspace segment, pending workspace commands run with options.dryRun set until the next instance command or the end of the sequence. Each workspace command is responsible for simulating its own changes. Dry runs cannot preview the full upgrade sequence because later workspace commands may depend on schema changes made by preceding instance commands.

Interrupting a run (Ctrl+C, SIGTERM)

Ctrl+C during an upgrade stops it gracefully: the workspace being processed finishes its commands, then the run stops instead of starting the next one. Ctrl+C again forces an immediate exit, leaving the command in progress unfinished.

Rerun the command to resume. Nothing is rolled back, and the run picks up from the last command recorded in upgradeMigration.

Expect the first Ctrl+C to look like it did nothing while a long step is running: it takes effect once the step ends.

Running detached

A foreground command dies with the shell that started it, so a dropped kubectl exec tunnel, a closed laptop or an expired VPN kills the run. scripts/command-background.sh wraps any nest command in its own session with no controlling terminal and streams the output from a log file instead. Use it for long runs (typically upgrade) over kubectl exec or SSH; foreground is still right locally and in CI.

yarn command:prod:background <command> [args]   # start detached, then stream the log
yarn command:prod:background:logs               # re-attach from another shell
yarn command:prod:background:stop               # graceful stop; --now immediate, --force SIGKILL

<command> [args] is forwarded verbatim to node dist/command/command, so yarn command:prod:background upgrade -w <id> is the detached form of yarn command:prod upgrade -w <id>. For upgrade the accepted args are -d/--dry-run, -v/--verbose, -w/--workspace-id <id> (repeatable), --start-from-workspace-id <id>, --workspace-count-limit <n>. The two workspace selectors are mutually exclusive. --include-slow is not among them, it belongs to run-instance-commands.

Ctrl+C detaches the stream only, and logs takes any number of concurrent readers. logs reports whether a run is alive before streaming, and on a finished one prints the verdict and the log tail rather than following a log that will never grow again: a segment can run for minutes without printing, so a silent log alone cannot tell a slow step from a dead process.

stop has three tiers, three explicit invocations with no timed escalation between them, since a segment can take many minutes and a timer would defeat the graceful path. Each prints the log tail after signalling. Only the node process is signalled, never the process group, which would also kill the wrapper that records the exit code. Graceful handling belongs to the command, not the wrapper: upgrade traps SIGTERM and stops at its next boundary, but a command without a handler dies immediately with the same exit 143.

Invocation Signal Effect
stop one SIGTERM stops at the next boundary, exit 143
stop --now two spaced SIGTERMs immediate exit, step in progress left unfinished
stop --force SIGKILL no boundary, a multi-transaction command may leave partial work

The wrapper outlives node and appends an EXIT=<code> line, which is what logs translates:

Log Meaning
EXIT=0 completed
EXIT=130 / EXIT=143 graceful stop on SIGINT / SIGTERM
EXIT=137 SIGKILLed, from stop --force or an OOM kill
any other EXIT= failed
no EXIT= line killed without a graceful stop, wrapper included (pod replaced, host lost)

A graceful stop is always safe to rerun: nothing is rolled back and the run resumes from the last command recorded in upgradeMigration, each workspace either fully done with its segment or untouched. Rerunning after a forced kill relies on commands being idempotent.

Limits of this mode:

  • Ctrl+C cannot reach a detached run, so stop is the only graceful entry point and 130 only ever appears on foreground runs.
  • Log and PID files live in /tmp in the container (TWENTY_COMMAND_LOG_FILE, TWENTY_COMMAND_PID_FILE) and are lost with the pod. There is a single PID and log file, so the wrapper runs one background command at a time per machine, enforced by a flock on <pid file>.lock held for the lifetime of the run.
  • The start refusal only sees this container. Nothing in upgrade prevents two concurrent sequences either: upgradeMigration has no in-progress state and the sequence runner takes no advisory lock. One upgrade at a time is an operational rule, not an enforced one.
  • Needs setsid, present in the runtime image and on Linux, absent on macOS where start refuses and points at the foreground command.
  • terminationGracePeriodSeconds does not apply. PID 1 in the command-runner pod is tail -f /dev/null, so on pod deletion the detached process is torn down without ever receiving SIGTERM.

Shipping a command for a future version (deferred drops)

You can write a command for a version listed in TWENTY_NEXT_VERSIONS — typically the second half of a zero-downtime migration, e.g. dropping a column one release after its replacement ships. Pass the target version to the generator:

npx nx run twenty-server:database:migrate:generate --name <name> --type fast --version 2.20.0

It registers and boots (versions are validated against TWENTY_ALL_VERSIONS) but stays dormant — the sequence only runs TWENTY_CROSS_UPGRADE_SUPPORTED_VERSIONS (previous + current). It activates automatically when nx version:bump promotes the version to current.

CI rejects changes to next version directories by default, so add the ci:allow-next-version-upgrade-mutation label to the PR to confirm the command is meant for a future version.

Caveat: @WasRemovedInUpgrade / @WasIntroducedInUpgrade are validated against the active sequence, so a decorator pointing at a still-dormant next-version command fails boot with unknown-step-name. For a deferred drop, keep the entity's WasRemovedInUpgrade<T> type wrapper now and add the decorator only once the version is current.

See the CI workflows for how upgrade commands are exercised in continuous integration.