The open alternative to Salesforce, designed for AI.
TypeScript crmcrm-systemcustomergood-first-issuegraphqlhacktoberfestjavascriptmarketingmonoreponestjsopen-sourcepostgresqlreactreactjssalestypescriptweb
Generalize upgrade:background into a background wrapper for any nest command (#25131)
Paul Rastoin732e1a29f046
diff --git a/packages/twenty-docker/twenty/Dockerfile b/packages/twenty-docker/twenty/Dockerfileindex 41a6280a2f3..23f4c2f2c9e 100644--- a/packages/twenty-docker/twenty/Dockerfile+++ b/packages/twenty-docker/twenty/Dockerfile@@ -116,11 +116,11 @@ COPY --chown=1000 --from=twenty-server-build /app/node_modules /app/node_modulesCOPY --chown=1000 --from=twenty-server-build /app/packages/twenty-server/package.json /app/packages/twenty-server/COPY --chown=1000 --from=twenty-server-build /app/packages/twenty-server/dist /app/packages/twenty-server/distCOPY --chown=1000 --from=twenty-server-build /app/packages/twenty-server/patches /app/packages/twenty-server/patches-# Shell entrypoints referenced by package.json scripts (upgrade:background*)+# Shell entrypoints referenced by package.json scripts (command:prod:background*)COPY --chown=1000 --from=twenty-server-build /app/packages/twenty-server/scripts /app/packages/twenty-server/scripts# COPY preserves the committed mode, but the scripts are also invoked directly# and not only through yarn, so do not let a dropped exec bit reach the pod-RUN chmod +x /app/packages/twenty-server/scripts/upgrade-background.sh+RUN chmod +x /app/packages/twenty-server/scripts/command-background.sh# Workspace packages (dist + package.json; node_modules symlinks resolve to these)COPY --chown=1000 --from=twenty-server-build /app/packages/twenty-shared/package.json /app/packages/twenty-shared/diff --git a/packages/twenty-server/docs/UPGRADE_COMMANDS.md b/packages/twenty-server/docs/UPGRADE_COMMANDS.mdindex 57b4e4117c4..4dee830fbe5 100644--- a/packages/twenty-server/docs/UPGRADE_COMMANDS.md+++ b/packages/twenty-server/docs/UPGRADE_COMMANDS.md@@ -144,19 +144,19 @@ Expect the first Ctrl+C to look like it did nothing while a long step is running### Running detached-A foreground `upgrade` dies with the shell that started it, so a dropped `kubectl exec` tunnel, a closed laptop or an expired VPN kills the run. `scripts/upgrade-background.sh` gives it its own session with no controlling terminal and streams the output from a log file instead. Use it for long upgrades over `kubectl exec` or SSH; foreground is still right locally and in CI.+A foreground command dies with the shell that started it, so a dropped `kubectl exec` tunnel, a closed laptop or an expired VPN kills the run. `scripts/command-background.sh` wraps any nest command in its own session with no controlling terminal and streams the output from a log file instead. Use it for long runs (typically `upgrade`) over `kubectl exec` or SSH; foreground is still right locally and in CI.```bash-yarn upgrade:background [args] # start detached, then stream the log-yarn upgrade:background:logs # re-attach from another shell-yarn upgrade:background:stop # graceful stop; --now immediate, --force SIGKILL+yarn command:prod:background <command> [args] # start detached, then stream the log+yarn command:prod:background:logs # re-attach from another shell+yarn command:prod:background:stop # graceful stop; --now immediate, --force SIGKILL```-`[args]` is forwarded verbatim to `upgrade`: `-d/--dry-run`, `-v/--verbose`, `-w/--workspace-id <id>` (repeatable), `--start-from-workspace-id <id>`, `--workspace-count-limit <n>`. The two workspace selectors are mutually exclusive. `--include-slow` is not among them, it belongs to `run-instance-commands`.+`<command> [args]` is forwarded verbatim to `node dist/command/command`, so `yarn command:prod:background upgrade -w <id>` is the detached form of `yarn command:prod upgrade -w <id>`. For `upgrade` the accepted args are `-d/--dry-run`, `-v/--verbose`, `-w/--workspace-id <id>` (repeatable), `--start-from-workspace-id <id>`, `--workspace-count-limit <n>`. The two workspace selectors are mutually exclusive. `--include-slow` is not among them, it belongs to `run-instance-commands`.Ctrl+C detaches the stream only, and `logs` takes any number of concurrent readers. `logs` reports whether a run is alive before streaming, and on a finished one prints the verdict and the log tail rather than following a log that will never grow again: a segment can run for minutes without printing, so a silent log alone cannot tell a slow step from a dead process.-`stop` has three tiers, three explicit invocations with no timed escalation between them, since a segment can take many minutes and a timer would defeat the graceful path. Each prints the log tail after signalling. Only the node process is signalled, never the process group, which would also kill the wrapper that records the exit code.+`stop` has three tiers, three explicit invocations with no timed escalation between them, since a segment can take many minutes and a timer would defeat the graceful path. Each prints the log tail after signalling. Only the node process is signalled, never the process group, which would also kill the wrapper that records the exit code. Graceful handling belongs to the command, not the wrapper: `upgrade` traps `SIGTERM` and stops at its next boundary, but a command without a handler dies immediately with the same exit `143`.| Invocation | Signal | Effect || --- | --- | --- |@@ -179,7 +179,7 @@ A graceful stop is always safe to rerun: nothing is rolled back and the run resuLimits of this mode:- Ctrl+C cannot reach a detached run, so `stop` is the only graceful entry point and `130` only ever appears on foreground runs.-- Log and PID files live in `/tmp` in the container (`TWENTY_UPGRADE_LOG_FILE`, `TWENTY_UPGRADE_PID_FILE`) and are lost with the pod.+- Log and PID files live in `/tmp` in the container (`TWENTY_COMMAND_LOG_FILE`, `TWENTY_COMMAND_PID_FILE`) and are lost with the pod. There is a single PID and log file, so the wrapper runs one background command at a time per machine, enforced by a `flock` on `<pid file>.lock` held for the lifetime of the run.- The start refusal only sees this container. Nothing in `upgrade` prevents two concurrent sequences either: `upgradeMigration` has no in-progress state and the sequence runner takes no advisory lock. One upgrade at a time is an operational rule, not an enforced one.- Needs `setsid`, present in the runtime image and on Linux, absent on macOS where `start` refuses and points at the foreground command.- `terminationGracePeriodSeconds` does not apply. PID 1 in the command-runner pod is `tail -f /dev/null`, so on pod deletion the detached process is torn down without ever receiving `SIGTERM`.diff --git a/packages/twenty-server/package.json b/packages/twenty-server/package.jsonindex 6b763aa415b..e56f3a78722 100644--- a/packages/twenty-server/package.json+++ b/packages/twenty-server/package.json@@ -12,9 +12,9 @@"database:init:prod": "node dist/database/scripts/setup-db.js && yarn database:migrate:prod --force --include-slow","database:migrate:prod": "node dist/command/command run-instance-commands","clickhouse:migrate:prod": "node dist/database/clickhouse/migrations/run-migrations.js",- "upgrade:background": "./scripts/upgrade-background.sh start",- "upgrade:background:logs": "./scripts/upgrade-background.sh logs",- "upgrade:background:stop": "./scripts/upgrade-background.sh stop",+ "command:prod:background": "./scripts/command-background.sh start",+ "command:prod:background:logs": "./scripts/command-background.sh logs",+ "command:prod:background:stop": "./scripts/command-background.sh stop","typeorm": "../../node_modules/typeorm/.bin/typeorm"},"dependencies": {diff --git a/packages/twenty-server/scripts/upgrade-background.sh b/packages/twenty-server/scripts/command-background.shsimilarity index 53%rename from packages/twenty-server/scripts/upgrade-background.shrename to packages/twenty-server/scripts/command-background.shindex ce81a183245..ac6a873ef94 100755--- a/packages/twenty-server/scripts/upgrade-background.sh+++ b/packages/twenty-server/scripts/command-background.sh@@ -1,25 +1,26 @@#!/bin/shset -eu-LOG_FILE="${TWENTY_UPGRADE_LOG_FILE:-/tmp/twenty-upgrade.log}"-PID_FILE="${TWENTY_UPGRADE_PID_FILE:-/tmp/twenty-upgrade.pid}"+LOG_FILE="${TWENTY_COMMAND_LOG_FILE:-/tmp/twenty-command.log}"+PID_FILE="${TWENTY_COMMAND_PID_FILE:-/tmp/twenty-command.pid}"+LOCK_FILE="$PID_FILE.lock"TAIL_LINES=20export LOG_FILE PID_FILE-upgrade_pid() { [ -s "$PID_FILE" ] && cat "$PID_FILE" || return 1; }+command_pid() { [ -s "$PID_FILE" ] && cat "$PID_FILE" || return 1; }stream() { exec tail -f "$LOG_FILE"; }-is_our_upgrade() {+is_our_command() {[ -r "/proc/$1/cmdline" ] || return 0tr '\0' ' ' < "/proc/$1/cmdline" 2>/dev/null | grep -q 'dist/command/command'}is_running() {- pid=$(upgrade_pid) && kill -0 "$pid" 2>/dev/null && is_our_upgrade "$pid"+ pid=$(command_pid) && kill -0 "$pid" 2>/dev/null && is_our_command "$pid"}detach_hint() {- echo "Ctrl+C detaches the stream only. Use 'yarn upgrade:background:stop' to stop the run."+ echo "Ctrl+C detaches the stream only. Use 'yarn command:prod:background:stop' to stop the run."}show_tail() {@@ -32,34 +33,48 @@ last_run() {code=$(grep '^EXIT=' "$LOG_FILE" 2>/dev/null | tail -n 1 | cut -d= -f2)case "$code" in0) echo "completed" ;;- 130) echo "stopped gracefully on SIGINT — rerun to resume" ;;- 143) echo "stopped gracefully on SIGTERM — rerun to resume" ;;- 137) echo "killed (SIGKILL) — partial work possible, rerun to resume" ;;+ 130) echo "stopped by SIGINT (graceful only if the command handles it)" ;;+ 143) echo "stopped by SIGTERM (graceful only if the command handles it)" ;;+ 137) echo "killed (SIGKILL), partial work possible" ;;"") echo "left no exit code — killed, or never started" ;;*) echo "failed (exit $code)" ;;esac}start() {+ [ $# -ge 1 ] || {+ echo "usage: yarn command:prod:background <command> [args]" >&2; exit 1+ }+command -v setsid > /dev/null || {echo "Failed to start: setsid not found, cannot detach the run from this terminal" >&2- echo "Use 'yarn command:prod upgrade' to run it in the foreground instead." >&2+ echo "Use 'yarn command:prod $1' to run it in the foreground instead." >&2exit 1}+ # The lock fd is inherited by the detached run and held until it exits, so+ # two concurrent starts cannot both pass the liveness check.+ exec 9>> "$LOCK_FILE"+ if command -v flock > /dev/null; then+ flock -n 9 || {+ echo "Failed to start: already running (pid $(command_pid || echo unknown)), one background command at a time" >&2+ exit 1+ }+ fiif is_running; then- echo "Failed to start: already running (pid $(upgrade_pid))" >&2; exit 1+ echo "Failed to start: already running (pid $(command_pid)), one background command at a time" >&2+ exit 1fi: > "$LOG_FILE"; rm -f "$PID_FILE"# Not exec: the wrapper has to outlive node to record EXIT=, and stop has to# signal node alone, never the process group this would otherwise share.setsid sh -c '- node dist/command/command upgrade "$@" &+ node dist/command/command "$@" &echo $! > "$PID_FILE"wait $!echo "EXIT=$?"- ' upgrade-background "$@" < /dev/null >> "$LOG_FILE" 2>&1 &+ ' command-background "$@" < /dev/null >> "$LOG_FILE" 2>&1 &i=0while [ ! -s "$PID_FILE" ] && [ "$i" -lt 10 ]; do i=$((i + 1)); sleep 1; done@@ -69,14 +84,17 @@ start() {exit 1}- echo "Running (pid $(upgrade_pid)), logging to $LOG_FILE"+ echo "Running (pid $(command_pid)), logging to $LOG_FILE"detach_hint+ # Release the lock here so only the run itself holds it: a reader left+ # attached to the stream must not block the next start.+ exec 9>&-stream}logs() {if is_running; then- echo "Running (pid $(upgrade_pid)), following $LOG_FILE" >&2+ echo "Running (pid $(command_pid)), following $LOG_FILE" >&2detach_hint >&2streamfi@@ -87,15 +105,15 @@ logs() {stop() {is_running || { echo "Not running, nothing to stop" >&2; rm -f "$PID_FILE"; exit 1; }- pid=$(upgrade_pid)+ pid=$(command_pid)case "${1:-}" in"")kill -TERM "$pid"- echo "SIGTERM sent to $pid, it finishes the step in progress then stops (exit 143)" >&2+ echo "SIGTERM sent to $pid, a command that handles it stops at its next boundary (exit 143)" >&2sleep 1show_tail- echo "Follow with 'yarn upgrade:background:logs'. Still stuck: 'stop --now'. Last resort: 'stop --force'." >&2+ echo "Follow with 'yarn command:prod:background:logs'. Still stuck: 'stop --now'. Last resort: 'stop --force'." >&2;;--now)kill -TERM "$pid"; sleep 2; kill -TERM "$pid" 2>/dev/null || true@@ -116,5 +134,5 @@ case "${1:-}" instart) shift; start "$@" ;;logs) logs ;;stop) shift; stop "$@" ;;- *) echo "usage: $0 {start [args]|logs|stop [--now|--force]}" >&2; exit 1 ;;+ *) echo "usage: $0 {start <command> [args]|logs|stop [--now|--force]}" >&2; exit 1 ;;esac