Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: escalate a stopped operator command to SIGKILL and keep it tracked until close

[._.]/ Adam Eivy3c1498f3f4ae
diff --git a/server/lib/README.md b/server/lib/README.md
index 1d09969544..e8f71285c4 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -431,7 +431,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `interactiveShellResolver.js` | `resolveInteractiveShell()` — picks the shell binary a PTY session runs (Shell page, agent TUI shells). On Windows the old `COMSPEC` default meant `cmd.exe`, where no command a user would type reaches another drive: `cd I:` silently PRINTS that drive's cwd and stays put, a bare `I:\` is "not recognized as a command", and `cd 'I:\path'` is a syntax error (no single-quote quoting) — so a Windows install with its repos on a second drive was stuck on `C:`. Resolves PORTOS_SHELL → newest `pwsh.exe` under `%ProgramFiles%\PowerShell\<major>` (majors enumerated and sorted numerically, so a future 8 needs no code change) → any `pwsh.exe` on PATH via `findCommandOnPath` (scoop/choco/Store shim) → `powershell.exe` 5.1 (ships with Windows) → `cmd.exe` last. PowerShell crosses drives with every form. On POSIX: PORTOS_SHELL → existing `SHELL` → `/bin/bash` `/usr/bin/bash` `/bin/sh` `/usr/bin/sh` then zsh if present → bare `bash`/`sh` via `findCommandOnPath` → bare `sh` last — never a hard-coded missing `/bin/zsh` (PM2/containers often lack `SHELL` and zsh; node-pty then `execvp`s and the Shell page shows only `execvp(3) failed`). Memoized; `_resetInteractiveShellCache()` for tests. `resolveInteractiveShellWith({platform, env, exists, readdir, findOnPath})` is the un-memoized injectable form for tests. |
| `openFolder.js` | `openFolderInSystemExplorer(localPath)` — cross-platform "open in Finder/Explorer/Nautilus" via detached spawn; child `error` handler prevents spawn failures from crashing the process. |
| `childProcess.js` | Drop-in `child_process` replacement that defaults every spawn to `windowsHide: true` — `spawn` / `spawnSync` / `fork` / `exec` / `execSync` / `execFile` / `execFileSync`, plus a `ChildProcess` re-export. **Server runtime code must import from here, never from `child_process` directly** (enforced by `childProcess.guards.test.js`): a console-less PM2 fork spawning a console child without `CREATE_NO_WINDOW` makes Windows hand the new console to Windows Terminal, which flashes a focus-stealing window. `exec`/`execFile` carry `util.promisify.custom` so `promisify(execFile)` still resolves to `{ stdout, stderr }`. An explicit `windowsHide: false` is respected. Background: `docs/WINDOWS_CONSOLE.md`. |
-| `bufferedSpawn.js` | `bufferedSpawn(cmd, args, opts)` (structured non-throwing result) + `bufferedSpawnOrThrow` (throwing adapter), plus `killProcessTree`, `resolveWindowsExecutable`, `prepareWindowsSafeSpawn`, `prepareCliSpawn(command, args, env)` (composed resolve+wrap for a `spawn()`-safe pair), `needsShell`, `IS_WIN32`, `WIN_CMD_SHIMS`, `MAX_OUTPUT_BYTES`, `guardChildStdin(child)` (attach the no-op `stdin` `'error'` listener BEFORE any `stdin.write`/`stdin.end` — a child that dies before reading stdin makes the pipe emit `EPIPE`, and an unlistened stream `'error'` outside the Express request lifecycle takes the whole server process down; call it at every spawn that writes to a child's stdin) + `deliverChildStdin(child, payload, label)` (its other half: write-then-`end()` with a synchronously-throwing `write` caught, the pipe destroyed so a reading child sees EOF rather than hanging, and the failure logged so an empty-prompt run that exits 0 isn't filed as clean), `spawnFailureDetail(result, fallback)` (the most useful sentence from a failed result — prefers the spawn error, then a stderr line, then a stdout line that isn't bare JSON punctuation, because a `--json` CLI prints its payload to stdout even when it exits non-zero) — shared buffered-spawn machinery with capped stdout/stderr, timeout SIGTERM plus fire-and-forget SIGKILL escalation (`killGraceMs`, default 8s), Windows `.cmd`/`.bat` shim resolution, and `taskkill /T /F` tree-kill. `killProcessTree(child, signal, { processGroup })` takes the POSIX group down too when the child was spawned `detached` (Windows always tree-kills). A killable that is NOT a `ChildProcess` — a node-pty `IPty` TUI session — is killed through its own `.kill()`, with **no signal on Windows**: node-pty throws `Signals not supported on windows.` for any signal argument, so a signalled kill there killed nothing and threw past the caller. Used by `appBuilder.js`, `appUpdater.js`, the CoS agent spawners, and `setupScriptRunner.js`. |
+| `bufferedSpawn.js` | `bufferedSpawn(cmd, args, opts)` (structured non-throwing result) + `bufferedSpawnOrThrow` (throwing adapter), plus `killProcessTree`, `terminateWithEscalation(child, { graceMs, label, processGroup })` (SIGTERM now, SIGKILL after the grace if the child is still running; returns a cancel for the caller's `close` handler — the one stop-escalation helper, used by the operator command runner's stop and by `bufferedSpawn`'s own timeout), `resolveWindowsExecutable`, `prepareWindowsSafeSpawn`, `prepareCliSpawn(command, args, env)` (composed resolve+wrap for a `spawn()`-safe pair), `needsShell`, `IS_WIN32`, `WIN_CMD_SHIMS`, `MAX_OUTPUT_BYTES`, `guardChildStdin(child)` (attach the no-op `stdin` `'error'` listener BEFORE any `stdin.write`/`stdin.end` — a child that dies before reading stdin makes the pipe emit `EPIPE`, and an unlistened stream `'error'` outside the Express request lifecycle takes the whole server process down; call it at every spawn that writes to a child's stdin) + `deliverChildStdin(child, payload, label)` (its other half: write-then-`end()` with a synchronously-throwing `write` caught, the pipe destroyed so a reading child sees EOF rather than hanging, and the failure logged so an empty-prompt run that exits 0 isn't filed as clean), `spawnFailureDetail(result, fallback)` (the most useful sentence from a failed result — prefers the spawn error, then a stderr line, then a stdout line that isn't bare JSON punctuation, because a `--json` CLI prints its payload to stdout even when it exits non-zero) — shared buffered-spawn machinery with capped stdout/stderr, timeout SIGTERM plus fire-and-forget SIGKILL escalation (`killGraceMs`, default 8s), Windows `.cmd`/`.bat` shim resolution, and `taskkill /T /F` tree-kill. `killProcessTree(child, signal, { processGroup })` takes the POSIX group down too when the child was spawned `detached` (Windows always tree-kills). A killable that is NOT a `ChildProcess` — a node-pty `IPty` TUI session — is killed through its own `.kill()`, with **no signal on Windows**: node-pty throws `Signals not supported on windows.` for any signal argument, so a signalled kill there killed nothing and threw past the caller. Used by `appBuilder.js`, `appUpdater.js`, the CoS agent spawners, and `setupScriptRunner.js`. |
| `setupScriptRunner.js` | `spawnSetupScript(envVars)` / `stopSetupScript(child)` / `SETUP_IMAGE_VIDEO_SCRIPT` — the one way to run `scripts/setup-image-video.sh` (shared by the Video Gen BYOV runtimes, the music engines and the MuScriptor venv). Runs it under `resolveBashBinary()` with a `toBashPath` script path, and on Windows presets the `PYTHON_BIN` the script would otherwise default to `python3` for. Cancel via `stopSetupScript`, which tree-kills so uv / pip / git die with bash. |
| `commandExists.js` | `commandExists(cmd, args = ['--version'], { timeoutMs = 5_000, env, cwd })` — does running `cmd args` succeed? A capability probe (spawn-based, with a deadline that settles even when the child ignores termination), not a PATH lookup like `processEnv.js`'s `whichFirst`; `env`/`cwd` let a caller check the exact child process configuration. Consolidates the two previously-private copies in `localLlm.js`/`ollamaManager.js`; callers probing a heavier CLI (e.g. `codeReview.js`'s reviewer-binary probe) pass a longer `timeoutMs`. Sibling `commandOutput(cmd, args, opts)` runs the same probe but returns the trimmed stdout (or `null` when it could not run / exited non-zero), so a caller can read a `--version` banner or a `models` listing without spawning the same child twice. |
| `runCliBounded.js` | `runCliBounded(cmd, args, { timeoutMs = 15000, ...spawnOptions })` — never-rejecting, never-hanging CLI runner resolving `{ code, stdout, stderr }`; kills a child still running at `timeoutMs` and resolves `code: -1` (stderr preserved, else `timed out`); pins PWD via `withSpawnCwdEnv`. Use it for every `gh`/`glab`/`git` call a scheduled job awaits (#11678). |
diff --git a/server/lib/bufferedSpawn.js b/server/lib/bufferedSpawn.js
index f8600b4e70..d6805595ab 100644
--- a/server/lib/bufferedSpawn.js
+++ b/server/lib/bufferedSpawn.js
@@ -195,6 +195,40 @@ export function killProcessTree(child, signal = 'SIGTERM', { processGroup = fals
}
+/**
+ * Stop a child gracefully with forced escalation: SIGTERM now, SIGKILL after
+ * `graceMs` if the child has not exited. The tree's one SIGTERM→SIGKILL
+ * pattern (the pm2 CLI runner and `bufferedSpawn`'s timeout path do the same
+ * dance): a child that traps or ignores SIGTERM — an npm wrapper, docker
+ * compose — must not outlive the caller's idea of "stopped" (#12021).
+ *
+ * Returns a cancel function for the caller's `close`/`error` handler so a
+ * child that did exit on SIGTERM never gets a stray SIGKILL aimed at a pid
+ * the OS may have reused. The timer is unref'd and never keeps the process
+ * alive. On Windows `killProcessTree` is already `taskkill /F`, so there is
+ * nothing to escalate to and the cancel is a no-op.
+ *
+ * @param {import('child_process').ChildProcess|{pid: number, kill?: Function}} child
+ * @param {{ graceMs?: number, label?: string, processGroup?: boolean }} [opts]
+ * @param {boolean} [isWin32] - platform override for tests
+ * @returns {() => void} cancel the pending SIGKILL escalation
+ */
+export function terminateWithEscalation(child, { graceMs = 5000, label = 'child process', processGroup = false } = {}, isWin32 = IS_WIN32) {
+ killProcessTree(child, 'SIGTERM', { processGroup }, isWin32);
+ if (isWin32) return () => {};
+ const timer = setTimeout(() => {
+ if (child.exitCode !== null || child.signalCode !== null) return;
+ console.warn(`⚠️ ${label} ignored SIGTERM for ${graceMs}ms — escalating to SIGKILL`);
+ try {
+ killProcessTree(child, 'SIGKILL', { processGroup }, isWin32);
+ } catch (err) {
+ console.error(`❌ ${label} SIGKILL escalation failed: ${err.message}`);
+ }
+ }, graceMs);
+ timer.unref?.();
+ return () => clearTimeout(timer);
+}
+
/**
* Spawn a command, buffer its (capped) stdout/stderr, enforce a timeout, and
* resolve a structured result. NEVER rejects — every terminal condition
@@ -245,20 +279,9 @@ export function bufferedSpawn(cmd, args, { cwd, env = process.env, timeoutMs, ki
? setTimeout(() => {
if (!settled) {
settled = true;
- killProcessTree(child, 'SIGTERM');
- if (!IS_WIN32) {
- const escalationTimer = setTimeout(() => {
- try {
- if (child.exitCode === null && child.signalCode === null) {
- console.log(`⚠️ ${cmd} didn't exit on SIGTERM — escalating to SIGKILL`);
- killProcessTree(child, 'SIGKILL');
- }
- } catch (err) {
- console.error(`❌ ${cmd} SIGKILL escalation failed: ${err.message}`);
- }
- }, killGraceMs);
- escalationTimer.unref?.();
- }
+ // Fire-and-forget: the timed-out result resolves now and the
+ // escalation timer is unref'd, so no cancel is held here.
+ terminateWithEscalation(child, { graceMs: killGraceMs, label: cmd });
resolve({ success: false, code: -1, signal: null, stdout, stderr, timedOut: true });
}
}, timeoutMs)
diff --git a/server/lib/bufferedSpawn.test.js b/server/lib/bufferedSpawn.test.js
index 38cc3819e0..e74bca06b8 100644
--- a/server/lib/bufferedSpawn.test.js
+++ b/server/lib/bufferedSpawn.test.js
@@ -1,5 +1,6 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { EventEmitter } from 'events';
+import { spawn as nodeSpawn } from 'node:child_process';
import { ChildProcess } from './childProcess.js';
import { mkdtemp, rm, writeFile } from 'fs/promises';
import { join } from 'path';
@@ -31,6 +32,7 @@ const {
spawnFailureDetail,
guardChildStdin,
deliverChildStdin,
+ terminateWithEscalation,
} = await import('./bufferedSpawn.js');
/**
@@ -478,6 +480,61 @@ describe('prepareCliSpawn (composed resolve+wrap — the CoS spawn fix, #2243)',
});
});
+describe.skipIf(IS_WIN32)('terminateWithEscalation (#12021)', () => {
+ // Real children: the contract is that a process which traps SIGTERM still
+ // dies, which a kill spy cannot prove.
+ const spawnReal = (script) => new Promise((resolve) => {
+ const child = nodeSpawn(process.execPath, ['-e', script], { stdio: ['ignore', 'pipe', 'ignore'] });
+ child.stdout.once('data', () => resolve(child));
+ });
+ const closed = (child) => new Promise((resolve) => child.once('close', (code, signal) => resolve({ code, signal })));
+
+ it('SIGKILLs a child that ignores SIGTERM once the grace elapses, and close fires', async () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const child = await spawnReal("process.on('SIGTERM', () => {}); setInterval(() => {}, 1000); process.stdout.write('ready');");
+ const cancel = terminateWithEscalation(child, { graceMs: 300, label: 'stubborn child' });
+ const result = await closed(child);
+ cancel();
+ expect(result.signal).toBe('SIGKILL');
+ expect(warn).toHaveBeenCalledWith(expect.stringContaining('stubborn child ignored SIGTERM for 300ms'));
+ warn.mockRestore();
+ });
+
+ it('leaves a child that honoured SIGTERM alone: cancel clears the escalation and no SIGKILL warning fires', async () => {
+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] });
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const child = makeFakeChild();
+ const cancel = terminateWithEscalation(child, { graceMs: 300, label: 'polite child' });
+ expect(child.kill).toHaveBeenCalledWith('SIGTERM');
+ // The child exits inside the grace window; the caller's close handler cancels.
+ child.signalCode = 'SIGTERM';
+ cancel();
+ vi.advanceTimersByTime(1000);
+ expect(child.kill).toHaveBeenCalledTimes(1);
+ expect(warn).not.toHaveBeenCalled();
+ warn.mockRestore();
+ });
+
+ it('skips the SIGKILL timer when a child already exited before the grace elapsed (no stray signal at a reused pid)', () => {
+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] });
+ const child = makeFakeChild();
+ terminateWithEscalation(child, { graceMs: 300 });
+ child.exitCode = 0;
+ vi.advanceTimersByTime(1000);
+ expect(child.kill).toHaveBeenCalledTimes(1);
+ });
+
+ it('on Windows the tree kill is already forced, so there is nothing to escalate', () => {
+ const child = makeFakeChild({ pid: 4242 });
+ const taskkill = makeFakeChild();
+ spawnMock.mockReturnValueOnce(taskkill);
+ const cancel = terminateWithEscalation(child, { graceMs: 300 }, true);
+ expect(spawnMock).toHaveBeenCalledWith('taskkill', ['/T', '/F', '/PID', '4242'], expect.anything());
+ expect(cancel).toBeTypeOf('function');
+ cancel();
+ });
+});
+
describe('bufferedSpawn — structured result', () => {
it('resolves success on a clean (code 0) exit and captures stdout/stderr', async () => {
const child = makeFakeChild();
diff --git a/server/lib/validation.js b/server/lib/validation.js
index ee73383467..702c4f7736 100644
--- a/server/lib/validation.js
+++ b/server/lib/validation.js
@@ -2848,7 +2848,13 @@ export const calendarAutoConfigureBodySchema = z.object({
email: optionalString,
requestId: z.string().min(1).max(128).regex(/^[A-Za-z0-9_-]+$/).optional(),
});
-export const commandExecuteBodySchema = z.object({ command: nonEmptyString, workspacePath: nullishString });
+// timeoutMs bounds an operator command run (#12021); absent means unbounded,
+// the right default for an interactive run the operator is watching.
+export const commandExecuteBodySchema = z.object({
+ command: nonEmptyString,
+ workspacePath: nullishString,
+ timeoutMs: z.number().int().positive().max(24 * 60 * 60 * 1000).optional(),
+});
export const dataArchiveBodySchema = z.object({
daysToKeep: z.preprocess((value) => (value == null ? undefined : value), z.coerce.number().finite().min(0).optional()),
});
diff --git a/server/routes/commands.js b/server/routes/commands.js
index 4c635753fa..cf436103be 100644
--- a/server/routes/commands.js
+++ b/server/routes/commands.js
@@ -14,7 +14,7 @@ const router = Router();
// POST /api/commands/execute - Execute a command
router.post('/execute', requireHostControl, asyncHandler(async (req, res) => {
- const { command, workspacePath } = validateRequest(commandExecuteBodySchema, req.body);
+ const { command, workspacePath, timeoutMs } = validateRequest(commandExecuteBodySchema, req.body);
// Validate workspacePath if provided: must exist, be a directory, and — after
// symlinks are followed — resolve within an allowed root. Using realpath here
@@ -57,7 +57,8 @@ router.post('/execute', requireHostControl, asyncHandler(async (req, res) => {
},
(result) => {
if (commandId) broadcastToCommandSubscribers(`command:${commandId}:complete`, result);
- }
+ },
+ { timeoutMs }
);
if (!commandId) {
diff --git a/server/routes/commands.test.js b/server/routes/commands.test.js
index 457bf3548a..a7f1bfb03c 100644
--- a/server/routes/commands.test.js
+++ b/server/routes/commands.test.js
@@ -86,6 +86,8 @@ function createChildProcess() {
const child = new EventEmitter();
child.stdout = new EventEmitter();
child.stderr = new EventEmitter();
+ child.exitCode = null;
+ child.signalCode = null;
child.kill = vi.fn();
return child;
}
@@ -505,6 +507,42 @@ describe('commands routes', () => {
expect(response.status).toBe(200);
expect(response.body).toEqual({ stopped: true });
expect(child.kill).toHaveBeenCalledWith('SIGTERM');
+
+ // The child has not closed yet (it may be ignoring SIGTERM): a repeat
+ // stop inside the grace window is absorbed, not reported as gone (#12021).
+ const again = await request(app)
+ .post(`/api/commands/${started.body.commandId}/stop`);
+ expect(again.status).toBe(200);
+ expect(again.body).toEqual({ stopped: true });
+ expect(child.kill).toHaveBeenCalledTimes(1);
+
+ child.emit('close', null, 'SIGTERM');
+ const afterClose = await request(app)
+ .post(`/api/commands/${started.body.commandId}/stop`);
+ expect(afterClose.status).toBe(404);
+ expect(afterClose.body.code).toBe('NOT_ACTIVE');
+ expect(broadcastToCommandSubscribers).toHaveBeenCalledWith(
+ `command:${started.body.commandId}:complete`,
+ expect.objectContaining({ success: false, signal: 'SIGTERM', stopped: true, error: 'Stopped (SIGTERM)' }),
+ );
+ });
+
+ it('accepts an optional per-command timeoutMs and rejects a non-positive one (#12021)', async () => {
+ const child = createChildProcess();
+ spawnMock.mockReturnValue(child);
+ const { app } = createApp();
+
+ const rejected = await request(app)
+ .post('/api/commands/execute')
+ .send({ command: 'pwd', timeoutMs: 0 });
+ expect(rejected.status).toBe(400);
+ expect(spawnMock).not.toHaveBeenCalled();
+
+ const started = await request(app)
+ .post('/api/commands/execute')
+ .send({ command: 'pwd', timeoutMs: 60_000 });
+ expect(started.status).toBe(202);
+ child.emit('close', 0);
});
it('returns NOT_ACTIVE for an unknown command id', async () => {
diff --git a/server/services/commands.js b/server/services/commands.js
index 2ed2acb683..dc32af4442 100644
--- a/server/services/commands.js
+++ b/server/services/commands.js
@@ -3,10 +3,25 @@ import { logAction } from './history.js';
import { ALLOWED_COMMANDS, redactOutput, validateCommand } from '../lib/commandSecurity.js';
import { safeChildProcessEnv } from '../lib/processEnv.js';
import { withSpawnCwdEnv } from '../lib/spawnCwd.js';
+import { terminateWithEscalation } from '../lib/bufferedSpawn.js';
-// Track active commands
+// SIGTERM grace before a stopped command is SIGKILLed (#12021).
+export const STOP_GRACE_MS = 5000;
+
+// commandId → { child, label, cancelEscalation, stopping, timedOut }. An entry lives
+// until the child's `close`/`error` fires, never from a stop request: a
+// child that ignores SIGTERM is still running and still ours to escalate.
const activeCommands = new Map();
+// Stop an active entry: SIGTERM now, SIGKILL after STOP_GRACE_MS if the child
+// is still alive. Idempotent — a second request during the grace window is
+// absorbed rather than re-signalling or reporting the command as gone.
+function stopEntry(entry) {
+ if (entry.stopping) return;
+ entry.stopping = true;
+ entry.cancelEscalation = terminateWithEscalation(entry.child, { graceMs: STOP_GRACE_MS, label: entry.label });
+}
+
/**
* Execute a shell command with safety checks
*
@@ -14,8 +29,12 @@ const activeCommands = new Map();
* 1. Base command must be in allowlist
* 2. Command cannot contain shell metacharacters that enable injection
* 3. Command executed via spawn with shell:false where possible
+ *
+ * `timeoutMs` (optional, default none — the operator watches an interactive
+ * run) bounds the run: at the deadline the child is stopped through the same
+ * SIGTERM→SIGKILL path as a manual stop and the completion reports `timedOut`.
*/
-export function executeCommand(command, workspacePath, onData, onComplete) {
+export function executeCommand(command, workspacePath, onData, onComplete, { timeoutMs = null } = {}) {
const commandId = Date.now().toString(36) + Math.random().toString(36).substr(2);
const validation = validateCommand(command);
@@ -50,7 +69,22 @@ export function executeCommand(command, workspacePath, onData, onComplete) {
shell: false
});
- activeCommands.set(commandId, child);
+ const entry = { child, label: `command '${command.substring(0, 50)}'`, cancelEscalation: null, stopping: false, timedOut: false };
+ activeCommands.set(commandId, entry);
+ const timeoutTimer = timeoutMs > 0
+ ? setTimeout(() => {
+ entry.timedOut = true;
+ console.warn(`⚠️ ${entry.label} exceeded ${timeoutMs}ms — stopping it`);
+ stopEntry(entry);
+ }, timeoutMs)
+ : null;
+ timeoutTimer?.unref?.();
+ const settle = () => {
+ settled = true;
+ clearTimeout(timeoutTimer);
+ entry.cancelEscalation?.();
+ activeCommands.delete(commandId);
+ };
// Every copy of the output leaves this function redacted (#12019): each
// streamed chunk on its own, and the accumulated buffer once more at close
@@ -68,34 +102,41 @@ export function executeCommand(command, workspacePath, onData, onComplete) {
onData?.(redactOutput(text), 'stderr');
});
- child.on('close', (code) => {
+ child.on('close', (code, signal) => {
if (settled) return;
- settled = true;
- activeCommands.delete(commandId);
+ settle();
const runtime = Date.now() - startTime;
const success = code === 0;
const redacted = redactOutput(output);
+ const error = success ? null
+ : entry.timedOut ? `Timed out after ${timeoutMs}ms`
+ : entry.stopping ? `Stopped (${signal || `exit code ${code}`})`
+ : signal ? `Killed by ${signal}` : `Exit code ${code}`;
logAction('command', null, command.substring(0, 50), {
command,
workspacePath,
exitCode: code,
+ signal,
runtime,
output: redacted.substring(0, 10000) // Truncate to prevent huge history
- }, success, success ? null : `Exit code ${code}`);
+ }, success, error);
onComplete?.({
success,
exitCode: code,
+ signal,
runtime,
- output: redacted
+ output: redacted,
+ ...(error ? { error } : {}),
+ ...(entry.stopping ? { stopped: true } : {}),
+ ...(entry.timedOut ? { timedOut: true } : {}),
});
});
child.on('error', (err) => {
if (settled) return;
- settled = true;
- activeCommands.delete(commandId);
+ settle();
logAction('command', null, command.substring(0, 50), { command, workspacePath }, false, err.message);
onComplete?.({
success: false,
@@ -108,16 +149,16 @@ export function executeCommand(command, workspacePath, onData, onComplete) {
}
/**
- * Stop a running command
+ * Stop a running command: SIGTERM, then SIGKILL after STOP_GRACE_MS if it is
+ * still alive. Returns true while the command is ours to stop — including a
+ * repeat request during the grace window, which is a no-op rather than
+ * "not active" — and false once `close` has retired the entry.
*/
export function stopCommand(commandId) {
- const child = activeCommands.get(commandId);
- if (child) {
- child.kill('SIGTERM');
- activeCommands.delete(commandId);
- return true;
- }
- return false;
+ const entry = activeCommands.get(commandId);
+ if (!entry) return false;
+ stopEntry(entry);
+ return true;
}
/**
diff --git a/server/services/commands.test.js b/server/services/commands.test.js
new file mode 100644
index 0000000000..5c449a558b
--- /dev/null
+++ b/server/services/commands.test.js
@@ -0,0 +1,120 @@
+import { EventEmitter } from 'events';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+const spawnMock = vi.hoisted(() => vi.fn());
+vi.mock('../lib/childProcess.js', async (importOriginal) => ({
+ ...await importOriginal(),
+ spawn: spawnMock,
+}));
+vi.mock('./history.js', () => ({ logAction: vi.fn() }));
+
+import { logAction } from './history.js';
+import { executeCommand, stopCommand, STOP_GRACE_MS } from './commands.js';
+
+// A child that only dies when the caller says so: `kill` records the signal
+// and the test decides when `close` fires, which is exactly the shape of an
+// npm wrapper or docker compose ignoring SIGTERM (#12021).
+function createChild() {
+ const child = new EventEmitter();
+ child.stdout = new EventEmitter();
+ child.stderr = new EventEmitter();
+ child.exitCode = null;
+ child.signalCode = null;
+ child.kill = vi.fn();
+ return child;
+}
+
+describe('executeCommand stop and timeout lifecycle (#12021)', () => {
+ let warn;
+ beforeEach(() => {
+ vi.useFakeTimers();
+ warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ spawnMock.mockReset();
+ logAction.mockReset();
+ });
+ afterEach(() => {
+ warn.mockRestore();
+ vi.useRealTimers();
+ });
+
+ it('keeps a stopped command active through the SIGTERM grace, escalates to SIGKILL, and retires it only on close', () => {
+ const child = createChild();
+ spawnMock.mockReturnValue(child);
+ const onComplete = vi.fn();
+ const id = executeCommand('pwd', null, null, onComplete);
+
+ expect(stopCommand(id)).toBe(true);
+ expect(child.kill).toHaveBeenCalledWith('SIGTERM');
+ // A second stop during the grace window is a no-op, not "not active".
+ expect(stopCommand(id)).toBe(true);
+ expect(child.kill).toHaveBeenCalledTimes(1);
+
+ vi.advanceTimersByTime(STOP_GRACE_MS - 1);
+ expect(child.kill).toHaveBeenCalledTimes(1);
+ vi.advanceTimersByTime(1);
+ expect(child.kill).toHaveBeenLastCalledWith('SIGKILL');
+ expect(warn).toHaveBeenCalledWith(expect.stringContaining('ignored SIGTERM'));
+ expect(onComplete).not.toHaveBeenCalled();
+
+ child.signalCode = 'SIGKILL';
+ child.emit('close', null, 'SIGKILL');
+ expect(onComplete).toHaveBeenCalledWith(expect.objectContaining({
+ success: false, exitCode: null, signal: 'SIGKILL', stopped: true, error: 'Stopped (SIGKILL)',
+ }));
+ expect(logAction).toHaveBeenCalledWith('command', null, 'pwd', expect.objectContaining({ signal: 'SIGKILL' }), false, 'Stopped (SIGKILL)');
+ expect(stopCommand(id)).toBe(false);
+ });
+
+ it('cancels the escalation when the child honours SIGTERM inside the grace window', () => {
+ const child = createChild();
+ spawnMock.mockReturnValue(child);
+ const onComplete = vi.fn();
+ const id = executeCommand('pwd', null, null, onComplete);
+
+ stopCommand(id);
+ child.signalCode = 'SIGTERM';
+ child.emit('close', null, 'SIGTERM');
+ vi.advanceTimersByTime(STOP_GRACE_MS * 2);
+ expect(child.kill).toHaveBeenCalledTimes(1);
+ expect(onComplete).toHaveBeenCalledTimes(1);
+ expect(warn).not.toHaveBeenCalled();
+ });
+
+ it('bounds a run with timeoutMs through the same stop path and reports timedOut', () => {
+ const child = createChild();
+ spawnMock.mockReturnValue(child);
+ const onComplete = vi.fn();
+ executeCommand('pwd', null, null, onComplete, { timeoutMs: 1000 });
+
+ vi.advanceTimersByTime(999);
+ expect(child.kill).not.toHaveBeenCalled();
+ vi.advanceTimersByTime(1);
+ expect(child.kill).toHaveBeenCalledWith('SIGTERM');
+ vi.advanceTimersByTime(STOP_GRACE_MS);
+ expect(child.kill).toHaveBeenLastCalledWith('SIGKILL');
+
+ child.emit('close', null, 'SIGKILL');
+ expect(onComplete).toHaveBeenCalledWith(expect.objectContaining({
+ success: false, timedOut: true, stopped: true, error: 'Timed out after 1000ms',
+ }));
+ });
+
+ it('runs unbounded by default and clears a pending timeout when the command exits on its own', () => {
+ const child = createChild();
+ spawnMock.mockReturnValue(child);
+ const onComplete = vi.fn();
+ executeCommand('pwd', null, null, onComplete, { timeoutMs: 1000 });
+ child.exitCode = 0;
+ child.emit('close', 0, null);
+ vi.advanceTimersByTime(5000);
+ expect(child.kill).not.toHaveBeenCalled();
+ expect(onComplete).toHaveBeenCalledWith(expect.objectContaining({ success: true, exitCode: 0 }));
+ expect(onComplete.mock.calls[0][0]).not.toHaveProperty('error');
+
+ const unbounded = createChild();
+ spawnMock.mockReturnValue(unbounded);
+ executeCommand('pwd', null, null, vi.fn());
+ vi.advanceTimersByTime(60 * 60 * 1000);
+ expect(unbounded.kill).not.toHaveBeenCalled();
+ });
+});