Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

21.1 KB
Raw Blame
import { spawn, ChildProcess } from './childProcess.js';
import { resolveWindowsExecutable, prepareWindowsSafeSpawn } from './aiToolkit/internal/windowsSafeSpawn.js';

// Keep the host's public API while sharing the toolkit's self-contained leaf.
export { resolveWindowsExecutable, prepareWindowsSafeSpawn };
import { withSpawnCwdEnv } from './spawnCwd.js';

/**
 * Shared buffered-spawn + Windows kill-tree machinery used by the app build and
 * update services (`server/services/appBuilder.js`, `server/services/appUpdater.js`).
 *
 * The two call sites historically carried near-identical copies of this logic;
 * they differed only in their result contract:
 *   - appBuilder maps failures to HTTP codes without try/catch, so it needs a
 *     structured non-throwing result.
 *   - appUpdater throws on failure.
 *
 * This module provides ONE structured-result core (`bufferedSpawn`) plus a thin
 * throwing adapter (`bufferedSpawnOrThrow`) so both contracts share the same
 * spawn / buffering / timeout / kill-tree code.
 *
 * Pure module-level constants and platform predicates are exported so callers
 * (and tests, on any platform) can reuse the shell-shim / kill decisions.
 */

export const IS_WIN32 = process.platform === 'win32';

// npm/npx are .cmd shims on Windows — enable shell only for these so cmd.exe
// can resolve them, without enabling shell metacharacter interpretation for
// native binaries (xcodebuild, swift, make, cargo, git, …).
export const WIN_CMD_SHIMS = new Set(['npm', 'npx']);

/**
 * True when a command must run through cmd.exe to be resolved on Windows.
 * Pure and platform-independent in shape — the `IS_WIN32` gate keeps it false
 * everywhere else.
 */
export const needsShell = (cmd) => IS_WIN32 && WIN_CMD_SHIMS.has(cmd);

/**
 * Compose `resolveWindowsExecutable` + `prepareWindowsSafeSpawn` into the single
 * `{ command, args }` pair a caller should hand to `spawn()` under the default
 * `shell: false`. This is the canonical fix for spawning a bare npm-installed
 * CLI provider (`opencode`, `codex`, `claude`, …) — a `.cmd`/`.bat` shim on
 * Windows — safely: resolve the bare name to its explicit-extension path, then
 * wrap a `.cmd`/`.bat` target as `cmd.exe /c <path> <args>` (see the two
 * helpers' docstrings for why a direct `.cmd` spawn fails post-CVE-2024-27980
 * and why `shell:true` is unsafe).
 *
 * Every terminal condition off-Windows is a no-op: `resolveWindowsExecutable`
 * returns `null` (so the bare `command` is kept) and `prepareWindowsSafeSpawn`
 * returns `{ command, args }` unchanged — POSIX callers get exactly what they
 * passed in.
 *
 * Resolution reads `searchEnv.PATH`/`.Path` — pass the actual env the child
 * will run under (after merging a provider's `envVars`) so a per-provider PATH
 * override is honored; defaults to `process.env`.
 *
 * The `server/services/runner.js`, `visionCli.js`, and `cliProviderRun.js`
 * paths pre-date this helper and still inline the two-step form; new callers
 * (the Chief-of-Staff agent spawners) use this instead.
 *
 * @param {string} command - bare command name, or an existing path
 * @param {string[]} args
 * @param {NodeJS.ProcessEnv} [searchEnv] - env to read PATH from; defaults to `process.env`
 * @param {boolean} [isWin32] - injectable for tests; defaults to the real platform
 * @returns {{ command: string, args: string[] }} launchable pair for `spawn()`
 */
export function prepareCliSpawn(command, args, searchEnv = process.env, isWin32 = IS_WIN32) {
  const resolved = resolveWindowsExecutable(command, isWin32, searchEnv) || command;
  return prepareWindowsSafeSpawn(resolved, args, isWin32, searchEnv);
}

// Cap buffered stdout/stderr so a runaway child can't exhaust memory; we only
// ever surface a tail of the output anyway.
export const MAX_OUTPUT_BYTES = 64 * 1024;

/**
 * Terminate a child process and its descendants.
 *
 * On Windows, SIGTERM kills the cmd.exe shim but orphans its child (npm), so we
 * use `taskkill /T /F` to take down the whole process tree. The taskkill spawn
 * is fire-and-forget: its own `error` is swallowed and the handle is unref'd so
 * it never keeps the event loop alive. Elsewhere, a plain SIGTERM suffices.
 *
 * `child.killed` is set synchronously on the win32 branch (mirroring what
 * Node's own `child.kill()` does for the POSIX branch) — callers elsewhere
 * gate re-entrant kill/abort handling on `.killed`, and `taskkill` runs in a
 * separate detached process that never touches the original ChildProcess
 * object, so without this the flag would stay `false` for the process's
 * entire lifetime and those guards would never engage on Windows.
 *
 * The win32 branch is gated on the target OWNING a `.kill()` of its own — some
 * callers (the aiToolkit runner's `stopRun`, via `registerExternalRun`) pass
 * this a killable that isn't a `child_process` spawn at all, e.g. a node-pty
 * `IPty` TUI session, which also exposes `.kill()`/`.pid`. A raw `taskkill`
 * against a pty's pid bypasses node-pty's own Windows teardown (releasing its
 * native ConPTY handle), leaking it — so any non-ChildProcess killable that
 * brings its own `.kill()` always uses it instead, on every platform.
 *
 * A bare `{ pid }` target is the third shape: a process this server never held
 * a handle to at all — `spawnDetached`'s Windows supervisor job, and the orphan
 * `reapDetached` finds through a control dir's pid file — so there is no
 * `.kill()` to prefer and the pid is the only address it has. It takes the same
 * tree-kill a ChildProcess does, which is what gives a detached Windows job the
 * "the runner's ffmpeg/python descendants die with it" guarantee POSIX gets
 * from a process-group signal (#4171, #6170).
 *
 * node-pty's Windows backend additionally throws `Signals not supported on
 * windows.` for ANY signal argument, so `kill(signal)` against a pty there never
 * kills anything — it throws past the caller, which typically logs it and moves
 * on while the PTY keeps running. That is why every CoS Runner TUI kill
 * (terminate, force-kill, pause, and the server's post-finalize relay) was a
 * silent no-op on Windows: the codex/claude PTY survived, held its worktree
 * locked, and stayed in the runner's active set — which the PortOS server
 * re-adopted on every orphan sweep and counted against the Update page's
 * "N CoS agents running" gate, pausing updates indefinitely.
 *
 * So a non-ChildProcess killable is offered the caller's signal FIRST and falls
 * back to a bare `kill()` only when the handle rejects it. Dropping the signal
 * unconditionally would be wrong for the other non-ChildProcess killables in
 * this codebase, which do accept one and forward it — `cosRunnerClient`'s TUI
 * proxy relays `{ signal }` over a socket, so an unconditional drop would
 * silently downgrade a force-kill to a graceful one. A bare `kill()` still tears
 * down node-pty's whole console process list, so nothing is lost where the
 * fallback does fire: the graceful/forced distinction simply is not expressible
 * for a Windows pty.
 *
 * `processGroup` is for a child spawned `detached` on POSIX: without it the
 * non-Windows branch signals a single pid, which leaves a shell's own children
 * (the uv / pip / git an installer script shells out to) running. With it the
 * negative pid signals the whole group, falling back to the single pid when the
 * group is already gone. Windows needs no flag — `taskkill /T` is always the
 * tree. Off by default because a non-detached child shares its PARENT's group,
 * where a group signal would reach the wrong processes.
 *
 * `signal` applies to the **POSIX** branch only (`SIGTERM` default → the
 * graceful-then-`SIGKILL`-escalation pattern callers expect). On Windows there
 * is no real POSIX signal — `taskkill /T /F` force-kills the whole tree
 * regardless — so the arg is ignored there. A caller that wraps a `.cmd`/`.bat`
 * shim via `prepareCliSpawn` (its child is a `cmd.exe /c …` parent) MUST use
 * this rather than `child.kill()`, or on Windows only the `cmd.exe` shim dies
 * and the real CLI child is orphaned.
 *
 * @param {import('child_process').ChildProcess|{pid: number, kill?: Function}} child - a ChildProcess, any killable exposing `.pid`/`.kill()`, or a bare `{ pid }`
 * @param {NodeJS.Signals} [signal] - POSIX signal to send (default `SIGTERM`); ignored on Windows
 * @param {{processGroup?: boolean}} [opts] - POSIX: signal the child's process group (it must have been spawned `detached`)
 * @param {boolean} [isWin32] - platform override, injected so the Windows branches are testable off Windows (same shape as `resolveWindowsExecutable`)
 */
export function killProcessTree(child, signal = 'SIGTERM', { processGroup = false } = {}, isWin32 = IS_WIN32) {
  const isChildProcess = child instanceof ChildProcess;
  // A target with no `.kill()` of its own is addressable only by pid — nothing
  // to prefer over the platform's own tree/group kill below.
  const isPidOnly = !isChildProcess && typeof child.kill !== 'function';
  if (isWin32 && child.pid && (isChildProcess || isPidOnly)) {
    child.killed = true;
    spawn('taskkill', ['/T', '/F', '/PID', String(child.pid)], { stdio: 'ignore' })
      .on('error', () => {})
      .unref();
    return;
  }
  if (processGroup && child.pid) {
    try {
      process.kill(-child.pid, signal);
      // Node sets `.killed` only from `child.kill()`, so a group signal would
      // otherwise leave it false and defeat every `if (!child.killed)` re-entry
      // guard that gates a second kill — exactly why the Windows branch above
      // sets it by hand too. Only for a real ChildProcess: a node-pty IPty has
      // no `killed` of its own, and inventing one would start short-circuiting
      // PTY teardown paths that deliberately re-kill.
      if (isChildProcess) child.killed = true;
      return;
    }
    catch { /* ESRCH — the group is already gone; fall through to the pid */ }
  }
  if (isPidOnly) {
    // POSIX, pid-only: the group signal above either landed or the group is
    // gone, so this is the plain per-pid signal — swallowing ESRCH exactly as
    // the group attempt does, since a target that already exited is not a
    // failure to report.
    try { process.kill(child.pid, signal); }
    catch { /* ESRCH — already gone */ }
    return;
  }
  if (isWin32 && !isChildProcess) {
    // node-pty rejects the signal outright on Windows (see the note above);
    // every other killable here accepts and forwards it, so ask first and only
    // fall back to the signal-free form for the handle that refuses. Retrying a
    // kill is harmless — the first attempt did nothing.
    try { child.kill(signal); }
    catch { child.kill(); }
    return;
  }
  child.kill(signal);
}


/**
 * Stop a child gracefully with forced escalation: SIGTERM now, SIGKILL after
 * `graceMs` if the child has not exited. The tree's one SIGTERM→SIGKILL
 * pattern (the pm2 CLI runner and `bufferedSpawn`'s timeout path do the same
 * dance): a child that traps or ignores SIGTERM — an npm wrapper, docker
 * compose — must not outlive the caller's idea of "stopped" (#12021).
 *
 * Returns a cancel function for the caller's `close`/`error` handler so a
 * child that did exit on SIGTERM never gets a stray SIGKILL aimed at a pid
 * the OS may have reused. The timer is unref'd and never keeps the process
 * alive. On Windows `killProcessTree` is already `taskkill /F`, so there is
 * nothing to escalate to and the cancel is a no-op.
 *
 * @param {import('child_process').ChildProcess|{pid: number, kill?: Function}} child
 * @param {{ graceMs?: number, label?: string, processGroup?: boolean }} [opts]
 * @param {boolean} [isWin32] - platform override for tests
 * @returns {() => void} cancel the pending SIGKILL escalation
 */
export function terminateWithEscalation(child, { graceMs = 5000, label = 'child process', processGroup = false } = {}, isWin32 = IS_WIN32) {
  killProcessTree(child, 'SIGTERM', { processGroup }, isWin32);
  if (isWin32) return () => {};
  const timer = setTimeout(() => {
    if (child.exitCode !== null || child.signalCode !== null) return;
    console.warn(`⚠️ ${label} ignored SIGTERM for ${graceMs}ms — escalating to SIGKILL`);
    try {
      killProcessTree(child, 'SIGKILL', { processGroup }, isWin32);
    } catch (err) {
      console.error(`❌ ${label} SIGKILL escalation failed: ${err.message}`);
    }
  }, graceMs);
  timer.unref?.();
  return () => clearTimeout(timer);
}

/**
 * Spawn a command, buffer its (capped) stdout/stderr, enforce a timeout, and
 * resolve a structured result. NEVER rejects — every terminal condition
 * (clean exit, non-zero exit, spawn error, timeout) resolves a result object.
 *
 * The result is a superset of what both call sites need:
 *   {
 *     success: boolean,        // true iff the process exited 0 within timeout
 *     code: number|null,       // exit code; -1 for timeout/spawn-error
 *     signal: string|null,     // termination signal, when applicable
 *     stdout: string,          // captured stdout (tail-capped, NOT trimmed)
 *     stderr: string,          // captured stderr (tail-capped, NOT trimmed)
 *     timedOut: boolean,       // true iff the timeout fired
 *     error?: Error,           // present only on a spawn 'error' event
 *   }
 *
 * Callers shape this into their own contract (e.g. an `output` tail string, an
 * `exitCode` alias, or a thrown Error) — see `bufferedSpawnOrThrow` and the
 * appBuilder result mappers.
 *
 * @param {string} cmd
 * @param {string[]} args
 * @param {object} options
 * @param {string} [options.cwd]
 * @param {NodeJS.ProcessEnv} [options.env] - complete child environment
 * @param {number} [options.timeoutMs] - kill + resolve as timed-out after this
 * @param {number} [options.killGraceMs=8000] - SIGTERM grace period before fire-and-forget SIGKILL cleanup
 * @param {boolean} [options.shell] - defaults to `needsShell(cmd)`
 * @returns {Promise<object>} structured result (never rejects)
 */
export function bufferedSpawn(cmd, args, { cwd, env = process.env, timeoutMs, killGraceMs = 8000, shell } = {}) {
  return new Promise((resolve) => {
    const child = spawn(cmd, args, {
      cwd,
      // Pin PWD to the spawn cwd — see withSpawnCwdEnv (#3193). Done in this
      // shared wrapper rather than at each caller so every present and future
      // bufferedSpawn user inherits it; `withSpawnCwdEnv` returns an unchanged
      // copy of process.env when no cwd was given, matching the previous
      // implicit-inherit behavior exactly.
      env: withSpawnCwdEnv(env, cwd),
      shell: shell === undefined ? needsShell(cmd) : shell,
    });
    let stdout = '';
    let stderr = '';
    let settled = false;

    const timer = timeoutMs
      ? setTimeout(() => {
          if (!settled) {
            settled = true;
            // Fire-and-forget: the timed-out result resolves now and the
            // escalation timer is unref'd, so no cancel is held here.
            terminateWithEscalation(child, { graceMs: killGraceMs, label: cmd });
            resolve({ success: false, code: -1, signal: null, stdout, stderr, timedOut: true });
          }
        }, timeoutMs)
      : null;

    const clear = () => { if (timer) clearTimeout(timer); };

    child.stdout.on('data', (d) => {
      stdout += d;
      if (stdout.length > MAX_OUTPUT_BYTES) stdout = stdout.slice(-MAX_OUTPUT_BYTES);
    });
    child.stderr.on('data', (d) => {
      stderr += d;
      if (stderr.length > MAX_OUTPUT_BYTES) stderr = stderr.slice(-MAX_OUTPUT_BYTES);
    });
    child.on('close', (code, signal) => {
      if (!settled) {
        settled = true;
        clear();
        resolve({ success: code === 0, code, signal, stdout, stderr, timedOut: false });
      }
    });
    child.on('error', (err) => {
      if (!settled) {
        settled = true;
        clear();
        resolve({ success: false, code: -1, signal: null, stdout, stderr, timedOut: false, error: err });
      }
    });
  });
}

/**
 * Throwing adapter over `bufferedSpawn` for call sites that want an exception on
 * failure (the appUpdater contract). Resolves `{ stdout, stderr }` on a clean
 * exit; rejects on spawn error, timeout, or non-zero exit.
 *
 * @param {string} cmd
 * @param {string[]} args
 * @param {object} options - same as `bufferedSpawn`, plus optional `timeoutLabel`
 * @param {string} [options.timeoutLabel] - prefix for the timeout error message (defaults to `cmd`)
 * @returns {Promise<{stdout: string, stderr: string}>}
 */
export async function bufferedSpawnOrThrow(cmd, args, options = {}) {
  const { timeoutLabel = cmd, ...spawnOpts } = options;
  const result = await bufferedSpawn(cmd, args, spawnOpts);
  if (result.error) throw result.error;
  if (result.timedOut) {
    throw new Error(`${timeoutLabel} timed out after ${(spawnOpts.timeoutMs ?? 0) / 1000}s`);
  }
  if (!result.success) {
    throw new Error(result.stderr.trim() || `${cmd} exited with code ${result.code}`);
  }
  return { stdout: result.stdout, stderr: result.stderr };
}

/**
 * The most useful sentence from a FAILED `bufferedSpawn` result.
 *
 * Callers of a `--json` CLI need this rather than a bare exit code, and the
 * naive "last line of stderr, else stdout" walk gets it wrong for exactly those
 * tools: they print their payload to stdout even when they exit non-zero, so the
 * last stdout line is a closing `}` or `]`. Prefers the spawn error, then a
 * stderr line, then a stdout line that isn't just JSON punctuation.
 *
 * @param {{error?: Error, stderr?: string, stdout?: string}} result - a `bufferedSpawn` result
 * @param {string} fallback - used when the command failed without saying anything
 * @returns {string}
 */
export function spawnFailureDetail(result, fallback) {
  if (result?.error?.message) return result.error.message;
  const lastLine = (text) => String(text || '').trim().split(/\r?\n/).filter(Boolean).pop();
  const stderrTail = lastLine(result?.stderr);
  if (stderrTail) return stderrTail;
  const stdoutTail = lastLine(result?.stdout);
  if (stdoutTail && !/^[}\]]+,?$/.test(stdoutTail)) return stdoutTail;
  return fallback;
}

/**
 * Attach the no-op `'error'` listener every spawned child's `stdin` needs before
 * anything writes to it.
 *
 * A child that dies before reading its stdin (a CLI that isn't on PATH, or one
 * that exits on a bad flag) makes the pipe emit `EPIPE`. A stream `'error'` with
 * no listener is re-thrown by Node, and every spawn site here runs OUTSIDE the
 * Express request lifecycle — there is no `next(err)` to bubble to, so the throw
 * takes down the single server process that owns every live agent run, PTY
 * shell, media job and socket. Swallowing it is correct: the child's own
 * `'error'`/`'exit'`/`'close'` handler is the authoritative settle point and
 * reports the real cause. It is logged rather than dropped, because a child
 * that exits 0 after refusing its prompt would otherwise be recorded as a clean
 * run with no trace of the prompt never having been delivered.
 *
 * A crash guard must never be the crash: a `spawn()` that failed command lookup
 * hands back a handle with no stdio at all, and a child configured with a
 * non-pipe stdin has no emitter to listen on either. Both are a no-op here
 * rather than a `TypeError` thrown from the very line meant to prevent one.
 *
 * @param {import('child_process').ChildProcess} child
 * @returns {import('child_process').ChildProcess} the same child, for chaining
 */
export function guardChildStdin(child) {
  if (typeof child?.stdin?.on === 'function') {
    child.stdin.on('error', (err) => {
      console.warn(`⚠️ child stdin closed before the prompt was delivered: ${err?.code || err?.message || err}`);
    });
  }
  return child;
}

/**
 * Deliver a prompt on a guarded child's `stdin` and close the pipe.
 *
 * Pair with `guardChildStdin`, which must already have run — this covers the
 * OTHER half of the same hazard, a `write()` that throws *synchronously* (an
 * already-destroyed pipe, or a payload that isn't a string/Buffer). That throw
 * escapes the spawn site the same way an unlistened `'error'` does, so it is
 * caught here.
 *
 * Two things then have to happen, and both were missed by the naive
 * swallow-and-continue: the pipe is destroyed, so a child that IS still reading
 * stdin sees EOF instead of waiting forever on a write that never lands; and
 * the failure is logged, because a provider that gets an empty prompt and exits
 * 0 would otherwise be filed as a clean run with nothing to explain the empty
 * output. The child's own `'error'`/`'close'` handler stays the authoritative
 * settle point, so this deliberately does not rethrow or settle anything.
 *
 * @param {import('child_process').ChildProcess} child
 * @param {string|Buffer|null} payload - written when non-null; pass null when the
 *   prompt was already delivered by argv or a temp file and stdin only needs closing
 * @param {string} label - names the run/agent in the failure log
 * @returns {boolean} true when the prompt was handed off cleanly
 */
export function deliverChildStdin(child, payload, label) {
  if (!child?.stdin) {
    console.error(`❌ ${label} has no stdin pipe — the prompt was not delivered`);
    return false;
  }
  try {
    if (payload != null) child.stdin.write(payload);
    child.stdin.end();
    return true;
  } catch (err) {
    console.error(`❌ ${label} stdin write failed, closing the pipe: ${err?.code || err?.message || err}`);
    child.stdin?.destroy();
    return false;
  }
}