Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Turn a raw node-pty spawn failure into a diagnosis a human can act on.
*
* node-pty reports every POSIX launch failure as the same opaque string —
* `posix_spawn failed: No such file or directory` — with no indication of WHICH
* file was missing. Three very different faults collapse into it:
*
* 1. the target executable is gone (already ruled out upstream: the CoS runner
* resolves it on the child PATH and runs a `--version` capability probe
* before it ever opens a PTY);
* 2. the `cwd` no longer exists — a worktree reaped out from under a spawn;
* 3. **node-pty's own runtime is broken** — on macOS/Linux `pty.fork()` execs a
* sibling `spawn-helper` binary that is read from disk on EVERY spawn, so an
* emptied `server/node_modules` breaks all future spawns while the already
* loaded binding keeps the runner process itself alive and healthy-looking.
*
* (3) is the one that cost a day: `npm ci` run inside a CoS worktree whose
* `server/node_modules` was symlinked at the primary checkout empties the
* SYMLINK TARGET and then installs into a fresh real directory in the worktree —
* see the "Never run `npm ci` … from inside a CoS worktree" rule in the root
* `AGENTS.md`, which exists to prevent this. The primary checkout is
* left with an empty `server/node_modules`, `spawn-helper` is gone, and every
* subsequent agent spawn fails identically forever. Classified as a generic
* `spawn-rejected` it looks transient, so the fleet retried it: every task type
* burned MAX_TASK_RETRIES, blocked itself, and filed a meta-investigation task
* that failed the same way.
*
* The probe is what separates (3) from the rest: re-launch a trivially-known-good
* command in a known-good directory. If THAT fails too, nothing is wrong with the
* caller's request — the PTY layer itself is unusable.
*/
import { existsSync } from 'fs';
import { withSpawnCwdEnv } from './spawnCwd.js';
/**
* Message prefix marking a broken PTY runtime. `agentTuiSpawning.js` matches it
* to classify the run as actionable-and-blocked instead of retrying, the same way
* it already special-cases `Command executable unavailable:`. Changing this string
* means changing that matcher.
*/
export const PTY_UNAVAILABLE_PREFIX = 'CoS Runner PTY unavailable:';
/**
* Message prefix for a spawn whose working directory vanished before launch.
*
* Deliberately NOT matched by `agentTuiSpawning.js`: unlike a broken PTY layer this
* is transient — each retry provisions a fresh worktree at a fresh path — so it
* falls through to the retrying `spawn-rejected`. Naming it separately is what
* keeps the two from being collapsed back together; a genuinely misconfigured cwd
* still surfaces by failing every attempt and blocking on MAX_TASK_RETRIES.
*/
export const PTY_WORKSPACE_MISSING_PREFIX = 'CoS Runner workspace missing:';
/**
* A command guaranteed to exist on the host, used only to prove the PTY layer can
* still fork at all. Never runs long enough to matter: the probe kills it.
*/
const PROBE_COMMAND = process.platform === 'win32' ? 'cmd.exe' : '/bin/echo';
const PROBE_ARGS = process.platform === 'win32' ? ['/c', 'exit'] : ['portos-pty-probe'];
/**
* Can node-pty still fork? Spawns {@link PROBE_COMMAND} in `probeCwd` and kills it
* immediately — we care only whether the constructor threw.
*
* @param {object} pty - the node-pty module (injected so this stays testable)
* @param {string} probeCwd - a directory known to exist (the runner's own root)
* @returns {boolean} true when the PTY layer is usable
*/
export function probePtyRuntime(pty, probeCwd) {
// Catching here is the point of the function, not an escape from the no-try/catch
// rule: "did this throw?" IS the answer being computed. It also runs while the
// caller is already handling a failure, so a probe that explodes on its own must
// not replace the original error with its own.
try {
const probe = pty.spawn(PROBE_COMMAND, PROBE_ARGS, {
name: 'xterm-256color',
cols: 80,
rows: 24,
cwd: probeCwd,
// The probe's child is a bare `echo`, which never reads PWD — but the
// tree-wide guard in spawnCwd.test.js rightly refuses to special-case that
// judgment, since it is the kind of thing that stops being true when
// someone swaps the probe command. Pin it like every other cwd-passing
// spawn rather than claiming an exemption (#3193).
env: withSpawnCwdEnv(process.env, probeCwd),
});
try {
probe.kill();
} catch {
// Already exited — `echo` is fast enough to beat the kill. Still a success.
}
return true;
} catch {
return false;
}
}
/**
* Explain a `pty.spawn` throw.
*
* Order matters: the workspace check is a cheap `existsSync` on the exact path the
* caller asked for, and a missing worktree is a per-request fault that says nothing
* about the runtime. Only once that is ruled out do we spend a probe fork asking
* whether the PTY layer is broken for everyone.
*
* @param {Error} err - what `pty.spawn` threw
* @param {object} options
* @param {string} options.cwd - the working directory the spawn requested
* @param {string} options.probeCwd - a directory known to exist, for the probe
* @param {(probeCwd: string) => boolean} options.runtimeProbe - returns whether the
* PTY layer still works; injected so callers own the node-pty import
* @returns {{ diagnosed: boolean, message: string }} `diagnosed: false` means the
* failure matched neither known fault, so the caller should let the original error
* bubble rather than dress an unknown cause in a confident explanation. Whether a
* named fault is worth RETRYING is the caller's policy, not this function's — the
* two differ: a reaped worktree clears on the next attempt (which provisions a
* fresh one), while a broken PTY layer reproduces forever.
*/
export function diagnosePtySpawnFailure(err, { cwd, probeCwd, runtimeProbe }) {
const raw = err?.message || String(err);
if (cwd && !existsSync(cwd)) {
return {
diagnosed: true,
message: `${PTY_WORKSPACE_MISSING_PREFIX} the working directory for this spawn no longer exists. It was probably removed (a reaped worktree) between the request and the launch. Original error: ${raw}`,
};
}
if (!runtimeProbe(probeCwd)) {
return {
diagnosed: true,
message: `${PTY_UNAVAILABLE_PREFIX} node-pty cannot fork any process, so this is not specific to the requested command. The usual cause is an emptied or partially installed \`server/node_modules\` — node-pty execs its \`spawn-helper\` binary from disk on every spawn. Repair it with \`npm install --prefix server\` and restart the runner (\`pm2 restart portos-cos\`). Original error: ${raw}`,
};
}
return { diagnosed: false, message: raw };
}