Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: bound gh calls in self-diagnostics and Layered Intelligence jobs with one shared runCliBounded (#11678)
[._.]/ Adam Eivy55020f533f94
diff --git a/server/lib/README.md b/server/lib/README.mdindex 5cb025a148..6765f9a282 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -432,6 +432,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `bufferedSpawn.js` | `bufferedSpawn(cmd, args, opts)` (structured non-throwing result) + `bufferedSpawnOrThrow` (throwing adapter), plus `killProcessTree`, `resolveWindowsExecutable`, `prepareWindowsSafeSpawn`, `prepareCliSpawn(command, args, env)` (composed resolve+wrap for a `spawn()`-safe pair), `needsShell`, `IS_WIN32`, `WIN_CMD_SHIMS`, `MAX_OUTPUT_BYTES`, `guardChildStdin(child)` (attach the no-op `stdin` `'error'` listener BEFORE any `stdin.write`/`stdin.end` — a child that dies before reading stdin makes the pipe emit `EPIPE`, and an unlistened stream `'error'` outside the Express request lifecycle takes the whole server process down; call it at every spawn that writes to a child's stdin) + `deliverChildStdin(child, payload, label)` (its other half: write-then-`end()` with a synchronously-throwing `write` caught, the pipe destroyed so a reading child sees EOF rather than hanging, and the failure logged so an empty-prompt run that exits 0 isn't filed as clean), `spawnFailureDetail(result, fallback)` (the most useful sentence from a failed result — prefers the spawn error, then a stderr line, then a stdout line that isn't bare JSON punctuation, because a `--json` CLI prints its payload to stdout even when it exits non-zero) — shared buffered-spawn machinery with capped stdout/stderr, timeout SIGTERM plus fire-and-forget SIGKILL escalation (`killGraceMs`, default 8s), Windows `.cmd`/`.bat` shim resolution, and `taskkill /T /F` tree-kill. `killProcessTree(child, signal, { processGroup })` takes the POSIX group down too when the child was spawned `detached` (Windows always tree-kills). A killable that is NOT a `ChildProcess` — a node-pty `IPty` TUI session — is killed through its own `.kill()`, with **no signal on Windows**: node-pty throws `Signals not supported on windows.` for any signal argument, so a signalled kill there killed nothing and threw past the caller. Used by `appBuilder.js`, `appUpdater.js`, the CoS agent spawners, and `setupScriptRunner.js`. || `setupScriptRunner.js` | `spawnSetupScript(envVars)` / `stopSetupScript(child)` / `SETUP_IMAGE_VIDEO_SCRIPT` — the one way to run `scripts/setup-image-video.sh` (shared by the Video Gen BYOV runtimes, the music engines and the MuScriptor venv). Runs it under `resolveBashBinary()` with a `toBashPath` script path, and on Windows presets the `PYTHON_BIN` the script would otherwise default to `python3` for. Cancel via `stopSetupScript`, which tree-kills so uv / pip / git die with bash. || `commandExists.js` | `commandExists(cmd, args = ['--version'], { timeoutMs = 5_000, env, cwd })` — does running `cmd args` succeed? A capability probe (spawn-based, with a deadline that settles even when the child ignores termination), not a PATH lookup like `processEnv.js`'s `whichFirst`; `env`/`cwd` let a caller check the exact child process configuration. Consolidates the two previously-private copies in `localLlm.js`/`ollamaManager.js`; callers probing a heavier CLI (e.g. `codeReview.js`'s reviewer-binary probe) pass a longer `timeoutMs`. Sibling `commandOutput(cmd, args, opts)` runs the same probe but returns the trimmed stdout (or `null` when it could not run / exited non-zero), so a caller can read a `--version` banner or a `models` listing without spawning the same child twice. |+| `runCliBounded.js` | `runCliBounded(cmd, args, { timeoutMs = 15000, ...spawnOptions })` — never-rejecting, never-hanging CLI runner resolving `{ code, stdout, stderr }`; kills a child still running at `timeoutMs` and resolves `code: -1` (stderr preserved, else `timed out`); pins PWD via `withSpawnCwdEnv`. Use it for every `gh`/`glab`/`git` call a scheduled job awaits (#11678). || `spawnCwd.js` | `resolveSpawnCwd(workspacePath, fallbackRoot, label)` — resolves and **logs** the working directory a run/agent spawns into (expanding `~`), and throws when a workspace was requested but is missing / not a directory. Behind `services/runner.js#resolveRunCwd`, which turns that throw into a normal failed-run record for the two spawning runners. Stops a bad app `repoPath` from silently spawning in the PortOS checkout (#3180). `usesCreativeDirectorScratchCwd(task)` / `creativeDirectorScratchCwd(agentId)` / `removeCreativeDirectorScratchCwd(agentId)` / `resolveAgentCliCwd({ workspacePath, fallbackRoot, task, agentId })` — Creative Director no-worktree tasks get a per-agent scratch cwd under `os.tmpdir()/portos-cd-cwd/<id>` (outside the PortOS git tree) instead of the PortOS root, so native CLI AGENTS.md / CLAUDE.md discovery cannot walk up into the repo (#4650). `removeCreativeDirectorScratchCwd` is the matching finalize cleanup. `withSpawnCwdEnv(env, cwd)` — returns a copy of `env` with `PWD` pinned to `cwd` (dropping stale case-variant keys), because `spawn({ cwd })` doesn't rewrite the inherited `PWD` and OpenCode resolves its project root as `process.env.PWD ?? process.cwd()` (#3193). Apply it at every spawn that names its own cwd — the shared wrappers (`bufferedSpawn`, `spawnDetached`) already do, so their callers inherit it. `spawnCwd.test.js` discovers cwd-passing spawns across `server/` and fails on any that neither pins nor is listed exempt.| `commandSecurity.js` | Two allowlists, one parser. `validateCommand(cmd)` gates the OPERATOR-driven runner against `ALLOWED_COMMANDS` (+ `validatePm2Command(args)`, which rejects daemon-wide `pm2 kill`/`startup`/`unstartup` and `<verb> all`). `validateUnattendedCommand(cmd)` gates the UNATTENDED lane (Layered Intelligence `cmd` sources) against the far narrower `UNATTENDED_READONLY_COMMANDS` — read-only inspection binaries only, no `npx`/`node`/`python`/`pip`/`curl`/`wget`/`brew`, since those execute network code with no shell metacharacter — then applies a per-binary SUBCOMMAND gate to the multi-purpose survivors, so `git reset --hard` / `git commit`, `find -delete` / `-exec`, and `gh`/`glab` writes (`api -X POST`, `pr merge`) are rejected while `git log`, `find -name` and `gh pr list` pass. Both share one parse + `DANGEROUS_SHELL_CHARS` body. Mirrored by the `agentGuard/` PATH shim for agentic paths. || `detachedSpawn.js` | `spawnDatabaseMaintenanceWorker(id, token)` launches only the fixed, one-use fenced offline-transfer worker and retains its supervisor evidence; it grants no ordinary launch bypass. `spawnDetached(bin, args, {controlDir,env,cwd,killProcessGroup?})` → ChildProcess-like handle for a long media job that SURVIVES `pm2 restart portos-server`. A pure-`sh` double-fork reparents the job to init (escaping pm2's PPID-based TreeKill — `detached:true` alone doesn't, since it only changes the process group); the server tails on-disk log files for `stdout`/`stderr`/`close`. Group-kill mode persists a marker so cancel, reattach, and orphan reaping terminate a group-leader wrapper plus every runtime child together. Windows reaches the same place through a powershell supervisor started by a launcher that exits (never `detached:true`, which there means DETACHED_PROCESS and leaves a console host no console to run in), writing the same control files — so both platforms survive the restart AND take the runner's children down on cancel (`taskkill /T /F` there, the process group on POSIX). Used by loraTraining + videoGen. Also exports `reattachDetached(controlDir)` / `isReattachable(controlDir)` to RE-ATTACH a survivor after a restart, `isDetachedRunning(controlDir, expectedProcess?)` with optional executable/argument validation for fixed-command control dirs, and `reapDetached` to checkpoint-kill one when re-attach isn't possible. `snapshotProcesses()` (POSIX pid/pgid/start/command table, fails closed on unreadable rows), `signalProcessGroup(pgid, signal)` and `isDetachedSupervisorCommand(command)` back database-maintenance descendant reconciliation; completed POSIX launches retire their registry record once two snapshots prove the launcher group empty. |diff --git a/server/lib/glabArgs.js b/server/lib/glabArgs.jsindex c43fb56256..8bab4e8b66 100644--- a/server/lib/glabArgs.js+++ b/server/lib/glabArgs.js@@ -1,7 +1,7 @@/*** Pure argv conventions for the GitLab CLI (`glab`) — no child-process access,* so the three runners that drive `glab` (services/gitlab.js#execGlab,- * perpetualWork.js#runCli, layeredIntelligence/runCli.js) can share one+ * perpetualWork.js#runCli, lib/runCliBounded.js) can share one* definition without inheriting each other's timeout policy. Mirrors* `cliProviderArgs.js` / `gitArgs.js`.*/diff --git a/server/lib/index.js b/server/lib/index.jsindex f73f033f38..2835d2e0ba 100644--- a/server/lib/index.js+++ b/server/lib/index.js@@ -333,6 +333,7 @@ export * from './paths.js';export * from './pathSafety.js';export * from './uploads.js';export * from './icloudFile.js';+export * from './runCliBounded.js';export * from './spawnCwd.js';export * from './sshHostAlias.js';export * from './schemaVersions.js';diff --git a/server/lib/runCliBounded.js b/server/lib/runCliBounded.jsnew file mode 100644index 0000000000..f99ab227b4--- /dev/null+++ b/server/lib/runCliBounded.js@@ -0,0 +1,37 @@+import { spawn } from './childProcess.js';+import { withSpawnCwdEnv } from './spawnCwd.js';++export const DEFAULT_CLI_TIMEOUT_MS = 15_000;++/**+ * Run a CLI, resolving `{ code, stdout, stderr }` — never rejects and never+ * hangs. A spawn error or a child still running after `timeoutMs` resolves+ * `code: -1` (the timed-out child is killed); stderr accumulated before the+ * fault is preserved, falling back to the error message / `timed out`.+ * `options` pass through to spawn; PWD is pinned to `options.cwd` (#3193).+ */+export function runCliBounded(cmd, args, { timeoutMs = DEFAULT_CLI_TIMEOUT_MS, ...options } = {}) {+ return new Promise((resolve) => {+ let stdout = '', stderr = '', settled = false, timer;+ const done = (result) => {+ if (settled) return;+ settled = true;+ clearTimeout(timer);+ resolve(result);+ };+ const child = spawn(cmd, args, {+ shell: false,+ ...options,+ env: withSpawnCwdEnv(options.env ?? process.env, options.cwd),+ });+ child.stdout?.on('data', d => { stdout += d.toString(); });+ child.stderr?.on('data', d => { stderr += d.toString(); });+ child.on('error', err => done({ code: -1, stdout, stderr: stderr || err.message }));+ child.on('close', code => done({ code, stdout, stderr }));+ timer = setTimeout(() => {+ try { child.kill(); } catch { /* already gone */ }+ done({ code: -1, stdout, stderr: stderr || 'timed out' });+ }, timeoutMs);+ timer.unref?.();+ });+}diff --git a/server/lib/runCliBounded.test.js b/server/lib/runCliBounded.test.jsnew file mode 100644index 0000000000..01f5a32832--- /dev/null+++ b/server/lib/runCliBounded.test.js@@ -0,0 +1,47 @@+import { describe, it, expect, vi, afterEach } from 'vitest';+import { EventEmitter } from 'events';++const children = [];+vi.mock('./childProcess.js', async (importOriginal) => ({+ ...(await importOriginal()),+ spawn: vi.fn(() => {+ const child = new EventEmitter();+ child.stdout = new EventEmitter();+ child.stderr = new EventEmitter();+ child.kill = vi.fn();+ children.push(child);+ return child;+ }),+}));++const { runCliBounded } = await import('./runCliBounded.js');++afterEach(() => { vi.useRealTimers(); children.length = 0; });++describe('runCliBounded', () => {+ it('kills a child that never exits and resolves a timeout result', async () => {+ vi.useFakeTimers();+ const p = runCliBounded('gh', ['issue', 'list'], { timeoutMs: 5000 });+ children[0].stderr.emit('data', Buffer.from('partial'));+ await vi.advanceTimersByTimeAsync(5000);+ expect(await p).toEqual({ code: -1, stdout: '', stderr: 'partial' });+ expect(children[0].kill).toHaveBeenCalled();+ });++ it('reports "timed out" when no stderr was produced', async () => {+ vi.useFakeTimers();+ const p = runCliBounded('gh', []);+ await vi.advanceTimersByTimeAsync(15_000);+ expect((await p).stderr).toBe('timed out');+ });++ it('resolves normally on close and on spawn error', async () => {+ const ok = runCliBounded('gh', []);+ children[0].stdout.emit('data', Buffer.from('out'));+ children[0].emit('close', 0);+ expect(await ok).toEqual({ code: 0, stdout: 'out', stderr: '' });+ const bad = runCliBounded('gh', []);+ children[1].emit('error', new Error('ENOENT'));+ expect(await bad).toMatchObject({ code: -1, stderr: 'ENOENT' });+ });+});diff --git a/server/lib/spawnCwd.test.js b/server/lib/spawnCwd.test.jsindex 31554f684b..6cd2395e45 100644--- a/server/lib/spawnCwd.test.js+++ b/server/lib/spawnCwd.test.js@@ -287,7 +287,6 @@ describe('every cwd-passing spawn pins PWD', () => {['lib/planIds.js', 'git only'],['services/repoCloner.js', 'git clone/fetch only'],['services/gitlab.js', 'glab only'],- ['services/perpetualWork.js', 'git/gh/glab probes only'],['services/pm2Standardizer.js', 'git + pm2 only'],['services/pm2.js', 'pm2 only'],['services/appDeployer.js', 'deploy shell commands, not an AI CLI'],@@ -353,7 +352,7 @@ describe('every cwd-passing spawn pins PWD', () => {// A spawn is in scope when its options carry a `cwd` — either written inline// (`spawn(cmd, args, { cwd, ... })`) or FORWARDED from a caller-supplied// options object (`spawn(cmd, args, { ...options })`, the shape of a generic- // wrapper like layeredIntelligence/runCli). The forwarding case matters as+ // wrapper like lib/runCliBounded). The forwarding case matters as// much as the inline one: a wrapper that passes a caller's cwd through is// exactly where an unpinned spawn hides from a scan that only reads literals.//@@ -430,7 +429,7 @@ describe('every cwd-passing spawn pins PWD', () => {// The forwarding-wrapper shape specifically — this one is invisible to a// scan that only reads inline `cwd:` literals, and missing it is what let an// unpinned caller-forwarded cwd through.- expect(found).toContain('services/layeredIntelligence/runCli.js');+ expect(found).toContain('lib/runCliBounded.js');});// Every EXEMPT entry must correspond to a file the scan actually finds.diff --git a/server/services/autonomousJobs/selfDiagnostics.js b/server/services/autonomousJobs/selfDiagnostics.jsindex fbd66dc9c0..28bfc16702 100644--- a/server/services/autonomousJobs/selfDiagnostics.js+++ b/server/services/autonomousJobs/selfDiagnostics.js@@ -25,8 +25,8 @@* takes injectable deps so tests drive it without a live `gh` or filesystem.*/-import { spawn } from '../../lib/childProcess.js';import { PATHS, safeJSONParse } from '../../lib/fileUtils.js'-import { withSpawnCwdEnv } from '../../lib/spawnCwd.js'+import { runCliBounded } from '../../lib/runCliBounded.js'+import { PATHS, safeJSONParse } from '../../lib/fileUtils.js'import { loadLearningData, computeWindowedStats } from '../taskLearning/store.js'// The label that marks the ONE persistent summary issue to reuse across runs, and@@ -151,25 +151,8 @@ export function buildDiagnosticsIssueBody(failing, { generatedAt = new Date().to// I/O layer — `gh` shell + the handler. Injectable deps keep them testable.// ----------------------------------------------------------------------------/** Run a CLI, resolving `{ code, stdout, stderr }` (never rejects). Pure I/O. */-function runCli(cmd, args, options = {}) {- return new Promise((resolve) => {- // Pin PWD to the spawn cwd — see withSpawnCwdEnv (#3193). `options` may- // carry a caller-supplied `cwd`, so this wrapper owns the pin for every- // caller (mirrors layeredIntelligence/runCli.js, which has the same shape).- const child = spawn(cmd, args, {- shell: false,- ...options,- env: withSpawnCwdEnv(options.env ?? process.env, options.cwd),- })- let stdout = ''- let stderr = ''- child.stdout?.on('data', d => { stdout += d.toString() })- child.stderr?.on('data', d => { stderr += d.toString() })- child.on('close', code => resolve({ code, stdout, stderr }))- child.on('error', err => resolve({ code: -1, stdout: '', stderr: err.message }))- })-}+// The injectable default `exec` for the gh calls below; bounded so a stalled gh can't wedge the job.+const runCli = runCliBounded/*** Locate OUR persistent summary issue to reuse. Matches only an open `monitoring`diff --git a/server/services/layeredIntelligence.js b/server/services/layeredIntelligence.jsindex adb3688704..84883280bd 100644--- a/server/services/layeredIntelligence.js+++ b/server/services/layeredIntelligence.js@@ -34,7 +34,6 @@ export * from './layeredIntelligence/awareness.js';export * from './layeredIntelligence/approvalFunnel.js';export * from './layeredIntelligence/gates.js';export * from './layeredIntelligence/prompt.js';-export * from './layeredIntelligence/runCli.js';export * from './layeredIntelligence/sources.js';export * from './layeredIntelligence/semanticDedup.js';export * from './layeredIntelligence/forgeFiler.js';diff --git a/server/services/layeredIntelligence/forgeFiler.js b/server/services/layeredIntelligence/forgeFiler.jsindex 72a6354838..e3cf148806 100644--- a/server/services/layeredIntelligence/forgeFiler.js+++ b/server/services/layeredIntelligence/forgeFiler.js@@ -10,7 +10,7 @@ import { safeJSONParse } from '../../lib/fileUtils.js';import { normalizeIssueState } from '../../lib/forgeIssueState.js';import { LI_LABEL, LI_BLOCKING_LABEL, LI_LABEL_SPEC, LI_BLOCKING_LABEL_SPEC } from './constants.js';import { slugMarker, extractSlugFromBody } from './dedup.js';-import { runCli } from './runCli.js';+import { runCliBounded as runCli } from '../../lib/runCliBounded.js';import { withGlabJson } from '../../lib/glabArgs.js';import { fileForgeIssue, ensureForgeIssueLabels } from '../appIssues.js';diff --git a/server/services/layeredIntelligence/runCli.js b/server/services/layeredIntelligence/runCli.jsdeleted file mode 100644index 0929f4311a..0000000000--- a/server/services/layeredIntelligence/runCli.js+++ /dev/null@@ -1,27 +0,0 @@-/**- * Layered Intelligence — shared CLI spawn primitive (#2842 split of- * layeredIntelligence.js). Its own module so the source gatherer and the forge- * filer can both use it without an import cycle.- */--import { spawn } from '../../lib/childProcess.js';-import { withSpawnCwdEnv } from '../../lib/spawnCwd.js';--/** Run a CLI, resolving `{ code, stdout, stderr }` (never rejects). */-export function runCli(cmd, args, options = {}) {- return new Promise((resolve) => {- // Pin PWD to the spawn cwd — see withSpawnCwdEnv (#3193). `options` may- // carry a caller-supplied `cwd`, so this generic wrapper owns the pin on- // every caller's behalf rather than each one remembering it.- const child = spawn(cmd, args, {- shell: false,- ...options,- env: withSpawnCwdEnv(options.env ?? process.env, options.cwd),- });- let stdout = '', stderr = '';- child.stdout?.on('data', d => { stdout += d.toString(); });- child.stderr?.on('data', d => { stderr += d.toString(); });- child.on('close', code => resolve({ code, stdout, stderr }));- child.on('error', err => resolve({ code: -1, stdout: '', stderr: err.message }));- });-}diff --git a/server/services/layeredIntelligence/sources.js b/server/services/layeredIntelligence/sources.jsindex 017485e48d..f2217d07fa 100644--- a/server/services/layeredIntelligence/sources.js+++ b/server/services/layeredIntelligence/sources.js@@ -3,7 +3,7 @@** Reads each configured source (planned work, files, HTTP, shell, LI task metrics)* with injectable deps so tests drive them without a live filesystem, network or- * forge CLI. Also owns the shared `runCli` primitive used by the forge filer.+ * forge CLI. Forge CLI calls go through lib/runCliBounded.js.*/import { join, resolve, relative, isAbsolute } from 'path';diff --git a/server/services/perpetualWork.js b/server/services/perpetualWork.jsindex 1661e0d1b8..b58382946e 100644--- a/server/services/perpetualWork.js+++ b/server/services/perpetualWork.js@@ -22,7 +22,7 @@* full recheck cadence on a blip.*/-import { spawn } from '../lib/childProcess.js';+import { runCliBounded } from '../lib/runCliBounded.js';import { readFile } from 'fs/promises';import { join } from 'path';import { emitLog } from './cosEvents.js';@@ -98,27 +98,9 @@ function redactCliDetail(stderr) {return detail.length > CLI_DETAIL_MAX_CHARS ? `${detail.slice(0, CLI_DETAIL_MAX_CHARS)}…` : detail;}-/**- * Best-effort CLI runner mirroring git.js#spawnCli (which isn't exported).- * Never rejects: a spawn error, non-zero exit, or timeout all resolve to a- * result object the detectors classify themselves.- */+/** Best-effort CLI runner: never rejects or hangs; the detectors classify the result themselves. */function runCli(cmd, args, cwd, env) {- return new Promise((resolve) => {- let stdout = '', stderr = '', settled = false;- const child = spawn(cmd, args, { cwd, shell: false, ...(env ? { env } : {}) });- const done = (result) => { if (!settled) { settled = true; clearTimeout(timer); resolve(result); } };- child.stdout.on('data', (d) => { stdout += d.toString(); });- child.stderr.on('data', (d) => { stderr += d.toString(); });- // Preserve whatever stderr accumulated before the spawn error / timeout —- // the CLI's actual fault (expired token, rate limit, network refused, wrong- // repo) lives there, and blanking it left every transient reason a bare- // fixed literal with no diagnostic content (#7551).- child.on('error', () => done({ code: -1, stdout, stderr }));- child.on('close', (code) => done({ code, stdout, stderr }));- const timer = setTimeout(() => { try { child.kill(); } catch { /* noop */ } done({ code: -1, stdout, stderr }); }, CLI_TIMEOUT_MS);- if (timer.unref) timer.unref();- });+ return runCliBounded(cmd, args, { cwd, env, timeoutMs: CLI_TIMEOUT_MS });}/**diff --git a/server/services/perpetualWork.test.js b/server/services/perpetualWork.test.jsindex 7cc2b47edd..d069cb5232 100644--- a/server/services/perpetualWork.test.js+++ b/server/services/perpetualWork.test.js@@ -882,7 +882,7 @@ describe('perpetualWork', () => {expect(resolveForgeTokenEnv).toHaveBeenCalledWith('/repo');const ghCalls = spawn.mock.calls.filter(([cmd]) => cmd === 'gh');expect(ghCalls.length).toBeGreaterThan(0);- expect(ghCalls.every(([, , options]) => !options.env)).toBe(true);+ expect(ghCalls.every(([, , options]) => options.env?.GH_TOKEN === process.env.GH_TOKEN)).toBe(true);});});@@ -899,7 +899,7 @@ describe('perpetualWork', () => {});await detectGitlabIssues(app, { issueAuthorFilter: 'any' });expect(resolveForgeTokenEnv).not.toHaveBeenCalled();- expect(spawn.mock.calls.filter(([cmd]) => cmd === 'glab').every(([, , options]) => !options.env)).toBe(true);+ expect(spawn.mock.calls.filter(([cmd]) => cmd === 'glab').every(([, , options]) => options.env?.GH_TOKEN === process.env.GH_TOKEN)).toBe(true);});it('normalizes iid + string labels and excludes MR-in-flight issues', async () => {