Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: attach child listeners before awaiting after spawn so a failed launch cannot crash the CoS Runner or wedge the GPU lane (#11761)

[._.]/ Adam Eivy5f541e668a19
diff --git a/scripts/ci-test-plan.js b/scripts/ci-test-plan.js
index 620fc7b98e..90bd9772ba 100644
--- a/scripts/ci-test-plan.js
+++ b/scripts/ci-test-plan.js
@@ -309,6 +309,9 @@ export const ALWAYS_RUN_TESTS = [
// Whole-tree scanner: any server file can log a failure through
// `console.log`, and the guard only ever sees it as a path string.
'server/logLevelConventions.test.js',
+ // Whole-tree scanner: any server file can spawn a child and await before
+ // attaching its error/exit listener (#11761), with no import edge to this file.
+ 'server/childListenerOrderConventions.test.js',
// Whole-tree scanner: any server/autofixer file can re-implement raw
// `pm2 jlist` execution/parsing outside the two designated readers (#8164).
'server/pm2JlistOwnership.guard.test.js',
diff --git a/server/AGENTS.md b/server/AGENTS.md
index 61df2416ba..98c8f94ccc 100644
--- a/server/AGENTS.md
+++ b/server/AGENTS.md
@@ -23,6 +23,10 @@ These apply to Express/service/model code. Universal constraints (no try/catch o
- **An `async` timer callback must own its rejections.** Inside `setTimeout(async …)` / `setInterval(async …)`, every `await` needs either an enclosing `try { … } catch` or a `.catch(…)` on the awaited chain. A timer fires outside the request lifecycle, so nothing holds the promise the callback returns and a rejection escapes as an unhandled rejection — fatal on Node >= 15. The CoS runner shipped that at boot: its orphan-cleanup timer awaited a state-file read/write, so one truncated `agents` file killed the runner seconds after start and PM2 restart-looped it against the same file (#5668). Log and continue in the catch unless the callback genuinely owns a failure verdict. `server/timerCallbackConventions.test.js` scans the whole server tree for this and carries no allowlist; the worked example is the watchdog in `server/services/mediaJobQueue/index.js`.
+## Child processes
+
+- **Attach a spawned child's terminal listener before the function next `await`s.** Node reports a failed spawn (ENOENT, EACCES, EMFILE) on `process.nextTick` and a fast-dying child exits while you are suspended: an unlistened `error` throws and takes the process down, an unlistened `close` strands the job `running`. Do the awaited work (a scanner read, a claim handoff) BEFORE `spawn()`, or call `bufferChildExit(child)` (`services/videoGen/generateVideoHelpers.js`) in the same tick and replay its result once the real handlers are wired (`services/imageGen/local.js`). The CoS Runner `/spawn` and `/spawn-tui` routes and the local image render each shipped this ordering bug (#11761, after #4617/#5655/#5791). `server/childListenerOrderConventions.test.js` scans the whole server tree for it and carries no allowlist.
+
## Socket handlers
- **An `async` Socket.IO handler must own its rejections.** Same rule as the timer callbacks above, same reason: Socket.IO hands the promise a listener returns to nobody, so a rejection escapes as an unhandled rejection and Node terminates the process — taking every agent run, PTY session and media job with it. Every `await` in an `async` handler under `server/sockets/` needs an enclosing `try { … } catch` (the shape `voice.js` uses on whole handler bodies) or a `.catch(…)` on the awaited chain (the shape `logs.js` uses on the lookups it falls back from). `server/sockets/asyncHandlerGuard.test.js` scans the directory for this and carries no allowlist; it names the file and the event string of any offender. Precedent: `voice:call:detach` shipped unguarded (#5661).
diff --git a/server/childListenerOrderConventions.test.js b/server/childListenerOrderConventions.test.js
new file mode 100644
index 0000000000..6b850488fb
--- /dev/null
+++ b/server/childListenerOrderConventions.test.js
@@ -0,0 +1,385 @@
+/**
+ * Repo-wide guard: a spawned child must have its terminal listener attached
+ * before the function next yields (#11761).
+ *
+ * ## The bug class
+ *
+ * const child = spawn(cli, args);
+ * const scan = await securityScans.read(id); // ← real file I/O
+ * child.on('error', onError); // ← too late
+ *
+ * Node reports a failed spawn (ENOENT, EACCES, EMFILE) through
+ * `process.nextTick`, and a fast-dying child exits while the function is
+ * suspended. An `error` with no listener THROWS and, outside the request
+ * lifecycle, takes the process down; a `close`/`exit` with no listener is lost
+ * and strands the job `running`. It shipped four times before this guard
+ * (#4617, #5655, #5791, and the CoS Runner `/spawn` + `/spawn-tui` + local
+ * image render sites of #11761).
+ *
+ * ## The rule
+ *
+ * In one function body, a `spawn(` / `pty.spawn(` / `spawnDetached(` result
+ * bound to a name must, before the next `await`, have one of:
+ *
+ * - `<name>.on('error', …)` (or `once` / `addListener` / `prependListener`),
+ * - `bufferChildExit(<name>, …)`,
+ * - `<name>.onExit(…)` (node-pty).
+ *
+ * The operand of that first `await` is evaluated before the function suspends,
+ * so a listener attached inside it (`await new Promise((r) => child.on('error', r))`)
+ * counts. A spawn with no `await` after it in its block has no yield window and
+ * passes.
+ *
+ * There is no allowlist: a new violation is a bug to fix by moving the awaited
+ * work BEFORE the spawn, or by calling `bufferChildExit(child)` in the same tick.
+ *
+ * ## What this guard CANNOT see
+ *
+ * Lexer-assisted, not scope-aware (see `lib/sourceScan.js`):
+ * - Only spawns bound by `name = spawn(…)` / `const name = await spawnDetached(…)`
+ * are tracked; a spawn passed straight into a helper or chained
+ * (`spawn(…).on(…)`) is not.
+ * - The scan stops at the end of the enclosing function body. It walks out of
+ * `try`/`if`/loop blocks (skipping a `try`'s `catch`) but not out of object
+ * literals or callbacks, where it conservatively reports nothing.
+ * - Awaits inside a nested `async` function body are not the spawning
+ * function's yield point and are skipped.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { execFileSync } from 'child_process';
+import { readFileSync } from 'fs';
+import { dirname, join } from 'path';
+import { fileURLToPath } from 'url';
+import { blankLiterals, matchBracket, parseCallbackAt } from './lib/sourceScan.js';
+
+const SERVER_ROOT = dirname(fileURLToPath(import.meta.url));
+
+// `name = [await] [ns.]spawn|spawnDetached(`. `spawnSync` is excluded on purpose:
+// it returns after the child exits and emits nothing.
+const SPAWN_BINDING = /([A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)*)\s*=\s*(?:await\s+)?(?:[A-Za-z_$][\w$]*\.)?(?:spawn|spawnDetached)\s*\(/g;
+
+const escapeRe = (text) => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+
+/** `[start, end)` of every nested async function body in `blanked`. */
+function asyncBodySpans(blanked) {
+ const spans = [];
+ for (const match of blanked.matchAll(/\basync\b/g)) {
+ // Arrow / `async function` spelling.
+ const parsed = parseCallbackAt(blanked, match.index, blanked.length);
+ if (parsed && blanked[parsed.start] === '{') {
+ spans.push([parsed.start, parsed.start + parsed.text.length]);
+ continue;
+ }
+ // Method spelling: `async name(params) {`.
+ const method = /^async\s+[\w$]+\s*\(/.exec(blanked.slice(match.index, match.index + 200));
+ if (!method) continue;
+ const paramsEnd = matchBracket(blanked, match.index + method[0].length - 1);
+ if (paramsEnd === -1) continue;
+ const open = blanked.slice(paramsEnd).search(/\S/);
+ if (open === -1 || blanked[paramsEnd + open] !== '{') continue;
+ const end = matchBracket(blanked, paramsEnd + open);
+ if (end !== -1) spans.push([paramsEnd + open, end]);
+ }
+ return spans;
+}
+
+/** Index of the `{` that opens the innermost block containing `pos`, or -1. */
+function enclosingOpen(blanked, pos) {
+ let depth = 0;
+ for (let i = pos - 1; i >= 0; i -= 1) {
+ if (blanked[i] === '}') depth += 1;
+ else if (blanked[i] === '{') {
+ if (depth === 0) return i;
+ depth -= 1;
+ }
+ }
+ return -1;
+}
+
+/**
+ * Whether the block opened at `open` is a control block of the SAME function
+ * (`try`, `catch`, `finally`, `if`, `else`, loops) rather than a function body or
+ * object literal. Returns 'try' for a try block so its catch clause can be skipped.
+ */
+function controlBlockKind(blanked, open) {
+ const before = blanked.slice(0, open).trimEnd();
+ if (/\btry$/.test(before)) return 'try';
+ if (/\b(?:else|finally|do)$/.test(before)) return 'control';
+ if (!before.endsWith(')')) return null;
+ let depth = 0;
+ for (let i = before.length - 1; i >= 0; i -= 1) {
+ if (before[i] === ')') depth += 1;
+ else if (before[i] === '(') {
+ depth -= 1;
+ if (depth === 0) {
+ return /\b(?:if|for|while|switch|catch)(?:\s+await)?\s*$/.test(before.slice(0, i)) ? 'control' : null;
+ }
+ }
+ }
+ return null;
+}
+
+/**
+ * Index of the first `await` after `from` that belongs to the spawning function,
+ * or -1 when none follows before the function body ends. Leaving a `try`/`if`/loop
+ * block keeps scanning (the spawn is commonly bound inside a `try`), and a
+ * `catch` clause is skipped: it only runs when the spawn itself threw.
+ */
+function nextAwait(blanked, from, spans) {
+ let depth = 0;
+ let scope = from;
+ for (let i = from; i < blanked.length; i += 1) {
+ // Only bodies that OPEN after the spawn are nested; the enclosing async
+ // function itself starts before it and its awaits are the ones that count.
+ const span = spans.find(([start, end]) => start >= from && i >= start && i < end);
+ if (span) { i = span[1] - 1; continue; }
+ const c = blanked[i];
+ if (c === '{') depth += 1;
+ else if (c === '}') {
+ depth -= 1;
+ if (depth < 0) {
+ const open = enclosingOpen(blanked, scope);
+ const kind = open === -1 ? null : controlBlockKind(blanked, open);
+ if (!kind) return -1;
+ scope = open;
+ depth = 0;
+ const clause = kind === 'try' ? /^\s*catch\b\s*(?:\([^)]*\))?\s*\{/.exec(blanked.slice(i + 1)) : null;
+ if (clause) i = matchBracket(blanked, i + clause[0].length) - 1;
+ }
+ } else if (c === 'a' && blanked.startsWith('await', i)
+ && !/[\w$]/.test(blanked[i - 1] || '') && !/[\w$]/.test(blanked[i + 5] || '')) {
+ return i;
+ }
+ }
+ return -1;
+}
+
+/** End of the expression an `await` at `awaitIndex` operates on (the `;` or closing bracket). */
+function awaitOperandEnd(blanked, awaitIndex) {
+ let depth = 0;
+ for (let i = awaitIndex + 5; i < blanked.length; i += 1) {
+ const c = blanked[i];
+ if (c === '(' || c === '[' || c === '{') depth += 1;
+ else if (c === ')' || c === ']' || c === '}') {
+ depth -= 1;
+ if (depth < 0) return i;
+ } else if (c === ';' && depth === 0) return i;
+ }
+ return blanked.length;
+}
+
+/** Every spawned-child binding in one file's source that awaits before wiring a terminal listener. */
+export function findLateChildListeners(src) {
+ const { code, blanked } = blankLiterals(src, { includeCommentOnly: true });
+ // `code` keeps literal text, so `'error'` is still readable; `blanked` is what
+ // await/bracket walks use.
+ const spans = asyncBodySpans(blanked);
+ const offenders = [];
+ for (const match of blanked.matchAll(SPAWN_BINDING)) {
+ const name = match[1];
+ const callEnd = matchBracket(blanked, match.index + match[0].length - 1);
+ if (callEnd === -1) continue;
+ const awaitAt = nextAwait(blanked, callEnd, spans);
+ if (awaitAt === -1) continue;
+ const window = code.slice(callEnd, awaitOperandEnd(blanked, awaitAt));
+ const n = escapeRe(name);
+ const attached = new RegExp(
+ `\\b${n}\\.(?:on|once|addListener|prependListener)\\(\\s*['"\`]error['"\`]`
+ + `|\\bbufferChildExit\\(\\s*${n}\\b`
+ + `|\\b${n}\\.onExit\\(`,
+ ).test(window);
+ if (attached) continue;
+ const line = blanked.slice(0, match.index).split('\n').length;
+ const awaited = blanked.slice(awaitAt, awaitOperandEnd(blanked, awaitAt)).replace(/\s+/g, ' ').trim();
+ offenders.push(`line ${line}: ${name} is awaited past (${awaited.slice(0, 80)}) before an error/exit listener`);
+ }
+ return offenders;
+}
+
+const trackedServerSources = () => execFileSync('git', ['ls-files', '*.js', '*.mjs', '*.cjs'], {
+ cwd: SERVER_ROOT,
+ encoding: 'utf8',
+ maxBuffer: 64 * 1024 * 1024,
+}).split('\n').filter((f) => f && !f.includes('.test.'));
+
+describe('spawned children get a terminal listener before the next await (#11761)', () => {
+ it('scans the server tree, every module extension included', () => {
+ const files = trackedServerSources();
+ expect(files.length).toBeGreaterThan(200);
+ expect(files.some((f) => f.endsWith('.mjs'))).toBe(true);
+ });
+
+ it('still recognizes the spawn sites it is meant to guard', () => {
+ // Without this the scan below could go vacuously green after a refactor
+ // renames the spawn helpers.
+ const sites = trackedServerSources().filter((file) => {
+ const src = readFileSync(join(SERVER_ROOT, file), 'utf8');
+ return /\b(?:spawn|spawnDetached)\s*\(/.test(src) && [...blankLiterals(src).matchAll(SPAWN_BINDING)].length > 0;
+ });
+ expect(sites).toEqual(expect.arrayContaining([
+ 'cos-runner/index.js',
+ 'services/imageGen/local.js',
+ 'services/videoGen/spawnWatch.js',
+ ]));
+ });
+
+ it('has no spawn whose listener waits behind an await', () => {
+ const violations = [];
+ for (const file of trackedServerSources()) {
+ const src = readFileSync(join(SERVER_ROOT, file), 'utf8');
+ if (!src.includes('spawn')) continue;
+ for (const hit of findLateChildListeners(src)) violations.push(`server/${file} ${hit}`);
+ }
+ expect(
+ violations,
+ 'These children are spawned and then the function awaits before any error/exit listener is '
+ + 'attached. Node reports a failed spawn on process.nextTick and a fast-dying child exits '
+ + 'during the await: an unlistened `error` throws and kills the process, an unlistened '
+ + '`close` strands the job running (#4617, #5655, #5791, #11761).\n'
+ + 'Fix: do the awaited work BEFORE the spawn, or call `bufferChildExit(child)` (videoGen/'
+ + 'generateVideoHelpers.js) in the same tick and replay its result once the real handlers are on.\n'
+ + `Offenders:\n ${violations.join('\n ')}`,
+ ).toEqual([]);
+ });
+});
+
+// Guards the guard: if the recognizer stops seeing the broken shape, the scan
+// above goes green and the bug class walks straight back in.
+describe('the child-listener-order recognizer', () => {
+ it('flags the pre-fix CoS Runner /spawn shape', () => {
+ expect(findLateChildListeners(`
+ const claudeProcess = spawn(cmd, args, { cwd });
+ const agent = {
+ process: claudeProcess,
+ securityScan: await securityScans.read(agentId),
+ };
+ claudeProcess.on('error', (err) => {});
+ `)).toHaveLength(1);
+ });
+
+ it('flags the pre-fix /spawn-tui shape (pty.spawn, exit subscribed after the read)', () => {
+ expect(findLateChildListeners(`
+ tuiProcess = pty.spawn(ptyCommand, ptyArgs, { cwd });
+ const agent = { securityScan: await securityScans.read(agentId) };
+ tuiProcess.onExit(handler);
+ `)).toHaveLength(1);
+ });
+
+ it('flags the pre-fix local image shape (handoff awaited before listeners)', () => {
+ expect(findLateChildListeners(`
+ try {
+ proc = spawn(bin, args, opts);
+ activeProcess = proc;
+ await heavyClaim.handoffTo?.(proc.pid);
+ } catch (err) { throw err; }
+ proc.on('error', onError);
+ `)).toHaveLength(1);
+ });
+
+ it('flags a spawnDetached result that waits and a member-bound child', () => {
+ expect(findLateChildListeners(`
+ firstProc = await spawnDetached(bin, args);
+ await claim.handoffTo(firstProc.pid);
+ firstProc.on('error', onError);
+ `)).toHaveLength(1);
+ expect(findLateChildListeners(`
+ state.child = spawn(bin);
+ await prepare();
+ state.child.on('error', onError);
+ `)).toHaveLength(1);
+ });
+
+ it('walks out of a try block but skips its catch clause', () => {
+ expect(findLateChildListeners(`
+ try {
+ child = spawn(bin);
+ } catch (err) {
+ await cleanup();
+ throw err;
+ }
+ child.on('error', onError);
+ `)).toEqual([]);
+ expect(findLateChildListeners(`
+ try {
+ child = spawn(bin);
+ } catch (err) {
+ throw err;
+ }
+ await prepare();
+ child.on('error', onError);
+ `)).toHaveLength(1);
+ });
+
+ it('treats an unrelated error listener on another object as no listener', () => {
+ expect(findLateChildListeners(`
+ const child = spawn(bin);
+ other.on('error', noop);
+ await prepare();
+ child.on('error', onError);
+ `)).toHaveLength(1);
+ });
+
+ it('accepts a listener, bufferChildExit or onExit attached before the await', () => {
+ expect(findLateChildListeners(`
+ const child = spawn(bin);
+ child.on('error', onError);
+ await prepare();
+ `)).toEqual([]);
+ expect(findLateChildListeners(`
+ proc = spawn(bin);
+ const take = bufferChildExit(proc, { bufferOutput: true });
+ await heavyClaim.handoffTo?.(proc.pid);
+ `)).toEqual([]);
+ expect(findLateChildListeners(`
+ const pty1 = pty.spawn(cmd, args);
+ pty1.onExit(handler);
+ await prepare();
+ `)).toEqual([]);
+ expect(findLateChildListeners(`
+ const child = spawn(bin);
+ child.once("error", onError);
+ await prepare();
+ `)).toEqual([]);
+ });
+
+ it('accepts a listener attached inside the first await operand', () => {
+ expect(findLateChildListeners(`
+ const child = spawn(bin);
+ await new Promise((resolve, reject) => {
+ child.on('error', reject);
+ child.on('close', resolve);
+ });
+ `)).toEqual([]);
+ });
+
+ it('accepts a spawn with no await after it in its block', () => {
+ expect(findLateChildListeners(`
+ async function run() {
+ await prepare();
+ const child = spawn(bin);
+ child.stdout.on('data', onData);
+ child.on('error', onError);
+ }
+ async function other() { await somethingElse(); }
+ `)).toEqual([]);
+ });
+
+ it('ignores awaits inside a nested async function and in comments or strings', () => {
+ expect(findLateChildListeners(`
+ const child = spawn(bin);
+ // await prepare();
+ const note = 'await prepare()';
+ const later = async () => { await prepare(); };
+ child.on('error', onError);
+ `)).toEqual([]);
+ });
+
+ it('does not treat spawnSync or a plain assignment named like spawn as a spawn', () => {
+ expect(findLateChildListeners(`
+ const result = spawnSync(bin);
+ const respawn = respawnCount + 1;
+ await prepare();
+ `)).toEqual([]);
+ });
+});
diff --git a/server/cos-runner/index.js b/server/cos-runner/index.js
index 8bd53aec9a..6a60a058ca 100644
--- a/server/cos-runner/index.js
+++ b/server/cos-runner/index.js
@@ -305,6 +305,9 @@ app.post('/spawn-tui', lifecycle.spawnRoute(async (req, res) => {
// preserves the shared escaping contract for paths/args passed to cmd.exe.
const { command: ptyCommand, args: ptyArgs } = prepareCliSpawn(executable, args, childEnv);
await securityScans.register(agentId, body.data.securityScanGrant);
+ // Read BEFORE pty.spawn so onData/onExit subscribe with no await after the
+ // spawn: a PTY that exits during the read would otherwise lose its exit (#11761).
+ const securityScan = await securityScans.read(agentId);
// Both checks above cleared the executable, so a throw here is about the PTY
// layer or the workspace — not the command. node-pty collapses those into one
// opaque `posix_spawn failed: No such file or directory`, which read as a
@@ -349,7 +352,7 @@ app.post('/spawn-tui', lifecycle.spawnRoute(async (req, res) => {
// node-pty reports pid 0 for ConPTY on Windows, so a pid probe is not a
// liveness signal. onExit is the runner's authority; GET /agents reads this.
exited: false,
- securityScan: await securityScans.read(agentId),
+ securityScan,
stopSecurityScan: () => securityScans.stop(agentId).catch(err => lifecycle.reportFailure(err)),
};
activeAgents.set(agentId, agent);
@@ -565,6 +568,10 @@ app.post('/spawn', lifecycle.spawnRoute(async (req, res) => {
// Spawn the CLI process
await securityScans.register(agentId, body.data.securityScanGrant);
+ // Read BEFORE spawn: Node reports a failed spawn (ENOENT/EACCES/EMFILE) via
+ // process.nextTick, so no await may sit between spawn() and the 'error'
+ // listener or the unhandled event kills the runner (#11761).
+ const securityScan = await securityScans.read(agentId);
const claudeProcess = spawn(spawnCommand, finalSpawnArgs, {
cwd,
shell: false,
@@ -596,7 +603,7 @@ app.post('/spawn', lifecycle.spawnRoute(async (req, res) => {
stripStdoutAnsi: isStreamJson ? (text) => text : createStreamingAnsiStripper(),
stripStderrAnsi: createStreamingAnsiStripper(),
workspacePath: cwd,
- securityScan: await securityScans.read(agentId),
+ securityScan,
stopSecurityScan: () => securityScans.stop(agentId).catch(err => lifecycle.reportFailure(err)),
};
activeAgents.set(agentId, agent);
diff --git a/server/cos-runner/spawnFailure.test.js b/server/cos-runner/spawnFailure.test.js
new file mode 100644
index 0000000000..8385dc502b
--- /dev/null
+++ b/server/cos-runner/spawnFailure.test.js
@@ -0,0 +1,86 @@
+import { afterAll, beforeAll, describe, expect, it } from 'vitest';
+import { spawn } from 'child_process';
+import { createServer } from 'net';
+import { mkdtempSync, rmSync } from 'fs';
+import { tmpdir } from 'os';
+import { dirname, join } from 'path';
+import { fileURLToPath } from 'url';
+import { io as connect } from 'socket.io-client';
+
+// index.js binds its server at import, so the only faithful way to exercise the
+// real /spawn ordering is to boot it as a child process. #11761: Node reports a
+// failed spawn on process.nextTick, and the route used to await the scanner
+// evidence read between spawn() and the 'error' listener — the unhandled
+// 'error' threw, the runner's uncaughtException handler exited the process, and
+// every other agent it supervised lost its owner.
+const RUNNER = join(dirname(fileURLToPath(import.meta.url)), 'index.js');
+
+const freePort = () => new Promise((resolve, reject) => {
+ const probe = createServer();
+ probe.once('error', reject);
+ probe.listen(0, '127.0.0.1', () => {
+ const { port } = probe.address();
+ probe.close(() => resolve(port));
+ });
+});
+
+describe('cos-runner /spawn survives a failed CLI launch (#11761)', () => {
+ let runner;
+ let dataRoot;
+ let base;
+ let socket;
+ let exited = false;
+
+ beforeAll(async () => {
+ dataRoot = mkdtempSync(join(tmpdir(), 'portos-runner-spawn-'));
+ const port = await freePort();
+ base = `http://127.0.0.1:${port}`;
+ // The runner is a standalone service: drop the runner-under-test markers so its
+ // own test-data isolation guard does not treat this boot as a test process.
+ const env = Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.startsWith('VITEST')));
+ runner = spawn(process.execPath, [RUNNER], {
+ env: { ...env, NODE_ENV: 'test', PORT: String(port), HOST: '127.0.0.1', PORTOS_DATA_ROOT: dataRoot },
+ stdio: 'ignore',
+ });
+ runner.once('exit', () => { exited = true; });
+ for (let attempt = 0; attempt < 100 && !exited; attempt += 1) {
+ const ready = await fetch(`${base}/health`).then((res) => res.ok, () => false);
+ if (ready) break;
+ await new Promise((resolve) => setTimeout(resolve, 100));
+ }
+ socket = connect(base, { transports: ['websocket'] });
+ await new Promise((resolve) => socket.once('connect', resolve));
+ }, 30_000);
+
+ afterAll(() => {
+ socket?.close();
+ runner?.kill('SIGKILL');
+ rmSync(dataRoot, { recursive: true, force: true });
+ });
+
+ it('reports agent:error, drops the agent and keeps the runner alive', async () => {
+ const errored = new Promise((resolve, reject) => {
+ socket.once('agent:error', resolve);
+ runner.once('exit', () => reject(new Error('the runner process died on a failed spawn')));
+ });
+ // A removed cwd makes the launch fail with ENOENT regardless of whether the
+ // CLI is installed, so this is deterministic on any machine.
+ const response = await fetch(`${base}/spawn`, {
+ method: 'POST',
+ headers: { 'content-type': 'application/json' },
+ body: JSON.stringify({
+ agentId: 'agent-spawn-failure',
+ taskId: 'task-spawn-failure',
+ prompt: 'example',
+ workspacePath: join(dataRoot, 'removed-workspace'),
+ cliCommand: 'claude',
+ cliArgs: [],
+ }),
+ });
+ expect(response.status, await response.clone().text()).toBe(200);
+ expect(await errored).toMatchObject({ agentId: 'agent-spawn-failure' });
+ expect(exited, 'the runner process must not die on a failed spawn').toBe(false);
+ const health = await fetch(`${base}/health`).then((res) => res.json());
+ expect(health).toMatchObject({ status: 'ok', activeAgents: 0 });
+ }, 20_000);
+});
diff --git a/server/services/imageGen/local.js b/server/services/imageGen/local.js
index b630dfa1f3..5ed85631ce 100644
--- a/server/services/imageGen/local.js
+++ b/server/services/imageGen/local.js
@@ -38,7 +38,7 @@ import { createLineReader } from '../../lib/streamLines.js';
import { claimHeavyLocalJob } from '../../lib/heavyJobClaim.js';
import { safeChildProcessOptions } from '../../lib/processEnv.js';
import { IMAGE_GEN_MODE, LOCAL_IMAGEGEN_DEFAULT_MODEL } from './modes.js';
-import { parseByteProgress, formatDownloadMessage } from '../videoGen/generateVideoHelpers.js';
+import { parseByteProgress, formatDownloadMessage, bufferChildExit } from '../videoGen/generateVideoHelpers.js';
const IS_WIN = process.platform === 'win32';
@@ -741,6 +741,7 @@ export async function generateImage({ pythonPath, prompt = '', negativePrompt =
let bin, args;
let proc;
let claimHandedOff = false;
+ let takeEarlyExit = null;
try {
// `renderPrompt` — the user's prompt plus any LoRA activation tokens it was
// missing (#4665). Identical to `prompt` when nothing was woven.
@@ -770,6 +771,10 @@ export async function generateImage({ pythonPath, prompt = '', negativePrompt =
activeJob = { ...meta, generationId: jobId, totalSteps: actualSteps, step: 0, progress: 0, currentImage: null, mode: IMAGE_GEN_MODE.LOCAL };
proc = spawn(bin, args, safeChildProcessOptions({ env: childEnv, stdio: ['ignore', 'pipe', 'pipe'] }));
+ // Same tick as the spawn, before the claim handoff below awaits file I/O: a
+ // child that dies inside that window would otherwise emit 'close' with no
+ // listener, leaving the job running and the GPU lane held forever (#11761).
+ takeEarlyExit = bufferChildExit(proc, { bufferOutput: true });
activeProcess = proc;
preparing = null;
await heavyClaim.handoffTo?.(proc.pid);
@@ -1256,6 +1261,17 @@ export async function generateImage({ pythonPath, prompt = '', negativePrompt =
});
});
+ // Every real listener is on and nothing has awaited since: hand over whatever
+ // the child emitted during the claim handoff so a fast-dying render still
+ // reaches its terminal handlers and releases the claim.
+ const earlyExit = takeEarlyExit();
+ if (earlyExit) {
+ console.warn(`⚠️ Image render child exited before it was wired [${shortId}]`);
+ for (const { stream, chunk } of earlyExit.output || []) proc[stream].emit('data', chunk);
+ if (earlyExit.type === 'error') proc.emit('error', earlyExit.error);
+ else if (earlyExit.type === 'close') proc.emit('close', earlyExit.code, earlyExit.signal);
+ }
+
return { jobId, filename, path: `/data/images/${filename}`, generationId: jobId, mode: IMAGE_GEN_MODE.LOCAL, model: modelId, seed: actualSeed };
}
diff --git a/server/services/imageGen/local.test.js b/server/services/imageGen/local.test.js
index c2f55aa4af..6fea9e923c 100644
--- a/server/services/imageGen/local.test.js
+++ b/server/services/imageGen/local.test.js
@@ -1370,3 +1370,34 @@ it('settles a refused close-event publication and releases the process and heavy
expect(attachSseClient(jobId, response)).toBe(false);
} finally { admissionFault.error = null; vi.useRealTimers(); }
});
+
+it('drives a child that dies during the heavy-claim handoff to a terminal state and releases the claim (#11761)', async () => {
+ mockResolveFlux2Python.mockReturnValue('/fake/venv-flux2/bin/python3');
+ mockIsFlux2VenvHealthy.mockResolvedValue(true);
+ const child = new EventEmitter();
+ child.stdout = new EventEmitter(); child.stderr = new EventEmitter();
+ child.pid = 4242;
+ mockSpawn.mockReturnValue(child);
+ const heavyJobClaim = await import('../../lib/heavyJobClaim.js');
+ // The handoff awaits real file I/O in production; here the child's stderr and
+ // `close` land inside that window, before any real listener is attached.
+ const claimSpy = vi.spyOn(heavyJobClaim, 'claimHeavyLocalJob').mockResolvedValueOnce({
+ ok: true,
+ release: releaseClaim,
+ handoffTo: async () => {
+ child.stderr.emit('data', Buffer.from('Traceback: venv aborted on import\n'));
+ child.emit('close', 1, null);
+ await new Promise((resolve) => setImmediate(resolve));
+ },
+ });
+ const { imageGenEvents } = await import('../imageGenEvents.js');
+ const { getActiveJob } = await import('./local.js');
+ const releasedBefore = releaseClaim.mock.calls.length;
+ const failed = new Promise((resolve) => imageGenEvents.once('failed', resolve));
+ try {
+ await generateImage({ modelId: 'qwen-image-2.1', prompt: 'dies at startup' });
+ expect((await failed).error).toContain('Traceback: venv aborted on import');
+ expect(getActiveJob()).toBeNull();
+ expect(releaseClaim.mock.calls.length).toBe(releasedBefore + 1);
+ } finally { claimSpy.mockRestore(); }
+});