Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: attach child listeners before awaiting after spawn so a failed launch cannot crash the CoS Runner or wedge the GPU lane (#11761)
[._.]/ Adam Eivy5f541e668a19
diff --git a/scripts/ci-test-plan.js b/scripts/ci-test-plan.jsindex 620fc7b98e..90bd9772ba 100644--- a/scripts/ci-test-plan.js+++ b/scripts/ci-test-plan.js@@ -309,6 +309,9 @@ export const ALWAYS_RUN_TESTS = [// Whole-tree scanner: any server file can log a failure through// `console.log`, and the guard only ever sees it as a path string.'server/logLevelConventions.test.js',+ // Whole-tree scanner: any server file can spawn a child and await before+ // attaching its error/exit listener (#11761), with no import edge to this file.+ 'server/childListenerOrderConventions.test.js',// Whole-tree scanner: any server/autofixer file can re-implement raw// `pm2 jlist` execution/parsing outside the two designated readers (#8164).'server/pm2JlistOwnership.guard.test.js',diff --git a/server/AGENTS.md b/server/AGENTS.mdindex 61df2416ba..98c8f94ccc 100644--- a/server/AGENTS.md+++ b/server/AGENTS.md@@ -23,6 +23,10 @@ These apply to Express/service/model code. Universal constraints (no try/catch o- **An `async` timer callback must own its rejections.** Inside `setTimeout(async …)` / `setInterval(async …)`, every `await` needs either an enclosing `try { … } catch` or a `.catch(…)` on the awaited chain. A timer fires outside the request lifecycle, so nothing holds the promise the callback returns and a rejection escapes as an unhandled rejection — fatal on Node >= 15. The CoS runner shipped that at boot: its orphan-cleanup timer awaited a state-file read/write, so one truncated `agents` file killed the runner seconds after start and PM2 restart-looped it against the same file (#5668). Log and continue in the catch unless the callback genuinely owns a failure verdict. `server/timerCallbackConventions.test.js` scans the whole server tree for this and carries no allowlist; the worked example is the watchdog in `server/services/mediaJobQueue/index.js`.+## Child processes++- **Attach a spawned child's terminal listener before the function next `await`s.** Node reports a failed spawn (ENOENT, EACCES, EMFILE) on `process.nextTick` and a fast-dying child exits while you are suspended: an unlistened `error` throws and takes the process down, an unlistened `close` strands the job `running`. Do the awaited work (a scanner read, a claim handoff) BEFORE `spawn()`, or call `bufferChildExit(child)` (`services/videoGen/generateVideoHelpers.js`) in the same tick and replay its result once the real handlers are wired (`services/imageGen/local.js`). The CoS Runner `/spawn` and `/spawn-tui` routes and the local image render each shipped this ordering bug (#11761, after #4617/#5655/#5791). `server/childListenerOrderConventions.test.js` scans the whole server tree for it and carries no allowlist.+## Socket handlers- **An `async` Socket.IO handler must own its rejections.** Same rule as the timer callbacks above, same reason: Socket.IO hands the promise a listener returns to nobody, so a rejection escapes as an unhandled rejection and Node terminates the process — taking every agent run, PTY session and media job with it. Every `await` in an `async` handler under `server/sockets/` needs an enclosing `try { … } catch` (the shape `voice.js` uses on whole handler bodies) or a `.catch(…)` on the awaited chain (the shape `logs.js` uses on the lookups it falls back from). `server/sockets/asyncHandlerGuard.test.js` scans the directory for this and carries no allowlist; it names the file and the event string of any offender. Precedent: `voice:call:detach` shipped unguarded (#5661).diff --git a/server/childListenerOrderConventions.test.js b/server/childListenerOrderConventions.test.jsnew file mode 100644index 0000000000..6b850488fb--- /dev/null+++ b/server/childListenerOrderConventions.test.js@@ -0,0 +1,385 @@+/**+ * Repo-wide guard: a spawned child must have its terminal listener attached+ * before the function next yields (#11761).+ *+ * ## The bug class+ *+ * const child = spawn(cli, args);+ * const scan = await securityScans.read(id); // ← real file I/O+ * child.on('error', onError); // ← too late+ *+ * Node reports a failed spawn (ENOENT, EACCES, EMFILE) through+ * `process.nextTick`, and a fast-dying child exits while the function is+ * suspended. An `error` with no listener THROWS and, outside the request+ * lifecycle, takes the process down; a `close`/`exit` with no listener is lost+ * and strands the job `running`. It shipped four times before this guard+ * (#4617, #5655, #5791, and the CoS Runner `/spawn` + `/spawn-tui` + local+ * image render sites of #11761).+ *+ * ## The rule+ *+ * In one function body, a `spawn(` / `pty.spawn(` / `spawnDetached(` result+ * bound to a name must, before the next `await`, have one of:+ *+ * - `<name>.on('error', …)` (or `once` / `addListener` / `prependListener`),+ * - `bufferChildExit(<name>, …)`,+ * - `<name>.onExit(…)` (node-pty).+ *+ * The operand of that first `await` is evaluated before the function suspends,+ * so a listener attached inside it (`await new Promise((r) => child.on('error', r))`)+ * counts. A spawn with no `await` after it in its block has no yield window and+ * passes.+ *+ * There is no allowlist: a new violation is a bug to fix by moving the awaited+ * work BEFORE the spawn, or by calling `bufferChildExit(child)` in the same tick.+ *+ * ## What this guard CANNOT see+ *+ * Lexer-assisted, not scope-aware (see `lib/sourceScan.js`):+ * - Only spawns bound by `name = spawn(…)` / `const name = await spawnDetached(…)`+ * are tracked; a spawn passed straight into a helper or chained+ * (`spawn(…).on(…)`) is not.+ * - The scan stops at the end of the enclosing function body. It walks out of+ * `try`/`if`/loop blocks (skipping a `try`'s `catch`) but not out of object+ * literals or callbacks, where it conservatively reports nothing.+ * - Awaits inside a nested `async` function body are not the spawning+ * function's yield point and are skipped.+ */++import { describe, it, expect } from 'vitest';+import { execFileSync } from 'child_process';+import { readFileSync } from 'fs';+import { dirname, join } from 'path';+import { fileURLToPath } from 'url';+import { blankLiterals, matchBracket, parseCallbackAt } from './lib/sourceScan.js';++const SERVER_ROOT = dirname(fileURLToPath(import.meta.url));++// `name = [await] [ns.]spawn|spawnDetached(`. `spawnSync` is excluded on purpose:+// it returns after the child exits and emits nothing.+const SPAWN_BINDING = /([A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)*)\s*=\s*(?:await\s+)?(?:[A-Za-z_$][\w$]*\.)?(?:spawn|spawnDetached)\s*\(/g;++const escapeRe = (text) => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');++/** `[start, end)` of every nested async function body in `blanked`. */+function asyncBodySpans(blanked) {+ const spans = [];+ for (const match of blanked.matchAll(/\basync\b/g)) {+ // Arrow / `async function` spelling.+ const parsed = parseCallbackAt(blanked, match.index, blanked.length);+ if (parsed && blanked[parsed.start] === '{') {+ spans.push([parsed.start, parsed.start + parsed.text.length]);+ continue;+ }+ // Method spelling: `async name(params) {`.+ const method = /^async\s+[\w$]+\s*\(/.exec(blanked.slice(match.index, match.index + 200));+ if (!method) continue;+ const paramsEnd = matchBracket(blanked, match.index + method[0].length - 1);+ if (paramsEnd === -1) continue;+ const open = blanked.slice(paramsEnd).search(/\S/);+ if (open === -1 || blanked[paramsEnd + open] !== '{') continue;+ const end = matchBracket(blanked, paramsEnd + open);+ if (end !== -1) spans.push([paramsEnd + open, end]);+ }+ return spans;+}++/** Index of the `{` that opens the innermost block containing `pos`, or -1. */+function enclosingOpen(blanked, pos) {+ let depth = 0;+ for (let i = pos - 1; i >= 0; i -= 1) {+ if (blanked[i] === '}') depth += 1;+ else if (blanked[i] === '{') {+ if (depth === 0) return i;+ depth -= 1;+ }+ }+ return -1;+}++/**+ * Whether the block opened at `open` is a control block of the SAME function+ * (`try`, `catch`, `finally`, `if`, `else`, loops) rather than a function body or+ * object literal. Returns 'try' for a try block so its catch clause can be skipped.+ */+function controlBlockKind(blanked, open) {+ const before = blanked.slice(0, open).trimEnd();+ if (/\btry$/.test(before)) return 'try';+ if (/\b(?:else|finally|do)$/.test(before)) return 'control';+ if (!before.endsWith(')')) return null;+ let depth = 0;+ for (let i = before.length - 1; i >= 0; i -= 1) {+ if (before[i] === ')') depth += 1;+ else if (before[i] === '(') {+ depth -= 1;+ if (depth === 0) {+ return /\b(?:if|for|while|switch|catch)(?:\s+await)?\s*$/.test(before.slice(0, i)) ? 'control' : null;+ }+ }+ }+ return null;+}++/**+ * Index of the first `await` after `from` that belongs to the spawning function,+ * or -1 when none follows before the function body ends. Leaving a `try`/`if`/loop+ * block keeps scanning (the spawn is commonly bound inside a `try`), and a+ * `catch` clause is skipped: it only runs when the spawn itself threw.+ */+function nextAwait(blanked, from, spans) {+ let depth = 0;+ let scope = from;+ for (let i = from; i < blanked.length; i += 1) {+ // Only bodies that OPEN after the spawn are nested; the enclosing async+ // function itself starts before it and its awaits are the ones that count.+ const span = spans.find(([start, end]) => start >= from && i >= start && i < end);+ if (span) { i = span[1] - 1; continue; }+ const c = blanked[i];+ if (c === '{') depth += 1;+ else if (c === '}') {+ depth -= 1;+ if (depth < 0) {+ const open = enclosingOpen(blanked, scope);+ const kind = open === -1 ? null : controlBlockKind(blanked, open);+ if (!kind) return -1;+ scope = open;+ depth = 0;+ const clause = kind === 'try' ? /^\s*catch\b\s*(?:\([^)]*\))?\s*\{/.exec(blanked.slice(i + 1)) : null;+ if (clause) i = matchBracket(blanked, i + clause[0].length) - 1;+ }+ } else if (c === 'a' && blanked.startsWith('await', i)+ && !/[\w$]/.test(blanked[i - 1] || '') && !/[\w$]/.test(blanked[i + 5] || '')) {+ return i;+ }+ }+ return -1;+}++/** End of the expression an `await` at `awaitIndex` operates on (the `;` or closing bracket). */+function awaitOperandEnd(blanked, awaitIndex) {+ let depth = 0;+ for (let i = awaitIndex + 5; i < blanked.length; i += 1) {+ const c = blanked[i];+ if (c === '(' || c === '[' || c === '{') depth += 1;+ else if (c === ')' || c === ']' || c === '}') {+ depth -= 1;+ if (depth < 0) return i;+ } else if (c === ';' && depth === 0) return i;+ }+ return blanked.length;+}++/** Every spawned-child binding in one file's source that awaits before wiring a terminal listener. */+export function findLateChildListeners(src) {+ const { code, blanked } = blankLiterals(src, { includeCommentOnly: true });+ // `code` keeps literal text, so `'error'` is still readable; `blanked` is what+ // await/bracket walks use.+ const spans = asyncBodySpans(blanked);+ const offenders = [];+ for (const match of blanked.matchAll(SPAWN_BINDING)) {+ const name = match[1];+ const callEnd = matchBracket(blanked, match.index + match[0].length - 1);+ if (callEnd === -1) continue;+ const awaitAt = nextAwait(blanked, callEnd, spans);+ if (awaitAt === -1) continue;+ const window = code.slice(callEnd, awaitOperandEnd(blanked, awaitAt));+ const n = escapeRe(name);+ const attached = new RegExp(+ `\\b${n}\\.(?:on|once|addListener|prependListener)\\(\\s*['"\`]error['"\`]`+ + `|\\bbufferChildExit\\(\\s*${n}\\b`+ + `|\\b${n}\\.onExit\\(`,+ ).test(window);+ if (attached) continue;+ const line = blanked.slice(0, match.index).split('\n').length;+ const awaited = blanked.slice(awaitAt, awaitOperandEnd(blanked, awaitAt)).replace(/\s+/g, ' ').trim();+ offenders.push(`line ${line}: ${name} is awaited past (${awaited.slice(0, 80)}) before an error/exit listener`);+ }+ return offenders;+}++const trackedServerSources = () => execFileSync('git', ['ls-files', '*.js', '*.mjs', '*.cjs'], {+ cwd: SERVER_ROOT,+ encoding: 'utf8',+ maxBuffer: 64 * 1024 * 1024,+}).split('\n').filter((f) => f && !f.includes('.test.'));++describe('spawned children get a terminal listener before the next await (#11761)', () => {+ it('scans the server tree, every module extension included', () => {+ const files = trackedServerSources();+ expect(files.length).toBeGreaterThan(200);+ expect(files.some((f) => f.endsWith('.mjs'))).toBe(true);+ });++ it('still recognizes the spawn sites it is meant to guard', () => {+ // Without this the scan below could go vacuously green after a refactor+ // renames the spawn helpers.+ const sites = trackedServerSources().filter((file) => {+ const src = readFileSync(join(SERVER_ROOT, file), 'utf8');+ return /\b(?:spawn|spawnDetached)\s*\(/.test(src) && [...blankLiterals(src).matchAll(SPAWN_BINDING)].length > 0;+ });+ expect(sites).toEqual(expect.arrayContaining([+ 'cos-runner/index.js',+ 'services/imageGen/local.js',+ 'services/videoGen/spawnWatch.js',+ ]));+ });++ it('has no spawn whose listener waits behind an await', () => {+ const violations = [];+ for (const file of trackedServerSources()) {+ const src = readFileSync(join(SERVER_ROOT, file), 'utf8');+ if (!src.includes('spawn')) continue;+ for (const hit of findLateChildListeners(src)) violations.push(`server/${file} ${hit}`);+ }+ expect(+ violations,+ 'These children are spawned and then the function awaits before any error/exit listener is '+ + 'attached. Node reports a failed spawn on process.nextTick and a fast-dying child exits '+ + 'during the await: an unlistened `error` throws and kills the process, an unlistened '+ + '`close` strands the job running (#4617, #5655, #5791, #11761).\n'+ + 'Fix: do the awaited work BEFORE the spawn, or call `bufferChildExit(child)` (videoGen/'+ + 'generateVideoHelpers.js) in the same tick and replay its result once the real handlers are on.\n'+ + `Offenders:\n ${violations.join('\n ')}`,+ ).toEqual([]);+ });+});++// Guards the guard: if the recognizer stops seeing the broken shape, the scan+// above goes green and the bug class walks straight back in.+describe('the child-listener-order recognizer', () => {+ it('flags the pre-fix CoS Runner /spawn shape', () => {+ expect(findLateChildListeners(`+ const claudeProcess = spawn(cmd, args, { cwd });+ const agent = {+ process: claudeProcess,+ securityScan: await securityScans.read(agentId),+ };+ claudeProcess.on('error', (err) => {});+ `)).toHaveLength(1);+ });++ it('flags the pre-fix /spawn-tui shape (pty.spawn, exit subscribed after the read)', () => {+ expect(findLateChildListeners(`+ tuiProcess = pty.spawn(ptyCommand, ptyArgs, { cwd });+ const agent = { securityScan: await securityScans.read(agentId) };+ tuiProcess.onExit(handler);+ `)).toHaveLength(1);+ });++ it('flags the pre-fix local image shape (handoff awaited before listeners)', () => {+ expect(findLateChildListeners(`+ try {+ proc = spawn(bin, args, opts);+ activeProcess = proc;+ await heavyClaim.handoffTo?.(proc.pid);+ } catch (err) { throw err; }+ proc.on('error', onError);+ `)).toHaveLength(1);+ });++ it('flags a spawnDetached result that waits and a member-bound child', () => {+ expect(findLateChildListeners(`+ firstProc = await spawnDetached(bin, args);+ await claim.handoffTo(firstProc.pid);+ firstProc.on('error', onError);+ `)).toHaveLength(1);+ expect(findLateChildListeners(`+ state.child = spawn(bin);+ await prepare();+ state.child.on('error', onError);+ `)).toHaveLength(1);+ });++ it('walks out of a try block but skips its catch clause', () => {+ expect(findLateChildListeners(`+ try {+ child = spawn(bin);+ } catch (err) {+ await cleanup();+ throw err;+ }+ child.on('error', onError);+ `)).toEqual([]);+ expect(findLateChildListeners(`+ try {+ child = spawn(bin);+ } catch (err) {+ throw err;+ }+ await prepare();+ child.on('error', onError);+ `)).toHaveLength(1);+ });++ it('treats an unrelated error listener on another object as no listener', () => {+ expect(findLateChildListeners(`+ const child = spawn(bin);+ other.on('error', noop);+ await prepare();+ child.on('error', onError);+ `)).toHaveLength(1);+ });++ it('accepts a listener, bufferChildExit or onExit attached before the await', () => {+ expect(findLateChildListeners(`+ const child = spawn(bin);+ child.on('error', onError);+ await prepare();+ `)).toEqual([]);+ expect(findLateChildListeners(`+ proc = spawn(bin);+ const take = bufferChildExit(proc, { bufferOutput: true });+ await heavyClaim.handoffTo?.(proc.pid);+ `)).toEqual([]);+ expect(findLateChildListeners(`+ const pty1 = pty.spawn(cmd, args);+ pty1.onExit(handler);+ await prepare();+ `)).toEqual([]);+ expect(findLateChildListeners(`+ const child = spawn(bin);+ child.once("error", onError);+ await prepare();+ `)).toEqual([]);+ });++ it('accepts a listener attached inside the first await operand', () => {+ expect(findLateChildListeners(`+ const child = spawn(bin);+ await new Promise((resolve, reject) => {+ child.on('error', reject);+ child.on('close', resolve);+ });+ `)).toEqual([]);+ });++ it('accepts a spawn with no await after it in its block', () => {+ expect(findLateChildListeners(`+ async function run() {+ await prepare();+ const child = spawn(bin);+ child.stdout.on('data', onData);+ child.on('error', onError);+ }+ async function other() { await somethingElse(); }+ `)).toEqual([]);+ });++ it('ignores awaits inside a nested async function and in comments or strings', () => {+ expect(findLateChildListeners(`+ const child = spawn(bin);+ // await prepare();+ const note = 'await prepare()';+ const later = async () => { await prepare(); };+ child.on('error', onError);+ `)).toEqual([]);+ });++ it('does not treat spawnSync or a plain assignment named like spawn as a spawn', () => {+ expect(findLateChildListeners(`+ const result = spawnSync(bin);+ const respawn = respawnCount + 1;+ await prepare();+ `)).toEqual([]);+ });+});diff --git a/server/cos-runner/index.js b/server/cos-runner/index.jsindex 8bd53aec9a..6a60a058ca 100644--- a/server/cos-runner/index.js+++ b/server/cos-runner/index.js@@ -305,6 +305,9 @@ app.post('/spawn-tui', lifecycle.spawnRoute(async (req, res) => {// preserves the shared escaping contract for paths/args passed to cmd.exe.const { command: ptyCommand, args: ptyArgs } = prepareCliSpawn(executable, args, childEnv);await securityScans.register(agentId, body.data.securityScanGrant);+ // Read BEFORE pty.spawn so onData/onExit subscribe with no await after the+ // spawn: a PTY that exits during the read would otherwise lose its exit (#11761).+ const securityScan = await securityScans.read(agentId);// Both checks above cleared the executable, so a throw here is about the PTY// layer or the workspace — not the command. node-pty collapses those into one// opaque `posix_spawn failed: No such file or directory`, which read as a@@ -349,7 +352,7 @@ app.post('/spawn-tui', lifecycle.spawnRoute(async (req, res) => {// node-pty reports pid 0 for ConPTY on Windows, so a pid probe is not a// liveness signal. onExit is the runner's authority; GET /agents reads this.exited: false,- securityScan: await securityScans.read(agentId),+ securityScan,stopSecurityScan: () => securityScans.stop(agentId).catch(err => lifecycle.reportFailure(err)),};activeAgents.set(agentId, agent);@@ -565,6 +568,10 @@ app.post('/spawn', lifecycle.spawnRoute(async (req, res) => {// Spawn the CLI processawait securityScans.register(agentId, body.data.securityScanGrant);+ // Read BEFORE spawn: Node reports a failed spawn (ENOENT/EACCES/EMFILE) via+ // process.nextTick, so no await may sit between spawn() and the 'error'+ // listener or the unhandled event kills the runner (#11761).+ const securityScan = await securityScans.read(agentId);const claudeProcess = spawn(spawnCommand, finalSpawnArgs, {cwd,shell: false,@@ -596,7 +603,7 @@ app.post('/spawn', lifecycle.spawnRoute(async (req, res) => {stripStdoutAnsi: isStreamJson ? (text) => text : createStreamingAnsiStripper(),stripStderrAnsi: createStreamingAnsiStripper(),workspacePath: cwd,- securityScan: await securityScans.read(agentId),+ securityScan,stopSecurityScan: () => securityScans.stop(agentId).catch(err => lifecycle.reportFailure(err)),};activeAgents.set(agentId, agent);diff --git a/server/cos-runner/spawnFailure.test.js b/server/cos-runner/spawnFailure.test.jsnew file mode 100644index 0000000000..8385dc502b--- /dev/null+++ b/server/cos-runner/spawnFailure.test.js@@ -0,0 +1,86 @@+import { afterAll, beforeAll, describe, expect, it } from 'vitest';+import { spawn } from 'child_process';+import { createServer } from 'net';+import { mkdtempSync, rmSync } from 'fs';+import { tmpdir } from 'os';+import { dirname, join } from 'path';+import { fileURLToPath } from 'url';+import { io as connect } from 'socket.io-client';++// index.js binds its server at import, so the only faithful way to exercise the+// real /spawn ordering is to boot it as a child process. #11761: Node reports a+// failed spawn on process.nextTick, and the route used to await the scanner+// evidence read between spawn() and the 'error' listener — the unhandled+// 'error' threw, the runner's uncaughtException handler exited the process, and+// every other agent it supervised lost its owner.+const RUNNER = join(dirname(fileURLToPath(import.meta.url)), 'index.js');++const freePort = () => new Promise((resolve, reject) => {+ const probe = createServer();+ probe.once('error', reject);+ probe.listen(0, '127.0.0.1', () => {+ const { port } = probe.address();+ probe.close(() => resolve(port));+ });+});++describe('cos-runner /spawn survives a failed CLI launch (#11761)', () => {+ let runner;+ let dataRoot;+ let base;+ let socket;+ let exited = false;++ beforeAll(async () => {+ dataRoot = mkdtempSync(join(tmpdir(), 'portos-runner-spawn-'));+ const port = await freePort();+ base = `http://127.0.0.1:${port}`;+ // The runner is a standalone service: drop the runner-under-test markers so its+ // own test-data isolation guard does not treat this boot as a test process.+ const env = Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.startsWith('VITEST')));+ runner = spawn(process.execPath, [RUNNER], {+ env: { ...env, NODE_ENV: 'test', PORT: String(port), HOST: '127.0.0.1', PORTOS_DATA_ROOT: dataRoot },+ stdio: 'ignore',+ });+ runner.once('exit', () => { exited = true; });+ for (let attempt = 0; attempt < 100 && !exited; attempt += 1) {+ const ready = await fetch(`${base}/health`).then((res) => res.ok, () => false);+ if (ready) break;+ await new Promise((resolve) => setTimeout(resolve, 100));+ }+ socket = connect(base, { transports: ['websocket'] });+ await new Promise((resolve) => socket.once('connect', resolve));+ }, 30_000);++ afterAll(() => {+ socket?.close();+ runner?.kill('SIGKILL');+ rmSync(dataRoot, { recursive: true, force: true });+ });++ it('reports agent:error, drops the agent and keeps the runner alive', async () => {+ const errored = new Promise((resolve, reject) => {+ socket.once('agent:error', resolve);+ runner.once('exit', () => reject(new Error('the runner process died on a failed spawn')));+ });+ // A removed cwd makes the launch fail with ENOENT regardless of whether the+ // CLI is installed, so this is deterministic on any machine.+ const response = await fetch(`${base}/spawn`, {+ method: 'POST',+ headers: { 'content-type': 'application/json' },+ body: JSON.stringify({+ agentId: 'agent-spawn-failure',+ taskId: 'task-spawn-failure',+ prompt: 'example',+ workspacePath: join(dataRoot, 'removed-workspace'),+ cliCommand: 'claude',+ cliArgs: [],+ }),+ });+ expect(response.status, await response.clone().text()).toBe(200);+ expect(await errored).toMatchObject({ agentId: 'agent-spawn-failure' });+ expect(exited, 'the runner process must not die on a failed spawn').toBe(false);+ const health = await fetch(`${base}/health`).then((res) => res.json());+ expect(health).toMatchObject({ status: 'ok', activeAgents: 0 });+ }, 20_000);+});diff --git a/server/services/imageGen/local.js b/server/services/imageGen/local.jsindex b630dfa1f3..5ed85631ce 100644--- a/server/services/imageGen/local.js+++ b/server/services/imageGen/local.js@@ -38,7 +38,7 @@ import { createLineReader } from '../../lib/streamLines.js';import { claimHeavyLocalJob } from '../../lib/heavyJobClaim.js';import { safeChildProcessOptions } from '../../lib/processEnv.js';import { IMAGE_GEN_MODE, LOCAL_IMAGEGEN_DEFAULT_MODEL } from './modes.js';-import { parseByteProgress, formatDownloadMessage } from '../videoGen/generateVideoHelpers.js';+import { parseByteProgress, formatDownloadMessage, bufferChildExit } from '../videoGen/generateVideoHelpers.js';const IS_WIN = process.platform === 'win32';@@ -741,6 +741,7 @@ export async function generateImage({ pythonPath, prompt = '', negativePrompt =let bin, args;let proc;let claimHandedOff = false;+ let takeEarlyExit = null;try {// `renderPrompt` — the user's prompt plus any LoRA activation tokens it was// missing (#4665). Identical to `prompt` when nothing was woven.@@ -770,6 +771,10 @@ export async function generateImage({ pythonPath, prompt = '', negativePrompt =activeJob = { ...meta, generationId: jobId, totalSteps: actualSteps, step: 0, progress: 0, currentImage: null, mode: IMAGE_GEN_MODE.LOCAL };proc = spawn(bin, args, safeChildProcessOptions({ env: childEnv, stdio: ['ignore', 'pipe', 'pipe'] }));+ // Same tick as the spawn, before the claim handoff below awaits file I/O: a+ // child that dies inside that window would otherwise emit 'close' with no+ // listener, leaving the job running and the GPU lane held forever (#11761).+ takeEarlyExit = bufferChildExit(proc, { bufferOutput: true });activeProcess = proc;preparing = null;await heavyClaim.handoffTo?.(proc.pid);@@ -1256,6 +1261,17 @@ export async function generateImage({ pythonPath, prompt = '', negativePrompt =});});+ // Every real listener is on and nothing has awaited since: hand over whatever+ // the child emitted during the claim handoff so a fast-dying render still+ // reaches its terminal handlers and releases the claim.+ const earlyExit = takeEarlyExit();+ if (earlyExit) {+ console.warn(`⚠️ Image render child exited before it was wired [${shortId}]`);+ for (const { stream, chunk } of earlyExit.output || []) proc[stream].emit('data', chunk);+ if (earlyExit.type === 'error') proc.emit('error', earlyExit.error);+ else if (earlyExit.type === 'close') proc.emit('close', earlyExit.code, earlyExit.signal);+ }+return { jobId, filename, path: `/data/images/${filename}`, generationId: jobId, mode: IMAGE_GEN_MODE.LOCAL, model: modelId, seed: actualSeed };}diff --git a/server/services/imageGen/local.test.js b/server/services/imageGen/local.test.jsindex c2f55aa4af..6fea9e923c 100644--- a/server/services/imageGen/local.test.js+++ b/server/services/imageGen/local.test.js@@ -1370,3 +1370,34 @@ it('settles a refused close-event publication and releases the process and heavyexpect(attachSseClient(jobId, response)).toBe(false);} finally { admissionFault.error = null; vi.useRealTimers(); }});++it('drives a child that dies during the heavy-claim handoff to a terminal state and releases the claim (#11761)', async () => {+ mockResolveFlux2Python.mockReturnValue('/fake/venv-flux2/bin/python3');+ mockIsFlux2VenvHealthy.mockResolvedValue(true);+ const child = new EventEmitter();+ child.stdout = new EventEmitter(); child.stderr = new EventEmitter();+ child.pid = 4242;+ mockSpawn.mockReturnValue(child);+ const heavyJobClaim = await import('../../lib/heavyJobClaim.js');+ // The handoff awaits real file I/O in production; here the child's stderr and+ // `close` land inside that window, before any real listener is attached.+ const claimSpy = vi.spyOn(heavyJobClaim, 'claimHeavyLocalJob').mockResolvedValueOnce({+ ok: true,+ release: releaseClaim,+ handoffTo: async () => {+ child.stderr.emit('data', Buffer.from('Traceback: venv aborted on import\n'));+ child.emit('close', 1, null);+ await new Promise((resolve) => setImmediate(resolve));+ },+ });+ const { imageGenEvents } = await import('../imageGenEvents.js');+ const { getActiveJob } = await import('./local.js');+ const releasedBefore = releaseClaim.mock.calls.length;+ const failed = new Promise((resolve) => imageGenEvents.once('failed', resolve));+ try {+ await generateImage({ modelId: 'qwen-image-2.1', prompt: 'dies at startup' });+ expect((await failed).error).toContain('Traceback: venv aborted on import');+ expect(getActiveJob()).toBeNull();+ expect(releaseClaim.mock.calls.length).toBe(releasedBefore + 1);+ } finally { claimSpy.mockRestore(); }+});