Skip to content

armgit / cindy

Public Imported from GitHub
0

Consider it done. The open-source AI agent that works out of the box · 想到,就能做到。开源、开箱即用的 AI Agent。

fix(pi): require process exit before receipt recovery

Chrisf97ebdb9af82
diff --git a/docs/research/pi-subagent-delegation.md b/docs/research/pi-subagent-delegation.md
index 442e1ca06d..aebbe4037e 100644
--- a/docs/research/pi-subagent-delegation.md
+++ b/docs/research/pi-subagent-delegation.md
@@ -19,10 +19,17 @@
`status.json` 仍是展示和控制的主记录。账号边界读取不到状态时,可以使用同目录的
完整终态 `result.json` 作为独立结束回执:必须匹配 `config.json` 的 runId、taskId、
parentSessionId、runDir 和全部 child/session/agent 身份,并确认所有子任务均为终态。
-该回执由 runner 在终止子进程、解除定时器后写出;续跑使用新 UUID,不复用旧 generation。
+该回执可能在仅尝试终止子进程后就写出,不能单独证明进程退出。恢复还必须通过独立 OS
+进程清单确认 run UUID 对应的 runner 与所有 native session ID 对应的 Pi 子进程均已消失,
+包括 runner 已退出但子进程仍存活的情况。清单查询失败、超时、截断,或无法确认本账号
+进程可见性时保留未知;Windows 使用系统的当前用户过滤取得 PID 集合,再逐项核验 CIM
+命令行,任一缺失或不可读都不放行,不把其他账号的受保护系统进程混入当前账号证明。
+仅排除本次启动的 `tasklist` 查询进程自身 PID,避免它正常退出后使清单永久不完整。
+查询是只读核验,不向清单里的 PID 发信号。同一次扫描复用清单,下次扫描重新读取。
+续跑使用新 run UUID;复用 native session ID 的活跃子进程仍保守视为未结束。
可读的运行态不被结果附件覆盖。缺少、损坏、超限、链接文件、身份不符或不完整的回执仍
保持未知并阻止账号交接,不能仅凭宿主退出、文件陈旧或 stop 无回执放行。恢复只读取证据,
-不修改或删除原状态、结果和对话文件,也不宣称验证了 OS 进程退出。
+不修改或删除原状态、结果和对话文件。
## 对标与取舍
diff --git a/packages/maker-core/src/agents/pi/__tests__/cindySubagentRunner.test.ts b/packages/maker-core/src/agents/pi/__tests__/cindySubagentRunner.test.ts
index 586091e612..821665fe3e 100644
--- a/packages/maker-core/src/agents/pi/__tests__/cindySubagentRunner.test.ts
+++ b/packages/maker-core/src/agents/pi/__tests__/cindySubagentRunner.test.ts
@@ -244,6 +244,8 @@ async function makeFixture(options: {
surviveStdinEnd?: boolean;
/** Model a wedged runner: publish status, never consume the stop mailbox. */
ignoreStopControl?: boolean;
+ /** Inject OS signal failures into the real runner, without faking its receipt. */
+ failChildTermination?: boolean;
/** Runner config of a shared-user (guest) hosted session. */
guestIsolation?: boolean;
/**
@@ -280,7 +282,7 @@ async function makeFixture(options: {
// the control mailbox. Written *into the run dir* so its argv path carries the
// run UUID, exactly as the production runner does — that path is the identity
// the host verifies before signalling.
- const runnerFile = options.ignoreStopControl
+ const runnerFile = options.ignoreStopControl || options.failChildTermination
? path.join(runDir, 'runner.cjs')
: await sharedRunnerFile();
const fakePiFile = path.join(root, 'fake-pi.cjs');
@@ -294,6 +296,22 @@ async function makeFixture(options: {
const tokensFile = path.join(root, 'tokens.jsonl');
const pidsFile = path.join(root, 'child-pids.jsonl');
const poisonedSessionDir = path.join(root, 'poisoned-session-dir');
+ if (options.failChildTermination) {
+ await writeFile(runnerFile, `
+const cp = require('node:child_process');
+const originalSpawn = cp.spawn;
+cp.spawn = function (...args) {
+ const child = originalSpawn(...args);
+ child.kill = function () { return false; };
+ return child;
+};
+const originalKill = process.kill.bind(process);
+process.kill = function (pid, signal) {
+ if (pid < 0) throw Object.assign(new Error('injected EPERM'), { code: 'EPERM' });
+ return originalKill(pid, signal);
+};
+` + CINDY_SUBAGENT_RUNNER_SOURCE, { mode: 0o700 });
+ }
if (options.ignoreStopControl) {
await writeFile(runnerFile, `
'use strict';
@@ -614,6 +632,34 @@ describe('Cindy durable PI Subagent runner', () => {
.toEqual(before);
});
+ it.skipIf(process.platform === 'win32')('refuses a real signal receipt until the orphan child has exited', async () => {
+ // SIGTERM handlers are a POSIX path; Windows abrupt termination cannot
+ // execute this handler. Keep the actual runner's signal/result code intact.
+ const fixture = await makeFixture({ hang: true, surviveStdinEnd: true, failChildTermination: true });
+ const childPid = await waitFor(async () => {
+ const content = await readFile(fixture.pidsFile, 'utf8').catch(() => '');
+ const pid = Number(content.trim().split('\n')[0]);
+ return Number.isSafeInteger(pid) && pid > 0 ? pid : null;
+ });
+ try {
+ fixture.child.kill('SIGTERM');
+ await waitForClose(fixture.child, fixture.stderr);
+ const receipt = JSON.parse(await readFile(path.join(fixture.runDir, 'result.json'), 'utf8'));
+ expect(receipt.state).toBe('stopped');
+ expect(receipt.tasks[0].status).toBe('stopped');
+ expect(process.kill(childPid, 0)).toBe(true);
+ await writeFile(path.join(fixture.runDir, 'status.json'), Buffer.alloc(72_506));
+ await expect(stopForAccountBoundary(fixture.root, { timeoutMs: 0 })).resolves.toBe(false);
+ // The read-only inventory must not kill a discovered process by its PID.
+ expect(process.kill(childPid, 0)).toBe(true);
+ } finally {
+ // This PID is from our own fixture, never from a product recovery scan.
+ try { process.kill(childPid, 'SIGKILL'); } catch { /* already gone */ }
+ }
+ await waitFor(async () => await stopForAccountBoundary(fixture.root, { timeoutMs: 0 }) ? true : null,
+ undefined, 'the independent process inventory to confirm the orphan exited');
+ });
+
// A profile prompt long enough to blow the platform spawn command line budget
// must fail closed as a readable task error. Without the guard the spawn
// itself throws (ENAMETOOLONG / EINVAL on Windows) and the run ends with an
diff --git a/packages/maker-core/src/agents/pi/__tests__/pi-subagent-runs.test.ts b/packages/maker-core/src/agents/pi/__tests__/pi-subagent-runs.test.ts
index 12622b2723..d2a8329401 100644
--- a/packages/maker-core/src/agents/pi/__tests__/pi-subagent-runs.test.ts
+++ b/packages/maker-core/src/agents/pi/__tests__/pi-subagent-runs.test.ts
@@ -1424,6 +1424,9 @@ describe('PI durable subagent run store', () => {
// terminal artifact before status. Existing unreadable-status cases have
// no receipt and must keep refusing the account handover.
async function terminalReceiptHome(write: (dir: string) => Promise<void>) {
+ // An independently successful OS inventory, with unrelated processes.
+ // Empty stdout is deliberately not evidence that the run has exited.
+ childProcess.spawnSync.mockImplementation(() => ({ status: 0, stdout: inventory(['node unrelated.js']) }));
const agentHome = await homeWithUnreadableRun(write);
const root = piSubagentRunRoot(agentHome, 'session-1');
const dir = path.join(root, runId);
@@ -1442,6 +1445,47 @@ describe('PI durable subagent run store', () => {
return { agentHome, root, dir, config, result };
}
+ function inventory(commands: string[]): string {
+ return process.platform === 'win32'
+ ? JSON.stringify({ complete: true, count: commands.length, commands })
+ : commands.map((command, i) => `${100 + i} ${command}`).join('\n');
+ }
+
+ it.each(['runner', 'orphan child'])('keeps a terminal receipt in scope while its %s is alive', async kind => {
+ const { agentHome, dir, config } = await terminalReceiptHome(async () => {});
+ const command = kind === 'runner'
+ ? `node ${path.join(dir, 'runner.cjs')} ${path.join(dir, 'config.json')}`
+ : `pi --mode rpc --session-id ${config.tasks[0]!.sessionId}`;
+ childProcess.spawnSync.mockImplementation(() => ({ status: 0, stdout: inventory([command]) }));
+ await expect(stopAllPiSubagentRunsForExit(agentHome, 0, { killUnresponsiveRunners: true })).resolves.toBe(false);
+ // Process discovery never grants permission to signal an unverified PID.
+ expect(childProcess.spawnSync.mock.calls.some(call => call[0] === 'taskkill')).toBe(false);
+ childProcess.spawnSync.mockImplementation(() => ({ status: 0, stdout: inventory(['node unrelated.js']) }));
+ await expect(stopAllPiSubagentRunsForExit(agentHome, 0)).resolves.toBe(true);
+ });
+
+ it.each([
+ ['empty output', { status: 0, stdout: '' }],
+ ['malformed output', { status: 0, stdout: 'unreadable process list' }],
+ ['failed probe', { status: 1, stdout: '' }],
+ ['timeout or overflow', { error: new Error('process inventory unavailable') }],
+ ] as const)('does not treat %s as process exit', async (_label, probe) => {
+ const { agentHome } = await terminalReceiptHome(async () => {});
+ childProcess.spawnSync.mockImplementation(() => probe);
+ await expect(stopAllPiSubagentRunsForExit(agentHome, 0)).resolves.toBe(false);
+ });
+
+ it.each([false, true])('requires every Windows process command line even when complete is %s', async complete => {
+ usePlatform('win32');
+ const { agentHome } = await terminalReceiptHome(async () => {});
+ childProcess.spawnSync.mockImplementation(() => ({
+ status: 0, stdout: JSON.stringify({ complete, count: 2, commands: ['node unrelated.js'] }),
+ }));
+ await expect(stopAllPiSubagentRunsForExit(agentHome, 0)).resolves.toBe(false);
+ childProcess.spawnSync.mockImplementation(() => ({ status: 0, stdout: inventory(['node unrelated.js']) }));
+ await expect(stopAllPiSubagentRunsForExit(agentHome, 0)).resolves.toBe(true);
+ });
+
it.each([
...cases,
['contains zero bytes', async (dir: string) => writeFile(path.join(dir, 'status.json'), Buffer.alloc(72_506))],
@@ -1453,8 +1497,15 @@ describe('PI durable subagent run store', () => {
.resolves.toBe(true);
await expect(stopPiSubagentRunsForAccountBoundary(root, { timeoutMs: 0 }))
.resolves.toBe(true);
- expect(await Promise.all(files.map(file => readFile(path.join(dir, file)).catch(() => null))))
- .toEqual(before);
+ // Native byte comparison keeps the 3 MiB oversize fixture cheap on CI;
+ // deep equality enumerates millions of Buffer indices and times out.
+ const after = await Promise.all(files.map(file => readFile(path.join(dir, file)).catch(() => null)));
+ for (let index = 0; index < files.length; index += 1) {
+ const original = before[index]!;
+ const current = after[index]!;
+ expect(current === null ? original === null : original !== null && current.equals(original), files[index])
+ .toBe(true);
+ }
expect(existsSync(path.join(dir, 'controls'))).toBe(false);
});
diff --git a/packages/maker-core/src/agents/pi/cindy-subagent-runner-source.ts b/packages/maker-core/src/agents/pi/cindy-subagent-runner-source.ts
index a7525f32ad..6ab362ef63 100644
--- a/packages/maker-core/src/agents/pi/cindy-subagent-runner-source.ts
+++ b/packages/maker-core/src/agents/pi/cindy-subagent-runner-source.ts
@@ -436,7 +436,8 @@ function main() {
// result.json is a best-effort attachment; status.json remains the primary
// record. Account-boundary recovery can validate this independent terminal
- // receipt if status is unreadable. Its write must never decide the outcome:
+ // receipt if status is unreadable AND the OS confirms no runner/child remains.
+ // A signal attempt alone is not exit evidence. Its write must never decide the outcome:
// letting it throw once
// turned a completed run into a published failure (the throw from the success
// path landed in the failure handler, which rewrote state and republished),
diff --git a/packages/maker-core/src/agents/pi/pi-subagent-runs.ts b/packages/maker-core/src/agents/pi/pi-subagent-runs.ts
index 0cbe1f0e98..dd0dc7ff72 100644
--- a/packages/maker-core/src/agents/pi/pi-subagent-runs.ts
+++ b/packages/maker-core/src/agents/pi/pi-subagent-runs.ts
@@ -3023,6 +3023,7 @@ const DEFAULT_KILL_BUDGET_MS = 8_000;
async function sweepableRunsUnderRoot(
root: string,
scope: PiSubagentSweepScope,
+ recoveryDeadline: number,
startTimeMemo?: ProcessStartTimeMemo,
): Promise<Array<{ runId: string; status: PiSubagentRunStatus | undefined }>> {
const runIds = await listRunDirectoryIds(root);
@@ -3035,10 +3036,14 @@ async function sweepableRunsUnderRoot(
diagnostics.filter((diagnostic) => diagnostic.kind === 'stale').map((diagnostic) => diagnostic.runId),
);
const sweepable: Array<{ runId: string; status: PiSubagentRunStatus | undefined }> = [];
+ const recoveryCandidates = new Map<string, string[]>();
for (const runId of runIds) {
const status = statuses.get(runId);
if ((status && isPiSubagentTerminal(status.state)) || staleRunIds.has(runId)) continue;
- if (!status && await hasTerminalResultReceipt(root, runId)) continue;
+ if (!status) {
+ const identities = await terminalResultProcessIdentities(root, runId);
+ if (identities) recoveryCandidates.set(runId, identities);
+ }
if (
scope.runtimeOwnerId !== undefined
&& status?.runtimeOwnerId !== undefined
@@ -3049,18 +3054,30 @@ async function sweepableRunsUnderRoot(
if (scope.hostPid !== undefined && !isSweepableByHost(status, scope.hostPid, startTimeMemo)) continue;
sweepable.push({ runId, status });
}
- return sweepable;
+ if (recoveryCandidates.size === 0) return sweepable;
+ // Read all candidate receipts BEFORE the inventory. Caching an earlier
+ // inventory while reading later receipts could miss a late runner launch.
+ // This one bounded, read-only probe is shared only within the current pass.
+ const processes = await readRecoveryProcessInventory(recoveryDeadline);
+ if (!processes) return sweepable;
+ return sweepable.filter(({ runId }) => {
+ const identities = recoveryCandidates.get(runId);
+ return !identities || processes.some(command => identities.some(id => command.includes(id)));
+ });
}
/**
- * The runner publishes result.json after killing/disarming its children, just
- * before the terminal status. That independent receipt survives a damaged
- * status file (#6694). Resume always creates a new UUID, never reuses this run.
- * Only a complete receipt for this exact config can retire an unreadable run;
- * age, a dead host, and an unanswered stop request are not completion evidence.
+ * A result survives a damaged status (#6694), but is NOT exit evidence: signal
+ * and partial-failure paths can publish it after an unsuccessful kill attempt.
+ * Require both a matching receipt and an independent OS inventory with no
+ * runner/child identities left. Resume creates a fresh run UUID; a resumed child
+ * with the same native session ID is conservatively considered still active.
* Keep both files untouched, including the corrupt status, for diagnosis.
*/
-async function hasTerminalResultReceipt(root: string, runId: string): Promise<boolean> {
+async function terminalResultProcessIdentities(
+ root: string,
+ runId: string,
+): Promise<string[] | null> {
try {
const runDir = path.join(root, runId);
const [config, value] = await Promise.all([
@@ -3071,14 +3088,14 @@ async function hasTerminalResultReceipt(root: string, runId: string): Promise<bo
|| config.parentSessionId !== path.basename(root)
|| typeof config.runDir !== 'string'
|| path.resolve(config.runDir) !== path.resolve(runDir)
- || !value || typeof value !== 'object' || Array.isArray(value)) return false;
+ || !value || typeof value !== 'object' || Array.isArray(value)) return null;
const result = value as Record<string, unknown>;
if (result.version !== 1 || result.runId !== runId || result.taskId !== config.taskId
|| !isState(result.state) || !isPiSubagentTerminal(result.state)
|| !finiteNonNegative(result.completedAt)
- || !Array.isArray(result.tasks) || result.tasks.length !== config.tasks.length) return false;
+ || !Array.isArray(result.tasks) || result.tasks.length !== config.tasks.length) return null;
const seen = new Set<string>();
- return result.tasks.every((value: unknown) => {
+ const matchingTasks = result.tasks.every((value: unknown) => {
if (!value || typeof value !== 'object' || Array.isArray(value)) return false;
const task = value as Record<string, unknown>;
if (typeof task.childId !== 'string' || !task.childId || seen.has(task.childId)) return false;
@@ -3090,9 +3107,93 @@ async function hasTerminalResultReceipt(root: string, runId: string): Promise<bo
&& task.sessionId === expected.sessionId && task.agent === expected.agent
&& isState(task.status) && isPiSubagentTerminal(task.status);
});
+ if (!matchingTasks) return null;
+ // The runner argv carries the UUID in its config/script path; every Pi
+ // child carries --session-id, including children surviving a runner exit.
+ // Matching only runner PIDs would miss precisely that orphan-child case.
+ return [runId, ...config.tasks.map(task => task.sessionId)];
} catch {
// Bounded reads reject missing, corrupt, oversized and linked artifacts.
- return false;
+ return null;
+ }
+}
+
+/**
+ * Inventory only for receipt recovery; never a source of PIDs to signal.
+ * Empty/failed/truncated probes are unknown, not proof of absence. Windows can
+ * omit command lines for protected processes in other accounts. Use the OS
+ * user filter first (tasklist /FI USERNAME, like POSIX ps -u), then require a
+ * readable command line for every selected PID. A missing/inaccessible own
+ * process keeps recovery closed; PID discovery never authorizes a kill.
+ */
+async function readRecoveryProcessInventory(deadline: number): Promise<string[] | null> {
+ try {
+ const remaining = Math.floor(deadline - Date.now());
+ if (remaining <= 0) return null;
+ if (process.platform === 'win32') {
+ const script = [
+ "$ErrorActionPreference = 'Stop'",
+ '$user = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name',
+ // tasklist reports itself, then exits before CIM runs. Track only our
+ // own probe PID so that its expected exit cannot block every recovery.
+ '$probe = [System.Diagnostics.Process]::new()',
+ '$probe.StartInfo.FileName = "$env:SystemRoot\\System32\\tasklist.exe"',
+ '$probe.StartInfo.Arguments = \'/V /FO CSV /NH /FI "USERNAME eq \' + $user + \'"\'',
+ '$probe.StartInfo.UseShellExecute = $false',
+ '$probe.StartInfo.CreateNoWindow = $true',
+ '$probe.StartInfo.RedirectStandardOutput = $true',
+ 'if (-not $probe.Start()) { throw "tasklist did not start" }',
+ '$probeId = $probe.Id',
+ '$csv = $probe.StandardOutput.ReadToEnd()',
+ '$probe.WaitForExit()',
+ '$probeExitCode = $probe.ExitCode',
+ '$probe.Dispose()',
+ 'if ($probeExitCode -ne 0) { throw "tasklist failed" }',
+ '$owned = @($csv | ConvertFrom-Csv -Header Name,ProcessId,SessionName,SessionNumber,Memory,Status,UserName,CPUTime,WindowTitle)',
+ '$processIds = @{}',
+ 'foreach ($row in $owned) {',
+ ' $number = 0',
+ ' if (-not [int]::TryParse($row.ProcessId, [ref]$number) -or $number -le 0) { throw "invalid process inventory" }',
+ ' if ($number -eq $probeId) { continue }',
+ ' $processIds[$number] = $true',
+ '}',
+ '$count = $processIds.Count',
+ '$commands = [System.Collections.Generic.List[string]]::new()',
+ '$complete = $true',
+ '$items = @(Get-CimInstance Win32_Process -ErrorAction Stop)',
+ 'foreach ($item in $items) {',
+ ' $number = [int]$item.ProcessId',
+ ' if (-not $processIds.ContainsKey($number)) { continue }',
+ ' if ([string]::IsNullOrWhiteSpace($item.CommandLine)) { $complete = $false; break }',
+ ' $commands.Add($item.CommandLine)',
+ ' $processIds.Remove($number)',
+ '}',
+ '@{ complete = ($complete -and $processIds.Count -eq 0); commands = @($commands.ToArray()); count = $count } | ConvertTo-Json -Compress',
+ ].join('\n');
+ const { stdout } = await execFileAsync('powershell.exe', ['-NoProfile', '-NonInteractive', '-Command', script], {
+ encoding: 'utf8', timeout: Math.min(DEFAULT_KILL_BUDGET_MS, remaining), maxBuffer: 4 * 1024 * 1024, windowsHide: true,
+ });
+ const value = JSON.parse(stdout);
+ if (value?.complete !== true || !Number.isSafeInteger(value.count) || value.count <= 0
+ || !Array.isArray(value.commands) || value.commands.length !== value.count
+ || value.commands.some((command: unknown) => typeof command !== 'string' || !command.trim())) return null;
+ return value.commands;
+ }
+ const uid = process.geteuid?.();
+ if (!Number.isSafeInteger(uid)) return null;
+ const { stdout } = await execFileAsync('ps', ['-ww', '-u', String(uid), '-o', 'pid=,args='], {
+ encoding: 'utf8', timeout: Math.min(KILL_PROBE_TIMEOUT_MS, remaining), maxBuffer: 4 * 1024 * 1024,
+ });
+ const lines = stdout.trim().split('\n');
+ const commands: string[] = [];
+ for (const line of lines) {
+ const match = /^\s*([1-9]\d*)\s+(.+)$/.exec(line);
+ if (!match || !match[2]!.trim()) return null;
+ commands.push(match[2]!);
+ }
+ return commands.length > 0 ? commands : null;
+ } catch {
+ return null;
}
}
@@ -3120,6 +3221,7 @@ async function stopPiSubagentRunsUnderRoots(
scope: PiSubagentSweepScope = {},
): Promise<boolean> {
const deadline = Date.now() + Math.max(0, Math.floor(timeoutMs));
+ const recoveryDeadline = deadline + (scope.killBudgetMs ?? DEFAULT_KILL_BUDGET_MS);
const requested = new Set<string>();
for (;;) {
// One memo per pass, discarded with it: within a single pass a pid cannot
@@ -3127,7 +3229,7 @@ async function stopPiSubagentRunsUnderRoots(
const startTimeMemo: ProcessStartTimeMemo = new Map();
let activeCount = 0;
for (const root of roots) {
- for (const { runId } of await sweepableRunsUnderRoot(root, scope, startTimeMemo)) {
+ for (const { runId } of await sweepableRunsUnderRoot(root, scope, recoveryDeadline, startTimeMemo)) {
activeCount += 1;
const key = `${root}:${runId}`;
if (requested.has(key)) continue;
@@ -3145,12 +3247,12 @@ async function stopPiSubagentRunsUnderRoots(
// spending the outgoing account and editing the workspace. Escalate to a
// verified kill; anything we cannot positively identify is left alone.
if (!scope.killUnresponsiveRunners) return false;
- const killDeadline = Date.now() + (scope.killBudgetMs ?? DEFAULT_KILL_BUDGET_MS);
+ const killDeadline = Math.min(recoveryDeadline, Date.now() + (scope.killBudgetMs ?? DEFAULT_KILL_BUDGET_MS));
let killedAll = true;
for (const root of roots) {
// Re-derived, so a run whose directory vanished in the meantime is gone
// from the set and counts as reclaimed.
- const work = await sweepableRunsUnderRoot(root, scope, startTimeMemo);
+ const work = await sweepableRunsUnderRoot(root, scope, killDeadline, startTimeMemo);
// In parallel: each reclaim is mostly waiting on its own process probe,
// and serialising them multiplied the worst case by the number of
// runners — enough to overrun a quit budget with only a few of them.