Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

Treat a defunct process as no evidence of a live server (#4754)

Nick Gomezd167c0252601
diff --git a/.changeset/removal-defunct-process-refusals.md b/.changeset/removal-defunct-process-refusals.md
new file mode 100644
index 000000000..767aa1aa0
--- /dev/null
+++ b/.changeset/removal-defunct-process-refusals.md
@@ -0,0 +1,7 @@
+---
+"@inkeep/open-knowledge": patch
+"@inkeep/open-knowledge-desktop": patch
+"@inkeep/open-knowledge-server": patch
+---
+
+`ok deinit`, `ok uninstall`, and the desktop uninstall flow no longer refuse to remove a project's local state over host uncertainty showing no evidence of a running server, such as an already-exited process. `ok clean` prunes a lock naming one. A live or unreadable server process still blocks both.
diff --git a/docs/content/reference/cli.mdx b/docs/content/reference/cli.mdx
index c013d9ac7..12f46b2e7 100644
--- a/docs/content/reference/cli.mdx
+++ b/docs/content/reference/cli.mdx
@@ -121,7 +121,7 @@ Each project runs its own local server. These commands let you see and stop them
| `ok stop` | Stop the server for the current directory — or pass a port, path, or PID to stop a specific server, or `all` to stop every server. Declines when something is still connected to the target, naming how many clients are attached; pass `--force` to stop it anyway. A server it cannot reach is stopped without the check |
| `ok clean` | Remove a stale lock file left behind by a crash (never touches live servers) |
-`ok clean` retains a lock and exits `1` if it cannot read it, it records another machine, or it has no recorded owner and its PID is still alive locally. The error names the lock and explains what to check. For an unverified owner, inspect the named local PID and stop it if it is an OpenKnowledge server; do not delete the lock while that process is running.
+`ok clean` retains a lock and exits `1` if it cannot read it, it records another machine, or it has no recorded owner and its PID is still running locally. A PID that has exited is not running even while it lingers as a defunct entry in the process table, so `ok clean` prunes the lock naming it. The error names the lock and explains what to check. For an unverified owner, inspect the named local PID and stop it if it is an OpenKnowledge server; do not delete the lock while that process is running.
`ok status`, `ok ps`, and the server-lock health check distinguish an unverified owner from a foreign host. `ok status --json` uses `state: "unverified-owner"` and includes the PID, with server liveness reported as unknown. `ok ps --json` uses the same status, `displayStatus: "unverified"`, and `null` for unrecorded host, port, and start time. The directory is the project root for current `.ok/local` and legacy `.ok` locks; it is `null` when a custom lock location does not establish a project root. Such records are not signal authority for `ok stop`, including with `--force`.
@@ -166,7 +166,7 @@ On Windows, confirming that a running server really is the process the lock name
`ok deinit` needs the project's `.ok/` directory; once that directory is gone, it reports nothing to remove. Remove any retained Pi trust entry manually using its reported exact key and file, as described in [installation details](./what-open-knowledge-writes.mdx#pi-folder-trust).
-Malformed filesystem residue does not by itself establish a running server. Cleanup can skip it after process and listener inspection succeeds and finds no associated live candidate. A read failure, incomplete inspection, or associated live server remains a blocker. Global-only uninstall can retain foreign-owned or ownership-unverified project state after the same local inspection; selecting project deinitialization keeps those ownership checks strict. Dead PID-only locks are recoverable. Human output lists root blockers once; JSON `failed[]` includes both direct failures and blocked dependent operations, with `blocked[]` providing the dependent subset.
+Malformed filesystem residue does not by itself establish a running server. Cleanup can skip it after process and listener inspection succeeds and finds no associated live candidate. A read failure, incomplete inspection, or associated live server remains a blocker. A process that has exited is not a live server, and its unreadable working directory does not make the inspection incomplete, even while the process lingers as a defunct entry in the process table. Global-only uninstall can retain foreign-owned or ownership-unverified project state after the same local inspection; selecting project deinitialization keeps those ownership checks strict. Dead PID-only locks are recoverable. Human output lists root blockers once; JSON `failed[]` includes both direct failures and blocked dependent operations, with `blocked[]` providing the dependent subset.
Both commands print their plan and ask before acting; `--dry-run` previews without changing anything. Once `ok uninstall` has finished successfully, an interactive run also asks one optional question about why you're leaving — you can skip it, and `--yes`, `--json`, and non-interactive runs never ask. Anything you do answer is [sent to us](/docs/reference/what-open-knowledge-writes#what-leaves-your-machine).
diff --git a/packages/cli/src/commands/clean.test.ts b/packages/cli/src/commands/clean.test.ts
index 2ca1a9584..825e60d6d 100644
--- a/packages/cli/src/commands/clean.test.ts
+++ b/packages/cli/src/commands/clean.test.ts
@@ -1,5 +1,10 @@
-import { describe, expect, test } from 'vitest';
+import type { ChildProcess } from 'node:child_process';
+import { existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
+import { hostname, tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterEach, describe, expect, test } from 'vitest';
import { buildCleanPlan, runClean } from './clean.ts';
+import { startDefunctProcess } from './defunct-process.test-helper.ts';
import type { LockState } from './lock-state.ts';
function alive(pid: number, port: number): LockState {
@@ -177,3 +182,44 @@ test('never prunes a malformed foreign-owned lock', () => {
expect(unlinked).toEqual([]);
expect(outcome.failed[0]?.error).toContain('owning machine');
});
+
+describe('runClean over a lock whose recorded process has exited but is unreaped', () => {
+ const children: ChildProcess[] = [];
+ let root: string | undefined;
+
+ afterEach(async () => {
+ for (const child of children.splice(0)) {
+ if (child.exitCode !== null || child.signalCode !== null) continue;
+ child.kill('SIGKILL');
+ await new Promise((resolve) => child.once('exit', resolve));
+ }
+ if (root !== undefined) rmSync(root, { recursive: true, force: true });
+ root = undefined;
+ });
+
+ test.skipIf(process.platform === 'win32')('prunes it, as a crash-stale lock', async () => {
+ root = realpathSync(mkdtempSync(join(tmpdir(), 'ok-clean-defunct-')));
+ const defunct = await startDefunctProcess(root, children);
+ expect(defunct.state).toMatch(/^Z/);
+
+ const lockDir = join(root, '.ok', 'local');
+ mkdirSync(lockDir, { recursive: true });
+ const lockPath = join(lockDir, 'server.lock');
+ writeFileSync(
+ lockPath,
+ JSON.stringify({
+ pid: defunct.pid,
+ hostname: hostname(),
+ port: 7391,
+ startedAt: new Date().toISOString(),
+ worktreeRoot: root,
+ }),
+ );
+
+ const outcome = runClean({ lockDir, log: () => {}, error: () => {} });
+
+ expect(outcome.failed).toEqual([]);
+ expect(outcome.pruned.map((target) => target.lockPath)).toEqual([lockPath]);
+ expect(existsSync(lockPath)).toBe(false);
+ });
+});
diff --git a/packages/cli/src/commands/clean.ts b/packages/cli/src/commands/clean.ts
index b726173bb..e6d8defd5 100644
--- a/packages/cli/src/commands/clean.ts
+++ b/packages/cli/src/commands/clean.ts
@@ -1,5 +1,5 @@
import { unlinkSync } from 'node:fs';
-import { type Config, resolveLockDir } from '@inkeep/open-knowledge-server';
+import { type Config, isLockProcessRunning, resolveLockDir } from '@inkeep/open-knowledge-server';
import { Command } from 'commander';
import { describeLockOwnershipRefusal, inspectLock, type LockState } from './lock-state.ts';
@@ -67,7 +67,8 @@ interface CleanOutcome {
}
export function runClean(deps: RunCleanDeps): CleanOutcome {
- const inspect = deps.inspect ?? (() => inspectLock(deps.lockDir, 'server'));
+ const inspect =
+ deps.inspect ?? (() => inspectLock(deps.lockDir, 'server', { isAlive: isLockProcessRunning }));
const unlink = deps.unlink ?? ((path) => unlinkSync(path));
const log = deps.log ?? ((msg) => console.log(msg));
const error = deps.error ?? ((msg) => console.error(msg));
diff --git a/packages/cli/src/commands/defunct-process.test-helper.ts b/packages/cli/src/commands/defunct-process.test-helper.ts
new file mode 100644
index 000000000..24c5e8219
--- /dev/null
+++ b/packages/cli/src/commands/defunct-process.test-helper.ts
@@ -0,0 +1,47 @@
+import { type ChildProcess, spawn } from 'node:child_process';
+import { mkdirSync, writeFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+
+export interface DefunctProcess {
+ pid: number;
+ state: string;
+}
+
+const HOLDER_SOURCE = [
+ "const { spawn, spawnSync } = require('node:child_process');",
+ "const { writeSync } = require('node:fs');",
+ "const child = spawn(process.execPath, ['-e', '0'], { stdio: 'ignore' });",
+ "let state = '';",
+ 'const deadline = Date.now() + 10000;',
+ 'while (Date.now() < deadline) {',
+ " const ps = spawnSync('ps', ['-p', String(child.pid), '-o', 'stat='], { encoding: 'utf-8' });",
+ " state = (ps.stdout || '').trim();",
+ " if (state.startsWith('Z')) break;",
+ ' Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 20);',
+ '}',
+ "writeSync(1, JSON.stringify({ pid: child.pid, state }) + '\\n');",
+ 'Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0);',
+ '',
+].join('\n');
+
+export async function startDefunctProcess(
+ scratchDir: string,
+ children: ChildProcess[],
+): Promise<DefunctProcess> {
+ const holderPath = join(scratchDir, 'defunct-holder.cjs');
+ mkdirSync(dirname(holderPath), { recursive: true });
+ writeFileSync(holderPath, HOLDER_SOURCE);
+ const holder = spawn(process.execPath, [holderPath], { stdio: ['ignore', 'pipe', 'inherit'] });
+ children.push(holder);
+ return await new Promise<DefunctProcess>((resolve, reject) => {
+ let buffered = '';
+ holder.stdout?.on('data', (chunk) => {
+ buffered += String(chunk);
+ const newline = buffered.indexOf('\n');
+ if (newline !== -1) resolve(JSON.parse(buffered.slice(0, newline)) as DefunctProcess);
+ });
+ holder.once('exit', (code) =>
+ reject(new Error(`Defunct-process holder exited before reporting (code ${code})`)),
+ );
+ });
+}
diff --git a/packages/cli/src/commands/removal-host-uncertainty.test.ts b/packages/cli/src/commands/removal-host-uncertainty.test.ts
new file mode 100644
index 000000000..fe42838dd
--- /dev/null
+++ b/packages/cli/src/commands/removal-host-uncertainty.test.ts
@@ -0,0 +1,315 @@
+import { type ChildProcess, type SpawnSyncReturns, spawn } from 'node:child_process';
+import { once } from 'node:events';
+import {
+ chmodSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ realpathSync,
+ rmSync,
+ writeFileSync,
+} from 'node:fs';
+import { hostname, tmpdir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { afterEach, beforeAll, beforeEach, describe, expect, test, vi } from 'vitest';
+import { startDefunctProcess } from './defunct-process.test-helper.ts';
+
+const spawnSyncMock = vi.fn();
+
+let realSpawnSync: typeof import('node:child_process').spawnSync;
+let buildDeinitPlan: typeof import('./removal-plan.ts').buildDeinitPlan;
+let runRemoval: typeof import('./removal-plan.ts').runRemoval;
+let isProcessAlive: typeof import('@inkeep/open-knowledge-server').isProcessAlive;
+let stopServerForRemoval: typeof import('./stop-for-removal.ts').stopServerForRemoval;
+
+beforeAll(async () => {
+ const realCp = await vi.importActual<typeof import('node:child_process')>('node:child_process');
+ realSpawnSync = realCp.spawnSync;
+ vi.doMock('node:child_process', () => ({ ...realCp, spawnSync: spawnSyncMock }));
+ ({ buildDeinitPlan, runRemoval } = await import('./removal-plan.ts'));
+ ({ isProcessAlive } = await import('@inkeep/open-knowledge-server'));
+ ({ stopServerForRemoval } = await import('./stop-for-removal.ts'));
+});
+
+interface HostProcess {
+ pid: number;
+ command: string;
+}
+
+interface HostPlan {
+ processes: HostProcess[];
+ cwdQueries?: Array<{ pids: string; result: SpawnSyncReturns<string> }>;
+}
+
+function spawnResult(overrides: Partial<SpawnSyncReturns<string>>): SpawnSyncReturns<string> {
+ return {
+ pid: 0,
+ output: [],
+ stdout: '',
+ stderr: '',
+ status: 0,
+ signal: null,
+ error: undefined,
+ ...overrides,
+ };
+}
+
+const LSOF_HEADER = 'COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n';
+
+describe.skipIf(process.platform === 'win32')(
+ 'removal under machine-global process-inspection uncertainty',
+ () => {
+ let root: string;
+ let project: string;
+ let home: string;
+ let escapedSpawns: string[];
+ const children: ChildProcess[] = [];
+
+ beforeEach(() => {
+ root = realpathSync(mkdtempSync(join(tmpdir(), 'ok-removal-host-')));
+ project = join(root, 'project');
+ home = join(root, 'home');
+ escapedSpawns = [];
+ write(join(project, '.ok', 'local', 'server.lock'), 'not json');
+ write(join(project, 'notes.md'), '# Keep');
+ spawnSyncMock.mockReset();
+ });
+
+ afterEach(async () => {
+ for (const child of children.splice(0)) {
+ if (child.exitCode !== null || child.signalCode !== null) continue;
+ const exited = once(child, 'exit');
+ child.kill('SIGKILL');
+ await exited;
+ }
+ spawnSyncMock.mockReset();
+ rmSync(root, { recursive: true, force: true });
+ expect(escapedSpawns).toEqual([]);
+ });
+
+ function write(path: string, bytes: string): void {
+ mkdirSync(dirname(path), { recursive: true });
+ writeFileSync(path, bytes);
+ }
+
+ function installHost(plan: HostPlan): void {
+ const scriptedCwds = new Map((plan.cwdQueries ?? []).map((q) => [q.pids, q.result]));
+ spawnSyncMock.mockImplementation(
+ (command: string, args: readonly string[] = [], options?: unknown) => {
+ if (command === 'pgrep') {
+ return spawnResult({ stdout: plan.processes.map((p) => `${p.pid}\n`).join('') });
+ }
+ if (command === 'ps' && args[0] === '-axo') {
+ const rows = plan.processes.map((p) => `${p.pid} ${p.command}`);
+ return spawnResult({ stdout: ` PID COMMAND\n${rows.join('\n')}\n` });
+ }
+ if (command === 'lsof' && args[0] === '-iTCP') {
+ return spawnResult({ stdout: LSOF_HEADER });
+ }
+ if (command === 'lsof' && args[0] === '-p') {
+ const scripted = scriptedCwds.get(String(args[1]));
+ if (scripted !== undefined) return scripted;
+ return realSpawnSync(command, args, options as never);
+ }
+ escapedSpawns.push(`${command} ${args.join(' ')}`);
+ return realSpawnSync(command, args, options as never);
+ },
+ );
+ }
+
+ async function startLiveProcess(cwd: string): Promise<number> {
+ mkdirSync(cwd, { recursive: true });
+ const child = spawn(
+ process.execPath,
+ ['-e', "process.send('ready'); setInterval(() => {}, 1000)"],
+ { cwd, stdio: ['ignore', 'ignore', 'ignore', 'ipc'] },
+ );
+ children.push(child);
+ await once(child, 'message');
+ if (child.pid === undefined) throw new Error('Live fixture failed to start');
+ return child.pid;
+ }
+
+ function detailOf(results: Array<{ detail?: string }>): string {
+ return results.map((result) => result.detail ?? '').join('\n');
+ }
+
+ function installFailingPsProbe(): string {
+ const binDir = join(root, 'bin');
+ mkdirSync(binDir, { recursive: true });
+ const shim = join(binDir, 'ps');
+ writeFileSync(shim, '#!/bin/sh\necho "ps: process state unavailable" >&2\nexit 1\n');
+ chmodSync(shim, 0o755);
+ return binDir;
+ }
+
+ test('routes a caller-supplied probe reporter through the fallback process scan', async () => {
+ const live = await startLiveProcess(project);
+ const scanOnly = await startLiveProcess(join(root, 'other'));
+ installHost({
+ processes: [
+ { pid: live, command: 'open-knowledge-server notes' },
+ { pid: scanOnly, command: 'open-knowledge-server notes' },
+ ],
+ cwdQueries: [
+ {
+ pids: `${live},${scanOnly}`,
+ result: spawnResult({ stdout: `p${live}\nn${project}\n` }),
+ },
+ ],
+ });
+ const binDir = installFailingPsProbe();
+ const originalPath = process.env.PATH;
+ process.env.PATH = `${binDir}:${originalPath ?? ''}`;
+ const reportedToCaller: number[] = [];
+ const stderrChunks: string[] = [];
+ const stderrSpy = vi.spyOn(process.stderr, 'write').mockImplementation((chunk: unknown) => {
+ stderrChunks.push(String(chunk));
+ return true;
+ });
+
+ try {
+ await expect(
+ stopServerForRemoval(join(project, '.ok', 'local'), {
+ probe: () => ({
+ onProbeFailure: (failure) => {
+ reportedToCaller.push(failure.pid);
+ },
+ }),
+ }),
+ ).rejects.toThrow(/Cannot verify shutdown/);
+ } finally {
+ stderrSpy.mockRestore();
+ process.env.PATH = originalPath;
+ }
+
+ const cwdQuery = spawnSyncMock.mock.calls.find(
+ ([command, args]: [string, readonly string[]]) => command === 'lsof' && args[0] === '-p',
+ );
+ expect(cwdQuery?.[1][1]).toBe(`${live},${scanOnly}`);
+ expect(reportedToCaller).toContain(scanOnly);
+ expect(reportedToCaller).toContain(live);
+ expect(stderrChunks.filter((chunk) => chunk.includes('[process-scan]'))).toEqual([]);
+ });
+
+ test('removes project state when the only host uncertainty is a defunct process', async () => {
+ const defunct = await startDefunctProcess(root, children);
+ expect(defunct.state).toMatch(/^Z/);
+ expect(isProcessAlive(defunct.pid)).toBe(true);
+ installHost({ processes: [{ pid: defunct.pid, command: 'open-knowledge-server notes' }] });
+
+ const outcome = await runRemoval(buildDeinitPlan(project, home));
+
+ expect(detailOf(outcome.failed)).toBe('');
+ expect(outcome.failed).toEqual([]);
+ expect(outcome.results[0]).toMatchObject({
+ status: 'skipped',
+ detail: expect.stringContaining('No server was stopped'),
+ });
+ expect(existsSync(join(project, '.ok'))).toBe(false);
+ expect(readFileSync(join(project, 'notes.md'), 'utf8')).toBe('# Keep');
+ });
+
+ test('removes project state when a lock-dir-marked process is defunct', async () => {
+ const defunct = await startDefunctProcess(root, children);
+ expect(defunct.state).toMatch(/^Z/);
+ expect(isProcessAlive(defunct.pid)).toBe(true);
+ const marker = Buffer.from(join(project, '.ok', 'local')).toString('base64url');
+ installHost({
+ processes: [
+ { pid: defunct.pid, command: `open-knowledge-server --ok-lock-dir-b64=${marker}` },
+ ],
+ });
+
+ const outcome = await runRemoval(buildDeinitPlan(project, home));
+
+ expect(detailOf(outcome.failed)).toBe('');
+ expect(outcome.failed).toEqual([]);
+ expect(existsSync(join(project, '.ok'))).toBe(false);
+ expect(readFileSync(join(project, 'notes.md'), 'utf8')).toBe('# Keep');
+ });
+
+ test('removes project state when a well-formed lock records a defunct process', async () => {
+ const defunct = await startDefunctProcess(root, children);
+ expect(defunct.state).toMatch(/^Z/);
+ expect(isProcessAlive(defunct.pid)).toBe(true);
+ write(
+ join(project, '.ok', 'local', 'server.lock'),
+ JSON.stringify({
+ pid: defunct.pid,
+ hostname: hostname(),
+ port: 7391,
+ startedAt: new Date().toISOString(),
+ worktreeRoot: project,
+ }),
+ );
+ installHost({ processes: [] });
+
+ const outcome = await runRemoval(buildDeinitPlan(project, home));
+
+ expect(detailOf(outcome.failed)).toBe('');
+ expect(outcome.failed).toEqual([]);
+ expect(existsSync(join(project, '.ok'))).toBe(false);
+ expect(readFileSync(join(project, 'notes.md'), 'utf8')).toBe('# Keep');
+ });
+
+ test('refuses removal while a live server is attributable to this project', async () => {
+ const server = await startLiveProcess(project);
+ expect(isProcessAlive(server)).toBe(true);
+ installHost({ processes: [{ pid: server, command: 'open-knowledge-server notes' }] });
+
+ const outcome = await runRemoval(buildDeinitPlan(project, home));
+
+ expect(detailOf(outcome.failed)).toContain('live process candidates');
+ expect(detailOf(outcome.failed)).toContain(String(server));
+ expect(existsSync(join(project, '.ok', 'local', 'server.lock'))).toBe(true);
+ expect(readFileSync(join(project, '.ok', 'local', 'server.lock'), 'utf8')).toBe('not json');
+ expect(readFileSync(join(project, 'notes.md'), 'utf8')).toBe('# Keep');
+ });
+
+ test('reports an unreadable process state once per run, not once per probing caller', async () => {
+ const server = await startLiveProcess(project);
+ installHost({ processes: [{ pid: server, command: 'open-knowledge-server notes' }] });
+ const binDir = installFailingPsProbe();
+ const diagnostics: string[] = [];
+ vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array) => {
+ diagnostics.push(String(chunk));
+ return true;
+ });
+
+ const originalPath = process.env.PATH;
+ process.env.PATH = `${binDir}:${originalPath ?? ''}`;
+ let outcome: Awaited<ReturnType<typeof runRemoval>>;
+ try {
+ outcome = await runRemoval(buildDeinitPlan(project, home));
+ } finally {
+ process.env.PATH = originalPath;
+ vi.restoreAllMocks();
+ }
+
+ expect(detailOf(outcome.failed)).toContain('live process candidates');
+ expect(detailOf(outcome.failed)).toContain(String(server));
+ const reports = diagnostics
+ .join('')
+ .split('\n')
+ .filter((line) => line.includes(`could not read the state of process ${server}`));
+ expect(reports).toHaveLength(1);
+ });
+
+ test('refuses removal while a live process keeps its working directory unreadable', async () => {
+ const opaque = await startLiveProcess(join(root, 'opaque-project'));
+ expect(isProcessAlive(opaque)).toBe(true);
+ installHost({
+ processes: [{ pid: opaque, command: 'open-knowledge-server notes' }],
+ cwdQueries: [{ pids: String(opaque), result: spawnResult({ stdout: '' }) }],
+ });
+
+ const outcome = await runRemoval(buildDeinitPlan(project, home));
+
+ expect(detailOf(outcome.failed)).toContain(String(opaque));
+ expect(existsSync(join(project, '.ok', 'local', 'server.lock'))).toBe(true);
+ expect(readFileSync(join(project, '.ok', 'local', 'server.lock'), 'utf8')).toBe('not json');
+ });
+ },
+);
diff --git a/packages/cli/src/commands/removal-plan.ts b/packages/cli/src/commands/removal-plan.ts
index a3f5c3bc8..ba883046d 100644
--- a/packages/cli/src/commands/removal-plan.ts
+++ b/packages/cli/src/commands/removal-plan.ts
@@ -4,6 +4,7 @@ import { PROJECT_SKILL_PROJECTION_PATHS } from '@inkeep/open-knowledge-core';
import { atomicWriteFileSync } from '@inkeep/open-knowledge-core/server';
import { resolveShadowDir } from '@inkeep/open-knowledge-core/shadow-repo-layout';
import {
+ createProbeFailureReporter,
type LockProcessScan,
resolveLockDir,
scanLockProcesses,
@@ -408,13 +409,15 @@ export async function runRemoval(
op.kind === 'stop-server' && !op.preserveProjectState ? [resolve(op.lockDir)] : [],
),
);
- const scan = deps.scanProcesses ?? scanLockProcesses;
+ const runProbe = createProbeFailureReporter();
+ const scan = deps.scanProcesses ?? (() => scanLockProcesses(runProbe));
let processScan: Promise<LockProcessScan> | undefined;
const stopServer =
deps.stopServer ??
((lockDir: string) =>
stopServerForRemoval(lockDir, {
preserveProjectState: plan.scope === 'uninstall' && !deinitLockDirs.has(resolve(lockDir)),
+ probe: runProbe,
scanProcesses: () => (processScan ??= scan()),
}));
diff --git a/packages/cli/src/commands/removal-process-start.test.ts b/packages/cli/src/commands/removal-process-start.test.ts
index 82044f90f..24f1e08b9 100644
--- a/packages/cli/src/commands/removal-process-start.test.ts
+++ b/packages/cli/src/commands/removal-process-start.test.ts
@@ -14,7 +14,7 @@ describe('readRemovalProcessStart', () => {
}),
).toBe(Date.parse('2026-09-08T20:21:34.000Z'));
expect(run).toHaveBeenCalledWith(
- '/bin/ps',
+ 'ps',
['-p', '123', '-o', 'lstart='],
expect.objectContaining({
timeout: 5000,
diff --git a/packages/cli/src/commands/removal-process-start.ts b/packages/cli/src/commands/removal-process-start.ts
index 83449dc44..1499e7a9c 100644
--- a/packages/cli/src/commands/removal-process-start.ts
+++ b/packages/cli/src/commands/removal-process-start.ts
@@ -86,7 +86,7 @@ export function readRemovalProcessStart(
windowsHide: true,
env: { ...env, LC_ALL: 'C', TZ: 'UTC0' },
};
- const raw = run('/bin/ps', ['-p', String(pid), '-o', 'lstart='], commandOptions).trim();
+ const raw = run('ps', ['-p', String(pid), '-o', 'lstart='], commandOptions).trim();
if (!/^\w{3}\s+\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+\d{4}$/.test(raw)) return null;
const startedAt = Date.parse(`${raw} UTC`);
return Number.isFinite(startedAt) ? startedAt : null;
diff --git a/packages/cli/src/commands/stop-for-removal.test.ts b/packages/cli/src/commands/stop-for-removal.test.ts
index 8bb469f26..6c113d5e3 100644
--- a/packages/cli/src/commands/stop-for-removal.test.ts
+++ b/packages/cli/src/commands/stop-for-removal.test.ts
@@ -71,6 +71,81 @@ describe('stopServerForRemoval', () => {
return pid;
}
+ async function startUnreapedServer(): Promise<number> {
+ const readyPath = join(dir, 'unreaped-ready');
+ const serverPath = join(dir, 'unreaped-server.cjs');
+ writeFileSync(
+ serverPath,
+ [
+ "const fs = require('node:fs');",
+ "process.on('SIGTERM', () => { process.exit(0); });",
+ "process.stdin.on('end', () => { process.exit(0); });",
+ 'process.stdin.resume();',
+ 'setInterval(() => {}, 1000);',
+ `fs.writeFileSync(${JSON.stringify(readyPath)}, 'ready');`,
+ '',
+ ].join('\n'),
+ );
+ const holderPath = join(dir, 'unreaped-holder.cjs');
+ writeFileSync(
+ holderPath,
+ [
+ "const { spawn } = require('node:child_process');",
+ "const { existsSync, writeSync } = require('node:fs');",
+ `const child = spawn(process.execPath, [${JSON.stringify(serverPath)}], { stdio: ['pipe', 'ignore', 'ignore'] });`,
+ 'const deadline = Date.now() + 10000;',
+ `while (Date.now() < deadline && !existsSync(${JSON.stringify(readyPath)})) {`,
+ ' Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 20);',
+ '}',
+ "writeSync(1, JSON.stringify({ pid: child.pid }) + '\\n');",
+ 'Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0);',
+ '',
+ ].join('\n'),
+ );
+ const holder = spawn(process.execPath, [holderPath], { stdio: ['ignore', 'pipe', 'inherit'] });
+ children.push(holder);
+ const pid = await new Promise<number>((resolvePid, reject) => {
+ let buffered = '';
+ holder.stdout?.on('data', (chunk) => {
+ buffered += String(chunk);
+ const newline = buffered.indexOf('\n');
+ if (newline !== -1) {
+ resolvePid((JSON.parse(buffered.slice(0, newline)) as { pid: number }).pid);
+ }
+ });
+ holder.once('exit', (code) =>
+ reject(new Error(`Unreaped-server holder exited before reporting (code ${code})`)),
+ );
+ });
+ writeFileSync(
+ lockFilePath(dir, 'server'),
+ JSON.stringify({ pid, hostname: hostname(), port: 0, startedAt: new Date().toISOString() }),
+ );
+ return pid;
+ }
+
+ function installPsCallLog(): { path: string; binDir: string } {
+ const binDir = join(dir, 'bin');
+ mkdirSync(binDir, { recursive: true });
+ const path = join(dir, 'ps-calls.log');
+ const realPs = execFileSync('sh', ['-c', 'command -v ps'], { encoding: 'utf8' }).trim();
+ const shim = join(binDir, 'ps');
+ writeFileSync(
+ shim,
+ `#!/bin/sh\nprintf '%s\\n' "$*" >> ${JSON.stringify(path)}\nexec ${realPs} "$@"\n`,
+ );
+ chmodSync(shim, 0o755);
+ writeFileSync(path, '');
+ return { path, binDir };
+ }
+
+ async function fixtureExit(pid: number): Promise<ChildProcess> {
+ const child = children.find((candidate) => candidate.pid === pid);
+ if (child === undefined) throw new Error(`No fixture child for pid ${pid}`);
+ if (child.exitCode === null && child.signalCode === null) await once(child, 'exit');
+ return child;
+ }
+
test('recovers a dead PID-only lock for deinit', async () => {
writeFileSync(lockFilePath(dir, 'server'), JSON.stringify({ pid: 4242 }));
expect(await stopServerForRemoval(dir, { isAlive: () => false })).toMatchObject({
@@ -558,7 +633,8 @@ describe('stopServerForRemoval', () => {
stopped: 1,
failed: [],
});
- expect(isProcessAlive(pid)).toBe(false);
+ const exited = await fixtureExit(pid);
+ expect(exited.signalCode ?? exited.exitCode).not.toBe(0);
});
test('does not signal a local process named by a foreign-host lock', async () => {
@@ -593,9 +669,30 @@ describe('stopServerForRemoval', () => {
const outcome = await stopServerForRemoval(dir, { timeoutMs: 2000, pollIntervalMs: 10 });
expect(outcome).toEqual({ stopped: 1, failed: [] });
expect(readFileSync(flushed, 'utf8')).toBe('saved');
- expect(isProcessAlive(pid)).toBe(false);
+ expect((await fixtureExit(pid)).exitCode).toBe(0);
});
+ test.skipIf(process.platform === 'win32')(
+ 'counts a process left defunct by the wait as stopped, without reading its state on every poll',
+ async () => {
+ const psCalls = installPsCallLog();
+ const pid = await startUnreapedServer();
+ const originalPath = process.env.PATH;
+ process.env.PATH = `${psCalls.binDir}:${originalPath ?? ''}`;
+ let outcome: Awaited<ReturnType<typeof stopServerForRemoval>>;
+ try {
+ outcome = await stopServerForRemoval(dir, { timeoutMs: 400, pollIntervalMs: 10 });
+ } finally {
+ process.env.PATH = originalPath;
+ }
+ expect(outcome).toEqual({ stopped: 1, failed: [] });
+ const stateReads = readFileSync(psCalls.path, 'utf8')
+ .split('\n')
+ .filter((call) => call.includes('stat=') && call.includes(String(pid)));
+ expect(stateReads.length).toBeLessThanOrEqual(2);
+ },
+ );
+
test.skipIf(process.platform === 'win32')(
'fails without escalating when the server ignores SIGTERM',
async () => {
diff --git a/packages/cli/src/commands/stop-for-removal.ts b/packages/cli/src/commands/stop-for-removal.ts
index 6cf7e3b52..a7fd05f5e 100644
--- a/packages/cli/src/commands/stop-for-removal.ts
+++ b/packages/cli/src/commands/stop-for-removal.ts
@@ -2,8 +2,11 @@ import { realpath } from 'node:fs/promises';
import { resolve } from 'node:path';
import { setTimeout } from 'node:timers/promises';
import {
+ createProbeFailureReporter,
+ isLockProcessRunning,
isProcessAlive,
type LockProcessScan,
+ type ProcessProbeOptions,
scanLockProcesses,
} from '@inkeep/open-knowledge-server';
import { getCliLogger } from '../cli-logger.ts';
@@ -17,6 +20,7 @@ import { runStop } from './stop.ts';
interface StopForRemovalOptions {
preserveProjectState?: boolean;
+ probe?: (pid: number) => ProcessProbeOptions;
scanProcesses?: () => Promise<LockProcessScan>;
timeoutMs?: number;
pollIntervalMs?: number;
@@ -32,7 +36,10 @@ export async function stopServerForRemoval(
lockDir: string,
options: StopForRemovalOptions = {},
): Promise<{ stopped: number; failed: Array<{ pid: number; error: string }>; skipped?: string }> {
- const isAlive = options.isAlive ?? isProcessAlive;
+ const probeOptions = options.probe ?? createProbeFailureReporter();
+ const isAlive =
+ options.isAlive ?? ((pid: number) => isLockProcessRunning(pid, probeOptions(pid)));
+ const hasProcessEntry = options.isAlive ?? isProcessAlive;
const state = inspectLock(lockDir, 'server', { isAlive });
const lockRecovery =
'Quit OpenKnowledge and stop any OpenKnowledge server processes. ' +
@@ -104,7 +111,7 @@ export async function stopServerForRemoval(
: foreignHost
? 'foreign-owned'
: 'malformed';
- const scan = await (options.scanProcesses ?? scanLockProcesses)();
+ const scan = await (options.scanProcesses ?? (() => scanLockProcesses(probeOptions)))();
const canonical = await realpath(lockDir).catch(() => resolve(lockDir));
const candidates = scan.candidates.filter(
(candidate) => candidate.lockDir === canonical && isAlive(candidate.pid),
@@ -203,13 +210,16 @@ export async function stopServerForRemoval(
const deadline = performance.now() + timeoutMs;
while (pending.size > 0) {
for (const pid of pending) {
- if (!isAlive(pid)) pending.delete(pid);
+ if (!hasProcessEntry(pid)) pending.delete(pid);
}
if (pending.size === 0) break;
const remaining = deadline - performance.now();
if (remaining <= 0) break;
await setTimeout(Math.min(options.pollIntervalMs ?? 50, remaining));
}
+ for (const pid of pending) {
+ if (!isAlive(pid)) pending.delete(pid);
+ }
for (const pid of pending) {
failed.push({ pid, error: `still running ${timeoutMs}ms after SIGTERM` });
}
diff --git a/packages/cli/src/commands/uninstall-lock-recovery.test.ts b/packages/cli/src/commands/uninstall-lock-recovery.test.ts
index 2cf074aed..e082b3c63 100644
--- a/packages/cli/src/commands/uninstall-lock-recovery.test.ts
+++ b/packages/cli/src/commands/uninstall-lock-recovery.test.ts
@@ -168,7 +168,9 @@ describe.skipIf(process.platform === 'win32')('uninstall lock recovery in an iso
const project = join(root, 'project');
write(join(project, '.ok', 'local', 'server.lock'), 'not json');
write(join(project, 'notes.md'), '# Keep');
- const outcome = await runRemoval(buildDeinitPlan(project, join(root, 'home')));
+ const outcome = await runRemoval(buildDeinitPlan(project, join(root, 'home')), {
+ scanProcesses: scanCandidatesWithoutHostUncertainty,
+ });
expect(outcome.failed).toEqual([]);
expect(outcome.results[0]).toMatchObject({
status: 'skipped',
diff --git a/packages/cli/src/commands/unverified-lock-owner.test.ts b/packages/cli/src/commands/unverified-lock-owner.test.ts
index f16183acf..de6510e21 100644
--- a/packages/cli/src/commands/unverified-lock-owner.test.ts
+++ b/packages/cli/src/commands/unverified-lock-owner.test.ts
@@ -1,6 +1,15 @@
-import { mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
+import {
+ chmodSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ realpathSync,
+ rmSync,
+ writeFileSync,
+} from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
+import { isLockProcessRunning, readProcessState } from '@inkeep/open-knowledge-server';
import { afterEach, beforeEach, expect, test, vi } from 'vitest';
import { runClean } from './clean.ts';
import { makeServerLockCheck } from './diagnose-health-checks/server-lock.ts';
@@ -21,7 +30,19 @@ beforeEach(() => {
lockPath = join(lockDir, 'server.lock');
writeFileSync(lockPath, raw);
});
-afterEach(() => rmSync(project, { recursive: true, force: true }));
+afterEach(() => {
+ vi.restoreAllMocks();
+ rmSync(project, { recursive: true, force: true });
+});
+
+function installBlindPsProbe(): string {
+ const binDir = join(project, 'bin');
+ mkdirSync(binDir, { recursive: true });
+ const shim = join(binDir, 'ps');
+ writeFileSync(shim, '#!/bin/sh\nexit 1\n');
+ chmodSync(shim, 0o755);
+ return binDir;
+}
const inspect = () => inspectLock(lockDir, 'server', { isAlive: () => true });
test('models a live PID-only lock without asserting absent metadata', () => {
@@ -151,3 +172,26 @@ test('global-only cleanup still refuses an associated live server candidate', as
).rejects.toThrow('unverified');
expect(readFileSync(lockPath, 'utf8')).toBe(raw);
});
+
+test.skipIf(process.platform === 'win32')(
+ 'reads a live foreign-owned PID as unverified, not dead, when the state probe cannot see it',
+ () => {
+ writeFileSync(lockPath, JSON.stringify({ pid: process.pid }));
+ vi.spyOn(process, 'kill').mockImplementation(() => {
+ throw Object.assign(new Error('kill EPERM'), { code: 'EPERM' });
+ });
+ const binDir = installBlindPsProbe();
+ const originalPath = process.env.PATH;
+ process.env.PATH = binDir;
+ try {
+ expect(readProcessState(process.pid)).toEqual({ status: 'gone' });
+ expect(inspectLock(lockDir, 'server', { isAlive: isLockProcessRunning })).toEqual({
+ status: 'unverified-owner',
+ lockPath,
+ pid: process.pid,
+ });
+ } finally {
+ process.env.PATH = originalPath;
+ }
+ },
+);
diff --git a/packages/server/src/index.ts b/packages/server/src/index.ts
index db6fed53d..d46b13ef8 100644
--- a/packages/server/src/index.ts
+++ b/packages/server/src/index.ts
@@ -429,12 +429,19 @@ export {
updateProcessLockPort,
} from './process-lock.ts';
export {
+ createProbeFailureReporter,
discoverLockDirs,
extractOkBinaryPath,
+ isDefunctProcess,
+ isLockProcessRunning,
type LockProcessScan,
+ type ProcessProbeFailure,
+ type ProcessProbeOptions,
+ type ProcessState,
type ProcessUsage,
processCommand,
processUsage,
+ readProcessState,
scanLockProcesses,
} from './process-scan.ts';
export {
diff --git a/packages/server/src/process-alive.ts b/packages/server/src/process-alive.ts
index 6c79050de..94e8adcef 100644
--- a/packages/server/src/process-alive.ts
+++ b/packages/server/src/process-alive.ts
@@ -1,15 +1,19 @@
-export function isProcessAlive(pid: number): boolean {
+export type ProcessLiveness = 'signalable' | 'foreign' | 'absent';
+
+export function readProcessLiveness(pid: number): ProcessLiveness {
try {
process.kill(pid, 0);
- return true;
+ return 'signalable';
} catch (err: unknown) {
- if (err && typeof err === 'object' && 'code' in err && err.code === 'EPERM') {
- return true;
- }
- return false;
+ if (err && typeof err === 'object' && 'code' in err && err.code === 'EPERM') return 'foreign';
+ return 'absent';
}
}
+export function isProcessAlive(pid: number): boolean {
+ return readProcessLiveness(pid) !== 'absent';
+}
+
export function isValidLockPid(value: unknown): value is number {
if (typeof value !== 'number') return false;
if (!Number.isInteger(value)) return false;
diff --git a/packages/server/src/process-scan.test.ts b/packages/server/src/process-scan.test.ts
index 776e7f64f..456facc18 100644
--- a/packages/server/src/process-scan.test.ts
+++ b/packages/server/src/process-scan.test.ts
@@ -2,6 +2,7 @@ import type { SpawnSyncReturns } from 'node:child_process';
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
const spawnSyncMock = vi.fn();
+const execFileSyncMock = vi.fn();
const existsSyncMock = vi.fn();
const readdirSyncMock = vi.fn();
const lstatSyncMock = vi.fn();
@@ -10,22 +11,36 @@ let scanLockProcesses: typeof import('./process-scan.ts').scanLockProcesses;
let discoverLockDirs: typeof import('./process-scan.ts').discoverLockDirs;
let findOkProcessPids: typeof import('./process-scan.ts').findOkProcessPids;
let readPidCwds: typeof import('./process-scan.ts').readPidCwds;
+let isDefunctProcess: typeof import('./process-scan.ts').isDefunctProcess;
+let isLockProcessRunning: typeof import('./process-scan.ts').isLockProcessRunning;
+let readProcessState: typeof import('./process-scan.ts').readProcessState;
let realCp: typeof import('node:child_process');
let realFs: typeof import('node:fs');
beforeAll(async () => {
realCp = await vi.importActual<typeof import('node:child_process')>('node:child_process');
realFs = await vi.importActual<typeof import('node:fs')>('node:fs');
- vi.doMock('node:child_process', () => ({ ...realCp, spawnSync: spawnSyncMock }));
+ execFileSyncMock.mockImplementation(realCp.execFileSync);
+ vi.doMock('node:child_process', () => ({
+ ...realCp,
+ spawnSync: spawnSyncMock,
+ execFileSync: execFileSyncMock,
+ }));
vi.doMock('node:fs', () => ({
...realFs,
existsSync: existsSyncMock,
readdirSync: readdirSyncMock,
lstatSync: lstatSyncMock,
}));
- ({ scanLockProcesses, discoverLockDirs, findOkProcessPids, readPidCwds } = await import(
- './process-scan.ts'
- ));
+ ({
+ scanLockProcesses,
+ discoverLockDirs,
+ findOkProcessPids,
+ readPidCwds,
+ isDefunctProcess,
+ isLockProcessRunning,
+ readProcessState,
+ } = await import('./process-scan.ts'));
});
function refuseUnmockedSpawn(command: string, args: readonly string[] = []): never {
@@ -516,11 +531,11 @@ describe('discoverLockDirs', () => {
(p: unknown) => p === lockDir || p === `${lockDir}/server.lock`,
);
readdirSyncSpy.mockImplementation((p: unknown) => {
- if (p === parent) return ['garth_nix'] as unknown as ReturnType<typeof fs.readdirSync>;
- return [] as unknown as ReturnType<typeof fs.readdirSync>;
+ if (p === parent) return ['garth_nix'] as unknown as ReturnType<typeof realFs.readdirSync>;
+ return [] as unknown as ReturnType<typeof realFs.readdirSync>;
});
lstatSyncSpy.mockImplementation(
- () => ({ isDirectory: () => true }) as unknown as ReturnType<typeof fs.lstatSync>,
+ () => ({ isDirectory: () => true }) as unknown as ReturnType<typeof realFs.lstatSync>,
);
try {
@@ -564,11 +579,11 @@ describe('discoverLockDirs', () => {
p === `${childLockDir}/server.lock`,
);
readdirSyncSpy.mockImplementation((p: unknown) => {
- if (p === parent) return ['garth_nix'] as unknown as ReturnType<typeof fs.readdirSync>;
- return [] as unknown as ReturnType<typeof fs.readdirSync>;
+ if (p === parent) return ['garth_nix'] as unknown as ReturnType<typeof realFs.readdirSync>;
+ return [] as unknown as ReturnType<typeof realFs.readdirSync>;
});
lstatSyncSpy.mockImplementation(
- () => ({ isDirectory: () => true }) as unknown as ReturnType<typeof fs.lstatSync>,
+ () => ({ isDirectory: () => true }) as unknown as ReturnType<typeof realFs.lstatSync>,
);
try {
@@ -606,11 +621,20 @@ describe('discoverLockDirs', () => {
});
describe('lock recovery process evidence', () => {
+ let stderrWrites: string[];
beforeEach(() => {
spawnSyncMock.mockReset();
+ execFileSyncMock.mockReset().mockImplementation(realCp.execFileSync);
+ stderrWrites = [];
+ vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array) => {
+ stderrWrites.push(String(chunk));
+ return true;
+ });
});
afterEach(() => {
+ vi.restoreAllMocks();
spawnSyncMock.mockReset();
+ execFileSyncMock.mockReset().mockImplementation(realCp.execFileSync);
});
it('does not mistake failed process enumeration for evidence of absence', async () => {
@@ -631,6 +655,17 @@ describe('lock recovery process evidence', () => {
'Could not enumerate TCP listeners with lsof',
]);
});
+ it('treats a listener query that exits non-zero with a table as incomplete enumeration', async () => {
+ spawnSyncMock.mockReturnValueOnce(makeSpawnResult({ status: 1 })).mockReturnValueOnce(
+ makeSpawnResult({
+ status: 1,
+ stdout: `COMMAND PID USER\nnode ${process.pid} user\n`,
+ }),
+ );
+ expect((await scanLockProcesses()).unavailable).toEqual([
+ 'Could not enumerate TCP listeners with lsof',
+ ]);
+ });
it('keeps explicit process provenance even without a lock file on disk', async () => {
const lockDir = '/missing/project/.ok/local';
spawnSyncMock
@@ -669,6 +704,149 @@ describe('lock recovery process evidence', () => {
`Could not read the working directory of process ${process.pid}`,
]);
});
+
+ it('retains a live candidate whose process-state probe could not answer', async () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('spawnSync ps ENOENT'), { code: 'ENOENT' });
+ });
+ spawnSyncMock
+ .mockReturnValueOnce(
+ makeSpawnResult({ stdout: `${process.pid} open-knowledge-server notes\n` }),
+ )
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }))
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }));
+
+ const scan = await scanLockProcesses();
+
+ const stateProbes = execFileSyncMock.mock.calls.filter((call) =>
+ (call[1] as string[] | undefined)?.includes(String(process.pid)),
+ );
+ expect(stateProbes).toHaveLength(1);
+ expect(scan.unavailable).toEqual([
+ `Could not read the working directory of process ${process.pid}`,
+ ]);
+ expect(stderrWrites.join('')).toContain(
+ `could not read the state of process ${process.pid} (ENOENT)`,
+ );
+ });
+ it('reports a process-state probe that ran and could not answer', async () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), {
+ status: 1,
+ stderr: 'ps: nosuchcol: keyword not found\nps: no valid keywords\n',
+ });
+ });
+ spawnSyncMock
+ .mockReturnValueOnce(
+ makeSpawnResult({ stdout: `${process.pid} open-knowledge-server notes\n` }),
+ )
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }))
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }));
+
+ const scan = await scanLockProcesses();
+
+ expect(scan.unavailable).toEqual([
+ `Could not read the working directory of process ${process.pid}`,
+ ]);
+ expect(stderrWrites.join('')).toContain(
+ `could not read the state of process ${process.pid} (ps: nosuchcol: keyword not found)`,
+ );
+ });
+
+ it('reports a probe failure whose reason follows a blank stderr line', async () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), {
+ status: 1,
+ stderr: '\nps: nosuchcol: keyword not found\nps: no valid keywords\n',
+ });
+ });
+ spawnSyncMock
+ .mockReturnValueOnce(
+ makeSpawnResult({ stdout: `${process.pid} open-knowledge-server notes\n` }),
+ )
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }))
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }));
+
+ const scan = await scanLockProcesses();
+
+ expect(scan.unavailable).toEqual([
+ `Could not read the working directory of process ${process.pid}`,
+ ]);
+ expect(stderrWrites.join('')).toContain(
+ `could not read the state of process ${process.pid} (ps: nosuchcol: keyword not found)`,
+ );
+ });
+
+ it('renders a multi-line probe failure as one diagnostic line', async () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), {
+ status: 1,
+ stderr: 'error: unknown user-defined format specifier "bogus"\n\nUsage:\n ps [options]\n',
+ });
+ });
+ spawnSyncMock
+ .mockReturnValueOnce(
+ makeSpawnResult({ stdout: `${process.pid} open-knowledge-server notes\n` }),
+ )
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }))
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }));
+
+ await scanLockProcesses();
+
+ const written = stderrWrites.join('');
+ expect(written).toContain(
+ `could not read the state of process ${process.pid} (error: unknown user-defined format specifier "bogus")`,
+ );
+ expect(written.trimEnd().split('\n')).toHaveLength(1);
+ });
+
+ it('keeps no uncertainty and stays silent when the process-state probe finds the pid already gone', async () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), { status: 1, stderr: '' });
+ });
+ spawnSyncMock
+ .mockReturnValueOnce(
+ makeSpawnResult({ stdout: `${process.pid} open-knowledge-server notes\n` }),
+ )
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }))
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }));
+
+ const scan = await scanLockProcesses();
+
+ const stateProbes = execFileSyncMock.mock.calls.filter((call) =>
+ (call[1] as string[] | undefined)?.includes(String(process.pid)),
+ );
+ expect(stateProbes).toHaveLength(1);
+ expect(scan.unavailable).toEqual([]);
+ expect(stderrWrites.join('')).toBe('');
+ });
+
+ it('warns once per process when one scan probes the same pid twice', async () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('spawnSync ps ENOENT'), { code: 'ENOENT' });
+ });
+ spawnSyncMock
+ .mockReturnValueOnce(
+ makeSpawnResult({
+ stdout: `${process.pid} open-knowledge-server notes\n${process.pid} node cli.mjs --ok-project-path=/notes\n`,
+ }),
+ )
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }))
+ .mockReturnValueOnce(makeSpawnResult({ status: 1 }));
+
+ await scanLockProcesses();
+
+ const stateProbes = execFileSyncMock.mock.calls.filter((call) =>
+ (call[1] as string[] | undefined)?.includes(String(process.pid)),
+ );
+ expect(stateProbes).toHaveLength(2);
+ const warnings = stderrWrites
+ .join('')
+ .split('\n')
+ .filter((line) => line.includes(`could not read the state of process ${process.pid}`));
+ expect(warnings).toHaveLength(1);
+ });
+
it('retains a live candidate the batched query could not answer for', async () => {
spawnSyncMock
.mockReturnValueOnce(
@@ -722,3 +900,133 @@ describe('lock recovery process evidence', () => {
});
});
});
+
+describe('isDefunctProcess', () => {
+ const realPlatform = process.platform;
+ let stderrWrites: string[];
+ beforeEach(() => {
+ execFileSyncMock.mockReset();
+ stderrWrites = [];
+ vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array) => {
+ stderrWrites.push(String(chunk));
+ return true;
+ });
+ });
+ afterEach(() => {
+ Object.defineProperty(process, 'platform', { value: realPlatform, configurable: true });
+ vi.restoreAllMocks();
+ execFileSyncMock.mockReset().mockImplementation(realCp.execFileSync);
+ });
+
+ it('skips the probe entirely on an unsupported platform', () => {
+ execFileSyncMock.mockReturnValue('SN \n');
+ expect(isDefunctProcess(4242)).toBe(false);
+ expect(execFileSyncMock).toHaveBeenCalledTimes(1);
+ Object.defineProperty(process, 'platform', { value: 'win32', configurable: true });
+ expect(isDefunctProcess(4242)).toBe(false);
+ expect(execFileSyncMock).toHaveBeenCalledTimes(1);
+ });
+ it('reads a zombie state as defunct', () => {
+ execFileSyncMock.mockReturnValue('Z+ \n');
+ expect(isDefunctProcess(4242)).toBe(true);
+ });
+ it('reads any other state as not defunct', () => {
+ execFileSyncMock.mockReturnValue('SN \n');
+ expect(isDefunctProcess(4242)).toBe(false);
+ });
+ it('reads a ps-confirmed absence as not defunct without reporting', () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), { status: 1, stderr: '' });
+ });
+ expect(isDefunctProcess(4242)).toBe(false);
+ expect(stderrWrites.join('')).toBe('');
+ });
+ it('reads a probe that could not run as not defunct and reports why', () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('spawnSync ps ENOENT'), { code: 'ENOENT' });
+ });
+ expect(isDefunctProcess(4242)).toBe(false);
+ expect(stderrWrites.join('')).toContain('could not read the state of process 4242 (ENOENT)');
+ });
+});
+
+describe('readProcessState', () => {
+ const realPlatform = process.platform;
+ let stderrWrites: string[];
+ beforeEach(() => {
+ execFileSyncMock.mockReset();
+ stderrWrites = [];
+ vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array) => {
+ stderrWrites.push(String(chunk));
+ return true;
+ });
+ });
+ afterEach(() => {
+ Object.defineProperty(process, 'platform', { value: realPlatform, configurable: true });
+ vi.restoreAllMocks();
+ execFileSyncMock.mockReset().mockImplementation(realCp.execFileSync);
+ });
+
+ it('tells a confirmed absence apart from a confirmed live process', () => {
+ execFileSyncMock.mockReturnValue('SN \n');
+ expect(readProcessState(4242)).toEqual({ status: 'running' });
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), { status: 1, stderr: '' });
+ });
+ expect(readProcessState(4242)).toEqual({ status: 'gone' });
+ });
+ it('tells a zombie apart from a probe that could not answer', () => {
+ execFileSyncMock.mockReturnValue('Z+ \n');
+ expect(readProcessState(4242)).toEqual({ status: 'defunct' });
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('spawnSync ps ENOENT'), { code: 'ENOENT' });
+ });
+ expect(readProcessState(4242)).toEqual({ status: 'unreadable', reason: 'ENOENT' });
+ });
+ it('reports an unsupported platform as unreadable without probing or writing', () => {
+ Object.defineProperty(process, 'platform', { value: 'win32', configurable: true });
+ expect(readProcessState(4242)).toEqual({
+ status: 'unreadable',
+ reason: 'no process-state query on win32',
+ });
+ expect(execFileSyncMock).not.toHaveBeenCalled();
+ expect(stderrWrites.join('')).toBe('');
+ });
+});
+
+describe('isLockProcessRunning', () => {
+ const realPlatform = process.platform;
+ beforeEach(() => {
+ execFileSyncMock.mockReset();
+ });
+ afterEach(() => {
+ Object.defineProperty(process, 'platform', { value: realPlatform, configurable: true });
+ vi.restoreAllMocks();
+ execFileSyncMock.mockReset().mockImplementation(realCp.execFileSync);
+ });
+
+ it('lets a probe-confirmed absence override a process that was alive a moment earlier', () => {
+ execFileSyncMock.mockReturnValue('SN \n');
+ expect(isLockProcessRunning(process.pid)).toBe(true);
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), { status: 1, stderr: '' });
+ });
+ expect(isLockProcessRunning(process.pid)).toBe(false);
+ });
+ it('keeps a live process running when the probe could not answer', () => {
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('spawnSync ps ENOENT'), { code: 'ENOENT' });
+ });
+ vi.spyOn(process.stderr, 'write').mockReturnValue(true);
+ expect(isLockProcessRunning(process.pid)).toBe(true);
+ });
+ it('keeps a foreign-owned process running when the probe cannot see it', () => {
+ vi.spyOn(process, 'kill').mockImplementation(() => {
+ throw Object.assign(new Error('kill EPERM'), { code: 'EPERM' });
+ });
+ execFileSyncMock.mockImplementation(() => {
+ throw Object.assign(new Error('Command failed: ps'), { status: 1, stderr: '' });
+ });
+ expect(isLockProcessRunning(4242)).toBe(true);
+ });
+});
diff --git a/packages/server/src/process-scan.ts b/packages/server/src/process-scan.ts
index 053280ef1..129d7e472 100644
--- a/packages/server/src/process-scan.ts
+++ b/packages/server/src/process-scan.ts
@@ -1,9 +1,12 @@
-import { spawnSync } from 'node:child_process';
+import { execFileSync, spawnSync } from 'node:child_process';
import { existsSync, lstatSync, readdirSync } from 'node:fs';
import { realpath } from 'node:fs/promises';
import { basename, isAbsolute, join, resolve } from 'node:path';
-import { withHiddenWindowsConsole } from './child-process-windows-hide.ts';
-import { isProcessAlive, isValidLockPid } from './process-alive.ts';
+import {
+ LOCAL_OP_PIPE_STDIO_OPTIONS,
+ withHiddenWindowsConsole,
+} from './child-process-windows-hide.ts';
+import { isValidLockPid, type ProcessLiveness, readProcessLiveness } from './process-alive.ts';
const SPAWN_TIMEOUT_MS = 2000;
const LOCK_SCAN_MAX_DEPTH = 3;
@@ -193,6 +196,85 @@ export function processUsage(pid: number): ProcessUsage | null {
return { cpuPercent, memPercent };
}
+export interface ProcessProbeFailure {
+ pid: number;
+ reason: string;
+}
+
+export interface ProcessProbeOptions {
+ onProbeFailure?: (failure: ProcessProbeFailure) => void;
+}
+
+function renderProbeFailure(failure: ProcessProbeFailure): string {
+ return `[process-scan] could not read the state of process ${failure.pid} (${failure.reason}); treating it as not defunct\n`;
+}
+
+export type ProcessState =
+ | { status: 'running' }
+ | { status: 'defunct' }
+ | { status: 'gone' }
+ | { status: 'unreadable'; reason: string };
+
+function reportUnreadableState(
+ pid: number,
+ reason: string,
+ options: ProcessProbeOptions,
+): ProcessState {
+ const failure: ProcessProbeFailure = { pid, reason };
+ if (options.onProbeFailure) options.onProbeFailure(failure);
+ else process.stderr.write(renderProbeFailure(failure));
+ return { status: 'unreadable', reason };
+}
+
+export function readProcessState(pid: number, options: ProcessProbeOptions = {}): ProcessState {
+ if (process.platform !== 'darwin' && process.platform !== 'linux') {
+ return { status: 'unreadable', reason: `no process-state query on ${process.platform}` };
+ }
+ try {
+ const state = execFileSync(
+ 'ps',
+ ['-p', String(pid), '-o', 'stat='],
+ withHiddenWindowsConsole({
+ encoding: 'utf8',
+ ...LOCAL_OP_PIPE_STDIO_OPTIONS,
+ timeout: SPAWN_TIMEOUT_MS,
+ env: { ...process.env, LC_ALL: 'C' },
+ }),
+ );
+ return state.trim().startsWith('Z') ? { status: 'defunct' } : { status: 'running' };
+ } catch (err: unknown) {
+ if (!(err instanceof Error)) return reportUnreadableState(pid, String(err), options);
+ const failure = err as NodeJS.ErrnoException & {
+ status?: number | null;
+ stderr?: Buffer | string | null;
+ };
+ const stderrText = String(failure.stderr ?? '');
+ if (typeof failure.status === 'number' && stderrText.trim() === '') return { status: 'gone' };
+ const diagnostic =
+ stderrText
+ .split('\n')
+ .find((line) => line.trim() !== '')
+ ?.trim() ?? '';
+ return reportUnreadableState(pid, diagnostic || failure.code || failure.message, options);
+ }
+}
+
+export function isDefunctProcess(pid: number, options: ProcessProbeOptions = {}): boolean {
+ return readProcessState(pid, options).status === 'defunct';
+}
+
+function psAbsenceOutranksLiveness(state: ProcessState, liveness: ProcessLiveness): boolean {
+ return state.status === 'gone' && liveness === 'signalable';
+}
+
+export function isLockProcessRunning(pid: number, options: ProcessProbeOptions = {}): boolean {
+ const liveness = readProcessLiveness(pid);
+ if (liveness === 'absent') return false;
+ const state = readProcessState(pid, options);
+ if (state.status === 'defunct') return false;
+ return !psAbsenceOutranksLiveness(state, liveness);
+}
+
function parsePidCwds(stdout: string): Map<number, string> {
const cwds = new Map<number, string>();
let pid: number | null = null;
@@ -376,7 +458,20 @@ export interface LockProcessScan {
unavailable: string[];
}
-export async function scanLockProcesses(): Promise<LockProcessScan> {
+export function createProbeFailureReporter(): (pid: number) => ProcessProbeOptions {
+ const warned = new Set<number>();
+ return (pid) => ({
+ onProbeFailure: (failure) => {
+ if (warned.has(pid)) return;
+ warned.add(pid);
+ process.stderr.write(renderProbeFailure(failure));
+ },
+ });
+}
+
+export async function scanLockProcesses(
+ probeOptions: (pid: number) => ProcessProbeOptions = createProbeFailureReporter(),
+): Promise<LockProcessScan> {
const scan: LockProcessScan = { candidates: [], unavailable: [] };
let entries: OkProcessEntry[];
try {
@@ -431,9 +526,14 @@ export async function scanLockProcesses(): Promise<LockProcessScan> {
for (const { pid, source } of pendingCwd) {
const cwd = cwds.get(pid);
if (cwd) await addProject(cwd, pid, source);
- else if (isProcessAlive(pid))
+ else if (isLockProcessRunning(pid, probeOptions(pid)))
scan.unavailable.push(`Could not read the working directory of process ${pid}`);
}
if (listenersUnavailable) scan.unavailable.push('Could not enumerate TCP listeners with lsof');
+ const defunctCandidates = new Set<number>();
+ for (const pid of new Set(scan.candidates.map((candidate) => candidate.pid))) {
+ if (isDefunctProcess(pid, probeOptions(pid))) defunctCandidates.add(pid);
+ }
+ scan.candidates = scan.candidates.filter((candidate) => !defunctCandidates.has(candidate.pid));
return scan;
}