Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

fix(desktop): capture where a frozen main thread was stuck (PRD-8894) (#5144)

Andrew Mikofalvya0c41a477457
diff --git a/.changeset/main-thread-stall-snapshot.md b/.changeset/main-thread-stall-snapshot.md
new file mode 100644
index 000000000..0121ec8ff
--- /dev/null
+++ b/.changeset/main-thread-stall-snapshot.md
@@ -0,0 +1,5 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+Bug reports sent after the desktop app froze or ended unexpectedly now record where the app was stuck and whether it exited on its own or was killed.
diff --git a/docs/content/reference/what-open-knowledge-writes.mdx b/docs/content/reference/what-open-knowledge-writes.mdx
index 2fc529408..af7ea06b0 100644
--- a/docs/content/reference/what-open-knowledge-writes.mdx
+++ b/docs/content/reference/what-open-knowledge-writes.mdx
@@ -51,6 +51,8 @@ On every platform the app writes app-level state and logs on launch. The table b
| `~/Library/Application Support/OpenKnowledge/bug-report-dirty-shutdown.json` | Dirty-shutdown sentinel — written on **every** launch, refreshed as it runs, and removed on a clean quit. The next launch reads it to tell an app crash (prompts you) from the endings that are not a crash: a reboot, an OS shutdown, or dying asleep (all suppressed and logged instead), an update install shutting the app down to replace its files (logged either way, and suppressed in almost every case: a shutdown the app had no chance to record, or one it cannot date because the previous session left no heartbeat, can still prompt so you are able to report it), and a death the app can date to more than a week ago, which is too stale to report usefully — the diagnostics a report would gather describe the day you sent it rather than the day it happened. A fresh crash dump still overrides those, the same as it does for a reboot, a suspend or an OS shutdown — except in the first 30 seconds after the app committed to quitting for an update install, where a dump is part of that same shutdown rather than a crash you could act on. A session that left neither a heartbeat nor a crash dump cannot be dated, so it is never treated as stale | Outside-project (home dir) |
| `~/Library/Application Support/OpenKnowledge/bug-report-crash-acks.json` | Records which crash-report invitations you already answered or dismissed, so the same crash never re-prompts. Answering is what retires a crash: an invitation you simply never saw — because it went unanswered for a day, or the window it was waiting on never appeared — is dropped rather than answered, so that crash can be offered again the next time you open the app | Outside-project (home dir) |
| `~/Library/Application Support/OpenKnowledge/bug-report-main-thread-liveness.json` | Main-thread liveness witness. A background worker rewrites it every five seconds while the app runs. The next launch can identify a main-thread block that had lasted at least 15 seconds at the last successful sample. A sample below that threshold does not rule out a shorter or recovered block, or a block after the worker stopped writing. Read only at the next launch; left in place on a clean quit and overwritten on the next run, through a `.tmp-` sibling during each write. The verdict lands in the desktop log a bug report carries; it does not change whether the app prompts you | Outside-project (home dir) |
+| `~/Library/Application Support/OpenKnowledge/bug-report-main-thread-stall.json` | Main-thread stall snapshot. When the liveness witness first sees the main thread blocked for at least 15 seconds, the same background worker writes this file at once, then briefly pauses the main thread to read where it is and resumes it straight away. It holds function names and script locations from the app's own code, and the Node.js runtime beneath it, at the moment of a freeze: at most 24 frames, never document content. A freeze inside native code only lets the pause land once the main thread returns to the app's code, so the file records how long that took, and a session that ended before then keeps the record written when the freeze was first seen. One snapshot per freeze; a later freeze in the same session overwrites it, through a `.tmp-` sibling during each write. A freeze the app recovers from is also noted in the desktop log as it recovers. Read at the next launch; left in place on a clean quit and overwritten by the next freeze. It does not change whether the app prompts you | Outside-project (home dir) |
+| `~/Library/Application Support/OpenKnowledge/bug-report-main-exit.json` | Main-process exit record. Written only when the main process exits without the app having marked a clean quit, for example when code ends the process directly instead of going through the normal quit path. It holds the session's identifier, the time, and the exit code. A crash or a forced kill never gets to write it, which is how the next launch tells those endings apart from an exit the app ran itself. Read only at the next launch, where the result lands in the desktop log a bug report carries; left in place and overwritten by the next such exit. It does not change whether the app prompts you | Outside-project (home dir) |
Stable registers `openknowledge://`; Beta registers `openknowledge-beta://`. Both check for updates — on launch and periodically while running — against separate Stable/Beta paths on the OpenKnowledge update service at `openknowledge.ai/updates`, which redirects to the matching GitHub release asset and counts the update per version and channel (Stable alone can fall back to GitHub directly if the proxy is unreachable). On macOS and Windows an update on your channel downloads in the background and installs on the next quit; on Linux — where there is no apt/rpm repository — the app downloads the new deb/rpm from the release feed and asks you to authorize the install (a `pkexec` password prompt) each time. You can also check on demand from **Check for updates…** in the app and Help menus. The channel is fixed by the product you installed. Older Beta installations named `OpenKnowledge` continue receiving Beta updates in place, preserving their existing settings and CLI commands; they do not silently become the separately installed `OpenKnowledge Beta` app. Reinstall Stable from `openknowledge.ai/download` or Beta from `openknowledge.ai/download/beta`; Windows and Linux installers are available from [GitHub Releases](https://github.com/inkeep/open-knowledge/releases). Override the feed with `OK_UPDATER_FEED_URL`.
diff --git a/packages/desktop/src/main/crash-detection.test.ts b/packages/desktop/src/main/crash-detection.test.ts
index bdbb0cdf4..e8f20c509 100644
--- a/packages/desktop/src/main/crash-detection.test.ts
+++ b/packages/desktop/src/main/crash-detection.test.ts
@@ -21,11 +21,13 @@ import {
INVITE_EXPIRE_AFTER_MS,
type InstallInFlight,
MAX_DECLINED_DEATHS,
+ type MainExitRecord,
+ parseMainExitRecord,
SENTINEL_HEARTBEAT_INTERVAL_MS,
STALE_CRASH_AFTER_MS,
startLocalCrashReporter,
} from './crash-detection.ts';
-import type { WatchdogRead } from './main-thread-watchdog.ts';
+import type { StallRead, StallSnapshot, WatchdogRead } from './main-thread-watchdog.ts';
import { buildMinidump } from './minidump.test-helper.ts';
import { type AppState, emptyState } from './state-store.ts';
@@ -63,6 +65,7 @@ interface Rig {
setInstallInFlight(inFlight: InstallInFlight | null): void;
setAppVersion(version: string): void;
setWatchdogRead(read: WatchdogRead): void;
+ setStallRead(read: StallRead): void;
watchdogStarts: string[];
watchdogStops(): number;
tick(): Date;
@@ -81,6 +84,7 @@ function makeRig(): Rig {
let bootSessionUuid: string | null = 'boot-epoch-a';
let installInFlight: InstallInFlight | null = null;
let watchdogRead: WatchdogRead = { kind: 'absent' };
+ let stallRead: StallRead = { kind: 'absent' };
const watchdogStarts: string[] = [];
let watchdogStops = 0;
let clockMs = Date.parse('2026-07-10T00:00:00.000Z');
@@ -133,6 +137,9 @@ function makeRig(): Rig {
setWatchdogRead(read: WatchdogRead) {
watchdogRead = read;
},
+ setStallRead(read: StallRead) {
+ stallRead = read;
+ },
watchdogStarts,
watchdogStops: () => watchdogStops,
tick() {
@@ -147,6 +154,7 @@ function makeRig(): Rig {
},
deps: {
sentinelPath: join(dir, 'user-data', 'bug-report-dirty-shutdown.json'),
+ mainExitPath: join(dir, 'user-data', 'bug-report-main-exit.json'),
ackStorePath: join(dir, 'user-data', 'bug-report-crash-acks.json'),
crashDumpsDir: join(dir, 'crash-dumps'),
appBundleRoot,
@@ -165,9 +173,11 @@ function makeRig(): Rig {
installInFlight: () => installInFlight,
mainThreadWatchdog: {
readPrevious: () => watchdogRead,
+ readPreviousStall: () => stallRead,
start: (bootId: string) => {
watchdogStarts.push(bootId);
watchdogRead = { kind: 'absent' };
+ stallRead = { kind: 'absent' };
return {
stop: () => {
watchdogStops += 1;
@@ -3092,6 +3102,37 @@ describe('whether the previous session died or hung', () => {
return line as Record<string, unknown>;
}
+ function stallFor(
+ rig: Rig,
+ outcome: 'pending' | 'captured',
+ stalledBeforeWitnessMs: number,
+ bootId = readSentinel(rig).bootId ?? '',
+ ): StallSnapshot {
+ const stallStartedAtMs = rig.nowMs() - stalledBeforeWitnessMs;
+ const request = {
+ schemaVersion: 1 as const,
+ bootId,
+ stallStartedAt: new Date(stallStartedAtMs).toISOString(),
+ requestedAt: new Date(stallStartedAtMs + 15_000).toISOString(),
+ blockedForMsAtRequest: 15_000,
+ };
+ const snapshot: StallSnapshot =
+ outcome === 'pending'
+ ? { ...request, outcome }
+ : {
+ ...request,
+ outcome,
+ capturedAt: new Date(stallStartedAtMs + 15_025).toISOString(),
+ pauseLatencyMs: 25,
+ frames: [
+ { functionName: 'rebuildIndex', url: 'file:///app/index.js', line: 3, column: 9 },
+ ],
+ framesTruncated: false,
+ };
+ rig.setStallRead({ kind: 'record', value: snapshot });
+ return snapshot;
+ }
+
test('the previous witness is read before this session starts its own', () => {
const rig = makeRig();
createCrashDetection(rig.deps).detectBootCrash();
@@ -3185,6 +3226,7 @@ describe('whether the previous session died or hung', () => {
sessionA.detectBootCrash();
sessionA.noteOsShutdown();
witnessFor(rig, 90_000);
+ stallFor(rig, 'captured', 90_000);
const lines = captureLines(rig);
expect(createCrashDetection(rig.deps).detectBootCrash()).toBeNull();
@@ -3193,6 +3235,94 @@ describe('whether the previous session died or hung', () => {
expect(breadcrumb?.livenessVerdict).toBe('blocked');
expect(breadcrumb?.mainThreadBlockedForMs).toBe(90_000);
expect(breadcrumb?.livenessEvidence).toBe('matched');
+ expect(breadcrumb?.mainThreadStall).toMatchObject({ outcome: 'captured', episode: 'final' });
+ expect(breadcrumb?.mainExitEvidence).toBe('absent');
+ });
+
+ test('a stall snapshot of the freeze the witness saw last rides the boot line as the final episode', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+ witnessFor(rig, 36_000);
+ const snapshot = stallFor(rig, 'captured', 36_000);
+ const lines = captureLines(rig);
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ const breadcrumb = bootBreadcrumb(lines);
+ expect(breadcrumb.mainThreadStallEvidence).toBe('matched');
+ expect(breadcrumb.mainThreadStall).toStrictEqual({
+ outcome: 'captured',
+ episode: 'final',
+ stallStartedAt: snapshot.stallStartedAt,
+ requestedAt: snapshot.requestedAt,
+ blockedForMsAtRequest: 15_000,
+ capturedAt: snapshot.outcome === 'captured' ? snapshot.capturedAt : null,
+ pauseLatencyMs: 25,
+ frames: [{ functionName: 'rebuildIndex', url: 'file:///app/index.js', line: 3, column: 9 }],
+ framesTruncated: false,
+ error: null,
+ });
+ });
+
+ test('a stall the session recovered from before dying is an earlier episode', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+ witnessFor(rig, 4_000);
+ stallFor(rig, 'captured', 600_000);
+ const lines = captureLines(rig);
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ const breadcrumb = bootBreadcrumb(lines);
+ expect(breadcrumb.livenessVerdict).toBe('died');
+ expect(breadcrumb.mainThreadStall).toMatchObject({ outcome: 'captured', episode: 'earlier' });
+ });
+
+ test('a freeze whose pause never landed boots as a pending stall', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+ witnessFor(rig, 36_000);
+ stallFor(rig, 'pending', 36_000);
+ const lines = captureLines(rig);
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootBreadcrumb(lines).mainThreadStall).toMatchObject({
+ outcome: 'pending',
+ episode: 'final',
+ pauseLatencyMs: null,
+ frames: null,
+ });
+ });
+
+ test('no stall snapshot spells the stall out as null with its evidence', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+ const lines = captureLines(rig);
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ const breadcrumb = bootBreadcrumb(lines);
+ expect(breadcrumb).toHaveProperty('mainThreadStall');
+ expect(breadcrumb.mainThreadStall).toBeNull();
+ expect(breadcrumb.mainThreadStallEvidence).toBe('absent');
+ });
+
+ test('a stall snapshot from another session or a torn one is not evidence', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+ stallFor(rig, 'captured', 36_000, 'a-much-older-boot');
+ const mismatched = captureLines(rig);
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ rig.setStallRead({ kind: 'unreadable' });
+ const torn = captureLines(rig);
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootBreadcrumb(mismatched).mainThreadStall).toBeNull();
+ expect(bootBreadcrumb(mismatched).mainThreadStallEvidence).toBe('boot-mismatch');
+ expect(bootBreadcrumb(torn).mainThreadStall).toBeNull();
+ expect(bootBreadcrumb(torn).mainThreadStallEvidence).toBe('unreadable');
});
test('the witness for this session starts under the bootId this boot armed', () => {
@@ -3223,3 +3353,165 @@ describe('whether the previous session died or hung', () => {
expect(rig.watchdogStops()).toBe(1);
});
});
+
+describe('how the previous main process exited', () => {
+ function bootLine(rig: Rig): Record<string, unknown> | undefined {
+ return rig.infos.filter((line) => line.event === 'crash-detection.boot').at(-1);
+ }
+
+ function exitRecord(overrides: Partial<Record<keyof MainExitRecord, unknown>> = {}): string {
+ return JSON.stringify({
+ schemaVersion: 1,
+ bootId: 'boot-1',
+ exitedAt: '2026-07-10T00:00:00.000Z',
+ exitCode: 7,
+ ...overrides,
+ });
+ }
+
+ test('an exit the quit path never marked clean is recorded for the next boot', () => {
+ const rig = makeRig();
+ const dying = createCrashDetection(rig.deps);
+ dying.detectBootCrash();
+ const bootId = readSentinel(rig).bootId;
+
+ dying.noteProcessExit(7);
+
+ const written = parseMainExitRecord(readFileSync(rig.deps.mainExitPath, 'utf8'));
+ expect(written).toMatchObject({ schemaVersion: 1, bootId, exitCode: 7 });
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootLine(rig)?.mainExit).toStrictEqual({
+ exitCode: 7,
+ exitedAt: written?.exitedAt,
+ });
+ expect(bootLine(rig)?.mainExitEvidence).toBe('matched');
+ });
+
+ test('an exit record cannot be attributed when the previous session left no readable sentinel', () => {
+ const rig = makeRig();
+ const dying = createCrashDetection(rig.deps);
+ dying.detectBootCrash();
+ dying.noteProcessExit(7);
+ writeFileSync(rig.deps.sentinelPath, 'torn-write-not-json');
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootLine(rig)?.mainExit).toBeNull();
+ expect(bootLine(rig)?.mainExitEvidence).toBe('no-previous-boot');
+ });
+
+ test('a clean quit leaves no exit record', () => {
+ const rig = makeRig();
+ const quitting = createCrashDetection(rig.deps);
+ quitting.detectBootCrash();
+
+ quitting.markCleanQuit();
+ quitting.noteProcessExit(0);
+
+ expect(existsSync(rig.deps.mainExitPath)).toBe(false);
+ });
+
+ test('an exit before this session armed its sentinel records nothing', () => {
+ const rig = makeRig();
+
+ createCrashDetection(rig.deps).noteProcessExit(1);
+
+ expect(existsSync(rig.deps.mainExitPath)).toBe(false);
+ });
+
+ test('a session killed without running its exit handler boots with the exit absent', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootLine(rig)).toHaveProperty('mainExit');
+ expect(bootLine(rig)?.mainExit).toBeNull();
+ expect(bootLine(rig)?.mainExitEvidence).toBe('absent');
+ });
+
+ test('an exit record left by an older session is not evidence about this death', () => {
+ const rig = makeRig();
+ const sessionA = createCrashDetection(rig.deps);
+ sessionA.detectBootCrash();
+ sessionA.noteProcessExit(9);
+ createCrashDetection(rig.deps).detectBootCrash();
+ expect(bootLine(rig)?.mainExitEvidence).toBe('matched');
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootLine(rig)?.mainExit).toBeNull();
+ expect(bootLine(rig)?.mainExitEvidence).toBe('boot-mismatch');
+ });
+
+ test('a torn exit record reads as unreadable and warns', () => {
+ const rig = makeRig();
+ createCrashDetection(rig.deps).detectBootCrash();
+ writeFileSync(rig.deps.mainExitPath, '{"schemaVersion":1,"bootId":');
+
+ createCrashDetection(rig.deps).detectBootCrash();
+
+ expect(bootLine(rig)?.mainExit).toBeNull();
+ expect(bootLine(rig)?.mainExitEvidence).toBe('unreadable');
+ expect(rig.warnings.map((w) => w.event)).toContain('crash-detection.main-exit-read-failed');
+ });
+
+ test('an exit record that cannot be written warns and does not throw', () => {
+ const rig = makeRig();
+ const blocker = join(rig.dir, 'not-a-directory');
+ writeFileSync(blocker, '');
+ rig.deps.mainExitPath = join(blocker, 'bug-report-main-exit.json');
+ const detection = createCrashDetection(rig.deps);
+ detection.detectBootCrash();
+
+ expect(() => detection.noteProcessExit(3)).not.toThrow();
+ expect(
+ rig.warnings.filter((w) => w.event === 'crash-detection.main-exit-write-failed'),
+ ).toEqual([expect.objectContaining({ exitCode: 3 })]);
+ });
+
+ test('a logger torn down before exit cannot make the exit handler throw', () => {
+ const rig = makeRig();
+ const blocker = join(rig.dir, 'not-a-directory');
+ writeFileSync(blocker, '');
+ rig.deps.mainExitPath = join(blocker, 'bug-report-main-exit.json');
+ const detection = createCrashDetection(rig.deps);
+ detection.detectBootCrash();
+ rig.deps.logger.warn = () => {
+ throw new Error('logger stream already closed');
+ };
+
+ expect(() => detection.noteProcessExit(3)).not.toThrow();
+ });
+
+ test('parseMainExitRecord round-trips a written record', () => {
+ expect(parseMainExitRecord(`${exitRecord()}\n`)).toStrictEqual({
+ schemaVersion: 1,
+ bootId: 'boot-1',
+ exitedAt: '2026-07-10T00:00:00.000Z',
+ exitCode: 7,
+ });
+ expect(parseMainExitRecord(exitRecord({ exitCode: -1 }))?.exitCode).toBe(-1);
+ });
+
+ test('parseMainExitRecord rejects every malformed field', () => {
+ for (const bad of [
+ { schemaVersion: 2 },
+ { schemaVersion: undefined },
+ { bootId: '' },
+ { bootId: 5 },
+ { exitedAt: 'yesterday' },
+ { exitedAt: undefined },
+ { exitCode: 1.5 },
+ { exitCode: '0' },
+ { exitCode: null },
+ ]) {
+ expect(parseMainExitRecord(exitRecord(bad)), JSON.stringify(bad)).toBeNull();
+ }
+ expect(parseMainExitRecord('null')).toBeNull();
+ expect(parseMainExitRecord('[]')).toBeNull();
+ expect(parseMainExitRecord('{"schemaVersion":1')).toBeNull();
+ });
+});
diff --git a/packages/desktop/src/main/crash-detection.ts b/packages/desktop/src/main/crash-detection.ts
index 7cda10587..d52c02776 100644
--- a/packages/desktop/src/main/crash-detection.ts
+++ b/packages/desktop/src/main/crash-detection.ts
@@ -16,10 +16,14 @@ import { asReportableAppVersion } from './crashed-app-version.ts';
import type { DesktopCrashProcessSnapshot } from './desktop-process-observability.ts';
import {
classifyPreviousLiveness,
+ classifyPreviousStall,
isFileMissingError,
livenessLogFields,
type MainThreadWatchdog,
type MainThreadWatchdogHandle,
+ type PreviousFileRead,
+ readPreviousFile,
+ stallLogFields,
} from './main-thread-watchdog.ts';
import {
classifyMinidumpCrashKind,
@@ -107,8 +111,53 @@ interface SentinelState {
appVersion?: string;
}
+export interface MainExitRecord {
+ readonly schemaVersion: 1;
+ readonly bootId: string;
+ readonly exitedAt: string;
+ readonly exitCode: number;
+}
+
+type MainExitEvidence = 'matched' | 'absent' | 'unreadable' | 'no-previous-boot' | 'boot-mismatch';
+
+interface MainExitLogFields {
+ mainExit: { exitCode: number; exitedAt: string } | null;
+ mainExitEvidence: MainExitEvidence;
+}
+
+export function parseMainExitRecord(raw: string): MainExitRecord | null {
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(raw);
+ } catch {
+ return null;
+ }
+ if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) return null;
+ const p = parsed as Record<string, unknown>;
+ if (p.schemaVersion !== 1) return null;
+ if (typeof p.bootId !== 'string' || p.bootId === '') return null;
+ if (typeof p.exitedAt !== 'string' || !Number.isFinite(Date.parse(p.exitedAt))) return null;
+ if (typeof p.exitCode !== 'number' || !Number.isInteger(p.exitCode)) return null;
+ return { schemaVersion: 1, bootId: p.bootId, exitedAt: p.exitedAt, exitCode: p.exitCode };
+}
+
+function mainExitLogFields(
+ read: PreviousFileRead<MainExitRecord>,
+ prevBootId: string | null,
+): MainExitLogFields {
+ if (read.kind !== 'record') return { mainExit: null, mainExitEvidence: read.kind };
+ if (prevBootId === null) return { mainExit: null, mainExitEvidence: 'no-previous-boot' };
+ if (read.value.bootId !== prevBootId)
+ return { mainExit: null, mainExitEvidence: 'boot-mismatch' };
+ return {
+ mainExit: { exitCode: read.value.exitCode, exitedAt: read.value.exitedAt },
+ mainExitEvidence: 'matched',
+ };
+}
+
export interface CrashDetectionDeps {
sentinelPath: string;
+ mainExitPath: string;
ackStorePath: string;
crashDumpsDir: string;
appBundleRoot: string;
@@ -118,7 +167,7 @@ export interface CrashDetectionDeps {
now(): Date;
currentBootSessionUuid(): string | null;
installInFlight?(span: { deathFromMs: number; deathToMs: number }): InstallInFlight | null;
- mainThreadWatchdog: Pick<MainThreadWatchdog, 'readPrevious' | 'start'>;
+ mainThreadWatchdog: Pick<MainThreadWatchdog, 'readPrevious' | 'readPreviousStall' | 'start'>;
logger: CrashLogger;
}
@@ -131,6 +180,7 @@ export interface InstallInFlight {
export interface CrashDetection {
detectBootCrash(): OkBugReportCrashDetectedEvent | null;
markCleanQuit(): void;
+ noteProcessExit(code: number): void;
noteAlive(): void;
noteOsShutdown(reasons?: readonly string[]): void;
noteSuspend(): void;
@@ -512,13 +562,27 @@ export function createCrashDetection(deps: CrashDetectionDeps): CrashDetection {
? installInFlight.handoffAt
: null;
- const livenessFields = livenessLogFields(
- classifyPreviousLiveness(
- deps.mainThreadWatchdog.readPrevious(),
+ const liveness = classifyPreviousLiveness(
+ deps.mainThreadWatchdog.readPrevious(),
+ prevBootId,
+ epochMsOrNull(prevLastAliveAt),
+ );
+ const previousSessionFields = {
+ ...livenessLogFields(liveness),
+ ...stallLogFields(
+ classifyPreviousStall(deps.mainThreadWatchdog.readPreviousStall(), prevBootId, liveness),
+ ),
+ ...mainExitLogFields(
+ readPreviousFile(
+ deps.mainExitPath,
+ parseMainExitRecord,
+ deps.logger,
+ 'crash-detection.main-exit-read-failed',
+ 'exit record',
+ ),
prevBootId,
- epochMsOrNull(prevLastAliveAt),
),
- );
+ };
const freshDumps: ClassifiedDump[] = freshMinidumpEntries().map((entry) => {
const ownership = classifyDump(entry.path);
@@ -676,7 +740,7 @@ export function createCrashDetection(deps: CrashDetectionDeps): CrashDetection {
suspendedAt: prevSuspendedAt,
pendingOsShutdownAt: prevPendingOsShutdownAt,
osShutdownReasons: prevOsShutdownReasons,
- ...livenessFields,
+ ...previousSessionFields,
};
if (reason === 'os-shutdown') {
deps.logger.warn(
@@ -740,7 +804,7 @@ export function createCrashDetection(deps: CrashDetectionDeps): CrashDetection {
}
: {}),
detectingAppVersion: deps.appVersion,
- ...livenessFields,
+ ...previousSessionFields,
},
'previous session ended uncleanly — arming report invitation',
);
@@ -784,6 +848,27 @@ export function createCrashDetection(deps: CrashDetectionDeps): CrashDetection {
}
},
+ noteProcessExit(code: number): void {
+ if (sentinel === null || cleanQuitMarked) return;
+ const record: MainExitRecord = {
+ schemaVersion: 1,
+ bootId: sentinel.bootId,
+ exitedAt: deps.now().toISOString(),
+ exitCode: code,
+ };
+ try {
+ mkdirSync(dirname(deps.mainExitPath), { recursive: true });
+ writeFileSync(deps.mainExitPath, `${JSON.stringify(record)}\n`);
+ } catch (err) {
+ try {
+ deps.logger.warn(
+ { event: 'crash-detection.main-exit-write-failed', err, exitCode: code },
+ 'could not record how the main process exited',
+ );
+ } catch {}
+ }
+ },
+
noteAlive(): void {
if (sentinel === null || cleanQuitMarked) return;
const nowAt = deps.now();
diff --git a/packages/desktop/src/main/index.ts b/packages/desktop/src/main/index.ts
index 7dba788aa..a1b48a77f 100644
--- a/packages/desktop/src/main/index.ts
+++ b/packages/desktop/src/main/index.ts
@@ -6024,6 +6024,7 @@ function bootPrimaryInstance(): void {
});
crashDetection = createCrashDetection({
sentinelPath: join(app.getPath('userData'), 'bug-report-dirty-shutdown.json'),
+ mainExitPath: join(app.getPath('userData'), 'bug-report-main-exit.json'),
ackStorePath: join(app.getPath('userData'), 'bug-report-crash-acks.json'),
crashDumpsDir: app.getPath('crashDumps'),
appBundleRoot: appBundleRootFromExecutable(app.getPath('exe')),
@@ -6051,10 +6052,14 @@ function bootPrimaryInstance(): void {
),
mainThreadWatchdog: createMainThreadWatchdog({
path: join(app.getPath('userData'), 'bug-report-main-thread-liveness.json'),
+ stallPath: join(app.getPath('userData'), 'bug-report-main-thread-stall.json'),
logger: getLogger('main-thread-watchdog'),
}),
logger: getLogger('crash-detection'),
});
+ process.on('exit', (code) => {
+ crashDetection?.noteProcessExit(code);
+ });
crashDetection.detectBootCrash();
rendererRecovery = createRendererRecovery({
now: () => Date.now(),
diff --git a/packages/desktop/src/main/ipc/bug-report.test.ts b/packages/desktop/src/main/ipc/bug-report.test.ts
index f06875376..63670e065 100644
--- a/packages/desktop/src/main/ipc/bug-report.test.ts
+++ b/packages/desktop/src/main/ipc/bug-report.test.ts
@@ -2584,6 +2584,7 @@ describe('handleBugReportCrashAck', () => {
let clockMs = Date.parse('2026-07-10T00:00:00.000Z');
const deps = {
sentinelPath: join(dir, 'sentinel.json'),
+ mainExitPath: join(dir, 'main-exit.json'),
ackStorePath: join(dir, 'crash-acks.json'),
crashDumpsDir: join(dir, 'dumps'),
appBundleRoot: join(dir, 'OpenKnowledge.app'),
@@ -2595,6 +2596,7 @@ describe('handleBugReportCrashAck', () => {
currentBootSessionUuid: () => 'boot-epoch-test',
mainThreadWatchdog: {
readPrevious: () => ({ kind: 'absent' }) as const,
+ readPreviousStall: () => ({ kind: 'absent' }) as const,
start: () => ({ stop: () => {} }),
},
logger: { info: () => {}, warn: () => {} },
diff --git a/packages/desktop/src/main/main-thread-watchdog.test.ts b/packages/desktop/src/main/main-thread-watchdog.test.ts
index 71f3baefd..6ca6eb1a4 100644
--- a/packages/desktop/src/main/main-thread-watchdog.test.ts
+++ b/packages/desktop/src/main/main-thread-watchdog.test.ts
@@ -1,16 +1,27 @@
+import { execFileSync } from 'node:child_process';
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
-import { join, resolve } from 'node:path';
+import { dirname, join, resolve } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
+import { Worker } from 'node:worker_threads';
import { afterEach, describe, expect, test } from 'vitest';
import {
+ boundStallFrames,
classifyPreviousLiveness,
+ classifyPreviousStall,
computeWatchdogTick,
createMainThreadWatchdog,
livenessLogFields,
+ type MainThreadWatchdog,
type MainThreadWatchdogHandle,
+ type PreviousSessionLiveness,
+ parseStallSnapshot,
parseWatchdogRecord,
+ STALL_SNAPSHOT_MAX_CHARS,
+ STALL_SNAPSHOT_MAX_FRAMES,
STALL_THRESHOLD_TICKS,
+ type StallSnapshot,
+ stallLogFields,
stallThresholdMs,
WATCHDOG_TICK_MS,
type WatchdogRecord,
@@ -33,6 +44,65 @@ function makeDir(): string {
return dir;
}
+function stallSnapshot(
+ outcome: StallSnapshot['outcome'] = 'captured',
+ overrides: Record<string, unknown> = {},
+): StallSnapshot {
+ const request = {
+ schemaVersion: 1 as const,
+ bootId: 'boot-1',
+ stallStartedAt: '2026-07-09T23:59:40.000Z',
+ requestedAt: '2026-07-09T23:59:55.000Z',
+ blockedForMsAtRequest: 15_000,
+ };
+ const byOutcome: Record<StallSnapshot['outcome'], StallSnapshot> = {
+ pending: { ...request, outcome: 'pending' },
+ captured: {
+ ...request,
+ outcome: 'captured',
+ capturedAt: '2026-07-09T23:59:55.030Z',
+ pauseLatencyMs: 27,
+ frames: [
+ {
+ functionName: 'rebuildIndex',
+ url: 'file:///app/out/main/index.js',
+ line: 812,
+ column: 5,
+ },
+ ],
+ framesTruncated: false,
+ },
+ failed: { ...request, outcome: 'failed', error: 'ERR_INSPECTOR_NOT_AVAILABLE: no inspector' },
+ };
+ return { ...byOutcome[outcome], ...overrides } as StallSnapshot;
+}
+
+function blockMainThreadFor(ms: number): void {
+ const end = Date.now() + ms;
+ while (Date.now() < end) {}
+}
+
+async function pollFor<T>(read: () => T | null, timeoutMs = 10_000): Promise<T | null> {
+ const deadline = Date.now() + timeoutMs;
+ while (Date.now() < deadline) {
+ const value = read();
+ if (value !== null) return value;
+ await delay(20);
+ }
+ return read();
+}
+
+function capturedStall(watchdog: MainThreadWatchdog, after = ''): StallSnapshot | null {
+ const read = watchdog.readPreviousStall();
+ if (read.kind !== 'record' || read.value.outcome !== 'captured') return null;
+ return read.value.stallStartedAt > after ? read.value : null;
+}
+
+function witnessedPings(watchdog: MainThreadWatchdog): WatchdogRecord | null {
+ const read = watchdog.readPrevious();
+ return read.kind === 'record' && read.record.mainTicksObserved >= 1 ? read.record : null;
+}
+
function record(overrides: Partial<WatchdogRecord> = {}): WatchdogRecord {
return {
schemaVersion: 1,
@@ -332,12 +402,362 @@ describe('livenessLogFields', () => {
});
});
+describe('boundStallFrames', () => {
+ function cdpFrame(functionName: string, url: string, lineNumber = 0, columnNumber = 0) {
+ return { functionName, url, location: { scriptId: '7', lineNumber, columnNumber } };
+ }
+
+ test('keeps names and script locations with one-based lines and columns', () => {
+ const bounded = boundStallFrames(
+ [cdpFrame('rebuildIndex', 'file:///app/out/main/index.js', 811, 4)],
+ new Map(),
+ STALL_SNAPSHOT_MAX_FRAMES,
+ STALL_SNAPSHOT_MAX_CHARS,
+ );
+
+ expect(bounded).toStrictEqual({
+ frames: [
+ {
+ functionName: 'rebuildIndex',
+ url: 'file:///app/out/main/index.js',
+ line: 812,
+ column: 5,
+ },
+ ],
+ framesTruncated: false,
+ });
+ });
+
+ test('a deep stack keeps its innermost frames and says it was cut', () => {
+ const deep = Array.from({ length: STALL_SNAPSHOT_MAX_FRAMES + 5 }, (_, i) =>
+ cdpFrame(`frame${i}`, 'file:///app/x.js'),
+ );
+
+ const bounded = boundStallFrames(deep, new Map(), STALL_SNAPSHOT_MAX_FRAMES, 256);
+
+ expect(bounded.frames).toHaveLength(STALL_SNAPSHOT_MAX_FRAMES);
+ expect(bounded.frames[0]?.functionName).toBe('frame0');
+ expect(bounded.framesTruncated).toBe(true);
+ });
+
+ test('a stack exactly at the cap is not marked as cut', () => {
+ const full = Array.from({ length: STALL_SNAPSHOT_MAX_FRAMES }, () => cdpFrame('f', 'u'));
+
+ expect(boundStallFrames(full, new Map(), STALL_SNAPSHOT_MAX_FRAMES, 256).framesTruncated).toBe(
+ false,
+ );
+ });
+
+ test('a frame without a url borrows the one its script was parsed with', () => {
+ const bounded = boundStallFrames(
+ [cdpFrame('anonymous', '')],
+ new Map([['7', 'node:internal/timers']]),
+ STALL_SNAPSHOT_MAX_FRAMES,
+ 256,
+ );
+
+ expect(bounded.frames[0]?.url).toBe('node:internal/timers');
+ });
+
+ test('long strings are clipped to the cap, keeping the end of a url', () => {
+ const url = `file:///${'a'.repeat(400)}/index.js`;
+
+ const bounded = boundStallFrames(
+ [cdpFrame('f'.repeat(400), url)],
+ new Map(),
+ STALL_SNAPSHOT_MAX_FRAMES,
+ 64,
+ );
+
+ expect(bounded.frames[0]?.functionName).toHaveLength(64);
+ expect(bounded.frames[0]?.url).toHaveLength(64);
+ expect(bounded.frames[0]?.url.endsWith('/index.js')).toBe(true);
+ });
+
+ test('malformed frames degrade to empty fields instead of throwing', () => {
+ const bounded = boundStallFrames([null, 7, { location: null }], new Map(), 24, 256);
+
+ expect(bounded.frames).toStrictEqual([
+ { functionName: '', url: '', line: 0, column: 0 },
+ { functionName: '', url: '', line: 0, column: 0 },
+ { functionName: '', url: '', line: 0, column: 0 },
+ ]);
+ expect(boundStallFrames('not frames', new Map(), 24, 256)).toStrictEqual({
+ frames: [],
+ framesTruncated: false,
+ });
+ });
+});
+
+describe('parseStallSnapshot', () => {
+ test('every outcome round-trips', () => {
+ for (const outcome of ['pending', 'captured', 'failed'] as const) {
+ const written = stallSnapshot(outcome);
+
+ expect(parseStallSnapshot(`${JSON.stringify(written)}\n`)).toStrictEqual(written);
+ }
+ });
+
+ test('every truncation of a snapshot reads as unparseable without throwing', () => {
+ const raw = JSON.stringify(stallSnapshot('captured'));
+
+ for (let end = 0; end < raw.length; end += 1) {
+ expect(parseStallSnapshot(raw.slice(0, end)), `truncated at ${end}`).toBeNull();
+ }
+ });
+
+ test('a snapshot missing the request it answers is rejected', () => {
+ for (const field of [
+ 'schemaVersion',
+ 'bootId',
+ 'stallStartedAt',
+ 'requestedAt',
+ 'blockedForMsAtRequest',
+ 'outcome',
+ ]) {
+ expect(
+ parseStallSnapshot(JSON.stringify({ ...stallSnapshot('pending'), [field]: undefined })),
+ field,
+ ).toBeNull();
+ }
+ expect(
+ parseStallSnapshot(JSON.stringify(stallSnapshot('pending', { outcome: 'paused' }))),
+ ).toBeNull();
+ expect(
+ parseStallSnapshot(JSON.stringify(stallSnapshot('pending', { stallStartedAt: 'soon' }))),
+ ).toBeNull();
+ expect(
+ parseStallSnapshot(JSON.stringify(stallSnapshot('pending', { blockedForMsAtRequest: -1 }))),
+ ).toBeNull();
+ expect(
+ parseStallSnapshot(JSON.stringify(stallSnapshot('pending', { schemaVersion: 2 }))),
+ ).toBeNull();
+ });
+
+ test('a captured snapshot must carry bounded frames and its pause latency', () => {
+ for (const field of ['capturedAt', 'pauseLatencyMs', 'frames', 'framesTruncated']) {
+ expect(
+ parseStallSnapshot(JSON.stringify({ ...stallSnapshot('captured'), [field]: undefined })),
+ field,
+ ).toBeNull();
+ }
+ const tooMany = Array.from({ length: STALL_SNAPSHOT_MAX_FRAMES + 1 }, () => ({
+ functionName: 'f',
+ url: 'u',
+ line: 1,
+ column: 1,
+ }));
+ expect(
+ parseStallSnapshot(JSON.stringify(stallSnapshot('captured', { frames: tooMany }))),
+ ).toBeNull();
+ const longUrl = 'u'.repeat(STALL_SNAPSHOT_MAX_CHARS + 1);
+ expect(
+ parseStallSnapshot(
+ JSON.stringify(
+ stallSnapshot('captured', {
+ frames: [{ functionName: 'f', url: longUrl, line: 1, column: 1 }],
+ }),
+ ),
+ ),
+ ).toBeNull();
+ expect(
+ parseStallSnapshot(
+ JSON.stringify(
+ stallSnapshot('captured', {
+ frames: [{ functionName: 'f', url: 'u', line: 1.5, column: 1 }],
+ }),
+ ),
+ ),
+ ).toBeNull();
+ expect(
+ parseStallSnapshot(JSON.stringify(stallSnapshot('captured', { pauseLatencyMs: -3 }))),
+ ).toBeNull();
+ });
+
+ test('a failed snapshot must say why', () => {
+ expect(parseStallSnapshot(JSON.stringify(stallSnapshot('failed', { error: '' })))).toBeNull();
+ expect(parseStallSnapshot(JSON.stringify(stallSnapshot('failed', { error: 3 })))).toBeNull();
+ });
+
+ test('a JSON value that is not an object is rejected', () => {
+ expect(parseStallSnapshot('null')).toBeNull();
+ expect(parseStallSnapshot('[]')).toBeNull();
+ });
+});
+
+describe('classifyPreviousStall', () => {
+ const tickMs = 5_000;
+ const witnessAt = '2026-07-10T00:00:00.000Z';
+
+ function blockedFor(blockedForMs: number): PreviousSessionLiveness {
+ return {
+ kind: 'blocked',
+ blockedForMs,
+ stallThresholdMs: tickMs * STALL_THRESHOLD_TICKS,
+ writtenAt: witnessAt,
+ };
+ }
+
+ function stallStartingAt(offsetFromWitnessMs: number): StallSnapshot {
+ return stallSnapshot('captured', {
+ stallStartedAt: new Date(Date.parse(witnessAt) + offsetFromWitnessMs).toISOString(),
+ });
+ }
+
+ test('a snapshot of the stall the witness saw last is the final episode', () => {
+ const snapshot = stallStartingAt(-36_000);
+
+ expect(
+ classifyPreviousStall({ kind: 'record', value: snapshot }, 'boot-1', blockedFor(36_000)),
+ ).toStrictEqual({ kind: 'matched', snapshot, episode: 'final' });
+ });
+
+ test('the final episode tolerates one tick of clock drift and no more', () => {
+ const liveness = blockedFor(36_000);
+
+ expect(
+ classifyPreviousStall(
+ { kind: 'record', value: stallStartingAt(-36_000 + tickMs) },
+ 'boot-1',
+ liveness,
+ ),
+ ).toMatchObject({ episode: 'final' });
+ expect(
+ classifyPreviousStall(
+ { kind: 'record', value: stallStartingAt(-36_000 - tickMs - 1) },
+ 'boot-1',
+ liveness,
+ ),
+ ).toMatchObject({ episode: 'earlier' });
+ });
+
+ test('a snapshot of a stall that started before the final one is an earlier episode', () => {
+ expect(
+ classifyPreviousStall(
+ { kind: 'record', value: stallStartingAt(-600_000) },
+ 'boot-1',
+ blockedFor(36_000),
+ ),
+ ).toMatchObject({ kind: 'matched', episode: 'earlier' });
+ });
+
+ test('a session the witness did not see blocked at the end only had earlier stalls', () => {
+ const snapshot = stallStartingAt(-36_000);
+
+ expect(
+ classifyPreviousStall({ kind: 'record', value: snapshot }, 'boot-1', {
+ kind: 'died',
+ blockedForMs: 6_000,
+ stallThresholdMs: 15_000,
+ writtenAt: witnessAt,
+ }),
+ ).toMatchObject({ episode: 'earlier' });
+ expect(
+ classifyPreviousStall({ kind: 'record', value: snapshot }, 'boot-1', {
+ kind: 'no-evidence',
+ why: 'absent',
+ }),
+ ).toMatchObject({ episode: 'earlier' });
+ });
+
+ test('a missing, torn, orphaned or foreign snapshot is no evidence', () => {
+ const liveness = blockedFor(36_000);
+ const snapshot = stallStartingAt(-36_000);
+
+ expect(classifyPreviousStall({ kind: 'absent' }, 'boot-1', liveness)).toStrictEqual({
+ kind: 'no-evidence',
+ why: 'absent',
+ });
+ expect(classifyPreviousStall({ kind: 'unreadable' }, 'boot-1', liveness)).toStrictEqual({
+ kind: 'no-evidence',
+ why: 'unreadable',
+ });
+ expect(
+ classifyPreviousStall({ kind: 'record', value: snapshot }, null, liveness),
+ ).toStrictEqual({ kind: 'no-evidence', why: 'no-previous-boot' });
+ expect(
+ classifyPreviousStall({ kind: 'record', value: snapshot }, 'boot-2', liveness),
+ ).toStrictEqual({ kind: 'no-evidence', why: 'boot-mismatch' });
+ });
+});
+
+describe('stallLogFields', () => {
+ test('a captured snapshot logs its frames, latency and episode', () => {
+ const snapshot = stallSnapshot('captured');
+
+ expect(stallLogFields({ kind: 'matched', snapshot, episode: 'final' })).toStrictEqual({
+ mainThreadStall: {
+ outcome: 'captured',
+ episode: 'final',
+ stallStartedAt: snapshot.stallStartedAt,
+ requestedAt: snapshot.requestedAt,
+ blockedForMsAtRequest: 15_000,
+ capturedAt: '2026-07-09T23:59:55.030Z',
+ pauseLatencyMs: 27,
+ frames: [
+ {
+ functionName: 'rebuildIndex',
+ url: 'file:///app/out/main/index.js',
+ line: 812,
+ column: 5,
+ },
+ ],
+ framesTruncated: false,
+ error: null,
+ },
+ mainThreadStallEvidence: 'matched',
+ });
+ });
+
+ test('a pause that never landed logs as pending with its capture fields spelled out as null', () => {
+ const fields = stallLogFields({
+ kind: 'matched',
+ snapshot: stallSnapshot('pending'),
+ episode: 'final',
+ });
+
+ expect(fields.mainThreadStall).toStrictEqual({
+ outcome: 'pending',
+ episode: 'final',
+ stallStartedAt: '2026-07-09T23:59:40.000Z',
+ requestedAt: '2026-07-09T23:59:55.000Z',
+ blockedForMsAtRequest: 15_000,
+ capturedAt: null,
+ pauseLatencyMs: null,
+ frames: null,
+ framesTruncated: null,
+ error: null,
+ });
+ });
+
+ test('a failed capture logs its reason', () => {
+ const fields = stallLogFields({
+ kind: 'matched',
+ snapshot: stallSnapshot('failed'),
+ episode: 'earlier',
+ });
+
+ expect(fields.mainThreadStall?.error).toBe('ERR_INSPECTOR_NOT_AVAILABLE: no inspector');
+ expect(fields.mainThreadStall?.frames).toBeNull();
+ });
+
+ test('absent evidence spells its null out rather than omitting the key', () => {
+ for (const why of ['absent', 'unreadable', 'no-previous-boot', 'boot-mismatch'] as const) {
+ expect(stallLogFields({ kind: 'no-evidence', why })).toStrictEqual({
+ mainThreadStall: null,
+ mainThreadStallEvidence: why,
+ });
+ }
+ });
+});
+
describe('the worker witness', () => {
test('it records the running session and the pings it heard from main', async () => {
const path = join(makeDir(), 'nested', 'watchdog.json');
const warnings: Record<string, unknown>[] = [];
const watchdog = createMainThreadWatchdog({
path,
+ stallPath: join(dirname(path), 'stall.json'),
logger: {
warn: (payload) => {
warnings.push(payload);
@@ -381,6 +801,7 @@ describe('the worker witness', () => {
const warnings: Record<string, unknown>[] = [];
const watchdog = createMainThreadWatchdog({
path: join(parentAsFile, 'watchdog.json'),
+ stallPath: join(parentAsFile, 'stall.json'),
logger: {
warn: (payload) => {
warnings.push(payload);
@@ -399,8 +820,174 @@ describe('the worker witness', () => {
test('an unreadable file reads as unreadable rather than absent', () => {
const dir = makeDir();
- const watchdog = createMainThreadWatchdog({ path: dir, logger: { warn: () => {} } });
+ const watchdog = createMainThreadWatchdog({
+ path: dir,
+ stallPath: dir,
+ logger: { warn: () => {} },
+ });
expect(watchdog.readPrevious()).toEqual({ kind: 'unreadable' });
+ expect(watchdog.readPreviousStall()).toEqual({ kind: 'unreadable' });
});
+
+ test('a main thread blocked past the threshold leaves a stack naming the blocking function', async () => {
+ const dir = makeDir();
+ const warnings: Record<string, unknown>[] = [];
+ const watchdog = createMainThreadWatchdog({
+ path: join(dir, 'watchdog.json'),
+ stallPath: join(dir, 'stall.json'),
+ logger: {
+ warn: (payload) => {
+ warnings.push(payload);
+ },
+ },
+ tickMs: 20,
+ });
+ handles.push(watchdog.start('boot-stall'));
+ expect(await pollFor(() => witnessedPings(watchdog))).not.toBeNull();
+ expect(watchdog.readPreviousStall()).toEqual({ kind: 'absent' });
+
+ blockMainThreadFor(1_500);
+ const first = await pollFor(() => capturedStall(watchdog));
+
+ if (first?.outcome !== 'captured') throw new Error('no captured stall snapshot');
+ expect(first.bootId).toBe('boot-stall');
+ expect(first.frames[0]?.functionName).toBe('blockMainThreadFor');
+ expect(first.frames[0]?.url).toContain('main-thread-watchdog.test');
+ expect(first.frames[0]?.line).toBeGreaterThan(0);
+ expect(first.blockedForMsAtRequest).toBeGreaterThanOrEqual(20 * STALL_THRESHOLD_TICKS);
+ expect(first.pauseLatencyMs).toBeLessThan(1_000);
+
+ await pollFor(() => {
+ const read = watchdog.readPrevious();
+ return read.kind === 'record' && read.record.blockedForMs < 20 ? read : null;
+ });
+ blockMainThreadFor(1_500);
+ const second = await pollFor(() => capturedStall(watchdog, first.stallStartedAt));
+ await delay(100);
+
+ expect(second?.outcome).toBe('captured');
+ expect(warnings.map((w) => w.event)).toEqual([
+ 'main-thread-watchdog.stall-captured',
+ 'main-thread-watchdog.stall-captured',
+ ]);
+ expect(warnings[0]?.snapshot).toStrictEqual(first);
+ });
+
+ test('a stall snapshot from the worker that does not validate leaves a warning', async () => {
+ const dir = makeDir();
+ const warnings: Record<string, unknown>[] = [];
+ const watchdog = createMainThreadWatchdog({
+ path: join(dir, 'watchdog.json'),
+ stallPath: join(dir, 'stall.json'),
+ logger: {
+ warn: (payload) => {
+ warnings.push(payload);
+ },
+ },
+ tickMs: 20,
+ });
+ handles.push(watchdog.start(''));
+ await delay(200);
+
+ blockMainThreadFor(1_500);
+ const warned = await pollFor(() =>
+ warnings.some((w) => w.event === 'main-thread-watchdog.stall-message-invalid') ? true : null,
+ );
+
+ expect(warned).toBe(true);
+ expect(warnings.map((w) => w.event)).not.toContain('main-thread-watchdog.stall-captured');
+ expect(
+ warnings.find((w) => w.event === 'main-thread-watchdog.stall-message-invalid'),
+ ).toMatchObject({
+ payloadType: 'object',
+ keys: expect.arrayContaining(['bootId', 'outcome']),
+ outcome: 'captured',
+ });
+ });
+
+ test("a stall that is another debugger's pause is recorded as failed and never resumed", async () => {
+ const dir = makeDir();
+ const watchdog = createMainThreadWatchdog({
+ path: join(dir, 'watchdog.json'),
+ stallPath: join(dir, 'stall.json'),
+ logger: { warn: () => {} },
+ tickMs: 200,
+ });
+ handles.push(watchdog.start('boot-debugger'));
+ expect(await pollFor(() => witnessedPings(watchdog))).not.toBeNull();
+
+ let lastMainTickAt = Date.now();
+ let longestMainGapMs = 0;
+ const mainTicks = setInterval(() => {
+ const now = Date.now();
+ longestMainGapMs = Math.max(longestMainGapMs, now - lastMainTickAt);
+ lastMainTickAt = now;
+ }, 5);
+ const developerDebugger = new Worker(
+ `
+ const { parentPort } = require('node:worker_threads');
+ const inspector = require('node:inspector');
+ const keepAlive = setInterval(() => {}, 1_000);
+ const session = new inspector.Session();
+ session.connectToMainThread();
+ let held = false;
+ session.on('Debugger.paused', () => {
+ if (held) return;
+ held = true;
+ setTimeout(() => {
+ session.post('Debugger.resume', () => {
+ parentPort.postMessage('released');
+ session.post('Debugger.disable', () => {
+ session.disconnect();
+ clearInterval(keepAlive);
+ });
+ });
+ }, 2_000);
+ });
+ session.post('Debugger.enable', () => session.post('Debugger.pause'));
+ `,
+ { eval: true },
+ );
+ await new Promise((resolve) => {
+ developerDebugger.once('message', resolve);
+ });
+ clearInterval(mainTicks);
+ await developerDebugger.terminate();
+ const read = await pollFor(() => {
+ const stall = watchdog.readPreviousStall();
+ return stall.kind === 'record' && stall.value.outcome !== 'pending' ? stall.value : null;
+ });
+
+ expect(longestMainGapMs).toBeGreaterThanOrEqual(1_900);
+ expect(read).toMatchObject({
+ outcome: 'failed',
+ error: 'the main thread was already paused by another debugger session',
+ });
+ });
+
+ test.skipIf(process.platform === 'win32')(
+ 'a stall in native code is on disk as pending before the pause lands, which records how late it landed',
+ async () => {
+ const dir = makeDir();
+ const stallPath = join(dir, 'stall.json');
+ const watchdog = createMainThreadWatchdog({
+ path: join(dir, 'watchdog.json'),
+ stallPath,
+ logger: { warn: () => {} },
+ tickMs: 20,
+ });
+ handles.push(watchdog.start('boot-native'));
+ expect(await pollFor(() => witnessedPings(watchdog))).not.toBeNull();
+
+ const seenWhileBlocked = execFileSync('/bin/sh', ['-c', 'sleep 1; cat "$0"', stallPath], {
+ encoding: 'utf8',
+ });
+ const landed = await pollFor(() => capturedStall(watchdog));
+
+ expect(parseStallSnapshot(seenWhileBlocked)?.outcome).toBe('pending');
+ if (landed?.outcome !== 'captured') throw new Error('the pause never landed');
+ expect(landed.pauseLatencyMs).toBeGreaterThanOrEqual(500);
+ },
+ );
});
diff --git a/packages/desktop/src/main/main-thread-watchdog.ts b/packages/desktop/src/main/main-thread-watchdog.ts
index ff6dbc024..22183e171 100644
--- a/packages/desktop/src/main/main-thread-watchdog.ts
+++ b/packages/desktop/src/main/main-thread-watchdog.ts
@@ -5,6 +5,12 @@ export const WATCHDOG_TICK_MS = 5_000;
export const STALL_THRESHOLD_TICKS = 3;
+export const STALL_SNAPSHOT_MAX_FRAMES = 24;
+
+export const STALL_SNAPSHOT_MAX_CHARS = 256;
+
+const STALL_ERROR_MAX_CHARS = 200;
+
export interface WatchdogRecord {
readonly schemaVersion: 1;
readonly bootId: string;
@@ -54,15 +60,73 @@ export interface LivenessLogFields {
| 'unwitnessed-main';
}
+export interface StallFrame {
+ readonly functionName: string;
+ readonly url: string;
+ readonly line: number;
+ readonly column: number;
+}
+
+interface StallSnapshotRequest {
+ readonly schemaVersion: 1;
+ readonly bootId: string;
+ readonly stallStartedAt: string;
+ readonly requestedAt: string;
+ readonly blockedForMsAtRequest: number;
+}
+
+export type StallSnapshot =
+ | (StallSnapshotRequest & { readonly outcome: 'pending' })
+ | (StallSnapshotRequest & {
+ readonly outcome: 'captured';
+ readonly capturedAt: string;
+ readonly pauseLatencyMs: number;
+ readonly frames: readonly StallFrame[];
+ readonly framesTruncated: boolean;
+ })
+ | (StallSnapshotRequest & { readonly outcome: 'failed'; readonly error: string });
+
+export type StallRead = PreviousFileRead<StallSnapshot>;
+
+type StallEpisode = 'final' | 'earlier';
+
+export type PreviousSessionStall =
+ | { kind: 'matched'; snapshot: StallSnapshot; episode: StallEpisode }
+ | { kind: 'no-evidence'; why: 'absent' | 'unreadable' | 'no-previous-boot' | 'boot-mismatch' };
+
+export interface StallLogFields {
+ mainThreadStall: {
+ outcome: StallSnapshot['outcome'];
+ episode: StallEpisode;
+ stallStartedAt: string;
+ requestedAt: string;
+ blockedForMsAtRequest: number;
+ capturedAt: string | null;
+ pauseLatencyMs: number | null;
+ frames: readonly StallFrame[] | null;
+ framesTruncated: boolean | null;
+ error: string | null;
+ } | null;
+ mainThreadStallEvidence:
+ | 'matched'
+ | Extract<PreviousSessionStall, { kind: 'no-evidence' }>['why'];
+}
+
export interface MainThreadWatchdogHandle {
stop(): void;
}
export interface MainThreadWatchdog {
readPrevious(): WatchdogRead;
+ readPreviousStall(): StallRead;
start(bootId: string): MainThreadWatchdogHandle;
}
+export type PreviousFileRead<T> =
+ | { kind: 'record'; value: T }
+ | { kind: 'absent' }
+ | { kind: 'unreadable' };
+
interface WatchdogLogger {
warn(payload: Record<string, unknown>, msg: string): void;
}
@@ -91,30 +155,77 @@ export function computeWatchdogTick(
};
}
-/* STOP: computeWatchdogTick is serialized into WORKER_SOURCE through toString(), so it must stay
- closure-free and import-free. WORKER_SOURCE is an eval string rather than a file-backed worker
- entry because electron-builder.yml unpacks node-pty's worker on the recorded ground that
- worker_threads cannot load JS from inside app.asar; a file-backed entry has to prove it loads
- from a packaged build before it replaces this. */
+export function boundStallFrames(
+ callFrames: unknown,
+ urlByScriptId: ReadonlyMap<string, string>,
+ maxFrames: number,
+ maxChars: number,
+): { frames: StallFrame[]; framesTruncated: boolean } {
+ const all: unknown[] = Array.isArray(callFrames) ? callFrames : [];
+ const text = (value: unknown): string => (typeof value === 'string' ? value : '');
+ const oneBased = (value: unknown): number =>
+ typeof value === 'number' && Number.isInteger(value) && value >= 0 ? value + 1 : 0;
+ const frames: StallFrame[] = [];
+ for (const raw of all.slice(0, maxFrames)) {
+ const frame = (typeof raw === 'object' && raw !== null ? raw : {}) as Record<string, unknown>;
+ const location = (
+ typeof frame.location === 'object' && frame.location !== null ? frame.location : {}
+ ) as Record<string, unknown>;
+ const url = text(frame.url) || (urlByScriptId.get(text(location.scriptId)) ?? '');
+ const functionName = text(frame.functionName);
+ frames.push({
+ functionName: functionName.slice(0, maxChars),
+ url: url.length > maxChars ? `...${url.slice(url.length - maxChars + 3)}` : url,
+ line: oneBased(location.lineNumber),
+ column: oneBased(location.columnNumber),
+ });
+ }
+ return { frames, framesTruncated: all.length > maxFrames };
+}
+
+/* STOP: computeWatchdogTick and boundStallFrames are serialized into WORKER_SOURCE through
+ toString(), so they must stay closure-free and import-free. WORKER_SOURCE is an eval string
+ rather than a file-backed worker entry because electron-builder.yml unpacks node-pty's worker
+ on the recorded ground that worker_threads cannot load JS from inside app.asar; a file-backed
+ entry has to prove it loads from a packaged build before it replaces this. */
const WORKER_SOURCE = `
const { mkdirSync, renameSync, unlinkSync, writeFileSync } = require('node:fs');
const { dirname } = require('node:path');
const { parentPort, workerData } = require('node:worker_threads');
-const { path, bootId, tickMs } = workerData;
+const { path, stallPath, bootId, tickMs } = workerData;
+const stallThresholdMs = tickMs * ${STALL_THRESHOLD_TICKS};
const computeTick = ${computeWatchdogTick.toString()};
+const boundFrames = ${boundStallFrames.toString()};
const state = {
lastMainAt: performance.now(),
lastTickAt: Date.now(),
mainTicksObserved: 0,
};
let reportedFailure = false;
+let stallEpisodeCaptured = false;
parentPort.on('message', () => {
state.lastMainAt = performance.now();
state.mainTicksObserved += 1;
+ stallEpisodeCaptured = false;
});
+function writeJsonAtomically(target, value) {
+ const tmpPath = target + '.tmp-' + process.pid + '-' + Date.now();
+ try {
+ mkdirSync(dirname(target), { recursive: true });
+ writeFileSync(tmpPath, JSON.stringify(value) + '\\n');
+ renameSync(tmpPath, target);
+ return null;
+ } catch (err) {
+ try {
+ unlinkSync(tmpPath);
+ } catch {}
+ return errorCode(err);
+ }
+}
+
function writeWitness() {
const record = computeTick(
state,
@@ -125,18 +236,95 @@ function writeWitness() {
bootId,
new Date().toISOString(),
);
- const tmpPath = path + '.tmp-' + process.pid + '-' + Date.now();
- try {
- mkdirSync(dirname(path), { recursive: true });
- writeFileSync(tmpPath, JSON.stringify(record) + '\\n');
- renameSync(tmpPath, path);
- } catch (err) {
+ const failedCode = writeJsonAtomically(path, record);
+ if (failedCode !== null && !reportedFailure) {
+ reportedFailure = true;
+ parentPort.postMessage({ type: 'write-failed', code: failedCode });
+ }
+ if (record.blockedForMs >= stallThresholdMs && !stallEpisodeCaptured) {
+ stallEpisodeCaptured = true;
+ requestStallSnapshot(record);
+ }
+}
+
+function requestStallSnapshot(record) {
+ const request = {
+ schemaVersion: 1,
+ bootId,
+ stallStartedAt: new Date(Date.parse(record.writtenAt) - record.blockedForMs).toISOString(),
+ requestedAt: record.writtenAt,
+ blockedForMsAtRequest: record.blockedForMs,
+ };
+ writeJsonAtomically(stallPath, Object.assign({}, request, { outcome: 'pending' }));
+ const requestedPerf = performance.now();
+ let session = null;
+ let settled = false;
+ let pauseRequested = false;
+ let ownPauseResumed = false;
+ const settle = (fields) => {
+ if (settled) return;
+ settled = true;
+ const snapshot = Object.assign({}, request, fields);
+ const writeError = writeJsonAtomically(stallPath, snapshot);
+ parentPort.postMessage({ type: 'stall-snapshot', snapshot, writeError });
+ if (session === null) return;
try {
- unlinkSync(tmpPath);
+ session.post('Debugger.disable');
} catch {}
- if (reportedFailure) return;
- reportedFailure = true;
- parentPort.postMessage({ type: 'write-failed', code: errorCode(err) });
+ try {
+ session.disconnect();
+ } catch {}
+ };
+ const fail = (err) => {
+ settle({ outcome: 'failed', error: describeError(err) });
+ };
+ try {
+ const inspector = require('node:inspector');
+ session = new inspector.Session();
+ session.connectToMainThread();
+ const urlByScriptId = new Map();
+ session.on('Debugger.scriptParsed', (message) => {
+ const params = message && message.params;
+ if (params && params.url) urlByScriptId.set(params.scriptId, params.url);
+ });
+ session.on('Debugger.paused', (message) => {
+ const params = message && message.params ? message.params : {};
+ if (!pauseRequested || ownPauseResumed) {
+ fail(new Error('the main thread was already paused by another debugger session'));
+ return;
+ }
+ ownPauseResumed = true;
+ const pauseLatencyMs = Math.round(performance.now() - requestedPerf);
+ try {
+ session.post('Debugger.resume');
+ } catch {}
+ if (settled) return;
+ const bounded = boundFrames(
+ params.callFrames,
+ urlByScriptId,
+ ${STALL_SNAPSHOT_MAX_FRAMES},
+ ${STALL_SNAPSHOT_MAX_CHARS},
+ );
+ settle({
+ outcome: 'captured',
+ capturedAt: new Date().toISOString(),
+ pauseLatencyMs,
+ frames: bounded.frames,
+ framesTruncated: bounded.framesTruncated,
+ });
+ });
+ session.post('Debugger.enable', (enableErr) => {
+ if (enableErr) {
+ fail(enableErr);
+ return;
+ }
+ pauseRequested = true;
+ session.post('Debugger.pause', (pauseErr) => {
+ if (pauseErr) fail(pauseErr);
+ });
+ });
+ } catch (err) {
+ fail(err);
}
}
@@ -146,6 +334,13 @@ setInterval(writeWitness, tickMs);
function errorCode(err) {
return err && typeof err.code === 'string' ? err.code : 'unknown';
}
+
+function describeError(err) {
+ const code = err && typeof err.code === 'string' ? err.code : '';
+ const message = err && typeof err.message === 'string' ? err.message : String(err);
+ const described = (code ? code + ': ' : '') + message;
+ return (described || 'unknown').slice(0, ${STALL_ERROR_MAX_CHARS});
+}
`;
export function isFileMissingError(err: unknown): boolean {
@@ -191,6 +386,123 @@ export function parseWatchdogRecord(raw: string): WatchdogRecord | null {
};
}
+function isIsoDate(value: unknown): value is string {
+ return typeof value === 'string' && Number.isFinite(Date.parse(value));
+}
+
+function isNonNegativeInteger(value: unknown): value is number {
+ return typeof value === 'number' && Number.isInteger(value) && value >= 0;
+}
+
+function validateStallFrame(value: unknown): StallFrame | null {
+ if (typeof value !== 'object' || value === null || Array.isArray(value)) return null;
+ const f = value as Record<string, unknown>;
+ if (typeof f.functionName !== 'string' || f.functionName.length > STALL_SNAPSHOT_MAX_CHARS) {
+ return null;
+ }
+ if (typeof f.url !== 'string' || f.url.length > STALL_SNAPSHOT_MAX_CHARS) return null;
+ if (!isNonNegativeInteger(f.line) || !isNonNegativeInteger(f.column)) return null;
+ return { functionName: f.functionName, url: f.url, line: f.line, column: f.column };
+}
+
+function validateStallSnapshot(value: unknown): StallSnapshot | null {
+ if (typeof value !== 'object' || value === null || Array.isArray(value)) return null;
+ const p = value as Record<string, unknown>;
+ if (p.schemaVersion !== 1) return null;
+ if (typeof p.bootId !== 'string' || p.bootId === '') return null;
+ if (!isIsoDate(p.stallStartedAt) || !isIsoDate(p.requestedAt)) return null;
+ if (!isNonNegativeFinite(p.blockedForMsAtRequest)) return null;
+ const request: StallSnapshotRequest = {
+ schemaVersion: 1,
+ bootId: p.bootId,
+ stallStartedAt: p.stallStartedAt,
+ requestedAt: p.requestedAt,
+ blockedForMsAtRequest: p.blockedForMsAtRequest,
+ };
+ switch (p.outcome) {
+ case 'pending':
+ return { ...request, outcome: 'pending' };
+ case 'failed':
+ if (typeof p.error !== 'string' || p.error === '') return null;
+ return { ...request, outcome: 'failed', error: p.error };
+ case 'captured': {
+ if (!isIsoDate(p.capturedAt)) return null;
+ if (!isNonNegativeFinite(p.pauseLatencyMs)) return null;
+ if (typeof p.framesTruncated !== 'boolean') return null;
+ if (!Array.isArray(p.frames) || p.frames.length > STALL_SNAPSHOT_MAX_FRAMES) return null;
+ const frames: StallFrame[] = [];
+ for (const raw of p.frames) {
+ const frame = validateStallFrame(raw);
+ if (frame === null) return null;
+ frames.push(frame);
+ }
+ return {
+ ...request,
+ outcome: 'captured',
+ capturedAt: p.capturedAt,
+ pauseLatencyMs: p.pauseLatencyMs,
+ frames,
+ framesTruncated: p.framesTruncated,
+ };
+ }
+ default:
+ return null;
+ }
+}
+
+export function parseStallSnapshot(raw: string): StallSnapshot | null {
+ try {
+ return validateStallSnapshot(JSON.parse(raw));
+ } catch {
+ return null;
+ }
+}
+
+export function classifyPreviousStall(
+ read: StallRead,
+ prevBootId: string | null,
+ liveness: PreviousSessionLiveness,
+): PreviousSessionStall {
+ if (read.kind === 'absent') return { kind: 'no-evidence', why: 'absent' };
+ if (read.kind === 'unreadable') return { kind: 'no-evidence', why: 'unreadable' };
+ const snapshot = read.value;
+ if (prevBootId === null) return { kind: 'no-evidence', why: 'no-previous-boot' };
+ if (snapshot.bootId !== prevBootId) return { kind: 'no-evidence', why: 'boot-mismatch' };
+ const finalStallStartedAtMs =
+ liveness.kind === 'blocked' ? Date.parse(liveness.writtenAt) - liveness.blockedForMs : null;
+ const tickMs =
+ liveness.kind === 'blocked' ? liveness.stallThresholdMs / STALL_THRESHOLD_TICKS : 0;
+ const episode: StallEpisode =
+ finalStallStartedAtMs !== null &&
+ Math.abs(finalStallStartedAtMs - Date.parse(snapshot.stallStartedAt)) <= tickMs
+ ? 'final'
+ : 'earlier';
+ return { kind: 'matched', snapshot, episode };
+}
+
+export function stallLogFields(stall: PreviousSessionStall): StallLogFields {
+ if (stall.kind === 'no-evidence') {
+ return { mainThreadStall: null, mainThreadStallEvidence: stall.why };
+ }
+ const { snapshot } = stall;
+ const captured = snapshot.outcome === 'captured' ? snapshot : null;
+ return {
+ mainThreadStall: {
+ outcome: snapshot.outcome,
+ episode: stall.episode,
+ stallStartedAt: snapshot.stallStartedAt,
+ requestedAt: snapshot.requestedAt,
+ blockedForMsAtRequest: snapshot.blockedForMsAtRequest,
+ capturedAt: captured?.capturedAt ?? null,
+ pauseLatencyMs: captured?.pauseLatencyMs ?? null,
+ frames: captured?.frames ?? null,
+ framesTruncated: captured?.framesTruncated ?? null,
+ error: snapshot.outcome === 'failed' ? snapshot.error : null,
+ },
+ mainThreadStallEvidence: 'matched',
+ };
+}
+
export function stallThresholdMs(record: WatchdogRecord): number {
return record.tickMs * STALL_THRESHOLD_TICKS;
}
@@ -243,8 +555,68 @@ export function livenessLogFields(liveness: PreviousSessionLiveness): LivenessLo
};
}
+function logStallSnapshot(logger: WatchdogLogger, raw: unknown, writeError: unknown): void {
+ const snapshot = validateStallSnapshot(raw);
+ if (snapshot === null) {
+ const fields = typeof raw === 'object' && raw !== null ? (raw as Record<string, unknown>) : {};
+ logger.warn(
+ {
+ event: 'main-thread-watchdog.stall-message-invalid',
+ payloadType: raw === null ? 'null' : typeof raw,
+ keys: Object.keys(fields).slice(0, 16),
+ outcome: typeof fields.outcome === 'string' ? fields.outcome.slice(0, 32) : null,
+ },
+ 'the main-thread watchdog sent a stall snapshot that did not validate',
+ );
+ return;
+ }
+ if (snapshot.outcome === 'pending') return;
+ const persistFailure = typeof writeError === 'string' ? writeError : null;
+ if (snapshot.outcome === 'failed') {
+ logger.warn(
+ { event: 'main-thread-watchdog.stall-capture-failed', snapshot, persistFailure },
+ 'the main thread stalled past the threshold but its stack could not be captured',
+ );
+ return;
+ }
+ logger.warn(
+ { event: 'main-thread-watchdog.stall-captured', snapshot, persistFailure },
+ 'the main thread stalled past the threshold and recovered; captured where it was',
+ );
+}
+
+export function readPreviousFile<T>(
+ path: string,
+ parse: (raw: string) => T | null,
+ logger: WatchdogLogger,
+ event: string,
+ noun: string,
+): PreviousFileRead<T> {
+ let raw: string;
+ try {
+ raw = readFileSync(path, 'utf8');
+ } catch (err) {
+ if (isFileMissingError(err)) return { kind: 'absent' };
+ logger.warn(
+ { event, ...watchdogFailureFields(err) },
+ `the previous session ${noun} could not be read`,
+ );
+ return { kind: 'unreadable' };
+ }
+ const value = parse(raw);
+ if (value === null) {
+ logger.warn(
+ { event, bytes: raw.length },
+ `the previous session ${noun} did not parse as a record`,
+ );
+ return { kind: 'unreadable' };
+ }
+ return { kind: 'record', value };
+}
+
export function createMainThreadWatchdog(opts: {
path: string;
+ stallPath: string;
logger: WatchdogLogger;
tickMs?: number;
}): MainThreadWatchdog {
@@ -252,26 +624,24 @@ export function createMainThreadWatchdog(opts: {
return {
readPrevious(): WatchdogRead {
- let raw: string;
- try {
- raw = readFileSync(opts.path, 'utf8');
- } catch (err) {
- if (isFileMissingError(err)) return { kind: 'absent' };
- opts.logger.warn(
- { event: 'main-thread-watchdog.read-failed', ...watchdogFailureFields(err) },
- 'the previous session witness file could not be read',
- );
- return { kind: 'unreadable' };
- }
- const record = parseWatchdogRecord(raw);
- if (record === null) {
- opts.logger.warn(
- { event: 'main-thread-watchdog.read-failed', bytes: raw.length },
- 'the previous session witness file did not parse as a record',
- );
- return { kind: 'unreadable' };
- }
- return { kind: 'record', record };
+ const read = readPreviousFile(
+ opts.path,
+ parseWatchdogRecord,
+ opts.logger,
+ 'main-thread-watchdog.read-failed',
+ 'witness file',
+ );
+ return read.kind === 'record' ? { kind: 'record', record: read.value } : read;
+ },
+
+ readPreviousStall(): StallRead {
+ return readPreviousFile(
+ opts.stallPath,
+ parseStallSnapshot,
+ opts.logger,
+ 'main-thread-watchdog.stall-read-failed',
+ 'stall snapshot file',
+ );
},
start(bootId: string): MainThreadWatchdogHandle {
@@ -279,7 +649,7 @@ export function createMainThreadWatchdog(opts: {
try {
worker = new Worker(WORKER_SOURCE, {
eval: true,
- workerData: { path: opts.path, bootId, tickMs },
+ workerData: { path: opts.path, stallPath: opts.stallPath, bootId, tickMs },
});
} catch (err) {
opts.logger.warn(
@@ -315,7 +685,16 @@ export function createMainThreadWatchdog(opts: {
let warnedWriteFailed = false;
worker.on('message', (message: unknown) => {
if (typeof message !== 'object' || message === null) return;
- const { type, code } = message as { type?: unknown; code?: unknown };
+ const { type, code, snapshot, writeError } = message as {
+ type?: unknown;
+ code?: unknown;
+ snapshot?: unknown;
+ writeError?: unknown;
+ };
+ if (type === 'stall-snapshot') {
+ logStallSnapshot(opts.logger, snapshot, writeError);
+ return;
+ }
if (type !== 'write-failed' || warnedWriteFailed) return;
warnedWriteFailed = true;
opts.logger.warn(
diff --git a/packages/desktop/tests/smoke/liveness-watchdog.e2e.ts b/packages/desktop/tests/smoke/liveness-watchdog.e2e.ts
index 4edb7c417..7ee9015e4 100644
--- a/packages/desktop/tests/smoke/liveness-watchdog.e2e.ts
+++ b/packages/desktop/tests/smoke/liveness-watchdog.e2e.ts
@@ -98,18 +98,13 @@ function killProcessTree(app: ElectronApplication): Promise<void> {
return exited;
}
-function freezeMainThread(app: ElectronApplication, ms: number): { settled: () => boolean } {
- let settled = false;
- const markSettled = (): void => {
- settled = true;
- };
- void app
- .evaluate((_electron, freezeMs) => {
+async function freezeMainThreadInATimerTask(app: ElectronApplication, ms: number): Promise<void> {
+ await app.evaluate((_electron, freezeMs) => {
+ setTimeout(function freezeMainThreadForStallCapture() {
const end = Date.now() + freezeMs;
while (Date.now() < end) {}
- }, ms)
- .then(markSettled, markSettled);
- return { settled: () => settled };
+ }, 0);
+ }, ms);
}
async function relaunchAndReadBreadcrumb(
@@ -142,11 +137,10 @@ test.describe('liveness watchdog separates a frozen main thread from a dead one
captureStderrFor(app, { cleanupDirs: [tmpHome] });
await waitForWitnessPings(tmpHome, 1);
- const freeze = freezeMainThread(app, FREEZE_MS);
+ await freezeMainThreadInATimerTask(app, FREEZE_MS);
await expect
.poll(() => readWitness(tmpHome)?.blockedForMs ?? 0, { timeout: 60_000 })
.toBeGreaterThanOrEqual(STALL_THRESHOLD_MS + 5_000);
- expect(freeze.settled()).toBe(false);
await killProcessTree(app);
const breadcrumb = await relaunchAndReadBreadcrumb(tmpHome, captureStderrFor);
@@ -154,6 +148,17 @@ test.describe('liveness watchdog separates a frozen main thread from a dead one
expect(breadcrumb.dirtyShutdown).toBe(true);
expect(breadcrumb.livenessVerdict).toBe('blocked');
expect(breadcrumb.mainThreadBlockedForMs).toBeGreaterThanOrEqual(STALL_THRESHOLD_MS);
+ const stall = breadcrumb.mainThreadStall as {
+ outcome: string;
+ episode: string;
+ frames: Array<{ functionName: string }> | null;
+ } | null;
+ expect(breadcrumb.mainThreadStallEvidence).toBe('matched');
+ expect(stall?.outcome).toBe('captured');
+ expect(stall?.episode).toBe('final');
+ expect(stall?.frames?.[0]?.functionName).toBe('freezeMainThreadForStallCapture');
+ expect(breadcrumb.mainExit).toBeNull();
+ expect(breadcrumb.mainExitEvidence).toBe('absent');
});
test('a main thread killed while responsive boots as died', async ({ captureStderrFor }) => {
@@ -169,6 +174,8 @@ test.describe('liveness watchdog separates a frozen main thread from a dead one
console.log('[liveness-watchdog] died breadcrumb', JSON.stringify(breadcrumb));
expect(breadcrumb.dirtyShutdown).toBe(true);
expect(breadcrumb.livenessVerdict).toBe('died');
+ expect(breadcrumb.mainThreadStall).toBeNull();
+ expect(breadcrumb.mainExitEvidence).toBe('absent');
});
test('Utility and renderer crashes remain recoverable before an external process-tree kill (PRD-8779)', async ({