Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

feat: attribute event-loop stalls to busy process, GC or host starvation (#12852)

[._.]/ Adam Eivy102147e540dc
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 305a8547f4..9ba6a105dc 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -593,6 +593,7 @@ CoS runtime records share one serialized state queue (`withStateLock` in `server
|------|------|
| `🚦 Merge admission operation=… status=settled …` | a settled admission transaction that took 1 s or more |
| `🚦 Merge admission … status=queue-timeout\|pending\|failed …` | always |
+| `❌ PM2 read failed (home=…): pm2 RPC timed out; … wallMs=… cpuMs=… load1=… cpus=… gcMs=… invCtx=…` | the first timed-out PM2 process-list read of a failure streak |
| `🐢 CoS state write slow outcome=… totalMs=… prepareMs agentsMs serializeMs writeMs replaceMs dirSyncMs loopLagMs loopLagMaxMs sizeKiB` | any `saveState` of 1 s or more |
| `🐢 CoS state queue slow hold heldMs=… waitMs=… loopLagMs=… loopLagMaxMs=…` | any operation that held the queue for 1 s or more |
@@ -600,7 +601,18 @@ Elapsed time inside an awaited phase is asynchronous wait plus delayed schedulin
- **`queueMs` high, `queueLagMs` near 0** — the state queue was held by other operations while the loop stayed responsive. Look for `🐢 CoS state queue slow hold` lines in the same window.
- **`persistenceMs` (or `trustedStateMs`) high, its `…LagMs` near 0** — asynchronous I/O wait: storage or fsync latency, or libuv threadpool queueing behind other file work. `agentsMs` (the changed `cos_agents` rows, written in one transaction), `writeMs` (temp write + file flush), `replaceMs` (rename) and `dirSyncMs` (directory flush) locate it; `readMs`/`parseMs` split the ownership read.
-- **`…LagMs` close to the phase time, or `timerLateMs` well above 0** — the event loop could not run: blocking JavaScript, garbage collection, host CPU starvation or process suspension. These numbers do not tell those apart; capture a CPU profile ([safely](#dynamic-import-callback-missing-after-profiling)) and host load at the time.
+- **`…LagMs` close to the phase time, or `timerLateMs` well above 0** — the event loop could not run: blocking JavaScript, garbage collection, host CPU starvation or process suspension. The attribution fields below tell those apart.
+
+**Lag attribution (#12852).** When an operation is charged more than 1,000 ms of loop lag, its line ends with `cpuMs=… load1=… cpus=… gcMs=… invCtx=…` measured over the same span. The PM2 timeout line carries them over the read, with `wallMs` as its span. Compare `cpuMs` with the span's wall time (`totalMs`, `heldMs` or `wallMs`):
+
+| Reading | Bucket | Next step |
+|---------|--------|-----------|
+| `cpuMs` near the wall time, `gcMs` a small share of it | **busy process**: blocking JavaScript | take one CPU profile on the next stall through the [safe inspector path](#dynamic-import-callback-missing-after-profiling); move the hot synchronous work off the loop, or chunk it |
+| `gcMs` a large share of `cpuMs` | **garbage collection** | reduce the allocation the profile names; a large `sizeKiB` on the state write points at snapshot cloning first |
+| `cpuMs` well below the wall time, `load1` at or above `cpus`, `invCtx` high | **host starvation**: other processes held the CPUs | reduce concurrent heavy work on the host (agent validation, full test suites), not PortOS code |
+| `cpuMs` and `load1` both low | **suspension**: the process was not scheduled at all (host sleep, a debugger, a stopped process) | check host power and debugger state for that window |
+
+`cpuMs` is this process's user plus system time, so worker threads can push it above the wall time. `load1` is the host's one-minute load average and trails a sudden spike. `gcMs` counts collections by when they started, and one finishing in a span's last loop turn may be missed. A span charged 1,000 ms of lag or less prints none of these fields.
Lag resolves to about one tick: the first 100 ms of a stall and any overdue stretch of 20 ms or less are not counted. `timerLateMs` is how late the 15 s admission deadline fired; anything beyond a few milliseconds of timer jitter proves scheduling delay on its own.
diff --git a/server/lib/README.md b/server/lib/README.md
index c77c580144..baa0476d97 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -942,5 +942,5 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
- `nativeTrash.js`: `moveToNativeTrash(paths)` recycles through OS facilities PortOS owns — one hidden, deadline-bound `osascript` (`NSFileManager.trashItemAtURL`, exported `MACOS_TRASH_SCRIPT`) per file on macOS so each gets Finder Put Back; one Windows PowerShell 5.1 `-EncodedCommand` run (shell32 `SHFileOperationW` with `FOF_ALLOWUNDO`, exported `WINDOWS_TRASH_SCRIPT`, paths in `PORTOS_TRASH_PATHS`, fixed-drive pre-flight) per batch; `trashXdg` per file on Linux. Refusals reject with a coded error (`TRASH_REFUSED` + per-file `report`, `TRASH_DEADLINE`, `TRASH_UNPARSABLE`, `TRASH_UNSUPPORTED_PLATFORM`); no local-directory or permanent-delete fallback.
- `nativeTrashXdg.js`: `trashXdg(path)` moves one file into the freedesktop Trash 1.0 directory serving its volume on `node:fs` alone (original names, RFC 2396 `Path=`, `O_EXCL` info reservation, NAME_MAX-bounded names, mount identity from `/proc/self/mountinfo` with a `st_dev` fallback so a bind mount gets its own volume trash instead of an `EXDEV`, physical-parent mount discovery, an owned trash directory is tightened to 0700 and `TRASH_DIR_UNSAFE` refuses a symlinked or foreign-owned one or a volume root whose entries other users could rename, a sticky shared `.Trash` counts only when root or the caller owns it and falls back to `.Trash-<uid>` when it cannot hold the per-user directory, refuses instead of copying across devices); `_topDirTrash(topDir, uid)`, `_parseMountInfo(text)` and `_mountOf(realDir, mounts)` are test hooks.
- `nativeTrash.js`: `moveToNativeTrash(paths)` executes the pinned package native recycling helpers on macOS/Windows with hidden windows and a deadline; Linux uses the package XDG adapter. No local-directory or permanent-delete fallback.
-- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329); `systemClock` (globals resolved at call time), `__createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).
+- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs, resources, attributeAboveMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329), and a span charged more than `attributeAboveMs` of lag carries `load` (process `cpuMs`, `gcMs`, `invCtx`, host `load1`/`cpus`) to tell a busy process from a starved one (#12852); `beginLoadSample(resources)`, `formatLoadAttribution(load)` and `systemResources` (the same counters for a caller without a span); `systemClock` (globals resolved at call time), `__createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).
- `pgJson.js`: text repair for values bound to PostgreSQL, which refuses a lone UTF-16 surrogate in `jsonb` and NUL in `jsonb`/`text` where every file store accepted both. `normalizePgString(s)` → `toWellFormed()` with NUL → U+FFFD (the same lossy repair a UTF-8 file write applied). `toPgJson(value)` → `JSON.stringify` that repairs string values and object keys; **use it, never bare `JSON.stringify`, for a parameter bound to a `json`/`jsonb` column or `::jsonb` cast** (`pgJsonConventions.test.js` enforces this). `normalizePgParams(params)` is applied by `db.js` `query()` and the `withTransaction` client: repairs string and array-element params (never unescapes `\^@`) and serializes plain-object params via `toPgJson`.
diff --git a/server/lib/eventLoopLagProbe.js b/server/lib/eventLoopLagProbe.js
index dd6490dc7e..54ec64f530 100644
--- a/server/lib/eventLoopLagProbe.js
+++ b/server/lib/eventLoopLagProbe.js
@@ -12,8 +12,17 @@
* Nothing is scheduled while no span is open. Resolution is about one interval,
* and an overdue stretch no longer than `toleranceMs` is ignored as timer
* jitter. Phase names are caller-supplied constants; summaries are numbers only.
+ *
+ * A span charged more than `attributeAboveMs` of lag also carries `load`
+ * (#12852): this process's CPU time, GC time and involuntary context switches
+ * over the span, plus host load, so lag can be told apart as busy-process
+ * (`cpuMs` near the span's wall time; `gcMs` names the GC share) versus
+ * starved-process (low `cpuMs`, `load1` at or above `cpus`).
*/
+import os from 'node:os';
+import { PerformanceObserver } from 'node:perf_hooks';
+
/** Resolves the globals at call time, so fake timers in tests govern it. */
export const systemClock = {
now: () => performance.now(),
@@ -52,10 +61,96 @@ export function __createManualClock() {
export const roundMs = (ms) => Math.min(Number.MAX_SAFE_INTEGER, Math.max(0, Math.round(ms)));
/**
- * @param {{ clock?: typeof systemClock, intervalMs?: number, toleranceMs?: number }} [options]
+ * GC time by occurrence, not delivery: Node queues `gc` entries to a later
+ * loop turn (`takeRecords()` does not see them synchronously), so each entry is
+ * charged to the open windows by its `startTime`. A GC in a span's last loop
+ * turn may arrive after the span is read. The observer runs only while a
+ * window is open, and entries older than every open window are dropped.
+ */
+function createGcMeter() {
+ const windows = new Set();
+ let entries = [];
+ let observer = null;
+ const record = (list) => {
+ for (const entry of list.getEntries()) entries.push([entry.startTime, entry.duration]);
+ };
+ return {
+ begin() {
+ const gcWindow = { since: performance.now() };
+ if (windows.size === 0) {
+ observer = new PerformanceObserver(record);
+ observer.observe({ entryTypes: ['gc'] });
+ }
+ windows.add(gcWindow);
+ let open = true;
+ return {
+ ms: () => entries.reduce((ms, [start, duration]) => (start >= gcWindow.since ? ms + duration : ms), 0),
+ end() {
+ if (!open) return;
+ open = false;
+ windows.delete(gcWindow);
+ if (windows.size === 0) {
+ observer.disconnect();
+ observer = null;
+ entries = [];
+ return;
+ }
+ let oldest = Infinity;
+ for (const { since } of windows) oldest = Math.min(oldest, since);
+ entries = entries.filter(([start]) => start >= oldest);
+ },
+ };
+ },
+ };
+}
+
+/** Process and host counters for lag attribution; each is resolved at call time. */
+export const systemResources = {
+ cpuUsage: () => process.cpuUsage(),
+ loadavg: () => os.loadavg(),
+ cpus: () => os.availableParallelism(),
+ involuntaryContextSwitches: () => process.resourceUsage().involuntaryContextSwitches,
+ gc: createGcMeter(),
+};
+
+/**
+ * Start counting this process's CPU, GC and involuntary context switches.
+ * `read()` returns the deltas so far plus host load; `end()` stops the GC
+ * window and is idempotent. Numbers only.
+ */
+export function beginLoadSample(resources = systemResources) {
+ const cpuStart = resources.cpuUsage();
+ const switchesStart = resources.involuntaryContextSwitches();
+ const gc = resources.gc.begin();
+ return {
+ read() {
+ const cpu = resources.cpuUsage();
+ const [load1] = resources.loadavg();
+ return {
+ cpuMs: roundMs((cpu.user - cpuStart.user + cpu.system - cpuStart.system) / 1000),
+ load1: Math.round(load1 * 100) / 100,
+ cpus: resources.cpus(),
+ gcMs: roundMs(gc.ms()),
+ invCtx: Math.max(0, resources.involuntaryContextSwitches() - switchesStart),
+ };
+ },
+ end: () => gc.end(),
+ };
+}
+
+/** ` cpuMs=… load1=… cpus=… gcMs=… invCtx=…` for a log line, or '' without a sample. */
+export const formatLoadAttribution = (load) => (load
+ ? ` cpuMs=${load.cpuMs} load1=${load.load1} cpus=${load.cpus} gcMs=${load.gcMs} invCtx=${load.invCtx}`
+ : '');
+
+/**
+ * @param {{ clock?: typeof systemClock, intervalMs?: number, toleranceMs?: number,
+ * resources?: typeof systemResources, attributeAboveMs?: number }} [options]
* @returns {{ begin: (phase: string) => { mark: (phase: string) => void, snapshot: () => object, end: () => object } }}
*/
-export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100, toleranceMs = 20 } = {}) {
+export function createEventLoopLagProbe({
+ clock = systemClock, intervalMs = 100, toleranceMs = 20, resources = systemResources, attributeAboveMs = 1000,
+} = {}) {
const spans = new Set();
let timer = null;
let dueAt = 0;
@@ -105,13 +200,14 @@ export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100,
lagMaxMs: roundMs(span.lagMaxMs),
phases: Object.fromEntries(Object.entries(span.phases)
.map(([name, { ms, lagMs }]) => [name, { ms: roundMs(ms), lagMs: roundMs(lagMs) }])),
+ load: span.lagMs > attributeAboveMs ? span.sample.read() : null,
});
return {
begin(phase) {
const at = clock.now();
if (timer === null) schedule(at);
- const span = { startedAt: at, accountedTo: at, lagMs: 0, lagMaxMs: 0, phases: {} };
+ const span = { startedAt: at, accountedTo: at, lagMs: 0, lagMaxMs: 0, phases: {}, sample: beginLoadSample(resources) };
enter(span, phase, at);
spans.add(span);
let ended = null;
@@ -133,6 +229,7 @@ export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100,
const now = clock.now();
close(span, now);
ended = summarize(span, now);
+ span.sample.end();
spans.delete(span);
if (spans.size === 0 && timer !== null) {
clock.clearTimer(timer);
diff --git a/server/routes/cosAgentRoutes.admissionQueue.test.js b/server/routes/cosAgentRoutes.admissionQueue.test.js
index 7b6704e286..3260c59f38 100644
--- a/server/routes/cosAgentRoutes.admissionQueue.test.js
+++ b/server/routes/cosAgentRoutes.admissionQueue.test.js
@@ -139,7 +139,7 @@ describe('merge admission state queue deadline', () => {
const admission = await response;
const logs = [...console.log.mock.calls, ...console.warn.mock.calls].flat();
for (const line of logs) {
- expect(line).toMatch(/^🚦 Merge admission operation=acquire status=(pending|settled) phase=(trustedState|origin|persistence) phaseMs=\d+ totalMs=\d+ queueMs=\d+ trustedStateMs=\d+ stateLoadMs=\d+ originMs=\d+ ownerRecoveryMs=\d+ persistenceMs=\d+ readMs=\d+ parseMs=\d+ prepareMs=\d+ agentsMs=\d+ serializeMs=\d+ writeMs=\d+ replaceMs=\d+ dirSyncMs=\d+ loopLagMs=\d+ loopLagMaxMs=\d+ queueLagMs=\d+ trustedStateLagMs=\d+ persistenceLagMs=\d+ timerLateMs=\d+$/);
+ expect(line).toMatch(/^🚦 Merge admission operation=acquire status=(pending|settled) phase=(trustedState|origin|persistence) phaseMs=\d+ totalMs=\d+ queueMs=\d+ trustedStateMs=\d+ stateLoadMs=\d+ originMs=\d+ ownerRecoveryMs=\d+ persistenceMs=\d+ readMs=\d+ parseMs=\d+ prepareMs=\d+ agentsMs=\d+ serializeMs=\d+ writeMs=\d+ replaceMs=\d+ dirSyncMs=\d+ loopLagMs=\d+ loopLagMaxMs=\d+ queueLagMs=\d+ trustedStateLagMs=\d+ persistenceLagMs=\d+ timerLateMs=\d+( cpuMs=\d+ load1=\d+(\.\d+)? cpus=\d+ gcMs=\d+ invCtx=\d+)?$/);
expect(line).not.toContain(admission.token);
expect(line).not.toContain(agentId);
expect(line).not.toContain('/repos');
diff --git a/server/services/cosMergeAdmission.diagnostics.test.js b/server/services/cosMergeAdmission.diagnostics.test.js
index f2cde89f28..26e2d23761 100644
--- a/server/services/cosMergeAdmission.diagnostics.test.js
+++ b/server/services/cosMergeAdmission.diagnostics.test.js
@@ -30,7 +30,7 @@ const deferred = () => {
const agentId = 'synthetic-parent';
const acquire = () => claimMergeAdmission({ agentId, action: 'acquire' });
const origin = { host: 'github.com', fullName: 'example/repo' };
-const LINE = /^🚦 Merge admission operation=[a-z-]+ status=[a-z-]+ phase=[A-Za-z]+( [A-Za-z]+Ms=\d+)+$/;
+const LINE = /^🚦 Merge admission operation=[a-z-]+ status=[a-z-]+ phase=[A-Za-z]+( [A-Za-z]+Ms=\d+)+( cpuMs=\d+ load1=\d+(\.\d+)? cpus=\d+ gcMs=\d+ invCtx=\d+)?$/;
const reported = () => [...console.log.mock.calls, ...console.warn.mock.calls, ...console.error.mock.calls].flat();
const fields = (line) => Object.fromEntries([...line.matchAll(/ ([A-Za-z]+)=(\S+)/g)]
.map(([, key, value]) => [key, /^\d+$/.test(value) ? Number(value) : value]));
@@ -81,6 +81,7 @@ describe('merge admission slow-operation diagnostics', () => {
expect((await pending).admitted).toBe(true);
expect(onlyLine()).toMatchObject({ operation: 'acquire', status: 'settled', totalMs: 5000,
queueMs: 5000, queueLagMs: 0, loopLagMs: 0, persistenceMs: 0, timerLateMs: 0 });
+ expect(reported()[0]).not.toContain('cpuMs=');
});
it('attributes slow asynchronous persistence I/O to its stage without loop lag', async () => {
@@ -112,7 +113,7 @@ describe('merge admission slow-operation diagnostics', () => {
// The first tick interval of a stall is indistinguishable from waiting for
// that tick, so lag is measured to within one 100 ms interval.
expect(onlyLine()).toMatchObject({ totalMs: 6000, persistenceMs: 6000, serializeMs: 6000,
- persistenceLagMs: 5900, loopLagMs: 5900, loopLagMaxMs: 5900 });
+ persistenceLagMs: 5900, loopLagMs: 5900, loopLagMaxMs: 5900, cpuMs: expect.any(Number), invCtx: expect.any(Number) });
});
it('identifies a deadline delivered late by a blocked loop, and the expired sweep never mutates', async () => {
diff --git a/server/services/cosMergeAdmission.js b/server/services/cosMergeAdmission.js
index 9e2a1e4e71..f90663509d 100644
--- a/server/services/cosMergeAdmission.js
+++ b/server/services/cosMergeAdmission.js
@@ -11,7 +11,7 @@ import { parsePullRequestUrl } from '../lib/gitForge.js';
import { isTruthyMeta } from '../lib/metadataFlags.js';
import { PR_COMPLETIONS, leavesPrForHuman, resolvePrCompletion } from '../lib/prDisposition.js';
import { repositoryCleanupMergeTarget } from '../lib/recoveryMergeTarget.js';
-import { createEventLoopLagProbe, phaseGroup, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';
+import { createEventLoopLagProbe, formatLoadAttribution, phaseGroup, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';
const retryAfterMs = 5000;
export const ADMISSION_QUEUE_TIMEOUT_MS = 15_000;
@@ -49,10 +49,10 @@ export const SLOW_ADMISSION_MS = 1000;
let clock = systemClock;
let lagProbe = createEventLoopLagProbe({ clock });
-/** Inject a deterministic clock; `null` restores the system clock. */
-export function __setMergeAdmissionClockForTests(next) {
+/** Inject a deterministic clock and optional resource counters; `null` restores the system ones. */
+export function __setMergeAdmissionClockForTests(next, resources) {
clock = next ?? systemClock;
- lagProbe = createEventLoopLagProbe({ clock });
+ lagProbe = createEventLoopLagProbe({ clock, resources });
}
const REPORTED_PHASES = ['queue', 'trustedState', 'stateLoad', 'origin', 'ownerRecovery', 'persistence'];
@@ -86,7 +86,7 @@ async function transaction(work, operation) {
const phases = REPORTED_PHASES.map((name) => `${name}Ms=${phaseGroup(summary, name).ms}`);
const stages = REPORTED_STAGES.map(([name, stage]) => `${stage}Ms=${summary.phases[`${name}.${stage}`]?.ms ?? 0}`);
const lag = REPORTED_LAG_PHASES.map((name) => `${name}LagMs=${phaseGroup(summary, name).lagMs}`);
- log(`🚦 Merge admission operation=${operation} status=${status} phase=${summary.phase.split('.')[0]} phaseMs=${summary.phaseMs} totalMs=${summary.totalMs} ${phases.join(' ')} ${stages.join(' ')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} ${lag.join(' ')} timerLateMs=${timerLateMs}`);
+ log(`🚦 Merge admission operation=${operation} status=${status} phase=${summary.phase.split('.')[0]} phaseMs=${summary.phaseMs} totalMs=${summary.totalMs} ${phases.join(' ')} ${stages.join(' ')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} ${lag.join(' ')} timerLateMs=${timerLateMs}${formatLoadAttribution(summary.load)}`);
};
const measure = async (name, fn) => {
span.mark(name);
diff --git a/server/services/cosState.js b/server/services/cosState.js
index 9df913c7a9..6680402580 100644
--- a/server/services/cosState.js
+++ b/server/services/cosState.js
@@ -9,7 +9,7 @@ import { readFile, readdir, rm, rename, stat } from 'fs/promises';
import { existsSync } from 'fs';
import { join, dirname, basename } from 'path';
import { createFileWriteQueue } from '../lib/fileWriteQueue.js';
-import { createEventLoopLagProbe, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';
+import { createEventLoopLagProbe, formatLoadAttribution, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';
import { ensureDirs, safeJSONParse, readJSONFileStrict, PATHS, atomicWrite } from '../lib/fileUtils.js';
import { isPlainObject } from '../lib/objects.js';
import { normalizeDomainAutonomy, getDomainMode } from '../lib/domainAutonomy.js';
@@ -65,10 +65,10 @@ export const SLOW_STATE_OPERATION_MS = 1000;
let stateClock = systemClock;
let stateLagProbe = createEventLoopLagProbe({ clock: stateClock });
-/** Inject a deterministic clock; `null` restores the system clock. */
-export function __setCosStateDiagnosticsClockForTests(clock) {
+/** Inject a deterministic clock and optional resource counters; `null` restores the system ones. */
+export function __setCosStateDiagnosticsClockForTests(clock, resources) {
stateClock = clock ?? systemClock;
- stateLagProbe = createEventLoopLagProbe({ clock: stateClock });
+ stateLagProbe = createEventLoopLagProbe({ clock: stateClock, resources });
}
export function withStateLock(fn) {
@@ -79,9 +79,9 @@ export function withStateLock(fn) {
try {
return await fn();
} finally {
- const { totalMs, lagMs, lagMaxMs } = span.end();
+ const { totalMs, lagMs, lagMaxMs, load } = span.end();
if (totalMs >= SLOW_STATE_OPERATION_MS) {
- console.log(`🐢 CoS state queue slow hold heldMs=${totalMs} waitMs=${waitMs} loopLagMs=${lagMs} loopLagMaxMs=${lagMaxMs}`);
+ console.log(`🐢 CoS state queue slow hold heldMs=${totalMs} waitMs=${waitMs} loopLagMs=${lagMs} loopLagMaxMs=${lagMaxMs}${formatLoadAttribution(load)}`);
}
}
});
@@ -726,7 +726,7 @@ export async function saveState(state, { onStage, changedAgentIds } = {}) {
const summary = span.end();
if (summary.totalMs >= SLOW_STATE_OPERATION_MS) {
const ms = (name) => summary.phases[name]?.ms ?? 0;
- console.log(`🐢 CoS state write slow outcome=${outcome} totalMs=${summary.totalMs} prepareMs=${ms('prepare')} agentsMs=${ms('agents')} serializeMs=${ms('serialize')} writeMs=${ms('write')} replaceMs=${ms('replace')} dirSyncMs=${ms('dirSync')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} sizeKiB=${Math.round(bytes / 1024)}`);
+ console.log(`🐢 CoS state write slow outcome=${outcome} totalMs=${summary.totalMs} prepareMs=${ms('prepare')} agentsMs=${ms('agents')} serializeMs=${ms('serialize')} writeMs=${ms('write')} replaceMs=${ms('replace')} dirSyncMs=${ms('dirSync')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} sizeKiB=${Math.round(bytes / 1024)}${formatLoadAttribution(summary.load)}`);
}
}
stateCache = stampGeneration(state, stateRecovery);
diff --git a/server/services/cosState.test.js b/server/services/cosState.test.js
index f74bc40095..2f6560784f 100644
--- a/server/services/cosState.test.js
+++ b/server/services/cosState.test.js
@@ -485,10 +485,19 @@ describe('live CoS restore boundary', () => {
// #12329: a slow state operation must separate durable-write stages and
// event-loop lag, logging only bounded numbers.
describe('state operation slow diagnostics', () => {
- it('reports each persistence stage and splits a slow write into I/O wait and loop lag', async () => {
+ it('reports each persistence stage, splits a slow write into I/O wait and loop lag, and attributes only lag over 1 s', async () => {
const store = await freshModule();
const clock = __createManualClock();
- store.__setCosStateDiagnosticsClockForTests(clock);
+ // Process/host counters a test moves by hand alongside the clock (#12852).
+ const counters = { cpuUs: 0, gcMs: 0, switches: 0 };
+ const resources = {
+ cpuUsage: () => ({ user: counters.cpuUs, system: 0 }),
+ loadavg: () => [7.5, 6, 5],
+ cpus: () => 8,
+ involuntaryContextSwitches: () => counters.switches,
+ gc: { begin: () => { const since = counters.gcMs; return { ms: () => counters.gcMs - since, end: () => {} }; } },
+ };
+ store.__setCosStateDiagnosticsClockForTests(clock, resources);
const warn = vi.spyOn(console, 'log').mockImplementation(() => {});
try {
const stages = [];
@@ -505,8 +514,13 @@ describe('state operation slow diagnostics', () => {
warn.mockClear();
await store.withStateLock(async () => clock.advance(200));
expect(warn).not.toHaveBeenCalled();
- await store.withStateLock(async () => clock.stall(1200));
- expect(warn.mock.calls.map(([line]) => line)).toEqual(['🐢 CoS state queue slow hold heldMs=1200 waitMs=0 loopLagMs=1100 loopLagMaxMs=1100']);
+ // Over 1 s of lag, the line attributes it: here the process itself was
+ // busy (cpuMs near heldMs), a quarter of it in GC.
+ await store.withStateLock(async () => {
+ clock.stall(1200);
+ Object.assign(counters, { cpuUs: counters.cpuUs + 1_150_000, gcMs: counters.gcMs + 300, switches: counters.switches + 42 });
+ });
+ expect(warn.mock.calls.map(([line]) => line)).toEqual(['🐢 CoS state queue slow hold heldMs=1200 waitMs=0 loopLagMs=1100 loopLagMaxMs=1100 cpuMs=1150 load1=7.5 cpus=8 gcMs=300 invCtx=42']);
} finally {
store.__setCosStateDiagnosticsClockForTests(null);
warn.mockRestore();
diff --git a/server/services/pm2.js b/server/services/pm2.js
index 45ec495d58..cdca02763c 100644
--- a/server/services/pm2.js
+++ b/server/services/pm2.js
@@ -9,6 +9,7 @@ import { homedir } from 'os';
import { atomicWrite, safeJSONParse, tryReadFile } from '../lib/fileUtils.js';
import { parseCommandArgs } from '../lib/commandSecurity.js';
import { parsePm2JlistStdout } from '../lib/pm2Jlist.js';
+import { beginLoadSample, formatLoadAttribution, roundMs } from '../lib/eventLoopLagProbe.js';
import { PATHS } from '../lib/paths.js';
import { serverSecretEnvKeys, withoutServerSecretsEnv } from '../lib/processEnv.js';
// The leaf, not readinessNotify.js: its observer's lazy imports would join
@@ -47,16 +48,17 @@ export function clearJlistCache(pm2Home = null) {
* Record a PM2 read failure and log it only on state change or reason change.
* @param {string|null} pm2Home The PM2 home being read
* @param {string} errorReason Formatted error description
+ * @param {string} [attribution] Numbers appended to the line, never part of the dedupe key
* @private
*/
-function logJlistFailure(pm2Home, errorReason) {
+function logJlistFailure(pm2Home, errorReason, attribution = '') {
const key = cacheKey(pm2Home);
const state = jlistFailureState.get(key);
const homeDesc = pm2Home ? 'custom' : 'default';
// Log only if no prior state, or if the reason changed
if (!state || state.lastError !== errorReason) {
- console.error(`❌ PM2 read failed (home=${homeDesc}): ${errorReason}`);
+ console.error(`❌ PM2 read failed (home=${homeDesc}): ${errorReason}${attribution}`);
jlistFailureState.set(key, { lastError: errorReason });
}
}
@@ -501,6 +503,10 @@ function fetchJlist(pm2Home = null) {
const inflight = jlistInflight.get(key);
if (inflight) return inflight;
+ // A timed-out read is the usual first symptom of a stalled loop, so it carries
+ // its own busy-versus-starved attribution (#12852).
+ const startedAt = performance.now();
+ const sample = beginLoadSample();
// Defer the read until the single-flight entry exists, even for a sync callback.
const promise = Promise.resolve().then(() => pm2Home
? execPm2(['jlist'], { env: buildEnv(pm2Home), timeoutMs: JLIST_TIMEOUT_MS })
@@ -514,9 +520,13 @@ function fetchJlist(pm2Home = null) {
jlistCache.set(key, { data: list, ts: Date.now() });
return list;
}).catch(error => {
- logJlistFailure(pm2Home, error.message || String(error));
+ const attribution = error?.timedOut
+ ? ` wallMs=${roundMs(performance.now() - startedAt)}${formatLoadAttribution(sample.read())}`
+ : '';
+ logJlistFailure(pm2Home, error.message || String(error), attribution);
return null;
}).finally(() => {
+ sample.end();
if (jlistInflight.get(key) === promise) jlistInflight.delete(key);
});
diff --git a/server/services/pm2.readFailureLog.test.js b/server/services/pm2.readFailureLog.test.js
index 174e5efd16..3bba338bcd 100644
--- a/server/services/pm2.readFailureLog.test.js
+++ b/server/services/pm2.readFailureLog.test.js
@@ -5,6 +5,7 @@ vi.mock('pm2', () => ({ default: { custom: function () { return mocks; } } }));
import { clearJlistCache, listProcessesStrict } from './pm2.js';
afterEach(() => {
+ vi.useRealTimers();
vi.clearAllMocks();
vi.restoreAllMocks();
clearJlistCache();
@@ -20,7 +21,7 @@ describe('PM2 read failure logging (#8431)', () => {
mocks.connect.mockImplementation(later((cb) => cb(new Error('connect ECONNREFUSED'))));
for (let i = 0; i < 3; i++) expect(await listProcessesStrict()).toBeNull();
- expect(readLines(errorSpy, 'PM2 read failed')).toHaveLength(1);
+ expect(readLines(errorSpy, 'PM2 read failed')).toEqual(['❌ PM2 read failed (home=default): connect ECONNREFUSED']);
mocks.connect.mockImplementation(later((cb) => cb(new Error('daemon timeout'))));
await listProcessesStrict();
@@ -33,4 +34,18 @@ describe('PM2 read failure logging (#8431)', () => {
await listProcessesStrict();
expect(readLines(logSpy, 'PM2 read recovered')).toHaveLength(1);
});
+
+ // #12852: the most common stall symptom carries its own busy-versus-starved attribution.
+ it('attributes an RPC timeout to process CPU, GC, host load and context switches', async () => {
+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'performance'] });
+ const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
+ mocks.connect.mockImplementation(() => {});
+
+ const read = listProcessesStrict();
+ await vi.advanceTimersByTimeAsync(15_000);
+ expect(await read).toBeNull();
+ expect(readLines(errorSpy, 'PM2 read failed')).toEqual([expect.stringMatching(
+ /^❌ PM2 read failed \(home=default\): pm2 RPC timed out; client disconnect requested, daemon outcome unknown wallMs=15000 cpuMs=\d+ load1=\d+(\.\d+)? cpus=\d+ gcMs=\d+ invCtx=\d+$/,
+ )]);
+ });
});