Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
feat: attribute event-loop stalls to busy process, GC or host starvation (#12852)
[._.]/ Adam Eivy102147e540dc
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 305a8547f4..9ba6a105dc 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -593,6 +593,7 @@ CoS runtime records share one serialized state queue (`withStateLock` in `server|------|------|| `🚦 Merge admission operation=… status=settled …` | a settled admission transaction that took 1 s or more || `🚦 Merge admission … status=queue-timeout\|pending\|failed …` | always |+| `❌ PM2 read failed (home=…): pm2 RPC timed out; … wallMs=… cpuMs=… load1=… cpus=… gcMs=… invCtx=…` | the first timed-out PM2 process-list read of a failure streak || `🐢 CoS state write slow outcome=… totalMs=… prepareMs agentsMs serializeMs writeMs replaceMs dirSyncMs loopLagMs loopLagMaxMs sizeKiB` | any `saveState` of 1 s or more || `🐢 CoS state queue slow hold heldMs=… waitMs=… loopLagMs=… loopLagMaxMs=…` | any operation that held the queue for 1 s or more |@@ -600,7 +601,18 @@ Elapsed time inside an awaited phase is asynchronous wait plus delayed schedulin- **`queueMs` high, `queueLagMs` near 0** — the state queue was held by other operations while the loop stayed responsive. Look for `🐢 CoS state queue slow hold` lines in the same window.- **`persistenceMs` (or `trustedStateMs`) high, its `…LagMs` near 0** — asynchronous I/O wait: storage or fsync latency, or libuv threadpool queueing behind other file work. `agentsMs` (the changed `cos_agents` rows, written in one transaction), `writeMs` (temp write + file flush), `replaceMs` (rename) and `dirSyncMs` (directory flush) locate it; `readMs`/`parseMs` split the ownership read.-- **`…LagMs` close to the phase time, or `timerLateMs` well above 0** — the event loop could not run: blocking JavaScript, garbage collection, host CPU starvation or process suspension. These numbers do not tell those apart; capture a CPU profile ([safely](#dynamic-import-callback-missing-after-profiling)) and host load at the time.+- **`…LagMs` close to the phase time, or `timerLateMs` well above 0** — the event loop could not run: blocking JavaScript, garbage collection, host CPU starvation or process suspension. The attribution fields below tell those apart.++**Lag attribution (#12852).** When an operation is charged more than 1,000 ms of loop lag, its line ends with `cpuMs=… load1=… cpus=… gcMs=… invCtx=…` measured over the same span. The PM2 timeout line carries them over the read, with `wallMs` as its span. Compare `cpuMs` with the span's wall time (`totalMs`, `heldMs` or `wallMs`):++| Reading | Bucket | Next step |+|---------|--------|-----------|+| `cpuMs` near the wall time, `gcMs` a small share of it | **busy process**: blocking JavaScript | take one CPU profile on the next stall through the [safe inspector path](#dynamic-import-callback-missing-after-profiling); move the hot synchronous work off the loop, or chunk it |+| `gcMs` a large share of `cpuMs` | **garbage collection** | reduce the allocation the profile names; a large `sizeKiB` on the state write points at snapshot cloning first |+| `cpuMs` well below the wall time, `load1` at or above `cpus`, `invCtx` high | **host starvation**: other processes held the CPUs | reduce concurrent heavy work on the host (agent validation, full test suites), not PortOS code |+| `cpuMs` and `load1` both low | **suspension**: the process was not scheduled at all (host sleep, a debugger, a stopped process) | check host power and debugger state for that window |++`cpuMs` is this process's user plus system time, so worker threads can push it above the wall time. `load1` is the host's one-minute load average and trails a sudden spike. `gcMs` counts collections by when they started, and one finishing in a span's last loop turn may be missed. A span charged 1,000 ms of lag or less prints none of these fields.Lag resolves to about one tick: the first 100 ms of a stall and any overdue stretch of 20 ms or less are not counted. `timerLateMs` is how late the 15 s admission deadline fired; anything beyond a few milliseconds of timer jitter proves scheduling delay on its own.diff --git a/server/lib/README.md b/server/lib/README.mdindex c77c580144..baa0476d97 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -942,5 +942,5 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub- `nativeTrash.js`: `moveToNativeTrash(paths)` recycles through OS facilities PortOS owns — one hidden, deadline-bound `osascript` (`NSFileManager.trashItemAtURL`, exported `MACOS_TRASH_SCRIPT`) per file on macOS so each gets Finder Put Back; one Windows PowerShell 5.1 `-EncodedCommand` run (shell32 `SHFileOperationW` with `FOF_ALLOWUNDO`, exported `WINDOWS_TRASH_SCRIPT`, paths in `PORTOS_TRASH_PATHS`, fixed-drive pre-flight) per batch; `trashXdg` per file on Linux. Refusals reject with a coded error (`TRASH_REFUSED` + per-file `report`, `TRASH_DEADLINE`, `TRASH_UNPARSABLE`, `TRASH_UNSUPPORTED_PLATFORM`); no local-directory or permanent-delete fallback.- `nativeTrashXdg.js`: `trashXdg(path)` moves one file into the freedesktop Trash 1.0 directory serving its volume on `node:fs` alone (original names, RFC 2396 `Path=`, `O_EXCL` info reservation, NAME_MAX-bounded names, mount identity from `/proc/self/mountinfo` with a `st_dev` fallback so a bind mount gets its own volume trash instead of an `EXDEV`, physical-parent mount discovery, an owned trash directory is tightened to 0700 and `TRASH_DIR_UNSAFE` refuses a symlinked or foreign-owned one or a volume root whose entries other users could rename, a sticky shared `.Trash` counts only when root or the caller owns it and falls back to `.Trash-<uid>` when it cannot hold the per-user directory, refuses instead of copying across devices); `_topDirTrash(topDir, uid)`, `_parseMountInfo(text)` and `_mountOf(realDir, mounts)` are test hooks.- `nativeTrash.js`: `moveToNativeTrash(paths)` executes the pinned package native recycling helpers on macOS/Windows with hidden windows and a deadline; Linux uses the package XDG adapter. No local-directory or permanent-delete fallback.-- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329); `systemClock` (globals resolved at call time), `__createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).+- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs, resources, attributeAboveMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329), and a span charged more than `attributeAboveMs` of lag carries `load` (process `cpuMs`, `gcMs`, `invCtx`, host `load1`/`cpus`) to tell a busy process from a starved one (#12852); `beginLoadSample(resources)`, `formatLoadAttribution(load)` and `systemResources` (the same counters for a caller without a span); `systemClock` (globals resolved at call time), `__createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).- `pgJson.js`: text repair for values bound to PostgreSQL, which refuses a lone UTF-16 surrogate in `jsonb` and NUL in `jsonb`/`text` where every file store accepted both. `normalizePgString(s)` → `toWellFormed()` with NUL → U+FFFD (the same lossy repair a UTF-8 file write applied). `toPgJson(value)` → `JSON.stringify` that repairs string values and object keys; **use it, never bare `JSON.stringify`, for a parameter bound to a `json`/`jsonb` column or `::jsonb` cast** (`pgJsonConventions.test.js` enforces this). `normalizePgParams(params)` is applied by `db.js` `query()` and the `withTransaction` client: repairs string and array-element params (never unescapes `\^@`) and serializes plain-object params via `toPgJson`.diff --git a/server/lib/eventLoopLagProbe.js b/server/lib/eventLoopLagProbe.jsindex dd6490dc7e..54ec64f530 100644--- a/server/lib/eventLoopLagProbe.js+++ b/server/lib/eventLoopLagProbe.js@@ -12,8 +12,17 @@* Nothing is scheduled while no span is open. Resolution is about one interval,* and an overdue stretch no longer than `toleranceMs` is ignored as timer* jitter. Phase names are caller-supplied constants; summaries are numbers only.+ *+ * A span charged more than `attributeAboveMs` of lag also carries `load`+ * (#12852): this process's CPU time, GC time and involuntary context switches+ * over the span, plus host load, so lag can be told apart as busy-process+ * (`cpuMs` near the span's wall time; `gcMs` names the GC share) versus+ * starved-process (low `cpuMs`, `load1` at or above `cpus`).*/+import os from 'node:os';+import { PerformanceObserver } from 'node:perf_hooks';+/** Resolves the globals at call time, so fake timers in tests govern it. */export const systemClock = {now: () => performance.now(),@@ -52,10 +61,96 @@ export function __createManualClock() {export const roundMs = (ms) => Math.min(Number.MAX_SAFE_INTEGER, Math.max(0, Math.round(ms)));/**- * @param {{ clock?: typeof systemClock, intervalMs?: number, toleranceMs?: number }} [options]+ * GC time by occurrence, not delivery: Node queues `gc` entries to a later+ * loop turn (`takeRecords()` does not see them synchronously), so each entry is+ * charged to the open windows by its `startTime`. A GC in a span's last loop+ * turn may arrive after the span is read. The observer runs only while a+ * window is open, and entries older than every open window are dropped.+ */+function createGcMeter() {+ const windows = new Set();+ let entries = [];+ let observer = null;+ const record = (list) => {+ for (const entry of list.getEntries()) entries.push([entry.startTime, entry.duration]);+ };+ return {+ begin() {+ const gcWindow = { since: performance.now() };+ if (windows.size === 0) {+ observer = new PerformanceObserver(record);+ observer.observe({ entryTypes: ['gc'] });+ }+ windows.add(gcWindow);+ let open = true;+ return {+ ms: () => entries.reduce((ms, [start, duration]) => (start >= gcWindow.since ? ms + duration : ms), 0),+ end() {+ if (!open) return;+ open = false;+ windows.delete(gcWindow);+ if (windows.size === 0) {+ observer.disconnect();+ observer = null;+ entries = [];+ return;+ }+ let oldest = Infinity;+ for (const { since } of windows) oldest = Math.min(oldest, since);+ entries = entries.filter(([start]) => start >= oldest);+ },+ };+ },+ };+}++/** Process and host counters for lag attribution; each is resolved at call time. */+export const systemResources = {+ cpuUsage: () => process.cpuUsage(),+ loadavg: () => os.loadavg(),+ cpus: () => os.availableParallelism(),+ involuntaryContextSwitches: () => process.resourceUsage().involuntaryContextSwitches,+ gc: createGcMeter(),+};++/**+ * Start counting this process's CPU, GC and involuntary context switches.+ * `read()` returns the deltas so far plus host load; `end()` stops the GC+ * window and is idempotent. Numbers only.+ */+export function beginLoadSample(resources = systemResources) {+ const cpuStart = resources.cpuUsage();+ const switchesStart = resources.involuntaryContextSwitches();+ const gc = resources.gc.begin();+ return {+ read() {+ const cpu = resources.cpuUsage();+ const [load1] = resources.loadavg();+ return {+ cpuMs: roundMs((cpu.user - cpuStart.user + cpu.system - cpuStart.system) / 1000),+ load1: Math.round(load1 * 100) / 100,+ cpus: resources.cpus(),+ gcMs: roundMs(gc.ms()),+ invCtx: Math.max(0, resources.involuntaryContextSwitches() - switchesStart),+ };+ },+ end: () => gc.end(),+ };+}++/** ` cpuMs=… load1=… cpus=… gcMs=… invCtx=…` for a log line, or '' without a sample. */+export const formatLoadAttribution = (load) => (load+ ? ` cpuMs=${load.cpuMs} load1=${load.load1} cpus=${load.cpus} gcMs=${load.gcMs} invCtx=${load.invCtx}`+ : '');++/**+ * @param {{ clock?: typeof systemClock, intervalMs?: number, toleranceMs?: number,+ * resources?: typeof systemResources, attributeAboveMs?: number }} [options]* @returns {{ begin: (phase: string) => { mark: (phase: string) => void, snapshot: () => object, end: () => object } }}*/-export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100, toleranceMs = 20 } = {}) {+export function createEventLoopLagProbe({+ clock = systemClock, intervalMs = 100, toleranceMs = 20, resources = systemResources, attributeAboveMs = 1000,+} = {}) {const spans = new Set();let timer = null;let dueAt = 0;@@ -105,13 +200,14 @@ export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100,lagMaxMs: roundMs(span.lagMaxMs),phases: Object.fromEntries(Object.entries(span.phases).map(([name, { ms, lagMs }]) => [name, { ms: roundMs(ms), lagMs: roundMs(lagMs) }])),+ load: span.lagMs > attributeAboveMs ? span.sample.read() : null,});return {begin(phase) {const at = clock.now();if (timer === null) schedule(at);- const span = { startedAt: at, accountedTo: at, lagMs: 0, lagMaxMs: 0, phases: {} };+ const span = { startedAt: at, accountedTo: at, lagMs: 0, lagMaxMs: 0, phases: {}, sample: beginLoadSample(resources) };enter(span, phase, at);spans.add(span);let ended = null;@@ -133,6 +229,7 @@ export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100,const now = clock.now();close(span, now);ended = summarize(span, now);+ span.sample.end();spans.delete(span);if (spans.size === 0 && timer !== null) {clock.clearTimer(timer);diff --git a/server/routes/cosAgentRoutes.admissionQueue.test.js b/server/routes/cosAgentRoutes.admissionQueue.test.jsindex 7b6704e286..3260c59f38 100644--- a/server/routes/cosAgentRoutes.admissionQueue.test.js+++ b/server/routes/cosAgentRoutes.admissionQueue.test.js@@ -139,7 +139,7 @@ describe('merge admission state queue deadline', () => {const admission = await response;const logs = [...console.log.mock.calls, ...console.warn.mock.calls].flat();for (const line of logs) {- expect(line).toMatch(/^🚦 Merge admission operation=acquire status=(pending|settled) phase=(trustedState|origin|persistence) phaseMs=\d+ totalMs=\d+ queueMs=\d+ trustedStateMs=\d+ stateLoadMs=\d+ originMs=\d+ ownerRecoveryMs=\d+ persistenceMs=\d+ readMs=\d+ parseMs=\d+ prepareMs=\d+ agentsMs=\d+ serializeMs=\d+ writeMs=\d+ replaceMs=\d+ dirSyncMs=\d+ loopLagMs=\d+ loopLagMaxMs=\d+ queueLagMs=\d+ trustedStateLagMs=\d+ persistenceLagMs=\d+ timerLateMs=\d+$/);+ expect(line).toMatch(/^🚦 Merge admission operation=acquire status=(pending|settled) phase=(trustedState|origin|persistence) phaseMs=\d+ totalMs=\d+ queueMs=\d+ trustedStateMs=\d+ stateLoadMs=\d+ originMs=\d+ ownerRecoveryMs=\d+ persistenceMs=\d+ readMs=\d+ parseMs=\d+ prepareMs=\d+ agentsMs=\d+ serializeMs=\d+ writeMs=\d+ replaceMs=\d+ dirSyncMs=\d+ loopLagMs=\d+ loopLagMaxMs=\d+ queueLagMs=\d+ trustedStateLagMs=\d+ persistenceLagMs=\d+ timerLateMs=\d+( cpuMs=\d+ load1=\d+(\.\d+)? cpus=\d+ gcMs=\d+ invCtx=\d+)?$/);expect(line).not.toContain(admission.token);expect(line).not.toContain(agentId);expect(line).not.toContain('/repos');diff --git a/server/services/cosMergeAdmission.diagnostics.test.js b/server/services/cosMergeAdmission.diagnostics.test.jsindex f2cde89f28..26e2d23761 100644--- a/server/services/cosMergeAdmission.diagnostics.test.js+++ b/server/services/cosMergeAdmission.diagnostics.test.js@@ -30,7 +30,7 @@ const deferred = () => {const agentId = 'synthetic-parent';const acquire = () => claimMergeAdmission({ agentId, action: 'acquire' });const origin = { host: 'github.com', fullName: 'example/repo' };-const LINE = /^🚦 Merge admission operation=[a-z-]+ status=[a-z-]+ phase=[A-Za-z]+( [A-Za-z]+Ms=\d+)+$/;+const LINE = /^🚦 Merge admission operation=[a-z-]+ status=[a-z-]+ phase=[A-Za-z]+( [A-Za-z]+Ms=\d+)+( cpuMs=\d+ load1=\d+(\.\d+)? cpus=\d+ gcMs=\d+ invCtx=\d+)?$/;const reported = () => [...console.log.mock.calls, ...console.warn.mock.calls, ...console.error.mock.calls].flat();const fields = (line) => Object.fromEntries([...line.matchAll(/ ([A-Za-z]+)=(\S+)/g)].map(([, key, value]) => [key, /^\d+$/.test(value) ? Number(value) : value]));@@ -81,6 +81,7 @@ describe('merge admission slow-operation diagnostics', () => {expect((await pending).admitted).toBe(true);expect(onlyLine()).toMatchObject({ operation: 'acquire', status: 'settled', totalMs: 5000,queueMs: 5000, queueLagMs: 0, loopLagMs: 0, persistenceMs: 0, timerLateMs: 0 });+ expect(reported()[0]).not.toContain('cpuMs=');});it('attributes slow asynchronous persistence I/O to its stage without loop lag', async () => {@@ -112,7 +113,7 @@ describe('merge admission slow-operation diagnostics', () => {// The first tick interval of a stall is indistinguishable from waiting for// that tick, so lag is measured to within one 100 ms interval.expect(onlyLine()).toMatchObject({ totalMs: 6000, persistenceMs: 6000, serializeMs: 6000,- persistenceLagMs: 5900, loopLagMs: 5900, loopLagMaxMs: 5900 });+ persistenceLagMs: 5900, loopLagMs: 5900, loopLagMaxMs: 5900, cpuMs: expect.any(Number), invCtx: expect.any(Number) });});it('identifies a deadline delivered late by a blocked loop, and the expired sweep never mutates', async () => {diff --git a/server/services/cosMergeAdmission.js b/server/services/cosMergeAdmission.jsindex 9e2a1e4e71..f90663509d 100644--- a/server/services/cosMergeAdmission.js+++ b/server/services/cosMergeAdmission.js@@ -11,7 +11,7 @@ import { parsePullRequestUrl } from '../lib/gitForge.js';import { isTruthyMeta } from '../lib/metadataFlags.js';import { PR_COMPLETIONS, leavesPrForHuman, resolvePrCompletion } from '../lib/prDisposition.js';import { repositoryCleanupMergeTarget } from '../lib/recoveryMergeTarget.js';-import { createEventLoopLagProbe, phaseGroup, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';+import { createEventLoopLagProbe, formatLoadAttribution, phaseGroup, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';const retryAfterMs = 5000;export const ADMISSION_QUEUE_TIMEOUT_MS = 15_000;@@ -49,10 +49,10 @@ export const SLOW_ADMISSION_MS = 1000;let clock = systemClock;let lagProbe = createEventLoopLagProbe({ clock });-/** Inject a deterministic clock; `null` restores the system clock. */-export function __setMergeAdmissionClockForTests(next) {+/** Inject a deterministic clock and optional resource counters; `null` restores the system ones. */+export function __setMergeAdmissionClockForTests(next, resources) {clock = next ?? systemClock;- lagProbe = createEventLoopLagProbe({ clock });+ lagProbe = createEventLoopLagProbe({ clock, resources });}const REPORTED_PHASES = ['queue', 'trustedState', 'stateLoad', 'origin', 'ownerRecovery', 'persistence'];@@ -86,7 +86,7 @@ async function transaction(work, operation) {const phases = REPORTED_PHASES.map((name) => `${name}Ms=${phaseGroup(summary, name).ms}`);const stages = REPORTED_STAGES.map(([name, stage]) => `${stage}Ms=${summary.phases[`${name}.${stage}`]?.ms ?? 0}`);const lag = REPORTED_LAG_PHASES.map((name) => `${name}LagMs=${phaseGroup(summary, name).lagMs}`);- log(`🚦 Merge admission operation=${operation} status=${status} phase=${summary.phase.split('.')[0]} phaseMs=${summary.phaseMs} totalMs=${summary.totalMs} ${phases.join(' ')} ${stages.join(' ')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} ${lag.join(' ')} timerLateMs=${timerLateMs}`);+ log(`🚦 Merge admission operation=${operation} status=${status} phase=${summary.phase.split('.')[0]} phaseMs=${summary.phaseMs} totalMs=${summary.totalMs} ${phases.join(' ')} ${stages.join(' ')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} ${lag.join(' ')} timerLateMs=${timerLateMs}${formatLoadAttribution(summary.load)}`);};const measure = async (name, fn) => {span.mark(name);diff --git a/server/services/cosState.js b/server/services/cosState.jsindex 9df913c7a9..6680402580 100644--- a/server/services/cosState.js+++ b/server/services/cosState.js@@ -9,7 +9,7 @@ import { readFile, readdir, rm, rename, stat } from 'fs/promises';import { existsSync } from 'fs';import { join, dirname, basename } from 'path';import { createFileWriteQueue } from '../lib/fileWriteQueue.js';-import { createEventLoopLagProbe, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';+import { createEventLoopLagProbe, formatLoadAttribution, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';import { ensureDirs, safeJSONParse, readJSONFileStrict, PATHS, atomicWrite } from '../lib/fileUtils.js';import { isPlainObject } from '../lib/objects.js';import { normalizeDomainAutonomy, getDomainMode } from '../lib/domainAutonomy.js';@@ -65,10 +65,10 @@ export const SLOW_STATE_OPERATION_MS = 1000;let stateClock = systemClock;let stateLagProbe = createEventLoopLagProbe({ clock: stateClock });-/** Inject a deterministic clock; `null` restores the system clock. */-export function __setCosStateDiagnosticsClockForTests(clock) {+/** Inject a deterministic clock and optional resource counters; `null` restores the system ones. */+export function __setCosStateDiagnosticsClockForTests(clock, resources) {stateClock = clock ?? systemClock;- stateLagProbe = createEventLoopLagProbe({ clock: stateClock });+ stateLagProbe = createEventLoopLagProbe({ clock: stateClock, resources });}export function withStateLock(fn) {@@ -79,9 +79,9 @@ export function withStateLock(fn) {try {return await fn();} finally {- const { totalMs, lagMs, lagMaxMs } = span.end();+ const { totalMs, lagMs, lagMaxMs, load } = span.end();if (totalMs >= SLOW_STATE_OPERATION_MS) {- console.log(`🐢 CoS state queue slow hold heldMs=${totalMs} waitMs=${waitMs} loopLagMs=${lagMs} loopLagMaxMs=${lagMaxMs}`);+ console.log(`🐢 CoS state queue slow hold heldMs=${totalMs} waitMs=${waitMs} loopLagMs=${lagMs} loopLagMaxMs=${lagMaxMs}${formatLoadAttribution(load)}`);}}});@@ -726,7 +726,7 @@ export async function saveState(state, { onStage, changedAgentIds } = {}) {const summary = span.end();if (summary.totalMs >= SLOW_STATE_OPERATION_MS) {const ms = (name) => summary.phases[name]?.ms ?? 0;- console.log(`🐢 CoS state write slow outcome=${outcome} totalMs=${summary.totalMs} prepareMs=${ms('prepare')} agentsMs=${ms('agents')} serializeMs=${ms('serialize')} writeMs=${ms('write')} replaceMs=${ms('replace')} dirSyncMs=${ms('dirSync')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} sizeKiB=${Math.round(bytes / 1024)}`);+ console.log(`🐢 CoS state write slow outcome=${outcome} totalMs=${summary.totalMs} prepareMs=${ms('prepare')} agentsMs=${ms('agents')} serializeMs=${ms('serialize')} writeMs=${ms('write')} replaceMs=${ms('replace')} dirSyncMs=${ms('dirSync')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} sizeKiB=${Math.round(bytes / 1024)}${formatLoadAttribution(summary.load)}`);}}stateCache = stampGeneration(state, stateRecovery);diff --git a/server/services/cosState.test.js b/server/services/cosState.test.jsindex f74bc40095..2f6560784f 100644--- a/server/services/cosState.test.js+++ b/server/services/cosState.test.js@@ -485,10 +485,19 @@ describe('live CoS restore boundary', () => {// #12329: a slow state operation must separate durable-write stages and// event-loop lag, logging only bounded numbers.describe('state operation slow diagnostics', () => {- it('reports each persistence stage and splits a slow write into I/O wait and loop lag', async () => {+ it('reports each persistence stage, splits a slow write into I/O wait and loop lag, and attributes only lag over 1 s', async () => {const store = await freshModule();const clock = __createManualClock();- store.__setCosStateDiagnosticsClockForTests(clock);+ // Process/host counters a test moves by hand alongside the clock (#12852).+ const counters = { cpuUs: 0, gcMs: 0, switches: 0 };+ const resources = {+ cpuUsage: () => ({ user: counters.cpuUs, system: 0 }),+ loadavg: () => [7.5, 6, 5],+ cpus: () => 8,+ involuntaryContextSwitches: () => counters.switches,+ gc: { begin: () => { const since = counters.gcMs; return { ms: () => counters.gcMs - since, end: () => {} }; } },+ };+ store.__setCosStateDiagnosticsClockForTests(clock, resources);const warn = vi.spyOn(console, 'log').mockImplementation(() => {});try {const stages = [];@@ -505,8 +514,13 @@ describe('state operation slow diagnostics', () => {warn.mockClear();await store.withStateLock(async () => clock.advance(200));expect(warn).not.toHaveBeenCalled();- await store.withStateLock(async () => clock.stall(1200));- expect(warn.mock.calls.map(([line]) => line)).toEqual(['🐢 CoS state queue slow hold heldMs=1200 waitMs=0 loopLagMs=1100 loopLagMaxMs=1100']);+ // Over 1 s of lag, the line attributes it: here the process itself was+ // busy (cpuMs near heldMs), a quarter of it in GC.+ await store.withStateLock(async () => {+ clock.stall(1200);+ Object.assign(counters, { cpuUs: counters.cpuUs + 1_150_000, gcMs: counters.gcMs + 300, switches: counters.switches + 42 });+ });+ expect(warn.mock.calls.map(([line]) => line)).toEqual(['🐢 CoS state queue slow hold heldMs=1200 waitMs=0 loopLagMs=1100 loopLagMaxMs=1100 cpuMs=1150 load1=7.5 cpus=8 gcMs=300 invCtx=42']);} finally {store.__setCosStateDiagnosticsClockForTests(null);warn.mockRestore();diff --git a/server/services/pm2.js b/server/services/pm2.jsindex 45ec495d58..cdca02763c 100644--- a/server/services/pm2.js+++ b/server/services/pm2.js@@ -9,6 +9,7 @@ import { homedir } from 'os';import { atomicWrite, safeJSONParse, tryReadFile } from '../lib/fileUtils.js';import { parseCommandArgs } from '../lib/commandSecurity.js';import { parsePm2JlistStdout } from '../lib/pm2Jlist.js';+import { beginLoadSample, formatLoadAttribution, roundMs } from '../lib/eventLoopLagProbe.js';import { PATHS } from '../lib/paths.js';import { serverSecretEnvKeys, withoutServerSecretsEnv } from '../lib/processEnv.js';// The leaf, not readinessNotify.js: its observer's lazy imports would join@@ -47,16 +48,17 @@ export function clearJlistCache(pm2Home = null) {* Record a PM2 read failure and log it only on state change or reason change.* @param {string|null} pm2Home The PM2 home being read* @param {string} errorReason Formatted error description+ * @param {string} [attribution] Numbers appended to the line, never part of the dedupe key* @private*/-function logJlistFailure(pm2Home, errorReason) {+function logJlistFailure(pm2Home, errorReason, attribution = '') {const key = cacheKey(pm2Home);const state = jlistFailureState.get(key);const homeDesc = pm2Home ? 'custom' : 'default';// Log only if no prior state, or if the reason changedif (!state || state.lastError !== errorReason) {- console.error(`❌ PM2 read failed (home=${homeDesc}): ${errorReason}`);+ console.error(`❌ PM2 read failed (home=${homeDesc}): ${errorReason}${attribution}`);jlistFailureState.set(key, { lastError: errorReason });}}@@ -501,6 +503,10 @@ function fetchJlist(pm2Home = null) {const inflight = jlistInflight.get(key);if (inflight) return inflight;+ // A timed-out read is the usual first symptom of a stalled loop, so it carries+ // its own busy-versus-starved attribution (#12852).+ const startedAt = performance.now();+ const sample = beginLoadSample();// Defer the read until the single-flight entry exists, even for a sync callback.const promise = Promise.resolve().then(() => pm2Home? execPm2(['jlist'], { env: buildEnv(pm2Home), timeoutMs: JLIST_TIMEOUT_MS })@@ -514,9 +520,13 @@ function fetchJlist(pm2Home = null) {jlistCache.set(key, { data: list, ts: Date.now() });return list;}).catch(error => {- logJlistFailure(pm2Home, error.message || String(error));+ const attribution = error?.timedOut+ ? ` wallMs=${roundMs(performance.now() - startedAt)}${formatLoadAttribution(sample.read())}`+ : '';+ logJlistFailure(pm2Home, error.message || String(error), attribution);return null;}).finally(() => {+ sample.end();if (jlistInflight.get(key) === promise) jlistInflight.delete(key);});diff --git a/server/services/pm2.readFailureLog.test.js b/server/services/pm2.readFailureLog.test.jsindex 174e5efd16..3bba338bcd 100644--- a/server/services/pm2.readFailureLog.test.js+++ b/server/services/pm2.readFailureLog.test.js@@ -5,6 +5,7 @@ vi.mock('pm2', () => ({ default: { custom: function () { return mocks; } } }));import { clearJlistCache, listProcessesStrict } from './pm2.js';afterEach(() => {+ vi.useRealTimers();vi.clearAllMocks();vi.restoreAllMocks();clearJlistCache();@@ -20,7 +21,7 @@ describe('PM2 read failure logging (#8431)', () => {mocks.connect.mockImplementation(later((cb) => cb(new Error('connect ECONNREFUSED'))));for (let i = 0; i < 3; i++) expect(await listProcessesStrict()).toBeNull();- expect(readLines(errorSpy, 'PM2 read failed')).toHaveLength(1);+ expect(readLines(errorSpy, 'PM2 read failed')).toEqual(['❌ PM2 read failed (home=default): connect ECONNREFUSED']);mocks.connect.mockImplementation(later((cb) => cb(new Error('daemon timeout'))));await listProcessesStrict();@@ -33,4 +34,18 @@ describe('PM2 read failure logging (#8431)', () => {await listProcessesStrict();expect(readLines(logSpy, 'PM2 read recovered')).toHaveLength(1);});++ // #12852: the most common stall symptom carries its own busy-versus-starved attribution.+ it('attributes an RPC timeout to process CPU, GC, host load and context switches', async () => {+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'performance'] });+ const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});+ mocks.connect.mockImplementation(() => {});++ const read = listProcessesStrict();+ await vi.advanceTimersByTimeAsync(15_000);+ expect(await read).toBeNull();+ expect(readLines(errorSpy, 'PM2 read failed')).toEqual([expect.stringMatching(+ /^❌ PM2 read failed \(home=default\): pm2 RPC timed out; client disconnect requested, daemon outcome unknown wallMs=15000 cpuMs=\d+ load1=\d+(\.\d+)? cpus=\d+ gcMs=\d+ invCtx=\d+$/,+ )]);+ });});