Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
feat: split slow CoS state and merge-admission time into I/O wait vs event-loop lag (#12329)
Adam Eivy8eabfcfff412
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex bf2f9fd47c..65b2a47ad9 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -553,6 +553,33 @@ cat data/apps.json | jq .2. **Many apps**: Pagination added in recent versions3. **Network latency**: Use local access when possible+### Agents Page or Merge Admission Stalls (Slow CoS State Operations)++**Symptom**: the Agents page is slow or unavailable while merge admission refuses with `state-queue-timeout`, PM2 reads time out, or peer probes fail around the same time.++CoS runtime records share one serialized state queue (`withStateLock` in `server/services/cosState.js`). Slow operations on it log bounded numbers only: no records, tokens, paths, hostnames or task text.++| Line | When |+|------|------|+| `🚦 Merge admission operation=… status=settled …` | a settled admission transaction that took 1 s or more |+| `🚦 Merge admission … status=queue-timeout\|pending\|failed …` | always |+| `🐢 CoS state write slow outcome=… totalMs=… prepareMs serializeMs writeMs replaceMs dirSyncMs loopLagMs loopLagMaxMs sizeKiB` | any `saveState` of 1 s or more |+| `🐢 CoS state queue slow hold heldMs=… waitMs=… loopLagMs=… loopLagMaxMs=…` | any operation that held the queue for 1 s or more |++Elapsed time inside an awaited phase is asynchronous wait plus delayed scheduling. While an instrumented operation is open, a 100 ms tick measures how long timers sat overdue (`server/lib/eventLoopLagProbe.js`) and charges that to the current phase as `…LagMs`. Read each slow phase as lag versus the rest:++- **`queueMs` high, `queueLagMs` near 0** — the state queue was held by other operations while the loop stayed responsive. Look for `🐢 CoS state queue slow hold` lines in the same window.+- **`persistenceMs` (or `trustedStateMs`) high, its `…LagMs` near 0** — asynchronous I/O wait: storage or fsync latency, or libuv threadpool queueing behind other file work. `writeMs` (temp write + file flush), `replaceMs` (rename) and `dirSyncMs` (directory flush) locate it; `readMs`/`parseMs` split the ownership read.+- **`…LagMs` close to the phase time, or `timerLateMs` well above 0** — the event loop could not run: blocking JavaScript, garbage collection, host CPU starvation or process suspension. These numbers do not tell those apart; capture a CPU profile ([safely](#dynamic-import-callback-missing-after-profiling)) and host load at the time.++Lag resolves to about one tick: the first 100 ms of a stall and any overdue stretch of 20 ms or less are not counted. `timerLateMs` is how late the 15 s admission deadline fired; anything beyond a few milliseconds of timer jitter proves scheduling delay on its own.++**Baseline.** In an isolated synthetic run (a temp directory, a ~1 MiB state-shaped document, real `atomicWrite`, no induced load), a full state write took 14 ms median and 35 ms worst over 20 runs (serialize ~2 ms, write ≤ 20 ms, directory flush ≤ 8 ms) with no measurable lag, and parsing took ~1–2 ms. Multi-second phases are therefore not explained by the size of the document.++**Status of #12329.** The incident that motivated these lines (a 35.9 s persistence phase, and a deadline that fired 9.4 s late) predates them. Whether its persistence time was storage latency or scheduling pressure is **unresolved**; the overdue deadline established scheduling delay only for the queue wait. Choose a latency fix only after a safely observed incident shows which bucket dominates, and record before/after numbers at that boundary. Do not induce overload on a live install, or run concurrent heavy suites, to obtain them.++**Log timestamps.** `LOG_DATE_FORMAT` in `ecosystem.config.cjs` already emits a real UTC offset (#11436). A PM2 process registered before that change keeps its old format, which labels local wall-clock time with a literal `Z`, until it is restarted from the ecosystem file as part of a normal, authorized rollout. Until then, correlate log lines by comparing a fresh line with its file modification time or the health endpoint's timestamp rather than trusting the suffix.+### High Memory Usage**Solution**:diff --git a/server/lib/README.md b/server/lib/README.mdindex 88364f20f7..807e59bb1b 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -924,3 +924,5 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub- `trashFiles.js`: `trashFiles(paths)` moves existing literal paths to the host OS Trash/Recycle Bin, guards test data, and propagates failures without permanent-delete fallback.- `nativeTrash.js`: `moveToNativeTrash(paths)` recycles through OS facilities PortOS owns — one hidden, deadline-bound `osascript` (`NSFileManager.trashItemAtURL`, exported `MACOS_TRASH_SCRIPT`) per file on macOS so each gets Finder Put Back; one Windows PowerShell 5.1 `-EncodedCommand` run (shell32 `SHFileOperationW` with `FOF_ALLOWUNDO`, exported `WINDOWS_TRASH_SCRIPT`, paths in `PORTOS_TRASH_PATHS`, fixed-drive pre-flight) per batch; `trashXdg` per file on Linux. Refusals reject with a coded error (`TRASH_REFUSED` + per-file `report`, `TRASH_DEADLINE`, `TRASH_UNPARSABLE`, `TRASH_UNSUPPORTED_PLATFORM`); no local-directory or permanent-delete fallback.- `nativeTrashXdg.js`: `trashXdg(path)` moves one file into the freedesktop Trash 1.0 directory serving its volume on `node:fs` alone (original names, RFC 2396 `Path=`, `O_EXCL` info reservation, NAME_MAX-bounded names, mount identity from `/proc/self/mountinfo` with a `st_dev` fallback so a bind mount gets its own volume trash instead of an `EXDEV`, physical-parent mount discovery, an owned trash directory is tightened to 0700 and `TRASH_DIR_UNSAFE` refuses a symlinked or foreign-owned one or a volume root whose entries other users could rename, a sticky shared `.Trash` counts only when root or the caller owns it and falls back to `.Trash-<uid>` when it cannot hold the per-user directory, refuses instead of copying across devices); `_topDirTrash(topDir, uid)`, `_parseMountInfo(text)` and `_mountOf(realDir, mounts)` are test hooks.+- `nativeTrash.js`: `moveToNativeTrash(paths)` executes the pinned package native recycling helpers on macOS/Windows with hidden windows and a deadline; Linux uses the package XDG adapter. No local-directory or permanent-delete fallback.+- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329); `systemClock` (globals resolved at call time), `createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).diff --git a/server/lib/eventLoopLagProbe.js b/server/lib/eventLoopLagProbe.jsnew file mode 100644index 0000000000..cb6698319d--- /dev/null+++ b/server/lib/eventLoopLagProbe.js@@ -0,0 +1,158 @@+/**+ * Scoped event-loop lag attribution for slow-operation diagnostics (#12329).+ *+ * A stopwatch around an awaited phase measures asynchronous wait (storage, the+ * libuv threadpool, a queue) plus delayed scheduling, and cannot tell them+ * apart. While at least one span is open, one shared tick is rescheduled every+ * `intervalMs`; the time it spends overdue is time the loop could not run+ * timers — blocking JavaScript, GC, host CPU starvation or process suspension.+ * Each span charges that overdue time to the phase it was in, so a slow phase+ * splits into `lagMs` (scheduling) and the remainder (asynchronous wait).+ *+ * Nothing is scheduled while no span is open. Resolution is about one interval,+ * and an overdue stretch no longer than `toleranceMs` is ignored as timer+ * jitter. Phase names are caller-supplied constants; summaries are numbers only.+ */++/** Resolves the globals at call time, so fake timers in tests govern it. */+export const systemClock = {+ now: () => performance.now(),+ setTimer: (fn, ms) => setTimeout(fn, ms),+ clearTimer: (timer) => clearTimeout(timer),+};++/**+ * A deterministic clock with the same contract. `advance(ms)` models a+ * responsive loop (every timer fires on schedule); `stall(ms)` models a blocked+ * one (time passes and overdue timers wait for the next `advance`).+ */+export function createManualClock() {+ let now = 0;+ let seq = 0;+ const timers = new Map();+ const nextDue = (limit) => [...timers].filter(([, t]) => t.at <= limit)+ .sort(([a, x], [b, y]) => x.at - y.at || a - b)[0];+ return {+ now: () => now,+ setTimer: (fn, ms) => { timers.set(++seq, { fn, at: now + ms }); return seq; },+ clearTimer: (id) => { timers.delete(id); },+ advance(ms) {+ const target = now + ms;+ for (let due = nextDue(target); due; due = nextDue(target)) {+ timers.delete(due[0]);+ now = Math.max(now, due[1].at);+ due[1].fn();+ }+ now = target;+ },+ stall(ms) { now += ms; },+ };+}++export const roundMs = (ms) => Math.min(Number.MAX_SAFE_INTEGER, Math.max(0, Math.round(ms)));++/**+ * @param {{ clock?: typeof systemClock, intervalMs?: number, toleranceMs?: number }} [options]+ * @returns {{ begin: (phase: string) => { mark: (phase: string) => void, snapshot: () => object, end: () => object } }}+ */+export function createEventLoopLagProbe({ clock = systemClock, intervalMs = 100, toleranceMs = 20 } = {}) {+ const spans = new Set();+ let timer = null;+ let dueAt = 0;++ // Charge the overdue part of [accountedTo, at] to the span's current phase.+ const account = (span, at) => {+ const from = Math.max(dueAt, span.accountedTo);+ span.accountedTo = Math.max(span.accountedTo, at);+ const lag = at - from;+ if (lag <= toleranceMs) return;+ span.phases[span.phase].lagMs += lag;+ span.lagMs += lag;+ span.lagMaxMs = Math.max(span.lagMaxMs, lag);+ };++ // The tick only observes; it must never keep the process alive.+ const schedule = (at) => {+ dueAt = at + intervalMs;+ timer = clock.setTimer(tick, intervalMs);+ timer?.unref?.();+ };++ function tick() {+ const at = clock.now();+ for (const span of spans) account(span, at);+ schedule(at);+ }++ const enter = (span, phase, at) => {+ span.phase = phase;+ span.phaseStartedAt = at;+ span.phaseEnteredAt = at;+ span.phases[phase] ??= { ms: 0, lagMs: 0 };+ };++ const close = (span, at) => {+ account(span, at);+ span.phases[span.phase].ms += at - span.phaseStartedAt;+ span.phaseStartedAt = at;+ };++ const summarize = (span, at) => ({+ phase: span.phase,+ phaseMs: roundMs(at - span.phaseEnteredAt),+ totalMs: roundMs(at - span.startedAt),+ lagMs: roundMs(span.lagMs),+ lagMaxMs: roundMs(span.lagMaxMs),+ phases: Object.fromEntries(Object.entries(span.phases)+ .map(([name, { ms, lagMs }]) => [name, { ms: roundMs(ms), lagMs: roundMs(lagMs) }])),+ });++ return {+ begin(phase) {+ const at = clock.now();+ if (timer === null) schedule(at);+ const span = { startedAt: at, accountedTo: at, lagMs: 0, lagMaxMs: 0, phases: {} };+ enter(span, phase, at);+ spans.add(span);+ let ended = null;+ return {+ mark(next) {+ if (ended) return;+ const now = clock.now();+ close(span, now);+ enter(span, next, now);+ },+ snapshot() {+ if (ended) return ended;+ const now = clock.now();+ close(span, now);+ return summarize(span, now);+ },+ end() {+ if (ended) return ended;+ const now = clock.now();+ close(span, now);+ ended = summarize(span, now);+ spans.delete(span);+ if (spans.size === 0 && timer !== null) {+ clock.clearTimer(timer);+ timer = null;+ }+ return ended;+ },+ };+ },+ };+}++/** Sum the named phase and its `name.` sub-phases from a span summary. */+export function phaseGroup(summary, name) {+ let ms = 0;+ let lagMs = 0;+ for (const [phase, value] of Object.entries(summary.phases)) {+ if (phase !== name && !phase.startsWith(`${name}.`)) continue;+ ms += value.ms;+ lagMs += value.lagMs;+ }+ return { ms, lagMs };+}diff --git a/server/lib/fileCore.js b/server/lib/fileCore.jsindex 41cd3e4c4a..9981ae2207 100644--- a/server/lib/fileCore.js+++ b/server/lib/fileCore.js@@ -258,13 +258,15 @@ export async function ensureDirs(dirs) {** @param {string} filePath - Destination file path* @param {string|object} data - String or JSON-serializable value+ * @param {{ onStage?: (stage: 'write'|'replace'|'dirSync') => void }} [options] -+ * `onStage` is called as each durable stage begins, for slow-write diagnostics.* @returns {Promise<void>}** @example* await atomicWrite(FILE, { version: 1, items: [] });* await atomicWrite(LOG_FILE, 'raw string content');*/-export async function atomicWrite(filePath, data) {+export async function atomicWrite(filePath, data, { onStage } = {}) {// #6176 — the filesystem analogue of db.js's row-write backstop. Refuses// before any bytes are produced, so a leaking suite cannot even leave a temp// file behind in the real tree.@@ -272,6 +274,7 @@ export async function atomicWrite(filePath, data) {// Buffer must pass through unchanged — JSON.stringify on a Buffer produces// `{"type":"Buffer","data":[...]}` which corrupts binary writes (PNG, etc.).const payload = typeof data === 'string' || Buffer.isBuffer(data) ? data : JSON.stringify(data, null, 2);+ onStage?.('write');await ensureDir(dirname(filePath));// Preserve the destination's existing permission bits. A plain// writeFile(existing) truncated the inode in place and kept its mode, but@@ -329,11 +332,13 @@ export async function atomicWrite(filePath, data) {// rename below never publishes an inode the disk has not yet received.await writeFile(tmp, payload, { ...(existingMode !== null ? { mode: existingMode } : {}), flush: true });if (existingMode !== null) await chmod(tmp, existingMode);+ onStage?.('replace');await replace();} catch (err) {await unlink(tmp).catch(() => {});throw err;}+ onStage?.('dirSync');await syncDirectory(dirname(filePath));}diff --git a/server/lib/index.js b/server/lib/index.jsindex 204b34bb79..1723fc8f3e 100644--- a/server/lib/index.js+++ b/server/lib/index.js@@ -908,3 +908,4 @@ export * from './securityScanEvidence.js';export * from './musicDesignerValidation.js';export * from './mediaJobPreparation.js';+export * from './eventLoopLagProbe.js';diff --git a/server/routes/cosAgentRoutes.admissionQueue.test.js b/server/routes/cosAgentRoutes.admissionQueue.test.jsindex f26739a30d..88b379b5f9 100644--- a/server/routes/cosAgentRoutes.admissionQueue.test.js+++ b/server/routes/cosAgentRoutes.admissionQueue.test.js@@ -139,7 +139,7 @@ describe('merge admission state queue deadline', () => {const admission = await response;const logs = [...console.log.mock.calls, ...console.warn.mock.calls].flat();for (const line of logs) {- expect(line).toMatch(/^🚦 Merge admission operation=acquire status=(pending|settled) phase=(trustedState|origin|persistence) phaseMs=\d+ totalMs=\d+ queueMs=\d+ trustedStateMs=\d+ stateLoadMs=\d+ originMs=\d+ ownerRecoveryMs=\d+ persistenceMs=\d+$/);+ expect(line).toMatch(/^🚦 Merge admission operation=acquire status=(pending|settled) phase=(trustedState|origin|persistence) phaseMs=\d+ totalMs=\d+ queueMs=\d+ trustedStateMs=\d+ stateLoadMs=\d+ originMs=\d+ ownerRecoveryMs=\d+ persistenceMs=\d+ readMs=\d+ parseMs=\d+ prepareMs=\d+ serializeMs=\d+ writeMs=\d+ replaceMs=\d+ dirSyncMs=\d+ loopLagMs=\d+ loopLagMaxMs=\d+ queueLagMs=\d+ trustedStateLagMs=\d+ persistenceLagMs=\d+ timerLateMs=\d+$/);expect(line).not.toContain(admission.token);expect(line).not.toContain(agentId);expect(line).not.toContain('/repos');diff --git a/server/services/cosMergeAdmission.diagnostics.test.js b/server/services/cosMergeAdmission.diagnostics.test.jsnew file mode 100644index 0000000000..1723d5ac53--- /dev/null+++ b/server/services/cosMergeAdmission.diagnostics.test.js@@ -0,0 +1,136 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import { createFileWriteQueue } from '../lib/fileWriteQueue.js';+import { createManualClock } from '../lib/eventLoopLagProbe.js';++// #12329: a slow admission phase must say whether its time went to waiting on+// the state queue, to asynchronous I/O, or to an event loop that could not run.+const fixture = vi.hoisted(() => ({ state: null, disk: null, queue: null, enqueued: null, save: null }));+vi.mock('./cosState.js', () => ({+ withStateLock: (fn) => { const result = fixture.queue(fn); fixture.enqueued?.(); return result; },+ loadState: async () => fixture.state,+ readMergeAdmissionStateForSafetyCheck: async ({ onStage } = {}) => {+ onStage?.('read');+ onStage?.('parse');+ return { trusted: true, ...structuredClone(fixture.disk) };+ },+ saveState: async (state, { onStage } = {}) => {+ await fixture.save?.(onStage);+ fixture.disk = structuredClone(state);+ fixture.state = state;+ },+}));+vi.mock('../lib/gitRemote.js', () => ({ getOriginInfo: async () => ({ host: 'github.com', fullName: 'example/repo' }) }));+import { claimMergeAdmission, setMergeAdmissionClockForTests, withPendingMergeAdmission } from './cosMergeAdmission.js';++const deferred = () => {+ let resolve;+ const promise = new Promise((yes) => { resolve = yes; });+ return { promise, resolve };+};+const agentId = 'synthetic-parent';+const acquire = () => claimMergeAdmission({ agentId, action: 'acquire' });+const origin = { host: 'github.com', fullName: 'example/repo' };+const LINE = /^🚦 Merge admission operation=[a-z-]+ status=[a-z-]+ phase=[A-Za-z]+( [A-Za-z]+Ms=\d+)+$/;+const reported = () => [...console.log.mock.calls, ...console.warn.mock.calls, ...console.error.mock.calls].flat();+const fields = (line) => Object.fromEntries([...line.matchAll(/ ([A-Za-z]+)=(\S+)/g)]+ .map(([, key, value]) => [key, /^\d+$/.test(value) ? Number(value) : value]));+const onlyLine = () => {+ const lines = reported();+ expect(lines).toHaveLength(1);+ expect(lines[0]).toMatch(LINE);+ return fields(lines[0]);+};+// The transaction imports cosState lazily; wait until it has actually enqueued.+// Wrapped so the async helper does not adopt (and wait on) the pending result.+const enqueue = async (start) => {+ const enqueued = deferred();+ fixture.enqueued = enqueued.resolve;+ const result = start();+ await enqueued.promise;+ fixture.enqueued = null;+ return { result };+};++let clock;+beforeEach(() => {+ clock = createManualClock();+ setMergeAdmissionClockForTests(clock);+ fixture.queue = createFileWriteQueue();+ fixture.save = null;+ fixture.state = { agents: { [agentId]: { id: agentId, taskId: 'synthetic-task', status: 'running',+ startedAt: '2026-01-01T00:00:00Z', metadata: { sourceWorkspace: '/repos/example', claimPicksOwnBranch: true } } }, mergeAdmissions: {} };+ fixture.disk = structuredClone(fixture.state);+ vi.spyOn(console, 'log').mockImplementation(() => {});+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ vi.spyOn(console, 'error').mockImplementation(() => {});+});+afterEach(() => { setMergeAdmissionClockForTests(null); vi.restoreAllMocks(); });++describe('merge admission slow-operation diagnostics', () => {+ it('stays silent for a fast settle and reports an ordinary blocked queue as lag-free queue wait', async () => {+ const fast = await acquire();+ expect(reported()).toEqual([]);+ await claimMergeAdmission({ agentId, action: 'release', token: fast.token, outcome: 'merged' });++ const gate = deferred();+ const holder = fixture.queue(() => gate.promise);+ const { result: pending } = await enqueue(acquire);+ clock.advance(5000);+ gate.resolve();+ await holder;+ expect((await pending).admitted).toBe(true);+ expect(onlyLine()).toMatchObject({ operation: 'acquire', status: 'settled', totalMs: 5000,+ queueMs: 5000, queueLagMs: 0, loopLagMs: 0, persistenceMs: 0, timerLateMs: 0 });+ });++ it('attributes slow asynchronous persistence I/O to its stage without loop lag', async () => {+ const entered = deferred();+ const io = deferred();+ fixture.save = async (onStage) => {+ for (const stage of ['prepare', 'serialize', 'write']) onStage(stage);+ entered.resolve();+ await io.promise;+ onStage('replace');+ onStage('dirSync');+ };+ const pending = acquire();+ await entered.promise;+ clock.advance(8000);+ io.resolve();+ expect((await pending).admitted).toBe(true);+ expect(onlyLine()).toMatchObject({ status: 'settled', totalMs: 8000, queueMs: 0, persistenceMs: 8000,+ writeMs: 8000, serializeMs: 0, dirSyncMs: 0, persistenceLagMs: 0, loopLagMs: 0 });+ });++ it('attributes blocking work inside persistence to loop lag', async () => {+ fixture.save = async (onStage) => {+ onStage('serialize');+ clock.stall(6000);+ onStage('write');+ };+ expect((await acquire()).admitted).toBe(true);+ // The first tick interval of a stall is indistinguishable from waiting for+ // that tick, so lag is measured to within one 100 ms interval.+ expect(onlyLine()).toMatchObject({ totalMs: 6000, persistenceMs: 6000, serializeMs: 6000,+ persistenceLagMs: 5900, loopLagMs: 5900, loopLagMaxMs: 5900 });+ });++ it('identifies a deadline delivered late by a blocked loop, and the expired sweep never mutates', async () => {+ const gate = deferred();+ const holder = fixture.queue(() => gate.promise);+ const before = structuredClone(fixture.disk);+ const work = vi.fn();+ const { result: pending } = await enqueue(() => withPendingMergeAdmission(origin, work));+ clock.advance(10_000);+ clock.stall(14_448);+ clock.advance(0);+ expect(await pending).toMatchObject({ admitted: false, reason: 'state-queue-timeout' });+ expect(onlyLine()).toMatchObject({ operation: 'sweep-acquire', status: 'queue-timeout', phase: 'queue',+ totalMs: 24_448, queueMs: 24_448, timerLateMs: 9448, queueLagMs: 14_348, loopLagMaxMs: 14_348 });+ gate.resolve();+ await holder;+ await fixture.queue(() => {});+ expect(work).not.toHaveBeenCalled();+ expect(fixture.disk).toEqual(before);+ });+});diff --git a/server/services/cosMergeAdmission.js b/server/services/cosMergeAdmission.jsindex 5d661c98fd..7483b9a60d 100644--- a/server/services/cosMergeAdmission.js+++ b/server/services/cosMergeAdmission.js@@ -10,6 +10,7 @@ import { parsePullRequestUrl } from '../lib/gitForge.js';import { isTruthyMeta } from '../lib/metadataFlags.js';import { PR_COMPLETIONS, leavesPrForHuman, resolvePrCompletion } from '../lib/prDisposition.js';import { repositoryCleanupMergeTarget } from '../lib/recoveryMergeTarget.js';+import { createEventLoopLagProbe, phaseGroup, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';const retryAfterMs = 5000;export const ADMISSION_QUEUE_TIMEOUT_MS = 15_000;@@ -39,36 +40,58 @@ const validLease = (lease, repository) => isPlainObject(lease) && lease.reposito: lease.kind === 'sweep' && typeof lease.serverOwner === 'string' && Number.isInteger(lease.pid) && lease.pid > 0);const expired = (lease) => lease.kind === 'agent' && Date.now() - Date.parse(lease.acquiredAt) >= AGENT_LEASE_MAX_HOLD_MS;+// A settled transaction is logged only at or above this duration; a deadline,+// pending or failed one always is. See docs/TROUBLESHOOTING.md for the fields.+export const SLOW_ADMISSION_MS = 1000;+let clock = systemClock;+let lagProbe = createEventLoopLagProbe({ clock });++/** Inject a deterministic clock; `null` restores the system clock. */+export function setMergeAdmissionClockForTests(next) {+ clock = next ?? systemClock;+ lagProbe = createEventLoopLagProbe({ clock });+}++const REPORTED_PHASES = ['queue', 'trustedState', 'stateLoad', 'origin', 'ownerRecovery', 'persistence'];+const REPORTED_STAGES = [['trustedState', 'read'], ['trustedState', 'parse'], ['persistence', 'prepare'],+ ['persistence', 'serialize'], ['persistence', 'write'], ['persistence', 'replace'], ['persistence', 'dirSync']];+const REPORTED_LAG_PHASES = ['queue', 'trustedState', 'persistence'];+async function transaction(work, operation) {// Lazy: almost every task-generator suite reaches prWatcher, even when no// pending merge exists. Admission needs runtime state only on the merge path.const { withStateLock, readMergeAdmissionStateForSafetyCheck, loadState, saveState } = await import('./cosState.js');- const queuedAt = performance.now();+ const queuedAt = clock.now();+ // Each phase runs until the next begins; `phase.stage` names are its stages.+ const span = lagProbe.begin('queue');let started = false;let cancelled = false;- let phase = 'queue';- let phaseStartedAt = queuedAt;- const timings = { queue: 0, trustedState: 0, stateLoad: 0, origin: 0, ownerRecovery: 0, persistence: 0 };- const elapsed = (since) => Math.min(Number.MAX_SAFE_INTEGER, Math.max(0, Math.round(performance.now() - since)));+ let timerLateMs = 0;// Only fixed operation/phase names and bounded numeric timings reach logs.const report = (status) => {+ const summary = status === 'pending' ? span.snapshot() : span.end();+ if (status === 'settled' && summary.totalMs < SLOW_ADMISSION_MS) return;const log = status === 'failed' ? console.error : status === 'settled' ? console.log : console.warn;- log(`🚦 Merge admission operation=${operation} status=${status} phase=${phase} phaseMs=${elapsed(phaseStartedAt)} totalMs=${elapsed(queuedAt)} queueMs=${timings.queue} trustedStateMs=${timings.trustedState} stateLoadMs=${timings.stateLoad} originMs=${timings.origin} ownerRecoveryMs=${timings.ownerRecovery} persistenceMs=${timings.persistence}`);+ const phases = REPORTED_PHASES.map((name) => `${name}Ms=${phaseGroup(summary, name).ms}`);+ const stages = REPORTED_STAGES.map(([name, stage]) => `${stage}Ms=${summary.phases[`${name}.${stage}`]?.ms ?? 0}`);+ const lag = REPORTED_LAG_PHASES.map((name) => `${name}LagMs=${phaseGroup(summary, name).lagMs}`);+ log(`🚦 Merge admission operation=${operation} status=${status} phase=${summary.phase.split('.')[0]} phaseMs=${summary.phaseMs} totalMs=${summary.totalMs} ${phases.join(' ')} ${stages.join(' ')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} ${lag.join(' ')} timerLateMs=${timerLateMs}`);};const measure = async (name, fn) => {- phase = name;- phaseStartedAt = performance.now();- try { return await fn(); }- finally { timings[name] = elapsed(phaseStartedAt); }+ span.mark(name);+ return fn();};+ const stage = (name) => (sub) => span.mark(`${name}.${sub}`);+ const overdue = () => roundMs(clock.now() - queuedAt - ADMISSION_QUEUE_TIMEOUT_MS);return new Promise((resolve, reject) => {const expire = () => {cancelled = true;- timings.queue = elapsed(queuedAt);report('queue-timeout');resolve(refuse('state-queue-timeout'));};- const timer = setTimeout(() => {+ // A late timer is itself evidence of event-loop delay: record by how much.+ const timer = clock.setTimer(() => {+ timerLateMs = overdue();if (!started) expire();else report('pending');}, ADMISSION_QUEUE_TIMEOUT_MS);@@ -76,10 +99,10 @@ async function transaction(work, operation) {// Check the clock as well: a delayed timer must not let an expired// callback mutate a lease when the state queue resumes first.if (cancelled) return;- if (performance.now() - queuedAt >= ADMISSION_QUEUE_TIMEOUT_MS) { expire(); return; }+ if (clock.now() - queuedAt >= ADMISSION_QUEUE_TIMEOUT_MS) { timerLateMs = overdue(); expire(); return; }started = true;- timings.queue = elapsed(queuedAt);- const snapshot = await measure('trustedState', () => readMergeAdmissionStateForSafetyCheck().catch(() => null));+ const snapshot = await measure('trustedState', () =>+ readMergeAdmissionStateForSafetyCheck({ onStage: stage('trustedState') }).catch(() => null));if (!snapshot?.trusted || !isPlainObject(snapshot.agents) || !isPlainObject(snapshot.mergeAdmissions)) {return refuse('ownership-unreadable');}@@ -93,7 +116,7 @@ async function transaction(work, operation) {// Copy before mutation: a failed atomic write must not admit from a cache// that falsely remembers a successful grant/release.const next = { ...state, agents: { ...state.agents }, mergeAdmissions: { ...snapshot.mergeAdmissions } };- return work(next, (value) => measure('persistence', () => saveState(value)), measure);+ return work(next, (value) => measure('persistence', () => saveState(value, { onStage: stage('persistence') })), measure);});// Once started, the queue owns the entire transaction through durable// settlement. The deadline diagnoses a slow phase but cannot return a@@ -102,7 +125,7 @@ async function transaction(work, operation) {if (!cancelled) { report('settled'); resolve(result); }}, (error) => {if (!cancelled) { report('failed'); reject(error); }- }).finally(() => clearTimeout(timer));+ }).finally(() => clock.clearTimer(timer));});}diff --git a/server/services/cosState.js b/server/services/cosState.jsindex 5a57e505af..1ac823034e 100644--- a/server/services/cosState.js+++ b/server/services/cosState.js@@ -9,6 +9,7 @@ import { readFile, readdir, rm, rename } from 'fs/promises';import { existsSync } from 'fs';import { join, dirname, basename } from 'path';import { createFileWriteQueue } from '../lib/fileWriteQueue.js';+import { createEventLoopLagProbe, roundMs, systemClock } from '../lib/eventLoopLagProbe.js';import { ensureDirs, safeJSONParse, readJSONFileStrict, PATHS, atomicWrite } from '../lib/fileUtils.js';import { isPlainObject } from '../lib/objects.js';import { normalizeDomainAutonomy, getDomainMode } from '../lib/domainAutonomy.js';@@ -51,7 +52,37 @@ export const ROOT_DIR = PATHS.root;// `withStateLock(...)` call sites are unchanged; the name is kept for that// reason. The queue additionally silences its tail so one rejected write can't// poison subsequent waiters (a strict improvement over the prior mutex).-export const withStateLock = createFileWriteQueue();+const stateQueue = createFileWriteQueue();++// Slow-operation diagnostics (#12329): only bounded numbers reach the log, and+// only for an operation at or above this duration. Each held operation's time+// is split into event-loop lag and the rest, so a stalled queue can be told+// apart from a busy host (see docs/TROUBLESHOOTING.md).+export const SLOW_STATE_OPERATION_MS = 1000;+let stateClock = systemClock;+let stateLagProbe = createEventLoopLagProbe({ clock: stateClock });++/** Inject a deterministic clock; `null` restores the system clock. */+export function setCosStateDiagnosticsClockForTests(clock) {+ stateClock = clock ?? systemClock;+ stateLagProbe = createEventLoopLagProbe({ clock: stateClock });+}++export function withStateLock(fn) {+ const queuedAt = stateClock.now();+ return stateQueue(async () => {+ const waitMs = roundMs(stateClock.now() - queuedAt);+ const span = stateLagProbe.begin('hold');+ try {+ return await fn();+ } finally {+ const { totalMs, lagMs, lagMaxMs } = span.end();+ if (totalMs >= SLOW_STATE_OPERATION_MS) {+ console.warn(`🐢 CoS state queue slow hold heldMs=${totalMs} waitMs=${waitMs} loopLagMs=${lagMs} loopLagMaxMs=${lagMaxMs}`);+ }+ }+ });+}// Config gets its OWN tail, so a settings read or write never queues behind the// runtime-record writes that dominate state.json (#6182). Same@@ -486,10 +517,12 @@ export async function loadState() {// action must distinguish "known empty" from "could not establish what is// there". `trusted: false` means the file exists but could not be read as an// object — every caller must treat that as "assume the worst", never as empty.-async function readStateForSafetyCheck() {+async function readStateForSafetyCheck({ onStage } = {}) {+ onStage?.('read');await ensureDirectories();if (!existsSync(STATE_FILE)) return { trusted: true, state: null };const content = await readFile(STATE_FILE, 'utf-8');+ onStage?.('parse');const state = parseStateFile(content);if (!state) return { trusted: false, state: null };return { trusted: true, state };@@ -497,8 +530,9 @@ async function readStateForSafetyCheck() {// Merge admission shares the runtime ownership store and its write queue.// Missing leases are legacy empty state; malformed leases are never defaulted.-export async function readMergeAdmissionStateForSafetyCheck() {- const { trusted, state } = await readStateForSafetyCheck();+// `onStage('read'|'parse')` marks each stage for slow-admission diagnostics.+export async function readMergeAdmissionStateForSafetyCheck({ onStage } = {}) {+ const { trusted, state } = await readStateForSafetyCheck({ onStage });return { trusted, agents: state?.agents ?? null, mergeAdmissions: state && Object.hasOwn(state, 'mergeAdmissions') ? state.mergeAdmissions : {} };}@@ -531,21 +565,39 @@ export async function readAgentsStateForSafetyCheck() {* can neither publish that copy nor leave `stateCache.config` undefined for the* next `state.config.…` reader. A caller wanting to CHANGE a setting must call* `updateConfig()`; `state.config` is a read-only view.+ *+ * `onStage` is called as each persistence stage begins (`prepare`, `serialize`,+ * `write`, `replace`, `dirSync`) for a caller's own slow-operation diagnostics.*/-export async function saveState(state) {+export async function saveState(state, { onStage } = {}) {assertCurrentGeneration(state, stateRecovery);stampGeneration(state, stateRecovery);+ const span = stateLagProbe.begin('prepare');+ const stage = (name) => { span.mark(name); onStage?.(name); };+ onStage?.('prepare');+ let bytes = 0;+ let outcome = 'failed';try {await ensureDirectories();state.config = await loadConfig();// Config lives in its own file; never re-serialize it onto the hot path.const persisted = { ...state };delete persisted.config;- await atomicWrite(STATE_FILE, persisted);+ stage('serialize');+ const payload = JSON.stringify(persisted, null, 2);+ bytes = Buffer.byteLength(payload);+ await atomicWrite(STATE_FILE, payload, { onStage: stage });+ outcome = 'saved';} catch (error) {stateCache = null;invalidateFailedWrite(stateRecovery);throw error;+ } finally {+ const summary = span.end();+ if (summary.totalMs >= SLOW_STATE_OPERATION_MS) {+ const ms = (name) => summary.phases[name]?.ms ?? 0;+ console.warn(`🐢 CoS state write slow outcome=${outcome} totalMs=${summary.totalMs} prepareMs=${ms('prepare')} serializeMs=${ms('serialize')} writeMs=${ms('write')} replaceMs=${ms('replace')} dirSyncMs=${ms('dirSync')} loopLagMs=${summary.lagMs} loopLagMaxMs=${summary.lagMaxMs} sizeKiB=${Math.round(bytes / 1024)}`);+ }}stateCache = stampGeneration(state, stateRecovery);}diff --git a/server/services/cosState.test.js b/server/services/cosState.test.jsindex 74d0f8021c..e2f470d1b0 100644--- a/server/services/cosState.test.js+++ b/server/services/cosState.test.js@@ -2,6 +2,7 @@ import { describe, it, expect, beforeEach, afterAll, vi } from 'vitest';import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'fs';import { join } from 'path';import { mockPathsDataRoot } from '../lib/mockPathsDataRoot.js';+import { createManualClock } from '../lib/eventLoopLagProbe.js';const { tempRoot, makeProxy, cleanup } = mockPathsDataRoot({ prefix: 'cos-state-test-' });@@ -439,3 +440,35 @@ describe('live CoS restore boundary', () => {expect(readFileSync(STATE_PATH, 'utf8')).toBe('{broken');});});++// #12329: a slow state operation must separate durable-write stages and+// event-loop lag, logging only bounded numbers.+describe('state operation slow diagnostics', () => {+ it('reports each persistence stage and splits a slow write into I/O wait and loop lag', async () => {+ const store = await freshModule();+ const clock = createManualClock();+ store.setCosStateDiagnosticsClockForTests(clock);+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});+ try {+ const stages = [];+ await store.saveState(await store.loadState(), { onStage: (stage) => {+ stages.push(stage);+ if (stage === 'write') clock.advance(1500);+ if (stage === 'dirSync') clock.stall(600);+ } });+ expect(stages).toEqual(['prepare', 'serialize', 'write', 'replace', 'dirSync']);+ expect(readJson(STATE_PATH).agents).toEqual({});+ expect(warn).toHaveBeenCalledTimes(1);+ expect(warn.mock.calls[0][0]).toMatch(/^🐢 CoS state write slow outcome=saved totalMs=2100 prepareMs=0 serializeMs=0 writeMs=1500 replaceMs=0 dirSyncMs=600 loopLagMs=500 loopLagMaxMs=500 sizeKiB=\d+$/);++ warn.mockClear();+ await store.withStateLock(async () => clock.advance(200));+ expect(warn).not.toHaveBeenCalled();+ await store.withStateLock(async () => clock.stall(1200));+ expect(warn.mock.calls.map(([line]) => line)).toEqual(['🐢 CoS state queue slow hold heldMs=1200 waitMs=0 loopLagMs=1100 loopLagMaxMs=1100']);+ } finally {+ store.setCosStateDiagnosticsClockForTests(null);+ warn.mockRestore();+ }+ });+});