Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix(mind): check provider liveness before interrupting a stale turn (#11287)
atomanticagentd39a8c63e3ef
diff --git a/server/lib/README.md b/server/lib/README.mdindex 49f8dee11d..d054698a7b 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -75,7 +75,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `agentRegistrationRecord.js` | The `registerAgent` payload one spawn writes, projected from its task — `buildAgentRegistration({ task, provider, … })` plus `resolveExecutionMode({ spawnHeadless, useRunner })` (`direct` / `runner` / `tui` / `runner-tui`, replacing a nested ternary chain). `agent.metadata` is a HAND-PICKED projection of `task.metadata`, never a spread: every downstream reader (run cards, `perpetualRefillPlan`, `declaresNoCommitCriterion`, the quota-burn continuation, `featureAgents`) sees the agent record alone, so an unlisted field does not survive the spawn — which is how `quotaBurnStepId` shipped persisted but unprojected (#6406) while this list was maintained inline in a 900-line orchestrator. Pure and synchronous: `primaryCheckoutBaseline` and `providerEndpoint` arrive as VALUES (both were `await`/service calls inline) keeping this leaf out of the agent-state and local-endpoint graphs. || `agentScratchPaths.js` | Runtime scratch PortOS itself writes into an agent worktree and never commits — `PUBLIC_REVIEW_INPUT_FILENAME` / `PUBLIC_REVIEW_PATCH_DIRNAME` (the public-review bundle materialized by `services/modelAbuseGuard.js`), the `AGENT_SCRATCH_PATHS` list, prefix-aware `matchesScratchRoot(path, roots)` and `isAgentScratchPath(path)`. `classifyWorktreeDirt` (`services/worktreeManager.js`) subtracts these for EVERY caller and uses the same matcher for its per-call `ignoredPaths`, so a worktree holding only scratch reads as clean end to end — before this, it was preserved by `removeWorktree`, held by `reapMergedWorktrees`, filed as a repo-state recovery task, and dispatched to a branch-reconcile coordinator, forever. Only STATIC names live here; a run-scoped name like `.agent-done-<agentId>` (`agentSentinel.js`) is passed per call instead. || `agentSentinel.js` | The `.agent-done` completion sentinel: `DONE_SENTINEL_NAME`, `doneSentinelName(agentId)` → the per-instance filename `.agent-done-<agentId>` (worktree-less agents share one workspace, so a shared name lets concurrent runs clobber — and finalize on — each other's signal), `doneSentinelCandidateNames(agentId)` → the canonical name plus the bounded one-character truncation recovery candidate for generated ids, `doneSentinelPath(workspacePath, agentId)` → the canonical path every producer resolves, `doneSentinelAgentId(filename)` → its inverse (`''` for the bare unscoped name, `null` for a non-sentinel), so the stale-sentinel sweep in `services/agentSentinelSweep.js` matches leftovers to runs the same way the producer named them, + pure `parseSentinelPayload(contents)` → `{ summary, payload }`. Back-compat — a plain-markdown sentinel yields `payload: null`; a JSON object yields its structured `payload` for a programmatic-I/O task type's `processTaskOutput` hook. `salvageSentinelPayload(contents)` (async) is the lenient second tier — runs `jsonExtract` over a fenced/prose-trailed/control-char-corrupted envelope so a less-capable model's near-valid output still surfaces its `payload` instead of being dropped. `extractSentinelPayloadFromTranscript(transcript, isPayload)` (async) is the third tier for programmatic-I/O types ONLY — scans the ANSI-stripped PTY transcript, newest balanced JSON block first, for a payload the model PRINTED instead of writing, and adopts it only if the owning hook's shape predicate accepts it (#3640). `PROGRAMMATIC_OUTPUT_COMPLETION_HEADING` names the briefing section that prints this run's absolute sentinel path — a task-type hook renders its prompt before spawn, so it points the agent at that section by name rather than at a filename it cannot know. |-| `persistentMind.js` | Pure persistent-CoS-mind runtime state: default-off durable shape, normalization, FIFO-message/self-wake selection, interrupted-wake requeue, accepted-route fingerprints/receipts, image-work rollback guard, bounded exponential backoff, watchdog staleness, and next-wake projection. |+| `persistentMind.js` | Pure persistent-CoS-mind runtime state: default-off durable shape, normalization, FIFO-message/self-wake selection, interrupted-wake requeue, accepted-route fingerprints/receipts, image-work rollback guard, bounded exponential backoff, watchdog staleness, and next-wake projection. Staleness alone never kills a turn: `services/persistentMindSupervisor.js` tracks each turn's in-flight provider calls and, before interrupting a stale heartbeat, checks liveness (an open call, and for Ollama-backed providers `/api/ps` still showing the model loaded); a live thought waits with no fixed cap, while a turn with no open call (e.g. after a process restart) or a gone provider is interrupted as before. || `persistentMindCapabilities.js` | Default-off persistent-mind action grants plus bounded typed CoS-task, private-Eidoverse, and self-cleanup contracts (app/provider/model/effort, PR completion policy, the optional exact provider/model task allowlist, and mind-owned context scopes). || `executionDurationKey.js` | Shared pure execution-duration key composers/parser, separator and absent-effort sentinel, and minimum execution sample threshold; preserves persisted keys for server writers and browser lookups. || `cosAgentListProjection.js` | What a CoS agent LIST response may carry. `toAgentListItem` / `toAgentListItems` drop the transcript and bound `metadata.taskDescription` to `AGENT_LIST_DESCRIPTION_CHARS` (2000, on a clause boundary via `textUtils.clampToCharLimit`), stamping `taskDescriptionTruncated` + the original `taskDescriptionLength` so the UI can hydrate the rest from `GET /api/cos/agents/:id`. That one field was 68% of the Agents tab payload — p50 is 75 characters but a pasted-prompt description runs past 50 KB, and `GET /api/cos/agents` sits on the first-paint critical path of every `/cos/*` tab and re-reads every 30s. Applied at the ROUTE layer only: `getAgents`/`getAgentsByDate` also feed the weekly digest and CoS reports, which need the whole text. |diff --git a/server/services/persistentMindSupervisor.js b/server/services/persistentMindSupervisor.jsindex fa691fe065..15686962e9 100644--- a/server/services/persistentMindSupervisor.js+++ b/server/services/persistentMindSupervisor.js@@ -49,7 +49,8 @@ export {import { cosEvents, emitLog } from './cosEvents.js';import { schedule, cancel, getEvent } from './eventScheduler.js';import { getDomainBudgetStatus, recordDomainUsage } from './domainUsage.js';-import { acquireLocalEndpointProviderSlot, localEndpointOfProvider } from './cosLocalEndpointSlots.js';+import { acquireLocalEndpointProviderSlot, localEndpointOfProvider, providerBaseUrl } from './cosLocalEndpointSlots.js';+import { isOllamaBackedProvider } from '../lib/aiToolkit/index.js';import { acquireCosActionReservation, acquireCosGlobalSlot } from './cosAdmissionReservations.js';import {createPersistentMindCallBoundary,@@ -105,6 +106,59 @@ let usageLimitProbeAttempt = 0;let lastWatchdogCheckAt = null;/** Wall time this process was last seen resuming from such a gap. */let watchdogResumedAt = null;+/**+ * Provider calls currently in flight, keyed by turn id. A stale heartbeat only+ * means the turn's pulse stopped; while a call is still open in THIS process+ * the thought may simply be long, so the watchdog consults this before it+ * interrupts. Empty after a process restart, so a turn orphaned by a restart+ * is still interrupted as before.+ */+const inFlightCalls = new Map();+const OLLAMA_LIVENESS_TIMEOUT_MS = 5_000;++async function providerLivenessProbe(provider, model) {+ if (!isOllamaBackedProvider(provider)) return { alive: true, reason: 'request open' };+ const base = String(providerBaseUrl(provider) || '').replace(/\/+$/, '').replace(/\/v1$/, '');+ if (!base) return { alive: true, reason: 'request open' };+ const response = await fetch(`${base}/api/ps`, { signal: AbortSignal.timeout(OLLAMA_LIVENESS_TIMEOUT_MS) }).catch(() => null);+ if (!response?.ok) return { alive: false, reason: 'Ollama is not responding' };+ const data = await response.json().catch(() => null);+ const models = Array.isArray(data?.models) ? data.models : [];+ const loaded = models.some((m) => !model || m.name === model || m.model === model);+ return loaded ? { alive: true, reason: 'model loaded' } : { alive: false, reason: `Ollama no longer has ${model || 'the model'} loaded` };+}+++function trackInFlightCall(turnId, provider, model, call) {+ return async (meta, run) => {+ const entry = { provider, model, startedAt: Date.now() };+ const calls = inFlightCalls.get(turnId) || new Set();+ calls.add(entry);+ inFlightCalls.set(turnId, calls);+ try {+ return await call(meta, run);+ } finally {+ calls.delete(entry);+ if (calls.size === 0) inFlightCalls.delete(turnId);+ }+ };+}++/**+ * Whether a stale-heartbeat turn still has a live provider call. Only a call+ * that is open in this process AND whose provider still answers counts.+ */+async function turnHasLiveCall(turnId) {+ const calls = inFlightCalls.get(turnId);+ if (!calls?.size) return { alive: false, reason: 'no provider call in flight' };+ for (const entry of calls) {+ const result = await Promise.resolve(providerLivenessProbe(entry.provider, entry.model))+ .catch((err) => ({ alive: false, reason: err?.message || 'liveness probe failed' }));+ if (result?.alive) return { alive: true, reason: result.reason };+ return { alive: false, reason: result?.reason || 'provider gone' };+ }+ return { alive: false, reason: 'no provider call in flight' };+}const nowIso = () => new Date().toISOString();const errorMessage = (error) => String(error?.message || error || 'Persistent mind turn failed')@@ -980,7 +1034,7 @@ async function runClaimedPersistentMindTurn(turn, mind, globalSlot) {effort: prepared.effort || null,signal: controller.signal,heartbeat: () => heartbeat(turn.id, generation),- callBoundary: callBoundary.call,+ callBoundary: trackInFlightCall(turn.id, prepared.provider, prepared.model || null, callBoundary.call),});const context = await preparePersistentMindContext({mindId: mind.mindId,@@ -1027,7 +1081,7 @@ async function runClaimedPersistentMindTurn(turn, mind, globalSlot) {signal: controller.signal,heartbeat: () => heartbeat(turn.id, generation),context,- callBoundary: callBoundary.call,+ callBoundary: trackInFlightCall(turn.id, prepared.provider, prepared.model || null, callBoundary.call),recordCapabilityEvent: ({ kind, id, data } = {}) => {const eventKind = kind === 'result' ? 'mind.capability.result' : 'mind.capability.request';const capabilityId = typeof id === 'string' && id ? id : randomUUID();@@ -1649,6 +1703,10 @@ export async function checkPersistentMindWatchdog(now = Date.now()) {if (watchdogResumedAt != null && now - watchdogResumedAt < PERSISTENT_MIND_LIMITS.WATCHDOG_STALE_MS) {return { interrupted: false, resumeGrace: true };}+ // A long thought is not a dead one: while the provider call is still open+ // and its provider still answers, keep waiting with no fixed cap.+ const liveness = await turnHasLiveCall(state.activeTurn.id);+ if (liveness.alive) return { interrupted: false, inFlight: true };const result = await interruptActiveTurn('Persistent mind turn heartbeat expired', 'interrupted', {retry: true,expectedTurnId: state.activeTurn.id,diff --git a/server/services/persistentMindSupervisor.test.js b/server/services/persistentMindSupervisor.test.jsindex 288c46e819..256cc1d415 100644--- a/server/services/persistentMindSupervisor.test.js+++ b/server/services/persistentMindSupervisor.test.js@@ -70,6 +70,7 @@ vi.mock('./domainUsage.js', () => ({vi.mock('./cosLocalEndpointSlots.js', () => ({acquireLocalEndpointProviderSlot: (...args) => mock.acquireSlot(...args),localEndpointOfProvider: (...args) => mock.localEndpointOfProvider(...args),+ providerBaseUrl: (provider) => provider?.endpoint || null,}));vi.mock('./agentRunEventLog.js', () => ({@@ -917,6 +918,50 @@ describe('persistent mind supervisor', () => {expect(mock.root.persistentMind.status).toBe('interrupted');});+ it.each([+ ['alive', () => Promise.resolve(new Response(JSON.stringify({ models: [{ name: 'example-model' }] }))), false],+ ['unloaded', () => Promise.resolve(new Response(JSON.stringify({ models: [] }))), true],+ ['gone', () => Promise.reject(new Error('ECONNREFUSED')), true],+ ])('watchdog with an Ollama call in flight and the runner %s', async (_label, psResponse, interrupts) => {+ let callStarted = false;+ const run = vi.fn(({ signal, callBoundary }) => callBoundary({ purpose: 'turn', round: 0 }, () => new Promise((resolve, reject) => {+ callStarted = true;+ signal.addEventListener('abort', () => reject(new Error('aborted')), { once: true });+ })));+ const probe = vi.fn(psResponse);+ vi.stubGlobal('fetch', probe);+ try {+ await supervisor.registerPersistentMindTurnAdapter({+ prepare: vi.fn(async () => ({ ok: true, provider: { id: 'example-cloud', ollamaBacked: true, endpoint: 'http://127.0.0.1:11434/v1' } })),+ run,+ });+ await supervisor.setPersistentMindEnabled(true);+ await supervisor.startPersistentMind();+ await supervisor.enqueuePersistentMindMessage({ id: 'message-1', text: 'Think for a long time.' });+ const drain = supervisor.drainPersistentMind();+ await vi.waitFor(() => expect(callStarted).toBe(true));+ // Far past the stale window: a long thought, not a suspend.+ mock.root.persistentMind.activeTurn.heartbeatAt = new Date(+ Date.now() - 10 * PERSISTENT_MIND_LIMITS.WATCHDOG_STALE_MS+ ).toISOString();+ const result = await supervisor.checkPersistentMindWatchdog();+ expect(probe).toHaveBeenCalledWith('http://127.0.0.1:11434/api/ps', expect.anything());+ if (interrupts) {+ expect(result).toEqual({ interrupted: true });+ await drain;+ expect(mock.root.persistentMind.activeTurn).toBeNull();+ } else {+ expect(result).toEqual({ interrupted: false, inFlight: true });+ expect(mock.root.persistentMind.activeTurn).not.toBeNull();+ await supervisor.interruptPersistentMindTurn?.();+ await supervisor.stopPersistentMind?.();+ await drain.catch(() => {});+ }+ } finally {+ vi.unstubAllGlobals();+ }+ });+it('watchdog restarts the stale window after a host suspend instead of killing the in-flight turn', async () => {const run = vi.fn(({ signal }) => new Promise((resolve, reject) => {signal.addEventListener('abort', () => reject(new Error('aborted')), { once: true });