Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix(mind): check provider liveness before interrupting a stale turn (#11287)

atomanticagentd39a8c63e3ef
diff --git a/server/lib/README.md b/server/lib/README.md
index 49f8dee11d..d054698a7b 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -75,7 +75,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `agentRegistrationRecord.js` | The `registerAgent` payload one spawn writes, projected from its task — `buildAgentRegistration({ task, provider, … })` plus `resolveExecutionMode({ spawnHeadless, useRunner })` (`direct` / `runner` / `tui` / `runner-tui`, replacing a nested ternary chain). `agent.metadata` is a HAND-PICKED projection of `task.metadata`, never a spread: every downstream reader (run cards, `perpetualRefillPlan`, `declaresNoCommitCriterion`, the quota-burn continuation, `featureAgents`) sees the agent record alone, so an unlisted field does not survive the spawn — which is how `quotaBurnStepId` shipped persisted but unprojected (#6406) while this list was maintained inline in a 900-line orchestrator. Pure and synchronous: `primaryCheckoutBaseline` and `providerEndpoint` arrive as VALUES (both were `await`/service calls inline) keeping this leaf out of the agent-state and local-endpoint graphs. |
| `agentScratchPaths.js` | Runtime scratch PortOS itself writes into an agent worktree and never commits — `PUBLIC_REVIEW_INPUT_FILENAME` / `PUBLIC_REVIEW_PATCH_DIRNAME` (the public-review bundle materialized by `services/modelAbuseGuard.js`), the `AGENT_SCRATCH_PATHS` list, prefix-aware `matchesScratchRoot(path, roots)` and `isAgentScratchPath(path)`. `classifyWorktreeDirt` (`services/worktreeManager.js`) subtracts these for EVERY caller and uses the same matcher for its per-call `ignoredPaths`, so a worktree holding only scratch reads as clean end to end — before this, it was preserved by `removeWorktree`, held by `reapMergedWorktrees`, filed as a repo-state recovery task, and dispatched to a branch-reconcile coordinator, forever. Only STATIC names live here; a run-scoped name like `.agent-done-<agentId>` (`agentSentinel.js`) is passed per call instead. |
| `agentSentinel.js` | The `.agent-done` completion sentinel: `DONE_SENTINEL_NAME`, `doneSentinelName(agentId)` → the per-instance filename `.agent-done-<agentId>` (worktree-less agents share one workspace, so a shared name lets concurrent runs clobber — and finalize on — each other's signal), `doneSentinelCandidateNames(agentId)` → the canonical name plus the bounded one-character truncation recovery candidate for generated ids, `doneSentinelPath(workspacePath, agentId)` → the canonical path every producer resolves, `doneSentinelAgentId(filename)` → its inverse (`''` for the bare unscoped name, `null` for a non-sentinel), so the stale-sentinel sweep in `services/agentSentinelSweep.js` matches leftovers to runs the same way the producer named them, + pure `parseSentinelPayload(contents)` → `{ summary, payload }`. Back-compat — a plain-markdown sentinel yields `payload: null`; a JSON object yields its structured `payload` for a programmatic-I/O task type's `processTaskOutput` hook. `salvageSentinelPayload(contents)` (async) is the lenient second tier — runs `jsonExtract` over a fenced/prose-trailed/control-char-corrupted envelope so a less-capable model's near-valid output still surfaces its `payload` instead of being dropped. `extractSentinelPayloadFromTranscript(transcript, isPayload)` (async) is the third tier for programmatic-I/O types ONLY — scans the ANSI-stripped PTY transcript, newest balanced JSON block first, for a payload the model PRINTED instead of writing, and adopts it only if the owning hook's shape predicate accepts it (#3640). `PROGRAMMATIC_OUTPUT_COMPLETION_HEADING` names the briefing section that prints this run's absolute sentinel path — a task-type hook renders its prompt before spawn, so it points the agent at that section by name rather than at a filename it cannot know. |
-| `persistentMind.js` | Pure persistent-CoS-mind runtime state: default-off durable shape, normalization, FIFO-message/self-wake selection, interrupted-wake requeue, accepted-route fingerprints/receipts, image-work rollback guard, bounded exponential backoff, watchdog staleness, and next-wake projection. |
+| `persistentMind.js` | Pure persistent-CoS-mind runtime state: default-off durable shape, normalization, FIFO-message/self-wake selection, interrupted-wake requeue, accepted-route fingerprints/receipts, image-work rollback guard, bounded exponential backoff, watchdog staleness, and next-wake projection. Staleness alone never kills a turn: `services/persistentMindSupervisor.js` tracks each turn's in-flight provider calls and, before interrupting a stale heartbeat, checks liveness (an open call, and for Ollama-backed providers `/api/ps` still showing the model loaded); a live thought waits with no fixed cap, while a turn with no open call (e.g. after a process restart) or a gone provider is interrupted as before. |
| `persistentMindCapabilities.js` | Default-off persistent-mind action grants plus bounded typed CoS-task, private-Eidoverse, and self-cleanup contracts (app/provider/model/effort, PR completion policy, the optional exact provider/model task allowlist, and mind-owned context scopes). |
| `executionDurationKey.js` | Shared pure execution-duration key composers/parser, separator and absent-effort sentinel, and minimum execution sample threshold; preserves persisted keys for server writers and browser lookups. |
| `cosAgentListProjection.js` | What a CoS agent LIST response may carry. `toAgentListItem` / `toAgentListItems` drop the transcript and bound `metadata.taskDescription` to `AGENT_LIST_DESCRIPTION_CHARS` (2000, on a clause boundary via `textUtils.clampToCharLimit`), stamping `taskDescriptionTruncated` + the original `taskDescriptionLength` so the UI can hydrate the rest from `GET /api/cos/agents/:id`. That one field was 68% of the Agents tab payload — p50 is 75 characters but a pasted-prompt description runs past 50 KB, and `GET /api/cos/agents` sits on the first-paint critical path of every `/cos/*` tab and re-reads every 30s. Applied at the ROUTE layer only: `getAgents`/`getAgentsByDate` also feed the weekly digest and CoS reports, which need the whole text. |
diff --git a/server/services/persistentMindSupervisor.js b/server/services/persistentMindSupervisor.js
index fa691fe065..15686962e9 100644
--- a/server/services/persistentMindSupervisor.js
+++ b/server/services/persistentMindSupervisor.js
@@ -49,7 +49,8 @@ export {
import { cosEvents, emitLog } from './cosEvents.js';
import { schedule, cancel, getEvent } from './eventScheduler.js';
import { getDomainBudgetStatus, recordDomainUsage } from './domainUsage.js';
-import { acquireLocalEndpointProviderSlot, localEndpointOfProvider } from './cosLocalEndpointSlots.js';
+import { acquireLocalEndpointProviderSlot, localEndpointOfProvider, providerBaseUrl } from './cosLocalEndpointSlots.js';
+import { isOllamaBackedProvider } from '../lib/aiToolkit/index.js';
import { acquireCosActionReservation, acquireCosGlobalSlot } from './cosAdmissionReservations.js';
import {
createPersistentMindCallBoundary,
@@ -105,6 +106,59 @@ let usageLimitProbeAttempt = 0;
let lastWatchdogCheckAt = null;
/** Wall time this process was last seen resuming from such a gap. */
let watchdogResumedAt = null;
+/**
+ * Provider calls currently in flight, keyed by turn id. A stale heartbeat only
+ * means the turn's pulse stopped; while a call is still open in THIS process
+ * the thought may simply be long, so the watchdog consults this before it
+ * interrupts. Empty after a process restart, so a turn orphaned by a restart
+ * is still interrupted as before.
+ */
+const inFlightCalls = new Map();
+const OLLAMA_LIVENESS_TIMEOUT_MS = 5_000;
+
+async function providerLivenessProbe(provider, model) {
+ if (!isOllamaBackedProvider(provider)) return { alive: true, reason: 'request open' };
+ const base = String(providerBaseUrl(provider) || '').replace(/\/+$/, '').replace(/\/v1$/, '');
+ if (!base) return { alive: true, reason: 'request open' };
+ const response = await fetch(`${base}/api/ps`, { signal: AbortSignal.timeout(OLLAMA_LIVENESS_TIMEOUT_MS) }).catch(() => null);
+ if (!response?.ok) return { alive: false, reason: 'Ollama is not responding' };
+ const data = await response.json().catch(() => null);
+ const models = Array.isArray(data?.models) ? data.models : [];
+ const loaded = models.some((m) => !model || m.name === model || m.model === model);
+ return loaded ? { alive: true, reason: 'model loaded' } : { alive: false, reason: `Ollama no longer has ${model || 'the model'} loaded` };
+}
+
+
+function trackInFlightCall(turnId, provider, model, call) {
+ return async (meta, run) => {
+ const entry = { provider, model, startedAt: Date.now() };
+ const calls = inFlightCalls.get(turnId) || new Set();
+ calls.add(entry);
+ inFlightCalls.set(turnId, calls);
+ try {
+ return await call(meta, run);
+ } finally {
+ calls.delete(entry);
+ if (calls.size === 0) inFlightCalls.delete(turnId);
+ }
+ };
+}
+
+/**
+ * Whether a stale-heartbeat turn still has a live provider call. Only a call
+ * that is open in this process AND whose provider still answers counts.
+ */
+async function turnHasLiveCall(turnId) {
+ const calls = inFlightCalls.get(turnId);
+ if (!calls?.size) return { alive: false, reason: 'no provider call in flight' };
+ for (const entry of calls) {
+ const result = await Promise.resolve(providerLivenessProbe(entry.provider, entry.model))
+ .catch((err) => ({ alive: false, reason: err?.message || 'liveness probe failed' }));
+ if (result?.alive) return { alive: true, reason: result.reason };
+ return { alive: false, reason: result?.reason || 'provider gone' };
+ }
+ return { alive: false, reason: 'no provider call in flight' };
+}
const nowIso = () => new Date().toISOString();
const errorMessage = (error) => String(error?.message || error || 'Persistent mind turn failed')
@@ -980,7 +1034,7 @@ async function runClaimedPersistentMindTurn(turn, mind, globalSlot) {
effort: prepared.effort || null,
signal: controller.signal,
heartbeat: () => heartbeat(turn.id, generation),
- callBoundary: callBoundary.call,
+ callBoundary: trackInFlightCall(turn.id, prepared.provider, prepared.model || null, callBoundary.call),
});
const context = await preparePersistentMindContext({
mindId: mind.mindId,
@@ -1027,7 +1081,7 @@ async function runClaimedPersistentMindTurn(turn, mind, globalSlot) {
signal: controller.signal,
heartbeat: () => heartbeat(turn.id, generation),
context,
- callBoundary: callBoundary.call,
+ callBoundary: trackInFlightCall(turn.id, prepared.provider, prepared.model || null, callBoundary.call),
recordCapabilityEvent: ({ kind, id, data } = {}) => {
const eventKind = kind === 'result' ? 'mind.capability.result' : 'mind.capability.request';
const capabilityId = typeof id === 'string' && id ? id : randomUUID();
@@ -1649,6 +1703,10 @@ export async function checkPersistentMindWatchdog(now = Date.now()) {
if (watchdogResumedAt != null && now - watchdogResumedAt < PERSISTENT_MIND_LIMITS.WATCHDOG_STALE_MS) {
return { interrupted: false, resumeGrace: true };
}
+ // A long thought is not a dead one: while the provider call is still open
+ // and its provider still answers, keep waiting with no fixed cap.
+ const liveness = await turnHasLiveCall(state.activeTurn.id);
+ if (liveness.alive) return { interrupted: false, inFlight: true };
const result = await interruptActiveTurn('Persistent mind turn heartbeat expired', 'interrupted', {
retry: true,
expectedTurnId: state.activeTurn.id,
diff --git a/server/services/persistentMindSupervisor.test.js b/server/services/persistentMindSupervisor.test.js
index 288c46e819..256cc1d415 100644
--- a/server/services/persistentMindSupervisor.test.js
+++ b/server/services/persistentMindSupervisor.test.js
@@ -70,6 +70,7 @@ vi.mock('./domainUsage.js', () => ({
vi.mock('./cosLocalEndpointSlots.js', () => ({
acquireLocalEndpointProviderSlot: (...args) => mock.acquireSlot(...args),
localEndpointOfProvider: (...args) => mock.localEndpointOfProvider(...args),
+ providerBaseUrl: (provider) => provider?.endpoint || null,
}));
vi.mock('./agentRunEventLog.js', () => ({
@@ -917,6 +918,50 @@ describe('persistent mind supervisor', () => {
expect(mock.root.persistentMind.status).toBe('interrupted');
});
+ it.each([
+ ['alive', () => Promise.resolve(new Response(JSON.stringify({ models: [{ name: 'example-model' }] }))), false],
+ ['unloaded', () => Promise.resolve(new Response(JSON.stringify({ models: [] }))), true],
+ ['gone', () => Promise.reject(new Error('ECONNREFUSED')), true],
+ ])('watchdog with an Ollama call in flight and the runner %s', async (_label, psResponse, interrupts) => {
+ let callStarted = false;
+ const run = vi.fn(({ signal, callBoundary }) => callBoundary({ purpose: 'turn', round: 0 }, () => new Promise((resolve, reject) => {
+ callStarted = true;
+ signal.addEventListener('abort', () => reject(new Error('aborted')), { once: true });
+ })));
+ const probe = vi.fn(psResponse);
+ vi.stubGlobal('fetch', probe);
+ try {
+ await supervisor.registerPersistentMindTurnAdapter({
+ prepare: vi.fn(async () => ({ ok: true, provider: { id: 'example-cloud', ollamaBacked: true, endpoint: 'http://127.0.0.1:11434/v1' } })),
+ run,
+ });
+ await supervisor.setPersistentMindEnabled(true);
+ await supervisor.startPersistentMind();
+ await supervisor.enqueuePersistentMindMessage({ id: 'message-1', text: 'Think for a long time.' });
+ const drain = supervisor.drainPersistentMind();
+ await vi.waitFor(() => expect(callStarted).toBe(true));
+ // Far past the stale window: a long thought, not a suspend.
+ mock.root.persistentMind.activeTurn.heartbeatAt = new Date(
+ Date.now() - 10 * PERSISTENT_MIND_LIMITS.WATCHDOG_STALE_MS
+ ).toISOString();
+ const result = await supervisor.checkPersistentMindWatchdog();
+ expect(probe).toHaveBeenCalledWith('http://127.0.0.1:11434/api/ps', expect.anything());
+ if (interrupts) {
+ expect(result).toEqual({ interrupted: true });
+ await drain;
+ expect(mock.root.persistentMind.activeTurn).toBeNull();
+ } else {
+ expect(result).toEqual({ interrupted: false, inFlight: true });
+ expect(mock.root.persistentMind.activeTurn).not.toBeNull();
+ await supervisor.interruptPersistentMindTurn?.();
+ await supervisor.stopPersistentMind?.();
+ await drain.catch(() => {});
+ }
+ } finally {
+ vi.unstubAllGlobals();
+ }
+ });
+
it('watchdog restarts the stale window after a host suspend instead of killing the in-flight turn', async () => {
const run = vi.fn(({ signal }) => new Promise((resolve, reject) => {
signal.addEventListener('abort', () => reject(new Error('aborted')), { once: true });