Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix([issue-12738]): wait for the agent migrations before reading a legacy state.json, and hold reaping when stored statuses are unreadable

Adam Eivyfef95c11b523
diff --git a/docs/STORAGE.md b/docs/STORAGE.md
index 80df4ac917..6813d0e6c0 100644
--- a/docs/STORAGE.md
+++ b/docs/STORAGE.md
@@ -159,7 +159,7 @@ The queues federate exactly as before: a full-sync peer advertises its working s
Direct editing of the Markdown files and the file watcher are retired (decision 2026-10-20, [plan](./plans/2026-10-20-cos-task-queue-to-postgres.md)); there is no runtime fallback to either. **Migration:** db-migration 018 imports `data/TASKS.md` and `data/COS-TASKS.md` (at the location the install's CoS config named, else the defaults) and, on installs that ran the interim completed-task archive, folds `cos_task_archive` in and drops it; it verifies that every parsed task landed as a row before committing. db-migration 019, which runs only after 018 has committed, renames each file to `<name>.imported` (the recovery copy; nothing reads it) and drops the `userTasksFile` / `cosTasksFile` config keys. Input-derived, so no `data.reference` seed ships for either file. Backed up by the Postgres dump. Repository: `server/services/cosTaskRepository.js` (in-memory backend only under the dev/test escape hatch); store: `server/services/cosTaskStore.js`.
-**CoS agent records** (`cos_agents`, `db-primary`, #12738): the agent working set — running agents plus finished ones `archiveStaleAgents()` has not yet moved to `data/cos/agents/<date>/` (retention `completedAgentRetentionMs`, 24 h by default) — is one row per agent, keyed by `agent_id`. The row's `agent` column is the record's exact JSON text (`json`, not `jsonb`, so bytes and key order round-trip and the merge-admission ownership compare stays exact); `task_id`, `status`, `started_at`, `completed_at` and `pid` are projections for queries, and `seq` keeps insertion order. `cosState` keeps its cached `state.agents` map, so callers are unchanged, and `saveState()` diffs it against each record's last stored text (`server/lib/cosAgentRows.js` `planAgentSave`) and writes only the changed rows in one transaction: a status or output update for one agent no longer rewrites every other agent, and the rest of `state.json` (config is in `config.json`) is rewritten only when it changed. `cos_agent_store.version` bumps with every write and a write names the version it was planned against; a stale cache (for example after a database restore) is refused with `COS_AGENTS_STALE` and reloads, rather than deleting rows it never saw. Safety readers (`readAgentsStateForSafetyCheck`, merge admission, the restore gate, abandoned-agent reconciliation) read the rows, never the cache, and an unreadable store is `trusted: false`. The sentinel sweep reads only `{ id, status }` (`listStoredAgentStatuses`). Older finished agents stay in the date-bucketed archive, reached by id through the agent index; they are never loaded wholesale. `state.json` keeps the small runtime remainder (running/paused flags, stats, Persistent Mind runtime, merge leases) behind `withStateLock`. **Intentionally machine-local — never federated.** **Migration:** db-migration 020 imports `state.json`'s `agents` map and verifies every record byte for byte (`agent::text` against its source serialization) before committing, so a failure stops the boot; db-migration 021, which runs only after 020 committed, re-verifies, keeps the original file as `state.json.imported` (the recovery copy; nothing reads it) and rewrites `state.json` without the key. There is no runtime fallback: a `state.json` that still carries the map (a restored pre-migration file) is ignored with a warning and the key is dropped on the next save. Input-derived, so no `data.reference` seed. Stale `state.json.<pid>.<ms>.<uuid>.tmp` files left by a write that was killed are swept once per process on first load (`sweepStaleStateTempFiles`: other processes' files older than a minute). Backed up by the Postgres dump. Repository: `server/services/cosAgentRepository.js` (in-memory backend, persisted to `data/cos-agents.json`, only under the dev/test escape hatch).
+**CoS agent records** (`cos_agents`, `db-primary`, #12738): the agent working set — running agents plus finished ones `archiveStaleAgents()` has not yet moved to `data/cos/agents/<date>/` (retention `completedAgentRetentionMs`, 24 h by default) — is one row per agent, keyed by `agent_id`. The row's `agent` column is the record's exact JSON text (`json`, not `jsonb`, so bytes and key order round-trip and the merge-admission ownership compare stays exact); `task_id`, `status`, `started_at`, `completed_at` and `pid` are projections for queries, and `seq` keeps insertion order. `cosState` keeps its cached `state.agents` map, so callers are unchanged, and `saveState()` diffs it against each record's last stored text (`server/lib/cosAgentRows.js` `planAgentSave`) and writes only the changed rows in one transaction: a status or output update for one agent no longer rewrites every other agent, and the rest of `state.json` (config is in `config.json`) is rewritten only when it changed. `cos_agent_store.version` bumps with every write and a write names the version it was planned against; a stale cache (for example after a database restore) is refused with `COS_AGENTS_STALE` and reloads, rather than deleting rows it never saw. Safety readers (`readAgentsStateForSafetyCheck`, merge admission, the restore gate, abandoned-agent reconciliation) read the rows, never the cache, and an unreadable store is `trusted: false`. The sentinel sweep reads only `{ id, status }` (`listStoredAgentStatuses`). Older finished agents stay in the date-bucketed archive, reached by id through the agent index; they are never loaded wholesale. `state.json` keeps the small runtime remainder (running/paused flags, stats, Persistent Mind runtime, merge leases) behind `withStateLock`. **Intentionally machine-local — never federated.** **Migration:** db-migration 020 imports `state.json`'s `agents` map and verifies every record byte for byte (`agent::text` against its source serialization) before committing, so a failure stops the boot; db-migration 021, which runs only after 020 committed, re-verifies, keeps the original file as `state.json.imported` (the recovery copy; nothing reads it) and rewrites `state.json` without the key. Boot starts CoS before the database phase runs these migrations, so a read of a `state.json` that still carries the map waits (bounded, 2 min, then `503 COS_AGENTS_MIGRATING`) until the boot reports the migrations settled; otherwise an early read would cache an empty agent set and its save would drop the map before 020 imported it. Once the map is gone nothing waits. There is no runtime fallback: a `state.json` that still carries the map after that (a restored pre-migration file) is ignored with a warning and the key is dropped on the next save. Input-derived, so no `data.reference` seed. Stale `state.json.<pid>.<ms>.<uuid>.tmp` files left by a write that was killed are swept once per process on first load (`sweepStaleStateTempFiles`: other processes' files older than a minute). Backed up by the Postgres dump. Repository: `server/services/cosAgentRepository.js` (in-memory backend, persisted to `data/cos-agents.json`, only under the dev/test escape hatch).
**Private security assessments** reuse the existing Review Hub item store for report prose and CoS agent archives for local source/transcripts. They add no store format or database migration. The source inventory stays in run memory; interruption before report validation requires a new assessment. Assessment tasks and all their archive files are excluded from peer federation, and shared socket notifications carry no report prose. Reports follow existing local backup and Review Hub retention; temporary sandbox homes are removed after completion. See [the assessment design and research](research/2026-09-06-private-security-models.md).
diff --git a/server/services/autonomousJobs.test.js b/server/services/autonomousJobs.test.js
index a5b098311e..368089d3e0 100644
--- a/server/services/autonomousJobs.test.js
+++ b/server/services/autonomousJobs.test.js
@@ -1252,12 +1252,12 @@ describe('autonomousJobs', () => {
expect(opts.activeAgentIds().has('live-agent-99')).toBe(true)
})
- it('handles an unreadable agent store gracefully', async () => {
+ it('skips every reap when the stored statuses cannot be read (a paused agent could look orphaned)', async () => {
listStoredAgentStatuses.mockRejectedValueOnce(new Error('database unavailable'))
- await expect(agentDataCleanup()).resolves.not.toThrow()
- expect(reapMergedWorktrees).toHaveBeenCalledOnce()
- expect(cleanupOrphanedWorktrees).toHaveBeenCalledOnce()
+ await expect(agentDataCleanup()).resolves.toMatchObject({ cleaned: 0, mergedReaped: 0, worktreesReaped: 0 })
+ expect(reapMergedWorktrees).not.toHaveBeenCalled()
+ expect(cleanupOrphanedWorktrees).not.toHaveBeenCalled()
})
})
})
diff --git a/server/services/autonomousJobs/scriptHandlers.js b/server/services/autonomousJobs/scriptHandlers.js
index bb157e1942..0f45027479 100644
--- a/server/services/autonomousJobs/scriptHandlers.js
+++ b/server/services/autonomousJobs/scriptHandlers.js
@@ -143,6 +143,14 @@ async function agentDataCleanup() {
return 0
})
+ // Without the stored statuses a paused agent is indistinguishable from an
+ // orphan, so everything below that deletes on "not live" waits for a run that
+ // can read them.
+ if (!storedStatuses) {
+ console.warn('⚠️ Agent data cleanup: stored agent statuses unavailable — skipping agent dir and worktree reaping')
+ return { cleaned: 0, sentinelsRemoved, mergedReaped: 0, worktreesReaped: 0, removalFailures: 0 }
+ }
+
const agentsDir = join(PATHS.cos, 'agents')
let cleaned = 0
if (existsSync(agentsDir)) {
diff --git a/server/services/bootstrap.js b/server/services/bootstrap.js
index 6e649980c1..f5cc94378d 100644
--- a/server/services/bootstrap.js
+++ b/server/services/bootstrap.js
@@ -76,6 +76,7 @@ import { initSpawner } from './subAgentSpawner.js';
import { initCertRenewer } from './certRenewer.js';
import * as cos from './cos.js';
+import { markCosAgentMigrationsSettled } from './cosState.js';
// Side-effect-free shared state module — importing it here does NOT pull in the
// agent-lifecycle orchestrator graph (that's why activeAgents lives in its own
// leaf module), so the shutdown handler can read the live agent set cheaply.
@@ -785,6 +786,8 @@ const runDatabaseBootPhase = () => runDatabasePhase({
return { dbReady, ensureSchema };
},
+ // Releases CoS reads that wait on the agent-record migrations (020/021) once the
+ // runner has had its turn — a failed migration exits the process instead.
migrate: ({ dbReady, ensureSchema }) => runDbAndCatalogMigrations({
dbReady,
ensureSchema,
@@ -814,7 +817,7 @@ const runDatabaseBootPhase = () => runDatabasePhase({
// Bytes + sidecars + video-history.json stay authoritative; this builds a
// queryable index over them. Idempotent, safe to run every boot.
initMediaAssetIndex: async () => (await import('./mediaAssetIndex/index.js')).initMediaAssetIndex()
- }),
+ }).finally(markCosAgentMigrationsSettled),
warmStores: () => warmMandatoryStores({
// Universe Builder PG warm (#1014): listIds() is the cheapest call that
diff --git a/server/services/cosState.js b/server/services/cosState.js
index d0a044ec5e..fd99d93e80 100644
--- a/server/services/cosState.js
+++ b/server/services/cosState.js
@@ -26,6 +26,7 @@ import {
import { cosEvents } from './cosEvents.js';
import { planAgentSave } from '../lib/cosAgentRows.js';
import { applyAgentSave, loadAgentRows } from './cosAgentRepository.js';
+import { isFileBackend } from '../lib/pgFileFacade.js';
import { ServerError } from '../lib/errorHandler.js';
import { DEFAULT_ALWAYS_APPROVE_KINDS } from './taskLearning/safetyKind.js';
@@ -459,6 +460,40 @@ let agentStore = null;
let lastStatePayload = null;
let warnedLegacyAgents = false;
+// db-migrations 020/021 move a legacy state.json `agents` map into cos_agents,
+// but boot starts CoS before the database phase runs them. A read of a file that
+// still carries the map would cache an empty agent set, and its next save would
+// drop the map before 020 could import it. So that read waits for the boot to
+// report the migrations settled (bounded, so an early awaited caller fails
+// loudly instead of deadlocking the boot). Steady state never waits: the map is
+// gone once 021 has run.
+const AGENT_MIGRATION_WAIT_MS = 120_000;
+let settleAgentMigrations;
+let agentMigrationsSettled = false;
+const agentMigrations = new Promise((resolve) => { settleAgentMigrations = resolve; });
+
+/** Called by the boot once the DB migrations have run (or were skipped by the escape hatch). */
+export function markCosAgentMigrationsSettled() {
+ agentMigrationsSettled = true;
+ settleAgentMigrations();
+}
+
+async function awaitAgentMigrations() {
+ if (agentMigrationsSettled || isFileBackend()) return;
+ let timer;
+ const timeout = new Promise((_, reject) => {
+ timer = setTimeout(() => reject(new ServerError('CoS agent records are still being migrated into the database. Retry once startup finishes.', {
+ status: 503, code: 'COS_AGENTS_MIGRATING',
+ })), AGENT_MIGRATION_WAIT_MS);
+ timer.unref?.();
+ });
+ try {
+ await Promise.race([agentMigrations, timeout]);
+ } finally {
+ clearTimeout(timer);
+ }
+}
+
function dropRuntimeCaches() {
stateCache = null;
agentStore = null;
@@ -541,6 +576,10 @@ export async function loadState() {
const content = await readFile(STATE_FILE, 'utf-8');
state = parseStateFile(content) ?? await recoverFromUnreadableState(content);
}
+ if (state && Object.hasOwn(state, 'agents') && !agentMigrationsSettled && !isFileBackend()) {
+ await awaitAgentMigrations();
+ return loadState();
+ }
const { store, agents } = await loadAgentStore();
// A concurrent read/write or restore may have superseded this read.
if (stateCache || generation !== stateRecovery.generation || restored !== restoreGeneration) return loadState();
@@ -648,7 +687,9 @@ export async function saveState(state, { onStage } = {}) {
try {
await ensureDirectories();
state.config = await loadConfig();
- agentStore ??= (await loadAgentStore()).store;
+ // A save with no cached read behind it goes through loadState first, so it
+ // waits out a pending agent migration (and loads the store) like any read.
+ if (!agentStore) await loadState();
// Agent records are rows: write only those whose serialization changed, so
// one agent's update never rewrites another's (#12738).
stage('agents');
diff --git a/server/services/cosState.test.js b/server/services/cosState.test.js
index 7ced40dd5b..2b98a204fb 100644
--- a/server/services/cosState.test.js
+++ b/server/services/cosState.test.js
@@ -7,6 +7,12 @@ import { __createManualClock } from '../lib/eventLoopLagProbe.js';
const { tempRoot, makeProxy, cleanup } = mockPathsDataRoot({ prefix: 'cos-state-test-' });
vi.mock('../lib/fileUtils.js', async (importOriginal) => makeProxy(await importOriginal()));
+// cosState's view of the backend choice; the repositories keep the real (in-memory) one.
+const backendMode = vi.hoisted(() => ({ postgres: false }));
+vi.mock('../lib/pgFileFacade.js', async (importOriginal) => {
+ const real = await importOriginal();
+ return { ...real, isFileBackend: () => !backendMode.postgres && real.isFileBackend() };
+});
// Observe the row writes cosState sends, against the real in-memory backend.
vi.mock('./cosAgentRepository.js', async (importOriginal) => {
const real = await importOriginal();
@@ -39,6 +45,7 @@ beforeEach(async () => {
rmSync(COS_DIR, { recursive: true, force: true });
mkdirSync(COS_DIR, { recursive: true });
(await import('./cosAgentRepository.js')).__resetCosAgentRepository();
+ backendMode.postgres = false;
});
describe('cosState persistence', () => {
@@ -594,3 +601,36 @@ describe('agent records in cos_agents', () => {
}
});
});
+
+// Boot starts CoS before the database phase runs db-migrations 020/021.
+describe('agent migration boot gate', () => {
+ const legacy = { 'agent-example-1': { id: 'agent-example-1', status: 'completed' } };
+
+ it('holds a read of a legacy agents map until the migrations settle, so no save can drop it first', async () => {
+ writeState({ running: false, agents: legacy });
+ backendMode.postgres = true;
+ const store = await freshModule();
+ let loaded = null;
+ const load = store.loadState().then((state) => { loaded = state; });
+ const save = store.saveState({ running: true, agents: {} });
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ expect(loaded).toBeNull();
+ // Nothing touched the file while the import was pending.
+ expect(readJson(STATE_PATH).agents).toEqual(legacy);
+
+ // What 020 + 021 leave behind: the rows, and a file without the map.
+ await seedAgents(legacy);
+ writeState({ running: false });
+ store.markCosAgentMigrationsSettled();
+ await Promise.all([load, save]);
+ expect(loaded.agents).toEqual(legacy);
+ });
+
+ it('never waits once the map is gone', async () => {
+ writeState({ running: false });
+ backendMode.postgres = true;
+ const store = await freshModule();
+ await seedAgents(legacy);
+ expect((await store.loadState()).agents).toEqual(legacy);
+ });
+});