Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(persistence): maintenance preflight re-stamps a device-only physical-root change like ordinary writes (#7760)

garrytanb0b909d0eab0
diff --git a/docs/architecture/key-files/core-persistence.md b/docs/architecture/key-files/core-persistence.md
index 8f90d77602..dbf24df20a 100644
--- a/docs/architecture/key-files/core-persistence.md
+++ b/docs/architecture/key-files/core-persistence.md
@@ -15,7 +15,7 @@ Managed facts extraction (withdrawn-candidate skip and recovery, conversation fa
- `src/core/persistence/{admin-lock,blocking-effects,writer-versions}.ts` — `admin-lock.ts` owns the opt-in brain-level writer admin lock: reserved config key `persistence.writer_admin_lock` (`{locked, set_at, host_id, generation}`), written only by local `gbrain sources writer lock|unlock` under the `persistence_brain` row `FOR UPDATE`; lock refuses while a prepared transfer is unaccepted. `assertWriterAdminUnlocked` runs inside the claim, managed claim, activation, shared-skill activation and transfer prepare/accept transactions for every caller (share lock on the brain row) and refuses with `writer_admin_locked`; only the ordinary first-write auto-claim (`claimWorktree(..., { automatic: true })` from page and memory mutations) is exempt. Generic `config set`/`unset`/`unset --pattern` refuse the key; engine migration copies it. It is not a security boundary and older binaries ignore it. `blocking-effects.ts` lists queued, running or recovering effects (`listBlockingEffects`) and builds `writer_not_quiesced` suggestions naming the first blocking request or effect with `gbrain sources writer status <source> --json`. `writer-versions.ts` stamps admitter version/host at admission and consumer version/host/`published_at` only on committed publication (migration 178, cutoff `persistence_brain.writer_version_cutoff` on the database clock); replay and compaction keep stamps. `sources writer status` reports `admin_lock`, `local_host_id`, `blocking_effects` and `writer_versions`; `src/commands/doctor/checks/writer-version.ts` is the seven-day `writer_version` advisory. `SEGMENTED_WRITER_FLOOR` (`page-body/writer-floor.ts`): a seven-day writer below it or unmarked keeps `pages.segmented_storage` off (`page-body/limits.ts`, doctor `segmented_storage`); marked rows of any role meet the floor (ingest log: two partial indexes, v254/v259). Tests: `test/persistence-writer-admin-lock.test.ts`, `test/persistence-writer-versions.test.ts`, `test/persistence-blocking-effects.test.ts`, `test/e2e/persistence-writer-{admin-lock,stamps}.test.ts`.
-- `src/core/persistence/{atom-maintenance,atom-retry,page-prepare,facts-maintenance,facts-prepare,prepared-maintenance}.ts` — resident dispatch for durable atom publication/completion, explicit atom retry (`retryManagedAtomBatch` takes the job's signal and deadline, stops between stages on caller cancel, deadline or cycle-lock loss, shares one deadline-bound `MaintenanceWriteWait` with the model rerun, and reports a stop after admission as `write_pending` or an `AbortError` carrying the receipts, never `cancelled`; #6622; a transcript item that does not resolve to the retry's source is refused `transcript_source_mismatch` before anything starts, #7869), authorized multi-entity fact-fence/backstop append, local synthesize/patterns/consolidation, and the cycle's lint repairs on a managed brain (`runLintCore` publishes each fix through `maintenancePreflight` + `publishMaintenancePage`; #5180, `test/managed-lint.test.ts`). `page-prepare.ts` keeps the stored provenance stamps on every remote page write, repo or not (`keepStoredProvenance`, `provenanceWriteTrust`: tier by the lower of caller and published frontmatter; #7873); it exports `fileMatchesSnapshot`, the canonical file-versus-page comparison `prepareFileTarget` applies, so lint can bind a repair to the revision it read (it takes the importer's path for the file: a frontmatter-less file under a directory-inference rule also matches when `readCanonicalAsImported` (BOM strip + `applyInference` at `scannerSourcePath`) reproduces the stored row, with an inferred type treated as undeclared, #6605; reconcile reads the file side through `reconcileFileReading` in `reconcile-state.ts`); the uncoordinated-local-edit refusal carries detail `file_database_drift` and a `sources reconcile <src> <slug> --brain host --preview` fix; a shared-origin delete: canonical-reconciliation.md. In `prepareFileTarget`, a live page whose canonical file is missing refuses `source_changed` unless it is declared or derive-phase database-only, or it is a `delete_page` (`deleting`, no content) of a page with no recorded `source_path` and no captured file URI: that delete plans no file, so it publishes database-only and its receipt reports no file or Git effect (`test/persistence-captured-file-target.test.ts`, `test/persistence-never-published-delete.test.ts`). `atomOwnerTarget` is the one owner decision for an atom session and for `atomDrainOwnerRefusal`, the auto-drain dispatch pre-check; an unbound Google/GitHub source publishes atom maintenance through the `connector_database` target (its `local_path` is connector state, not a checkout) and the publication refuses with `source_changed` when the source gained an owner after preflight. `submitMaintenance` scans every content-carrying intent with `scanCanonicalFences` (the check the canonical projection runs) over the whole submitted body and timeline before a request exists (#6278): a defect no adoption clears (a second facts fence in the timeline; `replaceOrInsertFactsFence` replaces only the first) is refused with its fence reason, so `fenceManaged` reports the page in `failed_pages` and the others still adopt, while a repair whose postimage is clean passes. `prepareFactFenceAdoption` names its steps for `enterClaimStep`, compares the fence cell with the legacy fact modulo whitespace, writes the cell's parsed text back into `facts.fact` in the same `UPDATE` that sets `row_num` (so the projection keeps the fact's id), and throws the registered `fence_unrenderable` (class `server`) as its last-line defence. `maintenancePreflight` creates the job's `MaintenanceWriteWait` (optional `deadlineAtMs`) and `submitMaintenance` uses it for new and replayed requests; its owner check goes through `checkOwner` / `ownerUnavailableError` (`owner-refusal.ts`), so a missing binding, a changed incarnation, another owner host, a draining or recovering worktree and a missing local or coordination path each refuse `owner_unavailable` with their own `reason`, and nothing claims or transfers ownership to get past them. #6325: `resumeManagedAtoms` returns `run | done | held`. The drain does not replay a failed batch: it resumes the next automatic attempt, whose run key is `digest([drainKey, failedRequestId, 'automatic'])` (one stable id per failure, so a crashed attempt resumes and two runs admit the same requests), and whose completion advances the failed drain-key checkpoint by the same compare-and-set an explicit retry uses. The completion counts one strike per completed failure on that checkpoint (`attempts`); at `MAX_DETERMINISTIC_FAILURES` it sets `held`, which `MANAGED_ATOM_DISCOVERY_SQL` excludes from discovery and the backlog and `listHeldManagedAtoms` reports (doctor `extract_atoms_held_failed`, code `held_failed`). Reading a held or failed checkpoint never strikes; storage and authority errors fail the attempt without completing it, so they never strike either. #7514 write-failure strikes and hold: [continued](core-persistence-continued.md). A content change is a new drain key, so the count starts over. An explicit retry (`session.retry`) ignores `held`, and its failure refusal (`extraction_failed`) names the recorded failure reason (malformed output or a response stopped before the end) and carries a paid `fix` (`gbrain jobs submit extract-atoms-drain --params '{sourceId, retryRequestId}'`) with a read-only `gbrain write-request` verify; the drain raises it only for a failed batch whose checkpoint is gone. Tests: `test/managed-atoms-retry.test.ts`. Managed facts extraction (the withdrawn-candidate skip and its recovery route, the conversation facts batches, the unmanaged derived index) continues in [Core Persistence (managed facts)](core-persistence-facts.md).
+- `src/core/persistence/{atom-maintenance,atom-retry,page-prepare,facts-maintenance,facts-prepare,prepared-maintenance}.ts` — resident dispatch for durable atom publication/completion, explicit atom retry (`retryManagedAtomBatch` takes the job's signal and deadline, stops between stages on caller cancel, deadline or cycle-lock loss, shares one deadline-bound `MaintenanceWriteWait` with the model rerun, and reports a stop after admission as `write_pending` or an `AbortError` carrying the receipts, never `cancelled`; #6622; a transcript item that does not resolve to the retry's source is refused `transcript_source_mismatch` before anything starts, #7869), authorized multi-entity fact-fence/backstop append, local synthesize/patterns/consolidation, and the cycle's lint repairs on a managed brain (`runLintCore` publishes each fix through `maintenancePreflight` + `publishMaintenancePage`; #5180, `test/managed-lint.test.ts`). `page-prepare.ts` keeps the stored provenance stamps on every remote page write, repo or not (`keepStoredProvenance`, `provenanceWriteTrust`: tier by the lower of caller and published frontmatter; #7873); it exports `fileMatchesSnapshot`, the canonical file-versus-page comparison `prepareFileTarget` applies, so lint can bind a repair to the revision it read (it takes the importer's path for the file: a frontmatter-less file under a directory-inference rule also matches when `readCanonicalAsImported` (BOM strip + `applyInference` at `scannerSourcePath`) reproduces the stored row, with an inferred type treated as undeclared, #6605; reconcile reads the file side through `reconcileFileReading` in `reconcile-state.ts`); the uncoordinated-local-edit refusal carries detail `file_database_drift` and a `sources reconcile <src> <slug> --brain host --preview` fix; a shared-origin delete: canonical-reconciliation.md. In `prepareFileTarget`, a live page whose canonical file is missing refuses `source_changed` unless it is declared or derive-phase database-only, or it is a `delete_page` (`deleting`, no content) of a page with no recorded `source_path` and no captured file URI: that delete plans no file, so it publishes database-only and its receipt reports no file or Git effect (`test/persistence-captured-file-target.test.ts`, `test/persistence-never-published-delete.test.ts`). `atomOwnerTarget` is the one owner decision for an atom session and for `atomDrainOwnerRefusal`, the auto-drain dispatch pre-check; an unbound Google/GitHub source publishes atom maintenance through the `connector_database` target (its `local_path` is connector state, not a checkout) and the publication refuses with `source_changed` when the source gained an owner after preflight. `submitMaintenance` scans every content-carrying intent with `scanCanonicalFences` (the check the canonical projection runs) over the whole submitted body and timeline before a request exists (#6278): a defect no adoption clears (a second facts fence in the timeline; `replaceOrInsertFactsFence` replaces only the first) is refused with its fence reason, so `fenceManaged` reports the page in `failed_pages` and the others still adopt, while a repair whose postimage is clean passes. `prepareFactFenceAdoption` names its steps for `enterClaimStep`, compares the fence cell with the legacy fact modulo whitespace, writes the cell's parsed text back into `facts.fact` in the same `UPDATE` that sets `row_num` (so the projection keeps the fact's id), and throws the registered `fence_unrenderable` (class `server`) as its last-line defence. `maintenancePreflight` creates the job's `MaintenanceWriteWait` (optional `deadlineAtMs`) and `submitMaintenance` uses it for new and replayed requests; its owner check goes through `checkOwner` / `ownerUnavailableError` (`owner-refusal.ts`), so a missing binding, a changed incarnation, another owner host, a draining or recovering worktree and a missing local or coordination path each refuse `owner_unavailable` with their own `reason`, and nothing claims or transfers ownership to get past them; a device-only physical-root change takes the ordinary write's locked, ownership-verified re-stamp through a no-wait `acquireWorktree` (#7760; a busy lock keeps the `physical_root_device_changed` refusal). #6325: `resumeManagedAtoms` returns `run | done | held`. The drain does not replay a failed batch: it resumes the next automatic attempt, whose run key is `digest([drainKey, failedRequestId, 'automatic'])` (one stable id per failure, so a crashed attempt resumes and two runs admit the same requests), and whose completion advances the failed drain-key checkpoint by the same compare-and-set an explicit retry uses. The completion counts one strike per completed failure on that checkpoint (`attempts`); at `MAX_DETERMINISTIC_FAILURES` it sets `held`, which `MANAGED_ATOM_DISCOVERY_SQL` excludes from discovery and the backlog and `listHeldManagedAtoms` reports (doctor `extract_atoms_held_failed`, code `held_failed`). Reading a held or failed checkpoint never strikes; storage and authority errors fail the attempt without completing it, so they never strike either. #7514 write-failure strikes and hold: [continued](core-persistence-continued.md). A content change is a new drain key, so the count starts over. An explicit retry (`session.retry`) ignores `held`, and its failure refusal (`extraction_failed`) names the recorded failure reason (malformed output or a response stopped before the end) and carries a paid `fix` (`gbrain jobs submit extract-atoms-drain --params '{sourceId, retryRequestId}'`) with a read-only `gbrain write-request` verify; the drain raises it only for a failed batch whose checkpoint is gone. Tests: `test/managed-atoms-retry.test.ts`. Managed facts extraction (the withdrawn-candidate skip and its recovery route, the conversation facts batches, the unmanaged derived index) continues in [Core Persistence (managed facts)](core-persistence-facts.md).
Atom completion admits replacement and revision-pinned stale-atom deletion children together. A failed, partial-input or zero-yield batch admits no retirement, and each retirement child refuses `revision_conflict` until every replacement page of its run is committed (`replacementPublished`), so a partly published batch leaves the prior generation live.
Each retirement stamps `retired_by`/`retired_at` into the tombstone in the same publication; `publishManagedAtoms` restores a retired atom of the same origin when an extraction produces it again, and a user-deleted atom stays deleted. Each committed retirement bumps the origin page's regeneration generation (`op_checkpoints` op `managed-atoms-generation`, part of the drain's run key only above 0, never of the explicit retry identity; the content whose extraction retired keeps the generation its accepted batch used, so that batch stays resumable) and clears that page's older `extract_atoms_page_state` rows and completed `managed-atoms` checkpoints in the same publication, so an A→B→A edit, even after a partly failed batch, re-extracts instead of replaying a retained receipt. Retirements are admitted in slug order.
Derived atom pages may regenerate or retire when their canonical file is already absent, while ordinary page writes retain missing-file refusal.
diff --git a/src/core/persistence/prepared-maintenance.ts b/src/core/persistence/prepared-maintenance.ts
index 7261994d51..0ff4759b61 100644
--- a/src/core/persistence/prepared-maintenance.ts
+++ b/src/core/persistence/prepared-maintenance.ts
@@ -4,7 +4,7 @@ import { boundedReads } from './bounded-reads.ts';
import { join } from 'node:path';
import type { BrainEngine, FactRow } from '../engine.ts';
import { loadConfig, type GBrainConfig } from '../config.ts';
-import { opError, type OperationContext, type OperationError } from '../ops/contract.ts';
+import { opError, OperationError, type OperationContext } from '../ops/contract.ts';
import { readFix, trustedCliRequired } from '../ops/op-fix.ts';
import type { Action } from '../agent-output.ts';
import { currentSubmissionAuthority } from '../minions/submission-authority.ts';
@@ -14,7 +14,7 @@ import { fenceCellReadBackProblem, formatFenceDate, parseFactsFence, renderFacts
import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts';
import { submissionAuthority, authorizeStoredRequest, authorizeWrite } from './authority.ts';
import { currentVerifiedLocalWriter, localHostId, registerLocalWriter } from './identity.ts';
-import { getWorktreeBinding, managedPersistenceEnabled, type WorktreeBinding } from './ownership.ts';
+import { acquireWorktree, getWorktreeBinding, managedPersistenceEnabled, type WorktreeBinding } from './ownership.ts';
import { checkOwner, ownerUnavailableError } from './owner-refusal.ts';
import { admitWrite, admitWriteInTransaction, assertReplayIntent, getWriteRequest, intentDigest, type WriteAdmission } from './journal.ts';
import { assertRefusalBreakerClosed } from './refusal-breaker.ts';
@@ -120,7 +120,15 @@ export async function maintenancePreflight(engine: BrainEngine, sourceId: string
{ fix: ownerStatusFix(sourceId) });
}
await nativeLockCapability();
- assertPhysicalRoot(owner.binding.local_path, { worktreeId: owner.binding.worktree_id, coordinationPath: owner.binding.coordination_path });
+ try {
+ assertPhysicalRoot(owner.binding.local_path, { worktreeId: owner.binding.worktree_id, coordinationPath: owner.binding.coordination_path });
+ } catch (error) {
+ // #7760: a device-only change takes the ordinary write's locked, ownership-verified re-stamp (#5604); a busy lock keeps the refusal.
+ if (!(error instanceof OperationError) || error.detail !== 'physical_root_device_changed') throw error;
+ const lock = await acquireWorktree(owner.binding, 0, undefined, engine);
+ if (!lock) throw error;
+ await lock.release();
+ }
}
if (!writeThrough) writer.databaseOnlyReason = 'disabled_by_config';
else if (!binding) writer.databaseOnlyReason = 'no_repo_configured';
diff --git a/test/persistence-physical-root-device.test.ts b/test/persistence-physical-root-device.test.ts
index bb4da18326..7fa248344d 100644
--- a/test/persistence-physical-root-device.test.ts
+++ b/test/persistence-physical-root-device.test.ts
@@ -11,6 +11,8 @@ import { runPersistenceAdministration } from '../src/core/persistence/administra
import { acquireWorktree, claimWorktree, getWorktreeBinding } from '../src/core/persistence/ownership.ts';
import { registerLocalWriter, withVerifiedLocalRegistration } from '../src/core/persistence/identity.ts';
import { assertPhysicalRoot } from '../src/core/persistence/physical-root.ts';
+import { maintenancePreflight } from '../src/core/persistence/prepared-maintenance.ts';
+import { tryAcquireNativeLock } from '../src/core/persistence/native-lock.ts';
import * as rootRecord from '../src/core/persistence/physical-root-record.ts';
import { PHYSICAL_ROOT_MARKER, physicalRootReservationPath } from '../src/core/persistence/physical-root-record.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
@@ -110,6 +112,28 @@ for (const kind of testBackends()) describe(`physical root device change (${kind
expect((await put(f, 'notes/second-write')).state).toBe('committed');
}), 60_000);
+ test('#7760: maintenance preflight takes the same locked device-only re-stamp; a busy lock refuses and leaves the stamp', () => fixture(async f => {
+ const previous = String(BigInt(f.live) + 1n);
+ const binding = await reboot(f.root, f.source, previous);
+ await engine.executeRaw("UPDATE persistence_topology_changes SET state='failed' WHERE $1::uuid=ANY(worktree_ids)", [binding.worktree_id]);
+ const preflight = () => withVerifiedLocalRegistration(engine, f.registration, () => maintenancePreflight(engine, f.source));
+ const held = (await tryAcquireNativeLock(binding.coordination_path!))!;
+ expect(held).not.toBeNull();
+ try {
+ await expect(preflight()).rejects.toMatchObject({ code: 'recovery_required', detail: 'physical_root_device_changed' });
+ expect(readJson(join(f.root, PHYSICAL_ROOT_MARKER)).device).toBe(previous);
+ expect(readJson(physicalRootReservationPath(f.root)).initialDevice).toBe(previous);
+ } finally { await held.release(); }
+ expect((await preflight())?.binding?.worktree_id).toBe(binding.worktree_id);
+ expect(readJson(join(f.root, PHYSICAL_ROOT_MARKER)).device).toBe(f.live);
+ expect(readJson(physicalRootReservationPath(f.root)).initialDevice).toBe(f.live);
+ const marker = join(f.root, PHYSICAL_ROOT_MARKER), original = readJson(marker);
+ writeJson(marker, { ...original, device: previous, inode: String(BigInt(original.inode) + 1n) });
+ await expect(preflight()).rejects.toMatchObject({ code: 'recovery_required' });
+ expect(readJson(marker).device).toBe(previous);
+ writeJson(marker, original);
+ }), 60_000);
+
test('a re-stamp interrupted after its database commit finishes on the next locked write', () => fixture(async f => {
const binding = await reboot(f.root, f.source, String(BigInt(f.live) + 1n));
const [owner] = await engine.executeRaw<{ manifest: { self_transfer: Record<string, any> } }>('SELECT manifest FROM persistence_worktrees WHERE id=$1::uuid', [binding.worktree_id]);