Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(facts): held-file refusals get the page checks; pending before mixed; superseded source page is re-extract; deterministic job rows (#7602 review)
garrytan52dae8474bf5
diff --git a/docs/architecture/key-files/core-utilities-2.md b/docs/architecture/key-files/core-utilities-2.mdindex b6a22897ed..5c4a6e49af 100644--- a/docs/architecture/key-files/core-utilities-2.md+++ b/docs/architecture/key-files/core-utilities-2.md@@ -13,7 +13,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/repair/captured-facts.ts` — `gbrain repair captured-facts`, explicit-only and preview-bound. `classifyCapturedFacts` (read-only; also the `captured_facts_active` count) takes active capture-lane facts and classes each by its session (`captureSessionId`: `sweep:corpus:<file>` resolves through `corpusFileSessionId`): `evidenced` when a gbrain claude-cli scratch project holds the session's harness transcript or its corpus file sits in `<corpus>.quarantine/`; `ambiguous` (paste candidate) when a retained corpus file of the session holds a paste block and at least 60% of the claim's content words occur there only inside paste blocks; `excluded:legitimate_duplicate` when the claim and entity also have an active row that is not proven suspect. Sessions with no transcript and no corpus file are counted `unclassifiable`. The preview hashes every candidate with the selection (`--include-ambiguous`) and saves one approved item per page through `preview-approval.ts`; the apply reclassifies each fact, expires database-only rows through one database-only `managed_maintenance_expire_captured_facts` request per page on a managed brain (owner-checked by `maintenancePreflight`; `prepareCapturedFactsExpiry`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain, and strikes fenced rows in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection expires them and a later page write cannot reactivate them. No withdrawal is recorded, so `remember` can save the same claim again. Pinned by `test/repair-captured-facts.test.ts` (+ Postgres arm).- `src/core/repair/redacted-facts.ts` — `gbrain repair redacted-facts` (#7125), explicit-only and preview-bound. `classifyRedactedFacts` (read-only) takes active facts whose text holds a redaction marker and runs `restoreNeutralized` (`think/sanitize.ts`, with the pre-fix `LEGACY_EVAL_SHELL` shape) against the page the fact's context names and the page holding it (`pageTexts`: an inline page's body + timeline, a segmented page's body through `readBodyRange` up to 16 MiB, then up to 50 `page_versions` bodies): `evidenced` when every proving text proves the same restoration, `ambiguous` (`conflicting_sources`) when a text holds two spans around the words or texts disagree, `unproven` (`no_source_page` / `not_in_source`) otherwise. A proved fenced row on a segmented page is `segmented` (`segmented_page`): listed with its restoration, never rewritten (`restoreFencedRows` reads and publishes inline bodies only). The preview hashes every candidate with its restoration and saves one approved item per page; the apply reclassifies each fact, rewrites database-only rows through one database-only `managed_maintenance_restore_redacted_facts` request per page on a managed brain (`prepareRedactedFactsRestore`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain (exact-text compare-and-set; context gets `restored: redacted-facts repair`), and rewrites fenced rows' claims in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection replaces the row. Pinned by `test/repair-redacted-facts.test.ts` (+ Postgres arm).- `src/core/repair/loop-facts.ts` — `gbrain repair loop-facts` (#5869), explicit-only and preview-bound (`--apply --expect <hash>`): candidates are `loopFactDrift` rows (closed `done`/`dropped` loops whose commitment fact is active, in the loop's source and shared with no open loop); each item is rechecked (`changed_since_preview`) and retired through `retireLoopFact`, the same coordinated strike-plus-expire publication `loops_close` uses.-- `src/core/repair/failed-writes.ts` — `gbrain repair failed-writes` (#5983), explicit-only and preview-bound: candidates are failed, uncompacted `persistence_requests` refused by the managed writer guard (`writer_coordinator_required`, or `storage_error` "Publication failed (P0001)"). Caller writes (`put_page` without an internal `kind`, `add_timeline_entry`, `remember`) are classified `replay` / `already_written` / `duplicate` / `superseded` / `unpinned_target` by later requests on the same page and by a committed replay request id, and `file_database_drift` (W4.11) when its last replay attempt was refused `source_changed` and the page has not been written since (`lastFailedAttempt`; the detail and an apply's `file_database_drift` outcome name `gbrain sources reconcile <source> <slug> --preview`; `test/repair-failed-writes-drift-w411.test.ts`); owner-produced writes are `producer_owned` residuals with the producer command (a failed `managed_file_repair` names both `gbrain repair frontmatter` and `gbrain repair fences`). #7602: fact rows (moved-entity refusals included) use `factsRecovery`: `recover:` (retry-dead reset) only while the page write's facts-absorb job is dead and the entity page is the one admitted, `pending:` while that job runs, `not recoverable:` for a deleted page, and `re-extract:` otherwise (replaced page, completed or missing job, exhausted follow-ups, an extraction whose open entity requests are not all moved-entity or all held-file refusals, a guard refusal). Only the preview reads fact rows (`failedWrites(..., { facts: true })`); the sync-time remember replay and apply-by-id keep the guard/drift query. The approved set records each target page's revision. Apply resumes an admitted replay attempt, otherwise reclassifies and rechecks the revision, re-runs `authorizeStoredRequest` on the failed row, and resubmits the retained intent on the original trust lane (`laneContext`: remote, take holders, delegated prefixes) through `submitPageMutation` / `submitRememberMutation`, with `put_page` bound to the previewed revision and `remember` pinned to its resolved subject, under a request id derived from the failed request id and attempt (`repair-failed-writes-5983.test.ts`). #5994: `laneContext` sets `ctx.replayAuthority` to the stored authority, so admission (`submissionAuthority` → `replayedAuthority` in `persistence/authority.ts`, and `requestPrincipalForContext`) reuses its principal, delegation, scopes and autoLinkTrusted for the same source incarnation and re-authorizes it live; a restricted write gets back its recorded job (`delegatedJobId`, or the id in a single legacy `wiki/agents/<id>/*` prefix, replayed on the legacy sandboxed path when its original was), and one with neither is accepted by `enforceSubagentSlugFence` (`ops/context.ts`) only through that marker with the exact non-empty stored allow-list. The fence still fails closed for every live dispatch. A put_page replay passes the stored `timeline_section` (#5969). The superseded / already-written / duplicate rule lives in `persistence/terminal-classify.ts`, shared with the #6650 terminal-failure readers.+- `src/core/repair/failed-writes.ts` — `gbrain repair failed-writes` (#5983), explicit-only and preview-bound: candidates are failed, uncompacted `persistence_requests` refused by the managed writer guard (`writer_coordinator_required`, or `storage_error` "Publication failed (P0001)"). Caller writes (`put_page` without an internal `kind`, `add_timeline_entry`, `remember`) are classified `replay` / `already_written` / `duplicate` / `superseded` / `unpinned_target` by later requests on the same page and by a committed replay request id, and `file_database_drift` (W4.11) when its last replay attempt was refused `source_changed` and the page has not been written since (`lastFailedAttempt`; the detail and an apply's `file_database_drift` outcome name `gbrain sources reconcile <source> <slug> --preview`; `test/repair-failed-writes-drift-w411.test.ts`); owner-produced writes are `producer_owned` residuals with the producer command (a failed `managed_file_repair` names both `gbrain repair frontmatter` and `gbrain repair fences`). #7602: fact rows (moved-entity refusals included) use `factsRecovery`: `recover:` (retry-dead reset) only while the page write's facts-absorb job is dead and the entity page is the one admitted, `pending:` while that job runs, `not recoverable:` for a deleted page, and `re-extract:` otherwise (replaced page, completed or missing job, exhausted follow-ups, a source page written again since the job's page write, which the job's own `readFactsBackstopJobPage` check skips as superseded, an extraction whose open entity requests are not all moved-entity or all held-file refusals, a guard refusal). Only the preview reads fact rows (`failedWrites(..., { facts: true })`); the sync-time remember replay and apply-by-id keep the guard/drift query. The approved set records each target page's revision. Apply resumes an admitted replay attempt, otherwise reclassifies and rechecks the revision, re-runs `authorizeStoredRequest` on the failed row, and resubmits the retained intent on the original trust lane (`laneContext`: remote, take holders, delegated prefixes) through `submitPageMutation` / `submitRememberMutation`, with `put_page` bound to the previewed revision and `remember` pinned to its resolved subject, under a request id derived from the failed request id and attempt (`repair-failed-writes-5983.test.ts`). #5994: `laneContext` sets `ctx.replayAuthority` to the stored authority, so admission (`submissionAuthority` → `replayedAuthority` in `persistence/authority.ts`, and `requestPrincipalForContext`) reuses its principal, delegation, scopes and autoLinkTrusted for the same source incarnation and re-authorizes it live; a restricted write gets back its recorded job (`delegatedJobId`, or the id in a single legacy `wiki/agents/<id>/*` prefix, replayed on the legacy sandboxed path when its original was), and one with neither is accepted by `enforceSubagentSlugFence` (`ops/context.ts`) only through that marker with the exact non-empty stored allow-list. The fence still fails closed for every live dispatch. A put_page replay passes the stored `timeline_section` (#5969). The superseded / already-written / duplicate rule lives in `persistence/terminal-classify.ts`, shared with the #6650 terminal-failure readers.- `src/core/repair/ontology-facts.ts` — `gbrain repair ontology-facts` (#6264), explicit-only and preview-bound (`--apply --expect <hash>`). `classifyOntologyFacts` (read-only; also doctor `ontology_facts_fenced`) takes ontology rows (`dimension IS NOT NULL`) carrying the pre-fix fence step's signature, a `row_num` or a `source_markdown_slug` that differs from `source` (`mergeOntologyFact` writes them equal and never sets `row_num`), and classes each `fenced` or `retired`, or excluded as `withdrawn` (`fact_withdrawals`), `consolidated` or `duplicate` (restoring would collide with the ontology dedup key). The apply sets `row_num` NULL, `source_markdown_slug = source` and `expired_at` NULL in one statement that rechecks row number, page, expiry and the dedup key (`changed_since_preview` otherwise), through `coordinatedDatabaseWrite('ontology_propose')` on the entity and page keys on a managed brain or one maintenance transaction otherwise; the withdrawal trigger re-expires a claim forgotten after the preview (`withdrawn`). Page text is never rewritten. Tests: `test/ontology-fence-sweep.test.ts`.- `src/core/repair/timeline-comments.ts` — `gbrain repair timeline-comments` (#6184), explicit-only. The plan lists pages with timeline rows whose source, summary or detail carries `<!--`/`-->` or whose body has a materialized bullet carrying markup (`dropCommentBullets`), with residuals `comment_only_rows`, `comment_bearing_rows`, `pages_with_comment_bullets`. Apply: one revision-bound `put_page` per page dropping those generated bullets (user-written bullets are left), then one transaction under the page lock (`withCoordinatedWrite` with maintenance attribution on managed brains) that deletes comment-only rows and rewrites the rest to their stripped tuple, or deletes them when the page already has it. Idempotent. `timeline.ts` warns `timeline_comment_markup` naming this kind while such rows exist. Rows on a segmented page are not planned (the apply rewrites a whole body): residuals `segmented_pages` / `segmented_rows` and a warning naming them.- `src/core/repair/aliases.ts` — `gbrain repair aliases` (#6619), explicit-only, projection publication, wired to doctor `dangling_aliases`. One item per (source, target slug) whose alias rows (`page_aliases.slug`, `slug_aliases.canonical_slug`) point at a slug with no page row at all; apply rechecks that under the page key lock and deletes both tables' rows for it, inside `withCoordinatedWrite` on a managed brain (both tables carry the managed writer guard). Aliases of soft-deleted pages are the `soft_deleted_target` residual; rename redirects (`alias_slug` = the missing slug) are never touched. Exports `ORPHANED_ALIAS_COUNTS_SQL` / `SOFT_DELETED_ALIAS_COUNT_SQL`, which `checkDanglingAliases` (`src/core/onboard/checks.ts`) reports apart, naming `repairPreviewCommand('aliases', {source})` per source instead of raw SQL. Test: `test/purge-aliases-6619.test.ts`.diff --git a/src/core/page-state/types.ts b/src/core/page-state/types.tsindex bd2b6a0f83..49e0fa3ff4 100644--- a/src/core/page-state/types.ts+++ b/src/core/page-state/types.ts@@ -78,15 +78,15 @@ export interface PageSnapshot {bodyGeneration?: number;}+/** A stale expected revision; stored on receipts (#7602 matches it for fact entity requests refused before the fix). */+export const PAGE_CHANGED_AFTER_READ_MESSAGE = 'The page changed after it was read. Read its current revision before retrying.';+/*** The three revision refusals. `expectedRevision` is the caller's precondition: a revision (stale → `revision_conflict`),* `null` for an asserted-absent page another writer created (→ `revision_conflict`), or `undefined` when the caller* supplied none and the page exists (#5385 → `revision_required`: the caller's omission, not a race, so a client* can read the page and resubmit instead of treating it as contention).*/-/** A stale expected revision; stored on receipts (#7602 matches it for fact entity requests refused before the fix). */-export const PAGE_CHANGED_AFTER_READ_MESSAGE = 'The page changed after it was read. Read its current revision before retrying.';-export class PageRevisionConflictError extends Error {readonly code: 'revision_conflict' | 'revision_required' | 'revision_backfill_pending';constructor(readonly expectedRevision: string | null | undefined, readonly currentRevision: string | null) {diff --git a/src/core/persistence/facts-maintenance.ts b/src/core/persistence/facts-maintenance.tsindex ea49e58692..521246e837 100644--- a/src/core/persistence/facts-maintenance.ts+++ b/src/core/persistence/facts-maintenance.ts@@ -327,7 +327,8 @@ export async function prepareManagedFactsSession(ctx: FactsBackstopCtx,* GBRA-130 A-36: codes one entity request ends on for reasons of its own page (changed, segmented, fence or content* refused). Automatic extraction reports them under `refused` and keeps the committed entities' counts. A file-check* refusal (`source_changed`) and a withdrawal refusal are not among them: they keep the batch refused, so the- * follow-up walks (#6048, #6506) re-admit their retained facts once the cause clears.+ * follow-up walks (#6048, #6506) re-admit their retained facts once the cause clears. An automatic extraction follows a+ * `revision_conflict` whose entity page moved (#7602, followEntityMoves) before it reports it.*/const ENTITY_REFUSAL_CODES: ReadonlySet<string> = new Set(['revision_conflict', 'invalid_fence', 'target_fence_malformed', 'page_body_segmented','invalid_params', 'page_identity_changed', 'page_not_found', 'request_too_large', 'payload_too_large']);diff --git a/src/core/repair/failed-writes.ts b/src/core/repair/failed-writes.tsindex efc81ac8f9..45b1c722c0 100644--- a/src/core/repair/failed-writes.ts+++ b/src/core/repair/failed-writes.ts@@ -57,6 +57,7 @@ import { PENDING_IMPORT, SYNC_HELD } from '../persistence/pending-import.ts';import { FACTS_ENTITY_MOVED_MESSAGES, isEntityMovedRefusal } from '../persistence/verb-errors.ts';import { MAX_FILE_REFUSAL_RETRIES } from '../persistence/facts-maintenance.ts';import { factsAbsorbWriteKey, factsRetryDeadCommand } from '../facts/dead-absorb-retry.ts';+import { readFactsBackstopJobPage } from '../persistence/effect-facts.ts';import { clearApprovedSet, loadApprovedSet, previewChangedError, previewHash, saveApprovedSet } from '../persistence/preview-approval.ts';import { resolveEntitySlugWithSource } from '../entities/resolve.ts';import { dispositionOf, PAGE_REMOVAL_OPERATIONS, supersessionOrderSql, supersessionSql } from '../persistence/terminal-classify.ts';@@ -91,7 +92,7 @@ async function factExtractions(engine: BrainEngine, sourceIds: string[], rows: Fif (!digests.length) return extractions;for (const row of await engine.executeRaw<ExtractionRequest>(`SELECT source_id, intent->>'inputDigest' AS input_digest, slug, state, error_code, error_message,intent->>'batchKey' AS batch_key FROM persistence_requests WHERE operation='extract_facts' AND source_id = ANY($1::text[])- AND intent->>'kind'='managed_facts_entity' AND intent->>'inputDigest' = ANY($2::text[])`, [sourceIds, digests])) {+ AND NOT compacted AND intent IS NOT NULL AND intent->>'kind'='managed_facts_entity' AND intent->>'inputDigest' = ANY($2::text[])`, [sourceIds, digests])) {const key = extractionKey(row.source_id, row.input_digest);extractions.set(key, [...(extractions.get(key) ?? []), row]);}@@ -118,30 +119,34 @@ function previewCommand(scope: RepairScope): string {* recovered by resetting that job once it is dead: its retry re-admits them at the entity page's current revision with* no model call, once the cause clears. Only an extraction whose open entity requests are all moved-entity refusals (the* entity walk) or all held-file refusals (the file walk) is re-admitted by one retry. A mixed extraction, a batch whose- * follow-ups were refused too, an entity page replaced since, a job that completed or is gone, a writer-guard refusal- * and a batch with no page write would replay the refusal or have no job to retry: those are extracted again.+ * follow-ups were refused too, an entity page replaced since, a job that completed or is gone, a source page written+ * again since the job's page write (the retry skips it as superseded), a writer-guard refusal and a batch with no page+ * write would replay the refusal or have no job to retry: those are extracted again. While the job still runs (a walk in+ * progress leaves follow-ups open), an entity page that is still the admitted one is listed `pending:`.*/async function factsRecovery(engine: BrainEngine, write: FailedWrite, extractions: Map<string, ExtractionRequest[]>): Promise<string> {const origin = (write.intent.origin as { slug?: string } | null | undefined)?.slug;const job = typeof write.intent.originalRequestId === 'string' ? write.intent.originalRequestId : null;const extractAgain = `re-extract: gbrain call extract_facts again on the current text of ${origin ?? 'the source turn'} with a new request_id (one model call)`;if (!job || write.state !== 'conflict') return extractAgain;+ const [page] = await engine.executeRaw<{ id: string }>('SELECT id::text FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [write.source_id, write.slug]);+ if (!page && write.page_id !== null) return `not recoverable: entity page ${write.slug} was deleted after these facts were admitted`;+ if ((page?.id ?? null) !== write.page_id) return `${extractAgain}; entity page ${write.slug} was replaced after these facts were admitted, so a job retry replays the refusal`;+ const key = factsAbsorbWriteKey(job);+ const [run] = await engine.executeRaw<{ status: string; data: Record<string, unknown> }>("SELECT status, data FROM minion_jobs WHERE name='facts-absorb' AND idempotency_key=$1 ORDER BY id DESC LIMIT 1", [key]);+ if (run && run.status !== 'dead' && run.status !== 'completed' && run.status !== 'cancelled') return `pending: its facts-absorb job (${key}) is ${run.status}; preview again once it ends`;const open = (extractions.get(extractionKey(write.source_id, write.intent.inputDigest)) ?? []).filter(row => row.state !== 'committed');if (!open.every(isEntityMovedRefusal) && !open.every(row => row.error_code === 'source_changed')) {return `${extractAgain}; its extraction also holds refusals one job retry does not re-admit together`;}- if (write.error_code === 'revision_conflict') {- const [page] = await engine.executeRaw<{ id: string }>('SELECT id::text FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [write.source_id, write.slug]);- if (!page && write.page_id !== null) return `not recoverable: entity page ${write.slug} was deleted after these facts were admitted`;- if ((page?.id ?? null) !== write.page_id) return `${extractAgain}; entity page ${write.slug} was replaced after these facts were admitted, so a job retry replays the refusal`;- if (new Set(open.filter(row => row.slug === write.slug).map(row => row.batch_key)).size > MAX_FILE_REFUSAL_RETRIES) {- return `${extractAgain}; its follow-up batches were refused as well, so a job retry replays the refusal`;- }+ if (new Set(open.filter(row => row.slug === write.slug).map(row => row.batch_key)).size > MAX_FILE_REFUSAL_RETRIES) {+ return `${extractAgain}; its follow-up batches were refused as well, so a job retry replays the refusal`;}- const key = factsAbsorbWriteKey(job);- const [run] = await engine.executeRaw<{ status: string }>("SELECT status FROM minion_jobs WHERE name='facts-absorb' AND idempotency_key=$1 ORDER BY id DESC LIMIT 1", [key]);- if (run && run.status !== 'dead' && run.status !== 'completed' && run.status !== 'cancelled') return `pending: its facts-absorb job (${key}) is ${run.status}; preview again once it ends`;if (run?.status !== 'dead') return `${extractAgain}; its facts-absorb job ${run ? `is ${run.status}` : 'is gone'}, so gbrain facts retry-dead has no dead job to reset`;+ const source = await readFactsBackstopJobPage(engine, run.data);+ if ('skipped' in source && (source.skipped === 'superseded' || source.skipped === 'page_missing')) {+ return `${extractAgain}; ${origin ?? 'the source page'} changed after the page write its facts-absorb job extracts, so a job retry skips it`;+ }return `recover: ${factsRetryDeadCommand('--reset', key)} (the next facts_drain pass retries the page's facts-absorb job, which re-admits the retained facts without a model call${write.error_code === 'source_changed' ? ' once the file check passes' : ''})`;}diff --git a/test/facts-absorb-stalled-receipt-7439.test.ts b/test/facts-absorb-stalled-receipt-7439.test.tsindex beb6c37499..498c684770 100644--- a/test/facts-absorb-stalled-receipt-7439.test.ts+++ b/test/facts-absorb-stalled-receipt-7439.test.ts@@ -80,7 +80,8 @@ for (const backend of testBackends()) {// why_new=#7602 covers revision_conflict listings only; seam=the job died is set by SQL (the reset needs a dead job); no_claim=does// not run the job retry itselfconst key = `facts-absorb:write:${write}`;- const job = await new MinionQueue(engine).add('facts-absorb', { slug: ORIGIN, sourceId: 'default', persistence_request_id: write }, { queue: 'default', idempotency_key: key });+ const origin = (await engine.readPageSnapshot(ORIGIN, { sourceId: 'default' }))!;+ const job = await new MinionQueue(engine).add('facts-absorb', { slug: ORIGIN, sourceId: 'default', persistence_request_id: write, page_id: origin.page.id, revision: origin.revision }, { queue: 'default', idempotency_key: key });await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE id=$1", [job.id]);const [listed, ...others] = await factsListing(engine);expect(others).toEqual([]);@@ -104,6 +105,26 @@ for (const backend of testBackends()) {expect(await factsListing(engine)).toEqual([]);}, { databaseUrl }), 120_000);+ // Value: protects=a held-file fact refusal whose entity page was deleted once the hold cleared is listed as not recoverable,+ // never the job retry (the file walk admits nothing for a deleted page, so the refusal replays until the job dies again);+ // fails_when=the page-identity checks apply to moved-entity refusals only; why_new=the row above keeps the page; seam=none;+ // no_claim=a purge compacts the refusal's receipt, so a purged-and-recreated page is covered by the #7602 replaced-page row+ test(`#7439 [${backend}]: a held-file fact refusal whose entity page was deleted after the hold cleared is not listed for the job retry`, () => managedBrain(async brain => {+ const { engine, ctx } = brain;+ const { file, pristine, write } = await refusedPageBatch(brain);+ const origin = (await engine.readPageSnapshot(ORIGIN, { sourceId: 'default' }))!;+ const job = await new MinionQueue(engine).add('facts-absorb', { slug: ORIGIN, sourceId: 'default', persistence_request_id: write, page_id: origin.page.id, revision: origin.revision }, { queue: 'default', idempotency_key: `facts-absorb:write:${write}` });+ await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE id=$1", [job.id]);+ writeFileSync(file, pristine);+ await withVerifiedLocalRegistration(engine, await registerLocalWriter(engine, 'cli'), async () => {+ const preview = await runReconcilePreview(engine, { source_id: 'default', slug: TARGET });+ expect((await runReconcileApply(engine, { source_id: 'default', slug: TARGET, preview: preview.preview, request_id: randomUUID() })).state).toBe('committed');+ });+ const snapshot = (await engine.readPageSnapshot(TARGET, { sourceId: 'default' }))!;+ await submitPageMutation(ctx, { operation: 'delete_page', params: { slug: TARGET, expected_revision: snapshot.revision, request_id: randomUUID() } });+ expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([`not recoverable: entity page ${TARGET} was deleted after these facts were admitted`]);+ }, { databaseUrl }), 120_000);+test(`#7439 [${backend}]: while the held file still differs, a retry keeps the refusal and admits nothing`, () => managedBrain(async brain => {const { engine } = brain;const { retry } = await refusedPageBatch(brain);diff --git a/test/facts-entity-moved-7602.test.ts b/test/facts-entity-moved-7602.test.tsindex 32137f2067..2b01ef0fcb 100644--- a/test/facts-entity-moved-7602.test.ts+++ b/test/facts-entity-moved-7602.test.ts@@ -22,16 +22,17 @@ import { join } from 'node:path';import { configureGateway, resetGateway, __setChatTransportForTests, type ChatResult } from '../src/core/ai/gateway.ts';import { FENCE_FACTS_INTENT } from '../src/core/cycle/extract-facts.ts';import type { FactsBackstopCtx } from '../src/core/facts/backstop.ts';-import { PAGE_CHANGED_AFTER_READ_MESSAGE } from '../src/core/page-state/types.ts';import { RECOVERY_VERSION } from '../src/core/markdown.ts';import { FACTS_DEFAULT_VISIBILITY_KEY } from '../src/core/facts/visibility.ts';import { makeFactsAbsorbHandler } from '../src/core/minions/handlers/facts-absorb.ts';import { jobRefusal } from '../src/core/minions/refusal.ts';+import { MinionQueue } from '../src/core/minions/queue.ts';import type { MinionJobContext } from '../src/core/minions/types.ts';import { prepareManagedFactsSession, publishManagedFacts, resumeManagedFacts } from '../src/core/persistence/facts-maintenance.ts';import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';import { maintenancePreflight, submitDatabaseMaintenanceIntent } from '../src/core/persistence/prepared-maintenance.ts';import { declarePersistenceProtocol } from '../src/core/persistence/protocol.ts';+import { FACTS_ENTITY_MOVED_MESSAGES } from '../src/core/persistence/verb-errors.ts';import { writeGitHold } from '../src/core/persistence/sync-holds.ts';import { resolveRepairScope } from '../src/core/repair/core.ts';import { repairRunner } from '../src/core/repair/registry.ts';@@ -103,6 +104,18 @@ const entityRequests = (engine: ManagedBrain['engine']) => engine.executeRaw<{ rconst claimB = (engine: ManagedBrain['engine']) => engine.executeRaw<{ fact: string; source_markdown_slug: string | null }>('SELECT fact, source_markdown_slug FROM facts WHERE fact=$1', [CLAIM_B]);+/**+ * The page write's facts-absorb job, by its idempotency key. The facts-backstop effect queues it asynchronously, so a test+ * that reads or sets its status adds it first (an add with the same key returns the job the effect queued, if any).+ */+async function factsAbsorbJob(engine: ManagedBrain['engine'], loser: Awaited<ReturnType<typeof extraction>>, status?: string) {+ const key = `facts-absorb:write:${loser.write}`;+ const job = await new MinionQueue(engine).add('facts-absorb', { slug: loser.slug, sourceId: 'default', source: 'mcp:put_page', persistence_request_id: loser.write,+ page_id: loser.snapshot.page.id, revision: loser.snapshot.revision }, { queue: 'default', idempotency_key: key });+ if (status) expect(await engine.executeRaw('UPDATE minion_jobs SET status=$2 WHERE id=$1 RETURNING id', [job.id, status])).toHaveLength(1);+ return key;+}+/** The `gbrain repair failed-writes` preview rows for the losing extraction's entity requests. */async function factsListing(engine: ManagedBrain['engine']) {const preview = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } }))@@ -191,6 +204,7 @@ for (const backend of testBackends()) {expect([retry.code, retry.retryable]).toEqual(['revision_conflict', false]);expect(await entityRequests(engine)).toEqual(requests);expect(await claimB(engine)).toEqual([]);+ await factsAbsorbJob(engine, loser, 'dead');const listing = await factsListing(engine);expect(listing).toHaveLength(4);for (const entry of listing) {@@ -201,6 +215,42 @@ for (const backend of testBackends()) {}}, { databaseUrl }), 120_000);+ // Value: protects=an extraction interrupted with exactly MAX_FILE_REFUSAL_RETRIES refused batches still has a follow-up left, so+ // repair lists the job reset and the reset's replay publishes; fails_when=the cap check counts the third open batch as exhausted+ // (`>=`); why_new=the rows above end with 2 or 4 open batches; seam=the run is interrupted by a throwing read before the third+ // follow-up; no_claim=the interruption is a thrown read, not a killed worker+ test(`#7602 [${backend}]: an extraction interrupted with three refused batches is still listed for the job reset, and the reset publishes`, () => managedBrain(async brain => {+ const { engine } = brain;+ const { move, loser } = await setup(brain, 3);+ const race = raceEntityReads(engine, 4, async n => { if (n === 3) throw new Error('interrupted before the third follow-up'); await move(n); });+ const interrupted = await publishManagedFacts(engine, await loser.session(), loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore);+ expect(interrupted.message).toBe('interrupted before the third follow-up');+ expect((await entityRequests(engine)).map(row => [row.state, row.error_code])).toEqual(Array(3).fill(['conflict', 'revision_conflict']));+ const key = await factsAbsorbJob(engine, loser, 'dead');+ expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual(Array(3).fill(`recover: gbrain facts retry-dead --reset '${key}'`));+ expect((await resumeManagedFacts(engine, await loser.session()))?.inserted).toBe(1);+ expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);+ expect(await factsListing(engine)).toEqual([]);+ }, { databaseUrl }), 120_000);++ // Value: protects=a refusal whose source page was written again after its job died is listed for re-extraction, not the job+ // reset (the retry skips a superseded page write); fails_when=factsRecovery offers recover: without the job's own page check;+ // why_new=the rows above keep the source page; seam=the job's status is set by SQL; no_claim=does not run the reset job+ test(`#7602 [${backend}]: a refusal whose source page was written again after its job died is listed for re-extraction`, () => managedBrain(async brain => {+ const { engine, ctx } = brain;+ const { move, loser } = await setup(brain, 1);+ const before = await loser.session();+ before.entityMoveRetry = false;+ const race = raceEntityReads(engine, 1, move);+ expect((await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore)).code).toBe('revision_conflict');+ const key = await factsAbsorbJob(engine, loser, 'dead');+ expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual([`recover: gbrain facts retry-dead --reset '${key}'`]);+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug: loser.slug, request_id: randomUUID(), expected_revision: loser.snapshot.revision,+ content: `---\ntype: meeting\ntitle: Example\n---\n${BODY} (rewritten)\n` } });+ expect((await engine.readPageSnapshot(loser.slug, { sourceId: 'default' }))!.revision).not.toBe(loser.snapshot.revision);+ expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([`re-extract: gbrain call extract_facts again on the current text of ${loser.slug} with a new request_id (one model call); ${loser.slug} changed after the page write its facts-absorb job extracts, so a job retry skips it`]);+ }, { databaseUrl }), 120_000);+// Value: protects=repair failed-writes offers the job reset only for a dead facts-absorb job, and the reset's replay publishes the// retained facts; fails_when=a waiting or completed job is listed with retry-dead (a command that finds no dead job), or the replay// does not publish; why_new=the rows above never list a recoverable refusal; seam=the job's status is set by SQL; no_claim=does not@@ -213,7 +263,7 @@ for (const backend of testBackends()) {const race = raceEntityReads(engine, 1, move);const refused = await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore);expect(refused.code).toBe('revision_conflict');- const key = `facts-absorb:write:${loser.write}`;+ const key = await factsAbsorbJob(engine, loser);const jobStatus = (status: string) => engine.executeRaw("UPDATE minion_jobs SET status=$2 WHERE name='facts-absorb' AND idempotency_key=$1 RETURNING id", [key, status]);expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([`pending: its facts-absorb job (${key}) is waiting; preview again once it ends`]);expect(await jobStatus('completed')).toHaveLength(1);@@ -283,28 +333,37 @@ for (const backend of testBackends()) {'re-extract: gbrain call extract_facts again on the current text of the source turn with a new request_id (one model call)']]);}, { databaseUrl }), 120_000);- // Value: protects=a move during preparation refused before the fix (stored with the page layer's stale-revision message) is- // listed with the job reset and re-admitted by it; fails_when=that message is missing from FACTS_ENTITY_MOVED_MESSAGES, so the- // receipt is never listed and a reset replays it; why_new=the row above stores the admission-time message; seam=the stored message- // is set by SQL to the pre-fix text; no_claim=does not reproduce the pre-fix preparation path itself- test(`#7602 [${backend}]: a pre-fix refusal stored with the stale-revision message is listed for the job reset and re-admitted`, () => managedBrain(async brain => {- const { engine } = brain;- const { move, loser } = await setup(brain, 1);- const before = await loser.session();- before.entityMoveRetry = false;- const race = raceEntityReads(engine, 1, move);- expect((await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore)).code).toBe('revision_conflict');- await engine.transaction(async tx => {- await declarePersistenceProtocol(tx);- expect(await tx.executeRaw("UPDATE persistence_requests SET error_message=$2 WHERE operation='extract_facts' AND slug=$1 AND state='conflict' RETURNING id", [TARGET, PAGE_CHANGED_AFTER_READ_MESSAGE])).toHaveLength(1);- });- const key = `facts-absorb:write:${loser.write}`;- expect(await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE name='facts-absorb' AND idempotency_key=$1 RETURNING id", [key])).toHaveLength(1);- expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual([`recover: gbrain facts retry-dead --reset '${key}'`]);- expect((await resumeManagedFacts(engine, await loser.session()))?.inserted).toBe(1);- expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);- expect(await factsListing(engine)).toEqual([]);- }, { databaseUrl }), 120_000);+ // Value: protects=a refusal stored with any moved-entity message (the admission, preparation, deduplication and coordinator kinds,+ // and the page layer's stale-revision message stored before the fix) is listed with the job reset and re-admitted by it, and+ // repair --apply leaves it alone; fails_when=a message is missing from FACTS_ENTITY_MOVED_MESSAGES, so the receipt is never listed+ // and a reset replays it, or apply replays a fact request; why_new=the row above stores the admission-time message only;+ // seam=the stored message is set by SQL; no_claim=does not reproduce each message's own refusal path+ for (const message of FACTS_ENTITY_MOVED_MESSAGES) {+ test(`#7602 [${backend}]: a refusal stored with '${message}' is listed for the job reset and re-admitted`, () => managedBrain(async brain => {+ const { engine } = brain;+ const { move, loser } = await setup(brain, 1);+ const before = await loser.session();+ before.entityMoveRetry = false;+ const race = raceEntityReads(engine, 1, move);+ expect((await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore)).code).toBe('revision_conflict');+ await engine.transaction(async tx => {+ await declarePersistenceProtocol(tx);+ expect(await tx.executeRaw("UPDATE persistence_requests SET error_message=$2 WHERE operation='extract_facts' AND slug=$1 AND state='conflict' RETURNING id", [TARGET, message])).toHaveLength(1);+ });+ const key = await factsAbsorbJob(engine, loser, 'dead');+ expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual([`recover: gbrain facts retry-dead --reset '${key}'`]);+ const requests = await entityRequests(engine);+ const scope = await resolveRepairScope(engine, 'default');+ const preview = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } })).run('failed-writes', scope, { explicit: true, sourceFlag: 'default' });+ const applied = await (await repairRunner(engine, { apply: true, logger: { info() {}, warn() {}, error() {} } })).run('failed-writes', scope,+ { explicit: true, sourceFlag: 'default', ...(preview.apply_command ? { expect: preview.apply_command.split('--expect ')[1] } : {}) });+ expect(applied.outcomes?.replayed ?? 0).toBe(0);+ expect(await entityRequests(engine)).toEqual(requests);+ expect((await resumeManagedFacts(engine, await loser.session()))?.inserted).toBe(1);+ expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);+ expect(await factsListing(engine)).toEqual([]);+ }, { databaseUrl }), 120_000);+ }// Value: protects=a re-admitted fact is never wider than the current default visibility, and keeps world under a world default;// fails_when=readmitMovedFacts re-admits the retained facts without narrowing them (or narrows unconditionally);@@ -398,7 +457,13 @@ for (const backend of testBackends()) {expect(race.moves()).toBe(1);expect(refused.code).toBe('source_changed');expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);- expect(await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE name='facts-absorb' AND idempotency_key=$1 RETURNING id", [`facts-absorb:write:${loser.write}`])).toHaveLength(1);+ const key = await factsAbsorbJob(engine, loser, 'waiting');+ expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([]);+ const pending = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } }))+ .run('failed-writes', await resolveRepairScope(engine, 'default'), { explicit: true, sourceFlag: 'default' });+ expect((pending.listing ?? []).filter(entry => entry.item.includes(' extract_facts')).map(entry => entry.detail))+ .toEqual([`pending: its facts-absorb job (${key}) is waiting; preview again once it ends`]);+ await factsAbsorbJob(engine, loser, 'dead');const preview = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } })).run('failed-writes', await resolveRepairScope(engine, 'default'), { explicit: true, sourceFlag: 'default' });expect((preview.listing ?? []).filter(entry => entry.item.includes(' extract_facts')).map(entry => [entry.item.split(' ')[0], entry.detail])).toEqual([@@ -426,6 +491,7 @@ for (const backend of testBackends()) {expect(refused.code).toBe('source_changed');expect((await entityRequests(engine)).map(row => [row.state, row.error_code])).toEqual([['conflict', 'revision_conflict'], ['conflict', 'source_changed']]);expect(await claimB(engine)).toEqual([]);+ await factsAbsorbJob(engine, loser, 'dead');expect((await factsListing(engine)).map(entry => entry.detail)).toEqual(Array(2).fill(`re-extract: gbrain call extract_facts again on the current text of ${loser.slug} with a new request_id (one model call); its extraction also holds refusals one job retry does not re-admit together`));}, { databaseUrl }), 120_000);