Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(facts): held-file refusals get the page checks; pending before mixed; superseded source page is re-extract; deterministic job rows (#7602 review)

garrytan52dae8474bf5
diff --git a/docs/architecture/key-files/core-utilities-2.md b/docs/architecture/key-files/core-utilities-2.md
index b6a22897ed..5c4a6e49af 100644
--- a/docs/architecture/key-files/core-utilities-2.md
+++ b/docs/architecture/key-files/core-utilities-2.md
@@ -13,7 +13,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/repair/captured-facts.ts` — `gbrain repair captured-facts`, explicit-only and preview-bound. `classifyCapturedFacts` (read-only; also the `captured_facts_active` count) takes active capture-lane facts and classes each by its session (`captureSessionId`: `sweep:corpus:<file>` resolves through `corpusFileSessionId`): `evidenced` when a gbrain claude-cli scratch project holds the session's harness transcript or its corpus file sits in `<corpus>.quarantine/`; `ambiguous` (paste candidate) when a retained corpus file of the session holds a paste block and at least 60% of the claim's content words occur there only inside paste blocks; `excluded:legitimate_duplicate` when the claim and entity also have an active row that is not proven suspect. Sessions with no transcript and no corpus file are counted `unclassifiable`. The preview hashes every candidate with the selection (`--include-ambiguous`) and saves one approved item per page through `preview-approval.ts`; the apply reclassifies each fact, expires database-only rows through one database-only `managed_maintenance_expire_captured_facts` request per page on a managed brain (owner-checked by `maintenancePreflight`; `prepareCapturedFactsExpiry`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain, and strikes fenced rows in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection expires them and a later page write cannot reactivate them. No withdrawal is recorded, so `remember` can save the same claim again. Pinned by `test/repair-captured-facts.test.ts` (+ Postgres arm).
- `src/core/repair/redacted-facts.ts` — `gbrain repair redacted-facts` (#7125), explicit-only and preview-bound. `classifyRedactedFacts` (read-only) takes active facts whose text holds a redaction marker and runs `restoreNeutralized` (`think/sanitize.ts`, with the pre-fix `LEGACY_EVAL_SHELL` shape) against the page the fact's context names and the page holding it (`pageTexts`: an inline page's body + timeline, a segmented page's body through `readBodyRange` up to 16 MiB, then up to 50 `page_versions` bodies): `evidenced` when every proving text proves the same restoration, `ambiguous` (`conflicting_sources`) when a text holds two spans around the words or texts disagree, `unproven` (`no_source_page` / `not_in_source`) otherwise. A proved fenced row on a segmented page is `segmented` (`segmented_page`): listed with its restoration, never rewritten (`restoreFencedRows` reads and publishes inline bodies only). The preview hashes every candidate with its restoration and saves one approved item per page; the apply reclassifies each fact, rewrites database-only rows through one database-only `managed_maintenance_restore_redacted_facts` request per page on a managed brain (`prepareRedactedFactsRestore`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain (exact-text compare-and-set; context gets `restored: redacted-facts repair`), and rewrites fenced rows' claims in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection replaces the row. Pinned by `test/repair-redacted-facts.test.ts` (+ Postgres arm).
- `src/core/repair/loop-facts.ts` — `gbrain repair loop-facts` (#5869), explicit-only and preview-bound (`--apply --expect <hash>`): candidates are `loopFactDrift` rows (closed `done`/`dropped` loops whose commitment fact is active, in the loop's source and shared with no open loop); each item is rechecked (`changed_since_preview`) and retired through `retireLoopFact`, the same coordinated strike-plus-expire publication `loops_close` uses.
-- `src/core/repair/failed-writes.ts` — `gbrain repair failed-writes` (#5983), explicit-only and preview-bound: candidates are failed, uncompacted `persistence_requests` refused by the managed writer guard (`writer_coordinator_required`, or `storage_error` "Publication failed (P0001)"). Caller writes (`put_page` without an internal `kind`, `add_timeline_entry`, `remember`) are classified `replay` / `already_written` / `duplicate` / `superseded` / `unpinned_target` by later requests on the same page and by a committed replay request id, and `file_database_drift` (W4.11) when its last replay attempt was refused `source_changed` and the page has not been written since (`lastFailedAttempt`; the detail and an apply's `file_database_drift` outcome name `gbrain sources reconcile <source> <slug> --preview`; `test/repair-failed-writes-drift-w411.test.ts`); owner-produced writes are `producer_owned` residuals with the producer command (a failed `managed_file_repair` names both `gbrain repair frontmatter` and `gbrain repair fences`). #7602: fact rows (moved-entity refusals included) use `factsRecovery`: `recover:` (retry-dead reset) only while the page write's facts-absorb job is dead and the entity page is the one admitted, `pending:` while that job runs, `not recoverable:` for a deleted page, and `re-extract:` otherwise (replaced page, completed or missing job, exhausted follow-ups, an extraction whose open entity requests are not all moved-entity or all held-file refusals, a guard refusal). Only the preview reads fact rows (`failedWrites(..., { facts: true })`); the sync-time remember replay and apply-by-id keep the guard/drift query. The approved set records each target page's revision. Apply resumes an admitted replay attempt, otherwise reclassifies and rechecks the revision, re-runs `authorizeStoredRequest` on the failed row, and resubmits the retained intent on the original trust lane (`laneContext`: remote, take holders, delegated prefixes) through `submitPageMutation` / `submitRememberMutation`, with `put_page` bound to the previewed revision and `remember` pinned to its resolved subject, under a request id derived from the failed request id and attempt (`repair-failed-writes-5983.test.ts`). #5994: `laneContext` sets `ctx.replayAuthority` to the stored authority, so admission (`submissionAuthority` → `replayedAuthority` in `persistence/authority.ts`, and `requestPrincipalForContext`) reuses its principal, delegation, scopes and autoLinkTrusted for the same source incarnation and re-authorizes it live; a restricted write gets back its recorded job (`delegatedJobId`, or the id in a single legacy `wiki/agents/<id>/*` prefix, replayed on the legacy sandboxed path when its original was), and one with neither is accepted by `enforceSubagentSlugFence` (`ops/context.ts`) only through that marker with the exact non-empty stored allow-list. The fence still fails closed for every live dispatch. A put_page replay passes the stored `timeline_section` (#5969). The superseded / already-written / duplicate rule lives in `persistence/terminal-classify.ts`, shared with the #6650 terminal-failure readers.
+- `src/core/repair/failed-writes.ts` — `gbrain repair failed-writes` (#5983), explicit-only and preview-bound: candidates are failed, uncompacted `persistence_requests` refused by the managed writer guard (`writer_coordinator_required`, or `storage_error` "Publication failed (P0001)"). Caller writes (`put_page` without an internal `kind`, `add_timeline_entry`, `remember`) are classified `replay` / `already_written` / `duplicate` / `superseded` / `unpinned_target` by later requests on the same page and by a committed replay request id, and `file_database_drift` (W4.11) when its last replay attempt was refused `source_changed` and the page has not been written since (`lastFailedAttempt`; the detail and an apply's `file_database_drift` outcome name `gbrain sources reconcile <source> <slug> --preview`; `test/repair-failed-writes-drift-w411.test.ts`); owner-produced writes are `producer_owned` residuals with the producer command (a failed `managed_file_repair` names both `gbrain repair frontmatter` and `gbrain repair fences`). #7602: fact rows (moved-entity refusals included) use `factsRecovery`: `recover:` (retry-dead reset) only while the page write's facts-absorb job is dead and the entity page is the one admitted, `pending:` while that job runs, `not recoverable:` for a deleted page, and `re-extract:` otherwise (replaced page, completed or missing job, exhausted follow-ups, a source page written again since the job's page write, which the job's own `readFactsBackstopJobPage` check skips as superseded, an extraction whose open entity requests are not all moved-entity or all held-file refusals, a guard refusal). Only the preview reads fact rows (`failedWrites(..., { facts: true })`); the sync-time remember replay and apply-by-id keep the guard/drift query. The approved set records each target page's revision. Apply resumes an admitted replay attempt, otherwise reclassifies and rechecks the revision, re-runs `authorizeStoredRequest` on the failed row, and resubmits the retained intent on the original trust lane (`laneContext`: remote, take holders, delegated prefixes) through `submitPageMutation` / `submitRememberMutation`, with `put_page` bound to the previewed revision and `remember` pinned to its resolved subject, under a request id derived from the failed request id and attempt (`repair-failed-writes-5983.test.ts`). #5994: `laneContext` sets `ctx.replayAuthority` to the stored authority, so admission (`submissionAuthority` → `replayedAuthority` in `persistence/authority.ts`, and `requestPrincipalForContext`) reuses its principal, delegation, scopes and autoLinkTrusted for the same source incarnation and re-authorizes it live; a restricted write gets back its recorded job (`delegatedJobId`, or the id in a single legacy `wiki/agents/<id>/*` prefix, replayed on the legacy sandboxed path when its original was), and one with neither is accepted by `enforceSubagentSlugFence` (`ops/context.ts`) only through that marker with the exact non-empty stored allow-list. The fence still fails closed for every live dispatch. A put_page replay passes the stored `timeline_section` (#5969). The superseded / already-written / duplicate rule lives in `persistence/terminal-classify.ts`, shared with the #6650 terminal-failure readers.
- `src/core/repair/ontology-facts.ts` — `gbrain repair ontology-facts` (#6264), explicit-only and preview-bound (`--apply --expect <hash>`). `classifyOntologyFacts` (read-only; also doctor `ontology_facts_fenced`) takes ontology rows (`dimension IS NOT NULL`) carrying the pre-fix fence step's signature, a `row_num` or a `source_markdown_slug` that differs from `source` (`mergeOntologyFact` writes them equal and never sets `row_num`), and classes each `fenced` or `retired`, or excluded as `withdrawn` (`fact_withdrawals`), `consolidated` or `duplicate` (restoring would collide with the ontology dedup key). The apply sets `row_num` NULL, `source_markdown_slug = source` and `expired_at` NULL in one statement that rechecks row number, page, expiry and the dedup key (`changed_since_preview` otherwise), through `coordinatedDatabaseWrite('ontology_propose')` on the entity and page keys on a managed brain or one maintenance transaction otherwise; the withdrawal trigger re-expires a claim forgotten after the preview (`withdrawn`). Page text is never rewritten. Tests: `test/ontology-fence-sweep.test.ts`.
- `src/core/repair/timeline-comments.ts` — `gbrain repair timeline-comments` (#6184), explicit-only. The plan lists pages with timeline rows whose source, summary or detail carries `<!--`/`-->` or whose body has a materialized bullet carrying markup (`dropCommentBullets`), with residuals `comment_only_rows`, `comment_bearing_rows`, `pages_with_comment_bullets`. Apply: one revision-bound `put_page` per page dropping those generated bullets (user-written bullets are left), then one transaction under the page lock (`withCoordinatedWrite` with maintenance attribution on managed brains) that deletes comment-only rows and rewrites the rest to their stripped tuple, or deletes them when the page already has it. Idempotent. `timeline.ts` warns `timeline_comment_markup` naming this kind while such rows exist. Rows on a segmented page are not planned (the apply rewrites a whole body): residuals `segmented_pages` / `segmented_rows` and a warning naming them.
- `src/core/repair/aliases.ts` — `gbrain repair aliases` (#6619), explicit-only, projection publication, wired to doctor `dangling_aliases`. One item per (source, target slug) whose alias rows (`page_aliases.slug`, `slug_aliases.canonical_slug`) point at a slug with no page row at all; apply rechecks that under the page key lock and deletes both tables' rows for it, inside `withCoordinatedWrite` on a managed brain (both tables carry the managed writer guard). Aliases of soft-deleted pages are the `soft_deleted_target` residual; rename redirects (`alias_slug` = the missing slug) are never touched. Exports `ORPHANED_ALIAS_COUNTS_SQL` / `SOFT_DELETED_ALIAS_COUNT_SQL`, which `checkDanglingAliases` (`src/core/onboard/checks.ts`) reports apart, naming `repairPreviewCommand('aliases', {source})` per source instead of raw SQL. Test: `test/purge-aliases-6619.test.ts`.
diff --git a/src/core/page-state/types.ts b/src/core/page-state/types.ts
index bd2b6a0f83..49e0fa3ff4 100644
--- a/src/core/page-state/types.ts
+++ b/src/core/page-state/types.ts
@@ -78,15 +78,15 @@ export interface PageSnapshot {
bodyGeneration?: number;
}
+/** A stale expected revision; stored on receipts (#7602 matches it for fact entity requests refused before the fix). */
+export const PAGE_CHANGED_AFTER_READ_MESSAGE = 'The page changed after it was read. Read its current revision before retrying.';
+
/**
* The three revision refusals. `expectedRevision` is the caller's precondition: a revision (stale → `revision_conflict`),
* `null` for an asserted-absent page another writer created (→ `revision_conflict`), or `undefined` when the caller
* supplied none and the page exists (#5385 → `revision_required`: the caller's omission, not a race, so a client
* can read the page and resubmit instead of treating it as contention).
*/
-/** A stale expected revision; stored on receipts (#7602 matches it for fact entity requests refused before the fix). */
-export const PAGE_CHANGED_AFTER_READ_MESSAGE = 'The page changed after it was read. Read its current revision before retrying.';
-
export class PageRevisionConflictError extends Error {
readonly code: 'revision_conflict' | 'revision_required' | 'revision_backfill_pending';
constructor(readonly expectedRevision: string | null | undefined, readonly currentRevision: string | null) {
diff --git a/src/core/persistence/facts-maintenance.ts b/src/core/persistence/facts-maintenance.ts
index ea49e58692..521246e837 100644
--- a/src/core/persistence/facts-maintenance.ts
+++ b/src/core/persistence/facts-maintenance.ts
@@ -327,7 +327,8 @@ export async function prepareManagedFactsSession(ctx: FactsBackstopCtx,
* GBRA-130 A-36: codes one entity request ends on for reasons of its own page (changed, segmented, fence or content
* refused). Automatic extraction reports them under `refused` and keeps the committed entities' counts. A file-check
* refusal (`source_changed`) and a withdrawal refusal are not among them: they keep the batch refused, so the
- * follow-up walks (#6048, #6506) re-admit their retained facts once the cause clears.
+ * follow-up walks (#6048, #6506) re-admit their retained facts once the cause clears. An automatic extraction follows a
+ * `revision_conflict` whose entity page moved (#7602, followEntityMoves) before it reports it.
*/
const ENTITY_REFUSAL_CODES: ReadonlySet<string> = new Set(['revision_conflict', 'invalid_fence', 'target_fence_malformed', 'page_body_segmented',
'invalid_params', 'page_identity_changed', 'page_not_found', 'request_too_large', 'payload_too_large']);
diff --git a/src/core/repair/failed-writes.ts b/src/core/repair/failed-writes.ts
index efc81ac8f9..45b1c722c0 100644
--- a/src/core/repair/failed-writes.ts
+++ b/src/core/repair/failed-writes.ts
@@ -57,6 +57,7 @@ import { PENDING_IMPORT, SYNC_HELD } from '../persistence/pending-import.ts';
import { FACTS_ENTITY_MOVED_MESSAGES, isEntityMovedRefusal } from '../persistence/verb-errors.ts';
import { MAX_FILE_REFUSAL_RETRIES } from '../persistence/facts-maintenance.ts';
import { factsAbsorbWriteKey, factsRetryDeadCommand } from '../facts/dead-absorb-retry.ts';
+import { readFactsBackstopJobPage } from '../persistence/effect-facts.ts';
import { clearApprovedSet, loadApprovedSet, previewChangedError, previewHash, saveApprovedSet } from '../persistence/preview-approval.ts';
import { resolveEntitySlugWithSource } from '../entities/resolve.ts';
import { dispositionOf, PAGE_REMOVAL_OPERATIONS, supersessionOrderSql, supersessionSql } from '../persistence/terminal-classify.ts';
@@ -91,7 +92,7 @@ async function factExtractions(engine: BrainEngine, sourceIds: string[], rows: F
if (!digests.length) return extractions;
for (const row of await engine.executeRaw<ExtractionRequest>(`SELECT source_id, intent->>'inputDigest' AS input_digest, slug, state, error_code, error_message,
intent->>'batchKey' AS batch_key FROM persistence_requests WHERE operation='extract_facts' AND source_id = ANY($1::text[])
- AND intent->>'kind'='managed_facts_entity' AND intent->>'inputDigest' = ANY($2::text[])`, [sourceIds, digests])) {
+ AND NOT compacted AND intent IS NOT NULL AND intent->>'kind'='managed_facts_entity' AND intent->>'inputDigest' = ANY($2::text[])`, [sourceIds, digests])) {
const key = extractionKey(row.source_id, row.input_digest);
extractions.set(key, [...(extractions.get(key) ?? []), row]);
}
@@ -118,30 +119,34 @@ function previewCommand(scope: RepairScope): string {
* recovered by resetting that job once it is dead: its retry re-admits them at the entity page's current revision with
* no model call, once the cause clears. Only an extraction whose open entity requests are all moved-entity refusals (the
* entity walk) or all held-file refusals (the file walk) is re-admitted by one retry. A mixed extraction, a batch whose
- * follow-ups were refused too, an entity page replaced since, a job that completed or is gone, a writer-guard refusal
- * and a batch with no page write would replay the refusal or have no job to retry: those are extracted again.
+ * follow-ups were refused too, an entity page replaced since, a job that completed or is gone, a source page written
+ * again since the job's page write (the retry skips it as superseded), a writer-guard refusal and a batch with no page
+ * write would replay the refusal or have no job to retry: those are extracted again. While the job still runs (a walk in
+ * progress leaves follow-ups open), an entity page that is still the admitted one is listed `pending:`.
*/
async function factsRecovery(engine: BrainEngine, write: FailedWrite, extractions: Map<string, ExtractionRequest[]>): Promise<string> {
const origin = (write.intent.origin as { slug?: string } | null | undefined)?.slug;
const job = typeof write.intent.originalRequestId === 'string' ? write.intent.originalRequestId : null;
const extractAgain = `re-extract: gbrain call extract_facts again on the current text of ${origin ?? 'the source turn'} with a new request_id (one model call)`;
if (!job || write.state !== 'conflict') return extractAgain;
+ const [page] = await engine.executeRaw<{ id: string }>('SELECT id::text FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [write.source_id, write.slug]);
+ if (!page && write.page_id !== null) return `not recoverable: entity page ${write.slug} was deleted after these facts were admitted`;
+ if ((page?.id ?? null) !== write.page_id) return `${extractAgain}; entity page ${write.slug} was replaced after these facts were admitted, so a job retry replays the refusal`;
+ const key = factsAbsorbWriteKey(job);
+ const [run] = await engine.executeRaw<{ status: string; data: Record<string, unknown> }>("SELECT status, data FROM minion_jobs WHERE name='facts-absorb' AND idempotency_key=$1 ORDER BY id DESC LIMIT 1", [key]);
+ if (run && run.status !== 'dead' && run.status !== 'completed' && run.status !== 'cancelled') return `pending: its facts-absorb job (${key}) is ${run.status}; preview again once it ends`;
const open = (extractions.get(extractionKey(write.source_id, write.intent.inputDigest)) ?? []).filter(row => row.state !== 'committed');
if (!open.every(isEntityMovedRefusal) && !open.every(row => row.error_code === 'source_changed')) {
return `${extractAgain}; its extraction also holds refusals one job retry does not re-admit together`;
}
- if (write.error_code === 'revision_conflict') {
- const [page] = await engine.executeRaw<{ id: string }>('SELECT id::text FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [write.source_id, write.slug]);
- if (!page && write.page_id !== null) return `not recoverable: entity page ${write.slug} was deleted after these facts were admitted`;
- if ((page?.id ?? null) !== write.page_id) return `${extractAgain}; entity page ${write.slug} was replaced after these facts were admitted, so a job retry replays the refusal`;
- if (new Set(open.filter(row => row.slug === write.slug).map(row => row.batch_key)).size > MAX_FILE_REFUSAL_RETRIES) {
- return `${extractAgain}; its follow-up batches were refused as well, so a job retry replays the refusal`;
- }
+ if (new Set(open.filter(row => row.slug === write.slug).map(row => row.batch_key)).size > MAX_FILE_REFUSAL_RETRIES) {
+ return `${extractAgain}; its follow-up batches were refused as well, so a job retry replays the refusal`;
}
- const key = factsAbsorbWriteKey(job);
- const [run] = await engine.executeRaw<{ status: string }>("SELECT status FROM minion_jobs WHERE name='facts-absorb' AND idempotency_key=$1 ORDER BY id DESC LIMIT 1", [key]);
- if (run && run.status !== 'dead' && run.status !== 'completed' && run.status !== 'cancelled') return `pending: its facts-absorb job (${key}) is ${run.status}; preview again once it ends`;
if (run?.status !== 'dead') return `${extractAgain}; its facts-absorb job ${run ? `is ${run.status}` : 'is gone'}, so gbrain facts retry-dead has no dead job to reset`;
+ const source = await readFactsBackstopJobPage(engine, run.data);
+ if ('skipped' in source && (source.skipped === 'superseded' || source.skipped === 'page_missing')) {
+ return `${extractAgain}; ${origin ?? 'the source page'} changed after the page write its facts-absorb job extracts, so a job retry skips it`;
+ }
return `recover: ${factsRetryDeadCommand('--reset', key)} (the next facts_drain pass retries the page's facts-absorb job, which re-admits the retained facts without a model call${write.error_code === 'source_changed' ? ' once the file check passes' : ''})`;
}
diff --git a/test/facts-absorb-stalled-receipt-7439.test.ts b/test/facts-absorb-stalled-receipt-7439.test.ts
index beb6c37499..498c684770 100644
--- a/test/facts-absorb-stalled-receipt-7439.test.ts
+++ b/test/facts-absorb-stalled-receipt-7439.test.ts
@@ -80,7 +80,8 @@ for (const backend of testBackends()) {
// why_new=#7602 covers revision_conflict listings only; seam=the job died is set by SQL (the reset needs a dead job); no_claim=does
// not run the job retry itself
const key = `facts-absorb:write:${write}`;
- const job = await new MinionQueue(engine).add('facts-absorb', { slug: ORIGIN, sourceId: 'default', persistence_request_id: write }, { queue: 'default', idempotency_key: key });
+ const origin = (await engine.readPageSnapshot(ORIGIN, { sourceId: 'default' }))!;
+ const job = await new MinionQueue(engine).add('facts-absorb', { slug: ORIGIN, sourceId: 'default', persistence_request_id: write, page_id: origin.page.id, revision: origin.revision }, { queue: 'default', idempotency_key: key });
await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE id=$1", [job.id]);
const [listed, ...others] = await factsListing(engine);
expect(others).toEqual([]);
@@ -104,6 +105,26 @@ for (const backend of testBackends()) {
expect(await factsListing(engine)).toEqual([]);
}, { databaseUrl }), 120_000);
+ // Value: protects=a held-file fact refusal whose entity page was deleted once the hold cleared is listed as not recoverable,
+ // never the job retry (the file walk admits nothing for a deleted page, so the refusal replays until the job dies again);
+ // fails_when=the page-identity checks apply to moved-entity refusals only; why_new=the row above keeps the page; seam=none;
+ // no_claim=a purge compacts the refusal's receipt, so a purged-and-recreated page is covered by the #7602 replaced-page row
+ test(`#7439 [${backend}]: a held-file fact refusal whose entity page was deleted after the hold cleared is not listed for the job retry`, () => managedBrain(async brain => {
+ const { engine, ctx } = brain;
+ const { file, pristine, write } = await refusedPageBatch(brain);
+ const origin = (await engine.readPageSnapshot(ORIGIN, { sourceId: 'default' }))!;
+ const job = await new MinionQueue(engine).add('facts-absorb', { slug: ORIGIN, sourceId: 'default', persistence_request_id: write, page_id: origin.page.id, revision: origin.revision }, { queue: 'default', idempotency_key: `facts-absorb:write:${write}` });
+ await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE id=$1", [job.id]);
+ writeFileSync(file, pristine);
+ await withVerifiedLocalRegistration(engine, await registerLocalWriter(engine, 'cli'), async () => {
+ const preview = await runReconcilePreview(engine, { source_id: 'default', slug: TARGET });
+ expect((await runReconcileApply(engine, { source_id: 'default', slug: TARGET, preview: preview.preview, request_id: randomUUID() })).state).toBe('committed');
+ });
+ const snapshot = (await engine.readPageSnapshot(TARGET, { sourceId: 'default' }))!;
+ await submitPageMutation(ctx, { operation: 'delete_page', params: { slug: TARGET, expected_revision: snapshot.revision, request_id: randomUUID() } });
+ expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([`not recoverable: entity page ${TARGET} was deleted after these facts were admitted`]);
+ }, { databaseUrl }), 120_000);
+
test(`#7439 [${backend}]: while the held file still differs, a retry keeps the refusal and admits nothing`, () => managedBrain(async brain => {
const { engine } = brain;
const { retry } = await refusedPageBatch(brain);
diff --git a/test/facts-entity-moved-7602.test.ts b/test/facts-entity-moved-7602.test.ts
index 32137f2067..2b01ef0fcb 100644
--- a/test/facts-entity-moved-7602.test.ts
+++ b/test/facts-entity-moved-7602.test.ts
@@ -22,16 +22,17 @@ import { join } from 'node:path';
import { configureGateway, resetGateway, __setChatTransportForTests, type ChatResult } from '../src/core/ai/gateway.ts';
import { FENCE_FACTS_INTENT } from '../src/core/cycle/extract-facts.ts';
import type { FactsBackstopCtx } from '../src/core/facts/backstop.ts';
-import { PAGE_CHANGED_AFTER_READ_MESSAGE } from '../src/core/page-state/types.ts';
import { RECOVERY_VERSION } from '../src/core/markdown.ts';
import { FACTS_DEFAULT_VISIBILITY_KEY } from '../src/core/facts/visibility.ts';
import { makeFactsAbsorbHandler } from '../src/core/minions/handlers/facts-absorb.ts';
import { jobRefusal } from '../src/core/minions/refusal.ts';
+import { MinionQueue } from '../src/core/minions/queue.ts';
import type { MinionJobContext } from '../src/core/minions/types.ts';
import { prepareManagedFactsSession, publishManagedFacts, resumeManagedFacts } from '../src/core/persistence/facts-maintenance.ts';
import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
import { maintenancePreflight, submitDatabaseMaintenanceIntent } from '../src/core/persistence/prepared-maintenance.ts';
import { declarePersistenceProtocol } from '../src/core/persistence/protocol.ts';
+import { FACTS_ENTITY_MOVED_MESSAGES } from '../src/core/persistence/verb-errors.ts';
import { writeGitHold } from '../src/core/persistence/sync-holds.ts';
import { resolveRepairScope } from '../src/core/repair/core.ts';
import { repairRunner } from '../src/core/repair/registry.ts';
@@ -103,6 +104,18 @@ const entityRequests = (engine: ManagedBrain['engine']) => engine.executeRaw<{ r
const claimB = (engine: ManagedBrain['engine']) => engine.executeRaw<{ fact: string; source_markdown_slug: string | null }>(
'SELECT fact, source_markdown_slug FROM facts WHERE fact=$1', [CLAIM_B]);
+/**
+ * The page write's facts-absorb job, by its idempotency key. The facts-backstop effect queues it asynchronously, so a test
+ * that reads or sets its status adds it first (an add with the same key returns the job the effect queued, if any).
+ */
+async function factsAbsorbJob(engine: ManagedBrain['engine'], loser: Awaited<ReturnType<typeof extraction>>, status?: string) {
+ const key = `facts-absorb:write:${loser.write}`;
+ const job = await new MinionQueue(engine).add('facts-absorb', { slug: loser.slug, sourceId: 'default', source: 'mcp:put_page', persistence_request_id: loser.write,
+ page_id: loser.snapshot.page.id, revision: loser.snapshot.revision }, { queue: 'default', idempotency_key: key });
+ if (status) expect(await engine.executeRaw('UPDATE minion_jobs SET status=$2 WHERE id=$1 RETURNING id', [job.id, status])).toHaveLength(1);
+ return key;
+}
+
/** The `gbrain repair failed-writes` preview rows for the losing extraction's entity requests. */
async function factsListing(engine: ManagedBrain['engine']) {
const preview = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } }))
@@ -191,6 +204,7 @@ for (const backend of testBackends()) {
expect([retry.code, retry.retryable]).toEqual(['revision_conflict', false]);
expect(await entityRequests(engine)).toEqual(requests);
expect(await claimB(engine)).toEqual([]);
+ await factsAbsorbJob(engine, loser, 'dead');
const listing = await factsListing(engine);
expect(listing).toHaveLength(4);
for (const entry of listing) {
@@ -201,6 +215,42 @@ for (const backend of testBackends()) {
}
}, { databaseUrl }), 120_000);
+ // Value: protects=an extraction interrupted with exactly MAX_FILE_REFUSAL_RETRIES refused batches still has a follow-up left, so
+ // repair lists the job reset and the reset's replay publishes; fails_when=the cap check counts the third open batch as exhausted
+ // (`>=`); why_new=the rows above end with 2 or 4 open batches; seam=the run is interrupted by a throwing read before the third
+ // follow-up; no_claim=the interruption is a thrown read, not a killed worker
+ test(`#7602 [${backend}]: an extraction interrupted with three refused batches is still listed for the job reset, and the reset publishes`, () => managedBrain(async brain => {
+ const { engine } = brain;
+ const { move, loser } = await setup(brain, 3);
+ const race = raceEntityReads(engine, 4, async n => { if (n === 3) throw new Error('interrupted before the third follow-up'); await move(n); });
+ const interrupted = await publishManagedFacts(engine, await loser.session(), loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore);
+ expect(interrupted.message).toBe('interrupted before the third follow-up');
+ expect((await entityRequests(engine)).map(row => [row.state, row.error_code])).toEqual(Array(3).fill(['conflict', 'revision_conflict']));
+ const key = await factsAbsorbJob(engine, loser, 'dead');
+ expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual(Array(3).fill(`recover: gbrain facts retry-dead --reset '${key}'`));
+ expect((await resumeManagedFacts(engine, await loser.session()))?.inserted).toBe(1);
+ expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);
+ expect(await factsListing(engine)).toEqual([]);
+ }, { databaseUrl }), 120_000);
+
+ // Value: protects=a refusal whose source page was written again after its job died is listed for re-extraction, not the job
+ // reset (the retry skips a superseded page write); fails_when=factsRecovery offers recover: without the job's own page check;
+ // why_new=the rows above keep the source page; seam=the job's status is set by SQL; no_claim=does not run the reset job
+ test(`#7602 [${backend}]: a refusal whose source page was written again after its job died is listed for re-extraction`, () => managedBrain(async brain => {
+ const { engine, ctx } = brain;
+ const { move, loser } = await setup(brain, 1);
+ const before = await loser.session();
+ before.entityMoveRetry = false;
+ const race = raceEntityReads(engine, 1, move);
+ expect((await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore)).code).toBe('revision_conflict');
+ const key = await factsAbsorbJob(engine, loser, 'dead');
+ expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual([`recover: gbrain facts retry-dead --reset '${key}'`]);
+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug: loser.slug, request_id: randomUUID(), expected_revision: loser.snapshot.revision,
+ content: `---\ntype: meeting\ntitle: Example\n---\n${BODY} (rewritten)\n` } });
+ expect((await engine.readPageSnapshot(loser.slug, { sourceId: 'default' }))!.revision).not.toBe(loser.snapshot.revision);
+ expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([`re-extract: gbrain call extract_facts again on the current text of ${loser.slug} with a new request_id (one model call); ${loser.slug} changed after the page write its facts-absorb job extracts, so a job retry skips it`]);
+ }, { databaseUrl }), 120_000);
+
// Value: protects=repair failed-writes offers the job reset only for a dead facts-absorb job, and the reset's replay publishes the
// retained facts; fails_when=a waiting or completed job is listed with retry-dead (a command that finds no dead job), or the replay
// does not publish; why_new=the rows above never list a recoverable refusal; seam=the job's status is set by SQL; no_claim=does not
@@ -213,7 +263,7 @@ for (const backend of testBackends()) {
const race = raceEntityReads(engine, 1, move);
const refused = await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore);
expect(refused.code).toBe('revision_conflict');
- const key = `facts-absorb:write:${loser.write}`;
+ const key = await factsAbsorbJob(engine, loser);
const jobStatus = (status: string) => engine.executeRaw("UPDATE minion_jobs SET status=$2 WHERE name='facts-absorb' AND idempotency_key=$1 RETURNING id", [key, status]);
expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([`pending: its facts-absorb job (${key}) is waiting; preview again once it ends`]);
expect(await jobStatus('completed')).toHaveLength(1);
@@ -283,28 +333,37 @@ for (const backend of testBackends()) {
're-extract: gbrain call extract_facts again on the current text of the source turn with a new request_id (one model call)']]);
}, { databaseUrl }), 120_000);
- // Value: protects=a move during preparation refused before the fix (stored with the page layer's stale-revision message) is
- // listed with the job reset and re-admitted by it; fails_when=that message is missing from FACTS_ENTITY_MOVED_MESSAGES, so the
- // receipt is never listed and a reset replays it; why_new=the row above stores the admission-time message; seam=the stored message
- // is set by SQL to the pre-fix text; no_claim=does not reproduce the pre-fix preparation path itself
- test(`#7602 [${backend}]: a pre-fix refusal stored with the stale-revision message is listed for the job reset and re-admitted`, () => managedBrain(async brain => {
- const { engine } = brain;
- const { move, loser } = await setup(brain, 1);
- const before = await loser.session();
- before.entityMoveRetry = false;
- const race = raceEntityReads(engine, 1, move);
- expect((await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore)).code).toBe('revision_conflict');
- await engine.transaction(async tx => {
- await declarePersistenceProtocol(tx);
- expect(await tx.executeRaw("UPDATE persistence_requests SET error_message=$2 WHERE operation='extract_facts' AND slug=$1 AND state='conflict' RETURNING id", [TARGET, PAGE_CHANGED_AFTER_READ_MESSAGE])).toHaveLength(1);
- });
- const key = `facts-absorb:write:${loser.write}`;
- expect(await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE name='facts-absorb' AND idempotency_key=$1 RETURNING id", [key])).toHaveLength(1);
- expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual([`recover: gbrain facts retry-dead --reset '${key}'`]);
- expect((await resumeManagedFacts(engine, await loser.session()))?.inserted).toBe(1);
- expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);
- expect(await factsListing(engine)).toEqual([]);
- }, { databaseUrl }), 120_000);
+ // Value: protects=a refusal stored with any moved-entity message (the admission, preparation, deduplication and coordinator kinds,
+ // and the page layer's stale-revision message stored before the fix) is listed with the job reset and re-admitted by it, and
+ // repair --apply leaves it alone; fails_when=a message is missing from FACTS_ENTITY_MOVED_MESSAGES, so the receipt is never listed
+ // and a reset replays it, or apply replays a fact request; why_new=the row above stores the admission-time message only;
+ // seam=the stored message is set by SQL; no_claim=does not reproduce each message's own refusal path
+ for (const message of FACTS_ENTITY_MOVED_MESSAGES) {
+ test(`#7602 [${backend}]: a refusal stored with '${message}' is listed for the job reset and re-admitted`, () => managedBrain(async brain => {
+ const { engine } = brain;
+ const { move, loser } = await setup(brain, 1);
+ const before = await loser.session();
+ before.entityMoveRetry = false;
+ const race = raceEntityReads(engine, 1, move);
+ expect((await publishManagedFacts(engine, before, loser.backstop, [fact(CLAIM_B)], 'private', loser.slug).catch(error => error).finally(race.restore)).code).toBe('revision_conflict');
+ await engine.transaction(async tx => {
+ await declarePersistenceProtocol(tx);
+ expect(await tx.executeRaw("UPDATE persistence_requests SET error_message=$2 WHERE operation='extract_facts' AND slug=$1 AND state='conflict' RETURNING id", [TARGET, message])).toHaveLength(1);
+ });
+ const key = await factsAbsorbJob(engine, loser, 'dead');
+ expect((await factsListing(engine)).map(entry => entry.detail?.split(' (')[0])).toEqual([`recover: gbrain facts retry-dead --reset '${key}'`]);
+ const requests = await entityRequests(engine);
+ const scope = await resolveRepairScope(engine, 'default');
+ const preview = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } })).run('failed-writes', scope, { explicit: true, sourceFlag: 'default' });
+ const applied = await (await repairRunner(engine, { apply: true, logger: { info() {}, warn() {}, error() {} } })).run('failed-writes', scope,
+ { explicit: true, sourceFlag: 'default', ...(preview.apply_command ? { expect: preview.apply_command.split('--expect ')[1] } : {}) });
+ expect(applied.outcomes?.replayed ?? 0).toBe(0);
+ expect(await entityRequests(engine)).toEqual(requests);
+ expect((await resumeManagedFacts(engine, await loser.session()))?.inserted).toBe(1);
+ expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);
+ expect(await factsListing(engine)).toEqual([]);
+ }, { databaseUrl }), 120_000);
+ }
// Value: protects=a re-admitted fact is never wider than the current default visibility, and keeps world under a world default;
// fails_when=readmitMovedFacts re-admits the retained facts without narrowing them (or narrows unconditionally);
@@ -398,7 +457,13 @@ for (const backend of testBackends()) {
expect(race.moves()).toBe(1);
expect(refused.code).toBe('source_changed');
expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);
- expect(await engine.executeRaw("UPDATE minion_jobs SET status='dead' WHERE name='facts-absorb' AND idempotency_key=$1 RETURNING id", [`facts-absorb:write:${loser.write}`])).toHaveLength(1);
+ const key = await factsAbsorbJob(engine, loser, 'waiting');
+ expect((await factsListing(engine)).map(entry => entry.detail)).toEqual([]);
+ const pending = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } }))
+ .run('failed-writes', await resolveRepairScope(engine, 'default'), { explicit: true, sourceFlag: 'default' });
+ expect((pending.listing ?? []).filter(entry => entry.item.includes(' extract_facts')).map(entry => entry.detail))
+ .toEqual([`pending: its facts-absorb job (${key}) is waiting; preview again once it ends`]);
+ await factsAbsorbJob(engine, loser, 'dead');
const preview = await (await repairRunner(engine, { apply: false, logger: { info() {}, warn() {}, error() {} } }))
.run('failed-writes', await resolveRepairScope(engine, 'default'), { explicit: true, sourceFlag: 'default' });
expect((preview.listing ?? []).filter(entry => entry.item.includes(' extract_facts')).map(entry => [entry.item.split(' ')[0], entry.detail])).toEqual([
@@ -426,6 +491,7 @@ for (const backend of testBackends()) {
expect(refused.code).toBe('source_changed');
expect((await entityRequests(engine)).map(row => [row.state, row.error_code])).toEqual([['conflict', 'revision_conflict'], ['conflict', 'source_changed']]);
expect(await claimB(engine)).toEqual([]);
+ await factsAbsorbJob(engine, loser, 'dead');
expect((await factsListing(engine)).map(entry => entry.detail)).toEqual(Array(2).fill(`re-extract: gbrain call extract_facts again on the current text of ${loser.slug} with a new request_id (one model call); its extraction also holds refusals one job retry does not re-admit together`));
}, { databaseUrl }), 120_000);