Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(facts): a follow-up passes the source-page check when only its own extraction moved the page (#7602 red team)
garrytan91cc63c05bb5
diff --git a/docs/architecture/key-files/core-persistence-facts.md b/docs/architecture/key-files/core-persistence-facts.mdindex d6f75a2947..62552f7d89 100644--- a/docs/architecture/key-files/core-persistence-facts.md+++ b/docs/architecture/key-files/core-persistence-facts.md@@ -5,7 +5,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGELManaged facts extraction: the entity-intent preparer and publisher, conversation facts batches and the unmanaged derived index. The extraction pipeline itself is in [Google and Loops (continued)](google-and-loops-continued.md) (`src/core/facts/backstop.ts`) and [Core Services (2/3)](core-services-2.md) (`src/core/facts/extract.ts`).- `src/core/facts/bulk-days.ts` — #7247 the one rule doctor `fact_bulk_dates` and `gbrain repair fact-claim-dates` share. `findBulkFactDays` (one read-only aggregate over active facts of active sources, optionally scoped) returns each UTC `valid_from` day holding at least `BULK_DAY_RULE` (200 facts, 25% of the source, 20 entities) with counts, top source tags and `dated_claims`. `claimDate` returns a claim's only `YYYY-MM-DD` token when it is a real date from 1900 to the day before; the aggregate's SQL applies the same token and bounds.-- `src/core/persistence/{facts-maintenance,facts-prepare}.ts` — the managed facts publisher and preparer behind the backstop's `publishManagedFacts` / `resumeManagedFacts` and the explicit fact writers (`managed-fact-write.ts`, `remember`); the shared maintenance dispatch they ride is in [Core Persistence](core-persistence.md). Each `ManagedFactsSession` carries one `MaintenanceWriteWait` (`maintenance-wait.ts`) built from the caller's `FactsBackstopCtx.deadlineAtMs` (the facts-absorb job passes its deadline; other callers get the 30 s base) and its abort signal: `collectManagedFacts` waits that long for each accepted request through `awaitWrite`, a cancelled job stops waiting at once, and a request still nonterminal after the wait leaves as `write_pending` with its receipt (#6778, `test/facts-absorb-publish-wait-6778.test.ts`). GBRA-130 A-36: `collectManagedFacts` awaits every row before it throws. An entity request that ended on a code of its own page (`ENTITY_REFUSAL_CODES`: `revision_conflict`, `invalid_fence`, `page_body_segmented`, …; not a file-check or withdrawal refusal, which the follow-up walks own) is reported as `refused` (slug, code, request id) on `ManagedFactsResult` and the `extract_facts` result when the session sets `reportRefusals` and another entity of the batch committed (the backstop sets it; `managed-fact-write.ts` and `remember` do not and keep the refusal), and the completion receipt refused only as incomplete is not a failure of its own; any other terminal row, then a pending one, then a refusal with nothing committed beside it, throws as its receipt. Test: `test/pathological/a-36-facts-batch-partial-refusal.test.ts` (PGLite and Postgres). `facts-maintenance.ts` / `facts-prepare.ts` (#6506): an automatic extraction's entity intent carries `withdrawn: 'skip'` (set by the backstop's `publishManagedFacts` call, never by `managed-fact-write.ts` or `remember`), under which `prepareManagedFactsMutation` skips a candidate `isFactWithdrawn` matches (after `assertFactNotPurged`, which still refuses `purged_content`) and counts it as `withdrawn_skipped` on the entity outcome, summed into the completion outcome, `ManagedFactsResult` and the `extract_facts` result (the unmanaged pipeline counts its own B-10 skip into the same field); without the flag the refusal stands. At locked validation the per-entry purge and withdrawal checks run before the page's fence re-validation, and under the flag a claim forgotten since preparation refuses `revision_conflict` with `error.withdrawn` (so the receipt's `error_detail.origin` is `withdrawal`); no `extract_facts` request is re-prepared. `resumeManagedFacts` runs `followWithdrawalRefusals` for an input-keyed session (`inputKeyed`, no request id): the `followFileRefusals` walk generalized as `followRefusals`, matching entity refusals by `withdrawnFromDetail(error_detail)` or the pre-detail `invalid_params` + `FACT_WITHDRAWN_MESSAGE` pair, re-admitting the retained facts under `digest([batchKey, 'withdrawal-retry', n])` with the flag (`readmitWithdrawnFacts`: a fact forgotten at its retained visibility keeps that visibility so preparation skips it instead of narrowing it to private), bounded by `MAX_FILE_REFUSAL_RETRIES`, historical receipts untouched. Tests: `test/managed-facts-withdrawn-skip-6506.test.ts`, `test/managed-facts-withdrawal-recovery-6506.test.ts` (PGLite and Postgres). #7602: an automatic extraction with no caller request id (`entityMoveRetry`, set by the backstop; explicit writers keep the refusal) also runs `followEntityMoves`, after admission in `publishManagedFacts` and on resume: an entity request refused `revision_conflict` because another write moved its entity page (`FACTS_ENTITY_MOVED_MESSAGES` in `verb-errors.ts`: changed after admission, changed during preparation, deduplication state changed, the coordinator's `PAGE_CHANGED_DURING_PREPARATION_MESSAGE`, which `coordinator.ts` and `group-publish.ts` throw, and the page layer's `PAGE_CHANGED_AFTER_READ_MESSAGE`, how a move during preparation was stored before the fix; `isEntityMovedRefusal` is the one predicate the walk and the repair preview share) is re-admitted under `digest([batchKey, 'entity-moved-retry', n])` at the page's current revision with its retained intent (`readmitMovedFacts`; a deleted or replaced page keeps the refusal), no model call, each batch settled first so a follow-up that loses again is followed in the same run, bounded by `MAX_FILE_REFUSAL_RETRIES`; a follow-up refused for another reason (a held file) is not re-admitted, and the walk carries every request it did not follow. A refusal `collectManagedFacts` still throws after the walks is marked `retryable: false`, and the facts-absorb handler dead-letters such a journaled refusal on its first attempt. Test: `test/facts-entity-moved-7602.test.ts` (PGLite and Postgres).+- `src/core/persistence/{facts-maintenance,facts-prepare}.ts` — the managed facts publisher and preparer behind the backstop's `publishManagedFacts` / `resumeManagedFacts` and the explicit fact writers (`managed-fact-write.ts`, `remember`); the shared maintenance dispatch they ride is in [Core Persistence](core-persistence.md). Each `ManagedFactsSession` carries one `MaintenanceWriteWait` (`maintenance-wait.ts`) built from the caller's `FactsBackstopCtx.deadlineAtMs` (the facts-absorb job passes its deadline; other callers get the 30 s base) and its abort signal: `collectManagedFacts` waits that long for each accepted request through `awaitWrite`, a cancelled job stops waiting at once, and a request still nonterminal after the wait leaves as `write_pending` with its receipt (#6778, `test/facts-absorb-publish-wait-6778.test.ts`). GBRA-130 A-36: `collectManagedFacts` awaits every row before it throws. An entity request that ended on a code of its own page (`ENTITY_REFUSAL_CODES`: `revision_conflict`, `invalid_fence`, `page_body_segmented`, …; not a file-check or withdrawal refusal, which the follow-up walks own) is reported as `refused` (slug, code, request id) on `ManagedFactsResult` and the `extract_facts` result when the session sets `reportRefusals` and another entity of the batch committed (the backstop sets it; `managed-fact-write.ts` and `remember` do not and keep the refusal), and the completion receipt refused only as incomplete is not a failure of its own; any other terminal row, then a pending one, then a refusal with nothing committed beside it, throws as its receipt. Test: `test/pathological/a-36-facts-batch-partial-refusal.test.ts` (PGLite and Postgres). `facts-maintenance.ts` / `facts-prepare.ts` (#6506): an automatic extraction's entity intent carries `withdrawn: 'skip'` (set by the backstop's `publishManagedFacts` call, never by `managed-fact-write.ts` or `remember`), under which `prepareManagedFactsMutation` skips a candidate `isFactWithdrawn` matches (after `assertFactNotPurged`, which still refuses `purged_content`) and counts it as `withdrawn_skipped` on the entity outcome, summed into the completion outcome, `ManagedFactsResult` and the `extract_facts` result (the unmanaged pipeline counts its own B-10 skip into the same field); without the flag the refusal stands. At locked validation the per-entry purge and withdrawal checks run before the page's fence re-validation, and under the flag a claim forgotten since preparation refuses `revision_conflict` with `error.withdrawn` (so the receipt's `error_detail.origin` is `withdrawal`); no `extract_facts` request is re-prepared. `resumeManagedFacts` runs `followWithdrawalRefusals` for an input-keyed session (`inputKeyed`, no request id): the `followFileRefusals` walk generalized as `followRefusals`, matching entity refusals by `withdrawnFromDetail(error_detail)` or the pre-detail `invalid_params` + `FACT_WITHDRAWN_MESSAGE` pair, re-admitting the retained facts under `digest([batchKey, 'withdrawal-retry', n])` with the flag (`readmitWithdrawnFacts`: a fact forgotten at its retained visibility keeps that visibility so preparation skips it instead of narrowing it to private), bounded by `MAX_FILE_REFUSAL_RETRIES`, historical receipts untouched. Tests: `test/managed-facts-withdrawn-skip-6506.test.ts`, `test/managed-facts-withdrawal-recovery-6506.test.ts` (PGLite and Postgres). #7602: an automatic extraction with no caller request id (`entityMoveRetry`, set by the backstop; explicit writers keep the refusal) also runs `followEntityMoves`, after admission in `publishManagedFacts` and on resume: an entity request refused `revision_conflict` because another write moved its entity page (`FACTS_ENTITY_MOVED_MESSAGES` in `verb-errors.ts`: changed after admission, changed during preparation, deduplication state changed, the coordinator's `PAGE_CHANGED_DURING_PREPARATION_MESSAGE`, which `coordinator.ts` and `group-publish.ts` throw, and the page layer's `PAGE_CHANGED_AFTER_READ_MESSAGE`, how a move during preparation was stored before the fix; `isEntityMovedRefusal` is the one predicate the walk and the repair preview share) is re-admitted under `digest([batchKey, 'entity-moved-retry', n])` at the page's current revision with its retained intent (`readmitMovedFacts`; a deleted or replaced page keeps the refusal), no model call, each batch settled first so a follow-up that loses again is followed in the same run, bounded by `MAX_FILE_REFUSAL_RETRIES`; a follow-up refused for another reason (a held file) is not re-admitted, and the walk carries every request it did not follow. A follow-up's source-page check in `facts-prepare.ts` accepts a source page moved only by a committed request of the same extraction (same principal and input digest, at the page's current revision), such as a fact the batch wrote onto its own source page; any other change still refuses `The source page changed during fact extraction.` A refusal `collectManagedFacts` still throws after the walks is marked `retryable: false`, and the facts-absorb handler dead-letters such a journaled refusal on its first attempt. Test: `test/facts-entity-moved-7602.test.ts` (PGLite and Postgres).- `src/core/facts/same-page-shadow.ts` — #6333 the shadow counter both fact writers share. `samePageRewording(engine, sourceId, fact, embedding)` re-reads the entity's nearest live same-visibility candidate (`listSupersessionCandidates`, the pool and filters of `decideSingleFact`'s cosine pass) and answers whether `wouldDedupSamePage` (capture-dedup.ts: same lane, same `source_session`, same visibility, cosine ≥ 0.85, no diverging guard token) would have called the fact a rewording; `warnSamePageShadow` is the one stderr spelling (`[facts-backstop] #6333 shadow: …; both kept`). `facts-prepare.ts` calls it for every candidate `decideSingleFact` did not match and counts `would_dedup_same_page` on the entity outcome, the completion outcome, `ManagedFactsResult` and the `extract_facts` result; the count is frozen in the receipt, so a replay reports it and does not recount. Nothing is dropped; this is the managed mirror of the unmanaged backstop's counter (`backstop.ts`), which never fired on a managed brain. Tests: `test/backstop-same-page-shadow-managed-6333.test.ts` (PGLite and Postgres), `test/backstop-same-page-shadow-6333.test.ts` (unmanaged).- `src/core/persistence/derived-facts.ts` — `writeDerivedFacts`: the unmanaged conversation fact index commits in one maintenance transaction under the maintenance principal; on a managed brain it refuses with `writer_coordinator_required`, because every managed facts/takes writer publishes a receipted maintenance request (`test/receipt-coverage.test.ts` lists every coordinated write site). Tests: `test/write-attribution-misc.test.ts`.- `src/core/facts/conversation-publication.ts` — managed conversation facts as database-only `managed_maintenance_conversation_facts` batch requests (`submitDatabaseMaintenanceIntent` on `ANCHOR_SLUG`, a dot-led slug no page can hold, dispatched from `prepareMaintenanceMutation`). Each page's frozen entry (`buildConversationPage`: `FrozenExtractedFact` rows, its generation and attempt, vectors only under the brain's facts embedding signature) joins the run's batch; `enqueueManagedPage` publishes it once `DEFAULT_BATCH_CAPS` is reached (25 pages, 10 with extracted facts, 8 MiB) and `flushManagedBatch` at the end of the run. The generation key (protocol, source incarnation, page id, revision, parser-input version token, `CONVERSATION_EXTRACTOR_VERSION`, selectors, newest extractor row id) is checked before model work by `startManagedPage` against `loadGenerationIndex` (the writer's unsettled and failed batches on the anchor, read once per run): a member of an unsettled batch waits and replays its receipt, a member of a retryably failed batch is resubmitted from its stored entry, a deterministic failure of a batch that held it alone or three failed batches block it (GBRA-130 A-51: `publishBatch` re-publishes a multi-page batch that failed deterministically in halves, so only the page that fails alone takes the code; `test/pathological/a-51-conversation-batch-booking.test.ts`); all without model calls. The batch request id digests its members' (generation, attempt) pairs; `submitConversationPages` raises every member's attempt past an id another write holds, or whose receipt lost its intent without committing (#7516: a redacted `purged_content` refusal or a compacted failure takes the attempt history `loadGenerationIndex` reads, so the same id came back on every run as `idempotency_conflict` after a paid extraction; test `test/conversation-facts-redacted-batch-7516.test.ts`, + Postgres arm). The preparer locks every member's key and applies each on its own (`applyEntry`): a deleted, replaced or moved page is skipped and reported, a page whose rows fail validation gets a durable `blocked:` not-extractable outcome, vectors under another signature are dropped; none fails the batch. `estimateConversationIntentBytes`/`conversationIntentByteLimit` skip a page whose largest entry cannot fit before any model call; `assertConversationAdmissionHeadroom` stops admitting at 80% of the writer's outstanding requests, reserved receipt bytes or lifetime ids (`maintenance_backpressure`, exit 12, naming the resource and a `--limit` that fits), up front for the run's planned batches and before each new batch. `publishConversationFacts` (the managed apply and the unmanaged writer) passes fact rows through the write gate at the page's derived tier and retires the prior batch only for a complete replacement (#7013: `full` scope, at least one fact, nothing held or rejected) or on a page with no active extractor facts, through `replaceConversationFacts`, which deletes the page's extractor rows (expiring referenced and keeping label-retired ones) and inserts the rows numbered above every remaining row, failing on a lost insert. Otherwise the prior facts stay: the outcome row replaces the page's prior outcome rows (so the page is not paid for again), a `window` run (`--since` dropped messages) adds its facts beside them, and a `partial` run (a `--segment-limit` cut, a page changed mid-run) publishes no facts over them. Each entry carries its `scope`; an entry frozen without one is `partial` without its outcome, `window` with a `--since` selector, else `full`. Tests: `test/pathological/h4-conversation-facts-zero-yield.test.ts` (managed and unmanaged, + Postgres arm). Tests: `test/conversation-facts-managed.test.ts` (+ Postgres arm), managed-connector contract cases `extract_conversation_facts*`.diff --git a/src/core/persistence/facts-prepare.ts b/src/core/persistence/facts-prepare.tsindex 1b56daebcc..ea0854c5f9 100644--- a/src/core/persistence/facts-prepare.ts+++ b/src/core/persistence/facts-prepare.ts@@ -124,9 +124,10 @@ export async function prepareManagedFactsMutation(engine: BrainEngine, row: Writ`Page ${p.origin.slug}, which these facts were extracted from, was deleted or replaced during extraction.`);if (origin.revision !== p.origin.revision) {const own = await tx.executeRaw(`SELECT id FROM persistence_requests WHERE operation='extract_facts' AND source_id=$1- AND source_incarnation=$2::uuid AND principal_kind=$3 AND principal_id=$4 AND COALESCE(intent->>'batchKey',outcome->>'batch_key')=$5+ AND source_incarnation=$2::uuid AND principal_kind=$3 AND principal_id=$4+ AND (COALESCE(intent->>'batchKey',outcome->>'batch_key')=$5 OR COALESCE(intent->>'inputDigest',outcome->>'input_digest')=$8)AND slug=$6 AND state='committed' AND outcome->>'revision'=$7`,- [row.source_id, row.source_incarnation, row.principal_kind, row.principal_id, p.batchKey, p.origin.slug, origin.revision]);+ [row.source_id, row.source_incarnation, row.principal_kind, row.principal_id, p.batchKey, p.origin.slug, origin.revision, p.inputDigest ?? null]);if (!own.length) throw factsRefusal('revision_conflict', 'The source page changed during fact extraction.', row,`Page ${p.origin.slug} changed after its facts were extracted, so facts from the older text were not published.`,`once it is final, run extract_facts again on the current text of ${p.origin.slug} with a new request_id.`);diff --git a/test/facts-entity-moved-7602.test.ts b/test/facts-entity-moved-7602.test.tsindex 2b01ef0fcb..44f52e1835 100644--- a/test/facts-entity-moved-7602.test.ts+++ b/test/facts-entity-moved-7602.test.ts@@ -44,6 +44,7 @@ const TARGET = 'people/alice-example';const CLAIM_B = 'Alice-example owns the synthetic onboarding checklist.';const OTHER = 'companies/acme-example';const CLAIM_OTHER = 'Acme-example runs the synthetic onboarding pilot.';+const CLAIM_SELF = 'The synthetic planning chat set the onboarding launch date.';const BODY = 'Notes from a synthetic planning chat about the onboarding checklist and the launch date, kept for the regression.';afterEach(() => { __setChatTransportForTests(null); resetGateway(); });@@ -172,6 +173,23 @@ for (const backend of testBackends()) {expect(await factsListing(engine)).toEqual([]);}, { databaseUrl }), 120_000);+ // Value: protects=the follow-up of a batch that also committed a fact on its own source page (a page write that extracts facts+ // about itself) publishes, because the source page moved only by that batch's own commit; fails_when=the follow-up is refused+ // 'The source page changed during fact extraction.' and the run throws; why_new=the row above extracts facts about the entity+ // only; seam=none; no_claim=one self fact and one moved entity+ test(`#7602 [${backend}]: a batch that also committed a fact on its own source page re-admits the moved entity in the same run`, () => managedBrain(async brain => {+ const { engine } = brain;+ const { move, loser } = await setup(brain, 1);+ const race = raceEntityReads(engine, 1, move);+ const first = await publishManagedFacts(engine, await loser.session(), loser.backstop, [{ ...fact(CLAIM_SELF), entity_slug: loser.slug }, fact(CLAIM_B)],+ 'private', loser.slug).finally(race.restore);+ expect(race.moves()).toBe(1);+ expect({ inserted: first.inserted, refused: first.refused }).toEqual({ inserted: 2, refused: undefined });+ expect((await engine.readPageSnapshot(loser.slug, { sourceId: 'default' }))!.revision).not.toBe(loser.snapshot.revision);+ expect(await claimB(engine)).toEqual([{ fact: CLAIM_B, source_markdown_slug: TARGET }]);+ expect((await entityRequests(engine)).map(row => [row.state, row.error_code])).toEqual([['conflict', 'revision_conflict'], ['committed', null]]);+ }, { databaseUrl }), 120_000);+// Value: protects=the third follow-up still runs and publishes when the page stops moving after three reads; fails_when=the walk// stops before MAX_FILE_REFUSAL_RETRIES follow-ups; why_new=the row below covers only the side where every follow-up loses;// seam=none; no_claim=the moves are direct publications from a read hook, not concurrent facts-absorb jobs