Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
A transcript re-ingest that drops stale parts scrubs what names them
garrytaned7b0c2ed6b1
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f9fef5222c..7f14e5766b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -32,12 +32,12 @@ identifiers and attribution are available in the pre-removal Git revision| What you did | Before | Now ||---|---|---|| A query or search with eval capture on, whose capture runs after the response, racing a page purge | the capture could write the purged slug back into `eval_candidates.retrieved_slugs` | the insert keeps only slugs whose page still exists, checked under the same row lock the feedback writers take |-| A connector, GitHub or Google sync deletes a page (`deletePages`) | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction |+| A connector, GitHub or Google sync deletes a page (`deletePages`), or a transcript re-ingest deletes a stale part | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction || `gbrain sources remove`, `sources purge`, the expired-archive sweep, or a managed `remove`/`purge` | the source's retrieval logs, ratings, volunteer rows and captured eval slugs stayed | deleted in the statement that deletes the source, and swept again right after for a write that raced it || `gbrain sources federate` or `sources restore` on a source with a remote-written `visibility: private` page and no readers | the warning named that page, though no-grant connections still cannot read it | the warning applies the read paths' privacy rules, unless `search.remote_private_pages` or `GBRAIN_REMOTE_PRIVATE_PAGES` relaxes them |- `src/core/feedback/store.ts`: `insertEvalCandidate` is the one eval capture insert both engines run.-- `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.+- `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. The transcript importer's stale-part delete now calls `deletePages`. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.- `src/core/source-delete.ts`: `removedSourceScrubCtes` runs inside `deleteSourceRow` and the managed lifecycle's source delete, then once more ungated. `retrieval_weights` and `mention_gazetteer_entries` already cascaded from `sources`.- `src/core/access/federation-exposure.ts`: the exposure read adds `privacyPartSql` unless `remotePrivacyRelaxed`.- Tests: `test/purge-scrub-followups-9068.test.ts` and `test/federation-exposure-private-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`). `test/eval-candidates.test.ts` now seeds the pages its round-trip captures, since a capture keeps only live slugs.diff --git a/docs/architecture/key-files/core-services-3.md b/docs/architecture/key-files/core-services-3.mdindex b314a056a4..ac65f34f00 100644--- a/docs/architecture/key-files/core-services-3.md+++ b/docs/architecture/key-files/core-services-3.md@@ -27,7 +27,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/verbs/usage-log.ts` — observability sidecar: one JSONL line per verb call at `~/.gbrain/integrations/memory-verbs/usage.jsonl` (gbrainPath — GBRAIN_HOME honored; `brainId()` = the resolved gbrain home). LOCAL ONLY, never uploaded, stats-only (lock-free 10MB rotation may drop lines; O_APPEND line-atomic, best-effort on Windows). `logVerbUsage` is fire-and-forget (never awaited, never throws); written from the DISPATCH layer so param-validation failures count. `readVerbUsage`/`earliestVerbUsageTs` feed `gbrain protocol stats` (incl. measured TTHW vs the init-stamped `protocol_installed_at`) and the doctor `memory_verbs_usage` check.-- `src/core/feedback/` — use-attributed retrieval feedback (guide: `docs/guides/retrieval-feedback.md`). `settings.ts` resolves `feedback.*` with a 30s cache (`effectiveInfluence` is 0 when off). `record.ts`: `recordAnswer(ctx, { op, pages, links })` mints `ans_<ULID>`, returns it at once and enqueues the event (pages with the retrieved `content_hash`, relational path edges) on a bounded write-behind queue (cap 1,000, drop-oldest counted, drained by the background-work registry as `retrieval-feedback`); the citation signal (rating 4, half alpha) rides the same queue. Only callers that may change shared ranking record: `canTeachSource` allows the owner's CLI (`remote === false`) and auth-less stdio, and remote callers with write scope on that source, no slug fence, no operation allowlist excluding `rate_answer`, not delegated. `recordThinkAnswer` marks cited pages (synthesis `ok`, bare slug unambiguous across gathered sources). `feedbackMetaFields` adds `answer_id`/`feedback` only to rateable answers (callers that cannot rate see unchanged output). `rate.ts` `rateAnswer` implements `rate_answer`: whole-answer ratings (think/synthesize cited pages, else gathered; search-shaped returned pages; plus path edges into rated pages) or targeted `pages: [{ ref, rating }]` (pages only); skips `stale_revision`/`unverifiable_revision`/`not_authorized_source`; receipts carry weight before/after and the multiplier; refusals are registered codes. `store.ts` owns all SQL (TEXT[] binds, booleans bound as text): `insertRetrievalEvents`, `applyRatings` (one statement: idempotent feedback insert per element, then an upsert whose `DO UPDATE` recomputes from the locked row, so concurrent ratings never lose an update), `readWeights`, `pruneRetrievalFeedback` (purge phase), `resetRetrievalWeights`, `feedbackStatus`, `scrubPurgedSlugs` (#8821: every hard delete of a page row, the page purge, the expired-tombstone purge and the engines' `deletePages` (connector, GitHub and Google deletes; #9068), removes a deleted slug's source-scoped rows as a page key or either end of an edge key, and drops it from `eval_candidates.retrieved_slugs` (`scrubCandidateSlugs`), in the delete's transaction; the page-purge receipt checks each store), `insertEvalCandidate` (#9068: the eval capture insert, keeping only slugs whose page row still exists, probed under the same key-share lock as the feedback writers). Pinned by `test/feedback-store.test.ts` and `test/e2e/feedback-parity.test.ts`.+- `src/core/feedback/` — use-attributed retrieval feedback (guide: `docs/guides/retrieval-feedback.md`). `settings.ts` resolves `feedback.*` with a 30s cache (`effectiveInfluence` is 0 when off). `record.ts`: `recordAnswer(ctx, { op, pages, links })` mints `ans_<ULID>`, returns it at once and enqueues the event (pages with the retrieved `content_hash`, relational path edges) on a bounded write-behind queue (cap 1,000, drop-oldest counted, drained by the background-work registry as `retrieval-feedback`); the citation signal (rating 4, half alpha) rides the same queue. Only callers that may change shared ranking record: `canTeachSource` allows the owner's CLI (`remote === false`) and auth-less stdio, and remote callers with write scope on that source, no slug fence, no operation allowlist excluding `rate_answer`, not delegated. `recordThinkAnswer` marks cited pages (synthesis `ok`, bare slug unambiguous across gathered sources). `feedbackMetaFields` adds `answer_id`/`feedback` only to rateable answers (callers that cannot rate see unchanged output). `rate.ts` `rateAnswer` implements `rate_answer`: whole-answer ratings (think/synthesize cited pages, else gathered; search-shaped returned pages; plus path edges into rated pages) or targeted `pages: [{ ref, rating }]` (pages only); skips `stale_revision`/`unverifiable_revision`/`not_authorized_source`; receipts carry weight before/after and the multiplier; refusals are registered codes. `store.ts` owns all SQL (TEXT[] binds, booleans bound as text): `insertRetrievalEvents`, `applyRatings` (one statement: idempotent feedback insert per element, then an upsert whose `DO UPDATE` recomputes from the locked row, so concurrent ratings never lose an update), `readWeights`, `pruneRetrievalFeedback` (purge phase), `resetRetrievalWeights`, `feedbackStatus`, `scrubPurgedSlugs` (#8821: every hard delete of a page row, the page purge, the expired-tombstone purge and the engines' `deletePages` (connector, GitHub and Google deletes, the transcript importer's stale parts; #9068), removes a deleted slug's source-scoped rows as a page key or either end of an edge key, and drops it from `eval_candidates.retrieved_slugs` (`scrubCandidateSlugs`), in the delete's transaction; the page-purge receipt checks each store), `insertEvalCandidate` (#9068: the eval capture insert, keeping only slugs whose page row still exists, probed under the same key-share lock as the feedback writers). Pinned by `test/feedback-store.test.ts` and `test/e2e/feedback-parity.test.ts`.- `src/core/ops/feedback.ts` — the `rate_answer` op (scope `write`, CLI `gbrain rate <answer_id> <rating>`), delegating to `feedback/rate.ts`. Answers carry `answer_id` from `ops/search.ts` (`search`, `query` via `buildRetrievalResponseMeta`), `ops/takes.ts` (`think`), `verbs.ts` (`synthesize`) and `ops/facts.ts` (`recall` page arm on the hybrid path).diff --git a/src/core/transcripts/ingest.ts b/src/core/transcripts/ingest.tsindex 083743e1bd..4e8e0672ed 100644--- a/src/core/transcripts/ingest.ts+++ b/src/core/transcripts/ingest.ts@@ -530,7 +530,7 @@ async function importSession(engine: BrainEngine, opts: TranscriptsIngestOpts, r// #5235: on a managed brain the stale part goes through the coordinator like the parts// themselves, bound to its revision; the direct delete is refused there.if (coordinator) await deleteStaleTranscriptPart(coordinator, engine, opts.sourceId, row.slug);- else await engine.deletePage(row.slug, { sourceId: opts.sourceId });+ else await engine.deletePages([row.slug], { sourceId: opts.sourceId });result.partsDeleted++;}}diff --git a/test/purge-scrub-followups-9068.test.ts b/test/purge-scrub-followups-9068.test.tsindex 9fb3d3d2b7..317814ba75 100644--- a/test/purge-scrub-followups-9068.test.ts+++ b/test/purge-scrub-followups-9068.test.ts@@ -6,21 +6,21 @@* page purge stores only slugs whose page still exists, and on Postgres one that runs while the purge is still open* waits on the page row and drops the slug; (b) the connector delete (deleteConnectorPage -> deletePages) scrubs* retrieval logs, ratings, weights, the volunteer log, the gazetteer and the eval candidates' captured slugs for- * that source in the delete's transaction;+ * that source in the delete's transaction, and so does the transcript stale-part delete, which now calls it;* (c) removing a source (unmanaged removeSource -> deleteSourceRow, and the managed lifecycle remove) drops every* such row of the source, and strips its slugs from eval candidates no other covered source still answers.* 2. Fails when: logEvalCandidate inserts slugs without probing the page row under a key-share lock, a hard delete or a* source removal leaves any of those rows behind, or a scrub reaches another source's rows or a slug another* source the candidate covers still holds.* 3. Existing coverage: purge-retrieval-feedback-8821 and purge-name-stores-9068 pin delete_page purge and the- * tombstone purge only; nothing pinned the capture, the connector delete, or source removal.+ * tombstone purge only; nothing pinned the capture, the connector and stale-part deletes, or source removal.* 4. Real seams: PGLite, and Postgres when DATABASE_URL is set (test/postgres-unit-arms.txt); the feedback store's- * writers, captureEvalCandidate under withResponseDeferral, the delete_page op, deleteConnectorPage, removeSource,+ * writers, captureEvalCandidate under withResponseDeferral, the delete_page op, deleteConnectorPage, runTranscriptsIngest, removeSource,* runManagedSourceLifecycle; repo-less brain, no provider calls.*/import { afterAll, beforeAll, describe, expect, test } from 'bun:test';import { randomUUID } from 'node:crypto';-import { mkdirSync, mkdtempSync, rmSync } from 'node:fs';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';import { join } from 'node:path';import type { BrainEngine } from '../src/core/engine.ts';@@ -35,6 +35,8 @@ import { captureEvalCandidate } from '../src/core/eval-capture.ts';import { withResponseDeferral } from '../src/core/background-work.ts';import { deleteConnectorPage } from '../src/core/connectors/api-source.ts';import { removeSource } from '../src/core/sources-ops.ts';+import { runTranscriptsIngest } from '../src/core/transcripts/ingest.ts';+import { buildTranscriptSlug } from '../src/core/transcripts/types.ts';import { registerLocalWriter } from '../src/core/persistence/identity.ts';import { claimWorktree } from '../src/core/persistence/ownership.ts';import { runManagedSourceLifecycle } from '../src/core/persistence/source-lifecycle.ts';@@ -64,9 +66,14 @@ async function page(engine: BrainEngine, sourceId: string, slug: string) {await importFromContent(engine, slug, `---\ntitle: ${slug}\ntype: note\n---\n# ${slug}\n\nBody text ${randomUUID()}.\n`, { noEmbed: true, sourceId });}-/** The page and OTHER in `sourceId`, and every store that names the page: a retrieval event (page and edge), page and edge ratings, a volunteer pointer, a gazetteer entry. */+/** The page and OTHER in `sourceId`, and every store that names the page (traces). */async function traced(engine: BrainEngine, sourceId: string, slug: string) {await page(engine, sourceId, slug);+ await traces(engine, sourceId, slug);+}++/** OTHER in `sourceId`, and for an existing page a retrieval event (page and edge), page and edge ratings, a volunteer pointer, a gazetteer entry. */+async function traces(engine: BrainEngine, sourceId: string, slug: string) {await page(engine, sourceId, OTHER);const id = randomUUID(), edge = `${slug}|mentions|${OTHER}`;await insertRetrievalEvents(engine, [{ id, client_id: 'local', op: 'query', links: [{ source_id: sourceId, edge_key: edge, to_slug: OTHER }],@@ -180,6 +187,39 @@ describe('#9068 the batch hard delete scrubs what names the page', () => {});+/** An openclaw session file with `count` large messages, so a low page ceiling splits it into parts. */+function session(dir: string, id: string, count: number): string {+ const filler = 'lorem widget fact '.repeat(220);+ const at = (i: number) => `2026-08-10T08:${String(Math.floor(i / 60)).padStart(2, '0')}:${String(i % 60).padStart(2, '0')}.000Z`;+ const lines = [JSON.stringify({ type: 'session', version: 3, id, timestamp: '2026-08-10T08:00:00.000Z', cwd: '/tmp' }),+ ...Array.from({ length: count }, (_, i) => JSON.stringify({ type: 'message', id: `m-${i}`, timestamp: at(i),+ message: { role: i % 2 === 0 ? 'user' : 'assistant', timestamp: at(i), content: [{ type: 'text', text: `marker-${i} ${filler}` }] } }))];+ const path = join(dir, `${id}-${count}.jsonl`);+ writeFileSync(path, lines.join('\n') + '\n');+ return path;+}++describe('#9068 the transcript stale-part delete scrubs what names the part', () => {+ test('a re-ingest that shrinks a split session deletes its stale parts with their rows', () => withEnv({ GBRAIN_HOME: home }, async () => {+ for (const engine of engines) {+ const dir = mkdtempSync(join(home, 'sessions-')), id = `shrink-${engine.kind}`;+ const ingest = (path: string) => runTranscriptsIngest(engine, { paths: [path], sourceId: 'default', userPatternsPath: '/nonexistent-patterns.txt' });+ await engine.setConfig('pages.max_bytes', '100000');+ let parts: number;+ try { parts = (await ingest(session(dir, id, 150))).pages.imported; } finally { await engine.unsetConfig('pages.max_bytes'); }+ expect(parts).toBeGreaterThan(1);+ const stale = `${buildTranscriptSlug('openclaw', '2026-08-10T08:00:00.000Z', { sessionId: id })}-p2`;+ await traces(engine, 'default', stale);+ const query = `stale part ${randomUUID()}`;+ await engine.logEvalCandidate(candidate(query, [stale, OTHER], ['default']));+ expect((await ingest(session(dir, id, 2))).partsDeleted).toBe(parts - 1);+ expect(await engine.executeRaw(`SELECT 1 FROM pages WHERE source_id = 'default' AND slug = $1`, [stale])).toHaveLength(0);+ expect({ engine: engine.kind, residue: await residue(engine, 'default', stale) }).toEqual({ engine: engine.kind, residue: NONE });+ expect(await capturedSlugs(engine, query)).toEqual([[OTHER]]);+ }+ }), 180_000);+});+describe('#9068 removing a source scrubs every row that names its pages', () => {test('unmanaged removeSource drops the source\'s rows and strips its slugs from eval candidates', () => withEnv({ GBRAIN_HOME: home }, async () => {for (const engine of engines) {