Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

A transcript re-ingest that drops stale parts scrubs what names them

garrytaned7b0c2ed6b1
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f9fef5222c..7f14e5766b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -32,12 +32,12 @@ identifiers and attribution are available in the pre-removal Git revision
| What you did | Before | Now |
|---|---|---|
| A query or search with eval capture on, whose capture runs after the response, racing a page purge | the capture could write the purged slug back into `eval_candidates.retrieved_slugs` | the insert keeps only slugs whose page still exists, checked under the same row lock the feedback writers take |
-| A connector, GitHub or Google sync deletes a page (`deletePages`) | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction |
+| A connector, GitHub or Google sync deletes a page (`deletePages`), or a transcript re-ingest deletes a stale part | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction |
| `gbrain sources remove`, `sources purge`, the expired-archive sweep, or a managed `remove`/`purge` | the source's retrieval logs, ratings, volunteer rows and captured eval slugs stayed | deleted in the statement that deletes the source, and swept again right after for a write that raced it |
| `gbrain sources federate` or `sources restore` on a source with a remote-written `visibility: private` page and no readers | the warning named that page, though no-grant connections still cannot read it | the warning applies the read paths' privacy rules, unless `search.remote_private_pages` or `GBRAIN_REMOTE_PRIVATE_PAGES` relaxes them |
- `src/core/feedback/store.ts`: `insertEvalCandidate` is the one eval capture insert both engines run.
-- `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.
+- `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. The transcript importer's stale-part delete now calls `deletePages`. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.
- `src/core/source-delete.ts`: `removedSourceScrubCtes` runs inside `deleteSourceRow` and the managed lifecycle's source delete, then once more ungated. `retrieval_weights` and `mention_gazetteer_entries` already cascaded from `sources`.
- `src/core/access/federation-exposure.ts`: the exposure read adds `privacyPartSql` unless `remotePrivacyRelaxed`.
- Tests: `test/purge-scrub-followups-9068.test.ts` and `test/federation-exposure-private-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`). `test/eval-candidates.test.ts` now seeds the pages its round-trip captures, since a capture keeps only live slugs.
diff --git a/docs/architecture/key-files/core-services-3.md b/docs/architecture/key-files/core-services-3.md
index b314a056a4..ac65f34f00 100644
--- a/docs/architecture/key-files/core-services-3.md
+++ b/docs/architecture/key-files/core-services-3.md
@@ -27,7 +27,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/verbs/usage-log.ts` — observability sidecar: one JSONL line per verb call at `~/.gbrain/integrations/memory-verbs/usage.jsonl` (gbrainPath — GBRAIN_HOME honored; `brainId()` = the resolved gbrain home). LOCAL ONLY, never uploaded, stats-only (lock-free 10MB rotation may drop lines; O_APPEND line-atomic, best-effort on Windows). `logVerbUsage` is fire-and-forget (never awaited, never throws); written from the DISPATCH layer so param-validation failures count. `readVerbUsage`/`earliestVerbUsageTs` feed `gbrain protocol stats` (incl. measured TTHW vs the init-stamped `protocol_installed_at`) and the doctor `memory_verbs_usage` check.
-- `src/core/feedback/` — use-attributed retrieval feedback (guide: `docs/guides/retrieval-feedback.md`). `settings.ts` resolves `feedback.*` with a 30s cache (`effectiveInfluence` is 0 when off). `record.ts`: `recordAnswer(ctx, { op, pages, links })` mints `ans_<ULID>`, returns it at once and enqueues the event (pages with the retrieved `content_hash`, relational path edges) on a bounded write-behind queue (cap 1,000, drop-oldest counted, drained by the background-work registry as `retrieval-feedback`); the citation signal (rating 4, half alpha) rides the same queue. Only callers that may change shared ranking record: `canTeachSource` allows the owner's CLI (`remote === false`) and auth-less stdio, and remote callers with write scope on that source, no slug fence, no operation allowlist excluding `rate_answer`, not delegated. `recordThinkAnswer` marks cited pages (synthesis `ok`, bare slug unambiguous across gathered sources). `feedbackMetaFields` adds `answer_id`/`feedback` only to rateable answers (callers that cannot rate see unchanged output). `rate.ts` `rateAnswer` implements `rate_answer`: whole-answer ratings (think/synthesize cited pages, else gathered; search-shaped returned pages; plus path edges into rated pages) or targeted `pages: [{ ref, rating }]` (pages only); skips `stale_revision`/`unverifiable_revision`/`not_authorized_source`; receipts carry weight before/after and the multiplier; refusals are registered codes. `store.ts` owns all SQL (TEXT[] binds, booleans bound as text): `insertRetrievalEvents`, `applyRatings` (one statement: idempotent feedback insert per element, then an upsert whose `DO UPDATE` recomputes from the locked row, so concurrent ratings never lose an update), `readWeights`, `pruneRetrievalFeedback` (purge phase), `resetRetrievalWeights`, `feedbackStatus`, `scrubPurgedSlugs` (#8821: every hard delete of a page row, the page purge, the expired-tombstone purge and the engines' `deletePages` (connector, GitHub and Google deletes; #9068), removes a deleted slug's source-scoped rows as a page key or either end of an edge key, and drops it from `eval_candidates.retrieved_slugs` (`scrubCandidateSlugs`), in the delete's transaction; the page-purge receipt checks each store), `insertEvalCandidate` (#9068: the eval capture insert, keeping only slugs whose page row still exists, probed under the same key-share lock as the feedback writers). Pinned by `test/feedback-store.test.ts` and `test/e2e/feedback-parity.test.ts`.
+- `src/core/feedback/` — use-attributed retrieval feedback (guide: `docs/guides/retrieval-feedback.md`). `settings.ts` resolves `feedback.*` with a 30s cache (`effectiveInfluence` is 0 when off). `record.ts`: `recordAnswer(ctx, { op, pages, links })` mints `ans_<ULID>`, returns it at once and enqueues the event (pages with the retrieved `content_hash`, relational path edges) on a bounded write-behind queue (cap 1,000, drop-oldest counted, drained by the background-work registry as `retrieval-feedback`); the citation signal (rating 4, half alpha) rides the same queue. Only callers that may change shared ranking record: `canTeachSource` allows the owner's CLI (`remote === false`) and auth-less stdio, and remote callers with write scope on that source, no slug fence, no operation allowlist excluding `rate_answer`, not delegated. `recordThinkAnswer` marks cited pages (synthesis `ok`, bare slug unambiguous across gathered sources). `feedbackMetaFields` adds `answer_id`/`feedback` only to rateable answers (callers that cannot rate see unchanged output). `rate.ts` `rateAnswer` implements `rate_answer`: whole-answer ratings (think/synthesize cited pages, else gathered; search-shaped returned pages; plus path edges into rated pages) or targeted `pages: [{ ref, rating }]` (pages only); skips `stale_revision`/`unverifiable_revision`/`not_authorized_source`; receipts carry weight before/after and the multiplier; refusals are registered codes. `store.ts` owns all SQL (TEXT[] binds, booleans bound as text): `insertRetrievalEvents`, `applyRatings` (one statement: idempotent feedback insert per element, then an upsert whose `DO UPDATE` recomputes from the locked row, so concurrent ratings never lose an update), `readWeights`, `pruneRetrievalFeedback` (purge phase), `resetRetrievalWeights`, `feedbackStatus`, `scrubPurgedSlugs` (#8821: every hard delete of a page row, the page purge, the expired-tombstone purge and the engines' `deletePages` (connector, GitHub and Google deletes, the transcript importer's stale parts; #9068), removes a deleted slug's source-scoped rows as a page key or either end of an edge key, and drops it from `eval_candidates.retrieved_slugs` (`scrubCandidateSlugs`), in the delete's transaction; the page-purge receipt checks each store), `insertEvalCandidate` (#9068: the eval capture insert, keeping only slugs whose page row still exists, probed under the same key-share lock as the feedback writers). Pinned by `test/feedback-store.test.ts` and `test/e2e/feedback-parity.test.ts`.
- `src/core/ops/feedback.ts` — the `rate_answer` op (scope `write`, CLI `gbrain rate <answer_id> <rating>`), delegating to `feedback/rate.ts`. Answers carry `answer_id` from `ops/search.ts` (`search`, `query` via `buildRetrievalResponseMeta`), `ops/takes.ts` (`think`), `verbs.ts` (`synthesize`) and `ops/facts.ts` (`recall` page arm on the hybrid path).
diff --git a/src/core/transcripts/ingest.ts b/src/core/transcripts/ingest.ts
index 083743e1bd..4e8e0672ed 100644
--- a/src/core/transcripts/ingest.ts
+++ b/src/core/transcripts/ingest.ts
@@ -530,7 +530,7 @@ async function importSession(engine: BrainEngine, opts: TranscriptsIngestOpts, r
// #5235: on a managed brain the stale part goes through the coordinator like the parts
// themselves, bound to its revision; the direct delete is refused there.
if (coordinator) await deleteStaleTranscriptPart(coordinator, engine, opts.sourceId, row.slug);
- else await engine.deletePage(row.slug, { sourceId: opts.sourceId });
+ else await engine.deletePages([row.slug], { sourceId: opts.sourceId });
result.partsDeleted++;
}
}
diff --git a/test/purge-scrub-followups-9068.test.ts b/test/purge-scrub-followups-9068.test.ts
index 9fb3d3d2b7..317814ba75 100644
--- a/test/purge-scrub-followups-9068.test.ts
+++ b/test/purge-scrub-followups-9068.test.ts
@@ -6,21 +6,21 @@
* page purge stores only slugs whose page still exists, and on Postgres one that runs while the purge is still open
* waits on the page row and drops the slug; (b) the connector delete (deleteConnectorPage -> deletePages) scrubs
* retrieval logs, ratings, weights, the volunteer log, the gazetteer and the eval candidates' captured slugs for
- * that source in the delete's transaction;
+ * that source in the delete's transaction, and so does the transcript stale-part delete, which now calls it;
* (c) removing a source (unmanaged removeSource -> deleteSourceRow, and the managed lifecycle remove) drops every
* such row of the source, and strips its slugs from eval candidates no other covered source still answers.
* 2. Fails when: logEvalCandidate inserts slugs without probing the page row under a key-share lock, a hard delete or a
* source removal leaves any of those rows behind, or a scrub reaches another source's rows or a slug another
* source the candidate covers still holds.
* 3. Existing coverage: purge-retrieval-feedback-8821 and purge-name-stores-9068 pin delete_page purge and the
- * tombstone purge only; nothing pinned the capture, the connector delete, or source removal.
+ * tombstone purge only; nothing pinned the capture, the connector and stale-part deletes, or source removal.
* 4. Real seams: PGLite, and Postgres when DATABASE_URL is set (test/postgres-unit-arms.txt); the feedback store's
- * writers, captureEvalCandidate under withResponseDeferral, the delete_page op, deleteConnectorPage, removeSource,
+ * writers, captureEvalCandidate under withResponseDeferral, the delete_page op, deleteConnectorPage, runTranscriptsIngest, removeSource,
* runManagedSourceLifecycle; repo-less brain, no provider calls.
*/
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { randomUUID } from 'node:crypto';
-import { mkdirSync, mkdtempSync, rmSync } from 'node:fs';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import type { BrainEngine } from '../src/core/engine.ts';
@@ -35,6 +35,8 @@ import { captureEvalCandidate } from '../src/core/eval-capture.ts';
import { withResponseDeferral } from '../src/core/background-work.ts';
import { deleteConnectorPage } from '../src/core/connectors/api-source.ts';
import { removeSource } from '../src/core/sources-ops.ts';
+import { runTranscriptsIngest } from '../src/core/transcripts/ingest.ts';
+import { buildTranscriptSlug } from '../src/core/transcripts/types.ts';
import { registerLocalWriter } from '../src/core/persistence/identity.ts';
import { claimWorktree } from '../src/core/persistence/ownership.ts';
import { runManagedSourceLifecycle } from '../src/core/persistence/source-lifecycle.ts';
@@ -64,9 +66,14 @@ async function page(engine: BrainEngine, sourceId: string, slug: string) {
await importFromContent(engine, slug, `---\ntitle: ${slug}\ntype: note\n---\n# ${slug}\n\nBody text ${randomUUID()}.\n`, { noEmbed: true, sourceId });
}
-/** The page and OTHER in `sourceId`, and every store that names the page: a retrieval event (page and edge), page and edge ratings, a volunteer pointer, a gazetteer entry. */
+/** The page and OTHER in `sourceId`, and every store that names the page (traces). */
async function traced(engine: BrainEngine, sourceId: string, slug: string) {
await page(engine, sourceId, slug);
+ await traces(engine, sourceId, slug);
+}
+
+/** OTHER in `sourceId`, and for an existing page a retrieval event (page and edge), page and edge ratings, a volunteer pointer, a gazetteer entry. */
+async function traces(engine: BrainEngine, sourceId: string, slug: string) {
await page(engine, sourceId, OTHER);
const id = randomUUID(), edge = `${slug}|mentions|${OTHER}`;
await insertRetrievalEvents(engine, [{ id, client_id: 'local', op: 'query', links: [{ source_id: sourceId, edge_key: edge, to_slug: OTHER }],
@@ -180,6 +187,39 @@ describe('#9068 the batch hard delete scrubs what names the page', () => {
});
+/** An openclaw session file with `count` large messages, so a low page ceiling splits it into parts. */
+function session(dir: string, id: string, count: number): string {
+ const filler = 'lorem widget fact '.repeat(220);
+ const at = (i: number) => `2026-08-10T08:${String(Math.floor(i / 60)).padStart(2, '0')}:${String(i % 60).padStart(2, '0')}.000Z`;
+ const lines = [JSON.stringify({ type: 'session', version: 3, id, timestamp: '2026-08-10T08:00:00.000Z', cwd: '/tmp' }),
+ ...Array.from({ length: count }, (_, i) => JSON.stringify({ type: 'message', id: `m-${i}`, timestamp: at(i),
+ message: { role: i % 2 === 0 ? 'user' : 'assistant', timestamp: at(i), content: [{ type: 'text', text: `marker-${i} ${filler}` }] } }))];
+ const path = join(dir, `${id}-${count}.jsonl`);
+ writeFileSync(path, lines.join('\n') + '\n');
+ return path;
+}
+
+describe('#9068 the transcript stale-part delete scrubs what names the part', () => {
+ test('a re-ingest that shrinks a split session deletes its stale parts with their rows', () => withEnv({ GBRAIN_HOME: home }, async () => {
+ for (const engine of engines) {
+ const dir = mkdtempSync(join(home, 'sessions-')), id = `shrink-${engine.kind}`;
+ const ingest = (path: string) => runTranscriptsIngest(engine, { paths: [path], sourceId: 'default', userPatternsPath: '/nonexistent-patterns.txt' });
+ await engine.setConfig('pages.max_bytes', '100000');
+ let parts: number;
+ try { parts = (await ingest(session(dir, id, 150))).pages.imported; } finally { await engine.unsetConfig('pages.max_bytes'); }
+ expect(parts).toBeGreaterThan(1);
+ const stale = `${buildTranscriptSlug('openclaw', '2026-08-10T08:00:00.000Z', { sessionId: id })}-p2`;
+ await traces(engine, 'default', stale);
+ const query = `stale part ${randomUUID()}`;
+ await engine.logEvalCandidate(candidate(query, [stale, OTHER], ['default']));
+ expect((await ingest(session(dir, id, 2))).partsDeleted).toBe(parts - 1);
+ expect(await engine.executeRaw(`SELECT 1 FROM pages WHERE source_id = 'default' AND slug = $1`, [stale])).toHaveLength(0);
+ expect({ engine: engine.kind, residue: await residue(engine, 'default', stale) }).toEqual({ engine: engine.kind, residue: NONE });
+ expect(await capturedSlugs(engine, query)).toEqual([[OTHER]]);
+ }
+ }), 180_000);
+});
+
describe('#9068 removing a source scrubs every row that names its pages', () => {
test('unmanaged removeSource drops the source\'s rows and strips its slugs from eval candidates', () => withEnv({ GBRAIN_HOME: home }, async () => {
for (const engine of engines) {