Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
doctor conversation_format_coverage: sample granular conversation types
garrytan6cd0af50d6f2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 18cebebfef..1fa67291b8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -17,6 +17,7 @@ identifiers and attribution are available in the pre-removal Git revision### Itemized changes- `timeline_history`: on a brain with more than 10,000 pages carrying database-only timeline rows, a pass keeps only the first 10,000 for its final recount. It used to report that recount as exact, and a pass whose kept pages had been repaired was recorded as clean while the rest still held database-only rows. Such a pass now reports its counts as a lower bound ("at least"), keeps warning, and is never trusted as clean. Reported by @Masashi-Ono0611 (#8511).+- `conversation_format_coverage`: the sample now includes pages stored under the granular conversation types the extractor already reads (`slack-thread`, `slack-dm-day`, `email-digest`), so unparseable transcripts on a brain that never ran the type-consolidation pack are reported instead of silently skipped. Reported by @Masashi-Ono0611 (#8519).## [0.60.620.0] - 2026-11-09diff --git a/docs/architecture/key-files/core-persistence-facts.md b/docs/architecture/key-files/core-persistence-facts.mdindex 28f34718f7..e8eb3490d1 100644--- a/docs/architecture/key-files/core-persistence-facts.md+++ b/docs/architecture/key-files/core-persistence-facts.md@@ -19,6 +19,7 @@ Managed facts extraction: the entity-intent preparer and publisher, conversation- `src/commands/doctor/checks/fact-bulk-dates.ts` — `fact_bulk_dates` (wave check, #7247): `findBulkFactDays` over the scoped sources (remote callers: their permitted sources); warns per day with fact and entity counts, top source tags and `dated_claims`, says the dates record loading, not occurrence, and names the `gbrain repair fact-claim-dates --source <id>` preview. Read-only.+- `src/commands/doctor/checks/conversation-coverage.ts` — `conversation_format_coverage`: samples up to 50 newest pages per concrete conversation type, the list the extractor reads (`pageTypesForAllowed(ALLOWED_TYPES)`, granular collector aliases such as `slack-thread` included, #8469), keeps the pages `isConversationFactsEligiblePage` admits, parses each without polish or fallback and warns above 10% unmatched transcript-claiming pages. Test: `test/doctor-conversation-granular-coverage-8469.test.ts`.- `src/core/repair/fact-claim-dates.ts` — `gbrain repair fact-claim-dates --source <id> [--day YYYY-MM-DD]` (#7247), explicit-only, preview-bound, destructive (`--yes`). Candidates: active facts on a bulk day (`findBulkFactDays`, or `--day`) whose claim has exactly one earlier ISO date (`claimDate`). A fence row's `valid_from` cell is rewritten in the page's one fence section and published by revision-bound `put_page` (the facts row is re-projected); a database-only fact is updated in one `maintenanceTransaction` on an unmanaged brain and counted `database_only_managed` on a managed one. Pages with an unparsed, split or segmented fence, or a rewrite that does not read back as the planned rows, are listed and kept. The hash binds revisions, row numbers and new dates; a fresh apply recomputes the plan and refuses `preview_changed` on any difference. Test: `test/repair-fact-claim-dates-7247.test.ts`.- `src/core/facts/fence-fact-append.ts` — `prepareFenceFactAppend(engine, row, config, snapshot, { fact, pendingTier, mode?, supersede? | rowNum }, signal)`: the one way a coordinator preparer puts a single fact into the facts fence of the request's page (`row.slug`); remember uses it, and so does a managed owner release of a held fact (`trust/release-publish.ts`, the `trust_release` operation: append for a held fact with an entity slug, index-existing for a hold naming `payload.row_num`, whose cell fields the owner previewed and the request binds: a moved cell refuses `preview_changed`). It returns `{ prepared, rowNum, fencesNormalized, apply(tx) }`: `prepared` is the page write to compose (its `file`, `exclusiveSources`, `validate`; null in index-existing), and `apply` runs in the publication transaction after the coordinator published the file, writing the page row (tier kept, `trust/fence-append.ts`) and then the facts row with `row_num` and `source_markdown_slug`, so a facts row never precedes its fence row in Git. `append` (default) normalizes the stored fence (#6188), renders the row with `fenceRowOf` (`unfenced-facts.ts`, the one renderer) at one past the highest row number in the fence and among the page's facts rows (#6696 C9), strikes `supersede.rowNum` when given, and prepares the page write `unprojected` (`preparePageMutation` option: the helper writes the facts row, so the page publishes without canonical projection whatever the operation) under `withPendingFenceRows` at `pendingTier`. `index-existing` takes a row the file already holds (a fence-guard hold's `payload.row_num`): the cell must pass `fenceCellReadBackProblem` against the fact, else it refuses `fence_row_changed` (reason `row_missing` / `struck` / `empty` / `visibility` / `text`) and writes nothing; otherwise the facts row derives from the cell (`extractFactsFromFenceText`, what a re-import projects) with the caller's embedding and session, and no file changes; the caller's observedRevision is `snapshot.revision`, which the coordinator re-checks under the page lock. An occupied row number fails the insert with `storage_error` naming the occupant. Tests: `test/fence-fact-append.test.ts` (PGLite and Postgres: append, supersede, index-existing from a real fence-guard hold, each refusal, and remember's fence row byte for byte against the earlier construction).diff --git a/src/commands/doctor/checks/conversation-coverage.ts b/src/commands/doctor/checks/conversation-coverage.tsindex f9f9c730b5..5c4afdbfea 100644--- a/src/commands/doctor/checks/conversation-coverage.ts+++ b/src/commands/doctor/checks/conversation-coverage.ts@@ -4,13 +4,13 @@* computeExtractAtomsBacklogCheck pattern. doctor.ts re-exports it under the* facade rule.*- * Scans up to 200 most-recent conversation-type pages (50 per allowed type),+ * Scans the most recent conversation-type pages (50 per concrete type, granular aliases included),* runs parseConversation in dry mode, reports per-pattern hit counts +* unmatched count. Warn at >10% unmatched with paste-ready hint pointing at* `gbrain conversation-parser scan <slug>`.*/import type { BrainEngine } from '../../../core/engine.ts';-import { ALLOWED_TYPES, isConversationFactsEligiblePage, requireParseableConversationFlag } from '../../../core/facts/conversation-types.ts';+import { ALLOWED_TYPES, isConversationFactsEligiblePage, pageTypesForAllowed, requireParseableConversationFlag } from '../../../core/facts/conversation-types.ts';import type { Check } from '../../doctor.ts';import { checkError } from '../check-fix.ts';@@ -24,10 +24,11 @@ export async function computeConversationFormatCoverageCheck(try {const { ConversationBodyTooLargeError, readConversationBodyForParsing } = await import('../../../core/conversation-parser/body.ts');const { parseConversation } = await import('../../../core/conversation-parser/parse.ts');- // Single source of truth for the conversation-facts type allowlist (#4135).- const allowedTypes = ALLOWED_TYPES;- // PageFilters supports singular `type` only; iterate the allowed types- // and cap at ~50/each to land at ~200 total max.+ // Single source of truth for the conversation-facts type allowlist (#4135),+ // expanded to the granular collector types the extractor reads (#8469).+ const allowedTypes = pageTypesForAllowed(ALLOWED_TYPES);+ // PageFilters supports singular `type` only; iterate the concrete types+ // and cap at TYPE_SAMPLE each.const sample: import('../../../core/types.ts').Page[] = [];// #5330: sample only pages the extractor would claim.const strict = await requireParseableConversationFlag(engine);diff --git a/test/doctor-conversation-granular-coverage-8469.test.ts b/test/doctor-conversation-granular-coverage-8469.test.tsnew file mode 100644index 0000000000..6dce45e068--- /dev/null+++ b/test/doctor-conversation-granular-coverage-8469.test.ts@@ -0,0 +1,42 @@+/**+ * #8469: conversation_format_coverage sampled pages by the canonical+ * conversation types only, while extraction expands them to the granular+ * collector types (`slack-thread`, `slack-dm-day`, `email-digest`) through+ * pageTypesForAllowed. Unparseable transcripts stored under a granular type+ * were never sampled, so the check reported nothing about them.+ */+import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { computeConversationFormatCoverageCheck } from '../src/commands/doctor/checks/conversation-coverage.ts';++let engine: PGLiteEngine;++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+}, 60_000);+afterAll(async () => { await engine.disconnect(); });+beforeEach(async () => { await engine.executeRaw('DELETE FROM pages'); });++async function putThread(frontmatter: Record<string, unknown> = {}): Promise<void> {+ await engine.putPage('slack/thread-acme-example', {+ type: 'slack-thread', title: 'Thread with alice-example', timeline: '', frontmatter,+ compiled_truth: ['## Transcript', '', '9.03.12 -- alice-example -- an invented turn', '9.04.55 -- bob-example -- another turn'].join('\n'),+ });+}++test('an unparseable transcript stored under a granular conversation type is sampled', async () => {+ await putThread();+ const check = await computeConversationFormatCoverageCheck(engine);+ expect(check.status).toBe('warn');+ expect(check.message).toContain('_no_match=1');+ expect(check.details).toMatchObject({ sampled: 1 });+});++test('a granular page opted out of extraction stays out of the sample', async () => {+ await putThread({ conversation_parseable: false });+ const check = await computeConversationFormatCoverageCheck(engine);+ expect(check.status).toBe('ok');+ expect(check.message).toContain('No conversation-type pages');+});