Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

doctor conversation_format_coverage: sample granular conversation types

garrytan6cd0af50d6f2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 18cebebfef..1fa67291b8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,7 @@ identifiers and attribution are available in the pre-removal Git revision
### Itemized changes
- `timeline_history`: on a brain with more than 10,000 pages carrying database-only timeline rows, a pass keeps only the first 10,000 for its final recount. It used to report that recount as exact, and a pass whose kept pages had been repaired was recorded as clean while the rest still held database-only rows. Such a pass now reports its counts as a lower bound ("at least"), keeps warning, and is never trusted as clean. Reported by @Masashi-Ono0611 (#8511).
+- `conversation_format_coverage`: the sample now includes pages stored under the granular conversation types the extractor already reads (`slack-thread`, `slack-dm-day`, `email-digest`), so unparseable transcripts on a brain that never ran the type-consolidation pack are reported instead of silently skipped. Reported by @Masashi-Ono0611 (#8519).
## [0.60.620.0] - 2026-11-09
diff --git a/docs/architecture/key-files/core-persistence-facts.md b/docs/architecture/key-files/core-persistence-facts.md
index 28f34718f7..e8eb3490d1 100644
--- a/docs/architecture/key-files/core-persistence-facts.md
+++ b/docs/architecture/key-files/core-persistence-facts.md
@@ -19,6 +19,7 @@ Managed facts extraction: the entity-intent preparer and publisher, conversation
- `src/commands/doctor/checks/fact-bulk-dates.ts` — `fact_bulk_dates` (wave check, #7247): `findBulkFactDays` over the scoped sources (remote callers: their permitted sources); warns per day with fact and entity counts, top source tags and `dated_claims`, says the dates record loading, not occurrence, and names the `gbrain repair fact-claim-dates --source <id>` preview. Read-only.
+- `src/commands/doctor/checks/conversation-coverage.ts` — `conversation_format_coverage`: samples up to 50 newest pages per concrete conversation type, the list the extractor reads (`pageTypesForAllowed(ALLOWED_TYPES)`, granular collector aliases such as `slack-thread` included, #8469), keeps the pages `isConversationFactsEligiblePage` admits, parses each without polish or fallback and warns above 10% unmatched transcript-claiming pages. Test: `test/doctor-conversation-granular-coverage-8469.test.ts`.
- `src/core/repair/fact-claim-dates.ts` — `gbrain repair fact-claim-dates --source <id> [--day YYYY-MM-DD]` (#7247), explicit-only, preview-bound, destructive (`--yes`). Candidates: active facts on a bulk day (`findBulkFactDays`, or `--day`) whose claim has exactly one earlier ISO date (`claimDate`). A fence row's `valid_from` cell is rewritten in the page's one fence section and published by revision-bound `put_page` (the facts row is re-projected); a database-only fact is updated in one `maintenanceTransaction` on an unmanaged brain and counted `database_only_managed` on a managed one. Pages with an unparsed, split or segmented fence, or a rewrite that does not read back as the planned rows, are listed and kept. The hash binds revisions, row numbers and new dates; a fresh apply recomputes the plan and refuses `preview_changed` on any difference. Test: `test/repair-fact-claim-dates-7247.test.ts`.
- `src/core/facts/fence-fact-append.ts` — `prepareFenceFactAppend(engine, row, config, snapshot, { fact, pendingTier, mode?, supersede? | rowNum }, signal)`: the one way a coordinator preparer puts a single fact into the facts fence of the request's page (`row.slug`); remember uses it, and so does a managed owner release of a held fact (`trust/release-publish.ts`, the `trust_release` operation: append for a held fact with an entity slug, index-existing for a hold naming `payload.row_num`, whose cell fields the owner previewed and the request binds: a moved cell refuses `preview_changed`). It returns `{ prepared, rowNum, fencesNormalized, apply(tx) }`: `prepared` is the page write to compose (its `file`, `exclusiveSources`, `validate`; null in index-existing), and `apply` runs in the publication transaction after the coordinator published the file, writing the page row (tier kept, `trust/fence-append.ts`) and then the facts row with `row_num` and `source_markdown_slug`, so a facts row never precedes its fence row in Git. `append` (default) normalizes the stored fence (#6188), renders the row with `fenceRowOf` (`unfenced-facts.ts`, the one renderer) at one past the highest row number in the fence and among the page's facts rows (#6696 C9), strikes `supersede.rowNum` when given, and prepares the page write `unprojected` (`preparePageMutation` option: the helper writes the facts row, so the page publishes without canonical projection whatever the operation) under `withPendingFenceRows` at `pendingTier`. `index-existing` takes a row the file already holds (a fence-guard hold's `payload.row_num`): the cell must pass `fenceCellReadBackProblem` against the fact, else it refuses `fence_row_changed` (reason `row_missing` / `struck` / `empty` / `visibility` / `text`) and writes nothing; otherwise the facts row derives from the cell (`extractFactsFromFenceText`, what a re-import projects) with the caller's embedding and session, and no file changes; the caller's observedRevision is `snapshot.revision`, which the coordinator re-checks under the page lock. An occupied row number fails the insert with `storage_error` naming the occupant. Tests: `test/fence-fact-append.test.ts` (PGLite and Postgres: append, supersede, index-existing from a real fence-guard hold, each refusal, and remember's fence row byte for byte against the earlier construction).
diff --git a/src/commands/doctor/checks/conversation-coverage.ts b/src/commands/doctor/checks/conversation-coverage.ts
index f9f9c730b5..5c4afdbfea 100644
--- a/src/commands/doctor/checks/conversation-coverage.ts
+++ b/src/commands/doctor/checks/conversation-coverage.ts
@@ -4,13 +4,13 @@
* computeExtractAtomsBacklogCheck pattern. doctor.ts re-exports it under the
* facade rule.
*
- * Scans up to 200 most-recent conversation-type pages (50 per allowed type),
+ * Scans the most recent conversation-type pages (50 per concrete type, granular aliases included),
* runs parseConversation in dry mode, reports per-pattern hit counts +
* unmatched count. Warn at >10% unmatched with paste-ready hint pointing at
* `gbrain conversation-parser scan <slug>`.
*/
import type { BrainEngine } from '../../../core/engine.ts';
-import { ALLOWED_TYPES, isConversationFactsEligiblePage, requireParseableConversationFlag } from '../../../core/facts/conversation-types.ts';
+import { ALLOWED_TYPES, isConversationFactsEligiblePage, pageTypesForAllowed, requireParseableConversationFlag } from '../../../core/facts/conversation-types.ts';
import type { Check } from '../../doctor.ts';
import { checkError } from '../check-fix.ts';
@@ -24,10 +24,11 @@ export async function computeConversationFormatCoverageCheck(
try {
const { ConversationBodyTooLargeError, readConversationBodyForParsing } = await import('../../../core/conversation-parser/body.ts');
const { parseConversation } = await import('../../../core/conversation-parser/parse.ts');
- // Single source of truth for the conversation-facts type allowlist (#4135).
- const allowedTypes = ALLOWED_TYPES;
- // PageFilters supports singular `type` only; iterate the allowed types
- // and cap at ~50/each to land at ~200 total max.
+ // Single source of truth for the conversation-facts type allowlist (#4135),
+ // expanded to the granular collector types the extractor reads (#8469).
+ const allowedTypes = pageTypesForAllowed(ALLOWED_TYPES);
+ // PageFilters supports singular `type` only; iterate the concrete types
+ // and cap at TYPE_SAMPLE each.
const sample: import('../../../core/types.ts').Page[] = [];
// #5330: sample only pages the extractor would claim.
const strict = await requireParseableConversationFlag(engine);
diff --git a/test/doctor-conversation-granular-coverage-8469.test.ts b/test/doctor-conversation-granular-coverage-8469.test.ts
new file mode 100644
index 0000000000..6dce45e068
--- /dev/null
+++ b/test/doctor-conversation-granular-coverage-8469.test.ts
@@ -0,0 +1,42 @@
+/**
+ * #8469: conversation_format_coverage sampled pages by the canonical
+ * conversation types only, while extraction expands them to the granular
+ * collector types (`slack-thread`, `slack-dm-day`, `email-digest`) through
+ * pageTypesForAllowed. Unparseable transcripts stored under a granular type
+ * were never sampled, so the check reported nothing about them.
+ */
+import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { computeConversationFormatCoverageCheck } from '../src/commands/doctor/checks/conversation-coverage.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+}, 60_000);
+afterAll(async () => { await engine.disconnect(); });
+beforeEach(async () => { await engine.executeRaw('DELETE FROM pages'); });
+
+async function putThread(frontmatter: Record<string, unknown> = {}): Promise<void> {
+ await engine.putPage('slack/thread-acme-example', {
+ type: 'slack-thread', title: 'Thread with alice-example', timeline: '', frontmatter,
+ compiled_truth: ['## Transcript', '', '9.03.12 -- alice-example -- an invented turn', '9.04.55 -- bob-example -- another turn'].join('\n'),
+ });
+}
+
+test('an unparseable transcript stored under a granular conversation type is sampled', async () => {
+ await putThread();
+ const check = await computeConversationFormatCoverageCheck(engine);
+ expect(check.status).toBe('warn');
+ expect(check.message).toContain('_no_match=1');
+ expect(check.details).toMatchObject({ sampled: 1 });
+});
+
+test('a granular page opted out of extraction stays out of the sample', async () => {
+ await putThread({ conversation_parseable: false });
+ const check = await computeConversationFormatCoverageCheck(engine);
+ expect(check.status).toBe('ok');
+ expect(check.message).toContain('No conversation-type pages');
+});