Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
doctor schema_pack_consistency: exempt gbrain-written pages by provenance
garrytan67328e5e2ef7
diff --git a/docs/architecture/key-files/schema-mutation.md b/docs/architecture/key-files/schema-mutation.mdindex 4ed9658390..c11a8b8ca5 100644--- a/docs/architecture/key-files/schema-mutation.md+++ b/docs/architecture/key-files/schema-mutation.md@@ -22,7 +22,7 @@ Key files:- `src/core/schema-pack/entity-types.ts` — #4772: the one answer to "is this stored type an entity" for the counters. `entityTypesFromPack(manifest)` returns the pack's `primitive: entity` type names (plus undeclared aliases of them) and the legacy names (`entity`, `person`, `company`, `organization`) the pack leaves undeclared; a pack that declares a legacy name under another primitive reclassified it and wins. `resolveEntityTypes(engine, scope)` resolves per source (every requested source id, or brain-wide: the brain's pack plus every registered source with a `schema_pack.source.<id>` override) into an `EntityTypeFilter` (default list + per-source lists that differ), and returns `pack_unavailable` with an empty filter when any pack in scope fails to load (the `best-effort.ts` null contract: empty filter, never hardcoded defaults). `entityTypePredicateSql(alias, filter)` renders it as a parameter-bound `sqlFragment`. #6953: `personTypesFromPack` / `resolvePersonTypes` answer "is this type person-like" the same way (per source, same `pack_unavailable` contract): `person` plus its alias closure (`closure.ts`), never primitive siblings; consumer `cycle/edge-contradictions.ts`: its `works_at` members are admitted before grouping and LIMIT only when the subject is person-like by its source's pack (a pack that does not load admits none) and the target is neither person-like nor a `NON_EMPLOYER_TYPES` type (untyped: `NON_EMPLOYER_PREFIXES`; both lists shared with `link-target-roles.ts`'s `targetTakesVerb`); an open proposal made before that rule is not re-checked when applied (`test/edge-employment-roles.test.ts` + Postgres twin; `personTypesFromPack` in `test/schema-pack-person-types-6953.test.ts`). Consumers: `engine-sql/health.ts` through `HealthDeps.entityTypes` (both engines delegate), `commands/doctor/checks/graph-health.ts` (`graph_coverage`, `orphan_ratio`), `onboard/checks.ts` (`entity_link_coverage`, `timeline_coverage`, and `visibleEntityCoverageSql` for `onboard/impact-capture.ts`; a pack that does not load warns `entity_types_unavailable`), `onboard/mcp-onboarding.ts` (the three entity counts are `null` with `entityTypesStatus: pack_unavailable`; the timeline result predicate) and `extract-ner.ts` (the target-type map; `pack_unavailable` when a source's pack does not load). `mentions/policy.ts` keeps its own linkable-type union. Pinned by `test/onboard-entity-types-pack-4772.test.ts`. Lazy-imports the pack loader so the engines can import it statically.- `src/core/schema-pack/best-effort.ts` — `loadActivePackBestEffort(ctx)` returns `ResolvedPack | null`. Single source of truth for the pack-aware wiring sites. `null` means EMPTY FILTER (NOT hardcoded defaults, which would silently violate the pack). Threads the tier-4 DB-plane `schema_pack` from `ctx.engine`, so every caller resolves the SAME pack `schema active` / `get_active_schema_pack` report. `readDbSchemaPack(engine?)` is the ONE spelling of that tier-4 read (null-safe, propagates DB read failures, trims missing/empty values to undefined) — shared by both loaders here, the CLI inspection verbs (`schema show/graph/lint/explain`, via `readDbSchemaPackConfig` in `src/commands/schema.ts`, which keeps the cfg-null gate so an unconfigured home never cold-creates a PGLite dir) and the MCP `schema_lint`/`schema_graph`/`schema_explain_type` ops. `loadActivePackForLocalEngine(engine)` is the `remote: false` + FILE-ONLY-config variant for local engine-backed surfaces. Pinned by `test/schema-pack-best-effort.test.ts`, `test/operations-schema-pack.test.ts`, `test/schema-cli-db-config.test.ts`.-- `src/core/schema-pack/type-usage.ts` — stored-type classifier behind the alias-footgun visibility surfaces: `classifyStoredType(type, pack)` → canonical | alias_of (with the canonical type + `path_prefixes[0]` filing directory) | undeclared, over a STRUCTURAL pack shape so import-file's thin `activePack` and the full manifest both satisfy it. `sanitizeTypeForDisplay` strips control chars + caps length (type strings come from frontmatter and get echoed into terminals); `renderTypeWarningSummary` renders the once-per-type-per-run lines. Consumers: `importFromContent` (advisory `ImportResult.type_warning` at the typeExplicit site — the type is still stored literally, zero filing change), sync/import summary aggregation (+ `SyncResult.type_warnings` so worker-driven syncs surface counts in job results), the `stored_type_is_alias`/`stored_type_undeclared` data-plane lint rules, all gated by config `schema.type_warnings` (default on; lint rules always active). `storedTypeMissesPack(type, pack)` (empty type, or undeclared) is the one predicate every pack-conformance surface uses: `findTypeOrphans` (`schema-pack/review.ts`), `schema_stats` coverage, doctor `schema_pack_consistency`, `gbrain lint`'s `type-undeclared` rule. `WRITER_OWNED_PAGE_TYPES` (`code`, `extract_receipt`: types gbrain's own writers stamp) are exempt from doctor `schema_pack_consistency`'s undeclared count, so a custom pack copied before gbrain-base-v2 declared them (#6818) is not reported as drift (#8815, `test/doctor-schema-pack-own-types-8815.test.ts`). Pinned by `test/schema-type-usage.test.ts` and `test/schema-pack-type-conformance.test.ts`.+- `src/core/schema-pack/type-usage.ts` — stored-type classifier behind the alias-footgun visibility surfaces: `classifyStoredType(type, pack)` → canonical | alias_of (with the canonical type + `path_prefixes[0]` filing directory) | undeclared, over a STRUCTURAL pack shape so import-file's thin `activePack` and the full manifest both satisfy it. `sanitizeTypeForDisplay` strips control chars + caps length (type strings come from frontmatter and get echoed into terminals); `renderTypeWarningSummary` renders the once-per-type-per-run lines. Consumers: `importFromContent` (advisory `ImportResult.type_warning` at the typeExplicit site — the type is still stored literally, zero filing change), sync/import summary aggregation (+ `SyncResult.type_warnings` so worker-driven syncs surface counts in job results), the `stored_type_is_alias`/`stored_type_undeclared` data-plane lint rules, all gated by config `schema.type_warnings` (default on; lint rules always active). `storedTypeMissesPack(type, pack)` (empty type, or undeclared) is the one predicate every pack-conformance surface uses: `findTypeOrphans` (`schema-pack/review.ts`), `schema_stats` coverage, doctor `schema_pack_consistency`, `gbrain lint`'s `type-undeclared` rule. `WRITER_OWNED_PAGE_SQL` exempts pages gbrain's own writers wrote, judged by provenance (a code importer page, `page_kind = 'code'`; a `dream_generated` `extract_receipt` under `extracts/`), from doctor `schema_pack_consistency`'s undeclared count, so a custom pack copied before gbrain-base-v2 declared those types (#6818) is not reported as drift, while a user's Markdown page typed `code` still is (#8815, `test/doctor-schema-pack-own-types-8815.test.ts`). Pinned by `test/schema-type-usage.test.ts` and `test/schema-pack-type-conformance.test.ts`.- `src/core/schema-pack/lint-rules.ts` — 14 pure rule functions. `withMutation`'s pre-write validation gate composes the 10 file-plane rules; the 4 DB-aware rules (`extractable_empty_corpus`, `mutation_count_anomaly`, `stored_type_is_alias`, `stored_type_undeclared`) need an engine (CLI `--with-db`; the stored-type pair accepts `LintOpts.sourceId` scoping, not yet threaded from the CLI). Single source of truth consumed by CLI lint + MCP `schema_lint` + the pre-write validation gate. File-plane rule `link_regex_catastrophic_backtrack` — advisory ReDoS pre-screen flagging the classic nested-quantifier shapes (`(a+)+`, `(a*)*`, `(a+)*`, `(\w+)+`) in a link_type's `inference.regex` via `NESTED_QUANTIFIER_RE`. WARNING not error: a hard reject would disable the whole pack on upgrade (pages fall back to legacy typing). The runtime input-length cap in `redos-guard.ts` is the actual safety net; this rule tells the pack author to fix the pattern.- `src/core/schema-pack/redos-guard.ts` + `src/core/schema-pack/link-inference.ts` — ReDoS hardening for pack inference regexes. `redos-guard.ts` provides `MAX_REGEX_INPUT_CHARS` (default 64_000, env `GBRAIN_MAX_REGEX_INPUT_CHARS`) — a hard input-length cap, the real runtime safety net (catastrophic backtracking needs a long input; a link-extraction `context` is normally a sentence or short paragraph). Over the cap, `runRegexBounded` throws the tagged `RegexInputTooLargeError` and the regex is skipped (degrade-to-mentions) without entering the `node:vm`. `link-inference.ts:inferLinkTypeFromPack` no-budget branch (test contexts) routes through `runRegexBounded` so the input-length cap + per-regex vm timeout (`PER_REGEX_TIMEOUT_MS = 50`) apply on every path. Defensive hardening + diagnostics; the deterministic ~3100-file sync-wedge root cause remains open. Pinned by `test/redos-hardening.test.ts` + `test/schema-pack-lint-rules.test.ts`.- `src/core/schema-pack/query-cache-invalidator.ts` — `invalidateQueryCache(engine, sourceId?)` DELETEs query_cache rows so cached search results bound to old page types don't survive a schema mutation.diff --git a/src/commands/doctor/schema-pack-checks.ts b/src/commands/doctor/schema-pack-checks.tsindex 26a7b9c98d..c85a179367 100644--- a/src/commands/doctor/schema-pack-checks.ts+++ b/src/commands/doctor/schema-pack-checks.ts@@ -11,7 +11,7 @@ import { statSync } from 'node:fs';import { redactConnectionInfo } from '../../core/audit/redact-connection-info.ts';import { formatPackBytes, OVERSIZED_PACK_RUNBOOK, schemaPackMaxBytes } from '../../core/schema-pack/loader.ts';import { loadActivePackForLocalEngine } from '../../core/schema-pack/best-effort.ts';-import { WRITER_OWNED_PAGE_TYPES, sanitizeTypeForDisplay, storedTypeMissesPack } from '../../core/schema-pack/type-usage.ts';+import { WRITER_OWNED_PAGE_SQL, sanitizeTypeForDisplay, storedTypeMissesPack } from '../../core/schema-pack/type-usage.ts';// =================================================================// v0.39 T7 + T9 — schema-pack doctor checks@@ -104,8 +104,8 @@ interface SourceConformance {* #5879: pages match the active pack only when their stored type is a* declared page type or an alias of one (`storedTypeMissesPack`, the same* classification as `schema lint --with-db`). Each source is graded against- * its own resolved pack. Any undeclared type warns, except gbrain's own- * page types (`WRITER_OWNED_PAGE_TYPES`, #8815); untyped pages warn at+ * its own resolved pack. Any undeclared type warns, except on pages gbrain's+ * own writers wrote (`WRITER_OWNED_PAGE_SQL`, #8815); untyped pages warn at* >= 10% of a source.*/export async function checkSchemaPackConsistency(@@ -113,14 +113,14 @@ export async function checkSchemaPackConsistency(opts: { sourceIds?: string[] } = {},): Promise<Check> {const scoped = opts.sourceIds !== undefined;- let rows: Array<{ src: string; type: string | null; n: string }>;+ let rows: Array<{ src: string; type: string | null; own: boolean; n: string }>;try {rows = await engine.executeRaw(- `SELECT COALESCE(source_id, 'default') AS src, type, COUNT(*)::text AS n- FROM pages- WHERE deleted_at IS NULL${scoped ? ' AND source_id = ANY($1::text[])' : ''}- GROUP BY source_id, type- ORDER BY source_id`,+ `SELECT COALESCE(p.source_id, 'default') AS src, p.type, ${WRITER_OWNED_PAGE_SQL} AS own, COUNT(*)::text AS n+ FROM pages p+ WHERE p.deleted_at IS NULL${scoped ? ' AND p.source_id = ANY($1::text[])' : ''}+ GROUP BY p.source_id, p.type, own+ ORDER BY p.source_id`,scoped ? [opts.sourceIds] : [],);} catch (e) {@@ -129,10 +129,10 @@ export async function checkSchemaPackConsistency(if (rows.length === 0) {return { name: 'schema_pack_consistency', status: 'ok', message: 'No pages in any source — schema consistency N/A.' };}- const bySource = new Map<string, Array<{ type: string | null; n: number }>>();+ const bySource = new Map<string, Array<{ type: string | null; own: boolean; n: number }>>();for (const r of rows) {const list = bySource.get(r.src) ?? [];- list.push({ type: r.type, n: Number(r.n) });+ list.push({ type: r.type, own: r.own === true, n: Number(r.n) });bySource.set(r.src, list);}const graded: SourceConformance[] = [];@@ -141,7 +141,7 @@ export async function checkSchemaPackConsistency(const pack = await loadActivePackForLocalEngine(engine, { sourceId: src });if (!pack) { unresolved.push(src); continue; }const undeclaredTypes = types- .filter((t): t is { type: string; n: number } => !!t.type && !WRITER_OWNED_PAGE_TYPES.has(t.type) && storedTypeMissesPack(t.type, pack.manifest))+ .filter((t): t is { type: string; own: boolean; n: number } => !!t.type && !t.own && storedTypeMissesPack(t.type, pack.manifest)).map((t) => ({ type: t.type, count: t.n })).sort((a, b) => b.count - a.count || a.type.localeCompare(b.type));graded.push({diff --git a/src/core/schema-pack/type-usage.ts b/src/core/schema-pack/type-usage.tsindex a483358f48..3e45a46300 100644--- a/src/core/schema-pack/type-usage.ts+++ b/src/core/schema-pack/type-usage.ts@@ -62,12 +62,15 @@ export function storedTypeMissesPack(type: string | null | undefined, pack: Type}/**- * Page types gbrain's own writers stamp outside the content taxonomy: `code`- * (the code importer) and `extract_receipt` (the extract receipt writer under- * extracts/). gbrain-base-v2 declares both since #6818; a custom pack copied- * before then lacks them, which is not the owner's schema drift (#8815).+ * SQL predicate over `pages p`: the page was written by one of gbrain's own+ * writers outside the content taxonomy, judged by how it was written rather+ * than by its type name. A code importer page carries page_kind 'code'; an+ * extract receipt is an `extract_receipt` under extracts/ stamped+ * dream_generated. gbrain-base-v2 declares both types since #6818; a custom+ * pack copied before then lacks them, which is not the owner's schema drift+ * (#8815). A user's Markdown page typed `code` matches neither.*/-export const WRITER_OWNED_PAGE_TYPES: ReadonlySet<string> = new Set(['code', 'extract_receipt']);+export const WRITER_OWNED_PAGE_SQL = `(p.page_kind = 'code' OR (p.type = 'extract_receipt' AND p.slug LIKE 'extracts/%' AND p.frontmatter->>'dream_generated' = 'true'))`;/*** Type strings come from user/agent frontmatter and get echoed into terminaldiff --git a/test/doctor-schema-pack-own-types-8815.test.ts b/test/doctor-schema-pack-own-types-8815.test.tsindex 078f1f0c1b..0bd4de3d30 100644--- a/test/doctor-schema-pack-own-types-8815.test.ts+++ b/test/doctor-schema-pack-own-types-8815.test.ts@@ -3,7 +3,9 @@* lacks as user schema drift, so a custom pack copied from gbrain-base-v2* before it declared gbrain's own page types (`extract_receipt`, `code`,* #6818) warned `page_type_undeclared` about pages gbrain itself writes.- * Those types are exempt; a user's own undeclared type still warns.+ * Pages gbrain wrote are exempt, judged by how they were written (a code+ * importer page, a dream_generated receipt under extracts/), not by type+ * name: a user's Markdown page typed `code` or `extract_receipt` still warns.*/import { afterAll, beforeAll, expect, test } from 'bun:test';import { mkdirSync, rmSync, writeFileSync } from 'node:fs';@@ -11,6 +13,7 @@ import { join } from 'node:path';import { PGLiteEngine } from '../src/core/pglite-engine.ts';import { checkSchemaPackConsistency } from '../src/commands/doctor/schema-pack-checks.ts';import { writeReceipt } from '../src/core/extract/receipt-writer.ts';+import { importCodeFile, importFromContent } from '../src/core/import-file.ts';import { emptyHome, withEnv } from './helpers/with-env.ts';const PACK = `api_version: gbrain-schema-pack-v1@@ -51,7 +54,7 @@ const consistency = () => withEnv({ GBRAIN_HOME: home, GBRAIN_SCHEMA_PACK: undeftest('gbrain-owned page types are not undeclared drift on a pack that predates them; a user type still is', async () => {await writeReceipt(engine, { kind: 'links', source_id: 'default', run_id: 'a1b2c3d4e5f6789abcdef', round: 'full', extracted_at: '2026-05-27T14:30:00.000Z', total_rows: 3, cost_usd: 0 });- await engine.putPage('src/example.ts', { type: 'code', title: 'example.ts', compiled_truth: 'export const x = 1;' }, { sourceId: 'default' });+ await importCodeFile(engine, 'src/example.ts', 'export const x = 1;\n', { noEmbed: true });const own = await consistency();expect(own.message).not.toContain('extract_receipt');expect(own.message).not.toContain("'code'");@@ -63,3 +66,19 @@ test('gbrain-owned page types are not undeclared drift on a pack that predates texpect(user.message).toContain('custom_fixture');expect(user.message).not.toContain('extract_receipt');});++test('a user Markdown page typed like a gbrain page is still undeclared drift', async () => {+ await engine.executeRaw('DELETE FROM pages');+ await writeReceipt(engine, { kind: 'links', source_id: 'default', run_id: 'b1c2d3e4f5a6789abcdef', round: 'full', extracted_at: '2026-05-27T14:30:00.000Z', total_rows: 3, cost_usd: 0 });+ await importCodeFile(engine, 'src/example.ts', 'export const x = 1;\n', { noEmbed: true });+ expect((await consistency()).status).toBe('ok');++ await importFromContent(engine, 'notes/snippet', '---\ntitle: Snippet\ntype: code\n---\nA pasted snippet.\n', { noEmbed: true });+ const code = await consistency();+ expect(code.status).toBe('warn');+ expect(code.message).toContain("'code' (1)");++ await importFromContent(engine, 'notes/receipt-like', '---\ntitle: Receipt like\ntype: extract_receipt\n---\nA user note.\n', { noEmbed: true });+ const receipt = await consistency();+ expect(receipt.message).toContain("'extract_receipt' (1)");+});