Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(git-gap): the census report refuses a source with a path whose ignore status Git cannot read, as before #7617 (#7618)
garrytan10412ead269f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex de1121752b..8b2453a1be 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -54,7 +54,7 @@ GBRAIN_MEMORY_TRACE=5 gbrain sync --source <source> # the same memory t- `src/core/export-snapshot.ts`: a batched export read looks each page up by its primary key, so `gbrain export` and the preview stay linear on large sources.- `src/commands/sources-canonical.ts`: progress lines, the partial report on signal, the push-pending notice.- `src/core/memory-trace.ts`: `GBRAIN_MEMORY_TRACE=<seconds>` (1 to 3600) writes a memory line on stderr during a conversion apply and during `gbrain sync`.-- `src/commands/sources-git-gap.ts`, `src/core/persistence/git-gap.ts`: the census reads a withheld page's file at its withheld path; a path whose ignore status Git cannot read no longer refuses the whole census, and `gbrain sources git-gap` and doctor's `db_ahead_of_git` count it as `ignore_status_unknown` instead.+- `src/commands/sources-git-gap.ts`, `src/core/persistence/git-gap.ts`: the census reads a withheld page's file at its withheld path; in the conversion plan a path whose ignore status Git cannot read no longer refuses the whole census, while `gbrain sources git-gap` and doctor's `db_ahead_of_git` still refuse that source and name the path.- `scripts/bench/sync-pass-memory.ts`: a memory probe for the strict-equivalent pass and the conversion (seed, run, cli, embed-seed).- Tests: `test/git-canonical-conversion-g8a.test.ts` (crash, refusal and withhold cases on PGLite and Postgres), `test/git-canonical-conversion-progress.test.ts`, `test/git-gap-census.test.ts`, and a Postgres smoke test for the memory probe.diff --git a/docs/designs/GIT_CANONICAL_MEMORY.md b/docs/designs/GIT_CANONICAL_MEMORY.mdindex b68dd6f33e..10369ac44c 100644--- a/docs/designs/GIT_CANONICAL_MEMORY.md+++ b/docs/designs/GIT_CANONICAL_MEMORY.md@@ -107,7 +107,7 @@ G8a ships it as `gbrain sources canonical <source-id> --to git|mixed [--dry-run| `unparseable` | The committed blob is not UTF-8 Markdown the importer accepts, or is a symlink. | A person fixes the file; conversion refuses the page until then. || `unsafe_path` | The recorded `source_path` maps to no safe file path. | Reconcile the path first. |-Without an import record (classic sync, or a page imported before records were written) the census cannot tell which side moved, so a page whose file changed in git counts as `database_ahead`; `database_ahead_without_import_record` says how many. The working tree is reported separately by raw bytes against HEAD (`clean`, `modified`, `deleted`, `untracked`, `absent`); a checkout whose attributes convert line endings reads as modified. In strict mode an uncommitted change means nothing. A page withheld from Git by its private transition (`withheld_source_path`) is compared with the file at that withheld path, where a page-prepare write would put it again. The census reads each batch's export views together (`readExportPages`, which looks pages up by primary key). A path whose ignore status Git cannot read (beyond a symlink, outside the checkout, a timed-out check) is classified as not ignored and counted under `ignore_status_unknown` by reason, which `gbrain sources git-gap` and doctor's `db_ahead_of_git` name; conversion discards it as `ignore_status_unknown`, and other paths in the batch are unaffected.+Without an import record (classic sync, or a page imported before records were written) the census cannot tell which side moved, so a page whose file changed in git counts as `database_ahead`; `database_ahead_without_import_record` says how many. The working tree is reported separately by raw bytes against HEAD (`clean`, `modified`, `deleted`, `untracked`, `absent`); a checkout whose attributes convert line endings reads as modified. In strict mode an uncommitted change means nothing. A page withheld from Git by its private transition (`withheld_source_path`) is compared with the file at that withheld path, where a page-prepare write would put it again. The census reads each batch's export views together (`readExportPages`, which looks pages up by primary key). A path whose ignore status Git cannot read (beyond a symlink, outside the checkout, a timed-out check) refuses the census report, so `gbrain sources git-gap` and doctor's `db_ahead_of_git` name the path and the reason and count nothing for that source. The conversion plan classifies it as not ignored and discards it as `ignore_status_unknown`; other paths in the batch are unaffected.The census also counts rows with no Git home, which G4 and G5 give a file form: active facts with no fence row, fenced facts whose fence page is missing or has no committed file, links whose `link_source` is not one a file derives (`markdown`, `frontmatter`, `mentions`, `wikilink-resolved`; `manual`, custom tags, `concept-provenance`, `atom-provenance` and legacy NULL rows all count), and timeline rows on pages with no committed file. Open sync holds are counted by code and parked effect targets by kind and code; strict mode retires both in G8.diff --git a/src/commands/doctor/checks/db-ahead-of-git.ts b/src/commands/doctor/checks/db-ahead-of-git.tsindex 5be42328bf..49ad97b69c 100644--- a/src/commands/doctor/checks/db-ahead-of-git.ts+++ b/src/commands/doctor/checks/db-ahead-of-git.ts@@ -20,7 +20,7 @@ export function gitGapScanTimeoutMs(): number {return Number.isFinite(n) && n > 0 ? n : 10_000;}-type SourceTotals = Pick<GitGapCensus, 'source_id' | 'complete' | 'pages' | 'states' | 'declared' | 'worktree' | 'ignore_status_unknown'>;+type SourceTotals = Pick<GitGapCensus, 'source_id' | 'complete' | 'pages' | 'states' | 'declared' | 'worktree'>;const nonzero = (record: Record<string, number>) => Object.entries(record).filter(([, n]) => n > 0).map(([k, n]) => `${n} ${k}`).join(', ') || 'no pages';@@ -36,8 +36,8 @@ async function runDbAheadOfGit(ctx: DoctorContext): Promise<Check[]> {for (const row of withCheckout) {if (Date.now() >= deadlineAt) { unread.push(row.id); continue; }try {- const { source_id, complete, pages, states, declared, worktree, ignore_status_unknown } = await censusGitGap(engine, row.id, { sample: 0, deadlineAt });- sources.push({ source_id, complete, pages, states, declared, worktree, ignore_status_unknown });+ const { source_id, complete, pages, states, declared, worktree } = await censusGitGap(engine, row.id, { sample: 0, deadlineAt });+ sources.push({ source_id, complete, pages, states, declared, worktree });} catch (error) {errors.push({ source_id: row.id, message: error instanceof Error ? error.message : String(error) });}@@ -45,9 +45,7 @@ async function runDbAheadOfGit(ctx: DoctorContext): Promise<Check[]> {const partial = [...sources.filter(source => !source.complete).map(source => source.source_id), ...unread];const details = { informational: true, sources, partial_sources: partial, errors, sources_without_git_checkout: rows.length - withCheckout.length,timeout_ms: gitGapScanTimeoutMs(), docs: GIT_GAP_DOCS };- const unknownOf = (source: SourceTotals) => Object.values(source.ignore_status_unknown).reduce((sum, n) => sum + n, 0);- const listed = sources.map(source => `${source.source_id}: ${nonzero(source.states)}${source.complete ? '' : ' (partial)'}`- + `${unknownOf(source) ? ` (ignore status unknown for ${unknownOf(source)})` : ''}`);+ const listed = sources.map(source => `${source.source_id}: ${nonzero(source.states)}${source.complete ? '' : ' (partial)'}`);const message = withCheckout.length? `Informational, nothing to do: the database compared with the commit at HEAD, per source with a Git checkout.${listed.length ? ` ${listed.join('; ')}.` : ''}`+ `${unread.length ? ` Not reached within the ${details.timeout_ms / 1000}s budget: ${unread.join(', ')}.` : ''}`diff --git a/src/commands/sources-git-gap.ts b/src/commands/sources-git-gap.tsindex cdd55c5e9a..302ecafee9 100644--- a/src/commands/sources-git-gap.ts+++ b/src/commands/sources-git-gap.ts@@ -63,8 +63,6 @@ export interface GitGapCensus {database_ahead_without_import_record: number;/** git_ahead pages whose row also moved since its last import: both sides changed. */git_ahead_database_also_moved: number;- /** Pages whose path Git could not say is ignored, by reason (IgnoreUnknownReason); they are classified as if not ignored. */- ignore_status_unknown: Record<string, number>;/** Pages of a source without a checkout that are not declared. Since G7a such a source is declared `database_only_source`, so this stays 0; kept for readers of the JSON. */not_compared: number;/** Working-tree files of the pages that may have a tracked file, by their raw bytes against HEAD; null without a checkout. */@@ -112,7 +110,7 @@ const zeros = <K extends string>(keys: readonly K[]) => Object.fromEntries(keys.function emptyCensus(sourceId: string, root: string | null, head: HeadTree | null): GitGapCensus {return { status: 'census', source_id: sourceId, checkout: root, head: head?.commit ?? null, complete: true, pages: 0, states: zeros(PAGE_STATES),- declared: zeros(DECLARED_CLASSES), declared_with_head_blob: 0, database_ahead_without_import_record: 0, git_ahead_database_also_moved: 0, ignore_status_unknown: {}, not_compared: 0,+ declared: zeros(DECLARED_CLASSES), declared_with_head_blob: 0, database_ahead_without_import_record: 0, git_ahead_database_also_moved: 0, not_compared: 0,worktree: root ? zeros(WORKTREE_STATES) : null,rows_without_git_home: { facts_unfenced: 0, facts_fence_page_missing: 0, facts_on_pages_without_file: { undeclared: 0, declared: 0 },links: { total: 0, by_link_source: {} }, timeline_on_pages_without_file: { undeclared: 0, declared: 0 } },@@ -189,16 +187,15 @@ async function censusBatch(tx: BrainEngine, batch: Key[], census: GitGapCensus,return { key, path, rel: path ? relative(checkout!.root, path).split(sep).join('/') : null };});if (checkout) {- // A path whose ignore status Git cannot read (one beyond a symlink) is classified as if not ignored and counted under- // ignore_status_unknown, so the report names it; the conversion plan discards it as ignore_status_unknown instead of adopting it.+ // A path whose ignore status Git cannot read (one beyond a symlink) is classified as if not ignored in the conversion+ // plan, which discards it as ignore_status_unknown instead of adopting it; the census report refuses, as it did before #7617.const status = await ignoredPaths(checkout.root, rows.flatMap(row => row.rel === null ? [] : [row.rel]));if (!status) throw refuse(`Git could not say which paths the checkout at ${checkout.root} ignores, so the declared classes are unknown.`,`Nothing changed. Check that ${checkout.root} is a Git checkout (git -C ${checkout.root} rev-parse HEAD), then rerun gbrain sources git-gap ${batch[0]!.source_id}.`);+ const [unknown] = status.unknown;+ if (unknown && !ctx.collect) throw refuse(`Git could not say whether ${unknown[0]} in the checkout at ${checkout.root} is ignored (${unknown[1]}), so the declared classes are unknown.`,+ `Nothing changed. Check the path with git -C ${checkout.root} check-ignore -- ${unknown[0]} (Git cannot check a path beyond a symlinked directory), then rerun gbrain sources git-gap ${batch[0]!.source_id}.`);ctx.policy.ignored = status.ignored;- for (const { rel } of rows) {- const reason = rel === null ? undefined : status.unknown.get(rel);- if (reason) census.ignore_status_unknown[reason] = (census.ignore_status_unknown[reason] ?? 0) + 1;- }}const noFile = { undeclared: [] as Key[], declared: [] as Key[] };const compare: Array<{ key: Key; path: string; rel: string; blob: TreeBlob; worktree: WorktreeState }> = [];@@ -337,8 +334,6 @@ export function describeGitGap(census: GitGapCensus): string {else if (!census.head) lines.push('The checkout has no commit yet, so no page has a committed file.');lines.push(`Pages: ${counts(census.states)}.`, `Declared database-only: ${counts(census.declared)}${census.declared_with_head_blob ? ` (${census.declared_with_head_blob} still committed at HEAD)` : ''}.`);if (census.database_ahead_without_import_record) lines.push(`${census.database_ahead_without_import_record} database_ahead page(s) have no import record, so the side that moved is unknown.`);- const unknown = Object.values(census.ignore_status_unknown).reduce((sum, n) => sum + n, 0);- if (unknown) lines.push(`${unknown} page(s) have a path whose ignore status Git could not read (${counts(census.ignore_status_unknown)}), so whether they are declared database-only is unknown; they are counted above as if not ignored.`);if (census.git_ahead_database_also_moved) lines.push(`${census.git_ahead_database_also_moved} git_ahead page(s) also changed in the database since their last import.`);if (census.worktree) lines.push(`Working tree against HEAD: ${counts(census.worktree)}.`);const rows = census.rows_without_git_home;diff --git a/test/git-gap-census.test.ts b/test/git-gap-census.test.tsindex 3c6f06b07d..a2847d0697 100644--- a/test/git-gap-census.test.ts+++ b/test/git-gap-census.test.ts@@ -221,7 +221,7 @@ for (const backend of testBackends()) describe(`git-gap census (git-canonical G1expect((await censusGitGap(engine, 'default', { sample: 0 })).declared.mirror_source).toBe(13);}, 120000);- test('#7618: a batch with a page beyond a symlink is classified, not refused', async () => {+ test('#7618: in the conversion plan, a batch with a page beyond a symlink is classified, not refused', async () => {const root = tempDir('git-gap-census-symlink-');git(root, 'init', '-q');await useCheckout(root);@@ -230,17 +230,19 @@ for (const backend of testBackends()) describe(`git-gap census (git-canonical G1commitAll(root, 'init');mkdirSync(join(root, 'real'));symlinkSync('real', join(root, 'link'));- const census = await censusGitGap(engine, 'default', { sample: 10 });+ const collected: Array<{ slug: string; state: string }> = [];+ const census = await censusGitGap(engine, 'default', { sample: 10, collect: page => collected.push(page) });expect(census).toMatchObject({ complete: true, pages: 3, states: { equal: 1, undeclared_database_only: 2 } });expect(census.sample.find(item => item.slug === 'link/beyond')).toMatchObject({ path: 'link/beyond.md', state: 'undeclared_database_only' });+ expect(collected.find(page => page.slug === 'link/beyond')).toMatchObject({ state: 'undeclared_database_only' });});/**- * Value: protects=a page whose ignore status Git refuses to read is reported as unknown by git-gap and doctor, never as a- * plain undeclared or worktree count; fails_when=the census drops ignoredPaths' unknown set (the pre-#7618 census refused- * instead); seam=none (a real check-ignore refusal beyond a symlink); no_claim=timeout and outside-repo reasons.+ * Value: protects=a page whose ignore status Git cannot read is never counted as a plain undeclared page by git-gap or doctor;+ * the report refuses as the pre-#7617 census did; fails_when=the report path classifies it as if not ignored;+ * seam=none (a real check-ignore refusal beyond a symlink); no_claim=timeout and outside-repo reasons.*/- test('#7618: a page whose ignore status Git cannot read is counted and named as unknown by git-gap and doctor', async () => {+ test('#7618: git-gap and doctor refuse a source with a page whose ignore status Git cannot read', async () => {const root = tempDir('git-gap-census-unknown-');git(root, 'init', '-q');await useCheckout(root);@@ -249,12 +251,12 @@ for (const backend of testBackends()) describe(`git-gap census (git-canonical G1commitAll(root, 'init');mkdirSync(join(root, 'real'));symlinkSync('real', join(root, 'link'));- const census = await censusGitGap(engine, 'default', { sample: 0 });- expect(census).toMatchObject({ pages: 2, ignore_status_unknown: { symlink: 1 } });- expect(describeGitGap(census)).toContain('1 page(s) have a path whose ignore status Git could not read (symlink 1)');+ const refusal = 'Git could not say whether link/beyond.md in the checkout at';+ await expect(censusGitGap(engine, 'default', { sample: 0 })).rejects.toMatchObject({ code: 'storage_error', message: expect.stringContaining(refusal) });const [check] = await dbAheadOfGitEntry.run({ engine, progress: { heartbeat: () => {} } } as never) as Array<{ message: string; details: any }>;- expect(check!.details.sources.find((s: { source_id: string }) => s.source_id === 'default')).toMatchObject({ ignore_status_unknown: { symlink: 1 } });- expect(check!.message).toContain('default: 1 equal, 1 undeclared_database_only (ignore status unknown for 1)');+ expect(check!.details.sources.find((s: { source_id: string }) => s.source_id === 'default')).toBeUndefined();+ expect(check!.details.errors).toEqual([expect.objectContaining({ source_id: 'default', message: expect.stringContaining('(symlink)') })]);+ expect(check!.message).toContain(`Could not compare: default (${refusal}`);});test('a repository filter, textconv or fsmonitor never runs: HEAD and the working tree are read without them', async () => {