Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(git-gap): the census report refuses a source with a path whose ignore status Git cannot read, as before #7617 (#7618)

garrytan10412ead269f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index de1121752b..8b2453a1be 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -54,7 +54,7 @@ GBRAIN_MEMORY_TRACE=5 gbrain sync --source <source> # the same memory t
- `src/core/export-snapshot.ts`: a batched export read looks each page up by its primary key, so `gbrain export` and the preview stay linear on large sources.
- `src/commands/sources-canonical.ts`: progress lines, the partial report on signal, the push-pending notice.
- `src/core/memory-trace.ts`: `GBRAIN_MEMORY_TRACE=<seconds>` (1 to 3600) writes a memory line on stderr during a conversion apply and during `gbrain sync`.
-- `src/commands/sources-git-gap.ts`, `src/core/persistence/git-gap.ts`: the census reads a withheld page's file at its withheld path; a path whose ignore status Git cannot read no longer refuses the whole census, and `gbrain sources git-gap` and doctor's `db_ahead_of_git` count it as `ignore_status_unknown` instead.
+- `src/commands/sources-git-gap.ts`, `src/core/persistence/git-gap.ts`: the census reads a withheld page's file at its withheld path; in the conversion plan a path whose ignore status Git cannot read no longer refuses the whole census, while `gbrain sources git-gap` and doctor's `db_ahead_of_git` still refuse that source and name the path.
- `scripts/bench/sync-pass-memory.ts`: a memory probe for the strict-equivalent pass and the conversion (seed, run, cli, embed-seed).
- Tests: `test/git-canonical-conversion-g8a.test.ts` (crash, refusal and withhold cases on PGLite and Postgres), `test/git-canonical-conversion-progress.test.ts`, `test/git-gap-census.test.ts`, and a Postgres smoke test for the memory probe.
diff --git a/docs/designs/GIT_CANONICAL_MEMORY.md b/docs/designs/GIT_CANONICAL_MEMORY.md
index b68dd6f33e..10369ac44c 100644
--- a/docs/designs/GIT_CANONICAL_MEMORY.md
+++ b/docs/designs/GIT_CANONICAL_MEMORY.md
@@ -107,7 +107,7 @@ G8a ships it as `gbrain sources canonical <source-id> --to git|mixed [--dry-run
| `unparseable` | The committed blob is not UTF-8 Markdown the importer accepts, or is a symlink. | A person fixes the file; conversion refuses the page until then. |
| `unsafe_path` | The recorded `source_path` maps to no safe file path. | Reconcile the path first. |
-Without an import record (classic sync, or a page imported before records were written) the census cannot tell which side moved, so a page whose file changed in git counts as `database_ahead`; `database_ahead_without_import_record` says how many. The working tree is reported separately by raw bytes against HEAD (`clean`, `modified`, `deleted`, `untracked`, `absent`); a checkout whose attributes convert line endings reads as modified. In strict mode an uncommitted change means nothing. A page withheld from Git by its private transition (`withheld_source_path`) is compared with the file at that withheld path, where a page-prepare write would put it again. The census reads each batch's export views together (`readExportPages`, which looks pages up by primary key). A path whose ignore status Git cannot read (beyond a symlink, outside the checkout, a timed-out check) is classified as not ignored and counted under `ignore_status_unknown` by reason, which `gbrain sources git-gap` and doctor's `db_ahead_of_git` name; conversion discards it as `ignore_status_unknown`, and other paths in the batch are unaffected.
+Without an import record (classic sync, or a page imported before records were written) the census cannot tell which side moved, so a page whose file changed in git counts as `database_ahead`; `database_ahead_without_import_record` says how many. The working tree is reported separately by raw bytes against HEAD (`clean`, `modified`, `deleted`, `untracked`, `absent`); a checkout whose attributes convert line endings reads as modified. In strict mode an uncommitted change means nothing. A page withheld from Git by its private transition (`withheld_source_path`) is compared with the file at that withheld path, where a page-prepare write would put it again. The census reads each batch's export views together (`readExportPages`, which looks pages up by primary key). A path whose ignore status Git cannot read (beyond a symlink, outside the checkout, a timed-out check) refuses the census report, so `gbrain sources git-gap` and doctor's `db_ahead_of_git` name the path and the reason and count nothing for that source. The conversion plan classifies it as not ignored and discards it as `ignore_status_unknown`; other paths in the batch are unaffected.
The census also counts rows with no Git home, which G4 and G5 give a file form: active facts with no fence row, fenced facts whose fence page is missing or has no committed file, links whose `link_source` is not one a file derives (`markdown`, `frontmatter`, `mentions`, `wikilink-resolved`; `manual`, custom tags, `concept-provenance`, `atom-provenance` and legacy NULL rows all count), and timeline rows on pages with no committed file. Open sync holds are counted by code and parked effect targets by kind and code; strict mode retires both in G8.
diff --git a/src/commands/doctor/checks/db-ahead-of-git.ts b/src/commands/doctor/checks/db-ahead-of-git.ts
index 5be42328bf..49ad97b69c 100644
--- a/src/commands/doctor/checks/db-ahead-of-git.ts
+++ b/src/commands/doctor/checks/db-ahead-of-git.ts
@@ -20,7 +20,7 @@ export function gitGapScanTimeoutMs(): number {
return Number.isFinite(n) && n > 0 ? n : 10_000;
}
-type SourceTotals = Pick<GitGapCensus, 'source_id' | 'complete' | 'pages' | 'states' | 'declared' | 'worktree' | 'ignore_status_unknown'>;
+type SourceTotals = Pick<GitGapCensus, 'source_id' | 'complete' | 'pages' | 'states' | 'declared' | 'worktree'>;
const nonzero = (record: Record<string, number>) => Object.entries(record).filter(([, n]) => n > 0).map(([k, n]) => `${n} ${k}`).join(', ') || 'no pages';
@@ -36,8 +36,8 @@ async function runDbAheadOfGit(ctx: DoctorContext): Promise<Check[]> {
for (const row of withCheckout) {
if (Date.now() >= deadlineAt) { unread.push(row.id); continue; }
try {
- const { source_id, complete, pages, states, declared, worktree, ignore_status_unknown } = await censusGitGap(engine, row.id, { sample: 0, deadlineAt });
- sources.push({ source_id, complete, pages, states, declared, worktree, ignore_status_unknown });
+ const { source_id, complete, pages, states, declared, worktree } = await censusGitGap(engine, row.id, { sample: 0, deadlineAt });
+ sources.push({ source_id, complete, pages, states, declared, worktree });
} catch (error) {
errors.push({ source_id: row.id, message: error instanceof Error ? error.message : String(error) });
}
@@ -45,9 +45,7 @@ async function runDbAheadOfGit(ctx: DoctorContext): Promise<Check[]> {
const partial = [...sources.filter(source => !source.complete).map(source => source.source_id), ...unread];
const details = { informational: true, sources, partial_sources: partial, errors, sources_without_git_checkout: rows.length - withCheckout.length,
timeout_ms: gitGapScanTimeoutMs(), docs: GIT_GAP_DOCS };
- const unknownOf = (source: SourceTotals) => Object.values(source.ignore_status_unknown).reduce((sum, n) => sum + n, 0);
- const listed = sources.map(source => `${source.source_id}: ${nonzero(source.states)}${source.complete ? '' : ' (partial)'}`
- + `${unknownOf(source) ? ` (ignore status unknown for ${unknownOf(source)})` : ''}`);
+ const listed = sources.map(source => `${source.source_id}: ${nonzero(source.states)}${source.complete ? '' : ' (partial)'}`);
const message = withCheckout.length
? `Informational, nothing to do: the database compared with the commit at HEAD, per source with a Git checkout.${listed.length ? ` ${listed.join('; ')}.` : ''}`
+ `${unread.length ? ` Not reached within the ${details.timeout_ms / 1000}s budget: ${unread.join(', ')}.` : ''}`
diff --git a/src/commands/sources-git-gap.ts b/src/commands/sources-git-gap.ts
index cdd55c5e9a..302ecafee9 100644
--- a/src/commands/sources-git-gap.ts
+++ b/src/commands/sources-git-gap.ts
@@ -63,8 +63,6 @@ export interface GitGapCensus {
database_ahead_without_import_record: number;
/** git_ahead pages whose row also moved since its last import: both sides changed. */
git_ahead_database_also_moved: number;
- /** Pages whose path Git could not say is ignored, by reason (IgnoreUnknownReason); they are classified as if not ignored. */
- ignore_status_unknown: Record<string, number>;
/** Pages of a source without a checkout that are not declared. Since G7a such a source is declared `database_only_source`, so this stays 0; kept for readers of the JSON. */
not_compared: number;
/** Working-tree files of the pages that may have a tracked file, by their raw bytes against HEAD; null without a checkout. */
@@ -112,7 +110,7 @@ const zeros = <K extends string>(keys: readonly K[]) => Object.fromEntries(keys.
function emptyCensus(sourceId: string, root: string | null, head: HeadTree | null): GitGapCensus {
return { status: 'census', source_id: sourceId, checkout: root, head: head?.commit ?? null, complete: true, pages: 0, states: zeros(PAGE_STATES),
- declared: zeros(DECLARED_CLASSES), declared_with_head_blob: 0, database_ahead_without_import_record: 0, git_ahead_database_also_moved: 0, ignore_status_unknown: {}, not_compared: 0,
+ declared: zeros(DECLARED_CLASSES), declared_with_head_blob: 0, database_ahead_without_import_record: 0, git_ahead_database_also_moved: 0, not_compared: 0,
worktree: root ? zeros(WORKTREE_STATES) : null,
rows_without_git_home: { facts_unfenced: 0, facts_fence_page_missing: 0, facts_on_pages_without_file: { undeclared: 0, declared: 0 },
links: { total: 0, by_link_source: {} }, timeline_on_pages_without_file: { undeclared: 0, declared: 0 } },
@@ -189,16 +187,15 @@ async function censusBatch(tx: BrainEngine, batch: Key[], census: GitGapCensus,
return { key, path, rel: path ? relative(checkout!.root, path).split(sep).join('/') : null };
});
if (checkout) {
- // A path whose ignore status Git cannot read (one beyond a symlink) is classified as if not ignored and counted under
- // ignore_status_unknown, so the report names it; the conversion plan discards it as ignore_status_unknown instead of adopting it.
+ // A path whose ignore status Git cannot read (one beyond a symlink) is classified as if not ignored in the conversion
+ // plan, which discards it as ignore_status_unknown instead of adopting it; the census report refuses, as it did before #7617.
const status = await ignoredPaths(checkout.root, rows.flatMap(row => row.rel === null ? [] : [row.rel]));
if (!status) throw refuse(`Git could not say which paths the checkout at ${checkout.root} ignores, so the declared classes are unknown.`,
`Nothing changed. Check that ${checkout.root} is a Git checkout (git -C ${checkout.root} rev-parse HEAD), then rerun gbrain sources git-gap ${batch[0]!.source_id}.`);
+ const [unknown] = status.unknown;
+ if (unknown && !ctx.collect) throw refuse(`Git could not say whether ${unknown[0]} in the checkout at ${checkout.root} is ignored (${unknown[1]}), so the declared classes are unknown.`,
+ `Nothing changed. Check the path with git -C ${checkout.root} check-ignore -- ${unknown[0]} (Git cannot check a path beyond a symlinked directory), then rerun gbrain sources git-gap ${batch[0]!.source_id}.`);
ctx.policy.ignored = status.ignored;
- for (const { rel } of rows) {
- const reason = rel === null ? undefined : status.unknown.get(rel);
- if (reason) census.ignore_status_unknown[reason] = (census.ignore_status_unknown[reason] ?? 0) + 1;
- }
}
const noFile = { undeclared: [] as Key[], declared: [] as Key[] };
const compare: Array<{ key: Key; path: string; rel: string; blob: TreeBlob; worktree: WorktreeState }> = [];
@@ -337,8 +334,6 @@ export function describeGitGap(census: GitGapCensus): string {
else if (!census.head) lines.push('The checkout has no commit yet, so no page has a committed file.');
lines.push(`Pages: ${counts(census.states)}.`, `Declared database-only: ${counts(census.declared)}${census.declared_with_head_blob ? ` (${census.declared_with_head_blob} still committed at HEAD)` : ''}.`);
if (census.database_ahead_without_import_record) lines.push(`${census.database_ahead_without_import_record} database_ahead page(s) have no import record, so the side that moved is unknown.`);
- const unknown = Object.values(census.ignore_status_unknown).reduce((sum, n) => sum + n, 0);
- if (unknown) lines.push(`${unknown} page(s) have a path whose ignore status Git could not read (${counts(census.ignore_status_unknown)}), so whether they are declared database-only is unknown; they are counted above as if not ignored.`);
if (census.git_ahead_database_also_moved) lines.push(`${census.git_ahead_database_also_moved} git_ahead page(s) also changed in the database since their last import.`);
if (census.worktree) lines.push(`Working tree against HEAD: ${counts(census.worktree)}.`);
const rows = census.rows_without_git_home;
diff --git a/test/git-gap-census.test.ts b/test/git-gap-census.test.ts
index 3c6f06b07d..a2847d0697 100644
--- a/test/git-gap-census.test.ts
+++ b/test/git-gap-census.test.ts
@@ -221,7 +221,7 @@ for (const backend of testBackends()) describe(`git-gap census (git-canonical G1
expect((await censusGitGap(engine, 'default', { sample: 0 })).declared.mirror_source).toBe(13);
}, 120000);
- test('#7618: a batch with a page beyond a symlink is classified, not refused', async () => {
+ test('#7618: in the conversion plan, a batch with a page beyond a symlink is classified, not refused', async () => {
const root = tempDir('git-gap-census-symlink-');
git(root, 'init', '-q');
await useCheckout(root);
@@ -230,17 +230,19 @@ for (const backend of testBackends()) describe(`git-gap census (git-canonical G1
commitAll(root, 'init');
mkdirSync(join(root, 'real'));
symlinkSync('real', join(root, 'link'));
- const census = await censusGitGap(engine, 'default', { sample: 10 });
+ const collected: Array<{ slug: string; state: string }> = [];
+ const census = await censusGitGap(engine, 'default', { sample: 10, collect: page => collected.push(page) });
expect(census).toMatchObject({ complete: true, pages: 3, states: { equal: 1, undeclared_database_only: 2 } });
expect(census.sample.find(item => item.slug === 'link/beyond')).toMatchObject({ path: 'link/beyond.md', state: 'undeclared_database_only' });
+ expect(collected.find(page => page.slug === 'link/beyond')).toMatchObject({ state: 'undeclared_database_only' });
});
/**
- * Value: protects=a page whose ignore status Git refuses to read is reported as unknown by git-gap and doctor, never as a
- * plain undeclared or worktree count; fails_when=the census drops ignoredPaths' unknown set (the pre-#7618 census refused
- * instead); seam=none (a real check-ignore refusal beyond a symlink); no_claim=timeout and outside-repo reasons.
+ * Value: protects=a page whose ignore status Git cannot read is never counted as a plain undeclared page by git-gap or doctor;
+ * the report refuses as the pre-#7617 census did; fails_when=the report path classifies it as if not ignored;
+ * seam=none (a real check-ignore refusal beyond a symlink); no_claim=timeout and outside-repo reasons.
*/
- test('#7618: a page whose ignore status Git cannot read is counted and named as unknown by git-gap and doctor', async () => {
+ test('#7618: git-gap and doctor refuse a source with a page whose ignore status Git cannot read', async () => {
const root = tempDir('git-gap-census-unknown-');
git(root, 'init', '-q');
await useCheckout(root);
@@ -249,12 +251,12 @@ for (const backend of testBackends()) describe(`git-gap census (git-canonical G1
commitAll(root, 'init');
mkdirSync(join(root, 'real'));
symlinkSync('real', join(root, 'link'));
- const census = await censusGitGap(engine, 'default', { sample: 0 });
- expect(census).toMatchObject({ pages: 2, ignore_status_unknown: { symlink: 1 } });
- expect(describeGitGap(census)).toContain('1 page(s) have a path whose ignore status Git could not read (symlink 1)');
+ const refusal = 'Git could not say whether link/beyond.md in the checkout at';
+ await expect(censusGitGap(engine, 'default', { sample: 0 })).rejects.toMatchObject({ code: 'storage_error', message: expect.stringContaining(refusal) });
const [check] = await dbAheadOfGitEntry.run({ engine, progress: { heartbeat: () => {} } } as never) as Array<{ message: string; details: any }>;
- expect(check!.details.sources.find((s: { source_id: string }) => s.source_id === 'default')).toMatchObject({ ignore_status_unknown: { symlink: 1 } });
- expect(check!.message).toContain('default: 1 equal, 1 undeclared_database_only (ignore status unknown for 1)');
+ expect(check!.details.sources.find((s: { source_id: string }) => s.source_id === 'default')).toBeUndefined();
+ expect(check!.details.errors).toEqual([expect.objectContaining({ source_id: 'default', message: expect.stringContaining('(symlink)') })]);
+ expect(check!.message).toContain(`Could not compare: default (${refusal}`);
});
test('a repository filter, textconv or fsmonitor never runs: HEAD and the working tree are read without them', async () => {