Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(rows-behind): check the scan deadline per row; doctor skips unproven confirmation; holds use the fresh refusal (#8071)
garrytan86261c6d3deb
diff --git a/docs/guides/repair.md b/docs/guides/repair.mdindex 601054f1bb..a6aee767f2 100644--- a/docs/guides/repair.md+++ b/docs/guides/repair.md@@ -1420,13 +1420,14 @@ importer itself, so a row that importing the committed file would leaveunchanged is current. A row is reported only when its revision is the one itslast sync import wrote. A row the database changed since that import belongsto [file drift](#file-drift) and is not reported here. A row with no import-record is counted as `unproven` and does not warn. Held paths, paths a later+record is counted as `unproven` from the content comparison alone and does not warn. Held paths, paths a latercommit changed (the next sync imports them) and declared database-only pagesare skipped. A file over the sync read bound is never read; its row isreported with the `file_too_large` refusal sync would hold it with. The scan-is bounded by `GBRAIN_DOCTOR_ROWS_BEHIND_TIMEOUT_MS` (default 10 s); a scan-the budget stopped, or a checkout Git cannot read, is `unknown`, never a-clean `ok`. Off the owner host, on a source without a checkout or a+is bounded by `GBRAIN_DOCTOR_ROWS_BEHIND_TIMEOUT_MS` (default 10 s), checked before+each row's comparison; a scan the budget stopped (doctor names the pages it+decided and the page id it stopped at), or a checkout Git cannot read, is+`unknown`, never a clean `ok`. Off the owner host, on a source without a checkout or acheckpoint, and on a repo-less brain (the database is the record there) thecheck is `ok` and not applicable.@@ -1446,11 +1447,12 @@ The preview writes nothing. Each row in the set has one action:- `hold:<code>`: importing the committed file is refused (for example amalformed fence), or the file is over the read bound(`hold:file_too_large`). The apply opens the hold sync would have opened,- with the refusal's code and message, and the usual+ with the code and message of the refusal the apply computes again, and the usual[held files](#held-files) steps clear it.Listed but left out of the set: rows with no import record-(`excluded:unproven`, joined with `--include-ambiguous`), rows whose+(`excluded:unproven`, joined with `--include-ambiguous`, which replaces any+database-only edit those rows hold), rows whoseworking-tree file differs from the committed one (`excluded:worktree_differs`:commit the edit and sync), and rows whose import would keep page data thefile does not carry (`excluded:import_overlay`; resolve them withdiff --git a/src/commands/doctor/checks/rows-behind.ts b/src/commands/doctor/checks/rows-behind.tsindex 7303d2f073..7010e02551 100644--- a/src/commands/doctor/checks/rows-behind.ts+++ b/src/commands/doctor/checks/rows-behind.ts@@ -5,7 +5,8 @@* at the checkpoint commit and confirms the difference with the importer.** warn when any source has a `behind` row: per-source counts, the first slugs and the read-only preview- * `gbrain repair rows-behind --source <id>`. Rows with no import record (`unproven`) are counted and never warn. ok when+ * `gbrain repair rows-behind --source <id>`. Rows with no import record (`unproven`) are counted from the content comparison+ * alone (no importer confirmation) and never warn. ok when* every source read completely and none is behind; a scan the budget stopped, or a checkout Git cannot read, is info* `unknown` (never a clean ok). Not applicable (ok, info) on a brain without managed sync, a source without a checkout or* checkpoint (a repo-less brain) and off the owner host. Bounded by `GBRAIN_DOCTOR_ROWS_BEHIND_TIMEOUT_MS` (default 10 s)@@ -32,17 +33,17 @@ async function runRowsBehind(ctx: DoctorContext): Promise<Check[]> {const deadlineAt = scanDeadline(ctx, rowsBehindScanTimeoutMs());const ids = doctorSourceIds(ctx) ?? (await engine.executeRaw<{ id: string }>('SELECT id FROM sources WHERE archived IS NOT TRUE ORDER BY id')).map(row => row.id);const scans: RowsBehindScan[] = [];- for (const id of ids) scans.push(await scanRowsBehind(engine, id, { deadlineAt }));+ for (const id of ids) scans.push(await scanRowsBehind(engine, id, { deadlineAt, confirmUnproven: false }));const known = scans.filter(scan => scan.status === 'known');const count = known.reduce((sum, scan) => sum + scan.behind_count, 0);const unproven = known.reduce((sum, scan) => sum + scan.unproven_count, 0);const partial = known.filter(scan => !scan.complete);const unknown = scans.filter(scan => scan.status === 'unknown');const details = { count, unproven, docs: ROWS_BEHIND_DOCS, timeout_ms: rowsBehindScanTimeoutMs(),- sources: scans.map(scan => ({ source_id: scan.source_id, status: scan.status, ...(scan.reason ? { reason: scan.reason } : {}), checkpoint: scan.checkpoint, complete: scan.complete,+ sources: scans.map(scan => ({ source_id: scan.source_id, status: scan.status, ...(scan.reason ? { reason: scan.reason } : {}), checkpoint: scan.checkpoint, complete: scan.complete, stopped_at: scan.stopped_at,pages: scan.pages, behind: scan.behind_count, unproven: scan.unproven_count, counts: scan.counts,samples: (ctx.remote ? [] : scan.behind).slice(0, SAMPLES).map(row => row.slug), unproven_samples: (ctx.remote ? [] : scan.unproven).slice(0, SAMPLES).map(row => row.slug) })) };- const unprovenNote = unproven ? ` ${unproven} more row(s) differ from their checkpoint file but have no import record, so the side that moved is unknown; gbrain repair rows-behind --source <id> --include-ambiguous lists and re-imports them.` : '';+ const unprovenNote = unproven ? ` ${unproven} more row(s) differ from their checkpoint file but have no import record, so the side that moved is unknown; gbrain repair rows-behind --source <id> --include-ambiguous lists and re-imports them, which replaces any database-only edit those rows hold.` : '';if (count) {const found = known.filter(scan => scan.behind_count);checks.push({ name: 'sync_rows_behind_checkpoint', status: 'warn', details,@@ -56,7 +57,7 @@ async function runRowsBehind(ctx: DoctorContext): Promise<Check[]> {}if (unknown.length || partial.length) {checks.push(infoCheck('sync_rows_behind_checkpoint', `Rows behind the sync checkpoint are unknown for ${[...unknown.map(scan => `${scan.source_id} (${scan.reason})`),- ...partial.map(scan => `${scan.source_id} (stopped after ${scan.pages} page(s) at the ${details.timeout_ms / 1000} s budget)`)].join(', ')}; this is not a clean result. `+ ...partial.map(scan => `${scan.source_id} (stopped after ${scan.pages} page(s)${scan.stopped_at !== null ? ` at page id ${scan.stopped_at}` : ''} at the ${details.timeout_ms / 1000} s budget)`)].join(', ')}; this is not a clean result. `+ `The full read: gbrain repair rows-behind --source <id> (writes nothing).${unprovenNote}`, 'unknown', undefined, details));return checks;}diff --git a/src/core/persistence/rows-behind.ts b/src/core/persistence/rows-behind.tsindex 625b507388..d12de42d23 100644--- a/src/core/persistence/rows-behind.ts+++ b/src/core/persistence/rows-behind.ts@@ -18,7 +18,10 @@* Bounded: the checkpoint tree is listed once (and HEAD's only when it differs), pages are read in id batches of* ROWS_BEHIND_BATCH in read-only transactions, committed blobs are read in bounded `cat-file --batch` calls, and a blob* over the sync read bound (SYNC_READ_BOUND, the page ceiling) is never read: its row is reported with the- * `file_too_large` refusal sync would hold it with. A deadline stops the scan between batches (`complete: false`).+ * `file_too_large` refusal sync would hold it with. The deadline is checked before each batch and before each row's+ * content comparison and confirmation (`complete: false`; `pages` and `counts` cover the rows decided before+ * `stopped_at`, the page id the scan stopped at). `confirmUnproven: false` (doctor) counts rows with no import record+ * from the content comparison alone, without running the importer's confirmation.* Applies only on the owner host of a managed source with a checkout and a checkpoint; everything else, a repo-less* brain included, is `not_applicable` with its reason. Read-only. Readers: doctor `sync_rows_behind_checkpoint` and* `gbrain repair rows-behind`.@@ -74,9 +77,11 @@ export interface RowsBehindScan {reason?: string;checkpoint: string | null;head: string | null;- /** False when the deadline stopped the scan; the counts cover the pages read. */+ /** False when the deadline stopped the scan; the counts cover the pages decided before `stopped_at`. */complete: boolean;- /** Live file-backed pages read. */+ /** The page id the deadline stopped the scan at (not compared), or null. */+ stopped_at: number | null;+ /** Live file-backed pages decided. */pages: number;behind: RowBehind[];unproven: RowBehind[];@@ -91,11 +96,12 @@ type Key = { id: string; slug: string; source_id: string; source_path: string; sfrontmatter: unknown; database_only_reason: string | null; segmented: boolean; lineage_private: boolean; revision: string };const empty = (sourceId: string, status: RowsBehindScan['status'], reason?: string): RowsBehindScan => ({ source_id: sourceId, status, ...(reason ? { reason } : {}),- checkpoint: null, head: null, complete: status !== 'unknown', pages: 0, behind: [], unproven: [], truncated: false, behind_count: 0, unproven_count: 0, counts: {} });+ checkpoint: null, head: null, complete: status !== 'unknown', stopped_at: null, pages: 0, behind: [], unproven: [], truncated: false, behind_count: 0, unproven_count: 0, counts: {} });const isMarkdown = (path: string) => /\.mdx?$/i.test(path);-export async function scanRowsBehind(engine: BrainEngine, sourceId: string, opts: { deadlineAt?: number; listLimit?: number } = {}): Promise<RowsBehindScan> {+export async function scanRowsBehind(engine: BrainEngine, sourceId: string,+ opts: { deadlineAt?: number; listLimit?: number; confirmUnproven?: boolean; now?: () => number } = {}): Promise<RowsBehindScan> {if (!(await managedPersistenceEnabled(engine))) return empty(sourceId, 'not_applicable', 'not_managed');const context = await sourceContext(engine, sourceId);if (!context.ok) return empty(sourceId, context.status, context.reason);@@ -115,11 +121,17 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string, optsconst scan: RowsBehindScan = { ...empty(sourceId, 'known'), checkpoint: checkpoint.commit, head: head.commit };const limit = opts.listLimit ?? ROWS_BEHIND_LIST_LIMIT;const count = (why: RowsBehindSkip) => { scan.counts[why] = (scan.counts[why] ?? 0) + 1; };+ const now = opts.now ?? Date.now;+ const late = () => opts.deadlineAt !== undefined && now() >= opts.deadlineAt;const screen = await repairScreenConfig(engine, sourceId);const withdrawals: ExportWithdrawal[] = await readExportWithdrawals(engine, sourceId);const incarnation = context.owner.source_incarnation;for (let cursor = '0'; ;) {- if (opts.deadlineAt !== undefined && Date.now() >= opts.deadlineAt) { scan.complete = false; break; }+ if (late()) { scan.complete = false; break; }+ let stop: number | null = null;+ const skipped: Array<[number, RowsBehindSkip]> = [];+ const skip = (id: string, why: RowsBehindSkip) => { skipped.push([Number(id), why]); };+ let ids: number[] = [];const candidates: Array<{ key: Key; rel: string; blob: TreeBlob; record: RowBehind['record']; content: Buffer | null; compared: ExportFileComparison | null }> = [];const done = await engine.transaction(async tx => {await tx.executeRaw('SET TRANSACTION READ ONLY');@@ -128,19 +140,19 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string, optsFROM pages WHERE source_id=$1 AND deleted_at IS NULL AND source_path IS NOT NULL AND pages.id > $2::bigint ORDER BY pages.id LIMIT ${ROWS_BEHIND_BATCH}`, [sourceId, cursor]);if (!batch.length) return true;cursor = batch[batch.length - 1]!.id;+ ids = batch.map(key => Number(key.id));const rows: Array<{ key: Key; rel: string; blob: TreeBlob }> = [];for (const key of batch) {- scan.pages++;const path = canonicalPath(context, key);- if (!path) { count('unsafe_path'); continue; }+ if (!path) { skip(key.id, 'unsafe_path'); continue; }const rel = relative(root, path).split(sep).join('/');- if (rel.startsWith('../') || rel === '..') { count('unsafe_path'); continue; }- if (!isMarkdown(rel)) { count('not_markdown'); continue; }+ if (rel.startsWith('../') || rel === '..') { skip(key.id, 'unsafe_path'); continue; }+ if (!isMarkdown(rel)) { skip(key.id, 'not_markdown'); continue; }const frontmatter = typeof key.frontmatter === 'string' ? JSON.parse(key.frontmatter) : key.frontmatter;- if (declaredDatabaseOnly({ ...key, frontmatter }, rel, policy)) { count('declared_database_only'); continue; }+ if (declaredDatabaseOnly({ ...key, frontmatter }, rel, policy)) { skip(key.id, 'declared_database_only'); continue; }const blob = checkpoint.blobs.get(gitPath(rel));- if (!blob) { count('no_file_at_checkpoint'); continue; }- if (blob.mode !== '100644' && blob.mode !== '100755') { count('not_regular_file'); continue; }+ if (!blob) { skip(key.id, 'no_file_at_checkpoint'); continue; }+ if (blob.mode !== '100644' && blob.mode !== '100755') { skip(key.id, 'not_regular_file'); continue; }rows.push({ key, rel, blob });}if (!rows.length) return batch.length < ROWS_BEHIND_BATCH;@@ -148,10 +160,10 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string, optsconst records = await readSyncImportProvenances(tx, sourceId, incarnation, rows.map(row => Number(row.key.id)));const compare: typeof rows = [];for (const row of rows) {- if (held.has(row.rel)) { count('held'); continue; }+ if (held.has(row.rel)) { skip(row.key.id, 'held'); continue; }const record = records.get(Number(row.key.id));- if (record?.revision != null && String(record.revision) !== row.key.revision) { count('database_moved'); continue; }- if (head !== checkpoint && head.blobs.get(gitPath(row.rel))?.oid !== row.blob.oid) { count('changed_after_checkpoint'); continue; }+ if (record?.revision != null && String(record.revision) !== row.key.revision) { skip(row.key.id, 'database_moved'); continue; }+ if (head !== checkpoint && head.blobs.get(gitPath(row.rel))?.oid !== row.blob.oid) { skip(row.key.id, 'changed_after_checkpoint'); continue; }compare.push(row);}const readable = compare.filter(row => row.blob.size <= SYNC_READ_BOUND);@@ -160,29 +172,52 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string, optsconst snapshots = await readExportPages(tx, readable.map(row => row.key), withdrawals);const byId = new Map(snapshots.map(snapshot => [String(snapshot.page.id), snapshot]));for (const row of compare) {+ if (late()) { stop = Number(row.key.id); break; }const recordState: RowBehind['record'] = records.get(Number(row.key.id))?.revision != null ? 'recorded' : 'none';if (row.blob.size > SYNC_READ_BOUND) { candidates.push({ ...row, record: recordState, content: null, compared: null }); continue; }const snapshot = byId.get(row.key.id);if (!snapshot) continue;const compared = row.key.segmented ? await compareSegmentedExportBlob(tx, snapshot, withdrawals, row.blob, checkpoint.format): compareInlineExportFile(bytes.get(row.blob.oid)!, scannerSourcePath(root, `${root}/${row.rel}`, context.mode), snapshot);- if (compared.state === 'equal') { count('current'); continue; }+ if (compared.state === 'equal') { skip(row.key.id, 'current'); continue; }candidates.push({ ...row, record: recordState, content: bytes.get(row.blob.oid) ?? null, compared });}return batch.length < ROWS_BEHIND_BATCH;});for (const candidate of candidates) {- const row = await confirm(engine, sourceId, root, screen, candidate);+ if (stop !== null && Number(candidate.key.id) >= stop) break;+ if (late()) { stop = Number(candidate.key.id); break; }+ const row = candidate.record === 'none' && opts.confirmUnproven === false ? unconfirmedRow(candidate) : await confirm(engine, sourceId, root, screen, candidate);if (typeof row === 'string') { count(row); continue; }const list = row.record === 'recorded' ? scan.behind : scan.unproven;if (row.record === 'recorded') scan.behind_count++; else scan.unproven_count++;if (list.length < limit) list.push(row); else scan.truncated = true;}+ for (const [id, why] of skipped) if (stop === null || id < stop) count(why);+ scan.pages += stop === null ? ids.length : ids.filter(id => id < stop!).length;+ if (stop !== null) { scan.complete = false; scan.stopped_at = stop; break; }if (done) break;}return scan;}+/** The hold fields of an import refusal (`prepareRepairPublication`'s `refused` status). */+export function rowRefusal(refusal: { code: string; message: string }): NonNullable<RowBehind['refusal']> {+ const { code, message } = refusal;+ const reason = 'reason' in refusal ? refusal.reason : undefined;+ const where = refusal as { key?: string; line?: number };+ return { code, message, ...(reason ? { reason: String(reason) } : {}), ...(where.key ? { key: where.key } : {}), ...(where.line !== undefined ? { line: where.line } : {}) };+}++/** A row with no import record, counted from the content comparison alone (doctor: no importer confirmation). */+function unconfirmedRow(candidate: { key: Key; rel: string; blob: TreeBlob; record: RowBehind['record'] }): RowBehind {+ const { key, rel, blob, record } = candidate;+ return { slug: key.slug, page_id: Number(key.id), revision: key.revision, path: rel, source_path: key.source_path, blob: blob.oid, size: blob.size, record,+ ...(blob.size > SYNC_READ_BOUND ? { refusal: tooLarge(blob.size) } : {}) };+}++const tooLarge = (size: number) => ({ code: 'file_too_large', message: `File too large (${size} bytes; the sync read bound is the ${SYNC_READ_BOUND}-byte page ceiling), so it was not read.` });+/*** A differing row confirmed by the importer: what the import of the checkpoint bytes would do to it. `current` when the* import is a no-op (the row holds that version after all); `unconfirmed` when the confirmation cannot run or the page@@ -193,7 +228,7 @@ async function confirm(engine: BrainEngine, sourceId: string, root: string, screconst { key, rel, blob, record } = candidate;const base: RowBehind = { slug: key.slug, page_id: Number(key.id), revision: key.revision, path: rel, source_path: key.source_path, blob: blob.oid, size: blob.size, record };if (candidate.content === null && blob.size > SYNC_READ_BOUND) {- return { ...base, refusal: { code: 'file_too_large', message: `File too large (${blob.size} bytes; the sync read bound is the ${SYNC_READ_BOUND}-byte page ceiling), so it was not read.` } };+ return { ...base, refusal: tooLarge(blob.size) };}const bytes = candidate.content ?? (await readBlobs(root, [{ oid: blob.oid, size: blob.size, mode: blob.mode }]))?.get(blob.oid) ?? null;const content = bytes?.toString('utf8') ?? null;@@ -202,12 +237,7 @@ async function confirm(engine: BrainEngine, sourceId: string, root: string, screconst snapshot = await engine.readPageSnapshot(key.slug, { sourceId, includeDeleted: true });if (!snapshot || String(snapshot.page.id) !== key.id || snapshot.revision !== key.revision) return 'unconfirmed';const publication = await prepareRepairPublication(engine, { sourceId, slug: key.slug, sourcePath: key.source_path, path: rel, root, content, snapshot, base: snapshot, ...screen });- if (publication.status === 'refused') {- const { code, message } = publication.refusal;- const reason = 'reason' in publication.refusal ? publication.refusal.reason : undefined;- const where = publication.refusal as { key?: string; line?: number };- return { ...base, refusal: { code, message, ...(reason ? { reason: String(reason) } : {}), ...(where.key ? { key: where.key } : {}), ...(where.line !== undefined ? { line: where.line } : {}) } };- }+ if (publication.status === 'refused') return { ...base, refusal: rowRefusal(publication.refusal) };if (publication.status === 'overlay') return publication.noop ? 'current' : { ...base, overlay: true };return publication.ready.noop ? 'current' : { ...base, digest: publication.digest };} catch {diff --git a/src/core/repair/rows-behind.ts b/src/core/repair/rows-behind.tsindex 55a7513e7d..eb6b226f3a 100644--- a/src/core/repair/rows-behind.ts+++ b/src/core/repair/rows-behind.ts@@ -9,7 +9,7 @@* projections a managed sync import uses, bound to the page revision and the previewed result; the file on disk already* holds those bytes, so no file is written, and the import record is rewritten). `hold:<code>`: the import of the* checkpoint file is refused (or it is over the sync read bound), so the apply opens the hold sync would have opened, with- * the refusal's code and message. Left out of the set and listed: rows with no import record (`excluded:unproven`, joined+ * the code and message of the refusal the apply recomputes (not the one saved with the preview). Left out of the set and listed: rows with no import record (`excluded:unproven`, joined* by --include-ambiguous), rows whose working-tree file differs from the checkpoint blob (`excluded:worktree_differs`: an* uncommitted edit; commit it and sync), and rows whose import would keep page data the file does not carry* (`excluded:import_overlay`). `--apply --expect <hash>` applies exactly that set; a row whose page, checkpoint or file@@ -27,7 +27,7 @@ import { acceptedPendingReceipt } from '../persistence/accepted-pending.ts';import { prepareRepairPublication, repairScreenConfig, submitManagedFileRepair } from '../persistence/file-repair.ts';import { readBlobs } from '../persistence/git-gap.ts';import { clearApprovedSet, loadApprovedSet, previewChangedError, previewHash, saveApprovedSet } from '../persistence/preview-approval.ts';-import { ROWS_BEHIND_DOCS, scanRowsBehind, type RowBehind } from '../persistence/rows-behind.ts';+import { ROWS_BEHIND_DOCS, rowRefusal, scanRowsBehind, type RowBehind } from '../persistence/rows-behind.ts';import { SYNC_READ_BOUND } from '../persistence/sync-blobs.ts';import { readGitHold, writeGitHold, type GitHoldRecord } from '../persistence/sync-holds.ts';import { afterCursor, repairRequestId, type RepairHandler, type RepairItem, type RepairItemOutcome, type RepairPlan, type RepairScope } from './core.ts';@@ -159,7 +159,7 @@ async function applyRow(ctx: OperationContext, row: ApprovedRow, embed: boolean): await prepareRepairPublication(engine, { sourceId: row.source_id, slug: row.slug, sourcePath: row.source_path, path: row.path, root: context.root, content, snapshot, base: snapshot, ...screen });if (row.action === 'hold') {if (publication && publication.status !== 'refused') return changed('import_no_longer_refused');- const refusal = row.refusal!;+ const refusal = publication?.status === 'refused' ? rowRefusal(publication.refusal) : row.refusal!;const held: Omit<GitHoldRecord, 'version' | 'held_at' | 'updated_at'> = { source_id: row.source_id, incarnation: row.owner.source_incarnation, path: row.path, source_path: row.source_path,slug: row.slug, page_id: row.page_id, code: refusal.code as GitHoldRecord['code'], message: refusal.message, upstream_version: bytes ? sha256(bytes) : null,observed_at: new Date().toISOString(), run_id: null, mode: 'managed',diff --git a/test/fixtures/goldens/doctor/content-pglite-json.json b/test/fixtures/goldens/doctor/content-pglite-json.jsonindex ceffbfa26f..95b2725792 100644--- a/test/fixtures/goldens/doctor/content-pglite-json.json+++ b/test/fixtures/goldens/doctor/content-pglite-json.json@@ -3550,6 +3550,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -3563,6 +3564,7 @@"samples": [],"source_id": "notes","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}diff --git a/test/fixtures/goldens/doctor/hot-checks-pglite-json.json b/test/fixtures/goldens/doctor/hot-checks-pglite-json.jsonindex 779d49aa3a..bd5b044130 100644--- a/test/fixtures/goldens/doctor/hot-checks-pglite-json.json+++ b/test/fixtures/goldens/doctor/hot-checks-pglite-json.json@@ -3770,6 +3770,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -3785,6 +3786,7 @@"samples": [],"source_id": "notes","status": "known",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -3798,6 +3800,7 @@"samples": [],"source_id": "plain","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}@@ -8607,6 +8610,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -8622,6 +8626,7 @@"samples": [],"source_id": "notes","status": "known",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -8635,6 +8640,7 @@"samples": [],"source_id": "plain","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}diff --git a/test/fixtures/goldens/doctor/hot-checks-postgres-json.json b/test/fixtures/goldens/doctor/hot-checks-postgres-json.jsonindex 5a698fd9fa..8a395cbf5f 100644--- a/test/fixtures/goldens/doctor/hot-checks-postgres-json.json+++ b/test/fixtures/goldens/doctor/hot-checks-postgres-json.json@@ -3773,6 +3773,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -3788,6 +3789,7 @@"samples": [],"source_id": "notes","status": "known",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -3801,6 +3803,7 @@"samples": [],"source_id": "plain","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}@@ -8611,6 +8614,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -8626,6 +8630,7 @@"samples": [],"source_id": "notes","status": "known",+ "stopped_at": null,"unproven": 0,"unproven_samples": []},@@ -8639,6 +8644,7 @@"samples": [],"source_id": "plain","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}diff --git a/test/fixtures/goldens/doctor/json-pglite-degraded.json b/test/fixtures/goldens/doctor/json-pglite-degraded.jsonindex e874a59509..84e8f0aeaa 100644--- a/test/fixtures/goldens/doctor/json-pglite-degraded.json+++ b/test/fixtures/goldens/doctor/json-pglite-degraded.json@@ -3252,6 +3252,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}diff --git a/test/fixtures/goldens/doctor/json-pglite-fresh.json b/test/fixtures/goldens/doctor/json-pglite-fresh.jsonindex 8d22e4e79a..bc718e0390 100644--- a/test/fixtures/goldens/doctor/json-pglite-fresh.json+++ b/test/fixtures/goldens/doctor/json-pglite-fresh.json@@ -3295,6 +3295,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}diff --git a/test/fixtures/goldens/doctor/json-postgres-fresh.json b/test/fixtures/goldens/doctor/json-postgres-fresh.jsonindex 2f4a9851e0..96067454bc 100644--- a/test/fixtures/goldens/doctor/json-postgres-fresh.json+++ b/test/fixtures/goldens/doctor/json-postgres-fresh.json@@ -3298,6 +3298,7 @@"samples": [],"source_id": "default","status": "not_applicable",+ "stopped_at": null,"unproven": 0,"unproven_samples": []}diff --git a/test/rows-behind-checkpoint-8071.test.ts b/test/rows-behind-checkpoint-8071.test.tsindex b6acef524d..bcd7a36bbd 100644--- a/test/rows-behind-checkpoint-8071.test.ts+++ b/test/rows-behind-checkpoint-8071.test.ts@@ -16,6 +16,8 @@* - a database edit made since the last import, a held path and a path changed after the checkpoint are not behind; a* row with no import record is listed apart and joins the set only with --include-ambiguous;* - a checkpoint file the import refuses is held with the refusal's code instead of imported;+ * - the scan checks its deadline inside a batch and reports how far it got; doctor counts unproven rows without the+ * importer's confirmation; a hold row applies the refusal the apply computes;* - repo-less: a brain with no checkout reports not_applicable (ok) and the preview lists nothing.** PGLite always; Postgres too when DATABASE_URL is set (test/e2e wrapper).@@ -37,6 +39,7 @@ import { rowsBehindEntry } from '../src/commands/doctor/checks/rows-behind.ts';import { categorizeCheck } from '../src/core/doctor-categories.ts';import { RECOVERY_VERSION } from '../src/core/markdown.ts';import { SYNC_READ_BOUND } from '../src/core/persistence/sync-blobs.ts';+import { PREVIEW_APPROVAL_OP } from '../src/core/persistence/preview-approval.ts';import { managedBrain } from './helpers/managed-brain.ts';import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';import { requirePostgresTestDatabase, testBackends } from './helpers/test-backends.ts';@@ -231,6 +234,51 @@ for (const backend of testBackends()) describe(`[${backend}] #8071: rows behindexpect(await doctor(engine)).toMatchObject({ status: 'ok', details: { count: 0 } });}, { databaseUrl, setup }), 240_000);+ test('the scan checks its deadline inside a batch and says how far it got; doctor counts unproven rows without confirming them', () => managedBrain(async ({ engine, root }) => {+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');+ await seedBehind(engine, root, Object.fromEntries(SLUGS.map(slug => [slug, { content: note(slug, `The redacted body of ${slug}.`), record: slug === 'notes/e' ? 'none' as const : 'checkpoint' as const }])));+ const full = await scanRowsBehind(engine, 'default');+ expect(full).toMatchObject({ complete: true, pages: 5, behind_count: 4, unproven_count: 1 });+ expect(full.unproven[0]!.digest).toBeDefined();++ // A clock that passes the deadline on its eighth read (one before the batch, five before the comparisons, one before+ // the first confirmation): the one batch of five rows must stop part way through, after one confirmed row.+ let reads = 0;+ const stopped = await scanRowsBehind(engine, 'default', { deadlineAt: 1, now: () => (reads++ < 7 ? 0 : 1) });+ expect(stopped.complete).toBe(false);+ expect(stopped.pages).toBeGreaterThan(0);+ expect(stopped.pages).toBeLessThan(5);+ expect(typeof stopped.stopped_at).toBe('number');+ expect(stopped.behind_count + stopped.unproven_count).toBeLessThan(5);+ expect(Object.values(stopped.counts).reduce((sum, n) => sum + n, 0) + stopped.behind_count + stopped.unproven_count).toBe(stopped.pages);++ const counted = await scanRowsBehind(engine, 'default', { confirmUnproven: false });+ expect(counted).toMatchObject({ complete: true, behind_count: 4, unproven_count: 1 });+ expect(counted.unproven[0]!.digest).toBeUndefined();+ const warned = await doctor(engine);+ expect(warned).toMatchObject({ status: 'warn', details: { count: 4, unproven: 1 } });+ expect(warned.message).toContain('replaces any database-only edit');++ const partial = await withEnv({ GBRAIN_DOCTOR_ROWS_BEHIND_TIMEOUT_MS: '1' }, async () => { await new Promise(resolve => setTimeout(resolve, 5)); return doctor(engine); });+ expect(partial).toMatchObject({ status: 'ok', details: { sources: [{ complete: false }] } });+ expect(partial.message).toMatch(/stopped after \d+ page\(s\)/);+ }, { databaseUrl, setup }), 240_000);++ test('a hold row applies the refusal the apply computes, not the one saved with the preview', () => managedBrain(async ({ engine, root }) => {+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');+ await seedBehind(engine, root, { 'notes/a': { content: '---\ntype: concept\ntitle: [unclosed\n---\n\nThe redacted body of notes/a.\n', record: 'checkpoint' } });+ const preview = await repair(engine);+ expect(listing(preview)).toEqual([['default:notes/a', 'hold:invalid_frontmatter']]);+ // Age the saved set: its refusal no longer matches what the import of the committed file says.+ await engine.executeRaw(`UPDATE op_checkpoints SET completed_keys = jsonb_set(completed_keys, '{0,items,0,refusal}', '{"code":"invalid_fence","message":"saved with an older preview"}'::jsonb)+ WHERE op=$1 AND completed_keys->0->>'hash'=$2`, [PREVIEW_APPROVAL_OP, preview.preview_hash]);+ const applied = await repair(engine, { apply: true, expect: preview.preview_hash });+ expect(applied.outcome_items?.map(item => [item.item, item.outcome])).toEqual([['default:notes/a', 'held']]);+ const hold = await readGitHold(engine, 'default', await incarnationOf(engine), 'notes/a.md');+ expect(hold?.code).toBe('invalid_frontmatter');+ expect(hold?.message).not.toContain('saved with an older preview');+ }, { databaseUrl, setup }), 240_000);+test('repo-less: a brain with no checkout is not applicable and the preview lists nothing', async () => {const home = (await import('node:fs')).mkdtempSync(join((await import('node:os')).tmpdir(), 'gbrain-rows-behind-'));await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {