Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(repair): repair_preview_timeout points at the read-only index preview and reads index states outside an expired deadline
garrytan0dd774a82c18
diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.mdindex e54f9e5dc6..64c564d3cd 100644--- a/docs/guides/troubleshooting.md+++ b/docs/guides/troubleshooting.md@@ -79,7 +79,7 @@| [`gbrain sync` refused `managed_sync_flag_refused`](write-refusals.md#managed_sync_flag_refused) | rerun the same command without the flag `detail` names (`--skip-failed` or `--include-gitignored`); fix a failed file and rerun the plain sync (`--retry-failed` is a deprecated alias), or commit an ignored file | agent | none | `gbrain sources status --json` || [Sync with the deprecated `--retry-failed` alias says failures "belong to a run with different sync options"](#retry-other-options) | run the command printed on each line: `resumable (replay …)` or `resumable (fresh discovery …)` reruns under the failed run's options, `safely orphaned` previews `gbrain repair managed-sync-orphans --source <id>` (apply with its hash), `unresolved` names what has to finish first | agent; the repair apply after the user agrees | the repair retires bookkeeping rows only | `gbrain sync status --source <id> --json` || [Managed sync blocked with `checkpoint_validation_timeout`](#checkpoint-validation-timeout) | `gbrain repair request-indexes --apply` when an index is missing or INVALID, then the printed `gbrain sync --source <id> --no-pull …` | brain host | none | `gbrain doctor --only persistence_request_indexes --json` |-| [`gbrain repair failed-writes` preview stopped with `repair_preview_timeout`](#repair-preview-timeout) | follow the refusal's `fix`: `gbrain repair request-indexes --apply` when it names an index, then the same preview | agent (indexes); brain host (a longer timeout) | none | `gbrain doctor --only persistence_request_indexes --json` |+| [`gbrain repair failed-writes` preview stopped with `repair_preview_timeout`](#repair-preview-timeout) | follow the refusal's `fix`: when it names an index, `gbrain repair request-indexes --json` previews the rebuild and `gbrain repair request-indexes --apply` runs it; then the same preview | agent (indexes); brain host (a longer timeout) | none | `gbrain doctor --only persistence_request_indexes --json` || Writes fail with `connection_inherited_read_only` (SQLSTATE 25006) | [reset the pooler's server connections, stop session-level `SET`, check role defaults](write-refusals.md#connection_inherited_read_only) | brain host | none | `gbrain doctor --json` || [Doctor warns `persistence_session_timeouts` (`session_timeouts_not_applied`)](#session-timeouts-not-applied) | `ALTER ROLE <gbrain role> SET statement_timeout = '5min'` on the database, or accept it: preparation reads carry their own bound | brain host | none (a role default; `destructive` only in that it changes a server setting) | `gbrain doctor --only persistence_session_timeouts --json` || `gbrain init` or `gbrain upgrade` stops with `schema_lock_blocked` naming a pid (or, before #5227, sat in `wait_event_type=Lock` with no output) | let the named session finish, or with the user's agreement end it on the database host: `SELECT pg_terminate_backend(<pid>)`; then rerun the same command ([ENGINES.md](../ENGINES.md#schema-lock-blocked)) | brain host, after the user agrees | `destructive` when it ends another session | the rerun completes; `gbrain doctor --json` shows the schema at head |@@ -132,7 +132,7 @@ Turning the `preparation_deadlines` switch off (`gbrain config set persistence.p<a id="checkpoint-validation-timeout"></a>**A managed sync is blocked with `checkpoint_validation_timeout`?** Run the printed commands on the brain host: `gbrain repair request-indexes --apply` when an index is missing or INVALID, then the printed `gbrain sync --source <id> --no-pull …`. Doctor's `persistence_request_growth` warns before lifetime request IDs run out.-<a id="repair-preview-timeout"></a>**The `gbrain repair failed-writes` preview stopped with `repair_preview_timeout`?** One of its reads of the write journal ran past the database statement timeout, so nothing was saved, no approval checkpoint was written and no page changed. The preview reads the journal in a few set reads (replay attempts by request id, later requests of each page, recovered fact requests) through `persistence_requests_replay_id` and `persistence_requests_slug_sequence`. When the refusal names one of them as missing, INVALID or still building, run `gbrain repair request-indexes --apply` (online, no user data changes), check with `gbrain doctor --only persistence_request_indexes --json`, and run the same preview again. When both indexes are valid, run the preview once on the brain host with a longer statement timeout for that command (`GBRAIN_STATEMENT_TIMEOUT=15min`; behind a transaction-mode pooler that setting is dropped, so raise the role's default as in [session timeouts](#session-timeouts-not-applied)). Older releases ended this preview with a bare `internal_error` on a large request table (#8707).+<a id="repair-preview-timeout"></a>**The `gbrain repair failed-writes` preview stopped with `repair_preview_timeout`?** One of its reads of the write journal ran past the database statement timeout, so nothing was saved, no approval checkpoint was written and no page changed. The preview reads the journal in a few set reads (replay attempts by request id, later requests of each page, recovered fact requests) through `persistence_requests_replay_id` and `persistence_requests_slug_sequence`. When the refusal names one of them as missing, INVALID or still building, preview the rebuild with `gbrain repair request-indexes --json`, run `gbrain repair request-indexes --apply` (online, no user data changes), check with `gbrain doctor --only persistence_request_indexes --json`, and run the same preview again. When both indexes are valid, run the preview once on the brain host with a longer statement timeout for that command (`GBRAIN_STATEMENT_TIMEOUT=15min`; behind a transaction-mode pooler that setting is dropped, so raise the role's default as in [session timeouts](#session-timeouts-not-applied)). Older releases ended this preview with a bare `internal_error` on a large request table (#8707).<a id="pglite-aborted"></a>**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](../ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](../INSTALL.md#pglite-crashes-at-startup-runtimeerror-aborted).diff --git a/src/core/repair/failed-writes.ts b/src/core/repair/failed-writes.tsindex c131c14173..fce9b1add1 100644--- a/src/core/repair/failed-writes.ts+++ b/src/core/repair/failed-writes.ts@@ -64,7 +64,7 @@ import { dispositionOf, PAGE_REMOVAL_OPERATIONS, supersessionOrderSql, supersessimport { isStatementTimeout } from '../persistence/bounded-reads.ts';import { readRequestIndexStates } from '../persistence/checkpoint-validation.ts';import { PERSISTENCE_REQUEST_REPLAY_ID_INDEX, PERSISTENCE_REQUEST_SLUG_SEQUENCE_INDEX } from '../persistence/schema.ts';-import { StatementDeadlineError } from '../statement-deadline.ts';+import { StatementDeadlineError, withStatementDeadline } from '../statement-deadline.ts';import { afterCursor, type RepairHandler, type RepairItem, type RepairItemOutcome, type RepairPlan, type RepairScope } from './core.ts';const REPLAYABLE = ['put_page', 'add_timeline_entry', 'remember'];@@ -426,7 +426,7 @@ async function previewTimeoutRefusal(engine: BrainEngine, error: unknown, scope:if (!isStatementTimeout(error) && !(error instanceof StatementDeadlineError)) return error;const command = previewCommand(scope);const argv = ['gbrain', 'repair', 'failed-writes', ...(scope.source_ids.length === 1 ? ['--source', scope.source_ids[0]!] : []), '--json'];- const unbuilt = (await readRequestIndexStates(engine, [PERSISTENCE_REQUEST_REPLAY_ID_INDEX, PERSISTENCE_REQUEST_SLUG_SEQUENCE_INDEX]).catch(() => []))+ const unbuilt = (await withStatementDeadline(null, () => readRequestIndexStates(engine, [PERSISTENCE_REQUEST_REPLAY_ID_INDEX, PERSISTENCE_REQUEST_SLUG_SEQUENCE_INDEX])).catch(() => [])).filter(index => index.state !== 'valid').map(index => `${index.name} (${index.state})`);const refusal = new OperationError('repair_preview_timeout','The failed-writes preview stopped: a read of the write journal ran past the database statement timeout. Nothing was saved and no page changed.',@@ -438,8 +438,8 @@ async function previewTimeoutRefusal(engine: BrainEngine, error: unknown, scope:? `Without ${unbuilt.join(' and ')} a lookup of the preview reads the whole request table.`: 'Every preview read is a set read over the request table; on this brain one of them took longer than the statement timeout allows.';refusal.fix = unbuilt.length- ? { argv: ['gbrain', 'repair', 'request-indexes', '--apply'], consent: [], actor: 'agent', requires_exclusive: false,- why: 'Builds the missing or INVALID request-table indexes online; it changes no user data.',+ ? { argv: ['gbrain', 'repair', 'request-indexes', '--json'], consent: [], actor: 'agent', requires_exclusive: false,+ why: 'Previews the request-table indexes a rebuild would build, without changing anything; its apply command builds them online.',verify: { argv: ['gbrain', 'doctor', '--only', 'persistence_request_indexes', '--json'] }, docs: 'docs/guides/repair.md#request-indexes' }: { argv, consent: [], actor: 'host_admin', requires_exclusive: false, why: 'Re-runs the read-only preview.',user_message: `The preview of refused writes did not finish within the database statement timeout. On the brain host, run ${command} with a longer timeout for that one command (GBRAIN_STATEMENT_TIMEOUT=15min; behind a transaction-mode pooler that setting is dropped, so raise the role's default instead).`,diff --git a/test/repair-failed-writes-bounded-8707.test.ts b/test/repair-failed-writes-bounded-8707.test.tsindex 408f111e35..f67fa61867 100644--- a/test/repair-failed-writes-bounded-8707.test.ts+++ b/test/repair-failed-writes-bounded-8707.test.ts@@ -18,6 +18,7 @@ import { digest } from '../src/core/persistence/digest.ts';import { FACTS_ENTITY_MOVED_MESSAGES } from '../src/core/persistence/verb-errors.ts';import { PERSISTENCE_REQUEST_REPLAY_ID_INDEX, PERSISTENCE_REQUEST_SLUG_SEQUENCE_INDEX } from '../src/core/persistence/schema.ts';import { repairRunner } from '../src/core/repair/registry.ts';+import { deadlineBoundEngine, StatementDeadlineError, withStatementDeadline, type StatementDeadline } from '../src/core/statement-deadline.ts';import { resolveRepairScope } from '../src/core/repair/core.ts';import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';import { testBackends } from './helpers/test-backends.ts';@@ -151,9 +152,23 @@ for (const backend of testBackends()) describe(`[${backend}] #8707 the failed-wrawait engine.executeRaw(`DROP INDEX ${PERSISTENCE_REQUEST_REPLAY_ID_INDEX.name}`);try {const refusal = await refused(timeout);- expect(refusal.fix).toMatchObject({ actor: 'agent', argv: ['gbrain', 'repair', 'request-indexes', '--apply'] });+ expect(refusal.fix).toMatchObject({ actor: 'agent', argv: ['gbrain', 'repair', 'request-indexes', '--json'] });expect(refusal.suggestion).toContain(`${PERSISTENCE_REQUEST_REPLAY_ID_INDEX.name} (missing)`);expect(refusal.suggestion).not.toContain(PERSISTENCE_REQUEST_SLUG_SEQUENCE_INDEX.name);+ // The caller's own deadline ends the read: the index states are still read, outside that expired deadline.+ const scope: StatementDeadline = { startedAt: Date.now(), deadlineAt: Date.now() + 60_000, expired: false };+ const bound = deadlineBoundEngine(engine);+ const expiring = new Proxy(bound, { get(target, key) {+ const value = Reflect.get(target, key, target);+ if (key !== 'executeRaw') return typeof value === 'function' ? value.bind(target) : value;+ return (sql: string, ...rest: unknown[]) => {+ if (sql.includes('request_id = ANY')) { scope.deadlineAt = Date.now() - 1; scope.expired = true; return Promise.reject(new StatementDeadlineError(1)); }+ return (value as Function).call(target, sql, ...rest);+ };+ } }) as BrainEngine;+ const deadlined = await withStatementDeadline(scope, () => refused(expiring));+ expect(deadlined.fix).toMatchObject({ actor: 'agent', argv: ['gbrain', 'repair', 'request-indexes', '--json'] });+ expect(deadlined.suggestion).toContain(`${PERSISTENCE_REQUEST_REPLAY_ID_INDEX.name} (missing)`);} finally { await engine.executeRaw(PERSISTENCE_REQUEST_REPLAY_ID_INDEX.sql); }}expect(await checkpoints()).toBe(before);