Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(refresh): #8072 a refresh's member syncs run governed with a deadline; refresh_sync_timeout

garrytanbc1c768f9b53
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index c1109ee612..85da49e76e 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -3551,6 +3551,16 @@ More: [docs/guides/on-commit-sync.md#refresh-required](../../docs/guides/on-comm
More: [docs/guides/write-refusals.md#refresh_source_changed](../../docs/guides/write-refusals.md#refresh_source_changed)
+### refresh_sync_timeout
+
+<a id="refresh_sync_timeout"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| A refresh's member sync committed nothing within the refresh sync budget. | A temporary condition (a lock, a pending write, a rate limit); the same request can succeed later. | Wait briefly, then retry the same request (writes: reuse the same request_id). | agent | `repeat the read that failed` | 1 | yes |
+
+More: [docs/guides/write-refusals.md#refresh_sync_timeout](../../docs/guides/write-refusals.md#refresh_sync_timeout)
+
### relay_disabled
<a id="relay_disabled"></a>
diff --git a/docs/guides/live-sync.md b/docs/guides/live-sync.md
index 0f2bdc09c3..b800d1d75f 100644
--- a/docs/guides/live-sync.md
+++ b/docs/guides/live-sync.md
@@ -653,11 +653,16 @@ vars — incident-time escape hatches, not everyday knobs.
`--wait-drain <seconds>` (default 60; `sources.refresh_drain_wait_ms`,
`GBRAIN_REFRESH_DRAIN_WAIT_MS`) and `--fetch-timeout-ms` (default 120000;
`sources.refresh_fetch_timeout_ms`, `GBRAIN_REFRESH_FETCH_TIMEOUT_MS`).
- The member syncs of one run stop starting work after
+ The member syncs of one run share a deadline,
`sources.refresh_sync_budget_ms` (default 300000; `GBRAIN_REFRESH_SYNC_BUDGET_MS`),
so a scheduled run never holds the refresh lock much longer than that:
a run past it reports `syncing` (members `deferred`) and the next
- `gbrain sources refresh <source-id>` continues where it stopped.
+ `gbrain sources refresh <source-id>` continues where it stopped. A member
+ whose sync committed nothing in the whole budget (a write no consumer
+ claims) is reported in `sync_blocked` as
+ [`refresh_sync_timeout`](write-refusals.md#refresh_sync_timeout); the
+ refresh then completes and releases its lock, and that source catches up
+ on its next sync.
A refresh whose process died (a crash, a replaced container) does not
wedge the checkout: the owner finishes what it can on its own, and the next
`gbrain sources refresh <source-id>` on the owner host finishes the rest
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index ef4f658c03..a9b2eeeeaf 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -553,6 +553,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.
| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |
| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). A refresh whose holder is recorded on another host is never taken over either: writer status shows its `liveness` as `unknown`, and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
| <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |
+| <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within the refresh sync budget (`sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. |
| <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. |
| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |
| <a id="worktree_refreshing"></a>`worktree_refreshing` | `worktree_refreshing` | A write, or a managed sync started by a cycle or another command, reached a checkout while a refresh is draining or merging it. It was not journaled. `detail` is `retry_after_ms=1000`. Writes to database-only memory are not affected. | Retry the same request ID after about a second. `gbrain sources writer status <source>` shows the refresh. A cycle's sync is retried by the next cycle. |
diff --git a/llms-full.txt b/llms-full.txt
index a007f2d123..76893d968d 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -7389,11 +7389,16 @@ vars — incident-time escape hatches, not everyday knobs.
`--wait-drain <seconds>` (default 60; `sources.refresh_drain_wait_ms`,
`GBRAIN_REFRESH_DRAIN_WAIT_MS`) and `--fetch-timeout-ms` (default 120000;
`sources.refresh_fetch_timeout_ms`, `GBRAIN_REFRESH_FETCH_TIMEOUT_MS`).
- The member syncs of one run stop starting work after
+ The member syncs of one run share a deadline,
`sources.refresh_sync_budget_ms` (default 300000; `GBRAIN_REFRESH_SYNC_BUDGET_MS`),
so a scheduled run never holds the refresh lock much longer than that:
a run past it reports `syncing` (members `deferred`) and the next
- `gbrain sources refresh <source-id>` continues where it stopped.
+ `gbrain sources refresh <source-id>` continues where it stopped. A member
+ whose sync committed nothing in the whole budget (a write no consumer
+ claims) is reported in `sync_blocked` as
+ [`refresh_sync_timeout`](write-refusals.md#refresh_sync_timeout); the
+ refresh then completes and releases its lock, and that source catches up
+ on its next sync.
A refresh whose process died (a crash, a replaced container) does not
wedge the checkout: the owner finishes what it can on its own, and the next
`gbrain sources refresh <source-id>` on the owner host finishes the rest
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 49e1969327..91c4396e69 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1373,7 +1373,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.514.0', id: 'company_db_only_keep_8014', text: 'A company-brain source now keeps a live page under a declared `storage.db_only` directory when its file is not in the approved manifest, and keeps every such page while its gbrain.yml storage section is invalid, as other managed sources do; it used to delete them. The decision reads gbrain.yml at the approved commit, never the working tree.' },
{ since: '0.60.514.0', id: 'redaction_marker_not_credential_7997', text: 'The transcript `labeled_credential` detector no longer reads an importer redaction marker (`login **<REDACTED:user-pattern>**`) as a typed user/password pair, so a correctly redacted page clears `gbrain transcripts audit-secrets`; a password written beside a marker is still found. The audit detector version moves to 3, so `gbrain doctor` asks for a fresh audit once.' },
{ since: 'TBD', id: 'oversize_hold_unread_8072', text: 'A Markdown file over 24 MiB that sync had to read whole to refuse (segmented storage on, held `file_too_large`, for example for a facts fence) is no longer read again by every sync: it stays held, unread, until it changes, gbrain is upgraded, or `gbrain sources retry-held <source>` schedules it. After changing a setting the hold message names, run `gbrain sources retry-held <source>`. A file held from its size alone still re-checks on every sync.' },
- { since: 'TBD', id: 'refresh_sync_budget_8072', text: 'The member syncs of one `gbrain sources refresh` run stop starting work after `sources.refresh_sync_budget_ms` (default 5 minutes; `GBRAIN_REFRESH_SYNC_BUDGET_MS`). A run past it reports `syncing` with the members it left `deferred`, and the next `gbrain sources refresh <source>` continues it, so a slow sync no longer holds the refresh lock across scheduled runs.' },
+ { since: 'TBD', id: 'refresh_sync_budget_8072', text: 'The member syncs of one `gbrain sources refresh` run stop starting work after `sources.refresh_sync_budget_ms` (default 5 minutes; `GBRAIN_REFRESH_SYNC_BUDGET_MS`). A run past it reports `syncing` with the members it left `deferred`, and the next `gbrain sources refresh <source>` continues it; a member that committed nothing in the whole budget is reported `refresh_sync_timeout` in `sync_blocked` and the refresh completes, so a stalled sync no longer holds the refresh lock across scheduled runs.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/error-catalogue.ts b/src/core/error-catalogue.ts
index e3d0089d55..531a5df2e3 100644
--- a/src/core/error-catalogue.ts
+++ b/src/core/error-catalogue.ts
@@ -87,6 +87,7 @@ export const ERROR_CATALOGUE = {
sync_in_progress: { code: 'sync_in_progress', docs: 'docs/guides/write-refusals.md#sync_in_progress' },
refresh_in_progress: { code: 'refresh_in_progress', docs: 'docs/guides/write-refusals.md#refresh_in_progress' },
refresh_drain_timeout: { code: 'refresh_drain_timeout', docs: 'docs/guides/write-refusals.md#refresh_drain_timeout' },
+ refresh_sync_timeout: { code: 'refresh_sync_timeout', docs: 'docs/guides/write-refusals.md#refresh_sync_timeout' },
refresh_source_changed: { code: 'refresh_source_changed', docs: 'docs/guides/write-refusals.md#refresh_source_changed' },
refresh_recovery_required: { code: 'refresh_recovery_required', docs: 'docs/guides/write-refusals.md#refresh_recovery_required' },
worktree_refreshing: { code: 'worktree_refreshing', docs: 'docs/guides/write-refusals.md#worktree_refreshing' },
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 686b0115f5..950eadc398 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -605,6 +605,7 @@ export const CODES = {
refresh_not_managed: { class: 'host_only', summary: "The source is not managed, so it cannot be refreshed." },
refresh_not_owner: { class: 'host_only', summary: "This process does not own the worktree it would refresh." },
refresh_source_changed: { class: 'caller', summary: "The source changed during the refresh." },
+ refresh_sync_timeout: { class: 'retryable', summary: "A refresh's member sync committed nothing within the refresh sync budget." },
relay_disabled: { class: 'caller', summary: "Google connect credential error: relay disabled.", docs: 'docs/guides/google-connect.md#troubleshooting' },
relay_session_expired: { class: 'caller', summary: "Google connect credential error: relay session expired.", docs: 'docs/guides/google-connect.md#troubleshooting' },
relay_unreachable: { class: 'retryable', summary: "Google connect credential error: relay unreachable.", docs: 'docs/guides/google-connect.md#troubleshooting' },
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index 34dac39b58..cd85139659 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -387,8 +387,8 @@ async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRo
for (let pass = 0; pass < MAX_MEMBER_SYNC_PASSES; pass++) {
if (await reachedTarget(git, root, row, await lastCommit(engine, sourceId))) break;
if (sync.spent()) { status = 'deferred'; break; }
- let result: SyncResult;
- try { result = await sync.run(engine, sourceId); }
+ let result: MemberSync;
+ try { result = await sync.member(engine, sourceId); }
catch (error) {
const failure = error instanceof OperationError ? error : opError('storage_error', error instanceof Error ? error.message : String(error),
`The managed sync of source ${sourceId} failed during refresh ${row.id}. Check its writer status, fix the cause, then resume with gbrain sync --source ${sourceId} --no-pull.`);
@@ -397,12 +397,9 @@ async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRo
break;
}
status = result.status;
- if (result.status === 'blocked_by_failures') {
- blocked.push({ source_id: sourceId, code: result.managedWrite?.write_error ?? result.failureCodes?.[0]?.code ?? 'sync_blocked',
- message: result.managedWrite?.message ?? 'The managed sync is blocked by a failed write.', resume: `gbrain sync --source ${sourceId} --no-pull` });
- break;
- }
- if (result.status === 'partial') await new Promise(resolve => setTimeout(resolve, DRAIN_POLL_MS));
+ if (result.blocked) { blocked.push(result.blocked); break; }
+ if (status === 'deferred') break;
+ if (status === 'partial') await new Promise(resolve => setTimeout(resolve, DRAIN_POLL_MS));
}
await recordUpstreamObservation(engine, sourceId);
synced.push({ source_id: sourceId, status, last_commit: await lastCommit(engine, sourceId) });
@@ -413,19 +410,51 @@ async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRo
{ synced, ...(blocked.length ? { sync_blocked: blocked } : {}) }));
return { synced, blocked, atTarget };
}
+type MemberSync = { status: string; blocked?: RefreshSyncBlocked };
/**
- * #8072: the member syncs of one refresh share one wall-clock budget, so a run never holds the refresh lock for longer than
- * that plus one entry: each pass is a managed-sync slice that yields `partial` (`writer_yield`) at an entry boundary once the
- * budget ran, and no pass starts after it. A member left behind reports `deferred`; the refresh stays `syncing` and the next
- * plain refresh (or `--resume`) continues it.
+ * #8072: the member syncs of one refresh share one deadline and run through the governed drain (sync-drain.ts) under the
+ * source's drain lease, as `gbrain sync` and a conversion's strict pass do, so no pass outlives the deadline by more than the
+ * drain's grace. A drain the deadline stopped after the member moved reports `deferred` (the refresh stays `syncing` and the
+ * next plain refresh or `--resume` continues it); one that moved nothing in the whole budget is blocked `refresh_sync_timeout`,
+ * naming the write it waited on, as is a drain its own rules blocked (`drain_stalled` and the rest), so the refresh completes
+ * with that member behind its target and releases the refresh lock instead of re-entering bare passes under it.
*/
async function boundedMemberSync(budgetMs: number) {
- const { performManagedSync } = await import('./sync-run.ts');
- const deadline = performance.now() + budgetMs;
+ const [{ drainManagedSync, engineStallProbe }, { headBlockerText }, { withDrainLease, leaseHeldResult, readDrainHolder }, { withSyncTail }, { LockUnavailableError, syncLockId }] = await Promise.all([
+ import('./sync-drain.ts'), import('./head-blocker.ts'), import('./sync-lease.ts'), import('./sync-tail.ts'), import('../db-lock.ts')]);
+ const deadlineAtMs = Date.now() + budgetMs;
+ const drain = async (engine: BrainEngine, sourceId: string): Promise<SyncResult> => {
+ const opts = { sourceId, noPull: true, deadlineAtMs };
+ try {
+ const drained = await withDrainLease(engine, sourceId, { holder: { kind: 'refresh', id: null }, remote: false },
+ signal => drainManagedSync(engine, signal ? { ...opts, signal } : opts, false));
+ return await withSyncTail(engine, sourceId, drained, { deadlineAtMs });
+ } catch (error) {
+ if (!(error instanceof LockUnavailableError) || error.lockId !== syncLockId(sourceId)) throw error;
+ return leaseHeldResult(sourceId, await readDrainHolder(engine, sourceId).catch(() => null));
+ }
+ };
return {
- spent: () => performance.now() >= deadline,
- run: (engine: BrainEngine, sourceId: string) => performManagedSync(engine, { sourceId, noPull: true, drainHolder: { kind: 'cli', id: null } },
- { maxPages: Number.MAX_SAFE_INTEGER, maxMs: Math.max(0, deadline - performance.now()) }),
+ spent: () => Date.now() >= deadlineAtMs,
+ member: async (engine: BrainEngine, sourceId: string): Promise<MemberSync> => {
+ const resume = `gbrain sync --source ${sourceId} --no-pull`;
+ const before = await lastCommit(engine, sourceId);
+ const result = await drain(engine, sourceId);
+ if (result.status === 'blocked_by_failures') return { status: result.status, blocked: { source_id: sourceId, resume,
+ code: result.managedWrite?.write_error ?? result.failureCodes?.[0]?.code ?? 'sync_blocked', message: result.managedWrite?.message ?? 'The managed sync is blocked by a failed write.' } };
+ const report = result.drain;
+ if (report?.outcome === 'blocked') {
+ const stall = report.stall;
+ return { status: 'blocked', blocked: { source_id: sourceId, resume, code: report.stop_reason ?? 'sync_blocked',
+ message: `The managed sync stopped ${report.stop_reason ?? 'blocked'}${stall ? ` on write request ${stall.request_id} (${stall.state}${stall.cause ? `, ${stall.cause}` : ''}, ${stall.stalled_seconds}s without progress)` : ''}; the refresh completed with this source behind its target.` } };
+ }
+ if (report?.outcome !== 'resumable' || report.stop_reason !== 'deadline') return { status: result.status };
+ if (report.passes === 0 || report.processed > 0 || await lastCommit(engine, sourceId) !== before) return { status: 'deferred' };
+ const head = await engineStallProbe(engine, sourceId).head?.().catch(() => null) ?? null;
+ const waiting = head ? `; its oldest unfinished write request ${head.write_request_id ?? head.request_id ?? 'unknown'} is ${head.head_state}${head.blocked_by ? `, blocked by ${headBlockerText(head.blocked_by)}` : ''}${head.consumer ? `, consumer ${head.consumer}` : ''}` : '';
+ return { status: 'blocked', blocked: { source_id: sourceId, resume, code: 'refresh_sync_timeout',
+ message: `The managed sync committed nothing within the refresh's sync budget of ${budgetMs} ms${waiting}. The refresh completed with this source behind its target; its next sync continues it.` } };
+ },
};
}
async function lastCommit(engine: BrainEngine, sourceId: string): Promise<string | null> {
@@ -472,9 +501,8 @@ async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshO
if (commit === plan.oldHead) { synced.push({ source_id: member, status: 'up_to_date', last_commit: plan.oldHead }); continue; }
if (sync.spent()) { synced.push({ source_id: member, status: 'deferred', last_commit: commit }); continue; }
try {
- const result = await sync.run(engine, member);
- if (result.status === 'blocked_by_failures') blocked.push({ source_id: member, code: result.managedWrite?.write_error ?? 'sync_blocked',
- message: result.managedWrite?.message ?? 'The managed sync is blocked by a failed write.', resume: `gbrain sync --source ${member} --no-pull` });
+ const result = await sync.member(engine, member);
+ if (result.blocked) blocked.push(result.blocked);
synced.push({ source_id: member, status: result.status, last_commit: await lastCommit(engine, member) });
} catch (error) {
blocked.push({ source_id: member, code: (error as OperationError).code ?? 'storage_error', message: (error as Error).message, resume: `gbrain sync --source ${member} --no-pull` });
diff --git a/test/oversize-hold-rescreen-8072.test.ts b/test/oversize-hold-rescreen-8072.test.ts
index 2771e2239f..4473e3e1ce 100644
--- a/test/oversize-hold-rescreen-8072.test.ts
+++ b/test/oversize-hold-rescreen-8072.test.ts
@@ -6,20 +6,29 @@
* block read, the hold's attempt count unchanged. `gbrain sources retry-held` (and a gbrain upgrade) still
* re-screens it; a hold decided from the size alone keeps re-screening every sync (one lstat).
* The member syncs of one refresh are bounded by `sources.refresh_sync_budget_ms`: past it the refresh stays
- * `syncing` and the next plain refresh finishes it.
+ * `syncing` and the next plain refresh finishes it. A member whose sync commits nothing in the whole budget (its write
+ * is never claimed: the `consumer:claiming` fault point holds the consumer) is reported `refresh_sync_timeout`, the
+ * refresh completes and its lock is free, with hundreds of preserved uncommitted paths in the checkout.
* 2. Fails when: every tick re-hashes and re-streams the unchanged held file (the hold re-screened on every sync), or a
- * refresh tick keeps syncing past its budget under the refresh lock.
- * Counts, not wall-clock time: write requests, `streamedImportWork` bytes and the hold's attempts.
+ * refresh tick keeps syncing past its budget under the refresh lock (on master the stalled refresh re-entered bare
+ * passes for about 300 s and did not return within the test's timeout).
+ * Counts, not wall-clock time: write requests, `streamedImportWork` bytes and the hold's attempts; the one time
+ * assertion is the stalled refresh's upper bound (budget + the drain's deadline grace + 30 s of slack).
* Repo-less brains have no managed sync or refresh (both need Git), so nothing here runs there.
*/
import { afterAll, beforeAll, expect, test } from 'bun:test';
+import { writeFileSync } from 'node:fs';
+import { join } from 'node:path';
import type { BrainEngine } from '../src/core/engine.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { PAGE_MAX_BYTES } from '../src/core/page-size-policy.ts';
import { streamedImportWork } from '../src/core/page-body/streamed-import.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
import { holdRescreenDue, readGitSourceHolds, type GitHoldRecord } from '../src/core/persistence/sync-holds.ts';
-import { refreshWorktree } from '../src/core/persistence/worktree-refresh.ts';
+import { refreshLiveness, refreshWorktree } from '../src/core/persistence/worktree-refresh.ts';
+import { installFaultHook } from '../src/core/persistence/fault-points.ts';
+import { localHostId } from '../src/core/persistence/identity.ts';
+import { DEADLINE_PASS_GRACE_MS } from '../src/core/persistence/sync-drain.ts';
import { retryHeld } from '../src/commands/sources-retry-held.ts';
import { VERSION } from '../src/version.ts';
import { conversationPage } from './helpers/conversation-page.ts';
@@ -42,6 +51,8 @@ afterAll(async () => {
await closePostgres?.();
});
+/** Hundreds of uncommitted paths outside the incoming diff, like the checkout that hit the hang; and a short budget for the stalled sync. */
+const PRESERVED = 300, STALL_BUDGET_MS = 20_000;
const each = (fn: (f: RefreshFixture) => Promise<void>) => withEnv(keyless, async () => { for (const engine of engines) await withRefreshFixture(engine, fn); });
/** Over the page ceiling, under the segmented ceiling, with a facts fence the streamed import refuses after reading the whole file. */
const fenced = `${conversationPage(PAGE_MAX_BYTES + 1024 * 1024, { seed: 8072 }).trimEnd()}\n\n## Facts\n\n<!--- gbrain:facts:begin -->\n| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |\n|---|---|---|---|---|---|---|---|---|---|\n| 1 | Example fact | fact | 1.0 | world | high | | | test | |\n<!--- gbrain:facts:end -->\n`;
@@ -105,3 +116,31 @@ test('#8072: a refresh past its sync budget stays syncing and the next plain ref
expect(await f.lastCommit(f.beta)).toBe(target);
expect((await f.refreshRows()).map(row => row.state)).toEqual(['completed']);
}), 300_000);
+
+test('#8072: a refresh whose member sync never settles ends refresh_sync_timeout within its budget, completes and releases the refresh lock', () => each(async f => {
+ for (let i = 0; i < PRESERVED; i++) writeFileSync(join(f.root, i % 2 ? 'alpha' : 'beta', `local-${i}.md`), page(`Local ${i}`, 'An uncommitted local note.'));
+ const target = f.push('alpha/two.md', page('Alpha two', 'An upstream observation.'));
+ await f.engine.setConfig('sources.refresh_sync_budget_ms', String(STALL_BUDGET_MS));
+ let stalls = 0, release!: () => void;
+ const gate = new Promise<void>(resolve => { release = resolve; });
+ installFaultHook(async point => { if (point === 'consumer:claiming') { stalls++; await gate; } });
+ try {
+ const started = performance.now();
+ const stalled = await refreshWorktree(f.engine, f.alpha);
+ expect(performance.now() - started).toBeLessThan(STALL_BUDGET_MS + DEADLINE_PASS_GRACE_MS + 30_000);
+ expect(stalls).toBeGreaterThan(0);
+ expect(stalled).toMatchObject({ status: 'sync_blocked', target_head: target });
+ expect(stalled.preserved_uncommitted).toHaveLength(PRESERVED);
+ expect(stalled.sync_blocked?.[0]).toMatchObject({ source_id: f.alpha, code: 'refresh_sync_timeout', resume: `gbrain sync --source ${f.alpha} --no-pull` });
+ expect(stalled.sync_blocked?.[0]?.message).toContain('is queued');
+ expect(f.head()).toBe(target);
+ expect(await f.lastCommit(f.alpha)).not.toBe(target);
+ expect((await f.refreshRows()).map(row => row.state)).toEqual(['completed']);
+ expect(await refreshLiveness(f.worktreeId, localHostId())).toBe('orphaned');
+ } finally { installFaultHook(undefined); release(); }
+
+ await f.engine.setConfig('sources.refresh_sync_budget_ms', '600000');
+ expect((await refreshWorktree(f.engine, f.alpha)).status).toBe('already_current');
+ expect(await f.lastCommit(f.alpha)).toBe(target);
+ expect(await f.lastCommit(f.beta)).toBe(target);
+}), 300_000);