Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(refresh): #8072 a deferred member completes the refresh; members share the sync budget fairly

garrytana27a67ba3bf4
diff --git a/docs/guides/live-sync.md b/docs/guides/live-sync.md
index b800d1d75f..940f3d39f4 100644
--- a/docs/guides/live-sync.md
+++ b/docs/guides/live-sync.md
@@ -653,16 +653,18 @@ vars — incident-time escape hatches, not everyday knobs.
`--wait-drain <seconds>` (default 60; `sources.refresh_drain_wait_ms`,
`GBRAIN_REFRESH_DRAIN_WAIT_MS`) and `--fetch-timeout-ms` (default 120000;
`sources.refresh_fetch_timeout_ms`, `GBRAIN_REFRESH_FETCH_TIMEOUT_MS`).
- The member syncs of one run share a deadline,
+ The member syncs of one run share a budget,
`sources.refresh_sync_budget_ms` (default 300000; `GBRAIN_REFRESH_SYNC_BUDGET_MS`),
- so a scheduled run never holds the refresh lock much longer than that:
- a run past it reports `syncing` (members `deferred`) and the next
- `gbrain sources refresh <source-id>` continues where it stopped. A member
- whose sync committed nothing in the whole budget (a write no consumer
- claims) is reported in `sync_blocked` as
- [`refresh_sync_timeout`](write-refusals.md#refresh_sync_timeout); the
- refresh then completes and releases its lock, and that source catches up
- on its next sync.
+ and each member gets an equal share of what is left, so a scheduled run
+ never holds the refresh lock much longer than that and one stalled source
+ cannot starve the others. A member that ran out of time is `deferred`:
+ the run reports `sync_deferred`, the refresh still completes (writes to
+ the checkout are admitted again) and the next
+ `gbrain sources refresh <source-id>` or sync continues from the member's
+ last commit. A member whose sync committed nothing in its share (a write no
+ consumer claims) is reported in `sync_blocked` as
+ [`refresh_sync_timeout`](write-refusals.md#refresh_sync_timeout) and
+ catches up on its next sync.
A refresh whose process died (a crash, a replaced container) does not
wedge the checkout: the owner finishes what it can on its own, and the next
`gbrain sources refresh <source-id>` on the owner host finishes the rest
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index a9b2eeeeaf..c862e75702 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -553,7 +553,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.
| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |
| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). A refresh whose holder is recorded on another host is never taken over either: writer status shows its `liveness` as `unknown`, and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
| <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |
-| <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within the refresh sync budget (`sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. |
+| <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within its share of the refresh sync budget (each member gets an equal share of what is left; `sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. |
| <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. |
| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |
| <a id="worktree_refreshing"></a>`worktree_refreshing` | `worktree_refreshing` | A write, or a managed sync started by a cycle or another command, reached a checkout while a refresh is draining or merging it. It was not journaled. `detail` is `retry_after_ms=1000`. Writes to database-only memory are not affected. | Retry the same request ID after about a second. `gbrain sources writer status <source>` shows the refresh. A cycle's sync is retried by the next cycle. |
diff --git a/llms-full.txt b/llms-full.txt
index 76893d968d..cf73d93140 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -7389,16 +7389,18 @@ vars — incident-time escape hatches, not everyday knobs.
`--wait-drain <seconds>` (default 60; `sources.refresh_drain_wait_ms`,
`GBRAIN_REFRESH_DRAIN_WAIT_MS`) and `--fetch-timeout-ms` (default 120000;
`sources.refresh_fetch_timeout_ms`, `GBRAIN_REFRESH_FETCH_TIMEOUT_MS`).
- The member syncs of one run share a deadline,
+ The member syncs of one run share a budget,
`sources.refresh_sync_budget_ms` (default 300000; `GBRAIN_REFRESH_SYNC_BUDGET_MS`),
- so a scheduled run never holds the refresh lock much longer than that:
- a run past it reports `syncing` (members `deferred`) and the next
- `gbrain sources refresh <source-id>` continues where it stopped. A member
- whose sync committed nothing in the whole budget (a write no consumer
- claims) is reported in `sync_blocked` as
- [`refresh_sync_timeout`](write-refusals.md#refresh_sync_timeout); the
- refresh then completes and releases its lock, and that source catches up
- on its next sync.
+ and each member gets an equal share of what is left, so a scheduled run
+ never holds the refresh lock much longer than that and one stalled source
+ cannot starve the others. A member that ran out of time is `deferred`:
+ the run reports `sync_deferred`, the refresh still completes (writes to
+ the checkout are admitted again) and the next
+ `gbrain sources refresh <source-id>` or sync continues from the member's
+ last commit. A member whose sync committed nothing in its share (a write no
+ consumer claims) is reported in `sync_blocked` as
+ [`refresh_sync_timeout`](write-refusals.md#refresh_sync_timeout) and
+ catches up on its next sync.
A refresh whose process died (a crash, a replaced container) does not
wedge the checkout: the owner finishes what it can on its own, and the next
`gbrain sources refresh <source-id>` on the owner host finishes the rest
diff --git a/src/commands/sources-refresh.ts b/src/commands/sources-refresh.ts
index d5288e3a46..b3d2f3f857 100644
--- a/src/commands/sources-refresh.ts
+++ b/src/commands/sources-refresh.ts
@@ -66,7 +66,7 @@ function describe(result: WorktreeRefreshResult): string {
export function printRefreshOutcome(outcome: { result: WorktreeRefreshResult } | { error: OperationError }, json: boolean): void {
if ('result' in outcome) {
console.log(json ? JSON.stringify(outcome.result) : describe(outcome.result));
- if (['sync_blocked', 'syncing'].includes(outcome.result.status)) setCliExitVerdict(1);
+ if (['sync_blocked', 'sync_deferred', 'syncing'].includes(outcome.result.status)) setCliExitVerdict(1);
return;
}
const error = outcome.error;
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 91c4396e69..bdc37346a0 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1373,7 +1373,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.514.0', id: 'company_db_only_keep_8014', text: 'A company-brain source now keeps a live page under a declared `storage.db_only` directory when its file is not in the approved manifest, and keeps every such page while its gbrain.yml storage section is invalid, as other managed sources do; it used to delete them. The decision reads gbrain.yml at the approved commit, never the working tree.' },
{ since: '0.60.514.0', id: 'redaction_marker_not_credential_7997', text: 'The transcript `labeled_credential` detector no longer reads an importer redaction marker (`login **<REDACTED:user-pattern>**`) as a typed user/password pair, so a correctly redacted page clears `gbrain transcripts audit-secrets`; a password written beside a marker is still found. The audit detector version moves to 3, so `gbrain doctor` asks for a fresh audit once.' },
{ since: 'TBD', id: 'oversize_hold_unread_8072', text: 'A Markdown file over 24 MiB that sync had to read whole to refuse (segmented storage on, held `file_too_large`, for example for a facts fence) is no longer read again by every sync: it stays held, unread, until it changes, gbrain is upgraded, or `gbrain sources retry-held <source>` schedules it. After changing a setting the hold message names, run `gbrain sources retry-held <source>`. A file held from its size alone still re-checks on every sync.' },
- { since: 'TBD', id: 'refresh_sync_budget_8072', text: 'The member syncs of one `gbrain sources refresh` run stop starting work after `sources.refresh_sync_budget_ms` (default 5 minutes; `GBRAIN_REFRESH_SYNC_BUDGET_MS`). A run past it reports `syncing` with the members it left `deferred`, and the next `gbrain sources refresh <source>` continues it; a member that committed nothing in the whole budget is reported `refresh_sync_timeout` in `sync_blocked` and the refresh completes, so a stalled sync no longer holds the refresh lock across scheduled runs.' },
+ { since: 'TBD', id: 'refresh_sync_budget_8072', text: 'The member syncs of one `gbrain sources refresh` run stop starting work after `sources.refresh_sync_budget_ms` (default 5 minutes; `GBRAIN_REFRESH_SYNC_BUDGET_MS`). Each member gets an equal share of what is left. A run past it reports `sync_deferred` with the members it left `deferred` and still completes the refresh, so writes to the checkout are admitted again and the next `gbrain sources refresh <source>` continues them; a member that committed nothing in its share is reported `refresh_sync_timeout` in `sync_blocked`. A stalled sync no longer holds the refresh lock across scheduled runs or starves the other members.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index cd85139659..93aab6ef82 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -64,7 +64,7 @@ export interface RefreshOptions {
export interface RefreshMemberSync { source_id: string; status: string; last_commit: string | null }
export interface RefreshSyncBlocked { source_id: string; code: string; message: string; resume: string }
export interface WorktreeRefreshResult {
- status: 'already_current' | 'completed' | 'sync_blocked' | 'dry_run' | 'syncing' | 'abandoned' | 'nothing_to_resume';
+ status: 'already_current' | 'completed' | 'sync_blocked' | 'sync_deferred' | 'dry_run' | 'syncing' | 'abandoned' | 'nothing_to_resume';
refresh_id: string | null; worktree_id: string; source_ids: string[]; old_head: string | null; target_head: string | null;
upstream_ref: string | null; preserved_uncommitted: string[]; incoming_files?: number;
synced: RefreshMemberSync[]; sync_blocked?: RefreshSyncBlocked[]; next?: string;
@@ -382,13 +382,13 @@ async function reachedTarget(git: Git, root: string, row: WorktreeRefreshRow, co
async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRow, root: string, budgetMs: number): Promise<{ synced: RefreshMemberSync[]; blocked: RefreshSyncBlocked[]; atTarget: boolean }> {
const sync = await boundedMemberSync(budgetMs);
const synced: RefreshMemberSync[] = [], blocked: RefreshSyncBlocked[] = [];
- for (const sourceId of row.source_ids) {
+ for (const [index, sourceId] of row.source_ids.entries()) {
let status = 'up_to_date';
for (let pass = 0; pass < MAX_MEMBER_SYNC_PASSES; pass++) {
if (await reachedTarget(git, root, row, await lastCommit(engine, sourceId))) break;
if (sync.spent()) { status = 'deferred'; break; }
let result: MemberSync;
- try { result = await sync.member(engine, sourceId); }
+ try { result = await sync.member(engine, sourceId, row.source_ids.length - index); }
catch (error) {
const failure = error instanceof OperationError ? error : opError('storage_error', error instanceof Error ? error.message : String(error),
`The managed sync of source ${sourceId} failed during refresh ${row.id}. Check its writer status, fix the cause, then resume with gbrain sync --source ${sourceId} --no-pull.`);
@@ -406,29 +406,32 @@ async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRo
}
let atTarget = true;
for (const member of synced) if (!await reachedTarget(git, root, row, member.last_commit)) { atTarget = false; break; }
- if (atTarget || blocked.length) await engine.transaction(tx => transition(tx, row, ['syncing'], 'completed',
- { synced, ...(blocked.length ? { sync_blocked: blocked } : {}) }));
+ // #8072: a deferred member completes the refresh too: the checkout is at the target and the member's next sync resumes from
+ // its last_commit, while a row left syncing would refuse every other write to the worktree until then.
+ const deferred = synced.filter(member => member.status === 'deferred').map(member => member.source_id);
+ if (atTarget || blocked.length || deferred.length) await engine.transaction(tx => transition(tx, row, ['syncing'], 'completed',
+ { synced, ...(blocked.length ? { sync_blocked: blocked } : {}), ...(deferred.length ? { sync_deferred: deferred } : {}) }));
return { synced, blocked, atTarget };
}
type MemberSync = { status: string; blocked?: RefreshSyncBlocked };
/**
- * #8072: the member syncs of one refresh share one deadline and run through the governed drain (sync-drain.ts) under the
- * source's drain lease, as `gbrain sync` and a conversion's strict pass do, so no pass outlives the deadline by more than the
- * drain's grace. A drain the deadline stopped after the member moved reports `deferred` (the refresh stays `syncing` and the
- * next plain refresh or `--resume` continues it); one that moved nothing in the whole budget is blocked `refresh_sync_timeout`,
- * naming the write it waited on, as is a drain its own rules blocked (`drain_stalled` and the rest), so the refresh completes
- * with that member behind its target and releases the refresh lock instead of re-entering bare passes under it.
+ * #8072: the member syncs of one refresh run through the governed drain (sync-drain.ts) under each source's drain lease, as
+ * `gbrain sync` and a conversion's strict pass do, within one budget: each member gets an equal share of what is left, so no
+ * pass outlives its share by more than the drain's grace and a member that never settles cannot starve the ones after it.
+ * A member the deadline stopped after it moved is `deferred`; one that committed nothing in its share is blocked
+ * `refresh_sync_timeout`, naming the write it waited on, as is a drain its own rules blocked (`drain_stalled` and the
+ * rest). Either way the refresh completes and releases its lock, and the member's next sync resumes from its last_commit.
*/
async function boundedMemberSync(budgetMs: number) {
const [{ drainManagedSync, engineStallProbe }, { headBlockerText }, { withDrainLease, leaseHeldResult, readDrainHolder }, { withSyncTail }, { LockUnavailableError, syncLockId }] = await Promise.all([
import('./sync-drain.ts'), import('./head-blocker.ts'), import('./sync-lease.ts'), import('./sync-tail.ts'), import('../db-lock.ts')]);
const deadlineAtMs = Date.now() + budgetMs;
- const drain = async (engine: BrainEngine, sourceId: string): Promise<SyncResult> => {
- const opts = { sourceId, noPull: true, deadlineAtMs };
+ const drain = async (engine: BrainEngine, sourceId: string, memberDeadlineAtMs: number): Promise<SyncResult> => {
+ const opts = { sourceId, noPull: true, deadlineAtMs: memberDeadlineAtMs };
try {
const drained = await withDrainLease(engine, sourceId, { holder: { kind: 'refresh', id: null }, remote: false },
signal => drainManagedSync(engine, signal ? { ...opts, signal } : opts, false));
- return await withSyncTail(engine, sourceId, drained, { deadlineAtMs });
+ return await withSyncTail(engine, sourceId, drained, { deadlineAtMs: memberDeadlineAtMs });
} catch (error) {
if (!(error instanceof LockUnavailableError) || error.lockId !== syncLockId(sourceId)) throw error;
return leaseHeldResult(sourceId, await readDrainHolder(engine, sourceId).catch(() => null));
@@ -436,10 +439,12 @@ async function boundedMemberSync(budgetMs: number) {
};
return {
spent: () => Date.now() >= deadlineAtMs,
- member: async (engine: BrainEngine, sourceId: string): Promise<MemberSync> => {
+ /** `left`: the members not yet synced, this one included; each gets an equal share of what remains, so one that never settles cannot starve the rest. */
+ member: async (engine: BrainEngine, sourceId: string, left: number): Promise<MemberSync> => {
const resume = `gbrain sync --source ${sourceId} --no-pull`;
+ const shareMs = Math.max(0, Math.floor((deadlineAtMs - Date.now()) / Math.max(1, left)));
const before = await lastCommit(engine, sourceId);
- const result = await drain(engine, sourceId);
+ const result = await drain(engine, sourceId, Date.now() + shareMs);
if (result.status === 'blocked_by_failures') return { status: result.status, blocked: { source_id: sourceId, resume,
code: result.managedWrite?.write_error ?? result.failureCodes?.[0]?.code ?? 'sync_blocked', message: result.managedWrite?.message ?? 'The managed sync is blocked by a failed write.' } };
const report = result.drain;
@@ -453,7 +458,7 @@ async function boundedMemberSync(budgetMs: number) {
const head = await engineStallProbe(engine, sourceId).head?.().catch(() => null) ?? null;
const waiting = head ? `; its oldest unfinished write request ${head.write_request_id ?? head.request_id ?? 'unknown'} is ${head.head_state}${head.blocked_by ? `, blocked by ${headBlockerText(head.blocked_by)}` : ''}${head.consumer ? `, consumer ${head.consumer}` : ''}` : '';
return { status: 'blocked', blocked: { source_id: sourceId, resume, code: 'refresh_sync_timeout',
- message: `The managed sync committed nothing within the refresh's sync budget of ${budgetMs} ms${waiting}. The refresh completed with this source behind its target; its next sync continues it.` } };
+ message: `The managed sync committed nothing within its ${shareMs} ms share of the refresh's sync budget${waiting}. The refresh completed with this source behind its target; its next sync continues it.` } };
},
};
}
@@ -463,10 +468,12 @@ async function lastCommit(engine: BrainEngine, sourceId: string): Promise<string
}
function finished(row: WorktreeRefreshRow, sync: { synced: RefreshMemberSync[]; blocked: RefreshSyncBlocked[]; atTarget: boolean }, preserved = row.preserved_uncommitted): WorktreeRefreshResult {
- const status = sync.blocked.length ? 'sync_blocked' : sync.atTarget ? 'completed' : 'syncing';
+ const deferred = sync.synced.some(member => member.status === 'deferred');
+ const status = sync.blocked.length ? 'sync_blocked' : sync.atTarget ? 'completed' : deferred ? 'sync_deferred' : 'syncing';
return { status, refresh_id: row.id, worktree_id: row.worktree_id, source_ids: row.source_ids, old_head: row.old_head, target_head: row.target_head,
upstream_ref: row.upstream_ref, preserved_uncommitted: preserved, synced: sync.synced,
...(sync.blocked.length ? { sync_blocked: sync.blocked, next: sync.blocked.map(item => item.resume).join(' ; ') } : {}),
+ ...(status === 'sync_deferred' ? { next: `gbrain sources refresh ${row.source_ids[0]}` } : {}),
...(status === 'syncing' ? { next: `gbrain sources refresh ${row.source_ids[0]} --resume` } : {}) };
}
@@ -495,13 +502,13 @@ async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshO
if (plan.oldHead === plan.targetHead) {
const synced: RefreshMemberSync[] = [], blocked: RefreshSyncBlocked[] = [];
const sync = await boundedMemberSync(syncBudgetMs);
- for (const member of plan.members) {
+ for (const [index, member] of plan.members.entries()) {
await recordUpstreamObservation(engine, member);
const commit = await lastCommit(engine, member);
if (commit === plan.oldHead) { synced.push({ source_id: member, status: 'up_to_date', last_commit: plan.oldHead }); continue; }
if (sync.spent()) { synced.push({ source_id: member, status: 'deferred', last_commit: commit }); continue; }
try {
- const result = await sync.member(engine, member);
+ const result = await sync.member(engine, member, plan.members.length - index);
if (result.blocked) blocked.push(result.blocked);
synced.push({ source_id: member, status: result.status, last_commit: await lastCommit(engine, member) });
} catch (error) {
@@ -509,7 +516,7 @@ async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshO
}
}
const deferred = synced.some(member => member.status === 'deferred');
- return { ...base, status: blocked.length ? 'sync_blocked' : 'already_current', synced, ...(blocked.length ? { sync_blocked: blocked, next: blocked.map(b => b.resume).join(' ; ') }
+ return { ...base, status: blocked.length ? 'sync_blocked' : deferred ? 'sync_deferred' : 'already_current', synced, ...(blocked.length ? { sync_blocked: blocked, next: blocked.map(b => b.resume).join(' ; ') }
: deferred ? { next: `gbrain sources refresh ${sourceId}` } : {}) };
}
const refreshLock = await acquireNativeLock(refreshLockPath(plan.binding.worktree_id), { timeoutMs: 1000 });
diff --git a/test/oversize-hold-rescreen-8072.test.ts b/test/oversize-hold-rescreen-8072.test.ts
index 4473e3e1ce..58ec74bd86 100644
--- a/test/oversize-hold-rescreen-8072.test.ts
+++ b/test/oversize-hold-rescreen-8072.test.ts
@@ -5,13 +5,16 @@
* and stays held, unread, on later ticks while its bytes are unchanged: no new write request for it, no pass-2
* block read, the hold's attempt count unchanged. `gbrain sources retry-held` (and a gbrain upgrade) still
* re-screens it; a hold decided from the size alone keeps re-screening every sync (one lstat).
- * The member syncs of one refresh are bounded by `sources.refresh_sync_budget_ms`: past it the refresh stays
- * `syncing` and the next plain refresh finishes it. A member whose sync commits nothing in the whole budget (its write
- * is never claimed: the `consumer:claiming` fault point holds the consumer) is reported `refresh_sync_timeout`, the
- * refresh completes and its lock is free, with hundreds of preserved uncommitted paths in the checkout.
+ * The member syncs of one refresh are bounded by `sources.refresh_sync_budget_ms`, each member with an equal share of
+ * what is left: past it the refresh reports `sync_deferred` and still completes, so writes are admitted while the
+ * members catch up. A member whose sync commits nothing in its share (its write is never claimed: the
+ * `consumer:claiming` fault point holds the consumer; or its pass hangs at `sync:after_freeze`) is reported
+ * `refresh_sync_timeout` and the next member still syncs in the same run; the refresh completes and its lock is
+ * free, with hundreds of preserved uncommitted paths in the checkout.
* 2. Fails when: every tick re-hashes and re-streams the unchanged held file (the hold re-screened on every sync), or a
* refresh tick keeps syncing past its budget under the refresh lock (on master the stalled refresh re-entered bare
- * passes for about 300 s and did not return within the test's timeout).
+ * passes for about 300 s and did not return within the test's timeout), a deferred member leaves the row `syncing`
+ * (every other write refused `worktree_refreshing`), or a stalled first member spends the whole budget.
* Counts, not wall-clock time: write requests, `streamedImportWork` bytes and the hold's attempts; the one time
* assertion is the stalled refresh's upper bound (budget + the drain's deadline grace + 30 s of slack).
* Repo-less brains have no managed sync or refresh (both need Git), so nothing here runs there.
@@ -24,6 +27,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { PAGE_MAX_BYTES } from '../src/core/page-size-policy.ts';
import { streamedImportWork } from '../src/core/page-body/streamed-import.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { performManagedSync } from '../src/core/persistence/sync-run.ts';
import { holdRescreenDue, readGitSourceHolds, type GitHoldRecord } from '../src/core/persistence/sync-holds.ts';
import { refreshLiveness, refreshWorktree } from '../src/core/persistence/worktree-refresh.ts';
import { installFaultHook } from '../src/core/persistence/fault-points.ts';
@@ -52,7 +56,7 @@ afterAll(async () => {
});
/** Hundreds of uncommitted paths outside the incoming diff, like the checkout that hit the hang; and a short budget for the stalled sync. */
-const PRESERVED = 300, STALL_BUDGET_MS = 20_000;
+const PRESERVED = 300, STALL_BUDGET_MS = 20_000, FAIR_BUDGET_MS = 20_000;
const each = (fn: (f: RefreshFixture) => Promise<void>) => withEnv(keyless, async () => { for (const engine of engines) await withRefreshFixture(engine, fn); });
/** Over the page ceiling, under the segmented ceiling, with a facts fence the streamed import refuses after reading the whole file. */
const fenced = `${conversationPage(PAGE_MAX_BYTES + 1024 * 1024, { seed: 8072 }).trimEnd()}\n\n## Facts\n\n<!--- gbrain:facts:begin -->\n| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |\n|---|---|---|---|---|---|---|---|---|---|\n| 1 | Example fact | fact | 1.0 | world | high | | | test | |\n<!--- gbrain:facts:end -->\n`;
@@ -100,21 +104,42 @@ test('#8072: a streamed file_too_large hold re-screens on retry or another gbrai
expect(holdRescreenDue(hold({}), false)).toBe(true);
});
-test('#8072: a refresh past its sync budget stays syncing and the next plain refresh finishes it', () => each(async f => {
+test('#8072: a refresh past its sync budget completes with its members deferred, admits writes, and reports a deferred member honestly', () => each(async f => {
const target = f.push('alpha/two.md', page('Alpha two', 'An upstream observation.'));
await f.engine.setConfig('sources.refresh_sync_budget_ms', '0');
const deferred = await refreshWorktree(f.engine, f.alpha);
- expect(deferred).toMatchObject({ status: 'syncing', target_head: target, next: `gbrain sources refresh ${f.alpha} --resume` });
+ expect(deferred).toMatchObject({ status: 'sync_deferred', target_head: target, next: `gbrain sources refresh ${f.alpha}` });
+ expect(deferred.synced.map(member => member.status)).toEqual(['deferred', 'deferred']);
expect(f.head()).toBe(target);
expect(await f.lastCommit(f.alpha)).not.toBe(target);
- expect((await f.refreshRows()).map(row => row.state)).toEqual(['syncing']);
+ expect((await f.refreshRows()).map(row => row.state)).toEqual(['completed']);
+
+ const again = await refreshWorktree(f.engine, f.alpha);
+ expect(again).toMatchObject({ status: 'sync_deferred', next: `gbrain sources refresh ${f.alpha}` });
+ expect(again.synced.map(member => member.status)).toEqual(['deferred', 'deferred']);
+ expect((await f.put(f.alpha, 'notes/during-catch-up', page('During', 'A write while the member catches up.'))).state ?? 'committed').toBe('committed');
await f.engine.setConfig('sources.refresh_sync_budget_ms', '600000');
- const finished = await refreshWorktree(f.engine, f.alpha);
- expect(finished.recovered).toMatchObject({ refresh_id: deferred.refresh_id, status: 'completed' });
- expect(await f.lastCommit(f.alpha)).toBe(target);
- expect(await f.lastCommit(f.beta)).toBe(target);
- expect((await f.refreshRows()).map(row => row.state)).toEqual(['completed']);
+ for (const source of [f.alpha, f.beta]) await performManagedSync(f.engine, { sourceId: source, noPull: true, noEmbed: true });
+ expect(await f.lastCommit(f.alpha)).toBe(f.head());
+ expect(await f.lastCommit(f.beta)).toBe(f.head());
+}), 300_000);
+
+test('#8072: a member whose sync never settles does not starve the next member of the same refresh', () => each(async f => {
+ const target = f.push('alpha/two.md', page('Alpha two', 'An upstream observation.'));
+ await f.engine.setConfig('sources.refresh_sync_budget_ms', String(FAIR_BUDGET_MS));
+ let stalls = 0, release!: () => void;
+ const gate = new Promise<void>(resolve => { release = resolve; });
+ installFaultHook(async (point, detail) => { if (point === 'sync:after_freeze' && detail.sourceId === f.alpha) { stalls++; await gate; } });
+ try {
+ const result = await refreshWorktree(f.engine, f.alpha);
+ expect(stalls).toBeGreaterThan(0);
+ expect(result.sync_blocked?.map(entry => [entry.source_id, entry.code])).toEqual([[f.alpha, 'refresh_sync_timeout']]);
+ expect(result.synced.find(member => member.source_id === f.beta)).toMatchObject({ last_commit: target });
+ expect(await f.lastCommit(f.beta)).toBe(target);
+ expect(await f.lastCommit(f.alpha)).not.toBe(target);
+ expect((await f.refreshRows()).map(row => row.state)).toEqual(['completed']);
+ } finally { installFaultHook(undefined); release(); }
}), 300_000);
test('#8072: a refresh whose member sync never settles ends refresh_sync_timeout within its budget, completes and releases the refresh lock', () => each(async f => {