Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
test(persistence): crash robot reruns a fault once when a race leaves the run one slot short (#8078)
garrytan6c646df6ec76
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2d03994232..3afb6da655 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,11 +14,15 @@ identifiers and attribution are available in the pre-removal Git revision**`gbrain doctor` now ends on its `--budget-ms` deadline when `git` hangs in an OpenClaw workspace.** The bootstrap entry checks push health by running `git status`, `git branch` and `git rev-list` on the install receipt's workspace, and on a workspace whose last push failed. Those calls ran synchronously with a 10-second limit each. While one hung, nothing else in the process could run, so the run overran its budget and every later check, the cheap incident checks included, started with no time left. They now run asynchronously, like the home-dir probes since #8979. Each call is killed with SIGKILL, together with its process group, at 10 seconds or at the entry's deadline, whichever comes first, and a call that fails or is cut short reads as unverified, as before. Every local probe's output is also capped at 1 MiB again: a probe that prints more than that is killed and reads as absent, which is what `execFile`'s buffer limit did before #8979.+**The Postgres crash robot no longer fails a full run when a race makes one fewer slot than its count run saw.** The robot first runs each schedule to count the seams it reaches, then crashes one run at a counted slot. In `sync_and_connector_race_direct_write`, the connector republish races a direct write, and the race legally settles two ways: one makes 5 publications, the other 6, with the git effects, preparations and commits following. A crash planned at the 6th publication was never reached by a run that took the 5-publication outcome, and the job failed with `fault_not_reached` (#8078). A run that misses its fault now reports how many slots of that seam it made. If it was exactly one short of the count run, it runs once more at its last slot, and the manifest records `retargeted` on the run and lists it under `faults_retargeted`. A run that made no slot, fell two or more short, or misses again still fails as `fault_not_reached`.+### Itemized changes- `src/commands/doctor/local-probe.ts`: `localProbe` moved here from `checks/local-audits.ts`. It takes `ownTimeoutMs` (default 5 s) and `windowsShell` (git runs without cmd.exe on Windows), and it kills a probe and returns null once its stdout passes `LOCAL_PROBE_MAX_OUTPUT_BYTES` (1 MiB).- `src/commands/doctor/bootstrap-checks.ts`: `bootstrapDoctorChecks(engine, entryDeadlineAt)` runs every git call through `localProbe` with a 10 s bound. `commitsAheadOfOrigin`, `treeMatchesOrigin` and `recordedPushFailureCheck` are async.- Tests: `test/doctor-bootstrap-git-deadline-8990.test.ts`. A `git` stand-in that ignores SIGTERM and waits on a 6 s child is killed within a 1000 ms budget, on both the workspace probe and the recorded-failure probe, and neither it nor its child survives (each fails on the previous code, at 18 s and 12 s). A probe that floods stdout is killed within 1.5 s and reads as absent, one that prints 2 MiB and exits reads as absent, and one under the cap returns its whole output (the first two fail without the cap).+- `scripts/persistence/crash-robot.ts`: the run role's fault hook reports `reached`, the slots of its seam it made. `scripts/persistence/robot-driver.ts`: a planned crash carries `counted` (the count run's slots), and `retargetable` reruns a run once when `counted - 1 <= reached < nth`.+- Tests: `test/persistence-crash-robot.test.ts`. A run one slot short crashes at its last slot and passes; a run two short, a run short again after the retarget, a run with no slot and a run with no counted plan still fail. Replaying the failed #9071 manifest on Postgres failed 3 of 3 before the fix and passed 3 of 3 after it.## [0.60.624.0] - 2026-11-09diff --git a/scripts/persistence/README.md b/scripts/persistence/README.mdindex 628942232e..443f093018 100644--- a/scripts/persistence/README.md+++ b/scripts/persistence/README.md@@ -146,7 +146,9 @@ key lists every run (schedule, seed, seam and occurrence or process fault),`fault_hook_errors` (a fault hook that threw instead of freezing the worker)and no `faults_not_reached` (a worker that finished without reaching itsrequested seam, outside the per-scenario `UNREACHED_FAULT_ALLOWLIST` in-`robot-driver.ts`); either kind of run also fails validation with a nonzero+`robot-driver.ts`; a run exactly one slot short of its count run, which a+concurrent group's other legal outcome makes, runs once more at its last slot+and is listed in `faults_retargeted`); either kind of run also fails validation with a nonzeroexit and is listed in `failing_runs` for `--replay`.A failing run prints two commands. `--replay=<manifest>` re-runs exactly thediff --git a/scripts/persistence/crash-robot.ts b/scripts/persistence/crash-robot.tsindex a110eaccb9..9809bb521d 100644--- a/scripts/persistence/crash-robot.ts+++ b/scripts/persistence/crash-robot.ts@@ -157,6 +157,8 @@ interface RobotState {}export interface RobotOutcome extends Omit<ScheduleResult, 'observations'> {crashed: boolean; fault?: RobotConfig['fault']; process?: ProcessFault; counts?: Record<string, number>; inFlight?: string[];+ /** A run that finished without reaching its fault: how many slots of the fault's seam it made (#8078). */+ reached?: number;/** What each caller observed, kept only for a run with violations. */trace?: string[];/** Known findings outside the safety classes, filed in TODOS.md: reported, never failing the gate. */@@ -283,17 +285,17 @@ export function fixedChunks(runner: typeof hardenedGitStream = hardenedGitStream* `fault_hook_error` event and the process freezes anyway; the driver counts* such a run as a gate failure.*/-export function robotFaultHook(fault: NonNullable<RobotConfig['fault']>): (point: FaultPoint, detail: FaultDetail) => void {+export function robotFaultHook(fault: NonNullable<RobotConfig['fault']>): ((point: FaultPoint, detail: FaultDetail) => void) & { reached(): number } {let seen = 0;const counts = seamCounter();- return (point, detail) => {+ return Object.assign((point: FaultPoint, detail: FaultDetail) => {const advanced = counts(point, detail);if (point !== fault.point || (fault.operation && detail.operation !== fault.operation) || !advanced) return;seen += advanced;if (seen < fault.nth) return;try { freeze({ event: 'fault', point, nth: seen, detail }); }catch (error) { freeze({ event: 'fault_hook_error', point, nth: seen, message: error instanceof Error ? error.message : String(error) }); }- };+ }, { reached: () => seen });}function finalize(config: RobotConfig, model: ReferenceModel, extra: Partial<RobotOutcome>): RobotOutcome {@@ -462,8 +464,9 @@ export async function robotWorker(role: 'count' | 'run' | 'recover', config: Rob__setSyncGitStreamForTests(fixedChunks());const counts: Record<string, number> = {};const counted = seamCounter();+ const faultHook = role === 'run' && config.fault ? robotFaultHook(config.fault) : undefined;const hook = role === 'count' ? (point: FaultPoint, detail: FaultDetail) => { const advanced = counted(point, detail); if (advanced) counts[point] = (counts[point] ?? 0) + advanced; }- : role === 'run' && config.fault ? robotFaultHook(config.fault) : undefined;+ : faultHook;if (hook) {world.claimPasses = 0;installFaultHook((point, detail) => { if (point === 'consumer:claiming') world.claimPasses!++; hook(point, detail); });@@ -475,7 +478,7 @@ export async function robotWorker(role: 'count' | 'run' | 'recover', config: Robif (config.process) { await processFault(config, world, model, checkouts); return finalize(config, model, { process: config.process }); }await runSteps(world, model, config.schedule, 0, (i, batch) => { next = i; inFlight = batch.map(d => d.id); writeState(config, world, model, checkouts, next, inFlight); });await finish(world, model, config.schedule.label);- return finalize(config, model, { fault: config.fault });+ return finalize(config, model, { fault: config.fault, ...(faultHook ? { reached: faultHook.reached() } : {}) });}if (role === 'count') {await runSteps(world, model, config.schedule);diff --git a/scripts/persistence/robot-driver.ts b/scripts/persistence/robot-driver.tsindex 4d1f9170d6..ac7f0f8d91 100644--- a/scripts/persistence/robot-driver.ts+++ b/scripts/persistence/robot-driver.ts@@ -65,6 +65,12 @@ export interface RobotRun {deferred?: RobotOutcome['deferred'];/** The worker finished without reaching its requested fault (#6584): nothing was crashed or recovered. */fault_not_reached?: boolean;+ /** Slots of the fault's seam the schedule's count run made, when the fault was planned from one. */+ counted?: number;+ /** Slots of the fault's seam made by a run that missed its fault. */+ reached?: number;+ /** The fault this run was planned with, and how many slots the run that missed it made (see `retargetable`). */+ retargeted?: { nth: number; reached: number };/** The worker's fault hook threw at the requested seam (`fault_hook_error`); the run proves nothing. */hook_error?: string;}@@ -78,6 +84,18 @@ export const UNREACHED_FAULT_ALLOWLIST: readonly { scenario: string; point: Faulexport function unreachedAllowed(run: Pick<RobotRun, 'schedule' | 'process' | 'fault'>): boolean {return UNREACHED_FAULT_ALLOWLIST.some(entry => entry.scenario === (run.process ?? run.schedule) && entry.point === run.fault?.point);}+/**+ * #8078: a schedule whose concurrent group settles one of two legal ways makes one slot more or fewer at some seams+ * (sync_and_connector_race_direct_write: the connector republish racing a direct write publishes 5 or 6 times, and+ * its git effects, preparations and commits follow), so a fault the count run planned at its last slot can be missed+ * by a run that took the other outcome. Such a run, short by exactly one slot of the count run, runs once more at the+ * last slot it did make. A run that made no slot, fell more than one short, or missed again fails as before.+ */+export function retargetable(run: RobotRun): boolean {+ const reached = run.reached ?? 0;+ return run.fault_not_reached === true && !run.retargeted && run.counted !== undefined && run.fault !== undefined+ && reached > 0 && reached >= run.counted - 1 && reached < run.fault.nth;+}/** A run fails the gate when it found violations, its hook threw, or it never reached a fault it was not allowed to miss. */export function runFailed(run: RobotRun): boolean {return run.violations.length > 0 || run.hook_error !== undefined || (run.fault_not_reached === true && !unreachedAllowed(run));@@ -146,14 +164,14 @@ export async function runRobotPhase(o: RobotOptions) {const index = o.databases.indexOf(database); if (index >= 0) o.databases.splice(index, 1);}const environment: Record<string, string> = o.pooledUrl ? { GBRAIN_PREPARE: 'false' } : {};- async function execute(schedule: Schedule, role: 'count' | 'run', extra: Partial<RobotConfig> = {}): Promise<RobotRun & { counts?: Record<string, number> }> {+ async function execute(schedule: Schedule, role: 'count' | 'run', extra: Partial<RobotConfig> = {}, planned: Pick<RobotRun, 'counted' | 'retargeted'> = {}): Promise<RobotRun & { counts?: Record<string, number> }> {const at = performance.now();const { database, ...place } = await fresh(`${schedule.label}-${role}`);const config: RobotConfig = { kind: o.engine, ...place, schedule, worktrees: o.worktrees ?? 2, statePath: join(place.root, 'state.json'), ...extra };const path = join(place.root, 'robot.json'); writeFileSync(path, JSON.stringify(config), { mode: 0o600 });const home = join(o.home, basename(place.root)); mkdirSync(home, { recursive: true });const base = { schedule: schedule.label, seed: schedule.seed, length: schedule.ops.length, ...(extra.fault ? { fault: extra.fault } : {}),- ...(extra.process ? { process: extra.process } : {}), ...(o.keep?.has(schedule.label) ? { ops: schedule.ops.map(d => d.id) } : {}) };+ ...(extra.process ? { process: extra.process } : {}), ...(o.keep?.has(schedule.label) ? { ops: schedule.ops.map(d => d.id) } : {}), ...planned };const child = o.spawn(path, home, 'robot', [role], environment); o.track(child);const outcome = async (): Promise<RobotRun & { counts?: Record<string, number> }> => {if (role === 'run' && extra.fault) {@@ -165,7 +183,8 @@ export async function runRobotPhase(o: RobotOptions) {}if ('done' in reached) {const result = reached.done.result as RobotOutcome;- return { ...base, crashed: false, fault_not_reached: true, violations: result.violations, deferred: result.deferred, trace: result.trace, duration_ms: performance.now() - at };+ return { ...base, crashed: false, fault_not_reached: true, ...(result.reached !== undefined ? { reached: result.reached } : {}),+ violations: result.violations, deferred: result.deferred, trace: result.trace, duration_ms: performance.now() - at };}await child.kill();const recovered = o.spawn(path, home, 'robot', ['recover'], environment); o.track(recovered);@@ -176,6 +195,11 @@ export async function runRobotPhase(o: RobotOptions) {return { ...base, crashed: false, violations: result.violations, deferred: result.deferred, trace: result.trace, counts: result.counts, duration_ms: performance.now() - at };};const run = await outcome();+ if (retargetable(run)) {+ await release(database);+ log(`${run.schedule} ${run.fault!.point}#${run.fault!.nth}: the run made ${run.reached} of the count run's ${run.counted} slots; crashing again at #${run.reached}`);+ return execute(schedule, role, { ...extra, fault: { ...extra.fault!, nth: run.reached! } }, { counted: run.counted, retargeted: { nth: run.fault!.nth, reached: run.reached! } });+ }if (!runFailed(run)) await release(database);return run;}@@ -187,7 +211,8 @@ export async function runRobotPhase(o: RobotOptions) {log(`${run.schedule} ${what}: ${verdict} (${Math.round(run.duration_ms)} ms)`);};const record = (run: RobotRun) => { runs.push(run); say(run); };- const go = (schedule: Schedule, role: 'count' | 'run', extra: Partial<RobotConfig> = {}) => execute(schedule, role, extra).then(run => { say(run); return run; });+ const go = (schedule: Schedule, role: 'count' | 'run', extra: Partial<RobotConfig> = {}, planned: Pick<RobotRun, 'counted'> = {}) =>+ execute(schedule, role, extra, planned).then(run => { say(run); return run; });const width = o.concurrency ?? ROBOT_CONCURRENCY;/** `each` over `items`, `width` at a time, each result at its item's index. The first failure stops every lane from taking another item. */async function pooled<T, R>(items: readonly T[], each: (item: T) => Promise<R>): Promise<R[]> {@@ -207,7 +232,8 @@ export async function runRobotPhase(o: RobotOptions) {if (o.replay) {for (const entry of o.replay) {const schedule = restrict(scheduleFor(entry.schedule, entry.seed, entry.length), entry.ops ? new Set(entry.ops) : o.keep?.get(entry.schedule));- record(await execute(schedule, entry.fault || entry.process ? 'run' : 'count', { ...(entry.fault ? { fault: entry.fault } : {}), ...(entry.process ? { process: entry.process } : {}) }));+ record(await execute(schedule, entry.fault || entry.process ? 'run' : 'count', { ...(entry.fault ? { fault: entry.fault } : {}), ...(entry.process ? { process: entry.process } : {}) },+ entry.counted !== undefined ? { counted: entry.counted } : {}));}return summarize(runs, started, o);}@@ -239,9 +265,9 @@ export async function runRobotPhase(o: RobotOptions) {plan.sort((x, y) => x.point.localeCompare(y.point) || turn(x) - turn(y));const firsts = plan.filter((x, i) => plan.findIndex(y => y.point === x.point) === i);const ordered = [...firsts, ...plan.filter(x => !firsts.includes(x))];- type Job = { schedule: Schedule; extra: Partial<RobotConfig> };+ type Job = { schedule: Schedule; extra: Partial<RobotConfig>; counted?: number };const crashes: Job[] = (everySeam ? ordered : firsts)- .map(item => ({ schedule: item.schedule, extra: { fault: { point: item.point, nth: 1 + Math.floor(rand() * item.count) } } }));+ .map(item => ({ schedule: item.schedule, extra: { fault: { point: item.point, nth: 1 + Math.floor(rand() * item.count) } }, counted: item.count }));const skippedFaults = new Set<ProcessFault>();const processes: Job[] = [];for (const fault of PROCESS_FAULTS) {@@ -258,12 +284,12 @@ export async function runRobotPhase(o: RobotOptions) {const points = (Object.entries(seededCounts[index]!.counts ?? {}) as [FaultPoint, number][]).filter(([point]) => !(o.engine === 'postgres' && leaseBoundSeam(point)));if (!points.length) return undefined;const [point, count] = points[Math.floor(rand() * points.length)]!;- return { schedule, extra: { fault: { point, nth: 1 + Math.floor(rand() * count) } } };+ return { schedule, extra: { fault: { point, nth: 1 + Math.floor(rand() * count) } }, counted: count };});// The process faults start first: a hung git child and a dead owner's claim lease mostly wait, so they// overlap the crash runs instead of adding their wait to the wall clock.const jobs = [...processes, ...crashes, ...seededCrashes.filter((job): job is Job => job !== undefined)];- const finished = await pooled(jobs, job => go(job.schedule, 'run', job.extra));+ const finished = await pooled(jobs, job => go(job.schedule, 'run', job.extra, job.counted !== undefined ? { counted: job.counted } : {}));const result = (job: Job) => finished[jobs.indexOf(job)]!;runs.push(...fixedCounts, ...crashes.map(result), ...processes.map(result));seeded.forEach((_, index) => { runs.push(seededCounts[index]!); const job = seededCrashes[index]; if (job) runs.push(result(job)); });@@ -283,6 +309,7 @@ function summarize(runs: RobotRun[], started: number, o: RobotOptions) {fault_hook_errors: runs.filter(r => r.hook_error !== undefined),faults_not_reached: runs.filter(r => r.fault_not_reached && !unreachedAllowed(r)),faults_not_reached_allowed: runs.filter(r => r.fault_not_reached && unreachedAllowed(r)),+ faults_retargeted: runs.filter(r => r.retargeted),sequences_x_crash_points: crashRuns.length, distinct_crash_points: [...points].sort(),process_faults: runs.filter(r => r.process).map(r => ({ fault: r.process, violations: r.violations.length })),violations, deferred: runs.flatMap(r => (r.deferred ?? []).map(v => ({ ...v, schedule: r.schedule, seed: r.seed, fault: r.fault }))),diff --git a/test/persistence-crash-robot.test.ts b/test/persistence-crash-robot.test.tsindex 25c9c75ce4..1982c2ae90 100644--- a/test/persistence-crash-robot.test.ts+++ b/test/persistence-crash-robot.test.ts@@ -24,7 +24,7 @@ import { ROBOT_CONCURRENCY, ROBOT_RANDOM_SEEDS, ROBOT_ROTATION_SEEDS, ROBOT_TOPOimport { robotCeilingUse, robotFailure, robotFullGate, robotOverBudget, runValidation, spawnWorker } from '../scripts/persistence/validate.ts';import { faultPoint, installFaultHook, type FaultDetail, type FaultPoint } from '../src/core/persistence/fault-points.ts';import { shrinkRun } from '../scripts/persistence/shrink.ts';-import { finish, fixedChunks, LEVEL_CHUNK_BYTES, runSchedule, runSteps, seamCounter } from '../scripts/persistence/crash-robot.ts';+import { finish, fixedChunks, LEVEL_CHUNK_BYTES, runSchedule, robotFaultHook, runSteps, seamCounter } from '../scripts/persistence/crash-robot.ts';import type { hardenedGitStream } from '../src/core/hardened-git.ts';import { installLockOrderTrace, lockOrderReport } from '../scripts/persistence/lock-order.ts';import { prepareTopology } from '../scripts/persistence/history-fixture.ts';@@ -358,11 +358,11 @@ describe('requested faults are reported, never silently skipped (#6584)', () =>}) as unknown as typeof spawnWorker;return { spawn, spawned };}- async function phase(spawn: typeof spawnWorker, fault: { point: FaultPoint; nth: number }) {+ async function phase(spawn: typeof spawnWorker, fault: { point: FaultPoint; nth: number }, planned: Pick<RobotRun, 'counted'> = {}) {const scratch = mkdtempSync(join(tmpdir(), 'gbrain-crash-robot-fake-'));try {return await runRobotPhase({ engine: 'pglite', seed: 0, seconds: 0, scratch, home: scratch, databases: [],- spawn, track: () => {}, replay: [replayRun(fault)], log: () => {} });+ spawn, track: () => {}, replay: [{ ...replayRun(fault), ...planned }], log: () => {} });} finally { rmSync(scratch, { recursive: true, force: true }); }}const clean = { result: { violations: [], deferred: [] } };@@ -597,6 +597,55 @@ await faultPoint('effect:git:mid', { requestId: 'r1', signal: cyclic as never })expect(robotFailure(robot.failing_runs, { engine: 'pglite' })).toContain('fault_not_reached');});+ const short = (reached: number) => ({ done: { result: { violations: [], deferred: [], reached } } });+ const publication = (nth: number) => ({ point: 'publication:before_publication' as FaultPoint, nth });++ test('the run hook reports how many slots of its seam it made', () => {+ const hook = robotFaultHook({ point: 'effect:git:mid', nth: 3 });+ hook('effect:git:mid', {}); hook('consumer:claiming', {}); hook('effect:git:mid', {});+ expect(hook.reached()).toBe(2);+ });++ test('#8078: a run one slot short of the count run (the race\'s other outcome) crashes once more at its last slot', async () => {+ const { spawn, spawned } = fakeSpawn([short(5), { fault: publication(5) }, { done: clean }]);+ const robot = await phase(spawn, publication(6), { counted: 6 });+ expect(spawned).toEqual([{ role: 'run', killed: false }, { role: 'run', killed: true }, { role: 'recover', killed: false }]);+ expect(robot.runs[0]).toMatchObject({ crashed: true, fault: publication(5), counted: 6, retargeted: { nth: 6, reached: 5 } });+ expect(robot.runs[0]!.fault_not_reached).toBeUndefined();+ expect({ crashed: robot.crashed_runs, retargeted: robot.faults_retargeted.length, failing: robot.failing_runs.length })+ .toEqual({ crashed: 1, retargeted: 1, failing: 0 });+ expect(robotFullGate(robot, 'full')).toBe(true);+ });++ test('#8078: a run two slots short of the count run fails as fault_not_reached', async () => {+ const { spawn, spawned } = fakeSpawn([short(4)]);+ const robot = await phase(spawn, publication(6), { counted: 6 });+ expect(spawned.map(s => s.role)).toEqual(['run']);+ expect(robot.runs[0]).toMatchObject({ crashed: false, fault_not_reached: true, reached: 4, counted: 6 });+ expect({ unreached: robot.faults_not_reached.length, retargeted: robot.faults_retargeted.length, failing: robot.failing_runs.length })+ .toEqual({ unreached: 1, retargeted: 0, failing: 1 });+ expect(robotFullGate(robot, 'full')).toBe(false);+ });++ test('#8078: a retargeted run that is short again fails; the retarget happens once', async () => {+ const { spawn, spawned } = fakeSpawn([short(5), short(4)]);+ const robot = await phase(spawn, publication(6), { counted: 6 });+ expect(spawned.map(s => s.role)).toEqual(['run', 'run']);+ expect(robot.runs[0]).toMatchObject({ crashed: false, fault_not_reached: true, fault: publication(5), reached: 4, retargeted: { nth: 6, reached: 5 } });+ expect(robot.failing_runs).toHaveLength(1);+ expect(robotFullGate(robot, 'full')).toBe(false);+ });++ test('#8078: a run that made no slot of its seam, or whose fault no count run planned, still fails', async () => {+ for (const [reached, planned] of [[0, { counted: 1 }], [5, {}]] as const) {+ const { spawn, spawned } = fakeSpawn([short(reached)]);+ const robot = await phase(spawn, publication(6), planned);+ expect(spawned.map(s => s.role)).toEqual(['run']);+ expect(robot.runs[0]).toMatchObject({ fault_not_reached: true, reached });+ expect(robot.failing_runs).toHaveLength(1);+ }+ });+test('a fault_hook_error kills the worker and fails the run and the full gate', async () => {const { spawn, spawned } = fakeSpawn([{ fault_hook_error: { point: 'effect:git:mid', nth: 1, message: 'JSON.stringify cannot serialize BigInt.' } }]);const robot = await phase(spawn, { point: 'effect:git:mid', nth: 1 });