Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

test(persistence): a short crash-robot run that found violations is never retargeted (#8078)

garrytanf4fca0fee7f3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3afb6da655..9f4203fc7c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,15 +14,15 @@ identifiers and attribution are available in the pre-removal Git revision
**`gbrain doctor` now ends on its `--budget-ms` deadline when `git` hangs in an OpenClaw workspace.** The bootstrap entry checks push health by running `git status`, `git branch` and `git rev-list` on the install receipt's workspace, and on a workspace whose last push failed. Those calls ran synchronously with a 10-second limit each. While one hung, nothing else in the process could run, so the run overran its budget and every later check, the cheap incident checks included, started with no time left. They now run asynchronously, like the home-dir probes since #8979. Each call is killed with SIGKILL, together with its process group, at 10 seconds or at the entry's deadline, whichever comes first, and a call that fails or is cut short reads as unverified, as before. Every local probe's output is also capped at 1 MiB again: a probe that prints more than that is killed and reads as absent, which is what `execFile`'s buffer limit did before #8979.
-**The Postgres crash robot no longer fails a full run when a race makes one fewer slot than its count run saw.** The robot first runs each schedule to count the seams it reaches, then crashes one run at a counted slot. In `sync_and_connector_race_direct_write`, the connector republish races a direct write, and the race legally settles two ways: one makes 5 publications, the other 6, with the git effects, preparations and commits following. A crash planned at the 6th publication was never reached by a run that took the 5-publication outcome, and the job failed with `fault_not_reached` (#8078). A run that misses its fault now reports how many slots of that seam it made. If it was exactly one short of the count run, it runs once more at its last slot, and the manifest records `retargeted` on the run and lists it under `faults_retargeted`. A run that made no slot, fell two or more short, or misses again still fails as `fault_not_reached`.
+**The Postgres crash robot no longer fails a full run when a race makes one fewer slot than its count run saw.** The robot first runs each schedule to count the seams it reaches, then crashes one run at a counted slot. In `sync_and_connector_race_direct_write`, the connector republish races a direct write, and the race legally settles two ways: one makes 5 publications, the other 6, with the git effects, preparations and commits following. A crash planned at the 6th publication was never reached by a run that took the 5-publication outcome, and the job failed with `fault_not_reached` (#8078). A run that misses its fault now reports how many slots of that seam it made. If it was exactly one short of the count run, it runs once more at its last slot, and the manifest records `retargeted` on the run and lists it under `faults_retargeted`. A run that found violations, made no slot, fell two or more short, or misses again still fails as `fault_not_reached`.
### Itemized changes
- `src/commands/doctor/local-probe.ts`: `localProbe` moved here from `checks/local-audits.ts`. It takes `ownTimeoutMs` (default 5 s) and `windowsShell` (git runs without cmd.exe on Windows), and it kills a probe and returns null once its stdout passes `LOCAL_PROBE_MAX_OUTPUT_BYTES` (1 MiB).
- `src/commands/doctor/bootstrap-checks.ts`: `bootstrapDoctorChecks(engine, entryDeadlineAt)` runs every git call through `localProbe` with a 10 s bound. `commitsAheadOfOrigin`, `treeMatchesOrigin` and `recordedPushFailureCheck` are async.
- Tests: `test/doctor-bootstrap-git-deadline-8990.test.ts`. A `git` stand-in that ignores SIGTERM and waits on a 6 s child is killed within a 1000 ms budget, on both the workspace probe and the recorded-failure probe, and neither it nor its child survives (each fails on the previous code, at 18 s and 12 s). A probe that floods stdout is killed within 1.5 s and reads as absent, one that prints 2 MiB and exits reads as absent, and one under the cap returns its whole output (the first two fail without the cap).
-- `scripts/persistence/crash-robot.ts`: the run role's fault hook reports `reached`, the slots of its seam it made. `scripts/persistence/robot-driver.ts`: a planned crash carries `counted` (the count run's slots), and `retargetable` reruns a run once when `counted - 1 <= reached < nth`.
-- Tests: `test/persistence-crash-robot.test.ts`. A run one slot short crashes at its last slot and passes; a run two short, a run short again after the retarget, a run with no slot and a run with no counted plan still fail. Replaying the failed #9071 manifest on Postgres failed 3 of 3 before the fix and passed 3 of 3 after it.
+- `scripts/persistence/crash-robot.ts`: the run role's fault hook reports `reached`, the slots of its seam it made. `scripts/persistence/robot-driver.ts`: a planned crash carries `counted` (the count run's slots), and `retargetable` reruns a run once when it found no violations, `counted` is set and `0 < reached`, `counted - 1 <= reached < nth`.
+- Tests: `test/persistence-crash-robot.test.ts`. A run one slot short crashes at its last slot and passes; a short run with violations, a run two short, a run short again after the retarget, a run with no slot and a run with no counted plan still fail. The failed #9071 manifest entry, given the `counted: 6` its count run saw, failed 4 of 4 on Postgres before the fix and passed 3 of 3 after it, crashing at the 5th publication.
## [0.60.624.0] - 2026-11-09
diff --git a/scripts/persistence/README.md b/scripts/persistence/README.md
index 443f093018..043bf94d06 100644
--- a/scripts/persistence/README.md
+++ b/scripts/persistence/README.md
@@ -146,10 +146,11 @@ key lists every run (schedule, seed, seam and occurrence or process fault),
`fault_hook_errors` (a fault hook that threw instead of freezing the worker)
and no `faults_not_reached` (a worker that finished without reaching its
requested seam, outside the per-scenario `UNREACHED_FAULT_ALLOWLIST` in
-`robot-driver.ts`; a run exactly one slot short of its count run, which a
-concurrent group's other legal outcome makes, runs once more at its last slot
-and is listed in `faults_retargeted`); either kind of run also fails validation with a nonzero
-exit and is listed in `failing_runs` for `--replay`.
+`robot-driver.ts`); either kind of run also fails validation with a nonzero
+exit and is listed in `failing_runs` for `--replay`. A run with no violations that
+misses its fault by exactly one slot of its count run, which a concurrent
+group's other legal outcome makes, runs once more at its last slot and is
+listed in `faults_retargeted`.
A failing run prints two commands. `--replay=<manifest>` re-runs exactly the
manifest's failing runs. `--shrink=<manifest>` removes ops from the first
diff --git a/scripts/persistence/robot-driver.ts b/scripts/persistence/robot-driver.ts
index ac7f0f8d91..09f2dad48c 100644
--- a/scripts/persistence/robot-driver.ts
+++ b/scripts/persistence/robot-driver.ts
@@ -89,11 +89,11 @@ export function unreachedAllowed(run: Pick<RobotRun, 'schedule' | 'process' | 'f
* (sync_and_connector_race_direct_write: the connector republish racing a direct write publishes 5 or 6 times, and
* its git effects, preparations and commits follow), so a fault the count run planned at its last slot can be missed
* by a run that took the other outcome. Such a run, short by exactly one slot of the count run, runs once more at the
- * last slot it did make. A run that made no slot, fell more than one short, or missed again fails as before.
+ * last slot it did make. A run that found violations, made no slot, fell more than one short, or missed again fails as before.
*/
export function retargetable(run: RobotRun): boolean {
const reached = run.reached ?? 0;
- return run.fault_not_reached === true && !run.retargeted && run.counted !== undefined && run.fault !== undefined
+ return run.fault_not_reached === true && !run.retargeted && run.counted !== undefined && run.fault !== undefined && run.violations.length === 0
&& reached > 0 && reached >= run.counted - 1 && reached < run.fault.nth;
}
/** A run fails the gate when it found violations, its hook threw, or it never reached a fault it was not allowed to miss. */
diff --git a/test/persistence-crash-robot.test.ts b/test/persistence-crash-robot.test.ts
index 1982c2ae90..8c0d6e95aa 100644
--- a/test/persistence-crash-robot.test.ts
+++ b/test/persistence-crash-robot.test.ts
@@ -617,6 +617,14 @@ await faultPoint('effect:git:mid', { requestId: 'r1', signal: cyclic as never })
expect(robotFullGate(robot, 'full')).toBe(true);
});
+ test('#8078: a short run that found violations is never retargeted; its violations fail the run', async () => {
+ const { spawn, spawned } = fakeSpawn([{ done: { result: { violations: [{ class: 'lost_write', detail: 'planted' }], deferred: [], reached: 5 } } }]);
+ const robot = await phase(spawn, publication(6), { counted: 6 });
+ expect(spawned.map(s => s.role)).toEqual(['run']);
+ expect(robot.violations.map(v => v.detail)).toEqual(['planted']);
+ expect({ retargeted: robot.faults_retargeted.length, failing: robot.failing_runs.length }).toEqual({ retargeted: 0, failing: 1 });
+ });
+
test('#8078: a run two slots short of the count run fails as fault_not_reached', async () => {
const { spawn, spawned } = fakeSpawn([short(4)]);
const robot = await phase(spawn, publication(6), { counted: 6 });