Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(autopilot): launchd paths on the home volume for an external brain; a pause is not a worker stall (#8024, from #8144)
garrytaned76db0760ea
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 81181fb27a..b364444463 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,6 +16,12 @@ identifiers and attribution are available in the pre-removal Git revisionNow `serve-restart-attempt.json` under the brain's `.gbrain` directory remembers the build the restart was tried on, so each build gets one attempt across relaunches. The autopilot restarts the serve service only when its plist or unit starts this brain's own serve wrapper, and audits any other case as `skipped` / `manual`. The restart command runs asynchronously with a 120-second cap.+**On macOS, an autopilot for a brain on an external volume now starts under launchd instead of exiting 78, and an autopilot pause no longer makes the worker restart itself every ten minutes or makes doctor call the held queue wedged.** launchd opens a job's program and its log files before the job runs. When those sat on a volume other than the home folder, as they do for a brain on an external `GBRAIN_HOME`, launchd ended every spawn with exit code 78 (`EX_CONFIG`) and no pid, even though the same wrapper ran fine by hand. `gbrain autopilot --install` now detects a brain home on another volume, has launchd start the wrapper through `/bin/bash`, and writes the job's logs to `~/Library/Logs/gbrain/<label>.log` and `.err`. The wrapper and every brain file stay under `GBRAIN_HOME`, and `--status` reads the new log path. A brain on the home volume gets the same plist as before. Re-run `--install` to move an existing job onto the new layout.++An autopilot pause (`gbrain autopilot pause`, or the marker a migration sets) stops the worker's job claims. The worker's own no-progress watchdog didn't know that: with jobs waiting and nothing completing, it emitted `stalled` and the autopilot restarted its worker about every ten minutes for as long as the pause lasted. Now the watchdog restarts its clock while either marker is present, so a long pause can't trip it the moment the pause clears. The database liveness probe keeps running, and a real stall after resume is still caught. Doctor's `wedged_queue` reports a queue held by a pause as a warning that says which pause holds it and how it clears, instead of failing with advice to restart the worker.++Contributed by @Masashi-Ono0611 (reports #8017 and #7966, and the approaches in #8018 and #7968).+### Itemized changes- `src/commands/upgrade-restart.ts`: the per-build attempt marker in `restartServeAfterSelfUpgrade`; the ownership check before `restartStaleServeService` collects processes; `hostExecutor` spawns asynchronously and kills a command past its timeout (120 s for the autopilot's serve restart, 600 s for `gbrain upgrade --restart-only` as before).@@ -23,6 +29,12 @@ Now `serve-restart-attempt.json` under the brain's `.gbrain` directory remembers- `src/core/upgrade-restart.ts`: `RestartExecutor.run` may return a promise, and `runRestartSteps` awaits each command.- Docs: the Restart section of `docs/guides/upgrades-auto-update.md` and the `upgrade-restart.ts` key-files entry.- Tests: `test/autopilot-serve-restart-8013.test.ts`.+- `src/core/autopilot-paths.ts`: `autopilotLaunchdHostPaths` compares the brain home's volume with the home folder's; on macOS `resolveAutopilotJob` takes its log paths from it and sets `launchdViaBash` for a brain on another volume.+- `src/commands/autopilot.ts` and `src/commands/autopilot/jobs.ts`: `generateLaunchdPlist` writes `/bin/bash` before the wrapper when `launchdViaBash` is set; `installLaunchd` creates the log directory; `plistWrapperPath` reads the wrapper after `/bin/bash`, so job ownership, status and uninstall keep working.+- `src/core/minions/worker.ts`: the non-supervised stall watchdog treats `autopilotPaused()` as a hold.+- `src/commands/doctor/checks/queue-jobs.ts`: `wedged_queue` returns `warn` with `details.paused` (`operator` or `cooperative`) while a pause marker is present.+- Docs: `docs/guides/live-sync.md` (log location for an external brain on macOS), `docs/guides/queue-operations-runbook.md` (a paused queue), and the `worker.ts` key-files entry.+- Tests: `test/autopilot-launchd-external-home-8017.test.ts` and `test/autopilot-pause-watchdog-7966.test.ts`.## [0.60.699.0] - 2026-11-12diff --git a/docs/architecture/key-files/core-minions-2.md b/docs/architecture/key-files/core-minions-2.mdindex 9012232caa..f7709ddfee 100644--- a/docs/architecture/key-files/core-minions-2.md+++ b/docs/architecture/key-files/core-minions-2.md@@ -49,6 +49,6 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/runtime-build.ts` — #7245: `runtimeInstallRoot()` (the package root this process loaded from; null without a readable package.json, as in a compiled binary), `installedVersion(root)` (the `gbrain` package.json version on disk now) and `serveBuild()` (`{version, installed_version, restart_pending}` for `/health`, disk read cached 10 s, never a path).- `src/core/minions/no-worker.ts` — queue honesty (agent-first operator wave E5). `queueWorkerAlive(queue, read?)` (a live registered worker serves the queue on this host; undefined when the registry is unreadable), `runWaitingJobsFix(engineKind, queue)` (PGLite: `gbrain jobs work [--queue Q]`, a foreground drain, `requires_exclusive`; Postgres: `gbrain jobs supervisor start`), `noWorkerNotice(...)` (code `no_worker`, kind `degraded`). Consumers: CLI `jobs submit` on PGLite refuses without `--follow` unless `--queue-only` (`src/commands/jobs/submit.ts`); `gbrain jobs work` on PGLite drains via `drainPgliteQueue` (`src/commands/jobs/work.ts`) and exits; MCP `submit_job` / `get_job_stats` emit the notice (`src/core/ops/jobs.ts`, PGLite fix wrapped by readiness `exclusiveFix` behind a running serve); `deriveWedgeSignal(wedge, {workerAlive})` returns `no_worker` and never calls that case wedged (`jobs stats`, `get_job_stats`); doctor `queue_health` counts PGLite waiting rows (`src/commands/doctor/checks/queue-jobs.ts`). Pinned by `test/queue-no-worker.test.ts`.-- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). The `ready` event follows final startup checks; listener failure stops startup before claims or timers. The renewal adapter latches typed configuration faults before the generic retry tick consumes them, including deadline verification. Configuration shutdown stops claims, tracks actual executions separately from scheduling slots, and releases only confirmed stopped work through `MinionQueue.releaseConfigurationJob` within one bounded shutdown envelope. The active/token-fenced release returns `released`, `no_op` or `unconfirmed`, leaves attempt/stall counters unchanged, and delays released work for 30 seconds; an unconfirmed stop or DB write falls back to lease expiry. Outstanding claims register their token before the post-claim configuration check so late claims cannot launch handlers. Aborted jobs call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`); `shutdownAbort` (instance field) fires on SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` (shell handler listens; non-shell handlers don't). Per-job timeout fires `abort.abort(new Error('timeout'))` then a 30s grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead if the handler ignores the abort signal (the generic abort listener fires for ANY abort reason). The `launchJob` lock-renewal block is a thin sync wrapper around the pure `runLockRenewalTick` from `src/core/minions/lock-renewal-tick.ts` (NEVER `setInterval(async () => await renewLock(...))` — that shape surfaces an unhandledRejection during PgBouncer rotation). Guarantees: (1) `cancelled` flag captured in the timer closure stops in-flight IIFEs writing misleading audit events after the job ended; (2) re-entrancy guard `tickInFlight` (skips counted as `overlapSkips` telemetry) + per-call `Promise.race` timeout with best-effort AbortSignal cancellation; (3) verify-before-evict: eviction requires a fenced-false (certain loss) or the `hardEvictMs` backstop — never bare local arithmetic; the per-job lease (`job.lock_duration_ms ?? opts.lockDuration`) drives the renewal state, and the cadence clamps to `min(lease/2, 60s)`; (4) explicit `.catch()` on the stored `executeJob(...).finally(...)` promise closes the second unhandledRejection vector; (5) exported `INFRASTRUCTURE_ABORT_REASONS = new Set(['lock-renewal-failed', 'lock-lost'])` so executeJob's catch skips `failJob` for these (PgBouncer blips don't dead-letter healthy jobs; the stall detector reclaims); worker configuration shutdown is not in this set — it is gated separately by the typed `configurationError` latch; (6) inFlight generation-safety — force-evict and the handler's finally delete the inFlight entry ONLY when it still carries their own lockToken (the token is the generation; an evicted execution's late delete cannot remove a same-worker re-claim's entry); (7) eviction observability — the worker keeps a `perf_hooks.monitorEventLoopDelay` histogram (reset on every successful renewal, sampled ns→ms at eviction) plus raw loadavg, and both the abort and grace-evict log lines carry cause / since-last-success / tick-lateness / overlap-skips / load / event-loop-delay from the stashed `abortMeta`. CI guard `scripts/check-worker-lock-renewal-shape.sh` (in `bun run verify`) asserts the bug pattern stays absent AND `launchJob` keeps calling `runLockRenewalTick`. Engine-ownership invariant: `start()` does NOT call `engine.disconnect()` on shutdown — the CLI handler in `src/commands/jobs.ts case 'work'` owns engine lifecycle via try/finally with loud error logging. RSS watchdog uses non-file-backed pages on Linux: exported `parseRssFromProcStatus(status)` (pure parser; field-presence regex so `RssAnon: 0 + RssShmem: 512` parses correctly) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5); the default `getRss` in `WorkerOpts` is `getAccurateRss`. `checkMemoryLimit` tracks peak RSS, fires an 80%-of-cap soft-warn (once per crossing, carrying peak + in-flight job kinds), and on exceed sets `_rssWatchdogTriggered=true` (exposed via `get rssWatchdogTriggered()`) so `jobs work`'s finally can `process.exit(WORKER_EXIT_RSS_WATCHDOG)` after disconnect (drain self-identifying instead of an opaque code-0 exit). The poll loop wraps `claim` in try/catch: on a retryable conn error it reconnects ONCE and continues to the next tick rather than blind-retrying (a retry after `UPDATE...RETURNING` committed but the socket died would double-claim). `LockRenewalDeps` is wired with `reconnect` when the engine supports it. The self-health DB-liveness probe runs EVEN under a supervisor (`GBRAIN_SUPERVISED=1`): the outer guard is `if (this.opts.healthCheckInterval > 0)` and only the STALL-detection block is wrapped in `if (!isSupervisedChild)` — so a supervised worker whose own pool dies self-exits `unhealthy(db_dead)` after `dbFailExitAfter` probes (the supervisor watches a different connection and can't see this worker's dead pool), while the supervisor's progress watchdog owns forward-progress. Pinned by the `test/worker-lock-renewal.test.ts` hermetic suite, `test/audit/lock-renewal-audit.test.ts`, `test/scripts/check-worker-lock-renewal-shape.test.ts`, `test/worker-shutdown-disconnect.test.ts` (asserts `disconnectSpy).not.toHaveBeenCalled()`), `test/worker-rss.test.ts` (11), `test/worker-supervised-db-probe.test.ts` (3). Per-job process isolation: `MinionWorkerOpts.jobIsolation` (`inline`|`process`) + `childCliInvocation`/`childTiniPath` swap `handler(context)` for `runJobInChild(...)` in executeJob — every reporting branch (completeJob, failJob, lease release, infra no-burn) is reused on the child outcome; when isolated the parent-side context is not built at all. Three extra no-burn child classes in the catch: `ChildSpawnInfraError` + `ChildWorkerShutdownError` (released, no attempt burned) and `ChildNotClaimedError` (the child proved the claim was already gone — reclaimed/cancelled before the handler ran — so nothing is recorded against it). The health probe delegates to `runDbProbe` (db-probe.ts) with a cancellation signal on every probe and emits a `verdict` (`pool_starved`/`server_unreachable`/`unknown`/`client_misconfigured`); typed configuration faults block immediately instead of burning the transient DB failure threshold. `getHandler(name)` is the read-only registry accessor run-child uses. Inline handlers run under `runWithJobSpend`; an isolated spend-authorized job passes `spendAuthorized` to the child runner's capability gate. #5062: `requestShutdown()` is the signal entry point (the `jobs work` signal owner calls it for SIGHUP/broken pipe too); a handler that ends because of the shutdown signal hands its claim back (`worker-shutdown.ts`: token-fenced, no attempt burned, `error_text = 'worker_shutdown: <handler error>'`); any handler error during shutdown is presumed shutdown-induced (`endedByShutdown` in `worker-shutdown.ts`) except `UnrecoverableError`, which dead-letters with its original text (W9F Decision 7; in-process and, for a child whose stop is confirmed, isolated; `test/worker-shutdown-error-matrix.test.ts`), and claims still running after the 30s drain are handed back the same way with `drainForced` set, which `jobs work` turns into exit `WORKER_EXIT_DRAIN_FORCED` (17). The RSS watchdog's `gracefulShutdown(reason)` aborts every `inFlight` job and records the reason; a claim whose `UPDATE … RETURNING` committed before the shutdown but whose result had not reached the poll loop is not in `inFlight` yet, so `launchJob` aborts it with the recorded reason as soon as it registers it, and the job takes the abort path instead of running unaborted through the drain (forced case in `test/worker-shutdown-error-matrix.test.ts`, which holds the claim result until the watchdog has fired). Each `inFlight` entry carries `shared` (another job ran beside it, or it registered after the drain began); the catch hands a watchdog-aborted shared job to `settleWatchdogAbort` (back to `waiting` uncharged, marked to run alone), so only a job that ran alone is charged `aborted: watchdog`, and it is marked too. A worker running a marked job claims nothing else, and a worker with jobs in flight claims no marked job. GBRA-133 2a.4: the idle poll wait (nothing claimed, nothing in flight) goes through `idleWaiter` (`src/core/minions/idle-wait.ts`), which the sync-queue wake ends early; `claimStarted()` runs right before each claim, and `stop()` and `requestShutdown()` end the wait at once. Before recording a handler's return, the worker reads it with `jobResultOutcome`: a reported failure ends the job through `failJob(..., 'failed', 0, result)` and a partial one completes, each storing `jobOutcomeResult`.+- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). The `ready` event follows final startup checks; listener failure stops startup before claims or timers. The renewal adapter latches typed configuration faults before the generic retry tick consumes them, including deadline verification. Configuration shutdown stops claims, tracks actual executions separately from scheduling slots, and releases only confirmed stopped work through `MinionQueue.releaseConfigurationJob` within one bounded shutdown envelope. The active/token-fenced release returns `released`, `no_op` or `unconfirmed`, leaves attempt/stall counters unchanged, and delays released work for 30 seconds; an unconfirmed stop or DB write falls back to lease expiry. Outstanding claims register their token before the post-claim configuration check so late claims cannot launch handlers. Aborted jobs call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`); `shutdownAbort` (instance field) fires on SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` (shell handler listens; non-shell handlers don't). Per-job timeout fires `abort.abort(new Error('timeout'))` then a 30s grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead if the handler ignores the abort signal (the generic abort listener fires for ANY abort reason). The `launchJob` lock-renewal block is a thin sync wrapper around the pure `runLockRenewalTick` from `src/core/minions/lock-renewal-tick.ts` (NEVER `setInterval(async () => await renewLock(...))` — that shape surfaces an unhandledRejection during PgBouncer rotation). Guarantees: (1) `cancelled` flag captured in the timer closure stops in-flight IIFEs writing misleading audit events after the job ended; (2) re-entrancy guard `tickInFlight` (skips counted as `overlapSkips` telemetry) + per-call `Promise.race` timeout with best-effort AbortSignal cancellation; (3) verify-before-evict: eviction requires a fenced-false (certain loss) or the `hardEvictMs` backstop — never bare local arithmetic; the per-job lease (`job.lock_duration_ms ?? opts.lockDuration`) drives the renewal state, and the cadence clamps to `min(lease/2, 60s)`; (4) explicit `.catch()` on the stored `executeJob(...).finally(...)` promise closes the second unhandledRejection vector; (5) exported `INFRASTRUCTURE_ABORT_REASONS = new Set(['lock-renewal-failed', 'lock-lost'])` so executeJob's catch skips `failJob` for these (PgBouncer blips don't dead-letter healthy jobs; the stall detector reclaims); worker configuration shutdown is not in this set — it is gated separately by the typed `configurationError` latch; (6) inFlight generation-safety — force-evict and the handler's finally delete the inFlight entry ONLY when it still carries their own lockToken (the token is the generation; an evicted execution's late delete cannot remove a same-worker re-claim's entry); (7) eviction observability — the worker keeps a `perf_hooks.monitorEventLoopDelay` histogram (reset on every successful renewal, sampled ns→ms at eviction) plus raw loadavg, and both the abort and grace-evict log lines carry cause / since-last-success / tick-lateness / overlap-skips / load / event-loop-delay from the stashed `abortMeta`. CI guard `scripts/check-worker-lock-renewal-shape.sh` (in `bun run verify`) asserts the bug pattern stays absent AND `launchJob` keeps calling `runLockRenewalTick`. Engine-ownership invariant: `start()` does NOT call `engine.disconnect()` on shutdown — the CLI handler in `src/commands/jobs.ts case 'work'` owns engine lifecycle via try/finally with loud error logging. RSS watchdog uses non-file-backed pages on Linux: exported `parseRssFromProcStatus(status)` (pure parser; field-presence regex so `RssAnon: 0 + RssShmem: 512` parses correctly) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5); the default `getRss` in `WorkerOpts` is `getAccurateRss`. `checkMemoryLimit` tracks peak RSS, fires an 80%-of-cap soft-warn (once per crossing, carrying peak + in-flight job kinds), and on exceed sets `_rssWatchdogTriggered=true` (exposed via `get rssWatchdogTriggered()`) so `jobs work`'s finally can `process.exit(WORKER_EXIT_RSS_WATCHDOG)` after disconnect (drain self-identifying instead of an opaque code-0 exit). The poll loop wraps `claim` in try/catch: on a retryable conn error it reconnects ONCE and continues to the next tick rather than blind-retrying (a retry after `UPDATE...RETURNING` committed but the socket died would double-claim). `LockRenewalDeps` is wired with `reconnect` when the engine supports it. The self-health DB-liveness probe runs EVEN under a supervisor (`GBRAIN_SUPERVISED=1`): the outer guard is `if (this.opts.healthCheckInterval > 0)` and only the STALL-detection block is wrapped in `if (!isSupervisedChild)` — so a supervised worker whose own pool dies self-exits `unhealthy(db_dead)` after `dbFailExitAfter` probes (the supervisor watches a different connection and can't see this worker's dead pool), while the supervisor's progress watchdog owns forward-progress. Pinned by the `test/worker-lock-renewal.test.ts` hermetic suite, `test/audit/lock-renewal-audit.test.ts`, `test/scripts/check-worker-lock-renewal-shape.test.ts`, `test/worker-shutdown-disconnect.test.ts` (asserts `disconnectSpy).not.toHaveBeenCalled()`), `test/worker-rss.test.ts` (11), `test/worker-supervised-db-probe.test.ts` (3). Per-job process isolation: `MinionWorkerOpts.jobIsolation` (`inline`|`process`) + `childCliInvocation`/`childTiniPath` swap `handler(context)` for `runJobInChild(...)` in executeJob — every reporting branch (completeJob, failJob, lease release, infra no-burn) is reused on the child outcome; when isolated the parent-side context is not built at all. Three extra no-burn child classes in the catch: `ChildSpawnInfraError` + `ChildWorkerShutdownError` (released, no attempt burned) and `ChildNotClaimedError` (the child proved the claim was already gone — reclaimed/cancelled before the handler ran — so nothing is recorded against it). The health probe delegates to `runDbProbe` (db-probe.ts) with a cancellation signal on every probe and emits a `verdict` (`pool_starved`/`server_unreachable`/`unknown`/`client_misconfigured`); typed configuration faults block immediately instead of burning the transient DB failure threshold. `getHandler(name)` is the read-only registry accessor run-child uses. Inline handlers run under `runWithJobSpend`; an isolated spend-authorized job passes `spendAuthorized` to the child runner's capability gate. #5062: `requestShutdown()` is the signal entry point (the `jobs work` signal owner calls it for SIGHUP/broken pipe too); a handler that ends because of the shutdown signal hands its claim back (`worker-shutdown.ts`: token-fenced, no attempt burned, `error_text = 'worker_shutdown: <handler error>'`); any handler error during shutdown is presumed shutdown-induced (`endedByShutdown` in `worker-shutdown.ts`) except `UnrecoverableError`, which dead-letters with its original text (W9F Decision 7; in-process and, for a child whose stop is confirmed, isolated; `test/worker-shutdown-error-matrix.test.ts`), and claims still running after the 30s drain are handed back the same way with `drainForced` set, which `jobs work` turns into exit `WORKER_EXIT_DRAIN_FORCED` (17). The RSS watchdog's `gracefulShutdown(reason)` aborts every `inFlight` job and records the reason; a claim whose `UPDATE … RETURNING` committed before the shutdown but whose result had not reached the poll loop is not in `inFlight` yet, so `launchJob` aborts it with the recorded reason as soon as it registers it, and the job takes the abort path instead of running unaborted through the drain (forced case in `test/worker-shutdown-error-matrix.test.ts`, which holds the claim result until the watchdog has fired). Each `inFlight` entry carries `shared` (another job ran beside it, or it registered after the drain began); the catch hands a watchdog-aborted shared job to `settleWatchdogAbort` (back to `waiting` uncharged, marked to run alone), so only a job that ran alone is charged `aborted: watchdog`, and it is marked too. A worker running a marked job claims nothing else, and a worker with jobs in flight claims no marked job. GBRA-133 2a.4: the idle poll wait (nothing claimed, nothing in flight) goes through `idleWaiter` (`src/core/minions/idle-wait.ts`), which the sync-queue wake ends early; `claimStarted()` runs right before each claim, and `stop()` and `requestShutdown()` end the wait at once. Before recording a handler's return, the worker reads it with `jobResultOutcome`: a reported failure ends the job through `failJob(..., 'failed', 0, result)` and a partial one completes, each storing `jobOutcomeResult`. #7966: while `autopilotPaused()` (either pause marker), the non-supervised stall watchdog restarts its no-progress clock each tick instead of emitting `stalled`; the DB liveness probe still runs. Doctor's `wedged_queue` reports a paused queue as `warn` (`details.paused`: operator or cooperative) without restart advice. Tests: `autopilot-pause-watchdog-7966.test.ts`.- `src/core/minions/watchdog-attribution.ts` — RSS watchdog attribution (GBRA-130 A-30). The watchdog measures the whole worker process, so a drain with several jobs in flight cannot name the job that grew: `settleWatchdogAbort` requeues each such job token-fenced with no attempt burned (`error_text` `WATCHDOG_DRAIN_RELEASE`) and appends `WATCHDOG_SUSPECT_MARK` once to its `stacktrace`, the job's durable history; a job that ran alone is marked and its caller charges it. An isolated child whose stop is unconfirmed, or any error during the release, leaves the claim to the stall sweep. `unmarkedJobSql` is the claim filter. `requeueCoRunnerStalls` requeues, marked and with no stall counted, the stalled claims of a worker that held other expired claims (lock tokens share the `<worker id>:` prefix, or a co-runner was already requeued with `worker_stalled_with_co_runners: <worker id>`); a marked or lone job's stall counts as before, so a kernel OOM kill gets the same attribution. Doctor's watchdog-kill counts (`error_text = 'aborted: watchdog'` on dead/failed rows) therefore count only jobs charged for a solo trip. Pinned by `test/pathological/a30-watchdog-attribution.test.ts` (both engines) and `test/worker-shutdown-error-matrix.test.ts`.diff --git a/docs/guides/live-sync.md b/docs/guides/live-sync.mdindex 8069b65f9e..0f5311366d 100644--- a/docs/guides/live-sync.md+++ b/docs/guides/live-sync.md@@ -779,7 +779,12 @@ Any other brain gets names with a suffix: `com.gbrain.autopilot.<suffix>`,and a crontab line ending in `# gbrain-autopilot:<suffix>`. The suffix is thefirst 8 hex characters of a random id that `--install` records in`<brain>/.gbrain/autopilot-install-id`. Every brain logs to its own-`<brain>/.gbrain/autopilot.log`.+`<brain>/.gbrain/autopilot.log`. On macOS, when the brain is on a volume other+than your home folder (an external `GBRAIN_HOME`), launchd can't start a job+whose program or log files are on that volume (it exits 78, `EX_CONFIG`). For+such a brain the job starts the wrapper through `/bin/bash` and logs to+`~/Library/Logs/gbrain/<label>.log` and `.err`. Re-run `--install` to move+an existing job onto this layout.`gbrain autopilot --status [--json]` prints the brain's job name, suffix,install-id path, wrapper path and log path (`job` in `--json`). `--status`diff --git a/docs/guides/queue-operations-runbook.md b/docs/guides/queue-operations-runbook.mdindex 8eb16db552..efbe7d9a39 100644--- a/docs/guides/queue-operations-runbook.md+++ b/docs/guides/queue-operations-runbook.md@@ -43,7 +43,11 @@ gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'````wedged_queue` is a per-queue health **error** (0 active_healthy + waiting > 0 +-stale completions). Manual fix if you ever need it:+stale completions). While an autopilot pause is in place (`gbrain autopilot+pause`, or the marker a migration sets), workers claim nothing, so the check+reports the held queue as a **warning** that names the resume step, and the+worker's stall watchdog stays quiet. Restarting a worker doesn't help then.+Manual fix if you ever need it:```bash# Restart the supervisor with a fresh pool. `start` alone runs in thediff --git a/src/commands/autopilot.ts b/src/commands/autopilot.tsindex 47e9fc67bd..f61f2657da 100644--- a/src/commands/autopilot.ts+++ b/src/commands/autopilot.ts@@ -1018,7 +1018,7 @@ export function generateLaunchdPlist(wrapperPath: string, home: string, job?: Au<dict><key>Label</key><string>${escapeXml(job?.launchdLabel ?? autopilotLaunchdLabel())}</string><key>ProgramArguments</key><array>- <string>${escapeXml(wrapperPath)}</string>+${(job?.launchdViaBash ? ['/bin/bash', wrapperPath] : [wrapperPath]).map(arg => ` <string>${escapeXml(arg)}</string>`).join('\n')}</array><key>WorkingDirectory</key><string>${escapeXml(home)}</string><key>RunAtLoad</key><true/>@@ -1044,7 +1044,7 @@ function installLaunchd(wrapperPath: string, home: string, repoPath: string, jobtry {const agentsDir = join(home, 'Library', 'LaunchAgents');- mkdirSync(agentsDir, { recursive: true });+ for (const dir of [agentsDir, dirname(job.logPath)]) mkdirSync(dir, { recursive: true });replaceLegacySharedJob(job, 'macos');writeFileSync(plistPath(job.launchdLabel), plist, { mode: 0o644 });// launchd rejects group/world-writable agent plists: bootstrap/load failsdiff --git a/src/commands/autopilot/jobs.ts b/src/commands/autopilot/jobs.tsindex 218ac3bba9..abfc2aca0d 100644--- a/src/commands/autopilot/jobs.ts+++ b/src/commands/autopilot/jobs.ts@@ -35,9 +35,9 @@ export function legacyStartScriptPath(): string {export const shellQuote = (s: string) => `'${s.replace(/'/g, "'\\''")}'`;export const unshellQuote = (s: string) => s.replace(/'\\''/g, "'");-/** The wrapper a launchd plist runs (its first ProgramArguments entry). */+/** The wrapper a launchd plist runs: its first ProgramArguments entry, or the second after `/bin/bash` (#8017). */export function plistWrapperPath(plist: string): string | null {- const m = /<key>ProgramArguments<\/key>\s*<array>\s*<string>([^<]*)<\/string>/.exec(plist);+ const m = /<key>ProgramArguments<\/key>\s*<array>\s*(?:<string>\/bin\/bash<\/string>\s*)?<string>([^<]*)<\/string>/.exec(plist);if (!m) return null;return m[1].replace(/"/g, '"').replace(/>/g, '>').replace(/</g, '<').replace(/&/g, '&');}diff --git a/src/commands/doctor/checks/queue-jobs.ts b/src/commands/doctor/checks/queue-jobs.tsindex 6b90998ca6..b6807e61fd 100644--- a/src/commands/doctor/checks/queue-jobs.ts+++ b/src/commands/doctor/checks/queue-jobs.ts@@ -5,6 +5,8 @@* doctor.ts) and buildChecks / doctorReportRemote consume them.*/import type { BrainEngine } from '../../../core/engine.ts';+import { existsSync } from 'node:fs';+import { autopilotOperatorPauseMarkerPath, autopilotPaused, autopilotPauseReason } from '../../../core/autopilot-paths.ts';import { resolveEnvNumber } from '../../../core/env-number.ts';import type { Check } from '../../doctor.ts';import { checkError, doctorVerify } from '../check-fix.ts';@@ -433,6 +435,19 @@ export async function computeWedgedQueueCheck(return { name: 'wedged_queue', status: 'warn', message: `Graduation exemption: ${wedged.map(w => w.label).join(', ')} carried from the quiesced source; workers start after cutover.`,details: { graduation_exempt: true, run_id: process.env.GBRAIN_GRADUATION_RUN, queues: wedged.map(w => w.queue) } };}+ // #7966: an autopilot pause stops every worker's claims, so the backlog is the hold, not a wedge: report it+ // without restart advice (a restart claims nothing while the marker stays).+ if (autopilotPaused()) {+ const operator = existsSync(autopilotOperatorPauseMarkerPath());+ return {+ name: 'wedged_queue',+ status: 'warn',+ message: `Held by ${operator ? 'the operator\'s autopilot pause' : 'the migration pause marker'} (${autopilotPauseReason()}): ${wedged.map(w => w.label).join('; ')}. ` ++ 'Workers claim no jobs while it is present, so a restart does not help; ' ++ (operator ? '`gbrain autopilot resume` releases the hold.' : 'it clears when the migration that set it finishes.'),+ details: { wedged_queues: wedged.length, paused: operator ? 'operator' : 'cooperative', threshold_minutes: thresholdMin },+ };+ }// #3063: "worker alive but not claiming work" would be a false claim// for a queue no worker process was ever subscribed to. Split the// wedged set by whether a registered live worker actually exists fordiff --git a/src/core/autopilot-paths.ts b/src/core/autopilot-paths.tsindex b5172beeb0..5e64a3ba13 100644--- a/src/core/autopilot-paths.ts+++ b/src/core/autopilot-paths.ts@@ -235,6 +235,8 @@ export interface AutopilotJob {wrapperPath: string;logPath: string;errPath: string;+ /** #8017: launchd starts the wrapper through /bin/bash because the brain home is on another volume (macOS only). */+ launchdViaBash?: boolean;installIdPath: string | null;/** configDir() at resolution time. */homeDir: string;@@ -249,6 +251,7 @@ export function resolveAutopilotJob(opts: { mint?: boolean } = {}): AutopilotJobsuffix = id ? id.slice(0, 8) : null;kind = suffix ? 'suffixed' : 'unassigned';}+ const logs = autopilotLaunchdHostPaths(homeDir, autopilotLaunchdLabel(suffix), homedir(), process.platform === 'darwin' ? undefined : () => 0);return {kind,suffix,@@ -257,13 +260,35 @@ export function resolveAutopilotJob(opts: { mint?: boolean } = {}): AutopilotJobstartScriptPath: join(homeDir, suffix ? `start-autopilot-${suffix}.sh` : 'start-autopilot.sh'),cronMarker: suffix ? `# gbrain-autopilot:${suffix}` : null,wrapperPath: join(homeDir, 'autopilot-run.sh'),- logPath: join(homeDir, 'autopilot.log'),- errPath: join(homeDir, 'autopilot.err'),+ logPath: logs.logPath,+ errPath: logs.errPath,+ ...(logs.viaBash ? { launchdViaBash: true } : {}),installIdPath: kind === 'default' ? null : autopilotInstallIdPath(),homeDir,};}+/**+ * #8017: launchd's spawn helper opens a job's Program and its log files before the job runs, and ends the job with+ * EX_CONFIG (78) when they sit on a volume other than the user's home (a brain on an external GBRAIN_HOME). Such a+ * brain gets launchd paths on the home volume: /bin/bash starts the wrapper, and the logs go to+ * ~/Library/Logs/gbrain/<label>.log and .err. A brain on the home volume, or one whose volume cannot be read, keeps+ * its logs in the brain home.+ */+export function autopilotLaunchdHostPaths(+ homeDir: string,+ label: string,+ hostHome: string = homedir(),+ dev: (path: string) => number = path => statSync(path).dev,+): { viaBash: boolean; logPath: string; errPath: string } {+ let external = false;+ try { external = dev(homeDir) !== dev(hostHome); } catch { /* unreadable: keep the brain-home layout */ }+ // nosemgrep: javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal -- hostHome is this user's home directory and the rest are fixed segments.+ const [logDir, base] = external ? [join(hostHome, 'Library', 'Logs', 'gbrain'), label] : [homeDir, 'autopilot'];+ // nosemgrep: javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal -- logDir is the brain's own config dir or ~/Library/Logs/gbrain; label passed autopilotLaunchdLabel's [A-Za-z0-9._-] grammar, which rejects separators and '..'.+ return { viaBash: external, logPath: join(logDir, `${base}.log`), errPath: join(logDir, `${base}.err`) };+}+/*** Who a job definition belongs to, judged by the wrapper path it runs (the* wrapper always lives at `<brain home>/autopilot-run.sh`):diff --git a/src/core/minions/worker.ts b/src/core/minions/worker.tsindex 3df88580d3..a5053a0fb3 100644--- a/src/core/minions/worker.ts+++ b/src/core/minions/worker.ts@@ -13,7 +13,7 @@ import { assertNoUnreviewedJobs, authorizeJobExecution, withSubmissionAuthority*/import { existsSync } from 'fs';-import { autopilotOperatorPauseMarkerPath, autopilotPausedMarkerPath } from '../autopilot-paths.ts';+import { autopilotOperatorPauseMarkerPath, autopilotPaused, autopilotPausedMarkerPath } from '../autopilot-paths.ts';import type { BrainEngine } from '../engine.ts';import type {MinionJob, MinionJobContext, MinionHandler, MinionWorkerOpts,@@ -728,7 +728,7 @@ export class MinionWorker extends EventEmitter {// detector too would double-act (and both emitting 'unhealthy' +// supervisor SIGTERM race). Bare `gbrain jobs work` keeps it.if (!isSupervisedChild) {- if (this.jobsCompleted > lastKnownCompleted) {+ if (this.jobsCompleted > lastKnownCompleted || autopilotPaused()) { // #7966: a pause holds claims, so its ticks restart the clocklastKnownCompleted = this.jobsCompleted;lastCompletionTime = Date.now();stallWarningSince = null;diff --git a/test/autopilot-launchd-external-home-8017.test.ts b/test/autopilot-launchd-external-home-8017.test.tsnew file mode 100644index 0000000000..9e649b0aa4--- /dev/null+++ b/test/autopilot-launchd-external-home-8017.test.ts@@ -0,0 +1,69 @@+/**+ * #8017: launchd's spawn helper opens the job's Program and its StandardOut/StandardError paths before the job runs.+ * When those sit on a volume other than the user's home (a brain on an external GBRAIN_HOME), launchd ends the job+ * with EX_CONFIG (78) and no pid, while the same wrapper runs fine by hand. The launchd-facing paths now stay on the+ * home volume (/bin/bash starts the wrapper; logs under ~/Library/Logs/gbrain) while the wrapper and the brain stay+ * under GBRAIN_HOME. A brain on the home volume keeps the plist it always had.+ */+import { describe, expect, test } from 'bun:test';+import { join } from 'node:path';+import { generateLaunchdPlist, plistWrapperPath } from '../src/commands/autopilot.ts';+import { autopilotLaunchdHostPaths, type AutopilotJob } from '../src/core/autopilot-paths.ts';++const HOST = '/Users/alice-example';+const EXTERNAL = '/Volumes/brain-disk/gbrain/.gbrain';+const LOCAL = `${HOST}/.gbrain`;+const LABEL = 'com.gbrain.autopilot.0a1b2c3d';+const devOf = (p: string) => (p.startsWith('/Volumes/') ? 2 : 1);++describe('autopilotLaunchdHostPaths (#8017)', () => {+ test('a brain on another volume gets launchd paths on the home volume', () => {+ expect(autopilotLaunchdHostPaths(EXTERNAL, LABEL, HOST, devOf)).toEqual({+ viaBash: true,+ logPath: join(HOST, 'Library', 'Logs', 'gbrain', `${LABEL}.log`),+ errPath: join(HOST, 'Library', 'Logs', 'gbrain', `${LABEL}.err`),+ });+ });++ test('a brain on the home volume keeps its logs in the brain home', () => {+ expect(autopilotLaunchdHostPaths(LOCAL, LABEL, HOST, devOf)).toEqual({+ viaBash: false, logPath: join(LOCAL, 'autopilot.log'), errPath: join(LOCAL, 'autopilot.err'),+ });+ });++ test('an unreadable volume keeps the existing layout', () => {+ const throwing = () => { throw new Error('ENOENT'); };+ expect(autopilotLaunchdHostPaths(EXTERNAL, LABEL, HOST, throwing).viaBash).toBe(false);+ });+});++describe('generateLaunchdPlist for an external brain (#8017)', () => {+ const job = (home: string): AutopilotJob => {+ const host = autopilotLaunchdHostPaths(home, LABEL, HOST, devOf);+ return {+ kind: 'suffixed', suffix: '0a1b2c3d', launchdLabel: LABEL, systemdUnit: 'gbrain-autopilot-0a1b2c3d.service',+ startScriptPath: join(home, 'start-autopilot-0a1b2c3d.sh'), cronMarker: '# gbrain-autopilot:0a1b2c3d',+ wrapperPath: join(home, 'autopilot-run.sh'), logPath: host.logPath, errPath: host.errPath,+ launchdViaBash: host.viaBash, installIdPath: join(home, 'autopilot-install-id'), homeDir: home,+ };+ };++ test('launchd starts /bin/bash with the wrapper, and logs on the home volume', () => {+ const plist = generateLaunchdPlist(join(EXTERNAL, 'autopilot-run.sh'), HOST, job(EXTERNAL));+ expect(plist).toMatch(/<key>ProgramArguments<\/key><array>\s*<string>\/bin\/bash<\/string>\s*<string>\/Volumes\/brain-disk\/gbrain\/\.gbrain\/autopilot-run\.sh<\/string>\s*<\/array>/);+ expect(plist).toContain(`<key>StandardOutPath</key><string>${HOST}/Library/Logs/gbrain/${LABEL}.log</string>`);+ expect(plist).toContain(`<key>StandardErrorPath</key><string>${HOST}/Library/Logs/gbrain/${LABEL}.err</string>`);+ expect(plist).not.toContain('<string>/Volumes/brain-disk/gbrain/.gbrain/autopilot.log</string>');+ });++ test('the wrapper it runs is still read back as this brain\'s, so status and uninstall keep working', () => {+ expect(plistWrapperPath(generateLaunchdPlist(join(EXTERNAL, 'autopilot-run.sh'), HOST, job(EXTERNAL)))).toBe(join(EXTERNAL, 'autopilot-run.sh'));+ });++ test('a brain on the home volume keeps the wrapper as the program', () => {+ const plist = generateLaunchdPlist(join(LOCAL, 'autopilot-run.sh'), HOST, job(LOCAL));+ expect(plist).not.toContain('<string>/bin/bash</string>');+ expect(plistWrapperPath(plist)).toBe(join(LOCAL, 'autopilot-run.sh'));+ expect(plist).toContain(`<key>StandardOutPath</key><string>${LOCAL}/autopilot.log</string>`);+ });+});diff --git a/test/autopilot-pause-watchdog-7966.test.ts b/test/autopilot-pause-watchdog-7966.test.tsnew file mode 100644index 0000000000..94cb121079--- /dev/null+++ b/test/autopilot-pause-watchdog-7966.test.ts@@ -0,0 +1,176 @@+/**+ * #7966: an autopilot pause (the operator pause or the cooperative migrate marker) stops the worker's claims, so the+ * worker's own no-progress watchdog must not read the hold as a stall, and doctor must not prescribe a worker restart+ * for a queue that the pause is holding. Real database liveness checks stay on, and a real stall is still detected.+ */+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { MinionWorker } from '../src/core/minions/worker.ts';+import { computeWedgedQueueCheck } from '../src/commands/doctor.ts';+import { waitFor } from './helpers/wait-for.ts';+import { withEnv } from './helpers/with-env.ts';++let engine: PGLiteEngine;+const dirs: string[] = [];++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({ database_url: '' });+ await engine.initSchema();+});+afterAll(async () => {+ await engine.disconnect();+ for (const d of dirs) rmSync(d, { recursive: true, force: true });+});+beforeEach(async () => { await engine.executeRaw('DELETE FROM minion_jobs'); });++const OPERATOR = 'autopilot-operator-paused';+const COOPERATIVE = 'autopilot-paused';++function brainHome(...markers: string[]): string {+ const home = mkdtempSync(join(tmpdir(), 'gbrain-pause-7966-'));+ dirs.push(home);+ mkdirSync(join(home, '.gbrain'), { recursive: true });+ for (const m of markers) writeFileSync(join(home, '.gbrain', m), 'test hold\n');+ return home;+}++function probeEngine(opts: { waiting: number; probe?: () => Promise<unknown> }) {+ return new Proxy(engine, {+ get(target, prop, receiver) {+ if (prop === 'executeRaw') {+ return async (sql: string, params?: unknown[]): Promise<unknown[]> => {+ if (/^\s*SELECT\s+1\s*$/i.test(sql)) return [await (opts.probe ?? (async () => ({ ok: 1 })))()];+ if (/count\(\*\).*minion_jobs.*WHERE\s+status\s*=\s*'waiting'/is.test(sql)) return [{ cnt: String(opts.waiting) }];+ return target.executeRaw(sql, params);+ };+ }+ return Reflect.get(target, prop, receiver);+ },+ }) as unknown as PGLiteEngine;+}++function worker(e: PGLiteEngine, stall: { warn: number; exit: number }) {+ const w = new MinionWorker(e, {+ queue: 'default', concurrency: 1, healthCheckInterval: 30, stallWarnAfterMs: stall.warn, stallExitAfterMs: stall.exit,+ dbFailExitAfter: 3, pollInterval: 20, stalledInterval: 10_000, maxRssMb: 0,+ });+ w.register('noop', async () => {});+ const events: Array<{ reason: string }> = [];+ w.on('unhealthy', info => { events.push(info); });+ return { w, events };+}++const sleep = (ms: number) => new Promise(r => setTimeout(r, ms));++describe('worker watchdog under an autopilot pause (#7966)', () => {+ for (const [label, markers] of [['operator pause', [OPERATOR]], ['cooperative pause', [COOPERATIVE]], ['both markers', [OPERATOR, COOPERATIVE]]] as const) {+ test(`${label}: waiting jobs and no progress are not a stall`, async () => {+ const home = brainHome(...markers);+ await withEnv({ GBRAIN_HOME: home, GBRAIN_SUPERVISED: undefined }, async () => {+ const { w, events } = worker(probeEngine({ waiting: 5 }), { warn: 50, exit: 100 });+ const run = w.start();+ await sleep(500);+ w.stop();+ await run;+ expect(events.filter(e => e.reason === 'stalled')).toHaveLength(0);+ });+ }, 10_000);+ }++ test('a long pause does not trip the watchdog right after resume, and a real stall afterwards still does', async () => {+ const home = brainHome(COOPERATIVE);+ await withEnv({ GBRAIN_HOME: home, GBRAIN_SUPERVISED: undefined }, async () => {+ const { w, events } = worker(probeEngine({ waiting: 5 }), { warn: 500, exit: 1_000 });+ const run = w.start();+ await sleep(1_500);+ rmSync(join(home, '.gbrain', COOPERATIVE));+ await sleep(400);+ const stalledSoonAfterResume = events.filter(e => e.reason === 'stalled').length;+ await waitFor(() => events.some(e => e.reason === 'stalled'), { timeoutMs: 5_000, label: 'real stall after resume' });+ w.stop();+ await run;+ expect(stalledSoonAfterResume).toBe(0);+ });+ }, 15_000);++ test('removing one of two markers keeps the hold', async () => {+ const home = brainHome(OPERATOR, COOPERATIVE);+ await withEnv({ GBRAIN_HOME: home, GBRAIN_SUPERVISED: undefined }, async () => {+ const { w, events } = worker(probeEngine({ waiting: 5 }), { warn: 50, exit: 100 });+ const run = w.start();+ await sleep(150);+ rmSync(join(home, '.gbrain', COOPERATIVE));+ await sleep(400);+ w.stop();+ await run;+ expect(events.filter(e => e.reason === 'stalled')).toHaveLength(0);+ });+ }, 10_000);++ test('a dead database is still reported while paused', async () => {+ const home = brainHome(OPERATOR);+ await withEnv({ GBRAIN_HOME: home, GBRAIN_SUPERVISED: undefined }, async () => {+ const { w, events } = worker(probeEngine({ waiting: 5, probe: async () => { throw new Error('connection terminated unexpectedly'); } }), { warn: 50, exit: 100 });+ const run = w.start();+ await waitFor(() => events.some(e => e.reason === 'db_dead'), { timeoutMs: 5_000, label: 'db_dead while paused' });+ w.stop();+ await run;+ });+ }, 10_000);++ test('the watchdog reads the markers and leaves them in place', async () => {+ const home = brainHome(OPERATOR, COOPERATIVE);+ await withEnv({ GBRAIN_HOME: home, GBRAIN_SUPERVISED: undefined }, async () => {+ const { w } = worker(probeEngine({ waiting: 5 }), { warn: 50, exit: 100 });+ const run = w.start();+ await sleep(200);+ w.stop();+ await run;+ });+ expect(Bun.file(join(home, '.gbrain', OPERATOR)).size).toBeGreaterThan(0);+ expect(Bun.file(join(home, '.gbrain', COOPERATIVE)).size).toBeGreaterThan(0);+ }, 10_000);+});++describe('doctor wedged_queue under an autopilot pause (#7966)', () => {+ const pgLike = () => ({ kind: 'postgres', executeRaw: engine.executeRaw.bind(engine) }) as unknown as BrainEngine;+ async function seedWedged(): Promise<void> {+ for (const [status, updated] of [['waiting', 'now()'], ['completed', "now() - interval '20 min'"]] as const) {+ await engine.executeRaw(+ `INSERT INTO minion_jobs (submission_authority, name, queue, status, updated_at) VALUES ('{"version":1,"kind":"application"}'::jsonb, 'cycle', 'default', $1, ${updated})`,+ [status],+ );+ }+ }+ const liveWorker = () => [{ queue: 'default' }];++ test('an operator pause reports the held queue as a warning that names the resume, never a restart', async () => {+ await seedWedged();+ const check = await withEnv({ GBRAIN_HOME: brainHome(OPERATOR) }, () => computeWedgedQueueCheck(pgLike(), { readWorkers: liveWorker }));+ expect(check.status).toBe('warn');+ expect(check.message).toContain("'default'");+ expect(check.message).toContain('gbrain autopilot resume');+ expect(check.message).not.toContain('supervisor stop');+ expect(check.details).toMatchObject({ paused: 'operator', wedged_queues: 1 });+ });++ test('a cooperative (migrate) pause reports the hold without a restart either', async () => {+ await seedWedged();+ const check = await withEnv({ GBRAIN_HOME: brainHome(COOPERATIVE) }, () => computeWedgedQueueCheck(pgLike(), { readWorkers: liveWorker }));+ expect(check.status).toBe('warn');+ expect(check.message).not.toContain('supervisor stop');+ expect(check.details).toMatchObject({ paused: 'cooperative' });+ });++ test('without a pause the wedged queue still fails with the restart advice', async () => {+ await seedWedged();+ const check = await withEnv({ GBRAIN_HOME: brainHome() }, () => computeWedgedQueueCheck(pgLike(), { readWorkers: liveWorker }));+ expect(check.status).toBe('fail');+ expect(check.message).toContain('supervisor stop');+ });+});