High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
docs: add browser auto-recovery design note for the soak lane
Bosch002e2d7f0971
diff --git a/tests/soak/AUTORECOVERY-DESIGN.md b/tests/soak/AUTORECOVERY-DESIGN.mdnew file mode 100644index 00000000..cd33f6dd--- /dev/null+++ b/tests/soak/AUTORECOVERY-DESIGN.md@@ -0,0 +1,35 @@+# Design: auto-recover from renderer-OOM degradation (follow-up)++Surfaced by `tests/soak/burst.sh`: under memory pressure Chrome OOM-kills a *renderer* but+the main browser process stays alive, so the instance is **functionally degraded** (new tabs+fail) yet the control plane never recovers it. In the burst run, 7/7 OOMs required an external+`pinchtab server restart`. (The 60s new-tab hang itself is already fixed — new-tab creation+now fails fast with a `503` naming OOM.)++## Root cause+The orchestrator emits crash events only at startup and process-exit, never continuously at+runtime:+- startup probe → `instance.error` (`internal/orchestrator/health.go`)+- process exit → `instance.stopped` (`finalizeInstanceExit`)++A running-but-degraded child (process alive, renderer dead) matches neither, so the autorestart+strategy — which acts only on `instance.error`/`instance.stopped`+(`internal/strategy/autorestart/restart.go`) — never triggers. Instances are child processes+(`orchestrator/runner.go`), so the degraded signal must cross the process boundary via the+child's `/health`.++## Plan (Option A — signal degraded → reuse the existing restart path)+1. **Child reports degraded** (low risk): a consecutive new-tab-OOM-failure counter on+ `Handlers`, incremented on the new-tab OOM 503 path in `runNavigate` (and the `CreateTab`+ OOM path), reset on any successful tab creation; exposed as `degraded` in the instance+ `/health`. Useful on its own (health reflects *functional capability*, not just liveness).+2. **Orchestrator runtime monitor** (sensitive): a goroutine that polls each running managed+ instance's `/health`; if `degraded` for K consecutive polls, emits `instance.error`.+ Guard against restart storms: K-consecutive requirement + the existing `MaxRestarts`/backoff+ in `handleCrash`; only the managed instance; skip while already restarting.+3. **Autorestart**: no change — `instance.error` already routes to `handleCrash` → restart.++## Verification+Reuse `burst.sh` but remove its own `server restart`; assert the instance self-recovers+(instance id changes once per OOM, `degraded` clears) without external intervention, and does+NOT restart-storm under steady light load.