Skip to content

Business operating system for Claude Code — 57 skills, 21 agents, smart daemon. Unified inbox (WhatsApp/Email/Slack/Telegram), autonomous PR merge, full-AWS monitoring, revenue (Stripe+RevenueCat), e-commerce (Shopify), marketing (Klaviyo/Meta/GA4), voice (Bland/ElevenLabs), APM (Datadog/NewRelic/OTEL), YOLO mode.

docs(crs): full-tuning execution plan for FRA relay (HEA-4672) (#611)

Aurora Capitalbee724e4e8b8
diff --git a/docs/runbooks/crs-full-tuning-plan.md b/docs/runbooks/crs-full-tuning-plan.md
new file mode 100644
index 00000000..99179979
--- /dev/null
+++ b/docs/runbooks/crs-full-tuning-plan.md
@@ -0,0 +1,269 @@
+# CRS Full-Tuning Execution Plan (FRA EC2)
+
+> Linear: [HEA-4672](https://linear.app/lifecycle-innovations/issue/HEA-4672) · Owner: <OWNER_NAME> · Created: 2026-06-19
+>
+> Scope: tune the **Claude Relay Service (CRS)** running on the Frankfurt EC2 box so the
+> agent fleet load-balances dynamically across the account pool, cools exhausted accounts
+> correctly, and survives concurrent-stream storms without `ConnectionRefused` cascades.
+> This plan is grounded in the **actual** CRS deployment as verified on the box on
+> 2026-06-19 — not generic relay advice.
+
+---
+
+## 0. Ground truth — the actual CRS deployment
+
+| Fact | Value (verified 2026-06-19) |
+|---|---|
+| Host | `dev-sandbox-fra` EC2 (`i-071b16e6328dbc876`, eu-central-1). SSM-managed; reachable via SSH alias `fra` (tailnet `100.107.38.70`) or `dev-sandbox-fra-cf` (CF-Access SSH) |
+| CRS dir | `/home/ec2-user/crs` — **a working copy on the box, NOT a git repo** (`git remote -v` empty). The Dockerfile/package.json/lock are on-disk; `docker-compose.yml` has `build: .` |
+| Relay container | `crs-claude-relay-1` (`weishaw/claude-relay-service`), bound `127.0.0.1:3005 → 3000`, status healthy |
+| Live image | `weishaw/claude-relay-service:cve-fix-20260615` (the CVE-clean node:22-alpine rebuild). **NOT** a `loadbalance-fix-*` tag — see §1 gap |
+| Redis | `crs-redis-1` (`redis:7-alpine`, :6379). Account state + concurrency counters + utilization live here |
+| Aux containers | redis-commander, prometheus, grafana (compose-defined) |
+| Account pool | ~10 Claude **Max OAuth** accounts: `pool-aurora/<OWNER_BRAND>/<USER>/chairman/foundation/<OWNER_BRAND>-personal/<OWNER_BRAND>-team` + `canary-lifecycle/support/sponsors`. Records: Redis hash `claude:account:<uuid>` |
+| Token refresh | `crs-token-feed.timer` (systemd `--user`, every 5 min) — FRA is the OAuth refresher-of-record |
+| Coarse priority cron | `dynamic-priority.py` (every 5 min) writes CRS `priority` from the same util formula |
+| Routing in | Fleet sessions route via overlay `~/.claude/crs-session-settings.json` (`ANTHROPIC_BASE_URL=…:3005/api` + `ANTHROPIC_AUTH_TOKEN=cr_…`). The `cr_` key MUST be `ANTHROPIC_AUTH_TOKEN`, never `ANTHROPIC_API_KEY` |
+
+### Key scheduler/relay files on disk
+- `src/services/scheduler/unifiedClaudeScheduler.js` — pool selection, concurrency gating, util load-balancing, overload/503 handling, rate-limit tracking. **(modified 2026-06-19 — newest file)**
+- `src/services/account/claudeAccountService.js` — account state, `markAccountRateLimited()`, util writeback (`claudeFiveHourResetsAt`, etc.)
+- `src/services/relay/claudeRelayService.js` — request proxy + 429/error mark sites (non-stream / stream-error / stream-end)
+- `config/config.js` (+ `config.example.js`, `models.js`) — defaults
+
+### What is ALREADY on disk (verified by grep)
+- **Util-based concurrency cap** — `_getMaxConcurrencyFromUtilization(util)` returns a per-account max concurrent stream count keyed on 5-hour utilization:
+
+ ```js
+ // src/services/scheduler/unifiedClaudeScheduler.js:593
+ _getMaxConcurrencyFromUtilization(utilization) {
+ const util = parseFloat(utilization) || 0
+ if (util < 20) return 8
+ if (util < 40) return 5
+ if (util < 60) return 3
+ if (util < 80) return 2
+ return 1
+ }
+ ```
+ Enforced at selection time (line ~801): `currentConcurrency = redis.getClaudeAccountConcurrency(account.id)`; account skipped if `currentConcurrency >= maxConcurrency`.
+- **Dedicated 503 when all accounts saturated** — `CONSOLE_ACCOUNT_CONCURRENCY_FULL` error code thrown when every eligible account is at its concurrency limit.
+- **Util load-balancing** — pool picks sorted by live utilization (`claudeFiveHourUtilization*0.4 + claudeSevenDayUtilization*0.6`), tie-break priority then `lastUsedAt` (the `📊 Load-balance pick` path).
+- **Rate-limit tracking** — `rateLimitEndAt` / `rateLimitResetTimestamp` per account, `overloaded` status handled.
+
+---
+
+## 1. ⚠️ The critical gap — credit-exhaustion cooldown is NOT on disk
+
+This is the **single highest-priority finding** and the most likely cause of repeat-account hammering.
+
+**Symptom (documented 2026-06-16):** an `out_of_credits` / `org_level_disabled` account
+gets re-selected every ~90 s and re-429'd (~40 wasted 429 calls/min). These 429s carry
+**no authoritative reset header** (`anthropic-ratelimit-unified-overage-disabled-reason: out_of_credits`,
+`overage-status: rejected`, body "credit balance too low"), so the relay only applied the
+short transient cooldown and re-hammered the dead account.
+
+**The 2026-06-16 fix (`loadbalance-fix-20260616`) introduced:**
+- `_detectCreditExhaustion(status, headers, body)` in `claudeRelayService.js`
+- wired into all 3× 429 mark sites → passes `{creditExhausted}` to `markAccountRateLimited()`
+- which cools the account until its `claudeFiveHourResetsAt` (if future) else
+ `CREDIT_EXHAUSTION_COOLDOWN_SECONDS` (default **3600 s / 1 h**); transient no-reset 429s
+ keep the short 90 s.
+
+**Verified 2026-06-19: NONE of `_detectCreditExhaustion`, `creditExhausted`, or
+`CREDIT_EXHAUSTION_COOLDOWN_SECONDS` exist on disk.** The live image is `cve-fix-20260615`,
+which predates this fix. **Conclusion: the CVE rebuild on 2026-06-15 produced an image that
+never carried the credit-cooldown fix, OR a later restore reverted it.** The util-LB +
+concurrency-cap code survived (it lives in `unifiedClaudeScheduler.js`), but the
+credit-cooldown patch in `claudeRelayService.js` / `claudeAccountService.js` did **not**.
+
+**This must be re-applied before any other tuning** — it is the root cause of the
+out_of_credits 429-hammer and wastes pool capacity on a billing-dead account.
+
+---
+
+## 2. Tuning workstreams
+
+### 2.1 Re-apply credit-exhaustion cooldown (P0)
+1. Confirm the diff is recoverable: check the box for `~/crs` git stash / `.bak` copies of
+ `claudeRelayService.js` and `claudeAccountService.js`, and for the
+ `loadbalance-fix-20260616` image (`docker images | grep loadbalance`). If neither exists,
+ re-implement from the spec in §1.
+2. Re-add `_detectCreditExhaustion(status, headers, body)` — strict match on
+ `out_of_credits` / `org_level_disabled` / `overage-status: rejected` / "credit balance too low".
+3. Wire it into the 3 × 429 mark sites in `claudeRelayService.js` (non-stream, stream-error,
+ stream-end) → pass `{ creditExhausted: true }`.
+4. In `claudeAccountService.js markAccountRateLimited(..., options)`: when `creditExhausted`,
+ set cooldown to `max(now → claudeFiveHourResetsAt, CREDIT_EXHAUSTION_COOLDOWN_SECONDS)`.
+5. Fire the `CLAUDE_OAUTH_CREDIT_EXHAUSTED` webhook so billing-dead accounts surface in ops.
+6. Leave transient (no-reset) 429s on the short cooldown (see §2.2 for the value).
+
+### 2.2 Scheduler / cooldown knob tuning
+The cooldowns currently live as **code defaults with no compose override** (verified: the
+`COOLDOWN`/`CREDIT_EXHAUSTION`/`UTILIZATION` env names are absent from `docker-compose.yml`
+and `.env`). Promote them to env knobs so they are tunable without a rebuild:
+
+| Knob | Current default | Recommended | Rationale |
+|---|---|---|---|
+| `CLAUDE_DEFAULT_RATELIMIT_COOLDOWN_SECONDS` | 90 | **90** (keep) | transient no-reset 429s; 90 s is fine for genuine short ratelimits |
+| `CREDIT_EXHAUSTION_COOLDOWN_SECONDS` | 3600 | **3600** (keep) | billing exhaustion; do not re-probe for 1 h |
+| `CLAUDE_UTILIZATION_LOAD_BALANCING` | ON | **ON** | dynamic least-util picks; `=false` reverts to pure priority |
+
+Action: add these three to `docker-compose.yml` `environment:` (with `${VAR:-default}` form,
+mirroring the existing `CRS_OAUTH_REAUTH_COOLDOWN_MS=${...:-1800000}` pattern) and to `.env`.
+This makes future tuning a `docker compose up -d` (env reload), not a code rebuild.
+
+### 2.3 Account-pool load-balancing verification
+- Confirm `_sortPrimaryClaudeAccountsBeforeFallback` is wired and the `📊 Load-balance pick`
+ log line fires on new (non-sticky) session assignment.
+- Confirm session stickiness is resolved **before** the util sort (sticky sessions must not
+ be re-balanced mid-conversation).
+- Confirm `dynamic-priority.py` cron is running and not erroring. **Known pre-existing bug:**
+ `NoneType .get` traceback on accounts with no usage object — non-blocking but should be
+ guarded so the cron doesn't half-update priorities.
+- Verify the dead account: `pool-<USER>` (`a9f4806a…`) was the out_of_credits trigger;
+ it needs a **credit top-up** to truly rejoin (5h/7d windows are not the constraint).
+ Until topped up, the credit-cooldown (§2.1) keeps it parked instead of hammered.
+
+### 2.4 Concurrency caps + overload protection (today's storm)
+Today a **ConnectionRefused storm** hit under ~17 concurrent agents; recovered via keepalive.
+The relay restarted (container up only minutes at audit time). Hardening:
+
+1. **Right-size the util→concurrency tiers.** Current tiers (`8/5/3/2/1`) are generous at low
+ util. With ~10 accounts × 8 = 80 theoretical concurrent low-util streams, the bottleneck is
+ **not** per-account concurrency — it is the node process / socket layer dropping connections
+ under burst. Validate the tiers hold the fleet's real peak (≈17–25 concurrent) without the
+ process refusing connections.
+2. **Bound the proxy socket pool.** `PROXY_MAX_SOCKETS` / `PROXY_MAX_FREE_SOCKETS` are present
+ in compose but **unset** (default unbounded). Set explicit ceilings (e.g.
+ `PROXY_MAX_SOCKETS=256`, `PROXY_MAX_FREE_SOCKETS=32`) so a burst can't exhaust file
+ descriptors → `ECONNREFUSED`.
+3. **Confirm the 503 backpressure path returns cleanly** rather than dropping the socket.
+ When all accounts are at concurrency limit the scheduler throws
+ `CONSOLE_ACCOUNT_CONCURRENCY_FULL`; verify the relay turns that into a **503 with
+ `Retry-After`**, not a connection reset — clients should back off, not see `ConnectionRefused`.
+4. **Container resilience.** Ensure `restart: unless-stopped` on the relay service and a
+ `healthcheck` so Docker auto-recovers a wedged process. Confirm the node process file-descriptor
+ `ulimit` (`docker inspect` → `Ulimits`/host `LimitNOFILE`) is high enough (≥ 65536) for the
+ socket count above.
+5. **Fleet-side ceiling.** The durable fix for "17 concurrent heavy streams" is **also** to cap
+ concurrent fleet dispatch (the spawner / keepalive), so the relay is never asked to hold more
+ live streams than the pool's summed concurrency budget. Document the agreed cap with Sam
+ before raising relay limits.
+
+### 2.5 Session-routing health
+- **Auth var:** every routed session must use `ANTHROPIC_AUTH_TOKEN=cr_…` (Bearer), never
+ `ANTHROPIC_API_KEY` (format-validated client-side → "Invalid API key format / Please run /login").
+- **Key-validity probe (authoritative — the relay decides):** malformed POST to
+ `…:3005/api/v1/messages` with the `cr_` key → **HTTP 400 = key accepted**, **401 = stale/rejected**.
+- **Canonical key:** macOS keychain `crs-fleet-key` (== Mac overlay token). Doppler
+ `claude-ops/prd CRS_FLEET_KEY` has drifted stale before — if the box 401s, re-sync from keychain
+ and re-push Doppler, else the next Doppler→box push re-breaks it.
+- **Preflight gate:** `~/.claude/scripts/crs-overlay-preflight.sh` must accept
+ `ANTHROPIC_AUTH_TOKEN` (it previously only checked `CLAUDE_CODE_OAUTH_TOKEN` and blocked ALL
+ dispatch). Confirm it still passes (HTTP 400 = exit 0).
+- **`✅ Cleared 401 error count` in relay logs is a SUCCESS** (counter cleared after a 200), not an
+ error. Do not blindly grep-count `401`. Genuine failures are `🔒 Invalid API key attempt`.
+
+---
+
+## 3. Execution sequence (do in this order)
+
+> All commands run **on the FRA box** (`ssh fra`). `$RP` = `$(docker exec crs-claude-relay-1 printenv REDIS_PASSWORD)`.
+
+1. **Snapshot for rollback** (see §5) — back up `docker-compose.yml`, tag the running image, note current account state.
+2. **Re-apply credit-cooldown** (§2.1) on-disk in `~/crs/src/...`. Do NOT bulk find-replace — edit the 3 mark sites + the service method surgically; grep each site first.
+3. **Promote cooldown/util knobs to env** (§2.2) in `docker-compose.yml` + `.env`.
+4. **Set socket-pool ceilings + restart policy + ulimit** (§2.4).
+5. **Rebuild + restart relay only:**
+ ```bash
+ cd ~/crs && docker compose build claude-relay && \
+ docker compose up -d claude-relay
+ ```
+ (Redis is NOT touched — preserve queues/account state.)
+6. **Run §4 verification.** If any check fails, **roll back via §5** (must complete < 2 min).
+7. Tag the validated image (e.g. `tuned-20260619`) and pin it in compose so a future restore is deterministic.
+
+---
+
+## 4. Verification (post-deploy)
+
+Run all of these; the deploy is good only if every one passes.
+
+1. **Health:** `curl -s -o /dev/null -w '%{http_code}' 127.0.0.1:3005/health` → `200`.
+2. **Key accepted:** malformed POST to `…:3005/api/v1/messages` with the `cr_` key → `400` (not 401).
+3. **Real relay:** a `claude -p "reply: OK" --model haiku` through the overlay → returns a completion (proves end-to-end forward + auth).
+4. **Credit-cooldown active (the P0 check):** in relay logs, confirm an `out_of_credits` 429
+ produces a long cooldown, NOT a 90 s one — grep for the credit-exhaustion mark + the
+ `CLAUDE_OAUTH_CREDIT_EXHAUSTED` webhook. Then confirm the exhausted account is **not
+ re-selected** within the cooldown window:
+ ```bash
+ docker logs crs-claude-relay-1 --since 10m 2>&1 | grep -iE "credit|out_of_credits|cooldown|Load-balance pick"
+ ```
+ No account should appear in a `Load-balance pick` while it is credit-cooled.
+5. **No repeat-hammer:** over a 5-min window, no single `claude:account:<uuid>` should accumulate
+ more than a handful of 429s. (Pre-fix baseline was ~40/min on the dead account.)
+6. **Load-balancing live:** under real fleet load, `📊 Load-balance pick` shows picks spread
+ across multiple low-util accounts, not one account pinned to 100% while others sit at 0%.
+7. **Concurrency gating:** force burst load (or replay the 17-agent peak) and confirm the relay
+ returns `503` + `Retry-After` when saturated — **no `ECONNREFUSED` / ConnectionRefused** in
+ client logs.
+8. **Container auto-recovery:** `docker inspect crs-claude-relay-1 --format '{{.HostConfig.RestartPolicy.Name}}'`
+ → `unless-stopped`; healthcheck status `healthy`.
+9. **Cron clean:** `dynamic-priority.py` last run logged no `NoneType` traceback.
+
+---
+
+## 5. Rollback procedure (must complete in < 2 minutes — test it first)
+
+> Redis is never cleared — account state and queues survive a relay rollback.
+
+```bash
+cd ~/crs
+
+# 1. Restore the pre-tuning compose (taken in §3 step 1)
+cp docker-compose.yml.bak-pretune-<TS> docker-compose.yml
+
+# 2. Re-pin the last-known-good image (the running image at audit = cve-fix-20260615;
+# OR the validated prior tag if one exists)
+# Edit compose `image:` → weishaw/claude-relay-service:cve-fix-20260615
+# (compose has `build: .`, so to force the prebuilt image either comment out `build:`
+# for the rollback, or `docker compose up -d --no-build claude-relay`)
+
+# 3. Restart relay ONLY (Redis untouched)
+docker compose up -d --no-build claude-relay
+
+# 4. Verify (< 30 s)
+curl -s -o /dev/null -w 'health=%{http_code}\n' 127.0.0.1:3005/health # expect 200
+```
+
+**Manual park/unpark a misbehaving account** (faster than a full rollback if only one account is bad):
+```bash
+RP=$(docker exec crs-claude-relay-1 printenv REDIS_PASSWORD)
+docker exec crs-redis-1 redis-cli -a "$RP" --no-auth-warning \
+ hset claude:account:<uuid> schedulable false # true to re-enable
+```
+> Note: `schedulable=false` only stops **new** session assignment — existing session-sticky
+> requests drain over their window.
+
+**Pre-flight the rollback before the real deploy:** do steps 1–4 against the current image once,
+time it, confirm < 2 min and health 200, then proceed with the tuning deploy.
+
+---
+
+## 6. Constraints & safety (carried from the agent brief)
+
+- **No destructive bulk edits** on scheduler code — grep/inspect each site before editing.
+- **Preserve Redis data** — never `FLUSHDB`/clear queues mid-fix.
+- **Rollback < 2 min, tested first.**
+- **Fleet impact:** changes affect every agent using the relay — coordinate the concurrency-cap
+ and fleet-dispatch-ceiling numbers with Sam before deploying.
+- **Never print** the `cr_` key, OAuth tokens, AWS secrets, or `REDIS_PASSWORD`.
+
+---
+
+## 7. Open items to confirm with Sam before deploy
+
+1. Agreed **fleet concurrent-dispatch ceiling** (relay-side cap is moot if the fleet over-dispatches).
+2. Whether the `loadbalance-fix-20260616` image/diff is recoverable on the box, or the
+ credit-cooldown must be re-implemented from §1 spec.
+3. Top-up status of `pool-<USER>` (billing-dead account) — keep parked vs. top up vs. remove from pool.