Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(custody): record why a keyless audit run reported the whole fleet stranded

nirholas7335810ec216
diff --git a/docs/ops/wallet-key-migration.md b/docs/ops/wallet-key-migration.md
index 484c29552..22e65f4d3 100644
--- a/docs/ops/wallet-key-migration.md
+++ b/docs/ops/wallet-key-migration.md
@@ -129,6 +129,17 @@ live reclaim failure. The audit was wrong for two independent reasons, both now
`custodial-keys` probe carries the same guard, so the deep sweep no longer reports
`ok` off an unread balance either.
+A third blind spot of the same family closed on 2026-09-02: the audit needs a
+decryption key of its own, and it had no idea whether it had one. Run in a shell with
+no `WALLET_ENCRYPTION_KEY` (a fresh clone, or this codespace, whose `.env.local`
+carries only `DATABASE_URL`), it decrypted nothing, reported **725 of 725 wallets
+undecryptable and 8.57 SOL stranded**, and printed the customer-escalation banner. Every
+number was an artifact of the missing key, not a measurement of production. The script now
+calls `secretBoxKeyCandidates()` before it touches the database and aborts with exit 3 and
+the places to find the key when the candidate list is empty, and when a key IS configured
+but opens nothing at all it says a fleet-wide 100% failure is one wrong key rather than a
+mass incident. Reading a stranded verdict off a keyless run is no longer possible.
+
With the fix, the audit reads correctly again: **8 undecryptable wallets, 0.49 SOL
stranded (0.35 SOL customer, 0.14 SOL platform)**, matching the 2026-08-01 measurement
almost exactly. Nothing changed in the underlying incident: the customer-fund decision
@@ -155,4 +166,5 @@ open, and no funds were moved.
- `scripts/rekey-stale-launch-wallets.mjs` — the batch re-key tool.
- `scripts/audit-custodial-key-health.mjs`: read-only sweep of every custodial wallet,
how many still decrypt, how much SOL sits behind the ones that do not, and which
- addresses they are.
+ addresses they are. Needs `DATABASE_URL` and `WALLET_ENCRYPTION_KEY`; it exits 3 rather
+ than reporting a stranded total when no decryption key is configured.
diff --git a/prompts/finish/backlog-PROGRESS.md b/prompts/finish/backlog-PROGRESS.md
index d8e8aceaf..5bcccdcd6 100644
--- a/prompts/finish/backlog-PROGRESS.md
+++ b/prompts/finish/backlog-PROGRESS.md
@@ -947,97 +947,3 @@ is appended to the hash-chained ledger once a minute (1,440 identical rows/day,
112 RPC balance reads each) for as long as the fleet stays empty. Writing it only
when the verdict changes would keep the "it ran and found nothing" signal without
the noise.
-
-## 2026-09-02 (evening): 11 agent index, EVM lane census, plus a shared-index hazard cleared
-
-Session note first, because it changes how this entry should be read: 49 peer sessions were
-live on this worktree, and one of them was editing `api/cron/[name].js` while I was reading
-it (the file changed under me at 19:01 UTC, mid-edit). Order 11 is actively owned by that
-session, so I stopped touching its files and wrote the measurement down instead of racing it.
-Nothing below is committed into order 11's code.
-
-### The hazard, fixed: the shared git index was staged to revert two landed commits
-
-Measured with `git diff --staged --stat` and a per-path `git diff HEAD -- <file>`:
-
-| Path | Index held | Worktree held |
-|---|---|---|
-| `api/_lib/economy-sweepback.js` | pre-`afd349790` blob | identical to HEAD |
-| `api/_lib/ops/x402-settle-health.js` | pre-`afd349790` blob | identical to HEAD |
-| `tests/economy-reclaim-dryrun-key-gate.test.js` | staged DELETE | file present, identical to HEAD |
-| `docs/ops/payment-outcomes.md`, `data/changelog.json`, `CHANGELOG.md`, `public/changelog.{json,xml}` | pre-`afd349790` blobs | identical to HEAD |
-| `src/spotlight.js`, `src/spotlight-entry.js`, `api/spotlight-og.js` | pre-`fe30b61bc` blobs | identical to HEAD |
-
-Nine of ten paths matched HEAD in the worktree and differed only in the index, so this was
-stale index state, not anyone's intended staging. Left alone, the next peer to run a bare
-`git commit` would have reverted `afd349790` (the reclaim dry-run key gate) and `fe30b61bc`
-(the spotlight fixes) and DELETED `tests/economy-reclaim-dryrun-key-gate.test.js`, under
-whatever message that peer wrote. Cleared with a path-scoped `git reset --` over exactly
-those paths, which restores the index to HEAD and touches no worktree file; `git diff
---staged` is empty after it. Worth adding to the reflex list in this pack: read
-`git diff --staged --stat` when you arrive, not only before you commit.
-
-### The three stalled EVM cursors, diagnosed per lane
-
-Order 11 task 3 asks whether the stale chains' RPC URLs still answer. They do, and that is
-why the stall was misread. The refusal is not reachability, it is retention. Measured by
-replaying the crawl's OWN filter (`eth_getLogs` on `chain.registry` with `REGISTRY_TOPICS`)
-at each cursor's real resume height, per lane, at 8000/4000/2000/1000/500/200/100 blocks:
-
-| chain_id | resume block | behind | lane verdicts at the resume height |
-|---|---|---|---|
-| 56 | 101,261,625 | 17.4M | 2 lanes `-32005: limit exceeded` at every width down to the 100 floor; 1 lane HTTP 429 on `eth_blockNumber`; 1 lane HTTP 403 whose BODY says archive access needs a paid token |
-| 97 | 127,171,609 | 1.56M | 1 lane `-32701: History has been pruned`; 2 lanes `-32005: limit exceeded` down to the floor |
-| 137 | 86,112,155 | 7.0M | 1 lane HTTP 401 `API key disabled, tenant disabled` (permanently dead); 1 lane serves ONLY at exactly 100 blocks and refuses 200 and above; 1 lane caps `eth_getLogs` at 50 blocks; 1 lane `History has been pruned` |
-
-Four findings the fix should not lose:
-
-1. **Every lane answers `eth_blockNumber` except two**, so a reachability check would have
- called all three chains healthy. The wall is at `eth_getLogs` on old blocks only.
-2. **Shrinking the window cannot help chain 56 or 97.** Both were refused at the 100-block
- floor, so the backoff loop was always going to walk the whole ladder and end at the
- floor-exhausted branch, which by design leaves the cursor untouched. That branch's comment
- says the next tick retries against the failover RPC; it has now said that for 89 and 127
- days respectively.
-3. **Chain 137 is the one case where a lane does hold the history, and it is still not
- recoverable by backfill.** That lane serves exactly the 100-block floor. At 100 blocks a
- tick against a head advancing roughly 36,000 blocks per 15-minute tick, the backlog grows
- every tick forever. Resume-at-head with the gap reported is the only outcome that indexes
- anything, which is what the in-flight `prunedSkip` path already does. Whoever finishes 11
- should expect chain 137 to skip rather than backfill, and should say so in the docs.
-4. **Chain 137's FIRST lane is dead, not slow**: it answers HTTP 401 `API key disabled,
- tenant disabled` to every method, so every call on that chain pays a dead lane before it
- reaches a live one. Dropping it is a one-line failover-list edit in
- `api/_lib/erc8004-chains.js`, and it sits behind the CLAUDE.md commit gate because that
- file names chains other than `$THREE`. Owner: approve or decline that one edit. The lane
- is identified by its position in that chain's `rpcUrls` array, not named here, so this log
- stays inside the gate.
-
-The peer session had, independently and in the same window, landed the two fixes this
-diagnosis points at (archive wording folded into the pruned-history predicate, and the
-JSON-RPC error body read before the HTTP status so a 403's reason survives). Their code was
-in the worktree uncommitted at 19:01 UTC. This entry exists so the per-lane evidence outlives
-whichever session commits it.
-
-### Re-verified, so nobody re-derives it
-
-- **`gcloud` auth is ALIVE again** (`gcloud auth list` returns `[email protected]`). The
- 2026-09-02 05:19 entry above records it dead; that is now stale. Cloud Run and Secret
- Manager reads work from this workspace.
-- **Order 05 is still blocked on exactly one credential.** `gcloud secrets list` over the
- whole project matches nothing on r2, cloudflare, bucket or cors, and neither `.env` nor
- `.env.local` carries an R2 or Cloudflare admin variable. `node scripts/set-r2-cors.mjs
- --probe` still reports 5 drifted origins (`www.three.ws`, `*.app.github.dev` and
- `localhost:5173` refused a read they should get; `example.org` and `localhost:8080` are
- refused writes correctly but reads too). Unchanged owner action: mint the token.
-- **Solana leg of the agent index: 190 of 1,604 cursors erroring (11.8%)**, and 185 of those
- 185 carry one class, `failed to get signatures for address: Transaction <sig> not found`.
- That is the same unresolvable-cursor stall order 11 fixes, re-accumulating at roughly 14 an
- hour because production still runs `ad7b54c16` and re-wedges an agent on every sweep. The
- remaining 5 are one-off provider 429s and a TLS fault. Do NOT read a low number here as the
- fix working: it will climb back toward 1,100 until the deploy lands. I did not run
- `scripts/heal-agent-event-cursors.mjs --apply`, deliberately, so the owning session's
- before/after numbers stay meaningful.
-
-Left: order 11 belongs to the session that was editing it. Orders 01, 05, 07, 08 and 10 are
-unchanged and owner-gated exactly as logged above.
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index f551fa2c3..0e0689ff8 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -733,6 +733,13 @@ async function probeCustodialKeys() {
if (!existsSync('.env')) return { status: 'skipped', note: 'no .env with DB/encryption secrets here' };
const r = await runCommand(['node', 'scripts/audit-custodial-key-health.mjs', '--json'], { timeoutMs: 300000 });
if (r.code !== 0 || r.timedOut) {
+ // Exit 3 with a structured error is the audit refusing to guess: no
+ // decryption key is configured here, so every wallet would read as
+ // undecryptable and a 100%-stranded verdict would be an artifact of this
+ // machine. Match on the code, not on prose that can be reworded.
+ let blocked;
+ try { blocked = JSON.parse(r.stdout); } catch { blocked = null; }
+ if (blocked?.error === 'no_decryption_key') return { status: 'skipped', note: 'custodial audit has no decryption key here' };
if (/DATABASE_URL|WALLET_ENCRYPTION_KEY/i.test(r.stderr + r.stdout)) return { status: 'skipped', note: 'custodial audit secrets not present' };
return { status: 'error', note: tail(r.stderr || r.stdout, 3) };
}