Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(owner-actions): re-measure the deploy gap at 5 commits, down from 112

nirholas015ab1705a0b
diff --git a/prompts/finish/_context/production-100-OWNER-ACTIONS.md b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
index b460ee0ce..f59360db9 100644
--- a/prompts/finish/_context/production-100-OWNER-ACTIONS.md
+++ b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
@@ -13,7 +13,7 @@ on it; some resolve themselves as other work lands.
| # | Action | Size | Unblocks | Detail lives in |
|---|---|---|---|---|
-| 1 | Approve the production deploy (or say "get production working", which is itself the approval). **The outage this row was written for is CLEARED.** Re-measured 2026-09-09 from the live origin: production serves `880bdcef8` on revision `three-ws-api-00420-ljh`, built 2026-09-08 19:05 UTC, and the `ERR_MODULE_NOT_FOUND` on `/app/services/home-relay/src/token.js` is gone: `/api/healthz` answers **HTTP 200** and `/api/home/status` and `/api/home/connections` answer 401 (auth required) rather than 500. What remains is ordinary staleness, 182 commits behind `main`, which is a routine cadence question and no longer a platform-health emergency. The 2026-09-04 text of this row, which described 96 commits and a 500ing healthz, is superseded. **Re-measured 2026-10-08 17:05 UTC:** production serves `fe2a8b24f` on `three-ws-api-00479-d7z`, **146 commits behind `main`, 90 of them fixes** (security, payouts, SSRF and XSS among them). Shipping them clears four live findings at once: the ring sweep's fee falls back to the treasury instead of the dry sponsor (`a9c996e72`, about 30 failed sweeps an hour since 2026-10-03), an EVM testnet identity-index cursor frozen since 2026-09-11 (`21fa42bb5`), two pages and their two docs pages advertised in the sitemap but answering 404 (added in `16f9d07f4`), and `/tty` holding its animation until the end for any client that accepts compression (`03b4f1189`). One migration must go with it: `20261008150000_revenue_direct_settle_backfill.sql` (pending; it matches 0 rows today, so applying it is safe, and the code half is what keeps the payout cron from paying a direct-to-seller sale twice). Commands, in order: `npm run db:status`, `npm run db:migrate`, `npm run deploy:gcp:full`. **Re-measured 2026-10-09 03:05 UTC by the triage sweep:** production still serves image `fe2a8b24f` (now on revision `three-ws-api-00480-mws`, an env-only roll that raised `DB_RETENTION_HIGH_WATER_MB` to 16384), **172 commits behind `main`**, and the `evolve` branch holds 85 more awaiting the runner's merge, six of them this sweep's production fixes: the leak scanner's lease and budget (`0abce6d69`, which ends about eight overlapping 600 to 900 s scans that were burning the Solana RPC quota), `-32020` RPC failover (`4b51a179c`), static 412/416 instead of 500 (`e01e43b3b`), the news knowledge write fix (`833407d4f`, 120 lost stories a day), and the reconstruct sweep's plan-limit close (`85d3b2034`). `20261008150000_revenue_direct_settle_backfill.sql` has since been applied; `npm run db:status` now lists two pending, `20261008190000_oauth_refresh_user_client_index.sql` and `20261008200000_mcp_client_analytics.sql` (rows 27 and 28). The 8 sitemap pages still answering 404 (`/grok`, `/grok-skill.md`, `/docs/grok`, `/docs/grok-bot`, `/dextools`, `/docs/dextools`, `/chart-companion`, `/docs/chart-companion`) are all deploy lag. **Re-measured 2026-10-10 20:06 UTC by the triage sweep:** production serves `5af439bc0` on `three-ws-api-00487-7mt`, built 2026-10-10 05:01 UTC, **45 commits behind `main`** (down from 172 on 2026-10-09; most of the gap already shipped), and the `evolve` branch holds 89 more. All 869 advertised pages, every cron (125) and the fleet (48 services) check out clean; the only pending migrations are rows 27 and 28's two files. **Re-measured 2026-10-11 04:16 UTC by the deep triage sweep:** production serves `3503381b1` on `three-ws-api-00491-fbl`, **112 commits behind `main`** (up from 45 on 2026-10-10). One new page-lag symptom from the gap: `/docs/tutorials/grok-bot-recipes` (added in `71c673f99`, after the running revision) answers 200 with `/docs`'s content instead of its own; this is deploy lag, not a routing bug (verified: `71c673f99` is not an ancestor of `3503381b1`). The fleet (48 services), 125 crons and DB are otherwise clean; the only pending migrations remain rows 27 and 28's two files. | one yes | Shipping whatever has landed on `main` since 2026-09-08 | [01-ship-readiness.md](../901-production-100-01-ship-readiness.md) |
+| 1 | Approve the production deploy (or say "get production working", which is itself the approval). **The outage this row was written for is CLEARED.** Re-measured 2026-09-09 from the live origin: production serves `880bdcef8` on revision `three-ws-api-00420-ljh`, built 2026-09-08 19:05 UTC, and the `ERR_MODULE_NOT_FOUND` on `/app/services/home-relay/src/token.js` is gone: `/api/healthz` answers **HTTP 200** and `/api/home/status` and `/api/home/connections` answer 401 (auth required) rather than 500. What remains is ordinary staleness, 182 commits behind `main`, which is a routine cadence question and no longer a platform-health emergency. The 2026-09-04 text of this row, which described 96 commits and a 500ing healthz, is superseded. **Re-measured 2026-10-08 17:05 UTC:** production serves `fe2a8b24f` on `three-ws-api-00479-d7z`, **146 commits behind `main`, 90 of them fixes** (security, payouts, SSRF and XSS among them). Shipping them clears four live findings at once: the ring sweep's fee falls back to the treasury instead of the dry sponsor (`a9c996e72`, about 30 failed sweeps an hour since 2026-10-03), an EVM testnet identity-index cursor frozen since 2026-09-11 (`21fa42bb5`), two pages and their two docs pages advertised in the sitemap but answering 404 (added in `16f9d07f4`), and `/tty` holding its animation until the end for any client that accepts compression (`03b4f1189`). One migration must go with it: `20261008150000_revenue_direct_settle_backfill.sql` (pending; it matches 0 rows today, so applying it is safe, and the code half is what keeps the payout cron from paying a direct-to-seller sale twice). Commands, in order: `npm run db:status`, `npm run db:migrate`, `npm run deploy:gcp:full`. **Re-measured 2026-10-09 03:05 UTC by the triage sweep:** production still serves image `fe2a8b24f` (now on revision `three-ws-api-00480-mws`, an env-only roll that raised `DB_RETENTION_HIGH_WATER_MB` to 16384), **172 commits behind `main`**, and the `evolve` branch holds 85 more awaiting the runner's merge, six of them this sweep's production fixes: the leak scanner's lease and budget (`0abce6d69`, which ends about eight overlapping 600 to 900 s scans that were burning the Solana RPC quota), `-32020` RPC failover (`4b51a179c`), static 412/416 instead of 500 (`e01e43b3b`), the news knowledge write fix (`833407d4f`, 120 lost stories a day), and the reconstruct sweep's plan-limit close (`85d3b2034`). `20261008150000_revenue_direct_settle_backfill.sql` has since been applied; `npm run db:status` now lists two pending, `20261008190000_oauth_refresh_user_client_index.sql` and `20261008200000_mcp_client_analytics.sql` (rows 27 and 28). The 8 sitemap pages still answering 404 (`/grok`, `/grok-skill.md`, `/docs/grok`, `/docs/grok-bot`, `/dextools`, `/docs/dextools`, `/chart-companion`, `/docs/chart-companion`) are all deploy lag. **Re-measured 2026-10-10 20:06 UTC by the triage sweep:** production serves `5af439bc0` on `three-ws-api-00487-7mt`, built 2026-10-10 05:01 UTC, **45 commits behind `main`** (down from 172 on 2026-10-09; most of the gap already shipped), and the `evolve` branch holds 89 more. All 869 advertised pages, every cron (125) and the fleet (48 services) check out clean; the only pending migrations are rows 27 and 28's two files. **Re-measured 2026-10-11 04:16 UTC by the deep triage sweep:** production serves `3503381b1` on `three-ws-api-00491-fbl`, **112 commits behind `main`** (up from 45 on 2026-10-10). One new page-lag symptom from the gap: `/docs/tutorials/grok-bot-recipes` (added in `71c673f99`, after the running revision) answers 200 with `/docs`'s content instead of its own; this is deploy lag, not a routing bug (verified: `71c673f99` is not an ancestor of `3503381b1`). The fleet (48 services), 125 crons and DB are otherwise clean; the only pending migrations remain rows 27 and 28's two files. **Re-measured 2026-10-11 12:32 UTC by the triage-lane sweep:** production serves `66a5e93bd` on `three-ws-api-00494-wh5`, built 2026-10-11 07:22 UTC, **only 5 commits behind `main`** (down sharply from 112 on 2026-10-11 04:16; the earlier large gap has shipped). The 5 outstanding commits are the desktop OS/shell/widgets feature set and the campus world update (`e33f529d8`, `33b5490c5`, `a48caf816`, `9c47f3ceb`, `86366a55e`). Rows 27 and 28's migrations are now applied; `npm run db:status` shows 2 new pending files, `20261011220000_agent_mention_policy.sql` (tracked by row 35) and `20261011230000_x402_ring_scan_checked_at.sql` (tracked by row 24's narrative), both additive. This sweep also committed two root-cause fixes to `main`-bound code that are NOT yet in this 5-commit gap: `65d6c4be8` (clink-fun MCP stream timeout misclassified as investigate) and `2f2fedf99` (a Pollinations 402 was putting the paid Replicate/TRELLIS lane into cooldown, contributing to `forge_generation: down`); the second one fixes a live production defect, so it is worth prioritizing in the next deploy. | one yes | Shipping whatever has landed on `main` since 2026-09-08, now including the forge-cooldown fix | [01-ship-readiness.md](../901-production-100-01-ship-readiness.md) |
| 2 | Fund the settle sponsor wallet `WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`. Healthz reports `x402_settle` down with `cause: sponsor_floor` (settle 5.9%, 3 of 51 paid attempts over 3 hours, 329 `no_solana_accept`) and the sniper worker out of capital (0.2307 SOL to refill its wallets, funding master holds 0.0250 SOL); no config change moves either. Re-read 2026-09-04 08:21 UTC against mainnet: the sponsor holds **0.001507661 SOL** (1,507,661 lamports), still below the settle floor, so the Solana accept stays withdrawn from every 402 challenge. Two months of re-measurement have only ever moved this number down; nothing on this machine can move it up. **Re-measured 2026-10-08 17:05 UTC:** the sponsor holds **651,828 lamports (0.000652 SOL)**, below rent-exempt (890,880), and healthz reads `x402_settle` down at 0.0% (0 of 32 paid attempts in 3 hours, 58 settles refused at the fee-wallet floor). Because the sponsor cannot co-sign, every 402 challenge advertises the Solana accept without `feePayer`, and the ring skips it: 151 `missing_fee_payer` skips in 24 hours on the MCP streaming health probe alone. Last successful settle: 2026-10-03 19:24 UTC. **Row 1's deploy alone restores the ring, no funding needed** (diagnosed 2026-10-08 by read-only simulation): the treasury pays its own sweep fee (5,001 lamports; it holds 0.060 SOL, about 11,800 sweeps) and moves its stranded 2.587 USDC back to the ring payer, which holds 341 atomics today, and ring settles are self-pay. What still needs this wallet: the sponsor-mode callers (the loop's inline entries and the MCP streaming health probe) and the facilitator's `sponsor_fee_unfunded` verify rejects. Clearing the 2,000,000-lamport floor takes at least **1,348,172 lamports**. **Re-measured 2026-10-09 03:05 UTC:** still **651,828 lamports**, unchanged; healthz reads `x402_settle` down at 0.0% (0 of 39 paid attempts in 3 hours, 37 `sweep_broadcast_failed`, 53 settles refused at the fee-wallet floor). The sweep failures are the ones row 1's deploy clears (`a9c996e72`). | ~1 SOL | x402 settle recovery, backlog 01 retirement, sniper | [../backlog/PROGRESS.md](backlog-PROGRESS.md), 2026-08-02 and 2026-09-01 entries |
| 3 | Decide the stranded customer funds: customers hold about 0.35 SOL behind the retired encryption key and cannot withdraw. Credit (recommended), contact first, or write off. The brief now exists with the measurement, the impossibility argument, the cost of each option and the exact commands: **[docs/ops/stranded-wallets.md](../../docs/ops/stranded-wallets.md)**. Measured 2026-09-02 from production's own records: the platform side is exactly two bot wallets holding 0.142875505 SOL (the reclaim cron has failed on them ~35,700 times each, most recently today), and one customer agent is confirmed by a `wallet_key_retired` withdraw failure (`GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm`, 0.250001 SOL). **Both customer wallets are now named** (2026-09-04): the keyed audit was run against production, so this row no longer waits on anything. `GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm` holds 0.250001 SOL and `HPL1LfuTdYDwtzJDzsnrmR2ngrrQwLTQyxJszCC4DHsN` holds 0.100001 SOL, 0.350002 SOL across exactly two wallet-auth accounts. Nine of 735 custodial wallets are sealed; the other five customer ones hold nothing. The brief carries the full table. Every input the decision needs is measured. | decision | Closing the support obligation | [docs/ops/stranded-wallets.md](../../docs/ops/stranded-wallets.md), [../backlog/PROGRESS.md](backlog-PROGRESS.md) |
| 4 | Rotate or explicitly accept three credentials, now that the storage half is done. **The config work shipped 2026-09-02**: the economy master key and 58 other plaintext credentials are Secret Manager references on `three-ws-api-00407-m7c`, zero plaintext credentials remain, healthz verified no worse and the master's signing path verified live. What is left is a judgment only the owner can make. (a) The master key was readable in the service config for some window, so decide whether to rotate it or accept that exposure, knowing who held project viewer access in that period; the exact owner-gated commands are in the runbook. (b) `WALLET_ENCRYPTION_KEY` and `JWT_SECRET` hold the SAME value on production, which passes the dedicated-key guard in `secret-box.js` while defeating what it guards; changing it is a re-key with real stranding risk, so it is not an agent's call. (c) `X402_SEED_SOLANA_SECRET_BASE58` holds the same value as `LAUNCHER_MASTER_SECRET_KEY_B64` though `scripts/wire-master-wallet.mjs` assigns those slots to different wallets; decide which is correct. | decision (+ one fund move if rotating) | 03 is otherwise closed | [docs/ops/wallet-key-migration.md](../../docs/ops/wallet-key-migration.md), 2026-09-02 section |