Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
chore(owner-actions): record the 2026-10-24 triage sweep's deploy gap and findings
nirholas4c1d1811fe51
diff --git a/prompts/finish/_context/production-100-OWNER-ACTIONS.md b/prompts/finish/_context/production-100-OWNER-ACTIONS.mdindex cbb0f27db..34942399a 100644--- a/prompts/finish/_context/production-100-OWNER-ACTIONS.md+++ b/prompts/finish/_context/production-100-OWNER-ACTIONS.md@@ -9,11 +9,11 @@ it and, if that unblocks a work order, say which one in the owning pack's `produRows carry the date they were added so staleness is visible. Re-verify a row before actingon it; some resolve themselves as other work lands.-## Open rows (2026-10-23 (triage-lane deep sweep): row 1 re-measured, deploy gap escalated to 535 commits against `threews/main` (up from 2 on 2026-10-14/15, production unchanged at `45c768d07`); the `/api/chat` 503 storm traced to deploy lag already fixed on HEAD, no new row; one `investigate` finding (`/api/pump/coin` 502, pump.fun Cloudflare anti-bot challenge) given a new self-healing signature (`pump-coin-lookup-upstream-502`) and fixed on `evolve` (`72b600f1c`); rows 2, 3, 4, 5, 6, 19, 23, 24, 36 and 47 re-confirmed unchanged; 2026-10-15 (triage-lane deep sweep, second pass): row 1 re-measured, deploy gap still 2 commits against `threews/main`, unchanged from 2026-10-14; two `investigate` findings reclassified into `KNOWN_SIGNATURES` and fixed on `evolve` (NVIDIA NVCF text-to-3d flakiness folded into the existing `three-ws-ai-providers-unavailable-5xx` signature, and a new self-healing `pump-trending-upstreams-502` signature for a confirmed transient double-upstream outage); row 63's duplicate number (PAY_VERDICT_HMAC_KEY, added same day by a concurrent agent) renumbered to 67; rows 24 and 36 re-confirmed unchanged against live `/api/healthz`; 2026-10-15 (order 522-integrate-23 close-out): row 66 added for the x402 outbound-receipts resolved-columns migration (`20261015110000_x402_outbound_resolved.sql`), not yet an active bug since zero outbound-spend rows exist today; 2026-10-15 (triage-lane deep sweep): row 65 added for the payment-session policy-upgrades migration (`20261015100000_payment_session_policy_upgrades.sql`), needed before the `655b5a491` code deploy to avoid a `42703 undefined_column` 500 on every payment-session create/execute call; row 1 re-measured, deploy gap still 2 commits against `threews/main`, unchanged from 2026-10-14; 2026-10-15 (order 518-integrate-19 close-out): row 63 added for setting a dedicated `PAY_VERDICT_HMAC_KEY` on the production Cloud Run service; not an active bug, the signed-verdict feature it covers already works today on a fallback secret, this just gives it its own rotatable key; 2026-10-14 (order 506-integrate-07 close-out): row 62 added for deploying the new `services/pump-claims` Cloud Run service and applying its `pump_fee_claims` migration to production; 2026-10-14 (order 503-integrate-04 close-out): row 61 added for pushing and archiving 7 retired repos (`three-ui`, `crypto-market-data`, `crypto-market-data-ts`, `crypto-data-aggregator`, `memescope-monday-directory`, `robinhood-chain-alert-bot`, `robinhood-chain-trading-bot`); 2026-10-14 (order 501-integrate-02 close-out): row 60 added for approving the `@nirholas/pump-sdk` bump to `^1.36.0` in `packages/avatar-agent-mcp/package.json`; 2026-10-14 (order 500-integrate-01 close-out): row 59 added for pushing every staged standalone mirror now that sync-standalone-repos.mjs stages instead of force-pushing; 2026-10-14 (order 404 second pass): correcting the 2026-10-11 entry below, the `x-mention-reply.js` `chainFor()` fix was not actually carried by merge commit `d77b89178`; checking out that commit's own `evolve` parent showed `chainFor()` still importing `resolveAgentBrain`/`modelChain` (the tool-calling chain shape, no `buildBody`), the exact bug row 34 flagged, still live. The real fix is this session's `evolve` commit `7ba9f9749` ("fix(x-mentions): give public replies a buildBody-shaped provider chain"), switching `chainFor()` to `llm.js`'s own `providerChain()` plus `loadOwnerGrokKey`/`agentDefaultModel`/`MODEL_CATALOG`; re-verified now genuinely passing with the same command the earlier entry cited (`RUN_LLM_TESTS=1 npx vitest run tests/x-mention-reply.llm.test.js`, 16/16 `source: 'model'`, no `buildBody` TypeError). Row 34's premise (the `evolve`/`main` divergence) is still correctly closed, just not for the reason originally stated; no row reopened. Order 060's matching status paragraph corrected the same way. 2026-10-13 (order 095 close-out): row 58 added for the Home Assistant integration's standalone repository, release, and the hacs/default and home-assistant/brands pull requests; 2026-10-13 (order 094 close-out): row 57 added for the WordPress.org/Framer/Webflow account signups and submissions for the three <agent-3d> site-builder embeds; 2026-10-13 (order 092 close-out): row 55 added for the VRoid Hub avatar-import OAuth app registration and migration; 2026-10-13 (triage-lane sweep, fifth pass): row 53 added for the Birdeye compute-units cap behind `pump-price-history-upstreams-502` (confirmed live, a billing/plan decision only); row 1 re-measured, deploy gap still 0 against the pushed `threews/main` ref and the 8 page-404/wrong-canonical findings this pass confirmed as deploy lag; row 47 re-measured, still a no-op (`COINGECKO_API_KEY` still absent); the previously-unclassified `/dist-lib/agent-3d.js` and `/agent-3d/latest/agent-3d.js` 500s ("Response size was too large") are the same bug row 48 already tracks, fixed on `evolve` in `f939d5ef1`, confirmed still green (`npx vitest run tests/lib-bundle-size.test.js tests/whisper-stt.test.js`, 16/16) and awaiting the pending deploy, no new row; 2026-10-13 (triage-lane sweep, fourth pass): row 1 re-measured, deploy gap still 0 against the pushed `threews/main` ref; the `ccv-ai-providers-unavailable-503` finding's Groq/OpenRouter/Vertex angles confirmed already covered by rows 6 and 20, no new row added; 2026-10-13 (order 087 close-out): row 51 added for the LiveKit and Pipecat avatar integrations, both built and verified end to end this session, with the PyPI publishes and the two upstream pull requests left as owner actions; 2026-10-13 (order 085 close-out): row 50 added for publishing the CC0 bulk animation library to production, and the Adobe Mixamo permission letter draft is ready to send as owner order 938; 2026-10-12 (order 084 close-out): row 49 added for the Tripo signed webhook's three linked steps (apply the `forge_webhook_deliveries` migration, set `TRIPO_WEBHOOK_SECRET`, register the dashboard webhook), not an active bug since the secret is unset and polling answers every Tripo job today; 2026-10-12 18:10 UTC (triage-lane sweep, third pass): row 48 added for the dist-lib CDN embed size fix (previously-unclassified `/dist-lib/agent-3d.js` 500s, root-caused and fixed on `evolve`, pure code change awaiting the next deploy); 2026-10-12 17:54 UTC (triage-lane sweep, second pass): row 1's same-day "11 behind" figure corrected to the real deploy gap of 0 (querying `/api/version` live, not `/api/healthz`, which has no commit field), row 47 added for CoinGecko's keyless tier being rate-limited (`COINGECKO_API_KEY` already absent from the service, so the finding's own pre-approved env fix is a no-op); 2026-10-12 (triage-lane sweep): row 1 re-measured against the canonical `threews` remote, 11 commits behind `main` (down from 28, local stale ref corrected), row 45 added for the HF image-provider (fal-ai/nscale) credit exhaustion, a separate pool from row 6; 2026-10-11 (best3d order 070 close-out): rows 43 and 44 added for the `model-partcrafter` and `model-holopart` first deploys; 2026-10-11 (triage-lane sweep): row 42 added for the MCP client-sessions nullable-column migration, an active production bug; row 1 re-measured, 28 commits behind `main` (up from 5); 2026-10-11 (best3d order 069 close-out): rows 40 and 41 added for the `model-rig-nonhuman` and `model-anigen` first deploys; 2026-10-11 (best3d order 068 close-out): row 39 added for the `model-pixal3d` first deploy; 2026-10-11 (best3d order 067 retry): row 38 added for the `model-trellis2` first deploy; 2026-10-11 (order 404 close-out): row 34 deleted, its premise cleared by merge commit `d77b89178` (the `evolve`/`main` reconciliation) which also carried the `x-mention-reply.js` `chainFor()` fix row 34 called out, verified live with `RUN_LLM_TESTS=1 npx vitest run tests/x-mention-reply.llm.test.js` (16/16 `source: 'model'`, no `buildBody` TypeError); 2026-10-11 (scout lane): row 36 added for the Cloudflare R2 flapping degraded/down pattern; 2026-10-11 (even later): row 24 re-measured again after order 400's ring-leak-scan tiering/lock fix shipped on `evolve`; 2026-10-11 (later): row 35 added for the agent mention-policy migration; 2026-10-11 04:16 UTC: rows 1 and 24 re-measured by the deep triage sweep; row 34 added for the evolve/main branch divergence; 2026-10-10: row 33 added for the xAI MCP e2e key; rows 1, 19 and 24 re-measured by the triage sweep; 2026-10-09: rows 1, 2 and 24 re-measured by the triage sweep, row 31 added for the GPU worker redeploy and row 32 for database table retention; row 30 added for the Grok Bot CLI setup rerun; 2026-10-08: row 26 added for the coin-gated MCP catalog regeneration; rows 1 and 2 re-measured by the triage sweep, row 24 added for the paid Solana RPC credits, row 25 added for the MCP connector OAuth deploy; 2026-09-30: row 23 added for the $THREE receiver wallets; 2026-09-29: row 19 re-measured after the agent-sniper deploy; 2026-09-09: row 19 added, and row 1 corrected because its outage premise no longer holds; rows 1, 2, 3, 9, 13, 15 and 16 re-measured 2026-09-04; the former row 18 was cleared that day and its number is not reused; the 2026-09-02 and 2026-08-09 passes are folded in below)+## Open rows (2026-10-24 (triage-lane deep sweep): row 1 re-measured, the 535-commit gap shipped and `three-ws-api` is now fully current against `threews/main` (production `660b5ad97`); row 69 added for `remesh-service` running a revision from 2026-07-16 that predates the already-shipped Blender-teardown-crash fix (`59d9aa9cd`), causing live `signal 11` crashes; a previously-unclassified `/api/chat` 503 finding (21 occurrences in 6h, traced to the vertex-gemini anchor hitting the already-tracked GCP billing hold, row 20) given a new owner-class signature (`chat-fallback-chain-exhausted-503`) and documented, fixed on `evolve` (`f6f116552`); the `x402-ring-leak-scan` cron's 100%-failure 120s timeout (missing `maxDurationMs` wiring into `wrapCron`) root-caused and fixed on `evolve` (`5814b5585`); the `forge-finalize` cron's serial-sweep timeout fixed on `evolve` (`376f160be`); `model-triposg`'s logged tracebacks confirmed to be its existing by-design opaque-error handling for caller-supplied image URLs (an expired third-party presigned URL, not a code defect), no action needed; 2026-10-23 (triage-lane deep sweep): row 1 re-measured, deploy gap escalated to 535 commits against `threews/main` (up from 2 on 2026-10-14/15, production unchanged at `45c768d07`); the `/api/chat` 503 storm traced to deploy lag already fixed on HEAD, no new row; one `investigate` finding (`/api/pump/coin` 502, pump.fun Cloudflare anti-bot challenge) given a new self-healing signature (`pump-coin-lookup-upstream-502`) and fixed on `evolve` (`72b600f1c`); rows 2, 3, 4, 5, 6, 19, 23, 24, 36 and 47 re-confirmed unchanged; 2026-10-15 (triage-lane deep sweep, second pass): row 1 re-measured, deploy gap still 2 commits against `threews/main`, unchanged from 2026-10-14; two `investigate` findings reclassified into `KNOWN_SIGNATURES` and fixed on `evolve` (NVIDIA NVCF text-to-3d flakiness folded into the existing `three-ws-ai-providers-unavailable-5xx` signature, and a new self-healing `pump-trending-upstreams-502` signature for a confirmed transient double-upstream outage); row 63's duplicate number (PAY_VERDICT_HMAC_KEY, added same day by a concurrent agent) renumbered to 67; rows 24 and 36 re-confirmed unchanged against live `/api/healthz`; 2026-10-15 (order 522-integrate-23 close-out): row 66 added for the x402 outbound-receipts resolved-columns migration (`20261015110000_x402_outbound_resolved.sql`), not yet an active bug since zero outbound-spend rows exist today; 2026-10-15 (triage-lane deep sweep): row 65 added for the payment-session policy-upgrades migration (`20261015100000_payment_session_policy_upgrades.sql`), needed before the `655b5a491` code deploy to avoid a `42703 undefined_column` 500 on every payment-session create/execute call; row 1 re-measured, deploy gap still 2 commits against `threews/main`, unchanged from 2026-10-14; 2026-10-15 (order 518-integrate-19 close-out): row 63 added for setting a dedicated `PAY_VERDICT_HMAC_KEY` on the production Cloud Run service; not an active bug, the signed-verdict feature it covers already works today on a fallback secret, this just gives it its own rotatable key; 2026-10-14 (order 506-integrate-07 close-out): row 62 added for deploying the new `services/pump-claims` Cloud Run service and applying its `pump_fee_claims` migration to production; 2026-10-14 (order 503-integrate-04 close-out): row 61 added for pushing and archiving 7 retired repos (`three-ui`, `crypto-market-data`, `crypto-market-data-ts`, `crypto-data-aggregator`, `memescope-monday-directory`, `robinhood-chain-alert-bot`, `robinhood-chain-trading-bot`); 2026-10-14 (order 501-integrate-02 close-out): row 60 added for approving the `@nirholas/pump-sdk` bump to `^1.36.0` in `packages/avatar-agent-mcp/package.json`; 2026-10-14 (order 500-integrate-01 close-out): row 59 added for pushing every staged standalone mirror now that sync-standalone-repos.mjs stages instead of force-pushing; 2026-10-14 (order 404 second pass): correcting the 2026-10-11 entry below, the `x-mention-reply.js` `chainFor()` fix was not actually carried by merge commit `d77b89178`; checking out that commit's own `evolve` parent showed `chainFor()` still importing `resolveAgentBrain`/`modelChain` (the tool-calling chain shape, no `buildBody`), the exact bug row 34 flagged, still live. The real fix is this session's `evolve` commit `7ba9f9749` ("fix(x-mentions): give public replies a buildBody-shaped provider chain"), switching `chainFor()` to `llm.js`'s own `providerChain()` plus `loadOwnerGrokKey`/`agentDefaultModel`/`MODEL_CATALOG`; re-verified now genuinely passing with the same command the earlier entry cited (`RUN_LLM_TESTS=1 npx vitest run tests/x-mention-reply.llm.test.js`, 16/16 `source: 'model'`, no `buildBody` TypeError). Row 34's premise (the `evolve`/`main` divergence) is still correctly closed, just not for the reason originally stated; no row reopened. Order 060's matching status paragraph corrected the same way. 2026-10-13 (order 095 close-out): row 58 added for the Home Assistant integration's standalone repository, release, and the hacs/default and home-assistant/brands pull requests; 2026-10-13 (order 094 close-out): row 57 added for the WordPress.org/Framer/Webflow account signups and submissions for the three <agent-3d> site-builder embeds; 2026-10-13 (order 092 close-out): row 55 added for the VRoid Hub avatar-import OAuth app registration and migration; 2026-10-13 (triage-lane sweep, fifth pass): row 53 added for the Birdeye compute-units cap behind `pump-price-history-upstreams-502` (confirmed live, a billing/plan decision only); row 1 re-measured, deploy gap still 0 against the pushed `threews/main` ref and the 8 page-404/wrong-canonical findings this pass confirmed as deploy lag; row 47 re-measured, still a no-op (`COINGECKO_API_KEY` still absent); the previously-unclassified `/dist-lib/agent-3d.js` and `/agent-3d/latest/agent-3d.js` 500s ("Response size was too large") are the same bug row 48 already tracks, fixed on `evolve` in `f939d5ef1`, confirmed still green (`npx vitest run tests/lib-bundle-size.test.js tests/whisper-stt.test.js`, 16/16) and awaiting the pending deploy, no new row; 2026-10-13 (triage-lane sweep, fourth pass): row 1 re-measured, deploy gap still 0 against the pushed `threews/main` ref; the `ccv-ai-providers-unavailable-503` finding's Groq/OpenRouter/Vertex angles confirmed already covered by rows 6 and 20, no new row added; 2026-10-13 (order 087 close-out): row 51 added for the LiveKit and Pipecat avatar integrations, both built and verified end to end this session, with the PyPI publishes and the two upstream pull requests left as owner actions; 2026-10-13 (order 085 close-out): row 50 added for publishing the CC0 bulk animation library to production, and the Adobe Mixamo permission letter draft is ready to send as owner order 938; 2026-10-12 (order 084 close-out): row 49 added for the Tripo signed webhook's three linked steps (apply the `forge_webhook_deliveries` migration, set `TRIPO_WEBHOOK_SECRET`, register the dashboard webhook), not an active bug since the secret is unset and polling answers every Tripo job today; 2026-10-12 18:10 UTC (triage-lane sweep, third pass): row 48 added for the dist-lib CDN embed size fix (previously-unclassified `/dist-lib/agent-3d.js` 500s, root-caused and fixed on `evolve`, pure code change awaiting the next deploy); 2026-10-12 17:54 UTC (triage-lane sweep, second pass): row 1's same-day "11 behind" figure corrected to the real deploy gap of 0 (querying `/api/version` live, not `/api/healthz`, which has no commit field), row 47 added for CoinGecko's keyless tier being rate-limited (`COINGECKO_API_KEY` already absent from the service, so the finding's own pre-approved env fix is a no-op); 2026-10-12 (triage-lane sweep): row 1 re-measured against the canonical `threews` remote, 11 commits behind `main` (down from 28, local stale ref corrected), row 45 added for the HF image-provider (fal-ai/nscale) credit exhaustion, a separate pool from row 6; 2026-10-11 (best3d order 070 close-out): rows 43 and 44 added for the `model-partcrafter` and `model-holopart` first deploys; 2026-10-11 (triage-lane sweep): row 42 added for the MCP client-sessions nullable-column migration, an active production bug; row 1 re-measured, 28 commits behind `main` (up from 5); 2026-10-11 (best3d order 069 close-out): rows 40 and 41 added for the `model-rig-nonhuman` and `model-anigen` first deploys; 2026-10-11 (best3d order 068 close-out): row 39 added for the `model-pixal3d` first deploy; 2026-10-11 (best3d order 067 retry): row 38 added for the `model-trellis2` first deploy; 2026-10-11 (order 404 close-out): row 34 deleted, its premise cleared by merge commit `d77b89178` (the `evolve`/`main` reconciliation) which also carried the `x-mention-reply.js` `chainFor()` fix row 34 called out, verified live with `RUN_LLM_TESTS=1 npx vitest run tests/x-mention-reply.llm.test.js` (16/16 `source: 'model'`, no `buildBody` TypeError); 2026-10-11 (scout lane): row 36 added for the Cloudflare R2 flapping degraded/down pattern; 2026-10-11 (even later): row 24 re-measured again after order 400's ring-leak-scan tiering/lock fix shipped on `evolve`; 2026-10-11 (later): row 35 added for the agent mention-policy migration; 2026-10-11 04:16 UTC: rows 1 and 24 re-measured by the deep triage sweep; row 34 added for the evolve/main branch divergence; 2026-10-10: row 33 added for the xAI MCP e2e key; rows 1, 19 and 24 re-measured by the triage sweep; 2026-10-09: rows 1, 2 and 24 re-measured by the triage sweep, row 31 added for the GPU worker redeploy and row 32 for database table retention; row 30 added for the Grok Bot CLI setup rerun; 2026-10-08: row 26 added for the coin-gated MCP catalog regeneration; rows 1 and 2 re-measured by the triage sweep, row 24 added for the paid Solana RPC credits, row 25 added for the MCP connector OAuth deploy; 2026-09-30: row 23 added for the $THREE receiver wallets; 2026-09-29: row 19 re-measured after the agent-sniper deploy; 2026-09-09: row 19 added, and row 1 corrected because its outage premise no longer holds; rows 1, 2, 3, 9, 13, 15 and 16 re-measured 2026-09-04; the former row 18 was cleared that day and its number is not reused; the 2026-09-02 and 2026-08-09 passes are folded in below)| # | Action | Size | Unblocks | Detail lives in ||---|---|---|---|---|-| 1 | Approve the production deploy (or say "get production working", which is itself the approval). **The outage this row was written for is CLEARED.** Re-measured 2026-09-09 from the live origin: production serves `880bdcef8` on revision `three-ws-api-00420-ljh`, built 2026-09-08 19:05 UTC, and the `ERR_MODULE_NOT_FOUND` on `/app/services/home-relay/src/token.js` is gone: `/api/healthz` answers **HTTP 200** and `/api/home/status` and `/api/home/connections` answer 401 (auth required) rather than 500. What remains is ordinary staleness, 182 commits behind `main`, which is a routine cadence question and no longer a platform-health emergency. The 2026-09-04 text of this row, which described 96 commits and a 500ing healthz, is superseded. **Re-measured 2026-10-08 17:05 UTC:** production serves `fe2a8b24f` on `three-ws-api-00479-d7z`, **146 commits behind `main`, 90 of them fixes** (security, payouts, SSRF and XSS among them). Shipping them clears four live findings at once: the ring sweep's fee falls back to the treasury instead of the dry sponsor (`a9c996e72`, about 30 failed sweeps an hour since 2026-10-03), an EVM testnet identity-index cursor frozen since 2026-09-11 (`21fa42bb5`), two pages and their two docs pages advertised in the sitemap but answering 404 (added in `16f9d07f4`), and `/tty` holding its animation until the end for any client that accepts compression (`03b4f1189`). One migration must go with it: `20261008150000_revenue_direct_settle_backfill.sql` (pending; it matches 0 rows today, so applying it is safe, and the code half is what keeps the payout cron from paying a direct-to-seller sale twice). Commands, in order: `npm run db:status`, `npm run db:migrate`, `npm run deploy:gcp:full`. **Re-measured 2026-10-09 03:05 UTC by the triage sweep:** production still serves image `fe2a8b24f` (now on revision `three-ws-api-00480-mws`, an env-only roll that raised `DB_RETENTION_HIGH_WATER_MB` to 16384), **172 commits behind `main`**, and the `evolve` branch holds 85 more awaiting the runner's merge, six of them this sweep's production fixes: the leak scanner's lease and budget (`0abce6d69`, which ends about eight overlapping 600 to 900 s scans that were burning the Solana RPC quota), `-32020` RPC failover (`4b51a179c`), static 412/416 instead of 500 (`e01e43b3b`), the news knowledge write fix (`833407d4f`, 120 lost stories a day), and the reconstruct sweep's plan-limit close (`85d3b2034`). `20261008150000_revenue_direct_settle_backfill.sql` has since been applied; `npm run db:status` now lists two pending, `20261008190000_oauth_refresh_user_client_index.sql` and `20261008200000_mcp_client_analytics.sql` (rows 27 and 28). The 8 sitemap pages still answering 404 (`/grok`, `/grok-skill.md`, `/docs/grok`, `/docs/grok-bot`, `/dextools`, `/docs/dextools`, `/chart-companion`, `/docs/chart-companion`) are all deploy lag. **Re-measured 2026-10-10 20:06 UTC by the triage sweep:** production serves `5af439bc0` on `three-ws-api-00487-7mt`, built 2026-10-10 05:01 UTC, **45 commits behind `main`** (down from 172 on 2026-10-09; most of the gap already shipped), and the `evolve` branch holds 89 more. All 869 advertised pages, every cron (125) and the fleet (48 services) check out clean; the only pending migrations are rows 27 and 28's two files. **Re-measured 2026-10-11 04:16 UTC by the deep triage sweep:** production serves `3503381b1` on `three-ws-api-00491-fbl`, **112 commits behind `main`** (up from 45 on 2026-10-10). One new page-lag symptom from the gap: `/docs/tutorials/grok-bot-recipes` (added in `71c673f99`, after the running revision) answers 200 with `/docs`'s content instead of its own; this is deploy lag, not a routing bug (verified: `71c673f99` is not an ancestor of `3503381b1`). The fleet (48 services), 125 crons and DB are otherwise clean; the only pending migrations remain rows 27 and 28's two files. **Re-measured 2026-10-11 12:32 UTC by the triage-lane sweep:** production serves `66a5e93bd` on `three-ws-api-00494-wh5`, built 2026-10-11 07:22 UTC, **only 5 commits behind `main`** (down sharply from 112 on 2026-10-11 04:16; the earlier large gap has shipped). The 5 outstanding commits are the desktop OS/shell/widgets feature set and the campus world update (`e33f529d8`, `33b5490c5`, `a48caf816`, `9c47f3ceb`, `86366a55e`). Rows 27 and 28's migrations are now applied; `npm run db:status` shows 2 new pending files, `20261011220000_agent_mention_policy.sql` (tracked by row 35) and `20261011230000_x402_ring_scan_checked_at.sql` (tracked by row 24's narrative), both additive. This sweep also committed two root-cause fixes to `main`-bound code that are NOT yet in this 5-commit gap: `65d6c4be8` (clink-fun MCP stream timeout misclassified as investigate) and `2f2fedf99` (a Pollinations 402 was putting the paid Replicate/TRELLIS lane into cooldown, contributing to `forge_generation: down`); the second one fixes a live production defect, so it is worth prioritizing in the next deploy. **Re-measured 2026-10-11 21:12 UTC by the triage-lane sweep:** production still serves `66a5e93bd` on `three-ws-api-00494-wh5`, **28 commits behind `main`** (`main` now at `9eda7992d`), up from 5 at 12:32 UTC because the desktop OS/shell/widgets/campus set from that measurement has been joined by new work: the x-content partner-spotlight feature (registry, daily scheduler, published IBM and AWS partner articles, the `/partners/benefits` page) and five dependabot bumps (`js-yaml`/`markdownlint-cli2`, `nanoid`, `maplibre-gl`, `torch` in `workers/texture`, `brace-expansion`, `source-map-js`, `moment`, `fast-uri`). `2f2fedf99` (the forge-cooldown fix) is still only on `evolve`, not yet on `main`; it remains worth prioritizing. The deep sweep found exactly one new pending migration, now tracked as row 42; rows 24 and 35's migrations are applied. The `/docs/desktop-widgets` 404 and `/docs/tutorials/grok-bot-recipes` wrong-canonical findings are confirmed deploy lag again (their catalog commits are not ancestors of `66a5e93bd`), not new routing bugs. **Re-measured 2026-10-12 by the triage-lane sweep, against the canonical `threews` remote (the local `main` ref in the `evolve` worktree was stale):** production (`/api/healthz`'s own `commitShort`) serves `c266e2591`, **11 commits behind `threews/main`** at `80e5e35f7`, down sharply from 28. The outstanding set is the Dissolve feature (a GPU particle-storm viewer for any 3D model) with its docs and page-catalog regeneration, a security-hardening batch (`8a56a44ab`: auth, wallet policy, SDP, vault, skill sandbox, MCP bridge URL guard, all with regression tests), a worker fix (`315e96f96`: block internal-host navigation in the screen pool, load text2motion weights with `weights_only`), an accessibility fix (`f772c8708`: icon-only button tooltips, gallery/create/header name-to-text matching), the Grid desktop world (`762269248`, `f35d07d93`, `a60d72dad`) and a new UI-lint tool (`568162d8d`). None of this sweep's own fixes (the Vertex/HF image-lane circuit breaker, the `forge-image-ladder-exhausted-502` and widened `r2-downstream-5xx` triage signatures) are in this 11-commit gap either; they are further back on `evolve`, behind the runner's usual fast-forward-to-`main` cadence. **Corrected 2026-10-12 17:54 UTC by the triage-lane sweep: the same-day "11 commits behind" figure above was a misread.** Querying the live `/api/version` endpoint directly (not `/api/healthz`, which carries no commit field) shows production still serves `771cfef01` on `three-ws-api-00496-78g`, built 2026-10-12 07:08 UTC. A fresh `git fetch threews` puts `threews/main` at `80e5e35f7`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds with exactly 1 commit between them, so the deployed revision is 1 commit AHEAD of the canonical remote `main`, not behind it. **The real deploy gap (commits on `main` not yet in production) is 0.** The 242 commits the deep sweep reports as "ahead" are `evolve`-branch commits (this worktree's local `main`, one commit ahead of pushed `threews/main`) not yet pushed, including this sweep's own `5afb045cf` (widened the `r2-downstream-5xx` triage signature to cover `/api/render/animate`, `/api/render/glb`, `/api/render/avatar-clip` and `/api/animations/clips/<id>`, all previously-unmatched symptoms of the same R2 flap row 36 already tracks); they are not a production staleness problem. **Re-measured 2026-10-13 by the triage-lane sweep:** production still serves `771cfef01` on `three-ws-api-00496-78g`. A fresh `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds, so the deployed revision is still exactly 1 commit AHEAD of the canonical remote `main`. **The real deploy gap remains 0.** The deep sweep's "283 commits ahead" figure is `evolve`-branch backlog (unpushed local commits, including `f939d5ef1`, the dist-lib CDN embed size fix documented in row 48) awaiting the runner's fast-forward, not a production staleness problem. This sweep found no new code-level defect to fix: the dist-lib bug is already root-caused, fixed and tested on `evolve`; the Solana-adjacent degraded subsystems (`agent_index`, `rpc_lanes`, `helius`, `sniper`, `three_token_rail`) and the `ccv-ai-providers-unavailable-503` finding (Groq's 200k tokens/day cap, the GCP billing hold, and the revoked OpenRouter key) are all pre-existing owner items already tracked in rows 23, 24, 19, 2, 20 and 6; none needed a new row. **Re-measured 2026-10-13 (triage-lane sweep, fifth pass):** production still serves `771cfef01` on `three-ws-api-00497-2ss`. `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` still succeeds, so the deployed revision remains exactly 1 commit AHEAD of `main`. **The real deploy gap stays 0.** The deep sweep's "291 commits ahead" figure is `evolve`-branch backlog (unpushed, includes `f939d5ef1` / row 48 and this pass's new row 53) awaiting the runner's fast-forward, not production staleness. The 8 sitemap-advertised pages answering 404 or wrong-canonical this pass (`/discord-activity`, `/docs/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/open-source-credits`, `/docs/voice-framework-integrations`, `/open-source`, `/docs/tutorials/grok-bot-recipes`) are all confirmed deploy lag (`git merge-base --is-ancestor <add-commit> 771cfef01` fails for each); no routing bug. **Re-measured 2026-10-14 by the triage-lane sweep (production health sweep order):** production serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews` puts `threews/main` at `43a5dba21`, and `git log --oneline 45c768d07..threews/main` lists exactly 2 commits (`43a5dba21`, an x-content proofs refresh, and `3480497ed`, the R2-NotEntitled storage-outage failover fix): **the real deploy gap is 2 commits, both routine, neither urgent.** The deep sweep's 16 failing-page findings (`/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/google-adk`, `/docs/open-source-credits`, `/docs/shopify-app`, `/docs/video-to-motion`, `/docs/voice-framework-integrations`, `/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion` and their docs pages, plus the `/docs/tutorials/grok-bot-recipes` wrong-canonical) are confirmed NOT part of this 2-commit gap: none of their routes exist in `threews/main`'s `data/pages.json` at all (checked directly), so they are unpushed `evolve`-branch backlog (334 commits ahead of `threews/main` in this worktree), the same already-tracked rows 49, 52, 54 and 55 cover (Tripo webhook, Discord Activity, Shopify app, VRoid Hub import). The 5 pending migrations `npm run db:status` lists (`mcp_client_sessions_client_nullable`, `forge_webhook_deliveries`, `discord_activity_rooms`, `shopify_app`, `vroid_hub_source`) are the same set tracked by rows 42, 49, 52, 54 and 55; none applied, consistent with those rows' guidance. The lone unclassified `investigate` finding, a single `HTTP 502 GET /api/v1/coins` on `cryptocurrency-cv` with no matching app-level error, has no isolated root cause distinguishable from the 300-plus already-tracked `ccv-ai-providers-unavailable-503` hits on the same service in the same window; not elevated to a new row. This sweep's one code-level fix, a missing diagnostic log on `/api/agents/:id/alpha/read`'s generic LLM-failure 502 branch (not in the deep sweep's own findings list, found by log analysis; test-first, committed on `evolve`), is orthogonal to this row and does not change the gap count. **Re-measured 2026-10-15 by the triage-lane deep sweep:** production still serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews` puts `threews/main` at `43a5dba21`, and `git log --oneline 45c768d07..threews/main` still lists exactly the same 2 commits (`43a5dba21`, `3480497ed`): **the real deploy gap remains 2 commits, unchanged from 2026-10-14.** The deep sweep's own "422 commits ahead" figure is `evolve`-branch backlog in this worktree, not production staleness; the same caveat row 1 has carried since 2026-10-12. This pass's 9 failing-page findings (`/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion`, `/docs/forge-workflows`, `/docs/standalone-repos`, `/docs/team-chat`, `/docs/tutorials/grok-bot-recipes`) are confirmed deploy lag the same way (none of their `data/pages.json` entries predate the deployed build; `git merge-base --is-ancestor` fails for each against `45c768d07`), consistent with rows 49/52/54/55/58. The `deep-db-migrations` probe's 8 pending files match rows already open above; none applied this pass, per the self-unblock playbook's no-dry-run warning on `npm run db:migrate`. Two `investigate` findings got code fixes instead of new rows: `/api/v1/ai/text-to-3d` 502s are the same NVIDIA NVCF free-tier capacity flakiness `three-ws-ai-providers-unavailable-5xx` already covers (rows 6/20), its matcher widened to include that route; `/api/pump/trending` and `/api/v1/pump/trending` 502s are a genuine but self-healing double-upstream outage (Birdeye and the pump.fun board both down in the same window, stale-cache fallback also exhausted), given a new self-healing signature so the monitor stops flagging it as `investigate`. Both fixes are committed on `evolve` (`90628d426`) with passing tests, pending the normal fast-forward to `main`. Rows 24 (`rpc_lanes: degraded, both paid lanes exhausted`) and 36 (`object_storage: down`, R2 "Please enable R2 through the Cloudflare Dashboard") re-confirmed unchanged against live `/api/healthz`; no new rows, no edits needed to either. **Re-measured 2026-10-23 by the triage-lane deep sweep:** production still serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews main` followed by `git log --oneline 45c768d07..threews/main | wc -l` returns **535**, a sharp escalation from the 2-commit gap recorded 2026-10-14/15: the deploy window simply widened while nothing shipped, not a new regression. The `deep-db-migrations` probe lists 11 pending files (`20261011220000_agent_mention_policy.sql`, `20261011230000_x402_ring_scan_checked_at.sql`, `20261015010000_mcp_client_sessions_client_nullable.sql`, `20261015040000_forge_webhook_deliveries.sql`, `20261015050000_discord_activity_rooms.sql`, `20261015060000_shopify_app.sql`, `20261015070000_vroid_hub_source.sql`, `20261015080000_pump_fee_claims.sql`, `20261015090000_agent_coin_earnings_timeline.sql`, `20261015100000_payment_session_policy_upgrades.sql`, `20261015110000_x402_outbound_resolved.sql`), none applied, per the self-unblock playbook's no-dry-run warning on `npm run db:migrate`. This pass's `/api/chat` 503 storm (groq's `FALLBACK_SIBLINGS` pointing at the dead model id `llama-3.3-70b-versatile`) is the same deploy-lag story, confirmed by `git show 45c768d07:api/chat.js`: HEAD already carries the fix (`groq: ['openai/gpt-oss-20b']`), so no new code change is needed, it ships with this row. One `investigate` finding got a code fix instead of a new row: a lone `/api/pump/coin` 502 traced via app logs to pump.fun's own Cloudflare anti-bot challenge page for that mint, with no last-good cache yet for it, same shape as the existing `pump-trending-upstreams-502` signature; given its own self-healing signature (`pump-coin-lookup-upstream-502`) so the monitor stops flagging it, committed on `evolve` (`72b600f1c`) with a passing test. The `deep-pages` probe now lists 20 failing pages, not the 9 from 2026-10-14/15: the original 9 (`/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion`, `/docs/forge-workflows`, `/docs/standalone-repos`, `/docs/team-chat`, `/docs/tutorials/grok-bot-recipes`) plus 11 newly-advertised pages added to `data/pages.json` since then (`/.well-known/skill.md`, `/apps/shopify`, `/discord-activity`, `/docs/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/google-adk`, `/docs/open-source-credits`, `/docs/shopify-app`, `/docs/video-to-motion`, `/docs/voice-framework-integrations`). Checked all 20 individually with `git merge-base --is-ancestor <adding-commit> 45c768d07`: every one of the 20 fails the check, confirming all 20 are deploy lag from the growing gap, zero are a routing bug. Rows 2, 3, 4, 5, 6, 19, 23, 24, 36 and 47 re-confirmed unchanged against live `/api/healthz` and service env; no new rows, no edits needed to any of them. | one yes | Shipping whatever has landed on `main` since 2026-09-08, now including the security-hardening batch and the forge image-lane breaker fix | [01-ship-readiness.md](../901-production-100-01-ship-readiness.md) |+| 1 | Approve the production deploy (or say "get production working", which is itself the approval). **The outage this row was written for is CLEARED.** Re-measured 2026-09-09 from the live origin: production serves `880bdcef8` on revision `three-ws-api-00420-ljh`, built 2026-09-08 19:05 UTC, and the `ERR_MODULE_NOT_FOUND` on `/app/services/home-relay/src/token.js` is gone: `/api/healthz` answers **HTTP 200** and `/api/home/status` and `/api/home/connections` answer 401 (auth required) rather than 500. What remains is ordinary staleness, 182 commits behind `main`, which is a routine cadence question and no longer a platform-health emergency. The 2026-09-04 text of this row, which described 96 commits and a 500ing healthz, is superseded. **Re-measured 2026-10-08 17:05 UTC:** production serves `fe2a8b24f` on `three-ws-api-00479-d7z`, **146 commits behind `main`, 90 of them fixes** (security, payouts, SSRF and XSS among them). Shipping them clears four live findings at once: the ring sweep's fee falls back to the treasury instead of the dry sponsor (`a9c996e72`, about 30 failed sweeps an hour since 2026-10-03), an EVM testnet identity-index cursor frozen since 2026-09-11 (`21fa42bb5`), two pages and their two docs pages advertised in the sitemap but answering 404 (added in `16f9d07f4`), and `/tty` holding its animation until the end for any client that accepts compression (`03b4f1189`). One migration must go with it: `20261008150000_revenue_direct_settle_backfill.sql` (pending; it matches 0 rows today, so applying it is safe, and the code half is what keeps the payout cron from paying a direct-to-seller sale twice). Commands, in order: `npm run db:status`, `npm run db:migrate`, `npm run deploy:gcp:full`. **Re-measured 2026-10-09 03:05 UTC by the triage sweep:** production still serves image `fe2a8b24f` (now on revision `three-ws-api-00480-mws`, an env-only roll that raised `DB_RETENTION_HIGH_WATER_MB` to 16384), **172 commits behind `main`**, and the `evolve` branch holds 85 more awaiting the runner's merge, six of them this sweep's production fixes: the leak scanner's lease and budget (`0abce6d69`, which ends about eight overlapping 600 to 900 s scans that were burning the Solana RPC quota), `-32020` RPC failover (`4b51a179c`), static 412/416 instead of 500 (`e01e43b3b`), the news knowledge write fix (`833407d4f`, 120 lost stories a day), and the reconstruct sweep's plan-limit close (`85d3b2034`). `20261008150000_revenue_direct_settle_backfill.sql` has since been applied; `npm run db:status` now lists two pending, `20261008190000_oauth_refresh_user_client_index.sql` and `20261008200000_mcp_client_analytics.sql` (rows 27 and 28). The 8 sitemap pages still answering 404 (`/grok`, `/grok-skill.md`, `/docs/grok`, `/docs/grok-bot`, `/dextools`, `/docs/dextools`, `/chart-companion`, `/docs/chart-companion`) are all deploy lag. **Re-measured 2026-10-10 20:06 UTC by the triage sweep:** production serves `5af439bc0` on `three-ws-api-00487-7mt`, built 2026-10-10 05:01 UTC, **45 commits behind `main`** (down from 172 on 2026-10-09; most of the gap already shipped), and the `evolve` branch holds 89 more. All 869 advertised pages, every cron (125) and the fleet (48 services) check out clean; the only pending migrations are rows 27 and 28's two files. **Re-measured 2026-10-11 04:16 UTC by the deep triage sweep:** production serves `3503381b1` on `three-ws-api-00491-fbl`, **112 commits behind `main`** (up from 45 on 2026-10-10). One new page-lag symptom from the gap: `/docs/tutorials/grok-bot-recipes` (added in `71c673f99`, after the running revision) answers 200 with `/docs`'s content instead of its own; this is deploy lag, not a routing bug (verified: `71c673f99` is not an ancestor of `3503381b1`). The fleet (48 services), 125 crons and DB are otherwise clean; the only pending migrations remain rows 27 and 28's two files. **Re-measured 2026-10-11 12:32 UTC by the triage-lane sweep:** production serves `66a5e93bd` on `three-ws-api-00494-wh5`, built 2026-10-11 07:22 UTC, **only 5 commits behind `main`** (down sharply from 112 on 2026-10-11 04:16; the earlier large gap has shipped). The 5 outstanding commits are the desktop OS/shell/widgets feature set and the campus world update (`e33f529d8`, `33b5490c5`, `a48caf816`, `9c47f3ceb`, `86366a55e`). Rows 27 and 28's migrations are now applied; `npm run db:status` shows 2 new pending files, `20261011220000_agent_mention_policy.sql` (tracked by row 35) and `20261011230000_x402_ring_scan_checked_at.sql` (tracked by row 24's narrative), both additive. This sweep also committed two root-cause fixes to `main`-bound code that are NOT yet in this 5-commit gap: `65d6c4be8` (clink-fun MCP stream timeout misclassified as investigate) and `2f2fedf99` (a Pollinations 402 was putting the paid Replicate/TRELLIS lane into cooldown, contributing to `forge_generation: down`); the second one fixes a live production defect, so it is worth prioritizing in the next deploy. **Re-measured 2026-10-11 21:12 UTC by the triage-lane sweep:** production still serves `66a5e93bd` on `three-ws-api-00494-wh5`, **28 commits behind `main`** (`main` now at `9eda7992d`), up from 5 at 12:32 UTC because the desktop OS/shell/widgets/campus set from that measurement has been joined by new work: the x-content partner-spotlight feature (registry, daily scheduler, published IBM and AWS partner articles, the `/partners/benefits` page) and five dependabot bumps (`js-yaml`/`markdownlint-cli2`, `nanoid`, `maplibre-gl`, `torch` in `workers/texture`, `brace-expansion`, `source-map-js`, `moment`, `fast-uri`). `2f2fedf99` (the forge-cooldown fix) is still only on `evolve`, not yet on `main`; it remains worth prioritizing. The deep sweep found exactly one new pending migration, now tracked as row 42; rows 24 and 35's migrations are applied. The `/docs/desktop-widgets` 404 and `/docs/tutorials/grok-bot-recipes` wrong-canonical findings are confirmed deploy lag again (their catalog commits are not ancestors of `66a5e93bd`), not new routing bugs. **Re-measured 2026-10-12 by the triage-lane sweep, against the canonical `threews` remote (the local `main` ref in the `evolve` worktree was stale):** production (`/api/healthz`'s own `commitShort`) serves `c266e2591`, **11 commits behind `threews/main`** at `80e5e35f7`, down sharply from 28. The outstanding set is the Dissolve feature (a GPU particle-storm viewer for any 3D model) with its docs and page-catalog regeneration, a security-hardening batch (`8a56a44ab`: auth, wallet policy, SDP, vault, skill sandbox, MCP bridge URL guard, all with regression tests), a worker fix (`315e96f96`: block internal-host navigation in the screen pool, load text2motion weights with `weights_only`), an accessibility fix (`f772c8708`: icon-only button tooltips, gallery/create/header name-to-text matching), the Grid desktop world (`762269248`, `f35d07d93`, `a60d72dad`) and a new UI-lint tool (`568162d8d`). None of this sweep's own fixes (the Vertex/HF image-lane circuit breaker, the `forge-image-ladder-exhausted-502` and widened `r2-downstream-5xx` triage signatures) are in this 11-commit gap either; they are further back on `evolve`, behind the runner's usual fast-forward-to-`main` cadence. **Corrected 2026-10-12 17:54 UTC by the triage-lane sweep: the same-day "11 commits behind" figure above was a misread.** Querying the live `/api/version` endpoint directly (not `/api/healthz`, which carries no commit field) shows production still serves `771cfef01` on `three-ws-api-00496-78g`, built 2026-10-12 07:08 UTC. A fresh `git fetch threews` puts `threews/main` at `80e5e35f7`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds with exactly 1 commit between them, so the deployed revision is 1 commit AHEAD of the canonical remote `main`, not behind it. **The real deploy gap (commits on `main` not yet in production) is 0.** The 242 commits the deep sweep reports as "ahead" are `evolve`-branch commits (this worktree's local `main`, one commit ahead of pushed `threews/main`) not yet pushed, including this sweep's own `5afb045cf` (widened the `r2-downstream-5xx` triage signature to cover `/api/render/animate`, `/api/render/glb`, `/api/render/avatar-clip` and `/api/animations/clips/<id>`, all previously-unmatched symptoms of the same R2 flap row 36 already tracks); they are not a production staleness problem. **Re-measured 2026-10-13 by the triage-lane sweep:** production still serves `771cfef01` on `three-ws-api-00496-78g`. A fresh `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds, so the deployed revision is still exactly 1 commit AHEAD of the canonical remote `main`. **The real deploy gap remains 0.** The deep sweep's "283 commits ahead" figure is `evolve`-branch backlog (unpushed local commits, including `f939d5ef1`, the dist-lib CDN embed size fix documented in row 48) awaiting the runner's fast-forward, not a production staleness problem. This sweep found no new code-level defect to fix: the dist-lib bug is already root-caused, fixed and tested on `evolve`; the Solana-adjacent degraded subsystems (`agent_index`, `rpc_lanes`, `helius`, `sniper`, `three_token_rail`) and the `ccv-ai-providers-unavailable-503` finding (Groq's 200k tokens/day cap, the GCP billing hold, and the revoked OpenRouter key) are all pre-existing owner items already tracked in rows 23, 24, 19, 2, 20 and 6; none needed a new row. **Re-measured 2026-10-13 (triage-lane sweep, fifth pass):** production still serves `771cfef01` on `three-ws-api-00497-2ss`. `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` still succeeds, so the deployed revision remains exactly 1 commit AHEAD of `main`. **The real deploy gap stays 0.** The deep sweep's "291 commits ahead" figure is `evolve`-branch backlog (unpushed, includes `f939d5ef1` / row 48 and this pass's new row 53) awaiting the runner's fast-forward, not production staleness. The 8 sitemap-advertised pages answering 404 or wrong-canonical this pass (`/discord-activity`, `/docs/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/open-source-credits`, `/docs/voice-framework-integrations`, `/open-source`, `/docs/tutorials/grok-bot-recipes`) are all confirmed deploy lag (`git merge-base --is-ancestor <add-commit> 771cfef01` fails for each); no routing bug. **Re-measured 2026-10-14 by the triage-lane sweep (production health sweep order):** production serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews` puts `threews/main` at `43a5dba21`, and `git log --oneline 45c768d07..threews/main` lists exactly 2 commits (`43a5dba21`, an x-content proofs refresh, and `3480497ed`, the R2-NotEntitled storage-outage failover fix): **the real deploy gap is 2 commits, both routine, neither urgent.** The deep sweep's 16 failing-page findings (`/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/google-adk`, `/docs/open-source-credits`, `/docs/shopify-app`, `/docs/video-to-motion`, `/docs/voice-framework-integrations`, `/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion` and their docs pages, plus the `/docs/tutorials/grok-bot-recipes` wrong-canonical) are confirmed NOT part of this 2-commit gap: none of their routes exist in `threews/main`'s `data/pages.json` at all (checked directly), so they are unpushed `evolve`-branch backlog (334 commits ahead of `threews/main` in this worktree), the same already-tracked rows 49, 52, 54 and 55 cover (Tripo webhook, Discord Activity, Shopify app, VRoid Hub import). The 5 pending migrations `npm run db:status` lists (`mcp_client_sessions_client_nullable`, `forge_webhook_deliveries`, `discord_activity_rooms`, `shopify_app`, `vroid_hub_source`) are the same set tracked by rows 42, 49, 52, 54 and 55; none applied, consistent with those rows' guidance. The lone unclassified `investigate` finding, a single `HTTP 502 GET /api/v1/coins` on `cryptocurrency-cv` with no matching app-level error, has no isolated root cause distinguishable from the 300-plus already-tracked `ccv-ai-providers-unavailable-503` hits on the same service in the same window; not elevated to a new row. This sweep's one code-level fix, a missing diagnostic log on `/api/agents/:id/alpha/read`'s generic LLM-failure 502 branch (not in the deep sweep's own findings list, found by log analysis; test-first, committed on `evolve`), is orthogonal to this row and does not change the gap count. **Re-measured 2026-10-15 by the triage-lane deep sweep:** production still serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews` puts `threews/main` at `43a5dba21`, and `git log --oneline 45c768d07..threews/main` still lists exactly the same 2 commits (`43a5dba21`, `3480497ed`): **the real deploy gap remains 2 commits, unchanged from 2026-10-14.** The deep sweep's own "422 commits ahead" figure is `evolve`-branch backlog in this worktree, not production staleness; the same caveat row 1 has carried since 2026-10-12. This pass's 9 failing-page findings (`/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion`, `/docs/forge-workflows`, `/docs/standalone-repos`, `/docs/team-chat`, `/docs/tutorials/grok-bot-recipes`) are confirmed deploy lag the same way (none of their `data/pages.json` entries predate the deployed build; `git merge-base --is-ancestor` fails for each against `45c768d07`), consistent with rows 49/52/54/55/58. The `deep-db-migrations` probe's 8 pending files match rows already open above; none applied this pass, per the self-unblock playbook's no-dry-run warning on `npm run db:migrate`. Two `investigate` findings got code fixes instead of new rows: `/api/v1/ai/text-to-3d` 502s are the same NVIDIA NVCF free-tier capacity flakiness `three-ws-ai-providers-unavailable-5xx` already covers (rows 6/20), its matcher widened to include that route; `/api/pump/trending` and `/api/v1/pump/trending` 502s are a genuine but self-healing double-upstream outage (Birdeye and the pump.fun board both down in the same window, stale-cache fallback also exhausted), given a new self-healing signature so the monitor stops flagging it as `investigate`. Both fixes are committed on `evolve` (`90628d426`) with passing tests, pending the normal fast-forward to `main`. Rows 24 (`rpc_lanes: degraded, both paid lanes exhausted`) and 36 (`object_storage: down`, R2 "Please enable R2 through the Cloudflare Dashboard") re-confirmed unchanged against live `/api/healthz`; no new rows, no edits needed to either. **Re-measured 2026-10-23 by the triage-lane deep sweep:** production still serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews main` followed by `git log --oneline 45c768d07..threews/main | wc -l` returns **535**, a sharp escalation from the 2-commit gap recorded 2026-10-14/15: the deploy window simply widened while nothing shipped, not a new regression. The `deep-db-migrations` probe lists 11 pending files (`20261011220000_agent_mention_policy.sql`, `20261011230000_x402_ring_scan_checked_at.sql`, `20261015010000_mcp_client_sessions_client_nullable.sql`, `20261015040000_forge_webhook_deliveries.sql`, `20261015050000_discord_activity_rooms.sql`, `20261015060000_shopify_app.sql`, `20261015070000_vroid_hub_source.sql`, `20261015080000_pump_fee_claims.sql`, `20261015090000_agent_coin_earnings_timeline.sql`, `20261015100000_payment_session_policy_upgrades.sql`, `20261015110000_x402_outbound_resolved.sql`), none applied, per the self-unblock playbook's no-dry-run warning on `npm run db:migrate`. This pass's `/api/chat` 503 storm (groq's `FALLBACK_SIBLINGS` pointing at the dead model id `llama-3.3-70b-versatile`) is the same deploy-lag story, confirmed by `git show 45c768d07:api/chat.js`: HEAD already carries the fix (`groq: ['openai/gpt-oss-20b']`), so no new code change is needed, it ships with this row. One `investigate` finding got a code fix instead of a new row: a lone `/api/pump/coin` 502 traced via app logs to pump.fun's own Cloudflare anti-bot challenge page for that mint, with no last-good cache yet for it, same shape as the existing `pump-trending-upstreams-502` signature; given its own self-healing signature (`pump-coin-lookup-upstream-502`) so the monitor stops flagging it, committed on `evolve` (`72b600f1c`) with a passing test. The `deep-pages` probe now lists 20 failing pages, not the 9 from 2026-10-14/15: the original 9 (`/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion`, `/docs/forge-workflows`, `/docs/standalone-repos`, `/docs/team-chat`, `/docs/tutorials/grok-bot-recipes`) plus 11 newly-advertised pages added to `data/pages.json` since then (`/.well-known/skill.md`, `/apps/shopify`, `/discord-activity`, `/docs/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/google-adk`, `/docs/open-source-credits`, `/docs/shopify-app`, `/docs/video-to-motion`, `/docs/voice-framework-integrations`). Checked all 20 individually with `git merge-base --is-ancestor <adding-commit> 45c768d07`: every one of the 20 fails the check, confirming all 20 are deploy lag from the growing gap, zero are a routing bug. Rows 2, 3, 4, 5, 6, 19, 23, 24, 36 and 47 re-confirmed unchanged against live `/api/healthz` and service env; no new rows, no edits needed to any of them. **Re-measured 2026-10-24 by the triage-lane deep sweep:** the 535-commit gap recorded 2026-10-23 has shipped. Production now serves `660b5ad973e1fae8d70479c0cef4c02b6223ba2b` on `three-ws-api-00504-xnh`, built 2026-10-24T05:10:44Z. `git fetch threews main` followed by `git rev-list --count 660b5ad97..threews/main` returns **0**: `three-ws-api` is fully current. Note this covers only `three-ws-api`; the other 19 Cloud Run services deploy independently and can lag far behind it (see row 69, `remesh-service` found running a revision from 2026-07-16, three months stale). | one yes | Shipping whatever has landed on `main` since 2026-09-08, now including the security-hardening batch and the forge image-lane breaker fix | [01-ship-readiness.md](../901-production-100-01-ship-readiness.md) || 2 | Fund the settle sponsor wallet `WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`. Healthz reports `x402_settle` down with `cause: sponsor_floor` (settle 5.9%, 3 of 51 paid attempts over 3 hours, 329 `no_solana_accept`) and the sniper worker out of capital (0.2307 SOL to refill its wallets, funding master holds 0.0250 SOL); no config change moves either. Re-read 2026-09-04 08:21 UTC against mainnet: the sponsor holds **0.001507661 SOL** (1,507,661 lamports), still below the settle floor, so the Solana accept stays withdrawn from every 402 challenge. Two months of re-measurement have only ever moved this number down; nothing on this machine can move it up. **Re-measured 2026-10-08 17:05 UTC:** the sponsor holds **651,828 lamports (0.000652 SOL)**, below rent-exempt (890,880), and healthz reads `x402_settle` down at 0.0% (0 of 32 paid attempts in 3 hours, 58 settles refused at the fee-wallet floor). Because the sponsor cannot co-sign, every 402 challenge advertises the Solana accept without `feePayer`, and the ring skips it: 151 `missing_fee_payer` skips in 24 hours on the MCP streaming health probe alone. Last successful settle: 2026-10-03 19:24 UTC. **Row 1's deploy alone restores the ring, no funding needed** (diagnosed 2026-10-08 by read-only simulation): the treasury pays its own sweep fee (5,001 lamports; it holds 0.060 SOL, about 11,800 sweeps) and moves its stranded 2.587 USDC back to the ring payer, which holds 341 atomics today, and ring settles are self-pay. What still needs this wallet: the sponsor-mode callers (the loop's inline entries and the MCP streaming health probe) and the facilitator's `sponsor_fee_unfunded` verify rejects. Clearing the 2,000,000-lamport floor takes at least **1,348,172 lamports**. **Re-measured 2026-10-09 03:05 UTC:** still **651,828 lamports**, unchanged; healthz reads `x402_settle` down at 0.0% (0 of 39 paid attempts in 3 hours, 37 `sweep_broadcast_failed`, 53 settles refused at the fee-wallet floor). The sweep failures are the ones row 1's deploy clears (`a9c996e72`). **Re-measured 2026-10-14 by the triage-lane sweep (production health sweep order):** sponsor still holds **651,828 lamports**, unchanged in five days; nothing on this machine moves it. But `x402_settle` itself is no longer down: healthz now reads `degraded` at **89.8% (149/166 paid attempts, 3 hours)**, and the 17 failures are `http_503×12, rpc_error×4, rpc_preflight_failed×1`, not `sponsor_floor` or `missing_fee_payer`. Row 1's self-pay fix did restore the ring as diagnosed above; the remaining ~10% rail faults trace to row 24's exhausted paid RPC lanes (`rpc_lanes: degraded, all 2 paid lanes exhausted`), a separate billing decision, not this wallet. This wallet is still needed for the sponsor-mode callers row 1's self-pay fix does not cover, so the funding ask stands, just smaller in blast radius than it was. | ~1 SOL | x402 settle recovery, backlog 01 retirement, sniper | [../backlog/PROGRESS.md](backlog-PROGRESS.md), 2026-08-02 and 2026-09-01 entries || 3 | Decide the stranded customer funds: customers hold about 0.35 SOL behind the retired encryption key and cannot withdraw. Credit (recommended), contact first, or write off. The brief now exists with the measurement, the impossibility argument, the cost of each option and the exact commands: **[docs/ops/stranded-wallets.md](../../docs/ops/stranded-wallets.md)**. Measured 2026-09-02 from production's own records: the platform side is exactly two bot wallets holding 0.142875505 SOL (the reclaim cron has failed on them ~35,700 times each, most recently today), and one customer agent is confirmed by a `wallet_key_retired` withdraw failure (`GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm`, 0.250001 SOL). **Both customer wallets are now named** (2026-09-04): the keyed audit was run against production, so this row no longer waits on anything. `GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm` holds 0.250001 SOL and `HPL1LfuTdYDwtzJDzsnrmR2ngrrQwLTQyxJszCC4DHsN` holds 0.100001 SOL, 0.350002 SOL across exactly two wallet-auth accounts. Nine of 735 custodial wallets are sealed; the other five customer ones hold nothing. The brief carries the full table. Every input the decision needs is measured. | decision | Closing the support obligation | [docs/ops/stranded-wallets.md](../../docs/ops/stranded-wallets.md), [../backlog/PROGRESS.md](backlog-PROGRESS.md) || 4 | Rotate or explicitly accept three credentials, now that the storage half is done. **The config work shipped 2026-09-02**: the economy master key and 58 other plaintext credentials are Secret Manager references on `three-ws-api-00407-m7c`, zero plaintext credentials remain, healthz verified no worse and the master's signing path verified live. What is left is a judgment only the owner can make. (a) The master key was readable in the service config for some window, so decide whether to rotate it or accept that exposure, knowing who held project viewer access in that period; the exact owner-gated commands are in the runbook. (b) `WALLET_ENCRYPTION_KEY` and `JWT_SECRET` hold the SAME value on production, which passes the dedicated-key guard in `secret-box.js` while defeating what it guards; changing it is a re-key with real stranding risk, so it is not an agent's call. (c) `X402_SEED_SOLANA_SECRET_BASE58` holds the same value as `LAUNCHER_MASTER_SECRET_KEY_B64` though `scripts/wire-master-wallet.mjs` assigns those slots to different wallets; decide which is correct. | decision (+ one fund move if rotating) | 03 is otherwise closed | [docs/ops/wallet-key-migration.md](../../docs/ops/wallet-key-migration.md), 2026-09-02 section |@@ -85,6 +85,7 @@ on it; some resolve themselves as other work lands.| 66 | **DONE 2026-10-23, applied by the order 522 close-out.** **Apply the x402 outbound-receipts resolved-columns migration** (added 2026-10-15, order 522-integrate-23 close-out). `api/_lib/migrations/20261015110000_x402_outbound_resolved.sql` adds `resolved_payer`, `resolved_payee`, `resolved_mint`, `resolved_amount_atomics`, `resolved_decimals`, `resolved_fee_lamports`, `resolved_slot`, `resolved_block_time`, `resolved_at` and `resolve_error` to `payment_session_executions`, plus a partial index (`pse_user_settled_resolved`) for the settled-and-unresolved lookup. `npm run db:status` lists it as pending today. Not an active production bug yet: zero outbound-spend rows exist in `payment_session_executions` right now, so nothing has tried to resolve one. It becomes load-bearing the moment the new `GET /api/x402/outbound-receipts` endpoint (`api/x402/outbound-receipts.js`, `api/_lib/x402/outbound-receipts.js`) is deployed and a real agent spend lands: `resolveAndCacheRow()`'s UPDATE would fail with Postgres `42703 undefined_column` the first time anyone views that row. Apply with `node scripts/apply-migrations.mjs --apply --file 20261015110000_x402_outbound_resolved.sql` (or `npm run db:migrate`) before or together with the deploy that ships this feature. Independent of row 65's migration; both are currently pending in the same `npm run db:status` list and can be applied in the same pass. | one migration apply, timed with the deploy | Lets the Receipt Vault's new Outbound tab cache chain-verified settlement data instead of erroring the first time it resolves a real row | [docs/payment-sessions.md](../../docs/payment-sessions.md) || 67 | Deploy `workers/model-triposplat` (`gcloud builds submit --config workers/model-triposplat/cloudbuild.yaml`, stage weights with `stage-assets.sh`), set `MODEL_TRIPOSPLAT_URL` on `three-ws-api` with `--update-env-vars`, then run `POST /api/forge-splat` on a real photo and open the returned splat in `/splat`. Added 2026-10-23, order 081 close-out. | minutes | Image-to-splat Forge lane live (order 081 DoD line 2) | `workers/model-triposplat/README.md` || 68 | world.three.ws has been unreachable since the backing Cloud Run service `hyperfy-world` was deleted via the Cloud Run API by `[email protected]` at 2026-10-23T02:48:38Z (confirmed via `gcloud logging read 'protoPayload.methodName="google.cloud.run.v1.Services.DeleteService"'`), a deliberate recent action, not drift. The `world.three.ws` domain mapping is itself still healthy and provisioned; it now points at a service that no longer exists, which is why it answers a generic 404 instead of a branded error. Decide: redeploy (`bash deploy/world/apply-hardening.sh`, fully scripted and idempotent, documented in `deploy/world/README.md`) or formally retire the surface (remove the domain mapping and any in-product links to it). `/api/healthz` now correctly reports this as `world: down` instead of masking it as `unknown` (order 413). | decision | world.three.ws restoration or retirement | `deploy/world/README.md` |+| 69 | **Redeploy `remesh-service`.** Added 2026-10-24 by the triage-lane deep sweep: production logs show `remesh-service` segfaulting ("Uncaught signal: 11") twice in 24h inside `_convert_to_fbx_preserving_rig` → `_blender_convert`, the exact "Blender dies in its own interpreter teardown after a fully written export" failure that commit `59d9aa9cd` (2026-08-11, "stop discarding a finished FBX because Blender crashed on the way out") already fixed by judging completion on the FACE_COUNT marker and a non-empty output file rather than the child's exit code. `gcloud run revisions describe remesh-service-00005-8rr --region us-central1` shows that revision was created 2026-07-16, five weeks before the fix landed: this is pure deploy lag on a service three.ws's own deploy cadence does not currently cover (row 1 tracks `three-ws-api`, which is current at `660b5ad97`, 0 commits behind `threews/main`; the 20-service Cloud Run fleet deploys independently per service). No code change needed, the fix already exists; only a rebuild and redeploy of this one GPU worker is owner-gated. Command: `gcloud builds submit --config workers/remesh/cloudbuild.yaml .` from the repo root, then confirm no new "Uncaught signal: 11" lines appear for `remesh-service` over the following 24h. | one build + one deploy | Stops rig-preserving FBX exports from occasionally crashing the whole `remesh-service` instance mid-request | `workers/remesh/main.py` (`_blender_convert`), [docs/ops/production-log-triage.md](../../docs/ops/production-log-triage.md) |Deleted 2026-09-01: the former row 5 (`WALLET_ENCRYPTION_KEY_PREVIOUS` policy), whichdeclared itself "delete on reading" and had outlived that by three weeks.