Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(prompts): re-measure OWNER-ACTIONS rows 1 and 36 from the production health sweep

nirholas1582492f498e
diff --git a/prompts/finish/_context/production-100-OWNER-ACTIONS.md b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
index 7d65bf3f0..da5ba1e55 100644
--- a/prompts/finish/_context/production-100-OWNER-ACTIONS.md
+++ b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
@@ -13,7 +13,7 @@ on it; some resolve themselves as other work lands.
| # | Action | Size | Unblocks | Detail lives in |
|---|---|---|---|---|
-| 1 | Approve the production deploy (or say "get production working", which is itself the approval). **The outage this row was written for is CLEARED.** Re-measured 2026-09-09 from the live origin: production serves `880bdcef8` on revision `three-ws-api-00420-ljh`, built 2026-09-08 19:05 UTC, and the `ERR_MODULE_NOT_FOUND` on `/app/services/home-relay/src/token.js` is gone: `/api/healthz` answers **HTTP 200** and `/api/home/status` and `/api/home/connections` answer 401 (auth required) rather than 500. What remains is ordinary staleness, 182 commits behind `main`, which is a routine cadence question and no longer a platform-health emergency. The 2026-09-04 text of this row, which described 96 commits and a 500ing healthz, is superseded. **Re-measured 2026-10-08 17:05 UTC:** production serves `fe2a8b24f` on `three-ws-api-00479-d7z`, **146 commits behind `main`, 90 of them fixes** (security, payouts, SSRF and XSS among them). Shipping them clears four live findings at once: the ring sweep's fee falls back to the treasury instead of the dry sponsor (`a9c996e72`, about 30 failed sweeps an hour since 2026-10-03), an EVM testnet identity-index cursor frozen since 2026-09-11 (`21fa42bb5`), two pages and their two docs pages advertised in the sitemap but answering 404 (added in `16f9d07f4`), and `/tty` holding its animation until the end for any client that accepts compression (`03b4f1189`). One migration must go with it: `20261008150000_revenue_direct_settle_backfill.sql` (pending; it matches 0 rows today, so applying it is safe, and the code half is what keeps the payout cron from paying a direct-to-seller sale twice). Commands, in order: `npm run db:status`, `npm run db:migrate`, `npm run deploy:gcp:full`. **Re-measured 2026-10-09 03:05 UTC by the triage sweep:** production still serves image `fe2a8b24f` (now on revision `three-ws-api-00480-mws`, an env-only roll that raised `DB_RETENTION_HIGH_WATER_MB` to 16384), **172 commits behind `main`**, and the `evolve` branch holds 85 more awaiting the runner's merge, six of them this sweep's production fixes: the leak scanner's lease and budget (`0abce6d69`, which ends about eight overlapping 600 to 900 s scans that were burning the Solana RPC quota), `-32020` RPC failover (`4b51a179c`), static 412/416 instead of 500 (`e01e43b3b`), the news knowledge write fix (`833407d4f`, 120 lost stories a day), and the reconstruct sweep's plan-limit close (`85d3b2034`). `20261008150000_revenue_direct_settle_backfill.sql` has since been applied; `npm run db:status` now lists two pending, `20261008190000_oauth_refresh_user_client_index.sql` and `20261008200000_mcp_client_analytics.sql` (rows 27 and 28). The 8 sitemap pages still answering 404 (`/grok`, `/grok-skill.md`, `/docs/grok`, `/docs/grok-bot`, `/dextools`, `/docs/dextools`, `/chart-companion`, `/docs/chart-companion`) are all deploy lag. **Re-measured 2026-10-10 20:06 UTC by the triage sweep:** production serves `5af439bc0` on `three-ws-api-00487-7mt`, built 2026-10-10 05:01 UTC, **45 commits behind `main`** (down from 172 on 2026-10-09; most of the gap already shipped), and the `evolve` branch holds 89 more. All 869 advertised pages, every cron (125) and the fleet (48 services) check out clean; the only pending migrations are rows 27 and 28's two files. **Re-measured 2026-10-11 04:16 UTC by the deep triage sweep:** production serves `3503381b1` on `three-ws-api-00491-fbl`, **112 commits behind `main`** (up from 45 on 2026-10-10). One new page-lag symptom from the gap: `/docs/tutorials/grok-bot-recipes` (added in `71c673f99`, after the running revision) answers 200 with `/docs`'s content instead of its own; this is deploy lag, not a routing bug (verified: `71c673f99` is not an ancestor of `3503381b1`). The fleet (48 services), 125 crons and DB are otherwise clean; the only pending migrations remain rows 27 and 28's two files. **Re-measured 2026-10-11 12:32 UTC by the triage-lane sweep:** production serves `66a5e93bd` on `three-ws-api-00494-wh5`, built 2026-10-11 07:22 UTC, **only 5 commits behind `main`** (down sharply from 112 on 2026-10-11 04:16; the earlier large gap has shipped). The 5 outstanding commits are the desktop OS/shell/widgets feature set and the campus world update (`e33f529d8`, `33b5490c5`, `a48caf816`, `9c47f3ceb`, `86366a55e`). Rows 27 and 28's migrations are now applied; `npm run db:status` shows 2 new pending files, `20261011220000_agent_mention_policy.sql` (tracked by row 35) and `20261011230000_x402_ring_scan_checked_at.sql` (tracked by row 24's narrative), both additive. This sweep also committed two root-cause fixes to `main`-bound code that are NOT yet in this 5-commit gap: `65d6c4be8` (clink-fun MCP stream timeout misclassified as investigate) and `2f2fedf99` (a Pollinations 402 was putting the paid Replicate/TRELLIS lane into cooldown, contributing to `forge_generation: down`); the second one fixes a live production defect, so it is worth prioritizing in the next deploy. **Re-measured 2026-10-11 21:12 UTC by the triage-lane sweep:** production still serves `66a5e93bd` on `three-ws-api-00494-wh5`, **28 commits behind `main`** (`main` now at `9eda7992d`), up from 5 at 12:32 UTC because the desktop OS/shell/widgets/campus set from that measurement has been joined by new work: the x-content partner-spotlight feature (registry, daily scheduler, published IBM and AWS partner articles, the `/partners/benefits` page) and five dependabot bumps (`js-yaml`/`markdownlint-cli2`, `nanoid`, `maplibre-gl`, `torch` in `workers/texture`, `brace-expansion`, `source-map-js`, `moment`, `fast-uri`). `2f2fedf99` (the forge-cooldown fix) is still only on `evolve`, not yet on `main`; it remains worth prioritizing. The deep sweep found exactly one new pending migration, now tracked as row 42; rows 24 and 35's migrations are applied. The `/docs/desktop-widgets` 404 and `/docs/tutorials/grok-bot-recipes` wrong-canonical findings are confirmed deploy lag again (their catalog commits are not ancestors of `66a5e93bd`), not new routing bugs. **Re-measured 2026-10-12 by the triage-lane sweep, against the canonical `threews` remote (the local `main` ref in the `evolve` worktree was stale):** production (`/api/healthz`'s own `commitShort`) serves `c266e2591`, **11 commits behind `threews/main`** at `80e5e35f7`, down sharply from 28. The outstanding set is the Dissolve feature (a GPU particle-storm viewer for any 3D model) with its docs and page-catalog regeneration, a security-hardening batch (`8a56a44ab`: auth, wallet policy, SDP, vault, skill sandbox, MCP bridge URL guard, all with regression tests), a worker fix (`315e96f96`: block internal-host navigation in the screen pool, load text2motion weights with `weights_only`), an accessibility fix (`f772c8708`: icon-only button tooltips, gallery/create/header name-to-text matching), the Grid desktop world (`762269248`, `f35d07d93`, `a60d72dad`) and a new UI-lint tool (`568162d8d`). None of this sweep's own fixes (the Vertex/HF image-lane circuit breaker, the `forge-image-ladder-exhausted-502` and widened `r2-downstream-5xx` triage signatures) are in this 11-commit gap either; they are further back on `evolve`, behind the runner's usual fast-forward-to-`main` cadence. **Corrected 2026-10-12 17:54 UTC by the triage-lane sweep: the same-day "11 commits behind" figure above was a misread.** Querying the live `/api/version` endpoint directly (not `/api/healthz`, which carries no commit field) shows production still serves `771cfef01` on `three-ws-api-00496-78g`, built 2026-10-12 07:08 UTC. A fresh `git fetch threews` puts `threews/main` at `80e5e35f7`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds with exactly 1 commit between them, so the deployed revision is 1 commit AHEAD of the canonical remote `main`, not behind it. **The real deploy gap (commits on `main` not yet in production) is 0.** The 242 commits the deep sweep reports as "ahead" are `evolve`-branch commits (this worktree's local `main`, one commit ahead of pushed `threews/main`) not yet pushed, including this sweep's own `5afb045cf` (widened the `r2-downstream-5xx` triage signature to cover `/api/render/animate`, `/api/render/glb`, `/api/render/avatar-clip` and `/api/animations/clips/<id>`, all previously-unmatched symptoms of the same R2 flap row 36 already tracks); they are not a production staleness problem. **Re-measured 2026-10-13 by the triage-lane sweep:** production still serves `771cfef01` on `three-ws-api-00496-78g`. A fresh `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds, so the deployed revision is still exactly 1 commit AHEAD of the canonical remote `main`. **The real deploy gap remains 0.** The deep sweep's "283 commits ahead" figure is `evolve`-branch backlog (unpushed local commits, including `f939d5ef1`, the dist-lib CDN embed size fix documented in row 48) awaiting the runner's fast-forward, not a production staleness problem. This sweep found no new code-level defect to fix: the dist-lib bug is already root-caused, fixed and tested on `evolve`; the Solana-adjacent degraded subsystems (`agent_index`, `rpc_lanes`, `helius`, `sniper`, `three_token_rail`) and the `ccv-ai-providers-unavailable-503` finding (Groq's 200k tokens/day cap, the GCP billing hold, and the revoked OpenRouter key) are all pre-existing owner items already tracked in rows 23, 24, 19, 2, 20 and 6; none needed a new row. **Re-measured 2026-10-13 (triage-lane sweep, fifth pass):** production still serves `771cfef01` on `three-ws-api-00497-2ss`. `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` still succeeds, so the deployed revision remains exactly 1 commit AHEAD of `main`. **The real deploy gap stays 0.** The deep sweep's "291 commits ahead" figure is `evolve`-branch backlog (unpushed, includes `f939d5ef1` / row 48 and this pass's new row 53) awaiting the runner's fast-forward, not production staleness. The 8 sitemap-advertised pages answering 404 or wrong-canonical this pass (`/discord-activity`, `/docs/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/open-source-credits`, `/docs/voice-framework-integrations`, `/open-source`, `/docs/tutorials/grok-bot-recipes`) are all confirmed deploy lag (`git merge-base --is-ancestor <add-commit> 771cfef01` fails for each); no routing bug. | one yes | Shipping whatever has landed on `main` since 2026-09-08, now including the security-hardening batch and the forge image-lane breaker fix | [01-ship-readiness.md](../901-production-100-01-ship-readiness.md) |
+| 1 | Approve the production deploy (or say "get production working", which is itself the approval). **The outage this row was written for is CLEARED.** Re-measured 2026-09-09 from the live origin: production serves `880bdcef8` on revision `three-ws-api-00420-ljh`, built 2026-09-08 19:05 UTC, and the `ERR_MODULE_NOT_FOUND` on `/app/services/home-relay/src/token.js` is gone: `/api/healthz` answers **HTTP 200** and `/api/home/status` and `/api/home/connections` answer 401 (auth required) rather than 500. What remains is ordinary staleness, 182 commits behind `main`, which is a routine cadence question and no longer a platform-health emergency. The 2026-09-04 text of this row, which described 96 commits and a 500ing healthz, is superseded. **Re-measured 2026-10-08 17:05 UTC:** production serves `fe2a8b24f` on `three-ws-api-00479-d7z`, **146 commits behind `main`, 90 of them fixes** (security, payouts, SSRF and XSS among them). Shipping them clears four live findings at once: the ring sweep's fee falls back to the treasury instead of the dry sponsor (`a9c996e72`, about 30 failed sweeps an hour since 2026-10-03), an EVM testnet identity-index cursor frozen since 2026-09-11 (`21fa42bb5`), two pages and their two docs pages advertised in the sitemap but answering 404 (added in `16f9d07f4`), and `/tty` holding its animation until the end for any client that accepts compression (`03b4f1189`). One migration must go with it: `20261008150000_revenue_direct_settle_backfill.sql` (pending; it matches 0 rows today, so applying it is safe, and the code half is what keeps the payout cron from paying a direct-to-seller sale twice). Commands, in order: `npm run db:status`, `npm run db:migrate`, `npm run deploy:gcp:full`. **Re-measured 2026-10-09 03:05 UTC by the triage sweep:** production still serves image `fe2a8b24f` (now on revision `three-ws-api-00480-mws`, an env-only roll that raised `DB_RETENTION_HIGH_WATER_MB` to 16384), **172 commits behind `main`**, and the `evolve` branch holds 85 more awaiting the runner's merge, six of them this sweep's production fixes: the leak scanner's lease and budget (`0abce6d69`, which ends about eight overlapping 600 to 900 s scans that were burning the Solana RPC quota), `-32020` RPC failover (`4b51a179c`), static 412/416 instead of 500 (`e01e43b3b`), the news knowledge write fix (`833407d4f`, 120 lost stories a day), and the reconstruct sweep's plan-limit close (`85d3b2034`). `20261008150000_revenue_direct_settle_backfill.sql` has since been applied; `npm run db:status` now lists two pending, `20261008190000_oauth_refresh_user_client_index.sql` and `20261008200000_mcp_client_analytics.sql` (rows 27 and 28). The 8 sitemap pages still answering 404 (`/grok`, `/grok-skill.md`, `/docs/grok`, `/docs/grok-bot`, `/dextools`, `/docs/dextools`, `/chart-companion`, `/docs/chart-companion`) are all deploy lag. **Re-measured 2026-10-10 20:06 UTC by the triage sweep:** production serves `5af439bc0` on `three-ws-api-00487-7mt`, built 2026-10-10 05:01 UTC, **45 commits behind `main`** (down from 172 on 2026-10-09; most of the gap already shipped), and the `evolve` branch holds 89 more. All 869 advertised pages, every cron (125) and the fleet (48 services) check out clean; the only pending migrations are rows 27 and 28's two files. **Re-measured 2026-10-11 04:16 UTC by the deep triage sweep:** production serves `3503381b1` on `three-ws-api-00491-fbl`, **112 commits behind `main`** (up from 45 on 2026-10-10). One new page-lag symptom from the gap: `/docs/tutorials/grok-bot-recipes` (added in `71c673f99`, after the running revision) answers 200 with `/docs`'s content instead of its own; this is deploy lag, not a routing bug (verified: `71c673f99` is not an ancestor of `3503381b1`). The fleet (48 services), 125 crons and DB are otherwise clean; the only pending migrations remain rows 27 and 28's two files. **Re-measured 2026-10-11 12:32 UTC by the triage-lane sweep:** production serves `66a5e93bd` on `three-ws-api-00494-wh5`, built 2026-10-11 07:22 UTC, **only 5 commits behind `main`** (down sharply from 112 on 2026-10-11 04:16; the earlier large gap has shipped). The 5 outstanding commits are the desktop OS/shell/widgets feature set and the campus world update (`e33f529d8`, `33b5490c5`, `a48caf816`, `9c47f3ceb`, `86366a55e`). Rows 27 and 28's migrations are now applied; `npm run db:status` shows 2 new pending files, `20261011220000_agent_mention_policy.sql` (tracked by row 35) and `20261011230000_x402_ring_scan_checked_at.sql` (tracked by row 24's narrative), both additive. This sweep also committed two root-cause fixes to `main`-bound code that are NOT yet in this 5-commit gap: `65d6c4be8` (clink-fun MCP stream timeout misclassified as investigate) and `2f2fedf99` (a Pollinations 402 was putting the paid Replicate/TRELLIS lane into cooldown, contributing to `forge_generation: down`); the second one fixes a live production defect, so it is worth prioritizing in the next deploy. **Re-measured 2026-10-11 21:12 UTC by the triage-lane sweep:** production still serves `66a5e93bd` on `three-ws-api-00494-wh5`, **28 commits behind `main`** (`main` now at `9eda7992d`), up from 5 at 12:32 UTC because the desktop OS/shell/widgets/campus set from that measurement has been joined by new work: the x-content partner-spotlight feature (registry, daily scheduler, published IBM and AWS partner articles, the `/partners/benefits` page) and five dependabot bumps (`js-yaml`/`markdownlint-cli2`, `nanoid`, `maplibre-gl`, `torch` in `workers/texture`, `brace-expansion`, `source-map-js`, `moment`, `fast-uri`). `2f2fedf99` (the forge-cooldown fix) is still only on `evolve`, not yet on `main`; it remains worth prioritizing. The deep sweep found exactly one new pending migration, now tracked as row 42; rows 24 and 35's migrations are applied. The `/docs/desktop-widgets` 404 and `/docs/tutorials/grok-bot-recipes` wrong-canonical findings are confirmed deploy lag again (their catalog commits are not ancestors of `66a5e93bd`), not new routing bugs. **Re-measured 2026-10-12 by the triage-lane sweep, against the canonical `threews` remote (the local `main` ref in the `evolve` worktree was stale):** production (`/api/healthz`'s own `commitShort`) serves `c266e2591`, **11 commits behind `threews/main`** at `80e5e35f7`, down sharply from 28. The outstanding set is the Dissolve feature (a GPU particle-storm viewer for any 3D model) with its docs and page-catalog regeneration, a security-hardening batch (`8a56a44ab`: auth, wallet policy, SDP, vault, skill sandbox, MCP bridge URL guard, all with regression tests), a worker fix (`315e96f96`: block internal-host navigation in the screen pool, load text2motion weights with `weights_only`), an accessibility fix (`f772c8708`: icon-only button tooltips, gallery/create/header name-to-text matching), the Grid desktop world (`762269248`, `f35d07d93`, `a60d72dad`) and a new UI-lint tool (`568162d8d`). None of this sweep's own fixes (the Vertex/HF image-lane circuit breaker, the `forge-image-ladder-exhausted-502` and widened `r2-downstream-5xx` triage signatures) are in this 11-commit gap either; they are further back on `evolve`, behind the runner's usual fast-forward-to-`main` cadence. **Corrected 2026-10-12 17:54 UTC by the triage-lane sweep: the same-day "11 commits behind" figure above was a misread.** Querying the live `/api/version` endpoint directly (not `/api/healthz`, which carries no commit field) shows production still serves `771cfef01` on `three-ws-api-00496-78g`, built 2026-10-12 07:08 UTC. A fresh `git fetch threews` puts `threews/main` at `80e5e35f7`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds with exactly 1 commit between them, so the deployed revision is 1 commit AHEAD of the canonical remote `main`, not behind it. **The real deploy gap (commits on `main` not yet in production) is 0.** The 242 commits the deep sweep reports as "ahead" are `evolve`-branch commits (this worktree's local `main`, one commit ahead of pushed `threews/main`) not yet pushed, including this sweep's own `5afb045cf` (widened the `r2-downstream-5xx` triage signature to cover `/api/render/animate`, `/api/render/glb`, `/api/render/avatar-clip` and `/api/animations/clips/<id>`, all previously-unmatched symptoms of the same R2 flap row 36 already tracks); they are not a production staleness problem. **Re-measured 2026-10-13 by the triage-lane sweep:** production still serves `771cfef01` on `three-ws-api-00496-78g`. A fresh `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` succeeds, so the deployed revision is still exactly 1 commit AHEAD of the canonical remote `main`. **The real deploy gap remains 0.** The deep sweep's "283 commits ahead" figure is `evolve`-branch backlog (unpushed local commits, including `f939d5ef1`, the dist-lib CDN embed size fix documented in row 48) awaiting the runner's fast-forward, not a production staleness problem. This sweep found no new code-level defect to fix: the dist-lib bug is already root-caused, fixed and tested on `evolve`; the Solana-adjacent degraded subsystems (`agent_index`, `rpc_lanes`, `helius`, `sniper`, `three_token_rail`) and the `ccv-ai-providers-unavailable-503` finding (Groq's 200k tokens/day cap, the GCP billing hold, and the revoked OpenRouter key) are all pre-existing owner items already tracked in rows 23, 24, 19, 2, 20 and 6; none needed a new row. **Re-measured 2026-10-13 (triage-lane sweep, fifth pass):** production still serves `771cfef01` on `three-ws-api-00497-2ss`. `git fetch threews` puts `threews/main` at `80e5e35f74302c85138bf6cabdad4f2db65b9ecf`, and `git merge-base --is-ancestor threews/main 771cfef01` still succeeds, so the deployed revision remains exactly 1 commit AHEAD of `main`. **The real deploy gap stays 0.** The deep sweep's "291 commits ahead" figure is `evolve`-branch backlog (unpushed, includes `f939d5ef1` / row 48 and this pass's new row 53) awaiting the runner's fast-forward, not production staleness. The 8 sitemap-advertised pages answering 404 or wrong-canonical this pass (`/discord-activity`, `/docs/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/open-source-credits`, `/docs/voice-framework-integrations`, `/open-source`, `/docs/tutorials/grok-bot-recipes`) are all confirmed deploy lag (`git merge-base --is-ancestor <add-commit> 771cfef01` fails for each); no routing bug. **Re-measured 2026-10-14 by the triage-lane sweep (production health sweep order):** production serves `45c768d07` on `three-ws-api-00500-b8q`. A fresh `git fetch threews` puts `threews/main` at `43a5dba21`, and `git log --oneline 45c768d07..threews/main` lists exactly 2 commits (`43a5dba21`, an x-content proofs refresh, and `3480497ed`, the R2-NotEntitled storage-outage failover fix): **the real deploy gap is 2 commits, both routine, neither urgent.** The deep sweep's 16 failing-page findings (`/discord-activity`, `/docs/engine-bridges`, `/docs/engine-qa`, `/docs/google-adk`, `/docs/open-source-credits`, `/docs/shopify-app`, `/docs/video-to-motion`, `/docs/voice-framework-integrations`, `/docs/vroid-hub-import`, `/docs/wordpress-embed`, `/import/vroid`, `/open-source`, `/video-to-motion` and their docs pages, plus the `/docs/tutorials/grok-bot-recipes` wrong-canonical) are confirmed NOT part of this 2-commit gap: none of their routes exist in `threews/main`'s `data/pages.json` at all (checked directly), so they are unpushed `evolve`-branch backlog (334 commits ahead of `threews/main` in this worktree), the same already-tracked rows 49, 52, 54 and 55 cover (Tripo webhook, Discord Activity, Shopify app, VRoid Hub import). The 5 pending migrations `npm run db:status` lists (`mcp_client_sessions_client_nullable`, `forge_webhook_deliveries`, `discord_activity_rooms`, `shopify_app`, `vroid_hub_source`) are the same set tracked by rows 42, 49, 52, 54 and 55; none applied, consistent with those rows' guidance. The lone unclassified `investigate` finding, a single `HTTP 502 GET /api/v1/coins` on `cryptocurrency-cv` with no matching app-level error, has no isolated root cause distinguishable from the 300-plus already-tracked `ccv-ai-providers-unavailable-503` hits on the same service in the same window; not elevated to a new row. This sweep's one code-level fix, a missing diagnostic log on `/api/agents/:id/alpha/read`'s generic LLM-failure 502 branch (not in the deep sweep's own findings list, found by log analysis; test-first, committed on `evolve`), is orthogonal to this row and does not change the gap count. | one yes | Shipping whatever has landed on `main` since 2026-09-08, now including the security-hardening batch and the forge image-lane breaker fix | [01-ship-readiness.md](../901-production-100-01-ship-readiness.md) |
| 2 | Fund the settle sponsor wallet `WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`. Healthz reports `x402_settle` down with `cause: sponsor_floor` (settle 5.9%, 3 of 51 paid attempts over 3 hours, 329 `no_solana_accept`) and the sniper worker out of capital (0.2307 SOL to refill its wallets, funding master holds 0.0250 SOL); no config change moves either. Re-read 2026-09-04 08:21 UTC against mainnet: the sponsor holds **0.001507661 SOL** (1,507,661 lamports), still below the settle floor, so the Solana accept stays withdrawn from every 402 challenge. Two months of re-measurement have only ever moved this number down; nothing on this machine can move it up. **Re-measured 2026-10-08 17:05 UTC:** the sponsor holds **651,828 lamports (0.000652 SOL)**, below rent-exempt (890,880), and healthz reads `x402_settle` down at 0.0% (0 of 32 paid attempts in 3 hours, 58 settles refused at the fee-wallet floor). Because the sponsor cannot co-sign, every 402 challenge advertises the Solana accept without `feePayer`, and the ring skips it: 151 `missing_fee_payer` skips in 24 hours on the MCP streaming health probe alone. Last successful settle: 2026-10-03 19:24 UTC. **Row 1's deploy alone restores the ring, no funding needed** (diagnosed 2026-10-08 by read-only simulation): the treasury pays its own sweep fee (5,001 lamports; it holds 0.060 SOL, about 11,800 sweeps) and moves its stranded 2.587 USDC back to the ring payer, which holds 341 atomics today, and ring settles are self-pay. What still needs this wallet: the sponsor-mode callers (the loop's inline entries and the MCP streaming health probe) and the facilitator's `sponsor_fee_unfunded` verify rejects. Clearing the 2,000,000-lamport floor takes at least **1,348,172 lamports**. **Re-measured 2026-10-09 03:05 UTC:** still **651,828 lamports**, unchanged; healthz reads `x402_settle` down at 0.0% (0 of 39 paid attempts in 3 hours, 37 `sweep_broadcast_failed`, 53 settles refused at the fee-wallet floor). The sweep failures are the ones row 1's deploy clears (`a9c996e72`). | ~1 SOL | x402 settle recovery, backlog 01 retirement, sniper | [../backlog/PROGRESS.md](backlog-PROGRESS.md), 2026-08-02 and 2026-09-01 entries |
| 3 | Decide the stranded customer funds: customers hold about 0.35 SOL behind the retired encryption key and cannot withdraw. Credit (recommended), contact first, or write off. The brief now exists with the measurement, the impossibility argument, the cost of each option and the exact commands: **[docs/ops/stranded-wallets.md](../../docs/ops/stranded-wallets.md)**. Measured 2026-09-02 from production's own records: the platform side is exactly two bot wallets holding 0.142875505 SOL (the reclaim cron has failed on them ~35,700 times each, most recently today), and one customer agent is confirmed by a `wallet_key_retired` withdraw failure (`GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm`, 0.250001 SOL). **Both customer wallets are now named** (2026-09-04): the keyed audit was run against production, so this row no longer waits on anything. `GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm` holds 0.250001 SOL and `HPL1LfuTdYDwtzJDzsnrmR2ngrrQwLTQyxJszCC4DHsN` holds 0.100001 SOL, 0.350002 SOL across exactly two wallet-auth accounts. Nine of 735 custodial wallets are sealed; the other five customer ones hold nothing. The brief carries the full table. Every input the decision needs is measured. | decision | Closing the support obligation | [docs/ops/stranded-wallets.md](../../docs/ops/stranded-wallets.md), [../backlog/PROGRESS.md](backlog-PROGRESS.md) |
| 4 | Rotate or explicitly accept three credentials, now that the storage half is done. **The config work shipped 2026-09-02**: the economy master key and 58 other plaintext credentials are Secret Manager references on `three-ws-api-00407-m7c`, zero plaintext credentials remain, healthz verified no worse and the master's signing path verified live. What is left is a judgment only the owner can make. (a) The master key was readable in the service config for some window, so decide whether to rotate it or accept that exposure, knowing who held project viewer access in that period; the exact owner-gated commands are in the runbook. (b) `WALLET_ENCRYPTION_KEY` and `JWT_SECRET` hold the SAME value on production, which passes the dedicated-key guard in `secret-box.js` while defeating what it guards; changing it is a re-key with real stranding risk, so it is not an agent's call. (c) `X402_SEED_SOLANA_SECRET_BASE58` holds the same value as `LAUNCHER_MASTER_SECRET_KEY_B64` though `scripts/wire-master-wallet.mjs` assigns those slots to different wallets; decide which is correct. | decision (+ one fund move if rotating) | 03 is otherwise closed | [docs/ops/wallet-key-migration.md](../../docs/ops/wallet-key-migration.md), 2026-09-02 section |
@@ -46,7 +46,7 @@ on it; some resolve themselves as other work lands.
| 32 | **Decide retention for the six large tables the database retention cron never prunes** (added 2026-10-09 by the evolve loop triage sweep). The Neon footprint reached 11,606 MB, past the 8192 MB high-water mark since 2026-09-09, so five write-heavy crons (`launcher-tick`, `coin-intel-observe`, `smart-money-rollup`, `recompute-reputation`, `intel-learn`) skipped every tick for a month. The sweep raised `DB_RETENTION_HIGH_WATER_MB` to 16384 (config-only, pre-approved; real cap `neon.max_cluster_size` is 16TB), which buys about six weeks at the measured ~114 MB a day. The growth is in tables outside the retention set, and pruning them deletes history, so it is not an agent's call: `oracle_training_set` 1.8 GB (1.37M rows), `x402_self_facilitator_log` 1.4 GB (3.0M rows; the code already names it a product decision about lifetime revenue totals), `firewall_decisions` 1.2 GB, `usage_events` 0.7 GB, `club_tips` 0.6 GB (1.2M rows), `ops_alerts` 0.5 GB (693k one-off `503 settlement_unavailable` rows from July to October). Name a window per table (or keep forever and accept the storage bill); an agent then adds each to `TIME_SERIES_TABLES` in `api/cron/db-retention.js`. | decision | Keeping the write crons running past mid-November without raising the mark again | [docs/ops/db-retention.md](../../../docs/ops/db-retention.md) |
| 33 | **Create a production xAI API key so a Grok model's live MCP calls against three.ws can be proven** (added 2026-10-10 by the evolve loop, x-grok order 039, which already owns this exact gate as order 927: `prompts/finish/927-x-grok-46-owner-grok-api-key.md`). Step 0 re-derivation this session found no `GROK_API_KEY`/`XAI_API_KEY` anywhere: not in `process.env`, not in `.env`/`.env.local`, not on the `three-ws-api` Cloud Run service (`node scripts/read-service-env.mjs '^(GROK\|XAI)_API_KEY$' --names` returned nothing). `scripts/xai-mcp-e2e.mjs` (npm script `e2e:xai-mcp`) is built and verified end to end in dry-run mode: it matches xAI's current Responses API remote-MCP schema field for field and names this gate when no key is found. Create a key at console.x.ai, then either set it directly (`gcloud run services update three-ws-api --region us-central1 --update-env-vars GROK_API_KEY=<key>`) or hand it to order 927 to wire through Secret Manager; afterward `npm run e2e:xai-mcp` should exit 0 against production. | minutes, console only | Closes x-grok order 039's one remaining Definition of Done line, and order 927 | [_context/x-grok-PROGRESS.md](x-grok-PROGRESS.md), 2026-10-10 entry |
| 35 | **Apply the agent mention-policy migration, then ship it with the next deploy** (added 2026-10-11 by the evolve loop, x-grok order 061, which is otherwise complete and retired). `api/_lib/migrations/20261011220000_agent_mention_policy.sql` adds `mention_replies_enabled`, `mention_replies_dry_run`, `mention_replies_daily_cap` and `mention_replies_allowed_intents` to `agent_x_policies`, the four fields the new "Answer mentions" panel on `pages/agent-edit.html` (Social tab) reads and writes. `npm run db:status` on 2026-10-11 lists it as the only pending migration. The read path (`GET /api/x/agents`) already works without it, since the query is `select *` and `rowToPolicy()` defaults the missing columns; only the write path breaks: `saveAgentPolicy`'s `INSERT` names the four columns explicitly, so a save attempt against the unmigrated table answers 500 with Postgres `42703 undefined_column`, verified live in a local run of the full UI against the real dev database on 2026-10-11 (every other state, no-connection/missing-scope/connected/populated-decisions, was verified working by temporarily granting `X_OAUTH_CLIENT_ID`/`SECRET` in `.env.local` only, never in production). Apply it with `node scripts/apply-migrations.mjs --apply --file 20261011220000_agent_mention_policy.sql`, or `npm run db:status` then `npm run db:migrate` (nothing else is pending as of this writing). `npm run db:check` blocks the next deploy on it until then. Separately, and not blocking this row: production has never set `X_OAUTH_CLIENT_ID`/`X_OAUTH_CLIENT_SECRET` at all (`node scripts/read-service-env.mjs '^X_OAUTH_CLIENT'` finds nothing), so the whole per-agent "connect this agent's own X account" flow (distinct from the company bot account's `X_API_KEY`/`X_ACCESS_TOKEN` posting credentials, which are set) answers "X connections are not enabled on this deployment" for every agent today; that is a pre-existing gap this order did not create and its settings UI already degrades to that state correctly. | one command | The "Answer mentions" save button in the agent settings UI | [x-grok-PROGRESS.md](x-grok-PROGRESS.md), 2026-10-11 entry for order 061 |
-| 36 | **Check Cloudflare R2's own status/dashboard for the bucket behind `S3_BUCKET`: it flaps between healthy and unreachable without any corresponding change on our side** (added 2026-10-11 by the evolve scout lane). Queried `ops_alerts` directly (columns: `signature, title, detail, severity, count, environment, first_seen, last_seen`): since 2026-09-08 the `object_storage` sensor (`api/_lib/ops/object-storage-health.js`) has logged roughly 167 "DEGRADED: Object storage (Cloudflare R2)" events (`bucket reachable but slow, signed list ~1095ms` against a 1000ms threshold) and about 18 "DOWN" events, interleaved with many "RESOLVED" events minutes apart showing a healthy 150 to 300ms. Reproduced live with production's own `S3_ACCESS_KEY_ID`/`S3_SECRET_ACCESS_KEY`/`S3_BUCKET`/`S3_ENDPOINT` (pulled via `node scripts/read-service-env.mjs` one variable at a time, used in-process, then deleted from disk immediately): `gatherObjectStorageHealth()` returned `{"status":"down","detail":"bucket unreachable: Please enable R2 through the Cloudflare Dashboard.","metrics":{"latencyMs":135,"credentialFault":false}}`, a 135ms response, i.e. the bucket answered fast but Cloudflare's own API told us R2 was not enabled, which cannot be true most of the time this account is clearly serving reads. Ruled out as a code cause: the credential is not rejected (`credentialFault:false`, no `SignatureDoesNotMatch`), and the AWS SDK's Node HTTP handler already defaults to keep-alive on (`keepAlive:true, maxSockets:50` in `@smithy/node-http-handler`), so this is not the classic missing-keep-alive latency bug. The probe itself is a single signed `ListObjectsV2Command` with no retry (`api/_lib/ops/object-storage-health.js:76-92`), so a genuinely transient one-off Cloudflare hiccup is indistinguishable from a sustained outage in this sensor today, which may be inflating the DOWN/DEGRADED counts above a one-retry version would show. Two independent things to do, neither owned by this reconciliation order: (a) check Cloudflare's own R2 dashboard/status page for the account behind `S3_BUCKET` for incidents correlated with the alert timestamps above (owner/dashboard access this agent does not have); (b) if the pattern looks like a true transient flap rather than sustained Cloudflare-side degradation, a future order could add one immediate retry before `gatherObjectStorageHealth()` classifies `down`/`degraded`, with a test proving a single synthetic failure followed by a synthetic success no longer pages. Not attempted here because the current evidence does not confidently distinguish "add a retry" from "mute a real recurring problem", and CLAUDE.md is explicit that a check does not get weakened without a dated, measured reason; this row exists so neither path gets silently skipped. 2026-10-13 (evolve scout lane, re-measured before considering it as a work order): queried `ops_alerts` again, grouped by failure text. The `"bucket unreachable: Please enable R2 through the Cloudflare Dashboard"` DOWN signature (the one live healthz reports right now) spans `first_seen` 2026-09-08T19:35Z to `last_seen` 2026-10-13T19:10Z (this instant), 58 alert rows, i.e. it has been recurring continuously for over a month and is still firing as this note is written: a sustained, ongoing condition, not a one-off transient blip. The retry idea in option (b) above does not apply to this dominant failure mode: retrying a single probe cannot paper over something that has been reproducing for 30-plus days, and doing so would be exactly the "mute a real recurring problem" case CLAUDE.md warns against, so no work order was written for it this pass. The narrower "DEGRADED: bucket reachable but slow" variant (option (b)'s original target) is a separate, lower-severity code path from this DOWN/unreachable one and was not re-measured this pass; a future pass that wants to pursue the retry idea should scope it to that DEGRADED signature specifically and re-confirm its flap pattern first, never the DOWN/unreachable signature above. 2026-10-13 (order 400, evolve loop, severity escalation): re-measured again, 18:10-18:30 UTC. The "flaps between healthy and unreachable" framing at the top of this row is now stale and understates the severity: the last `RESOLVED: Object storage (Cloudflare R2)` row is 2026-10-11T06:55:05Z, and every hourly sensor tick since then has been `DOWN` with zero resolves, a 35-plus-hour unbroken streak at measurement time (now 60-plus hours as this note is written, still ongoing). Two things that were working as designed have also failed: the public `*.r2.dev` domain itself now answers a Cloudflare-branded 403 (an edge-level block on the whole R2 product for this account, not a bucket-level 404/403), and `/cdn/<key>` (`api/cdn-object.js`), whose job is to fail over from the signed read to that same public domain, now 502s because both of its legs are down simultaneously. The "reads fail over to the public bucket domain meanwhile" hint text in `api/_lib/ops/object-storage-health.js` is temporarily false while this persists. Enabling R2 in the Cloudflare dashboard for the account behind `S3_BUCKET` is now urgent, not routine; this row's classification stays owner/Cloudflare-dashboard-only, unchanged. Separately, order 400 implemented the retry this row had deferred, but scoped narrowly: one extra attempt after a 200-500ms backoff, inside a single `gatherObjectStorageHealth()` invocation (`api/_lib/ops/object-storage-health.js`, tests in `tests/object-storage-health.test.js`). This does not contradict the 2026-10-13 re-measurement above that ruled out retry for the dominant DOWN signature: one retry a few hundred milliseconds later mathematically cannot turn 60-plus hours of genuine downtime into `ok`, so it cannot mute this sustained outage. It only absorbs a single transient network blip inside one invocation, which is the DEGRADED-flap case this row originally proposed it for, scoped exactly as the warning above required. | Cloudflare dashboard check (owner): this is the one for the account behind `S3_BUCKET`. The retry idea is ruled out for the dominant failure mode; do not file it as a work order without re-scoping to the DEGRADED-only signature | Every avatar, thumbnail and generated GLB reads through this bucket; `/cdn/*` and the forge both degrade when it flaps | `ops_alerts` table, queried 2026-10-11 and re-queried 2026-10-13 |
+| 36 | **Check Cloudflare R2's own status/dashboard for the bucket behind `S3_BUCKET`: it flaps between healthy and unreachable without any corresponding change on our side** (added 2026-10-11 by the evolve scout lane). Queried `ops_alerts` directly (columns: `signature, title, detail, severity, count, environment, first_seen, last_seen`): since 2026-09-08 the `object_storage` sensor (`api/_lib/ops/object-storage-health.js`) has logged roughly 167 "DEGRADED: Object storage (Cloudflare R2)" events (`bucket reachable but slow, signed list ~1095ms` against a 1000ms threshold) and about 18 "DOWN" events, interleaved with many "RESOLVED" events minutes apart showing a healthy 150 to 300ms. Reproduced live with production's own `S3_ACCESS_KEY_ID`/`S3_SECRET_ACCESS_KEY`/`S3_BUCKET`/`S3_ENDPOINT` (pulled via `node scripts/read-service-env.mjs` one variable at a time, used in-process, then deleted from disk immediately): `gatherObjectStorageHealth()` returned `{"status":"down","detail":"bucket unreachable: Please enable R2 through the Cloudflare Dashboard.","metrics":{"latencyMs":135,"credentialFault":false}}`, a 135ms response, i.e. the bucket answered fast but Cloudflare's own API told us R2 was not enabled, which cannot be true most of the time this account is clearly serving reads. Ruled out as a code cause: the credential is not rejected (`credentialFault:false`, no `SignatureDoesNotMatch`), and the AWS SDK's Node HTTP handler already defaults to keep-alive on (`keepAlive:true, maxSockets:50` in `@smithy/node-http-handler`), so this is not the classic missing-keep-alive latency bug. The probe itself is a single signed `ListObjectsV2Command` with no retry (`api/_lib/ops/object-storage-health.js:76-92`), so a genuinely transient one-off Cloudflare hiccup is indistinguishable from a sustained outage in this sensor today, which may be inflating the DOWN/DEGRADED counts above a one-retry version would show. Two independent things to do, neither owned by this reconciliation order: (a) check Cloudflare's own R2 dashboard/status page for the account behind `S3_BUCKET` for incidents correlated with the alert timestamps above (owner/dashboard access this agent does not have); (b) if the pattern looks like a true transient flap rather than sustained Cloudflare-side degradation, a future order could add one immediate retry before `gatherObjectStorageHealth()` classifies `down`/`degraded`, with a test proving a single synthetic failure followed by a synthetic success no longer pages. Not attempted here because the current evidence does not confidently distinguish "add a retry" from "mute a real recurring problem", and CLAUDE.md is explicit that a check does not get weakened without a dated, measured reason; this row exists so neither path gets silently skipped. 2026-10-13 (evolve scout lane, re-measured before considering it as a work order): queried `ops_alerts` again, grouped by failure text. The `"bucket unreachable: Please enable R2 through the Cloudflare Dashboard"` DOWN signature (the one live healthz reports right now) spans `first_seen` 2026-09-08T19:35Z to `last_seen` 2026-10-13T19:10Z (this instant), 58 alert rows, i.e. it has been recurring continuously for over a month and is still firing as this note is written: a sustained, ongoing condition, not a one-off transient blip. The retry idea in option (b) above does not apply to this dominant failure mode: retrying a single probe cannot paper over something that has been reproducing for 30-plus days, and doing so would be exactly the "mute a real recurring problem" case CLAUDE.md warns against, so no work order was written for it this pass. The narrower "DEGRADED: bucket reachable but slow" variant (option (b)'s original target) is a separate, lower-severity code path from this DOWN/unreachable one and was not re-measured this pass; a future pass that wants to pursue the retry idea should scope it to that DEGRADED signature specifically and re-confirm its flap pattern first, never the DOWN/unreachable signature above. 2026-10-13 (order 400, evolve loop, severity escalation): re-measured again, 18:10-18:30 UTC. The "flaps between healthy and unreachable" framing at the top of this row is now stale and understates the severity: the last `RESOLVED: Object storage (Cloudflare R2)` row is 2026-10-11T06:55:05Z, and every hourly sensor tick since then has been `DOWN` with zero resolves, a 35-plus-hour unbroken streak at measurement time (now 60-plus hours as this note is written, still ongoing). Two things that were working as designed have also failed: the public `*.r2.dev` domain itself now answers a Cloudflare-branded 403 (an edge-level block on the whole R2 product for this account, not a bucket-level 404/403), and `/cdn/<key>` (`api/cdn-object.js`), whose job is to fail over from the signed read to that same public domain, now 502s because both of its legs are down simultaneously. The "reads fail over to the public bucket domain meanwhile" hint text in `api/_lib/ops/object-storage-health.js` is temporarily false while this persists. Enabling R2 in the Cloudflare dashboard for the account behind `S3_BUCKET` is now urgent, not routine; this row's classification stays owner/Cloudflare-dashboard-only, unchanged. Separately, order 400 implemented the retry this row had deferred, but scoped narrowly: one extra attempt after a 200-500ms backoff, inside a single `gatherObjectStorageHealth()` invocation (`api/_lib/ops/object-storage-health.js`, tests in `tests/object-storage-health.test.js`). This does not contradict the 2026-10-13 re-measurement above that ruled out retry for the dominant DOWN signature: one retry a few hundred milliseconds later mathematically cannot turn 60-plus hours of genuine downtime into `ok`, so it cannot mute this sustained outage. It only absorbs a single transient network blip inside one invocation, which is the DEGRADED-flap case this row originally proposed it for, scoped exactly as the warning above required. 2026-10-14 (triage-lane sweep, production health sweep order): re-measured via live `/api/healthz`: `object_storage` still reports `down`, `"bucket unreachable: Please enable R2 through the Cloudflare Dashboard."`, and `forge_generation` is now also `down` ("no generation has STARTED in 196m") as a direct downstream symptom. The last `RESOLVED` row remains 2026-10-11T06:55:05Z, so this is now an unbroken streak of roughly 66 hours with zero resolves, up from the 60-plus hours measured in the order 400 note above. Still owner/Cloudflare-dashboard-only; no new row, no code change (the retry already shipped in order 400 cannot and should not mute a multi-day outage). | Cloudflare dashboard check (owner): this is the one for the account behind `S3_BUCKET`. The retry idea is ruled out for the dominant failure mode; do not file it as a work order without re-scoping to the DEGRADED-only signature | Every avatar, thumbnail and generated GLB reads through this bucket; `/cdn/*` and the forge both degrade when it flaps | `ops_alerts` table, queried 2026-10-11, 2026-10-13 and 2026-10-14 |
| 37 | **Ship the cron time-budget fix with the next deploy** (added 2026-10-11 by the evolve loop, order 403, otherwise complete and retired). `wrapCron()` in `api/_lib/http.js` now races every cron handler against a real time budget (`maxDurationMs`, wired from each handler's own `export const maxDuration * 1000`, defaulting to 120000ms) and answers a clean `503 { reason: 'cron_timeout' }` instead of hanging until Cloud Run's blind 900s ingress kill, which is what `/api/cron/x402-ring-leak-scan` was hitting repeatedly before order 400 landed its own fix. Verified locally: a real `http.createServer` hosting `wrapCron` with an 800ms budget and a never-resolving handler answered `503 cron_timeout` at 0.809s (not 900s, not instantly); the new `tests/http-cron-timeout.test.js` (4 tests) and the added `wrapCron` case in `tests/http-db-unavailable-boundary.test.js` pass 10/10, and the timeout test was proven red (indefinite hang) against the pre-fix code via a path-scoped `git stash` before the fix landed. No migration, no env var, nothing else needed; this is a pure code fix sitting on `evolve` waiting for row 1's routine deploy cadence to carry it to `main` and production. Ship with `npm run deploy:gcp:full` (row 1 already tracks the general "approve the next deploy" gate; this row exists only so the new behavior is named for verification). Post-deploy verification: re-run the step 0 query from `prompts/finish/403-auto-cron-timeout-enforcement.md` (`gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" textPayload:"maximum request timeout"' --freshness=24h --project aerial-vehicle-466722-p5`) after the deploy window; a healthy rollout shows that query going quiet (or near-quiet) while `textPayload:"exceeded its"` picks up any genuine stuck cron with its name attached, cross-referenced against order 400's `x402-ring-leak-scan` fix so a recurrence is distinguished from a different handler entirely. | one deploy (already covered by row 1's cadence) | Clean, diagnosable cron timeouts platform-wide instead of a blank "maximum request timeout" | `docs/ops/production-log-triage.md`, "HTTP timeout with no cron context" section |
| 38 | **Deploy `model-trellis2` (the TRELLIS.2 GPU worker), then point `MODEL_TRELLIS2_URL` at it** (re-measured 2026-10-11 by the evolve loop, best3d order 067, built and tested on 2026-10-09, not yet deployed). `workers/model-trellis2/` is complete (Dockerfile, `cloudbuild.yaml` pinned to `three-ws-build@`, 64 of 65 local tests passing, 1 skipped), the Forge lane registry already makes `trellis2` the default image-to-3D backend with the old chain (`trellis_selfhost`, `hunyuan3d`, `trellis`, `modly`, `triposg`) as failover, and the territory router already keeps the EU, the UK and South Korea off every Tencent lane regardless. Confirmed live today: `gcloud run services describe model-trellis2 --region us-central1` answers "Cannot find service", and `three-ws-api`'s env has no `MODEL_TRELLIS2_URL`, so the registry's default lane is currently unreachable and every image-to-3D request falls through to the next lane in the chain, same as the 2026-10-09 measurement. This is a production deploy (new Cloud Run GPU service, gate 2), so no agent runs it. Commands, in order: `gcloud builds submit --config workers/model-trellis2/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5 .` from the repo root, then `gcloud run services update three-ws-api --region us-central1 --update-env-vars=MODEL_TRELLIS2_URL=<service-url>` (never `--set-env-vars`, which replaces the whole set). After that, run one real image-to-3D generation and confirm the `forge_creations` row for it shows `backend: 'trellis2'`, and validate the returned GLB with `packages/glb-tools`; both are the two remaining Definition of done lines in the order file. | one build + one env update + one test generation | Closes best3d order 067; makes TRELLIS.2 (MIT, EU/UK/KR-safe) the live default instead of the untested fallback chain | [067-best3d-01-trellis2-default-lane.md](../067-best3d-01-trellis2-default-lane.md), [best3d-PROGRESS.md](best3d-PROGRESS.md) |
| 39 | **Deploy `model-pixal3d` (the Pixal3D fidelity-lane GPU worker), then point `MODEL_PIXAL3D_URL` at it** (added 2026-10-11 by the evolve loop, best3d order 068, built and tested same day). `workers/model-pixal3d/` is complete (Dockerfile, `cloudbuild.yaml` pinned to `three-ws-build@`, 81 of 82 local tests passing, 1 skipped), `api/forge.js`/`api/_lib/forge-tiers.js`/`api/_providers/gcp.js` already register and wire the `pixal3d` backend (1-4 photos, `view_labels`, `fill_other_sides`, opt-in only: never a named default or a failover target), and `src/forge.js`/`pages/forge.html` already ship the "Match my photo" engine button and panel. Confirmed live today: `gcloud run services describe model-pixal3d --region us-central1` answers "Cannot find service", and `three-ws-api`'s env has no `MODEL_PIXAL3D_URL`, so the lane reports `backend_unconfigured` today and the engine button does not render in the catalog (verified against a synthetic catalog entry in a local browser session, since the real worker isn't deployed yet). This is a production deploy (new Cloud Run GPU service, gate 2), so no agent runs it. Commands, in order (from `workers/model-pixal3d/README.md`): `workers/model-pixal3d/stage_weights.sh` (one time, idempotent weight mirror), then `gcloud builds submit --config workers/model-pixal3d/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5 --substitutions=SHORT_SHA=manual$(date +%s) .` from the repo root, then `gcloud run services update three-ws-api --region us-central1 --update-env-vars MODEL_PIXAL3D_URL=$(gcloud run services describe model-pixal3d --region us-central1 --format='value(status.url)')` (never `--set-env-vars`, which replaces the whole set). After that, run one real 3-image generation with `backend: "pixal3d"` and confirm the `forge_creations` row records all three input URLs and the returned GLB is valid; that is the order's last unverified Definition of done line. | one weight stage + one build + one env update + one test generation | Closes best3d order 068; makes "Match my photo" (pixel-aligned fidelity to the caller's own photos, MIT + Apache-2.0) live in the Forge engine grid | [068-best3d-02-pixal3d-multiview.md](../068-best3d-02-pixal3d-multiview.md), [best3d-PROGRESS.md](best3d-PROGRESS.md) |