Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(triage): add the forge-image-ladder-exhausted-502 signature for /api/3d/generate

nirholas06d1667d8542
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 19eb9aca7..a9f9b57c6 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -1323,6 +1323,38 @@ ERROR 502 POST /api/pump/launch-agent
---
+## 🔴 HTTP 502 `/api/3d/generate`: `forge-image-ladder-exhausted-502` (§forge-image-ladder-exhausted)
+
+- **Source:** `/api/3d/generate` self-calls `/api/forge`, whose reference-image
+ step ([api/_lib/forge-reference-image.js](../../api/_lib/forge-reference-image.js))
+ tries Vertex, then falls through to the text-to-image ladder
+ ([api/_mcp3d/text-to-image.js](../../api/_mcp3d/text-to-image.js)): NIM FLUX,
+ HF fal-ai, HF nscale, keyless Pollinations, Replicate. A 502 means every rung
+ the request reached failed.
+- **What it means:** root-caused on 2026-10-12. Vertex 403s on the GCP billing
+ dunning hold (row 20); HF fal-ai and HF nscale both 402 "no remaining
+ credits" (a separate credit pool from the LLM lanes' `HF_TOKEN` usage in
+ row 6, needs its own funding, see OWNER-ACTIONS). Before the 2026-10-12 fix
+ ("stop the Vertex billing hold and HF credit exhaustion from re-paying their
+ own timeout on every image request"), none of those three dead lanes had a
+ circuit breaker, so every request re-paid their full timeouts before ever
+ reaching Pollinations, often blowing the shared ladder budget before
+ Pollinations ran at all, which is what turned an owner-gated billing/credit
+ issue into a request-killing 502 storm instead of a quiet fallback to the
+ free lane.
+- **Resolve (owner, billing/credential):** fund the Vertex billing hold (row
+ 20) and/or top up the HF Inference Providers credit balance (fal-ai,
+ nscale). Either one clears most of this family; both together clear it.
+ Nothing further to fix in code unless a *new* rung fails, or Pollinations
+ itself starts failing with no paid Replicate token configured as a backstop:
+ check `npm run logs -- -s three-ws-api --app --since 1h --grep "falling
+ back"` for which rung is actually down before assuming it is the same two
+ causes above.
+- **Monitor signature:** `forge-image-ladder-exhausted-502` in
+ [scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), classified `owner`.
+
+---
+
## 🟢 HTTP 503 `/api/agents/<id>/solana/networth`: `agent-networth-rpc-unavailable-503` (§solana-rpc)
- **Source:** `handleNetWorth()` in
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index 4e3c653d2..f3599572d 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -371,6 +371,33 @@ const KNOWN_HTTP_SIGNATURES = [
class: 'owner',
action: `/api/vision and /api/tts/synthesize each walk a provider chain (NVIDIA, then Vertex Gemini, then OpenAI) and answer an honest 502/503 when every rung fails. The GCP billing hold "Lightning dunning decision is deny" on aerial-vehicle-466722-p5 blocks the Vertex/Gemini rung (production-100-OWNER-ACTIONS.md row 20) and the OpenAI key is dead (row 6); together that leaves only the NVIDIA rung, which the forge_generation health sensor shows intermittently failing on free-tier NVCF capacity. Nothing to fix in code: restoring either billing-held provider clears this family, same as the identical cryptocurrency-cv pattern (ccv-ai-providers-unavailable-503). ${RUNBOOK} §three-ws-ai-providers-unavailable.`,
},
+ {
+ id: 'forge-image-ladder-exhausted-502',
+ // 2026-10-12 triage-lane sweep: /api/3d/generate self-calls /api/forge,
+ // whose reference-image step (api/_lib/forge-reference-image.js) tries
+ // Vertex, then falls through to the text-to-image.js ladder (NIM FLUX,
+ // HF fal-ai, HF nscale, keyless Pollinations, Replicate). Root-caused one
+ // occurrence end to end: Vertex 403 (the GCP billing dunning hold, row
+ // 20), NIM FLUX content-filtered on that prompt, HF fal-ai and HF nscale
+ // both 402 "no remaining credits" (production-100-OWNER-ACTIONS.md: fold
+ // into row 6 or add a dedicated row; the LLM lanes' HF_TOKEN credit is a
+ // separate pool from this image lane's). Before the 2026-10-12 fix
+ // (commit "stop the Vertex billing hold and HF credit exhaustion from
+ // re-paying their own timeout on every image request"), none of those
+ // three dead lanes had a circuit breaker, so EVERY request re-paid their
+ // full timeouts before ever reaching Pollinations, often blowing the
+ // shared ladder budget before Pollinations ran at all and surfacing a 502
+ // instead of a served image. That amplification is fixed: a 401/402/403
+ // now parks the lane for AUTH_COOLDOWN_SECONDS (5 min), so a request
+ // should reach Pollinations fast. A 502 after the fix means Pollinations
+ // itself failed (its own free-tier throttle) or NIM was also down with no
+ // fallback left, which would be rare, not the storm this was. Nothing
+ // further to fix in code unless Pollinations itself is now the one
+ // failing; that is a new, different signature.
+ test: (g) => g.service === 'three-ws-api' && g.status >= 500 && g.path === '/api/3d/generate',
+ class: 'owner',
+ action: `/api/3d/generate's provider chain (Vertex → NIM FLUX → HF fal-ai → HF nscale → Pollinations → Replicate) answers 502 generation_failed when every rung fails. Check which rung: curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="ai_providers" or .name=="object_storage")', and grep the app logs for "falling back"/"falling through" in the same window (npm run logs -- -s three-ws-api --app --since 1h --grep "falling back"). Vertex billing (row 20) and HF image-provider credits are owner-gated funding; nothing to fix in code unless a NEW rung (not Vertex/NIM/HF/Pollinations) is the one failing, or Pollinations itself is down with no paid Replicate token configured as a backstop. ${RUNBOOK} §forge-image-ladder-exhausted.`,
+ },
{
id: 'agent-networth-rpc-unavailable-503',
// api/agents/solana-wallet.js's handleNetWorth explicitly documents this: