Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(triage): classify /api/chat 503s under the known Vertex billing hold
nirholasf6f116552642
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.mdindex 36a2fc559..44ebb9f7b 100644--- a/docs/ops/production-log-triage.md+++ b/docs/ops/production-log-triage.md@@ -1437,6 +1437,38 @@ ERROR 502 POST /api/pump/launch-agent---+## 🔴 HTTP 503 `/api/chat`: `chat-fallback-chain-exhausted-503` (§chat-fallback-chain-exhausted)++- **Source:** `/api/chat`'s failover chain (groq, then every openrouter+ sibling, then nvidia, then the `vertex-gemini` anchor) answers 503+ `rate_limited` only once every rung including the anchor has failed+ ([api/chat.js](../../api/chat.js), the final branch around line 936). The+ `vertex-gemini` anchor is deliberately exempt from the fallback cooldown and+ the `MAX_FALLBACK_ATTEMPTS` cap specifically so a healthy anchor should never+ let chat 503 (see `buildFallbackChain` in the same file).+- **What it means:** added 2026-10-24 after a deep sweep found 21 `/api/chat`+ 503s in 6h. Traced one occurrence end to end via `npm run logs`: groq 429,+ every openrouter sibling 429, nvidia 404 ("Function ... Not found for+ account"), then the vertex-gemini anchor 403 on the same GCP billing dunning+ hold ("Lightning dunning decision is deny" on `aerial-vehicle-466722-p5`,+ row 20) that blocks the anchor for `three-ws-ai-providers-unavailable-5xx`+ above. With the anchor itself billing-blocked there is nothing left to fail+ over to. The nvidia 404 is a separate, narrower signal: that `NVIDIA_API_KEY`+ account has lost access to the configured NIM function server-side (not a+ stale model id in this repo; the model string is unchanged and groq's prior+ 429 already shows free-tier demand, not a routing bug, is the bottleneck).+- **Resolve (owner, billing/credential):** restoring the Vertex billing hold+ (row 20) clears this family the same way it clears+ `three-ws-ai-providers-unavailable-5xx`. Separately, check the NVIDIA NIM+ account behind `NVIDIA_API_KEY` at the build.nvidia.com console if the 404+ "Not found for account" line persists; that is an upstream account-state+ check, not a code fix. Nothing to fix in code unless a *new* rung (not+ groq/openrouter/nvidia/vertex-gemini) is the one failing.+- **Monitor signature:** `chat-fallback-chain-exhausted-503` in+ [scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), classified `owner`.++---+## 🔴 HTTP 502 `/api/3d/generate`: `forge-image-ladder-exhausted-502` (§forge-image-ladder-exhausted)- **Source:** `/api/3d/generate` self-calls `/api/forge`, whose reference-imagediff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjsindex b31934766..4dac08cbb 100644--- a/scripts/gcp-triage.mjs+++ b/scripts/gcp-triage.mjs@@ -438,6 +438,26 @@ const KNOWN_HTTP_SIGNATURES = [class: 'owner',action: `/api/vision, /api/tts/synthesize and /api/v1/ai/text-to-3d each walk a provider chain (NVIDIA, then Vertex Gemini, then OpenAI) and answer an honest 502/503 when every rung fails. The GCP billing hold "Lightning dunning decision is deny" on aerial-vehicle-466722-p5 blocks the Vertex/Gemini rung (production-100-OWNER-ACTIONS.md row 20) and the OpenAI key is dead (row 6); together that leaves only the NVIDIA rung, which the forge_generation health sensor shows intermittently failing on free-tier NVCF capacity. Nothing to fix in code: restoring either billing-held provider clears this family, same as the identical cryptocurrency-cv pattern (ccv-ai-providers-unavailable-503). ${RUNBOOK} §three-ws-ai-providers-unavailable.`,},+ {+ id: 'chat-fallback-chain-exhausted-503',+ // 2026-10-24 triage-lane sweep: 21 /api/chat 503s in 6h. Traced one+ // occurrence end to end via npm run logs: groq 429 -> every openrouter+ // sibling 429 -> nvidia 404 "Function ... Not found for account" -> the+ // vertex-gemini anchor (api/chat.js's deliberately-unevictable last rung,+ // see buildFallbackChain) 403 "Lightning dunning decision is deny" on+ // aerial-vehicle-466722-p5, same billing hold as+ // three-ws-ai-providers-unavailable-5xx (row 20). Once the anchor itself+ // is billing-blocked there is nothing left to fail over to, so chat.js's+ // final branch (api/chat.js ~line 936) correctly answers 503 rate_limited+ // instead of a fake reply. The nvidia 404 is a separate, narrower signal+ // (that NVAPI key's account lost access to the configured NIM function,+ // not a stale id in this repo: the model string is unchanged and groq's+ // prior 429 already proves free-tier demand is the bottleneck); it is+ // owner-side NVIDIA account state, not fixable by editing code here.+ test: (g) => g.service === 'three-ws-api' && g.status === 503 && g.path === '/api/chat',+ class: 'owner',+ action: `/api/chat's failover chain (groq -> openrouter siblings -> nvidia -> vertex-gemini anchor) answers 503 rate_limited only once every rung including the vertex-gemini anchor has failed. The anchor is the credits-funded lane with no free-tier cap, deliberately exempt from the fallback cooldown/cap specifically so chat should not 503 while it is healthy; a 503 here means the anchor itself is down, which the GCP billing hold "Lightning dunning decision is deny" on aerial-vehicle-466722-p5 explains (production-100-OWNER-ACTIONS.md row 20, same cause as three-ws-ai-providers-unavailable-5xx). Separately check whether nvidia is answering 404 "Function ... Not found for account" in the same window (npm run logs -- -s three-ws-api --app --since 6h --grep "Not found for account"): that is the configured NVIDIA_API_KEY's account losing access to its NIM function and needs the owner to check the build.nvidia.com console, not a code fix. Nothing to fix in code unless a NEW rung (not groq/openrouter/nvidia/vertex-gemini) is the one failing. ${RUNBOOK} §chat-fallback-chain-exhausted.`,+ },{id: 'forge-image-ladder-exhausted-502',// 2026-10-12 triage-lane sweep: /api/3d/generate self-calls /api/forge,diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.jsindex bd89fd36c..53fe68306 100644--- a/tests/gcp-triage.test.js+++ b/tests/gcp-triage.test.js@@ -148,6 +148,17 @@ describe('gcp-triage HTTP classification', () => {expect(finding.class).toBe('owner');});+ // 2026-10-24 triage-lane sweep: 21 /api/chat 503s in 6h, previously+ // unclassified (fell through to investigate) despite sharing the same+ // root cause as three-ws-ai-providers-unavailable-5xx (the GCP billing+ // hold blocking the vertex-gemini anchor, row 20 in+ // production-100-OWNER-ACTIONS.md).+ it('classifies a chat fallback-chain exhaustion as owner-gated', () => {+ const [finding] = buildFindings([request({ status: 503, path: '/api/chat' })]);+ expect(finding.signature).toBe('chat-fallback-chain-exhausted-503');+ expect(finding.class).toBe('owner');+ });+it('classifies a holder-distribution RPC exhaustion 503 as self-healing', () => {const [finding] = buildFindings([request({ status: 503, path: '/api/crypto/holders' })]);expect(finding.signature).toBe('holders-rpc-exhausted-503');