Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): classify /api/chat 503s under the known Vertex billing hold

nirholasf6f116552642
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 36a2fc559..44ebb9f7b 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -1437,6 +1437,38 @@ ERROR 502 POST /api/pump/launch-agent
---
+## 🔴 HTTP 503 `/api/chat`: `chat-fallback-chain-exhausted-503` (§chat-fallback-chain-exhausted)
+
+- **Source:** `/api/chat`'s failover chain (groq, then every openrouter
+ sibling, then nvidia, then the `vertex-gemini` anchor) answers 503
+ `rate_limited` only once every rung including the anchor has failed
+ ([api/chat.js](../../api/chat.js), the final branch around line 936). The
+ `vertex-gemini` anchor is deliberately exempt from the fallback cooldown and
+ the `MAX_FALLBACK_ATTEMPTS` cap specifically so a healthy anchor should never
+ let chat 503 (see `buildFallbackChain` in the same file).
+- **What it means:** added 2026-10-24 after a deep sweep found 21 `/api/chat`
+ 503s in 6h. Traced one occurrence end to end via `npm run logs`: groq 429,
+ every openrouter sibling 429, nvidia 404 ("Function ... Not found for
+ account"), then the vertex-gemini anchor 403 on the same GCP billing dunning
+ hold ("Lightning dunning decision is deny" on `aerial-vehicle-466722-p5`,
+ row 20) that blocks the anchor for `three-ws-ai-providers-unavailable-5xx`
+ above. With the anchor itself billing-blocked there is nothing left to fail
+ over to. The nvidia 404 is a separate, narrower signal: that `NVIDIA_API_KEY`
+ account has lost access to the configured NIM function server-side (not a
+ stale model id in this repo; the model string is unchanged and groq's prior
+ 429 already shows free-tier demand, not a routing bug, is the bottleneck).
+- **Resolve (owner, billing/credential):** restoring the Vertex billing hold
+ (row 20) clears this family the same way it clears
+ `three-ws-ai-providers-unavailable-5xx`. Separately, check the NVIDIA NIM
+ account behind `NVIDIA_API_KEY` at the build.nvidia.com console if the 404
+ "Not found for account" line persists; that is an upstream account-state
+ check, not a code fix. Nothing to fix in code unless a *new* rung (not
+ groq/openrouter/nvidia/vertex-gemini) is the one failing.
+- **Monitor signature:** `chat-fallback-chain-exhausted-503` in
+ [scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), classified `owner`.
+
+---
+
## 🔴 HTTP 502 `/api/3d/generate`: `forge-image-ladder-exhausted-502` (§forge-image-ladder-exhausted)
- **Source:** `/api/3d/generate` self-calls `/api/forge`, whose reference-image
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index b31934766..4dac08cbb 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -438,6 +438,26 @@ const KNOWN_HTTP_SIGNATURES = [
class: 'owner',
action: `/api/vision, /api/tts/synthesize and /api/v1/ai/text-to-3d each walk a provider chain (NVIDIA, then Vertex Gemini, then OpenAI) and answer an honest 502/503 when every rung fails. The GCP billing hold "Lightning dunning decision is deny" on aerial-vehicle-466722-p5 blocks the Vertex/Gemini rung (production-100-OWNER-ACTIONS.md row 20) and the OpenAI key is dead (row 6); together that leaves only the NVIDIA rung, which the forge_generation health sensor shows intermittently failing on free-tier NVCF capacity. Nothing to fix in code: restoring either billing-held provider clears this family, same as the identical cryptocurrency-cv pattern (ccv-ai-providers-unavailable-503). ${RUNBOOK} §three-ws-ai-providers-unavailable.`,
},
+ {
+ id: 'chat-fallback-chain-exhausted-503',
+ // 2026-10-24 triage-lane sweep: 21 /api/chat 503s in 6h. Traced one
+ // occurrence end to end via npm run logs: groq 429 -> every openrouter
+ // sibling 429 -> nvidia 404 "Function ... Not found for account" -> the
+ // vertex-gemini anchor (api/chat.js's deliberately-unevictable last rung,
+ // see buildFallbackChain) 403 "Lightning dunning decision is deny" on
+ // aerial-vehicle-466722-p5, same billing hold as
+ // three-ws-ai-providers-unavailable-5xx (row 20). Once the anchor itself
+ // is billing-blocked there is nothing left to fail over to, so chat.js's
+ // final branch (api/chat.js ~line 936) correctly answers 503 rate_limited
+ // instead of a fake reply. The nvidia 404 is a separate, narrower signal
+ // (that NVAPI key's account lost access to the configured NIM function,
+ // not a stale id in this repo: the model string is unchanged and groq's
+ // prior 429 already proves free-tier demand is the bottleneck); it is
+ // owner-side NVIDIA account state, not fixable by editing code here.
+ test: (g) => g.service === 'three-ws-api' && g.status === 503 && g.path === '/api/chat',
+ class: 'owner',
+ action: `/api/chat's failover chain (groq -> openrouter siblings -> nvidia -> vertex-gemini anchor) answers 503 rate_limited only once every rung including the vertex-gemini anchor has failed. The anchor is the credits-funded lane with no free-tier cap, deliberately exempt from the fallback cooldown/cap specifically so chat should not 503 while it is healthy; a 503 here means the anchor itself is down, which the GCP billing hold "Lightning dunning decision is deny" on aerial-vehicle-466722-p5 explains (production-100-OWNER-ACTIONS.md row 20, same cause as three-ws-ai-providers-unavailable-5xx). Separately check whether nvidia is answering 404 "Function ... Not found for account" in the same window (npm run logs -- -s three-ws-api --app --since 6h --grep "Not found for account"): that is the configured NVIDIA_API_KEY's account losing access to its NIM function and needs the owner to check the build.nvidia.com console, not a code fix. Nothing to fix in code unless a NEW rung (not groq/openrouter/nvidia/vertex-gemini) is the one failing. ${RUNBOOK} §chat-fallback-chain-exhausted.`,
+ },
{
id: 'forge-image-ladder-exhausted-502',
// 2026-10-12 triage-lane sweep: /api/3d/generate self-calls /api/forge,
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index bd89fd36c..53fe68306 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -148,6 +148,17 @@ describe('gcp-triage HTTP classification', () => {
expect(finding.class).toBe('owner');
});
+ // 2026-10-24 triage-lane sweep: 21 /api/chat 503s in 6h, previously
+ // unclassified (fell through to investigate) despite sharing the same
+ // root cause as three-ws-ai-providers-unavailable-5xx (the GCP billing
+ // hold blocking the vertex-gemini anchor, row 20 in
+ // production-100-OWNER-ACTIONS.md).
+ it('classifies a chat fallback-chain exhaustion as owner-gated', () => {
+ const [finding] = buildFindings([request({ status: 503, path: '/api/chat' })]);
+ expect(finding.signature).toBe('chat-fallback-chain-exhausted-503');
+ expect(finding.class).toBe('owner');
+ });
+
it('classifies a holder-distribution RPC exhaustion 503 as self-healing', () => {
const [finding] = buildFindings([request({ status: 503, path: '/api/crypto/holders' })]);
expect(finding.signature).toBe('holders-rpc-exhausted-503');