Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): classify cryptocurrency.cv AI-provider outages as owner billing, not code faults

nirholas9ee87d38582f
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 83c263a79..c95de7467 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -451,6 +451,50 @@ HTTP 502/503 GET|POST /api/x402/*, /api/mcp ua: threews-x402-autonomous/1.0 or
---
+## 🔴 HTTP 503 on cryptocurrency.cv AI routes: `ccv-ai-providers-unavailable` / `ccv-ai-providers-unavailable-503`
+
+```
+HTTP 503 GET https://cryptocurrency.cv/api/narratives
+AI provider groq unavailable (AIRateLimitError), trying next provider... rate-limited / out of quota (429)
+AI provider gemini unavailable (AIAuthError) ... [403 Forbidden] Lightning dunning decision is deny for project: projects/93741856042
+```
+
+- **Service:** `cryptocurrency-cv` (serves https://cryptocurrency.cv). Source
+ lives in its own repository, `nirholas/cryptocurrency.cv`, not in this one.
+ Its AI routes walk one provider chain in `src/lib/ai-provider.ts`.
+- **Root cause (2026-10-08 sweep): every provider is out at once, and each for
+ a billing reason, not a code one.**
+ - Groq: the free tier's 200k tokens/day budget is spent. Before the
+ 2026-10-08 fix, `/api/sentiment` re-ran a ~7k-token prompt on every poll, so
+ one client polling a single asset (91 calls in 6h) spent the day's budget by
+ itself.
+ - Gemini and Vertex: refused with `Lightning dunning decision is deny`, a
+ billing hold on project `aerial-vehicle-466722-p5` (number 93741856042). It
+ reads like an IAM error and is not one; no key or role change clears it.
+ - OpenRouter: both `OPENROUTER_FALLBACK_KEYS` belong to one account whose
+ usage exceeds its credits. Revisions built before 2026-09-30 never even
+ read that variable.
+- **What users see after the 2026-10-08 fixes:** `/api/sentiment` answers 200
+ from a keyword reading of the headlines (`meta.method: "keyword"`,
+ `meta.degraded: true`), `/api/narratives` serves its last good analysis for
+ up to an hour, and every AI route caches one analysis for five minutes, so a
+ poller no longer drains the provider. A 503 now only appears on an AI route
+ with nothing cached, carries `Retry-After`, and no longer echoes the
+ provider's raw error (which used to include the Groq organization id).
+- **Resolve (owner, billing): restore any ONE provider.** The chain recovers on
+ the next request with no redeploy. Cheapest first: clear the GCP billing hold
+ on `aerial-vehicle-466722-p5` (restores Gemini and Vertex, and with them the
+ okx-chat-bot's Vertex lane), top up the OpenRouter account, or move
+ `GROQ_API_KEY` to a paid Groq tier.
+- **Confirm:** `curl -s https://cryptocurrency.cv/api/narratives` returns
+ narratives, and `curl -s 'https://cryptocurrency.cv/api/sentiment' | jq .meta.method`
+ prints `"ai"`.
+- **Monitor signatures:** `ccv-ai-providers-unavailable` (app log) and
+ `ccv-ai-providers-unavailable-503` (request log) in
+ [scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), both classified `owner`.
+
+---
+
## 🟡 HTTP 502 on `/api/coin/*` — `coingecko-quota-exhausted-502`
```
@@ -844,8 +888,11 @@ HTTP 503 GET /api/community/worlds body: {"error":"cc_unconfigured","error_des
("Holder check is offline") with a working one-click path into the coin's
open world. Before 2026-08-08 this was the one dead end: the gate showed
the raw upstream string with retry-only actions that could never succeed.
- - `/clash/*` pins the battle tabs on a human message and stops the 5s poll
- on the first `cc_unconfigured` answer.
+ - `/clash/state` and `/clash/leaderboard` fail over to the same trending
+ feed (since 2026-10-08), with the roster frozen per round, so the arena
+ stays playable. They answer `cc_unconfigured` only when trending is down
+ too, and the page then pins the battle tabs on a human message and stops
+ the 5s poll.
- **Resolve (owner, credential):** provision a CoinCommunities API key
(api.coin-communities.xyz), then:
`gcloud run services update three-ws-api --region us-central1 --update-env-vars CC_API_KEY=<key>`.
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index d5c91c857..cecd9f8ee 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -103,6 +103,15 @@ const KNOWN_SIGNATURES = [
class: 'self-healing',
action: `Cache circuit breaker riding out an Upstash blip; reads keep serving. Optional: same-region cache store. ${RUNBOOK} §cache.`,
},
+ {
+ // Service-scoped, so it sits ahead of the broad helius-backoff match
+ // ("rate-limited") that would otherwise claim these lines first.
+ id: 'ccv-ai-providers-unavailable',
+ match: /AI provider \w+ unavailable|\[ai\] every provider unavailable|\[sentiment\] AI analysis unavailable/i,
+ services: ['cryptocurrency-cv'],
+ class: 'owner',
+ action: `cryptocurrency.cv walked its whole AI chain and every provider refused: Groq's free tier spent its daily token budget, Gemini and Vertex answer "Lightning dunning decision is deny" (a billing hold on project aerial-vehicle-466722-p5, not an IAM or key problem), and the OpenRouter account behind OPENROUTER_FALLBACK_KEYS is out of credit. /api/sentiment degrades to a keyword reading and /api/narratives to its last good analysis, so this is noise until the owner restores one provider. ${RUNBOOK} §ccv-ai-providers-unavailable.`,
+ },
{
id: 'helius-backoff',
match: /helius quota|rate.?limited|refresh deferred \(transient upstream\)/i,
@@ -241,6 +250,17 @@ const KNOWN_HTTP_SIGNATURES = [
class: 'owner',
action: `/api/community/worlds returns its designed 503 cc_unconfigured: CC_API_KEY exists nowhere (Cloud Run env, .env, Secret Manager — swept 2026-07-26). The coin-worlds lobby stays empty until the owner provisions a CoinCommunities API key (api.coin-communities.xyz), then: gcloud run services update three-ws-api --region us-central1 --update-env-vars CC_API_KEY=<key>. Harmless noise until then. ${RUNBOOK} §cc-unconfigured.`,
},
+ {
+ id: 'ccv-ai-providers-unavailable-503',
+ // The AI routes on cryptocurrency.cv answer a designed 503 with
+ // Retry-After when no provider can serve and nothing is cached.
+ test: (g) =>
+ g.service === 'cryptocurrency-cv' &&
+ g.status === 503 &&
+ /^\/api\/(v1\/)?(sentiment|narratives|summarize|digest|entities|factcheck|clickbait|ask|signals|relationships|ai\/|analytics\/influencers|onchain\/events)/.test(g.path),
+ class: 'owner',
+ action: `A cryptocurrency.cv AI route found every provider out (Groq daily token budget spent, Gemini and Vertex refused by the GCP billing hold "Lightning dunning decision is deny", OpenRouter out of credit) with no cached analysis to serve. Nothing in code is broken and no redeploy fixes it: the owner restores ONE provider (clear the billing hold on aerial-vehicle-466722-p5, top up OpenRouter, or move GROQ_API_KEY to a paid tier) and the routes recover on the next request. Confirm with: curl -s https://cryptocurrency.cv/api/narratives. ${RUNBOOK} §ccv-ai-providers-unavailable.`,
+ },
{
id: 'watsonx-unconfigured-503',
test: (g) => g.status === 503 && (g.path === '/api/galaxy' || g.path.startsWith('/api/galaxy/')),
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index 5c0b478ae..d7f6951ad 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -33,6 +33,14 @@ describe('gcp-triage classify', () => {
expect(sig?.id).toBe('db-storage-cap');
});
+ it('classifies the cryptocurrency-cv AI chain exhaustion as owner-gated only on that service', () => {
+ const line = 'AI provider groq unavailable (AIRateLimitError), trying next provider... rate-limited / out of quota (429)';
+ const onCcv = classify(line, ['cryptocurrency-cv']);
+ expect(onCcv?.id).toBe('ccv-ai-providers-unavailable');
+ expect(onCcv?.class).toBe('owner');
+ expect(classify(line, ['three-ws-api'])?.id).not.toBe('ccv-ai-providers-unavailable');
+ });
+
it('returns null for an unknown message', () => {
expect(classify('some brand new failure nobody has seen', ['three-ws-api'])).toBeNull();
});
@@ -56,6 +64,18 @@ describe('gcp-triage HTTP classification', () => {
expect(finding.class).toBe('owner');
});
+ it('classifies a cryptocurrency-cv AI route 503 as owner-gated', () => {
+ const [finding] = buildFindings([request({ service: 'cryptocurrency-cv', status: 503, path: '/api/narratives' })]);
+ expect(finding.signature).toBe('ccv-ai-providers-unavailable-503');
+ expect(finding.class).toBe('owner');
+ });
+
+ it('leaves a cryptocurrency-cv 503 on a non-AI route for investigation', () => {
+ const [finding] = buildFindings([request({ service: 'cryptocurrency-cv', status: 503, path: '/api/news' })]);
+ expect(finding.signature).not.toBe('ccv-ai-providers-unavailable-503');
+ expect(finding.class).toBe('investigate');
+ });
+
it('classifies pump curve fallback exhaustion as self-healing', () => {
const [finding] = buildFindings([request({ status: 502, path: '/api/pump/curve' })]);
expect(finding.signature).toBe('pump-curve-rpc-unavailable');