Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): classify holder-distribution 503s as RPC-exhaustion self-healing, not investigate

nirholasf6b9f5343537
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index a5909e4ea..f3de1c106 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -911,6 +911,14 @@ ws error: Unexpected server response: 301
`pump-curve-rpc-unavailable` (self-healing). Escalate it only when it persists
after `healthz.rpc_lanes` shows at least one healthy lane; no transaction was
broadcast and retrying cannot spend user funds.
+- **Holder-distribution symptom:** `HTTP 503 GET /api/crypto/holders` answers
+ `upstream_unavailable` when both holder paths (Helius, then keyless
+ `getTokenLargestAccounts`) fail for a mint with no last-good report cached
+ yet (24h TTL), common for a mint nobody queried before during a lane
+ outage. The monitor classifies this `holders-rpc-exhausted-503`
+ (self-healing). Escalate only if it persists after `rpc_lanes` reads ok;
+ [api/_lib/crypto-token-holders.js](../../api/_lib/crypto-token-holders.js)
+ is the composition logic if it does.
---
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index 7e49a2e39..5b9119ac4 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -288,6 +288,17 @@ const KNOWN_HTTP_SIGNATURES = [
class: 'owner',
action: `A cryptocurrency.cv AI route found every provider out (Groq daily token budget spent, Gemini and Vertex refused by the GCP billing hold "Lightning dunning decision is deny", OpenRouter out of credit) with no cached analysis to serve. Nothing in code is broken and no redeploy fixes it: the owner restores ONE provider (clear the billing hold on aerial-vehicle-466722-p5, top up OpenRouter, or move GROQ_API_KEY to a paid tier) and the routes recover on the next request. Confirm with: curl -s https://cryptocurrency.cv/api/narratives. ${RUNBOOK} §ccv-ai-providers-unavailable.`,
},
+ {
+ id: 'holders-rpc-exhausted-503',
+ // composeTokenHolders() returns upstream_down, and the handler answers 503,
+ // only when the mint read and every holder path (Helius, then keyless
+ // getTokenLargestAccounts) failed AND no last-good copy was cached for this
+ // mint+limit yet, overwhelmingly a brand-new/rarely-queried mint hit while
+ // every RPC lane is exhausted (healthz rpc_lanes), not a code fault.
+ test: (g) => g.service === 'three-ws-api' && g.status === 503 && g.path === '/api/crypto/holders',
+ class: 'self-healing',
+ action: `The free holder-distribution route exhausted Helius and the keyless RPC fallback for a mint with no cached last-good report, so it answered an honest 503 upstream_unavailable rather than a guess. Check healthz rpc_lanes/helius: this fires in bursts while every paid Solana RPC lane is cooling down (see ${RUNBOOK} "Every paid Solana RPC lane exhausted at once"). Investigate only if it persists after rpc_lanes reads ok. ${RUNBOOK} §solana-rpc.`,
+ },
{
id: 'watsonx-unconfigured-503',
test: (g) => g.status === 503 && (g.path === '/api/galaxy' || g.path.startsWith('/api/galaxy/')),
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index 75a651587..77671b2c3 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -105,6 +105,12 @@ describe('gcp-triage HTTP classification', () => {
expect(finding.signature).toBe('pump-price-history-upstreams-502');
expect(finding.class).toBe('env-action');
});
+
+ it('classifies a holder-distribution RPC exhaustion 503 as self-healing', () => {
+ const [finding] = buildFindings([request({ status: 503, path: '/api/crypto/holders' })]);
+ expect(finding.signature).toBe('holders-rpc-exhausted-503');
+ expect(finding.class).toBe('self-healing');
+ });
});
describe('gcp-triage page failure summary', () => {