Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(triage): classify designed kol-trades, sentiment-pulse and redis-proxy upstream failures as self-healing so the sweep stops flagging them
nirholas6b52cfc9dbe4
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.mdindex 4abfcde98..36a2fc559 100644--- a/docs/ops/production-log-triage.md+++ b/docs/ops/production-log-triage.md@@ -1469,6 +1469,18 @@ ERROR 502 POST /api/pump/launch-agent---+## 🟢 HTTP 502 `/api/kol/trades`: `kol-trades-upstream-502` (§solana-rpc)++`api/kol/trades.js` answers a designed 502 with `Retry-After: 15` when `fetchKolTrades()` fails and no last-good copy exists for the mint. It fires in the same bursts as the holders 503 while every Solana RPC lane cools down (healthz `rpc_lanes`). No action; investigate only if it persists after `rpc_lanes` reads ok.++## 🟢 HTTP 502 `/api/social/sentiment-pulse`: `sentiment-pulse-pumpfun-502` (§pump-trending-upstreams)++By design (`api/social/sentiment-pulse.js`): a pump.fun outage with no caller-supplied texts answers 502 `upstream_unavailable`. Recovers when pump.fun answers.++## 🟢 HTTP 5xx on `three-ws-redis-proxy`: `redis-proxy-upstash-timeout` (§redis-proxy)++2026-10-23 22:33Z burst: the proxy's `GenServer.call` to Upstash timed out (`SET rpccool:v1`). The API's cache and rate limiter fall back to Postgres/in-memory and healthz `cache` stays ok, so nothing user-facing breaks. Investigate only if it outlasts a burst. See also `redis-proxy-srh-crash` and the optional dedicated cache store in the owner checklist.+## 🟢 HTTP 503 `/api/agents/<id>/solana/networth`: `agent-networth-rpc-unavailable-503` (§solana-rpc)- **Source:** `handleNetWorth()` indiff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjsindex 9b3e70ac7..b31934766 100644--- a/scripts/gcp-triage.mjs+++ b/scripts/gcp-triage.mjs@@ -475,6 +475,31 @@ const KNOWN_HTTP_SIGNATURES = [class: 'self-healing',action: `The agent net-worth "look" endpoint exhausted every Solana RPC path for a wallet it has never read successfully, so it answered a designed 503 rpc_unavailable with Retry-After rather than a fake baseline (api/agents/solana-wallet.js handleNetWorth, comment above the catch block). Check healthz rpc_lanes/helius: fires in the same bursts as holders-rpc-exhausted-503 while every paid Solana RPC lane cools down (production-100-OWNER-ACTIONS.md row 24). Investigate only if it persists after rpc_lanes reads ok. ${RUNBOOK} §solana-rpc.`,},+ {+ id: 'kol-trades-upstream-502',+ // api/kol/trades.js answers err.status || 502 with Retry-After: 15 only after+ // fetchKolTrades() failed AND no last-good copy existed for the mint.+ test: (g) => g.service === 'three-ws-api' && g.status === 502 && g.path === '/api/kol/trades',+ class: 'self-healing',+ action: `The KOL trades route could not reach its Solana data lanes and had no cached last-good for this mint, so it answered a designed retryable 502 with Retry-After. Check healthz rpc_lanes: fires in the same bursts as holders-rpc-exhausted-503. Investigate only if it persists after rpc_lanes reads ok. ${RUNBOOK} §solana-rpc.`,+ },+ {+ id: 'sentiment-pulse-pumpfun-502',+ // api/social/sentiment-pulse.js: "A pump.fun outage with no caller-supplied+ // texts answers 502" by design (upstream_unavailable).+ test: (g) => g.service === 'three-ws-api' && g.status === 502 && g.path === '/api/social/sentiment-pulse',+ class: 'self-healing',+ action: `pump.fun was unreachable and the caller supplied no texts, so the sentiment pulse answered its designed 502 upstream_unavailable. Recovers when pump.fun answers. Investigate only if it persists across several sweeps. ${RUNBOOK} §pump-trending-upstreams.`,+ },+ {+ id: 'redis-proxy-upstash-timeout',+ // 2026-10-23: a 22:33Z burst of 5xx on three-ws-redis-proxy (GenServer.call+ // time out toward Upstash, SET rpccool:v1) while every RPC lane was cooling;+ // callers fall back to Postgres/in-memory (healthz cache.metrics.totalSetFailures).+ test: (g) => g.service === 'three-ws-redis-proxy' && g.status >= 500,+ class: 'self-healing',+ action: `The Redis proxy timed out talking to Upstash (GenServer.call time out); the API's cache and rate limiter fall back to Postgres/in-memory and healthz cache stays ok. Investigate only if it persists beyond a burst: npm run logs -- -s three-ws-redis-proxy --errors --app --since 6h. ${RUNBOOK} §redis-proxy.`,+ },{id: 'ccv-whales-500',test: (g) => g.service === 'cryptocurrency-cv' && g.status >= 500 && g.path === '/api/whales',