Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Production log triage
Every recurring error/warning signature that shows up in a production log
export, mapped to its root cause, the exact resolution, and who can
apply it. Built from the three-ws-character-studio export on 2026-07-03 and
re-confirmed against the three.ws export on 2026-07-05 (same population plus the
two storage-pressure signatures added below — still no code defects). The
signatures are unchanged after the 2026-07-07 move to Google Cloud Run; only the
places you read logs and set env vars changed (Cloud Run + Cloud Scheduler now,
per docs/ops/gcp-production.md).
The headline finding, so nobody re-derives it: none of these are code defects. Each line is the platform's own graceful-degradation or fail-closed machinery working correctly — a fallback firing, a circuit breaker holding, a guard refusing to spend. They are all resolved by an environment / billing / activation action on the Cloud Run service env or in an upstream dashboard, not by a code change. Silencing any of them in code would hide a real production signal, so don't.
Severity legend: 🔴 owner decision (money / security / billing) · 🟡 set an env var or add quota · 🟢 self-healing, no action needed.
This table is now machine-checked.
npm run triage:gcp(gcp-logs.md) sweeps WARNING+ logs across the whole Cloud Run fleet, matches them against these signatures, and emits a classified action plan; agents run the loop via the/gcp-triageskill. When you document a new signature here, also add it toKNOWN_SIGNATURESin scripts/gcp-triage.mjs. For the whole-surface answer to "what's wrong with three.ws?", runnpm run triage:gcp:deep: on top of the log sweep it probes the deployed version, TLS expiry, fleet revision readiness, every advertised page on the live site, cron drift and liveness, pending DB migrations, service-wallet floors, and custodial-key health, all read-only and concurrent.
You no longer need a log export to see most of this. The platform now self-reports internal-dependency health: /status renders it with a plain-language fix for each degradation, and
/api/healthzcarries a machine-readablesubsystemsblock (database, cache, rate limiter, Helius RPC, the whole Solana RPC lane tier (rpc_lanes), x402 ring, x402 settlement success, Forge 3D generation, object storage, world, sniper, the OKX chat bot, x402 config). The uptime cron (api/cron/uptime-check.js) parks a snapshot each tick and re-pages a degradation that persists. Source of the roll-up: api/_lib/ops/subsystem-health.js. This table remains the deep reference for what each state means and how to clear it.
🔴 [ring-invariants] SPEND PATH DISABLED in x402-autonomous-loop
guard env violated:
• X402_CHARITY_AUDIT_BPS = <unset> (expected 0)
• X402_FACILITATOR_URL_SOLANA / X402_SELF_FACILITATOR_ENABLED = enabled=false url=…payai… (expected self)
- Source: api/_lib/x402/ring-allowlist.js
assertRingSpendInvariants, called each tick by api/cron/x402-autonomous-loop.js. - What it means: the autonomous spend loop is enabled but the closed-loop
guard env is only half-configured —
X402_EXTERNAL_ENABLED=falseis set (good), butX402_CHARITY_AUDIT_BPSis unset and the Solana facilitator still points at an external host. The loop fails CLOSED: no money moves. It logserrorand fires one throttled critical alert per hour because a partially-off guard on a money path is exactly what you want screamed at you. - This is not a false alarm — it accurately reports an unfinished ring activation. Resolve it by finishing or pausing, not by muting.
Resolve — pick one (owner):
- Pause cleanly until you're ready to arm the ring (recommended if you are
not actively activating): set
X402_AUTONOMOUS_ENABLED=false. The loop then returnsskippedwith no error and no alert (guard check never runs). - Finish arming the ring (moves real USDC — deliberate go-live): set the
documented safe values from .env.example §x402-ring —
X402_CHARITY_AUDIT_BPS=0,X402_SELF_FACILITATOR_ENABLED=true, and either unsetX402_FACILITATOR_URL_SOLANAor point it athttps://three.ws/api/x402-facilitator. Acceptance criteria: the guard env set is complete and the coverage sweep settles every catalog entry.
🔴 [world-health] world is UNPROTECTED — ADMIN_CODE is not set
- Source: api/cron/world-health.js.
- What it means: the
world.three.wsCloud Run service is serving withoutADMIN_CODE, so every visitor has build rights. Logged aswarning. - Resolve (owner): set
ADMIN_CODEon the world service and re-rundeploy/world/apply-hardening.sh. It's a security credential the owner must choose and store — not something to auto-generate here.
🔴 The request signature we calculated does not match the signature you provided (r2-credential)
Signature. Any of: a 502 from POST /api/forge or /api/gpt-forge carrying that
sentence, [cdn-object] signed read failed, serving public bucket domain,
[avatars/glb] signed read failed, streaming public bucket domain, [forge] object storage rejected the generation, [register/prep] object storage rejected the manifest, or object_storage: down in /api/healthz.
The request-log-only form is HTTP 503 POST /api/forge-upload. The monitor
classifies that as r2-upload-unavailable (owner) because the route has only
two 503 outcomes: storage is absent, or its signing credential is rejected.
Confirm which one from healthz.object_storage; do not spend a code-debugging
cycle on a deliberately honest storage preflight response.
The deep monitor runs its log, HTTP/TLS, fleet, page, cron, database and wallet probes concurrently. Keep child processes asynchronous: a synchronous Cloud SDK read blocks the Node event loop and turns healthy network probes into a cluster of false timeouts. Cloud SDK inventory calls also get a three-minute ceiling because concurrent credential/config reads may serialize.
What it is. Cloudflare R2 is refusing our signed requests, and the SECRET is wrong. Every signed operation fails at once, read and write.
A REVOKED or deleted access key id is a different fault with the same blast radius,
and it does not say InvalidAccessKeyId: R2 answers a bare Unauthorized (measured
2026-09-08 against an unknown key id). Read the code off the error, not the sentence.
isStorageInfrastructureError matches both shapes, so the failovers below fire either
way; before 2026-09-08 it matched only the rotated-secret shape and a revoked key
reached every caller as 502 upstream_error instead.
Blast radius. Re-probed live on 2026-09-07 15:44-16:05 UTC, because the lanes do NOT all fail the same way and treating them as one outage sends you after the wrong thing:
- text→3D is dead.
POST /api/forge {"prompt":...}answers502 generation_failedand surfaces R2'sSignatureDoesNotMatchsentence verbatim to the caller. The reference image cannot be parked, so the run dies before it starts. Same on the ChatGPT surfaces (they share the bucket, not the endpoint). - image→3D from a pasted URL still works, non-durably. It reaches
status:"done"with a real GLB on the GCS raw-mesh bucket (storage.googleapis.com/ three-ws-avatar-reconstructions/, verified 200model/gltf-binary), because that hop never touches R2. ButmaterializeCreationcannot copy it, so the reply carriesdurable:false: no R2 copy, no DB creation row, no gallery entry, and the mesh is only as long-lived as the raw bucket. This is the workaround to give users. - Uploads cannot land, which is what users actually report. The R2 CORS preflight
passes, then the browser's
PUTanswers403 SignatureDoesNotMatchwith NOAccess-Control-Allow-Originon the 403 itself, so the browser hides the status andfetchthrows with nothing to read. Until 2026-09-08POST /api/forge-uploadpresigned fine (200) regardless, so/forgecould only say "Network error during upload" and a user had no way to tell our outage from their own signal. The route now asksobjectStorageUsable()(api/_lib/r2.js) whether the bucket will accept an upload before it mints a URL, and answers503 storage_unavailablewhen it will not, which the page renders as "Uploads are down right now. Try again shortly."api/print/upload.jsdoes the same. Treat either phrase in a user report as this outage until proven otherwise: the old one means the fix is not deployed yet. - Every avatar on the site disappears for signed-in users, which is the widest
symptom and the one an anonymous check cannot see.
GET /api/avatars/:id/glb502s, so the avatar widget that rides along on nearly every page swaps the user's model for therobotplaceholder and logs[walk] avatar "<id>" failed to load. The authenticated page sweep (npm run audit:web:login, thennpm run audit:web) found this on 2026-09-08 as a confirmed error on page after page; it was the only real error class in that sweep. That route now streams the public bucket domain through itself rather than redirecting, because it exists precisely to add the wildcard CORS header that the public r2.dev domain does not send, so a 302 there would turn the 502 into a CORS failure for the embed SDK and every cross-originGLTFLoader. - Agent registration cannot store its manifest.
/cdn/*cannot read an object and answers502 upstream_erroruntil the public-bucket fallback (36b67b8a8,99c521446,2ab1cb56a) is actually deployed; those commits sat onmainunshipped while production ran8770c06c2.
First seen 2026-09-07, ~00:19 UTC: generation stopped dead for nearly five hours
and was reported by users in Telegram, not by us, because nothing here had a signal
for it. Two gaps were closed in the same change and both are worth knowing about:
forge_generation read ok, 89% success throughout (its ledger only contains
generations that got far enough to write a row, so a total stop looked like a quiet
hour), and isStorageInfrastructureError matched only the compact
SignatureDoesNotMatch code, which appears in err.name and never in the sentence
above. object_storage now probes the credential directly and the forge sensor now
reports a stall.
Resolution (owner). Re-set the secret on the Cloud Run service:
gcloud run services update three-ws-api --region us-central1 \
--update-env-vars S3_SECRET_ACCESS_KEY=<secret> # or --update-secrets for a Secret Manager ref
Two traps that produce this exact error and look like a correct value:
- On R2 the Secret Access Key is the SHA-256 digest of the API token, not the
token value. Pasting the token itself signs into
SignatureDoesNotMatch. - A trailing newline in the stored secret fails identically.
env.jstrims credential-class values now, so a padded value can only bite a consumer that readsprocess.envdirectly.
Verify with curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")', then one POST /api/forge {"prompt":"cube"}.
No code change routes around it. The read paths fail over to the public bucket
domain (/cdn/* by redirect, /api/avatars/:id/glb by streaming it through), which
is rate-limited and uncached, so it is a degradation and not a fix. Writes cannot fail
over at all.
🔴 [text-to-image] replicate billing/credit failure: insufficient credit
- Source: the forge text→image lane.
- What it means: the Replicate account is out of credit. The forge already
degrades to the free NVIDIA NIM lane (see the
[forge] paid … lane unavailable; degrading to free NVIDIA NIMline), so image generation keeps working at lower fidelity. - Resolve (owner): add credit at
replicate.com/account/billing. No code change — the fallback is already correct.
🔴 502 wave on /api/x402/* from threews-x402-autonomous/1.0 (ring-duplicate-signature)
HTTP 502 POST /api/x402/<any endpoint> ua: threews-x402-autonomous/1.0 (~334/day since 2026-07-09)
- Source: the x402 autonomous ring paying its own endpoints; the settle step fails, so the paid handler returns 502 with a settle_failed body. No app-level ERROR accompanies it (the handler responds, nothing crashes).
- Root cause (diagnosed 2026-07-17): same-priced ring payments created in the same tick share a blockhash and compile to byte-identical Solana transactions, hence the same signature. Only the first lands; the rest die at preflight as already-processed, surfaced by the RPC as a bare "Transaction simulation failed" with empty logs.
- Triage tip: settle_failed with empty simulation logs plus interleaved on-chain successes means duplicate signature, not RPC or blockhash trouble.
- Fixed and deployed (2026-07-17): commit
93430b4fbadds an auto-nonce topayX402so every payment is byte-unique, and reports a precisebroadcast_failed:already_processedreason via agetSignatureStatusesprobe. The wave stopped on the revision carrying the fix, so the dedicated monitor signature (ring-duplicate-signature-502) has been retired from scripts/gcp-triage.mjs; ring self-pay 5xx now classifies underx402-wallets-dry-5xx(next section), which is a funding shape, not this bug. - If this exact shape recurs (settle_failed, empty simulation logs,
interleaved on-chain successes) on a revision carrying
93430b4fb, treat it as a regression and re-investigate the payment builder; do not fund wallets for it, and do not let the wallets-dry classification mask it. - Now watched continuously: this wave went undetected for eight days
because every sensor read "ring armed / discovery up" while a third of
settles silently failed. The x402 settlement success subsystem
(api/_lib/ops/x402-settle-health.js)
closes that blind spot: it reads the ring's own
x402_autonomous_logoutcomes over a 3h window and reports the settle SUCCESS RATE, counting only payment-rail faults (5xx / 402 / RPC / broadcast / confirm) and excluding caller errors (http_400/404/405/409), benign guards (cap, low USDC), and downstream notes. It rolls into/api/healthz,/status, and the uptime-cron escalation (pages on first sight, re-pages hourly, clears on recovery). States:ok≥ 90% settle rate,degraded50–90%,down< 50%,unknownbelow 20 settle attempts in-window (a quiet ring is not a fault). A NEW settle-fault reason is caught automatically; a new benign reason that shows up as a fault is the one thing to teach it — add the token to the fault/exclusion rules in that module, the same learn-once loop asKNOWN_SIGNATUREShere.
🔴 5xx storm on paid x402 routes from platform agent traffic (x402-wallets-dry)
HTTP 502/503 GET|POST /api/x402/*, /api/mcp ua: threews-x402-autonomous/1.0 or threews-x402-seed/1.0
Source: the platform's own agent economy (ring, seeders, autonomous buyers) paying its own endpoints while something in the settle path fails.
Three distinct causes look identical in the HTTP logs (2026-07-28: all three fired in one window). Diagnose from app logs before acting:
npm run logs -- -s three-ws-api --app --grep "fee_wallet_below_floor" --since 1hhits = the sponsor fee wallet drifted underX402_SPONSOR_SOL_FLOOR_LAMPORTS. Self-heal territory: runPOST /api/cron/treasury-topup; itsreclaimIdleAgentSolleg refunds the fee wallet from idle agent SOL (commitd6a7bf2a4). No owner money needed unless every reclaim source reportsat_or_below_floorand the run reportsskipped_floor_held_sol0. A floor skip carries its numbers (at_or_below_floor:<held><<keep>), so a non-zero held total means the SOL is sitting on a platform wallet behind a configured floor, not that the fleet is empty; review that wallet'sminSolbefore asking for capital. Abelow_min_sweep:<excess><<min>skip is a wallet above its keep line whose excess is underECONOMY_SWEEPBACK_MIN_SOL(default 0.01 SOL): dust the sweep declines to move, released by that minimum rather than by a floor. See economy-master.md "A floor skip says whether the wallet is empty or fenced".--grep "data_unavailable"hits = paid endpoints refunding honestly (no charge) because no market source could answer. Since 2026-07-28 crypto-intel prices bonding-curve pump.fun mints from the pump.fun feed (the sniper-intel pipeline passesmint), so a recurrence at volume means a data-source outage, not wallets.--grep "broadcast_failed"hits = a settle transaction failed simulation; the reason now keeps the simulation-log tail (the actual cause).insufficient fundson the token transfer = the ring payer's USDC float dipped below per-tick volume;POST /api/cron/economy-rebalancerestores it (expectresults[].status: "swapped").
Symptom map:
502= fee wallet below its SOL floor or settle broadcast failed;503= the payer's self-pay refused or a data_unavailable refund;402from a ring agent = its buyer wallet is out of USDC.The quiet variant: no storm at all, just a settle-rate collapse (
no_solana_accept). Observed 2026-07-29 16:00 UTC. The sponsor drifts under its floor and buildRequirements() WITHDRAWS the Solana accept from every 402 challenge (sponsorKnownBelowFloor()), so the ring, which is Solana-only, never gets to attempt a payment. There is no 5xx storm to grep, because nothing is being rejected. The fingerprint is: settlements collapse (that hour: 735/h to 13/h) while rail faults stay at their normal level (~100/h all day), andno_solana_acceptinx402_autonomous_loggoes from exactly 0 to 300-400/h. Confirm the mechanism from outside the box, no logs needed:curl -s https://three.ws/api/x402/three-intel | jq '.accepts[].network'. Onlyeip155:8453back, nosolana:mainnet, means the accept is withdrawn. Fix is the same free self-heal as cause 1 above (treasury-topup). Since 2026-07-30 healthz reports this itself:x402_settle.metrics.causeissponsor_floor(vsfee_governorwhen the wallet fee governor is pacing the window, orrailfor real payment-rail faults) anddetailnames the withdrawal, so the sensor no longer points at the facilitator for a funding problem.The third variant, and the one that reads as the quiet variant but is not: the accept is still advertised and every settle is REFUSED at the floor. Measured 2026-09-09.
causeissponsor_floorhere too and the fix is the same wallet, but the evidence lives in the opposite place, so checkmetrics.mechanismbefore you follow either hint:metrics.mechanismWhat is happening Where the proof is accept_withdrawnsponsorKnownBelowFloor()dropped the Solana accept, so no payment is ever attemptedthe 402 challenge ( /api/x402/three-intel), NOT the facilitator: it never saw these callssettle_refusedthe accept is on every challenge, buyers sign real payments, our facilitator turns each away at the floor gate the facilitator's reject book: fee_wallet_below_floor:<held><<floor>names the wallet and both numberspacedthe wallet fee governor spent today's budget on purpose GET /api/x402/runway-labgate_bypassa caller outside the ring signs and verifies payments the governor then refuses at settle, without asking the admission gate first ( metrics.gateBypasscounts them)x402_self_facilitator_log: group the window'sverifyrows bypayerandamount_atomic, then match the request logs by user agentrailgenuine payment-rail faults the settle logs, per the greps above gate_bypasswas production on 2026-09-29, invisible underx402_settle: ok, 97%: 43,848 of 46,375 facilitator settle attempts in 24h werefee_runway_exhausted, and 95% of them were the x402 seed cron (threews-x402-seed/1.0, 60 sponsor-mode $0.001 tips every two minutes against a budget that funded about one). The fix is in the caller, not the budget: funding the fee wallet only moves the line such a caller runs into.Two things the sensor deliberately does not count. Rows from pipelines that never pay (
UNPAID_PROBE_PIPELINESinapi/_lib/ops/x402-settle-health.js, today the external uptime monitorreliability) are dropped before the rate is taken: an unpaid HEAD probe timing out on a slow third-party host is not our rail. And anhttp_502cluster whose facilitator rows readsignature_already_settledwith a matchingokrow for the sametx_sigseconds earlier is a nonce collision (two same-amount payments built against one blockhash), not the chain: see "Every payment picks its own nonce" indocs/x402-ring-economy.md.On 2026-09-09 production was
settle_refusedverbatim: 0no_solana_accept, 164fee_wallet_below_floor:1167627<2000000, and 126http_402recorded by the ring on the paid replay (a 402 there means the payment we built was rejected, which is why the reject book and not the challenge is the place to look). Before the split, that window printedSolana accept withdrawn (0 no_solana_accept …), a clause contradicting its own counter, and told the reader not to start at the facilitator.The loud variant: a rail-shaped storm that is really a dry sponsor (
InsufficientFundsForRent). Observed 2026-08-28. This is the quiet variant's opposite and it is easy to misread, because every symptom points at the rail. The sponsor held 0.000899107 SOL against the 0.02 SOL floor, which is 0.0000082 SOL of spendable headroom, less than two transaction fees, so every transaction it fee-paid failed at simulation:sponsor_fee_unfunded:<sponsor pubkey>on the verify path (spelledsimulation_failed:{"InsufficientFundsForRent":{"account_index":0}}before 2026-09-09, see below) andsweep_broadcast_failed:Simulation failed ... account (0) with insufficient funds for renton the sweep path. Account index 0 of a compiled Solana message is the fee payer, so that error is never about the buyer's USDC: it is the sponsor being too poor to sign.Since 2026-09-09 the verify path gives that verdict its own reason class, so you can read it straight off
/api/healthz(x402.self_facilitator.verify.reject_reasons, which groups on the token before the first:):Class Who is out of SOL What to do sponsor_fee_unfundedour sponsor ( X402_FEE_PAYER_SOLANA)fund it; this is the platform's problem payer_fee_unfundedthe buyer's own wallet in a self-pay settle nothing; not ours to fund simulation_failedneither: some other revert read the suffix, it carries the raw error Under the old single
simulation_failedspelling those three shared one bucket, and becauseisRailFault()matches/simulation/the platform's own dry sponsor was also counted as a payment-rail fault. Measured on 2026-09-09, that bucket sat at the top of the verify book with 1,438 rejects in 24 hours and every row was the sponsor.Why the quiet variant's guard did not catch it:
sponsorKnownBelowFloor()was only ever written bygetBalance, and all four paid Solana RPC lanes were over quota in the same window.refreshSponsorFloorState()fails open on an RPC error by design, so the guard never learned the sponsor was dry and the ring spent three hours attempting payments that could not settle: 95 attempts, 0 settled. Since 2026-08-28 the settle path reads the verdict off the failure itself (noteSponsorRentFailure()in self-facilitator.js), which costs no RPC call and works precisely when the RPC is too degraded to answer one, and the sensor counts a fee-payer rent failure as a floor signal rather than a rail fault. Before that fix healthz reportedcause: railand its hint sent the reader to duplicate signatures and RPC preflight.Confirm in one command, no logs needed:
curl -s https://api.mainnet-beta.solana.com -X POST -H 'content-type: application/json' -d '{"jsonrpc":"2.0","id":1,"method":"getBalance","params":["WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW"]}'Under 20,000,000 lamports is below the floor; under ~900,000 the wallet cannot pay a single fee. Fix is the owner top-up below.Not a code bug. The economy-rebalance keypair crash (assigned
loadSignerKeypair's wrapper tokeypair, read.publicKeyof undefined) is fixed and live in commitbb02839f9. WhenPOST /api/cron/economy-rebalance(BearerCRON_SECRET) answersskipped: insufficient_sol_surplus, the wallets genuinely hold nothing to swap; at the 94-calls/min ring shape the burn is ~1-1.4 SOL/day.Resolve (owner, money — only after all three greps above are exhausted): send SOL (or USDC) to the economy master
WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW. The treasury-topup cron distributes to the engines within minutes and economy-rebalance restores the payer's USDC float. Recovery is visible as: rebalance returnsresults[].status: "swapped", healthzx402_settleback took, the 5xx storm stops.Alternative to daily funding: throttle the ring to funded runway (
X402_RING_TICK_CONCURRENCY, cadence and cap knobs on the Cloud Run service) so burn matches what the owner wants to spend.Monitor signature:
x402-wallets-dry-5xxin scripts/gcp-triage.mjs, classifiedowner. It matches anythreews-*user agent, because the ring calls its own paid routes under several:threews-x402-autonomous/1.0,threews-x402-seed,threews-ring-agent/<persona>(persona agents),threews-x402-wallet-monitorandthreews-x402-thumbnail-regen. Matching only the first two used to leak the rest intoinvestigateon every sweep with the same root cause.NOT this class:
fee_runway_exhausted. A settle refused withfee_runway_exhausted:<spent>+<next>><budget>is the wallet fee governor pacing a platform wallet's daily fee burn to its funded runway — a designed throttle, not an outage, and no action is required. Volume resumes at UTC midnight or immediately when the wallet is topped up (the budget scales with live balance). See docs/x402-ring-economy.md "The wallet fee governor". Onlyfee_wallet_below_floorandinsufficient_sol_surplusmean the wallets are actually dry.It answers 503
settlement_unavailable, not 502. Until 2026-08-06 this refusal was the one deliberate throttle the settle path's retryable branch missed, so it went out as502 settle_failedand read as an outage to every buyer, trust monitor and internal pipeline: 15,619 of the autonomous loop's 20,030http_502rows in the preceding 48 hours were this one reason. A502 settle_failedcluster is therefore now genuinely unexplained and worth investigating; a503 settlement_unavailablecluster is the governor or the sponsor floor, and the reason token tells you which. The autonomous loop also checks fee admission before it pays now, so a paced wallet shows up asfee_runway_exhaustedskips inx402_autonomous_lograther than as failed paid calls.The sensor reads through the status codes now.
x402_autonomous_logis reason-blind for refusals that arrive over HTTP (the row sayshttp_503or, pre-2026-08-06,http_502; the reason lives only inx402_self_facilitator_log.reject_reason). Since 2026-08-07 the healthzx402_settlesensor reconciles those status-only 5xx rows against the facilitator book for the same window, re-attributing them tofee_governor/sponsor_floorbefore it computes the rate. Sometrics.causeis trustworthy even for governed refusals recorded as bare 5xx, the settle rate judges the rail alone, and acause: "rail"verdict now genuinely means the rail. (The 2026-08-05 storm readdown/railat settle 26.1% under exactly this blindness; the same data now reads as governor pacing with a healthy rail.)
🔴 HTTP 503 on cryptocurrency.cv AI routes: ccv-ai-providers-unavailable / ccv-ai-providers-unavailable-503
HTTP 503 GET https://cryptocurrency.cv/api/narratives
AI provider groq unavailable (AIRateLimitError), trying next provider... rate-limited / out of quota (429)
AI provider gemini unavailable (AIAuthError) ... [403 Forbidden] Lightning dunning decision is deny for project: projects/93741856042
- Service:
cryptocurrency-cv(serves https://cryptocurrency.cv). Source lives in its own repository,nirholas/cryptocurrency.cv, not in this one. Its AI routes walk one provider chain insrc/lib/ai-provider.ts. - Root cause (2026-10-08 sweep): every provider is out at once, and each for
a billing reason, not a code one.
- Groq: the free tier's 200k tokens/day budget is spent. Before the
2026-10-08 fix,
/api/sentimentre-ran a ~7k-token prompt on every poll, so one client polling a single asset (91 calls in 6h) spent the day's budget by itself. - Gemini and Vertex: refused with
Lightning dunning decision is deny, a billing hold on projectaerial-vehicle-466722-p5(number 93741856042). It reads like an IAM error and is not one; no key or role change clears it. - OpenRouter: both
OPENROUTER_FALLBACK_KEYSbelong to one account whose usage exceeds its credits. Revisions built before 2026-09-30 never even read that variable.
- Groq: the free tier's 200k tokens/day budget is spent. Before the
2026-10-08 fix,
- What users see after the 2026-10-08 fixes:
/api/sentimentanswers 200 from a keyword reading of the headlines (meta.method: "keyword",meta.degraded: true),/api/narrativesserves its last good analysis for up to an hour, and every AI route caches one analysis for five minutes, so a poller no longer drains the provider. A 503 now only appears on an AI route with nothing cached, carriesRetry-After, and no longer echoes the provider's raw error (which used to include the Groq organization id). - Resolve (owner, billing): restore any ONE provider. The chain recovers on
the next request with no redeploy. Cheapest first: clear the GCP billing hold
on
aerial-vehicle-466722-p5(restores Gemini and Vertex, and with them the okx-chat-bot's Vertex lane), top up the OpenRouter account, or moveGROQ_API_KEYto a paid Groq tier. - Confirm:
curl -s https://cryptocurrency.cv/api/narrativesreturns narratives, andcurl -s 'https://cryptocurrency.cv/api/sentiment' | jq .meta.methodprints"ai". - Monitor signatures:
ccv-ai-providers-unavailable(app log) andccv-ai-providers-unavailable-503(request log) in scripts/gcp-triage.mjs, both classifiedowner.
🟡 HTTP 502 on /api/coin/* — coingecko-quota-exhausted-502
HTTP 502 GET /api/coin/detail?id=<coin-id>
body: {"error":"upstream_error","error_description":"coin data is unavailable right now — retry shortly"}
Source: api/_lib/coingecko.js
geckoFetch()exhausted every rung — live fetch, in-memory stale buffer (30 min) and the durable Upstash last-good copy (6 h) — so the handler surfaced its 502.What it usually means: the demo KEY, not the upstream. CoinGecko's demo tier caps at 10,000 calls per month. Once the cap is hit, every request carrying
COINGECKO_API_KEYgets a 429 (error_code: 10006) for the rest of the billing period — while the identical request without the key is still answered by the keyless public tier. An exhausted key is therefore strictly worse than no key. On 2026-07-28 it took/api/coin/detail,/tickersand/exchangeto a hard 502 for hours; the pages that survived only did so off their durable last-good copies.Confirm in one call (
$KEYfrom the Cloud Run env):curl -s -H "x-cg-demo-api-key: $KEY" https://api.coingecko.com/api/v3/key # {"status":{"error_code":10006,"error_message":"You've reached 10,000 calls limit. …"}}Self-healing since 2026-07-28.
geckoFetchnow treats the key as a resource that can go bad: a keyed 401/403/429 benches the key for 15 minutes and immediately retries the same URL keyless, andgeckoHeaders()(shared with api/_lib/market-fallbacks.js) stops attaching a benched key at all. A monthly quota reset or a key upgrade recovers on its own with no redeploy. Pinned by tests/coingecko-key-health.test.js.Resolve: a lingering 502 after that means the keyless tier is throttled too (Cloud Run egress IPs are shared and CoinGecko rate-limits per IP). Stop paying the wasted round trip, config-only and pre-approved:
gcloud run services update three-ws-api --region us-central1 \ --remove-env-vars COINGECKO_API_KEYThen tell the owner the key needs a paid tier (or to wait for the monthly reset). Re-add with
--update-env-vars COINGECKO_API_KEY=<key>once renewed.Monitor signature:
coingecko-quota-exhausted-502in scripts/gcp-triage.mjs, classifiedenv-action.
🟡 HTTP 502 on /api/pump/price-history: pump-price-history-upstreams-502
HTTP 502 GET /api/pump/price-history?mint=<mint>&interval=15m&from=<unix>&to=<unix>
body: {"error":"upstream_error","error_description":"Price history is unavailable for this coin right now"}
- Source: api/pump/price-history.js walked every rung (Birdeye via api/_lib/birdeye.js, GeckoTerminal, pump.fun candles) and held no last-good candles for the mint and interval.
- Not this: a coin with no market anywhere. Until 2026-10-08 such a coin
could 502 here: the Birdeye rung's GeckoTerminal fallback had already been
told "no pools", then the handler re-asked the identical URL, drew a 429 on
the shared egress and read the throttle as an outage. The pool lookup now
shares api/_lib/market/ohlcv.js
topPoolForToken(), so that coin answers404 no_market. - What a 502 now usually means: Birdeye is out of compute units and
GeckoTerminal is throttled at the same moment. Confirm Birdeye:
curl -s -H "X-API-KEY: $BIRDEYE_API_KEY" -H "x-chain: solana" "https://public-api.birdeye.so/defi/ohlcv?address=<mint>&type=15m&time_from=<from>&time_to=<to>".{"success":false,"message":"Compute units usage limit exceeded"}(seen 2026-10-08) is the plan's quota, and only the owner can reset or upgrade it. The chart keeps working on GeckoTerminal meanwhile. - Monitor signature:
pump-price-history-upstreams-502in scripts/gcp-triage.mjs, classifiedenv-action.
🟢 HTTP 502 on /api/pump/trending, /api/v1/pump/trending: pump-trending-upstreams-502 (§pump-trending-upstreams)
HTTP 502 GET /api/pump/trending?limit=48&rich=1
- Source: api/pump/trending.js reads Birdeye
first, then falls back to the pump.fun board through the shared,
cross-instance reader api/_lib/pump-feed-fetch.js,
which already retries once honoring
Retry-Afterand serves a stale board for up to 10 minutes after both live sources fail. A 502 only fires when Birdeye, pump.fun, AND the stale cache are all empty at once. - What it means: usually the same Birdeye compute-unit exhaustion as
pump-price-history-upstreams-502landing at the same moment pump.fun's frontend feed is also rate-limiting the shared Cloud Run egress IP, on an instance cold enough to hold no stale board yet. Self-healing: the 30s live cache and 10-minute stale window recover on the next request once either upstream answers. - Added: 2026-10-15 deep sweep found 2 occurrences with no app-level crash log, matching this exact "both upstreams down, no stale board" shape rather than a code fault.
- Monitor signature:
pump-trending-upstreams-502in scripts/gcp-triage.mjs, classifiedself-healing.
🟢 HTTP 502 on /api/pump/coin: pump-coin-lookup-upstream-502 (§pump-coin-lookup-upstream)
HTTP 502 GET /api/pump/coin?mint=<mint>&miss=empty
- Source:
handleCoin()in api/pump/[action].js fetches/coins/<mint>from pump.fun and falls back to the last-good cached copy (serveStaleCoin) on any non-2xx/non-404 upstream response. A 502upstream_failedonly fires when pump.fun rejects the request AND no last-good copy exists yet for that specific mint. - What it means: confirmed 2026-10-23 by reading the app logs for the sampled mint: pump.fun answered with its own Cloudflare anti-bot "Just a moment..." challenge page (HTTP 403), not a real pump.fun-side 404 or 500. Self-healing: the next successful fetch repopulates the cache for that mint.
- Added: 2026-10-23 triage-lane sweep found 1 occurrence with no app-level crash log, matching the known pump.fun Cloudflare-challenge shape rather than a code fault.
- Monitor signature:
pump-coin-lookup-upstream-502in scripts/gcp-triage.mjs, classifiedself-healing.
🟡 429 Client Error … huggingface.co → LocalEntryNotFoundError on a model worker — hf-hub-rate-limited
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url:
https://huggingface.co/facebook/dino-vitb16/resolve/main/config.json
We had to rate limit your IP (2600:1900:0:2d0e::2801).
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub …
ERROR: Application startup failed. Exiting.
Container called exit(3).
Default STARTUP TCP probe failed 1 time consecutively for container "server-1" on port 8080.
Source: a model worker whose model construction pulls a file from huggingface.co at startup. 2026-07-28 case:
model-triposr, whose TripoSRDINOSingleImageTokenizercallshf_hub_download("facebook/dino-vitb16", "config.json")while building the graph — the TripoSR checkpoint itself was already local on the gcsfuse mount, but that one 454-byte config was not.What it means: anonymous HF pulls are rate-limited per IP, and Cloud Run egress IPs are shared with the rest of the region. When HF says no, the app never listens on
$PORT, the startup probe fails, and Cloud Run kills the revision — a crash loop that scales to zero and 503s every request. No boot path may depend on a live huggingface.co fetch.The token trap: the pinned
huggingface_hubreads the legacyHUGGING_FACE_HUB_TOKENenv var. SettingHF_TOKENalone changes nothing (verified: the retry 429'd identically).Resolve (config-only, pre-approved). Stage the file into the HF cache layout inside the already-mounted weights bucket, then point the worker at it.
<sha>is the repo's current commit fromhttps://huggingface.co/api/models/<org>/<repo>(fieldsha):# 1. build the cache layout locally SHA=$(curl -s https://huggingface.co/api/models/facebook/dino-vitb16 | jq -r .sha) D=hub/models--facebook--dino-vitb16 mkdir -p "$D/refs" "$D/snapshots/$SHA" printf '%s' "$SHA" > "$D/refs/main" curl -sL -o "$D/snapshots/$SHA/config.json" \ "https://huggingface.co/facebook/dino-vitb16/resolve/$SHA/config.json" # 2. stage it (the bucket is gcsfuse-mounted at /weights on every model worker) gsutil -m cp -r hub gs://three-ws-model-weights/hf-cache/ # 3. point the worker at it, offline gcloud run services update model-triposr --region us-central1 \ --update-env-vars HF_HOME=/weights/hf-cache,HF_HUB_OFFLINE=1,HUGGING_FACE_HUB_TOKEN=<HF_TOKEN from .env>Plain files work —
hf_hub_downloadonly checksrefs/<revision>for the commit hash and thenos.path.exists()on the snapshot path, so the blob symlinks a real HF cache uses are not required.Monitor signature:
hf-hub-rate-limitedin scripts/gcp-triage.mjs, classifiedenv-action.
🟡 503 with "exceeded its quota limit for … nvidia_l4_gpu_allocation" (gpu-quota-starved)
HTTP 503 GET /health (model-* service) textPayload: The request failed because the project exceeded its quota limit for run.googleapis.com/nvidia_l4_gpu_allocation_no_zonal_redundancy
- Source: every L4 GPU engine draws from ONE regional pool
(
NvidiaL4GpuAllocNoZonalRedundancyPerProjectRegion, granted 3 in us-central1). When warmmin-instancesacross the fleet pin all of them, a min-0 service can never allocate: every request, including the Cloud Scheduler health ping, 503s without an instance ever starting. - Resolve (env-action, pre-approved): find the idle holder first. Measure
real job traffic (POST requests, not liveness pings) for each warm GPU
service over a few days; a warm L4 with zero jobs is dead weight. Free it:
gcloud run services update <service> --region us-central1 --min-instances=0. 2026-07-26 case:model-hunyuan3dheld a warm L4 with 0 jobs in 3 days whilemodel-text2motion503'd for hours; freeing it brought text2motion back (health 200, ~14 s cold start) with no code change. - Quota raise: a preference to 16 is filed and reconciling:
gcloud alpha quotas preferences list --project=aerial-vehicle-466722-p5(l4-no-zonal-us-central1-8). us-east4 also holds 3 granted L4s if a lane is ever worth porting. Fleet map and the do-not-repeat lessons: docs/ops/gcp-credits-plan.md. - Monitor signature:
gpu-quota-starvedin scripts/gcp-triage.mjs, classifiedenv-action(matches on the request entry's textPayload).
🟡 Container terminated on signal 9. on model-trellis (gpu-worker-sigkill)
WARNING [model-trellis] The request failed because either the HTTP response was malformed or connection to the instance had an error.
WARNING [model-trellis] Container terminated on signal 9.
- Source: an out-of-memory kill in the middle of a job. The L4 instance is
already at Cloud Run's 32 GiB ceiling for that GPU, and Cloud Run does not
print its usual "Memory limit of N MiB exceeded" line for these instances, so
the bare SIGKILL is the only trace (the generic
run-oomsignature never fires). The 5xx just before it is whatever poll was in flight when the container died; the caller treats a poll error as "still running". - How it was proven (2026-10-08): twelve kills in five days, every one on
the single warm instance. The revision's memory utilization climbed in steps
after heavy jobs and never came back (0.46, 0.56, 0.71, 0.81, 0.88, 0.91), and
each kill landed seconds after the job log printed
After remove invisible faces, i.e. inside the UV unwrap and texture bake. Read the curve yourself:TOKEN=$(gcloud auth print-access-token) curl -s -G -H "Authorization: Bearer $TOKEN" \ "https://monitoring.googleapis.com/v3/projects/aerial-vehicle-466722-p5/timeSeries" \ --data-urlencode 'filter=metric.type="run.googleapis.com/container/memory/utilizations" AND resource.labels.service_name="model-trellis" AND resource.labels.location="us-central1"' \ --data-urlencode "interval.startTime=$(date -u -d '-24 hours' +%FT%TZ)" \ --data-urlencode "interval.endTime=$(date -u +%FT%TZ)" \ --data-urlencode "aggregation.alignmentPeriod=1200s" \ --data-urlencode "aggregation.perSeriesAligner=ALIGN_PERCENTILE_50" - User impact before the fix: the killed job and the one queued behind it
kept a durable
runningrecord for 30 minutes, long after the forge client stopped polling, so lane failover never redispatched them (forge_creationsrows ended astask orphanedor stuck ingenerating). - Fix (code, in
workers/model-trellis):MALLOC_ARENA_MAX=2plusmalloc_trimafter every job stop glibc hoarding freed heap per executor thread; the 3 GiB of staged weights in the in-memory/tmpare deleted once the model is on the GPU; an instance that has still grown past 80% of its limit drains and restarts between jobs ("recycling instance" in its log) instead of dying inside one; and every live task heartbeats its record, so an orphan now fails within ~3 minutes while the client is still polling and the forge fails it over. Seeworkers/model-trellis/instance_health.py. - If it recurs after that deploy: it is a regression or a new leak. Each job
now logs
memory after job: …; find where the step happens. - Monitor signature:
gpu-worker-sigkillin scripts/gcp-triage.mjs, scoped tomodel-trellis, classifiedinvestigate.
🟡 Revision never Ready on a single-GPU service: "Startup probes timed out after 4m" with no container logs
Ready condition status changed to False for Revision model-hunyuan3d-21-rtx-000NN-xxx with message:
Container failed to become healthy. Startup probes timed out after 4m (1 attempts with a period of 4m each).
- Source: not the probe budget and not the image.
model-hunyuan3d-21-rtxruns on the RTX PRO 6000 quota, which Cloud Run enforces at one GPU in us-central1, and the serving revision holds that GPU withmin-instances=1. A rolling update needs a second GPU instance for the new revision before traffic can move, so that instance never starts: its log shows onlyStarting new instanceonce a minute, with no container output at all, until the 4 minute startup window expires. Traffic stays on the old revision, so users are unaffected, but the change never ships. The 2026-09-18 attempt (revision 00002, addingU2NET_HOME) sat in this state for three weeks. - Resolve (config-only, pre-approved; ~10 minutes without this lane while the
new instance loads, forge fails over meanwhile): release the GPU, roll, then
restore autoscaling. Manual scaling is a service-level setting, so neither the
first nor the last step creates a revision:
gcloud run services update model-hunyuan3d-21-rtx --region us-central1 --scaling=0 gcloud run services update model-hunyuan3d-21-rtx --region us-central1 --update-env-vars KEY=VALUE # or deploy the new image gcloud run services update model-hunyuan3d-21-rtx --region us-central1 --scaling=auto --max=1workers/model-hunyuan3d/cloudbuild.hunyuan21rtx.yamlruns this same sequence around its deploy, and restores autoscaling even when the deploy fails. Applied 2026-10-08: revisionmodel-hunyuan3d-21-rtx-00003-jmsbecame Ready in 35 s. - Monitor: the deep sweep's fleet readiness probe (
npm run triage:gcp:deep) reports the service as not Ready.
🟡 [x402-audit] insert failed … db query exceeded 3000ms deadline
- Source: api/_lib/x402/audit-log.js
logPaymentEvent. - What it means: the Neon DB is saturated, so a best-effort audit write timed
out at its 3 s fast-fail budget. This is fire-and-forget — the payment was
already decided and the response already sent (the accompanying
402is the normal x402 challenge, not a failure). The log is throttled to one line/minute with a suppressed-count digest by design. - Amplification is now fixed in code. Audit writes no longer fire one Neon
insert per request.
logPaymentEventbuffers each event to a Redis list and a once-a-minute batch flusher (flushAuditBuffer, drained by api/cron/flush-usage-events.js and the QStash job) drains them as one multi-row INSERT — the same buffer→flush path usage events use. So a slow-DB spell no longer self-amplifies into a storm of concurrent single-row writes; this line now appears only if the batched flush itself hits a genuinely down Neon, and only when Redis is also absent does it fall back to the old bounded direct insert. A retention sweep in api/cron/db-retention.js also keeps the ledger trimmed (X402_AUDIT_RETENTION_DAYS, default 90) so the dashboard aggregates that scan it stay fast. - Resolve (owner, capacity): the root cause is still DB headroom — scale the
Neon compute or add a pooler so writes settle quickly. For more durability
headroom on the direct-insert fallback, raise
X402_AUDIT_WRITE_TIMEOUT_MS(500–15000, default 3000). Losing these rows loses only telemetry, never a payment. Ring spend status (live / paused / guard violations) is now also visible in/api/healthzunderx402.ring.
🟡 [cron] <name> skipped — db at storage cap (<size>MB ≥ <high-water>MB); retention will reclaim space
[cron] launcher-tick skipped — db at storage cap (593MB ≥ 470MB); retention will reclaim space
Source: api/_lib/http.js
wrapCron({ requireWriteCapacity: true }), viaisStoragePressured()in api/_lib/db.js. Emitted by the write-heavy crons that opt into the preflight:launcher-tick,coin-intel-observe,smart-money-rollup,recompute-reputation,intel-learn.What it means: the Neon branch is over its high-water mark (
DB_RETENTION_HIGH_WATER_MB, default 470). Rather than run a full write-tick that would fail per-row with SQLSTATE 53100 and flood the logs, each write-heavy cron preflight-skips with a single warn and a healthy heartbeat (uptime reads it as up, not stalled). api/cron/db-retention.js runs every 15 min, tightens its retention window to the floor under pressure, DELETEs + VACUUMs, and the next tick resumes once size drops back under the mark. In the 2026-07-05 export db-retention was scheduled and returning200(~3.5 s/run) the whole window — the valve is working; the branch is simply sitting above the mark because the live data footprint exceeds it and Neon's storage GC is not instant.Check the mark against the REAL cap first (2026-07-29). The high-water mark is a config value, not a measurement, and it has been set to a phantom cap before. Confirm both numbers before touching anything else:
psql "$DATABASE_URL" -c 'SHOW neon.max_cluster_size' # the real ceiling psql "$DATABASE_URL" -c 'SELECT pg_size_pretty(pg_database_size(current_database()))' # the live footprint npm run logs -- --all --grep "project size limit" --since 7d # real 53100s, if anyIf
max_cluster_sizeis far above the mark and there are noproject size limiterrors, the skips are self-inflicted and the fix is to raise the mark (config-only, pre-approved). On 2026-07-29 that read 16TB against a ~2.5 GB footprint while the mark sat at 3072, so the mark was raised to8192; on 2026-10-09 the footprint had reached 11.6 GB and it was raised again to16384. See db-retention.md for the sizing rule.This gate has starved the money path. On 2026-07-28 at 18:42 the branch crossed a mark set equal to the assumed cap, and 56 crons skipped in one minute — including
economy-rebalance, which refills the x402 fee wallet. The fee wallet then starved and every settle returnedfee_wallet_below_floorfor four hours.economy-rebalanceis deliberately not gated any more; do not re-addrequireWriteCapacityto a cron on the funding path.Resolve (capacity): the write crons stay skipped only while
pg_database_size > high-water. In order: (a) raiseDB_RETENTION_HIGH_WATER_MBif the real cap has headroom (the usual answer, see above); (b) raise the Neon compute/storage plan if it genuinely does not; (c) tightenPUMP_INTEL_RETENTION_DAYS/PUMP_INTEL_MIN_RETENTION_DAYSto shed the firehose faster. None is a code change — the gate and the valve are covered by tests/cron-storage-backoff.test.js.
🟢 ws error: Unexpected server response: 301
ws error: Unexpected server response: 301
Source:
@solana/web3.js's internalrpc-websocketsclient, under the pump.fun trade firehose (api/_lib/pump-onchain-trades.js).What it means: an RPC lane in the shared chain serves JSON-RPC happily over HTTP but refuses the WebSocket upgrade. rpc-websockets treats every failure as transient and reconnects in a tight background loop for the life of a warm instance, so one bad lane produces ~100 identical lines an hour (measured 2026-07-29:
solana.leorpc.com→ 301,solana-mainnet.gateway.tatum.io→ 402).Already fixed — this should be self-healing now. Each lane's socket is probed once before use, and a structural refusal (301/302/307/308, 401/402/403/404/405/410/501) benches that lane for the process while a transient one (429, 5xx, reset) benches it for 5 minutes. The bench policy is a pure function (
classifyWsFailure) covered by tests/pump-onchain-ws-lanes.test.js. Expect at most onews lane benched (structural): <host>line per lane per process instead of the storm.If the storm returns: a lane is failing in a way the classifier reads as transient when it is really permanent. Probe it directly and add its status to the structural list:
curl -s -o /dev/null -w '%{http_code}\n' -m 8 \ -H 'Connection: Upgrade' -H 'Upgrade: websocket' \ -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \ https://<rpc-host>
🟢 {"stage":"index-delegations","warning":"time-budget-exceeded","elapsedMs":…,"stoppedAtBlock":…}
- Source: api/cron/[name].js, the delegations indexer (
IDX_TIME_BUDGET_MS, 22 s — a conservative per-invocation budget that checkpoints the cursor well before any request timeout). - What it means: the indexer hit its per-invocation time budget mid-backfill, so it saved the cursor at
stoppedAtBlockand returned rather than risk a 504. The next tick resumes exactly where it stopped. This is a checkpoint, not a failure — logged aswarningby design. A run of these back-to-back just means the indexer is draining a block backlog (cold cursor starts a day back); it stops once the cursor catches the confirmed head. - Resolve: 🟢 nothing required — it self-heals as the backlog drains. If it never stops over many hours, the RPC pool is too slow to keep pace; add a faster
IDX_RPC_URLSendpoint or shrink the per-chain block cap so each tick makes more progress.
🟢 [cache] redis SET failed / degraded / circuit opened … memory fallback
- Source: api/_lib/cache.js.
- What it means: Upstash REST SETs are timing out (a store in a region far
from the function region is the usual cause). The cache adapter already: fails
fast at
CACHE_REDIS_CMD_TIMEOUT_MS, opens a circuit breaker after 5 consecutive failures, adds a SET-suppression gate so degraded writes skip Redis entirely, and throttles every warning to one line/minute. Reads keep being served; nothing is on the request critical path. - Resolve: 🟢 nothing required — it self-heals when Upstash recovers (you'll
see
redis SET recovered). 🟡 optional: move the cache to a same-region store, or provision a dedicatedUPSTASH_CACHE_REST_URL/TOKENso best-effort cache writes don't contend with the fail-closed rate limiter, or bumpCACHE_REDIS_CMD_TIMEOUT_MSfor a distant store.
🟢 [three-holders-snapshot] refresh deferred (transient upstream): Solana error #8100002
- Source: api/cron/three-holders-snapshot.js.
- What it means: Helius DAS returned a 429 (rate limit). The cron classifies
it as transient (via
isRpcRateLimitedon the structured status code), logs awarningnot anerror, leaves the prior good snapshot intact, and self-heals on the next 5-minute tick. Public reads are unaffected. - Resolve: 🟢 nothing required. 🟡 if it's frequent, raise the Helius plan/quota.
🟢 [balances] helius quota/rate-limited — skipping it … using public RPC
- Source: api/_lib/balances.js (and the token-market path).
- What it means: Helius hit
max usage reached; the code backs off Helius for a few minutes and serves from the public Solana RPC in the meantime. - Resolve: 🟢 nothing required — the public-RPC fallback is working. 🟡 raise the Helius quota to avoid the degraded window.
- But do not read this line as "only Helius is affected." It watches one
provider; see the
rpc_lanessection below for the whole-tier view.
🔴 Every paid Solana RPC lane exhausted at once (rpc_lanes degraded)
/api/healthz → subsystems.rpc_lanes: degraded
"all 3 paid lanes exhausted; serving from 6 free lanes"
- Source:
checkRpcLanes()in api/_lib/ops/subsystem-health.js, readingrpcLaneHealth()from api/_lib/solana/connection.js. - What it means: every endpoint the platform PAYS for is parked in quota
cooldown simultaneously, so all Solana traffic is running on free public
nodes. The platform stays UP (that is the failover chain working), but free
nodes are aggressively throttled, and the symptom set is diffuse and easy to
misdiagnose: intermittent 502/503/504 on on-chain routes,
broadcast_failedsettles, slow balance reads, sniper and pump routes timing out. - Why this sensor exists. On 2026-07-29 the Helius plan
(
-32429 max usage reached), QuickNode's daily cap (-32003 daily request limit reached) and Alchemy's monthly cap (429 Monthly capacity limit exceeded) were ALL exhausted at the same time, and nothing surfaced it: the only RPC sensor watched Helius, through per-instance memory, so a freshly started instance reportedpremium RPC healthywhile every premium lane was returning errors. Verify any lane by hand with a single call:curl -s "<endpoint>" -X POST -H 'Content-Type: application/json' \ -d '{"jsonrpc":"2.0","id":1,"method":"getBalance","params":["WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW"]}' - Resolve (owner, money): top up or upgrade the exhausted plans (Helius, QuickNode, Alchemy). Nothing else clears a quota; the cooldowns simply expire and re-trip while traffic exceeds the free tiers.
- Check for a scanner burning the quota before topping up. Measured
2026-10-08:
/api/cron/x402-ring-leak-scan(api/cron/x402-ring-leak-scan.js) walked its full 2,006-wallet registry sequentially on every run, with no budget and no single-flight lock, fired both by its own Cloud Scheduler*/10job and byeconomy-tick.jsevery 2 minutes. Result: about 8 overlapping full scans at all times, roughly 46,000getSignaturesForAddresscalls an hour from one read-only monitor, run latency p50 765s against a 300s declared budget, and 504s on about 7% of runs. Recognize it withgcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" httpRequest.requestUrl:"x402-ring-leak-scan"' --freshness=3h --limit=500 --format='value(timestamp,httpRequest.status,httpRequest.latency,httpRequest.userAgent)' --project aerial-vehicle-466722-p5: many overlapping entries withuserAgentthreews-economy-tick/1.0alongsideGoogle-Cloud-Scheduler, and latencies in the hundreds of seconds. Fixed 2026-10-11: the handler now holds a lease (acquireLock), runs under a 200s wall-clock budget, scans role wallets plus ledger-active wallets plus an oldest-checked_at-first rotation instead of the whole registry every time, and was removed fromeconomy-tick.js'sTARGETS. A healthy run now shows p50 under 60s with nothreews-economy-tick/1.0caller in the log query above. - Do NOT "fix" it by promoting the reserve.
SOLANA_RPC_LAST_RESORT_URLSexists to keep a metered endpoint in reserve; naming that same URL inSOLANA_RPC_URLorQUICKNODE_RPC_URLmakes it the primary (the chain dedupes to the first occurrence) and burns the reserve first. That exact misconfiguration was live until 2026-07-29. See docs/solana.md for the full priority contract. - Repointing the primary at a free node hides this sensor on older builds.
checkRpcLanes()counts whichever URLSOLANA_RPC_URLnames as paid capacity, so pointing it at a keyless node during an outage makes the paid tier read healthy at the exact moment all of it is dark. Cross-check by probing each provider by hand with thegetBalancecall above before trusting a greenrpc_lanesrow. - Whole-tier runbook: solana-rpc-lanes.md has the one-sweep probe over every configured lane, the measured per-lane capability matrix (which free node refuses which method, and why PublicNode as primary silently breaks holder gating), the rotate-vs-fail classification contract, and the recovery commands.
- Bonding-curve symptom:
HTTP 502 GET /api/pump/curve(and the v1 alias) answersupstream_erroronly after the read-only RPC/Jupiter fallback is exhausted. The monitor classifies an isolated occurrence aspump-curve-rpc-unavailable(self-healing). Escalate it only when it persists afterhealthz.rpc_lanesshows at least one healthy lane; no transaction was broadcast and retrying cannot spend user funds. - Holder-distribution symptom:
HTTP 503 GET /api/crypto/holdersanswersupstream_unavailablewhen both holder paths (Helius, then keylessgetTokenLargestAccounts) fail for a mint with no last-good report cached yet (24h TTL), common for a mint nobody queried before during a lane outage. The monitor classifies thisholders-rpc-exhausted-503(self-healing). Escalate only if it persists afterrpc_lanesreads ok; api/_lib/crypto-token-holders.js is the composition logic if it does. - Bonding-status symptom (2026-10-15 deep sweep):
HTTP 503 GET /api/crypto/bondinganswersupstream_unavailablewhen the pump.fun feed is temporarily unreachable, the identical designed backoff contractgetBondingStatus()documents in its own handler comment ("Never 500: answer 503 + Retry-After so the agent backs off and retries, per the free-API contract"). A single hit with no matching app-level error log line fell through toinvestigateevery sweep before this signature was added. The monitor now classifies itbonding-upstream-unavailable-503(self-healing). Escalate only if it persists for the same mint after the pump.fun public frontend itself is reachable again; api/crypto/bonding.js and api/_lib/pump-bonding.js are the handler and composition logic. Regression-covered in tests/gcp-triage.test.js ("classifies a bonding-curve upstream outage as self-healing").
🟡 HTTP timeout with no cron context
The request has been terminated because it has reached the maximum request
timeout
- Source (pre-2026-10-11): nothing.
maxDurationexports onapi/cron/*.jshandlers were a real Vercel-enforced per-function budget; nothing replaced that enforcement after the 2026-07-07 Cloud Run migration, so a hang anywhere inside a cron handler ran unbounded until Cloud Run's own 900s ingress timeout (--timeout 900in server/cloudbuild.yaml) killed the connection. That log line carries no cron name, no elapsed time, and no heartbeat write, which is why/api/cron/x402-ring-leak-scan(see therpc_lanesentry above) looked like a blank wall until it was traced by hand. - Fixed 2026-10-11:
wrapCron()in api/_lib/http.js now races every cron handler against its own time budget (maxDurationMs, wired from each file'sexport const maxDuration * 1000, falling back toDEFAULT_CRON_MAX_DURATION_MS(120000ms) when a handler declares none). On timeout it answers a clean503 { ok: false, reason: 'cron_timeout', cron: '<name>', budget_ms: <n> }, writes the heartbeat (cron:heartbeat:<name>) as{ ok: false, timed_out: true }so uptime tooling sees the tick as failed rather than stalled, and fires a singlesendOpsAlertdeduplicated by signaturecron:timeout:<name>. Node has no generic way to preempt arbitrary in-flight async work, so the handler itself keeps running in the background; a late settlement after the timeout has already answered is a no-op (it can never overwrite thetimed_outheartbeat or touch the already-sent response). - Read it:
gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" textPayload:"exceeded its"' --freshness=24h --project aerial-vehicle-466722-p5surfaces the new[cron] <name> exceeded its <n>ms budget, answering 503 before Cloud Run's own timeout kills the connectionwarn line with the cron name attached. A dashboard or alert firing on the ops-alert signaturecron:timeout:<name>names the exact handler; a bare "maximum request timeout" with no matchingcron:timeout:*alert is now a sign the hang is outsidewrapCronentirely (a non-cron API route, orserver/index.mjsitself), not inside a cron tick. - Resolve: treat a recurring
cron:timeout:<name>the same as any other cron bug: find why that handler's own work (a slow RPC lane, an unbounded DB scan, a Puppeteer render) is exceeding its declaredmaxDuration, not by raising the budget first. RaisingmaxDurationonly widens the window before Cloud Run's 900s kill reappears if the underlying work keeps growing.
🟡 [solana-rpc] https://<host> 403, cooling 30m, failing over
- Source:
cooldownMsFor()in api/_lib/solana/connection.js. - What it means: depends entirely on the body, and the two cases need
opposite responses.
- A real credential failure (bad or expired key). The lane is benched for 30 minutes and stays benched. Fix the key.
- A refused call shape. Some keyless nodes answer 403 to reject ONE
request while serving everything else. PublicNode does this for
getTokenAccountsByOwnerfiltered byprogramId:{"code":-32602,"message":"Request blocked. Details: blocked parameter: params.1.programId"}.
- Why it mattered: token and USDC balance readers make that call constantly,
so when it was sized as a 30-minute auth bench, the primary evicted itself on
its own routine traffic and every Solana call cascaded down the chain onto the
exhausted paid lanes. Measured 2026-07-30, this was the last remaining
solana-rpcline in production after the primary was repointed. - Resolve: 🟢 nothing required for the blocked-call-shape case. Since
commit
955146961the rotating fetch recognises the refusal and demotes just that (lane, method) pair (markMethodDemotion) with no lane cooldown at all, so the node keeps serving every other call shape; the exportedmarkEndpointCooldownpath still sizes a shape refusal at 30 s instead of the 30 min auth bench. The monitor classifies the short-cooldown line assolana-rpc-policy-block(self-healing) in scripts/gcp-triage.mjs. 🔴 for the credential case, rotate the key on that provider. - Tell them apart:
A body namingnpm run logs -- -s three-ws-api --grep "403" --since 1hRequest blocked/blocked parameteris the harmless case. Anything else is a key problem.
🟢 [forge] paid TRELLIS lane unavailable (N); degrading text→3D to free NVIDIA NIM / nim flux failed, falling back: nim flux timed out
- Source: the forge generation router.
- What it means: the paid TRELLIS lane returned a 402/timeout, so forge fell back to the free NVIDIA NIM lane, and when a specific NIM model timed out it fell back again. Layered fallbacks — generation keeps succeeding.
- Resolve: 🟢 nothing required. Tied to the same Replicate billing item above if you want the paid high-fidelity lane back.
🟢 Error: Invalid name account provided (at @bonfida/spl-name-service → resolveName in pay-by-name.js) — sns-name-not-found
- Source: api/x402/pay-by-name.js
resolveName(), the.solresolution branch callingsns.resolve(conn, bare). - What it means: the x402 pay-by-name resolver was asked for a
.solname that has no on-chain name account (a name that does not exist). Theawait sns.resolve(...)call is wrapped intry/catchand correctly returnsnull, so the caller gets a clean404 not_found— there is no user impact. TheERRORline appears anyway because@bonfida/spl-name-serviceruns several lookup strategies in parallel: the awaited promise rejects and is caught, but a sibling promise also rejects and is left un-consumed, so Node surfaces the orphan rejection with an async stack that still points back throughresolveName→handleResolve. It is caught chatter, not a fault. - Resolve: 🟢 nothing required. It fires on a steady low cadence from callers
probing names that are not registered. If it ever spikes, check who is
resolving (request logs for
/api/x402/pay-by-name) rather than the resolver.
🟢 HTTP 503 /api/community/*, /api/clash* — cc_unconfigured (cc-unconfigured-503)
HTTP 503 GET /api/community/worlds body: {"error":"cc_unconfigured","error_description":"CoinCommunities is not configured"}
- Source: every
api/community/*andapi/clash/*handler throws the designedUnconfiguredError503 from api/_lib/coin-communities.js becauseCC_API_KEYis not set. Re-swept 2026-08-08: the key still exists nowhere, not on the Cloud Run service (checked all 200 env vars), not in.env/.env.local, not in Secret Manager. Nothing is crashing, and every surface has a designed answer for it:/worldsfails over to the live pump.fun trending feed and tags the responsesource: "pump-trending", so the world picker stays populated./messagesrenders the town's unavailable state and a locked composer ("Posting opens soon"); the live feed stays open./holder-passdrives the Play holders gate to itsunavailablestate ("Holder check is offline") with a working one-click path into the coin's open world. Before 2026-08-08 this was the one dead end: the gate showed the raw upstream string with retry-only actions that could never succeed./clash/stateand/clash/leaderboardfail over to the same trending feed (since 2026-10-08), with the roster frozen per round, so the arena stays playable. They answercc_unconfiguredonly when trending is down too, and the page then pins the battle tabs on a human message and stops the 5s poll.
- Resolve (owner, credential): provision a CoinCommunities API key
(api.coin-communities.xyz), then:
gcloud run services update three-ws-api --region us-central1 --update-env-vars CC_API_KEY=<key>. - Monitor signature:
cc-unconfigured-503in scripts/gcp-triage.mjs, classifiedowner.
🟢 HTTP 503 /api/galaxy — watsonx_unavailable (watsonx-unconfigured-503)
- Source: api/galaxy.js. The Agent Galaxy positions its stars with IBM Granite embeddings on watsonx.ai; without credentials there is no semantic space to place anything in, so it answers a designed 503 rather than inventing coordinates.
- What it means:
WATSONX_API_KEYandWATSONX_PROJECT_ID(orWATSONX_SPACE_ID) exist nowhere — not on the Cloud Run service, not in.env, not in Secret Manager (swept 2026-07-29). - No broken user path. src/galaxy.js matches this
exact error and renders a designed empty state ("IBM Granite isn't connected …
once watsonx credentials are configured, the universe lights up here"), so the
/galaxypage explains itself instead of failing. Nothing to fix in code. - Resolve (owner, credential): provision watsonx credentials at
cloud.ibm.com, then:gcloud run services update three-ws-api --region us-central1 --update-env-vars WATSONX_API_KEY=<key>,WATSONX_PROJECT_ID=<id>. - Monitor signature:
watsonx-unconfigured-503in scripts/gcp-triage.mjs, classifiedowner.
🟢 HTTP 503 /health on a model worker, UA Google-Cloud-Scheduler — worker-coldstart-health-503
- Source: the keep-warm Cloud Scheduler probe hitting a GPU/model worker (model-text2motion, model-hunyuan3d-21, model-rig, ...) while the instance is cold-booting. These workers stream hundreds of MB of weights before their server reports ready, so a probe landing in that 30-60s window gets a 503.
- What it means: nothing is wrong; the probe exists precisely to absorb this cold start so a user never pays it. Verified 2026-07-26 on model-text2motion: 503 at 21:xx, "text2motion ready" logged 30 seconds later.
- Resolve: 🟢 nothing required. Investigate only if the SAME service shows
this across several consecutive sweeps, which means it is crash-looping
instead of finishing boot (
npm run logs -- -s <service> --since 1h). - Monitor signature:
worker-coldstart-health-503in scripts/gcp-triage.mjs,self-healing.
🟢 [pump/launch-agent] send failed ... pre-broadcast simulation failed — pump-launch-sim-rejected
ERROR 502 POST /api/pump/launch-agent
[pump/launch-agent] send failed Error: pre-broadcast simulation failed: {"InstructionError":[1,{"Custom":6015}]}
- Source: api/pump/[action].js, action
launch-agent. The handler simulates every launch transaction before broadcasting it. - What it means: the simulation REFUSED a launch that would have failed on-chain (pump program custom errors: curve state raced another buyer, slippage moved, metadata rejected). That refusal protects the user's fees; the caller gets a clean 502 and a retry lands (2026-07-26 case: the same user's 201 followed within a minute).
- Resolve: 🟢 nothing required for isolated occurrences. Investigate if
the 502 group on
/api/pump/launch-agentbecomes sustained or one wallet repeats the failure many times (then decode the specific custom error code against the pump program's IDL). - Monitor signature:
pump-launch-sim-rejectedin scripts/gcp-triage.mjs,self-healing.
🟢 Error: seat reservation expired. (Colyseus, three-ws-multiplayer) — colyseus-seat-expired
- Source:
@colyseus/ws-transportWebSocketServer.onConnection, on thethree-ws-multiplayerservice. - What it means: matchmaking handed the client a seat, and the client did not complete the WebSocket upgrade within the reservation TTL (slow network, a closed tab, a mobile browser backgrounding the page mid-join). The server correctly refuses the stale ticket; the client's next join request gets a fresh seat. Observed cadence is a handful per week, which is normal churn for public rooms.
- Resolve: 🟢 nothing required. Investigate only if it spikes together with real "can't join" reports (then suspect load-balancer or WebSocket upgrade latency in front of the service, not Colyseus itself).
- Monitor signature:
colyseus-seat-expiredin scripts/gcp-triage.mjs,self-healing.
🟢 Uncaught signal: 10, pid=…, tid=…, fault_addr=0. (three-ws-redis-proxy ONLY) — redis-proxy-srh-crash
- Source: the pinned third-party image
hiett/serverless-redis-http:0.0.10(SRH, the Upstash-protocol HTTP proxy in front of Redis) running asthree-ws-redis-proxy. - What it means: the SRH runtime aborts sporadically (observed ~3x/day) and
Cloud Run restarts the instance.
minScale 2keeps a warm sibling serving during the restart, and the API's cache layer rides the blip via its circuit breaker + in-memory fallback (see §cache above) — healthzcachestaysokthrough these events, so there is no user impact at the observed rate. - Resolve: 🟢 nothing required day-to-day. The durable fix is bumping the
SRH image tag on
three-ws-redis-proxy, which changes a running binary and therefore waits for owner approval like any deploy. Investigate only if the crash rate climbs to many per hour or healthzcachedegrades. - Scope note: the monitor classifies this only for
three-ws-redis-proxy. An identicalUncaught signalline from any other service staysinvestigate— do not generalize this signature. - Monitor signature:
redis-proxy-srh-crashin scripts/gcp-triage.mjs,self-healing(service-scoped via itsservicesfield).
🟢 Truncated response body. Usually implies that the request timed out... (cryptocurrency-cv ONLY): ccv-sse-stream-timeout
- Source: Cloud Run itself (
run.googleapis.com/varlog/system), on thecryptocurrency-cvservice. The service's source lives in its own repository,nirholas/cryptocurrency.cv. - What it means:
/api/news/streamis a Server-Sent Events feed. A client holds it open until Cloud Run's 300 s request timeout closes the response, and Cloud Run logs every response it closes that way. Measured 2026-10-08: 12 of 12 sampled lines joined bytraceto aGET /api/news/streamrequest that ran 300.99 to 301.00 s, about one every two minutes. Browsers'EventSourcereconnects on its own, so the reader sees an unbroken feed. - Resolve: 🟢 nothing required. To confirm a line, join it to its request:
gcloud logging read 'trace="<trace from the warning>" httpRequest:*' --freshness=24h --format='value(httpRequest.requestUrl,httpRequest.latency)'. Investigate only if a trace lands on a route that is not a stream. - Scope note: classified only for
cryptocurrency-cv. The same line from any other service staysinvestigate, because there it can mean a handler that never finished its response. - Monitor signature:
ccv-sse-stream-timeoutin scripts/gcp-triage.mjs,self-healing(service-scoped via itsservicesfield).
🟢 Truncated response body. Usually implies that the request timed out... (clink-fun ONLY): clink-mcp-stream-timeout
- Source: Cloud Run itself (
run.googleapis.com/varlog/system), on theclink-funservice (serves https://paired.exchange). Its image (clink/web) builds from its own repository, not this one. - What it means:
/mcpis an MCP streamable-HTTP endpoint. A client holds the connection open until Cloud Run's 300 s request timeout closes the response, and Cloud Run logs every response it closes that way. Measured 2026-10-11: 4 of 4 sampled lines joined bytraceto aGET /mcprequest that ran 300.99 to 301.00 s, status 200, user agentrokmcp-collector/0.2. MCP clients reconnect on their own, so this is not a dropped session. - Resolve: 🟢 nothing required. To confirm a line, join it to its request:
gcloud logging read 'trace="<trace from the warning>" httpRequest:*' --freshness=24h --format='value(httpRequest.requestUrl,httpRequest.latency)'. Investigate only if a trace lands on a route that is not/mcp. - Scope note: classified only for
clink-fun. The same line from any other service staysinvestigate, because there it can mean a handler that never finished its response. - Monitor signature:
clink-mcp-stream-timeoutin scripts/gcp-triage.mjs,self-healing(service-scoped via itsservicesfield).
🔴 5xx fan-out across /api/forge, /api/gpt-forge, /api/glb, avatars, /api/3d/inspect, /api/print/quote, /cdn/*, /api/render/*, /api/animations/clips/<id>, /api/avatar/render, /api/avatar/capabilities: r2-downstream-5xx (§r2-downstream)
- Source: every route that reads or writes through
api/_lib/r2.js (
api/forge.js,api/gpt-forge.js,api/objects/library.js,api/avatars/[id].js), the entire/cdn/*namespace (routed byvercel.json's single/cdn/(.+)catch-all to api/cdn-object.js, which always reads R2; no subpath is an exception), two generic caller-supplied-URL fetch tools (api/print/quote.js,api/3d/inspect.js, 502fetch_failed) that happen to be fetching R2-hosted GLBs in practice because forge/gpt-forge persist generated models there, and (added 2026-10-12) the avatar-render family that fetches an avatar's ownmodel_urlthroughfetchModel/getObjectBufferinstead ofapi/_lib/r2.js:api/render/animate.js(502render_failed),api/render/glb.jsandapi/render/avatar-clip.js(502, same lib), and GETapi/animations/[id].js(/api/animations/clips/<id>, 502storage_errorfromgetObjectBuffer). Added 2026-10-25:api/avatar/render.js(502render_failedfromavatar-render.js'srenderAvatarImage) andapi/avatar/capabilities.js(502fetch_failedfromavatar-capabilities.js'sinspectGlbCapabilities), the same avatar.model_url fetch pattern under two more public route names. - What it means: Cloudflare R2 unreachable (healthz
object_storagedown/degraded, tracked at production-100-OWNER-ACTIONS.md row 36) fans out into 5xx on this whole family at once with zero matching app-level crash logs, because each handler returns a clean upstream-failure status rather than throwing. Triage-lane sweep 2026-10-11 saw all nine original routes fire together inside the same window; the 2026-10-12 sweep reproduced it live for/api/render/animateby curling the failing avatar's ownmodel_url(https://pub-<account>.r2.dev/u/<owner>/<slug>.glb) directly and getting the same403Cloudflare answers for every other route in this family. 2026-10-13 (order 400): this was previously described as a flap (manyRESOLVEDevents minutes apart); re-measurement found a 35-plus-hour unbrokenDOWNstreak since the last resolve at 2026-10-11T06:55:05Z (60-plus hours and still ongoing by the time this note was written), with the public*.r2.devfallback domain also edge-blocked (Cloudflare-branded 403) and/cdn/*'s own signed-read-then-public-fallback therefore 502ing on both legs at once. Treat a live recurrence as a sustained outage, not a flap, until row 36 says otherwise. Order 400 also added one bounded retry (200-500ms backoff, inside a singlegatherObjectStorageHealth()call) so a single transient blip no longer pages asDOWN, and routed most generated thumbnail/GLB URLs returned by the API through/cdn/instead of a raw*.r2.devURL, so a future single-leg flap (only the signed path or only the public domain degraded) gets/cdn/'s existing failover instead of no protection at all. - Resolve: check healthz first:
curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'.down/degradedconfirms this family and the fix is the owner-side Cloudflare dashboard check in row 36. Ifobject_storagereadsok, stop treating this as one signature and read the specific route's handler: the R2-direct routes point at a real credential/bucket problem, the fetch-tool and avatar-render routes point at whatever host the sample URL actually names (confirm it is really.r2.devbefore blaming R2). - Monitor signature:
r2-downstream-5xxin scripts/gcp-triage.mjs, classifiedowner. Regression-covered in tests/gcp-triage.test.js ("classifies an animated-render R2 fetch failure..." / "...a clip-detail R2 fetch failure..." / "...an avatar-render R2 fetch failure..." / "...an avatar-capabilities R2 fetch failure..."). - Worker-side app-log variant (same outage, different shape): a GPU/mesh
worker that routes a caller-supplied model URL through
worker_security.py'sfetch_remote_bytes(remesh-serviceconfirmed 2026-10-15; any worker vendoring that module can hit it) logs a Python traceback ending inhttpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://pub-<account>.r2.dev/...'when the fetched model lives on R2's public.r2.devdomain during the flap. The worker already catches this cleanly (safe_errormarks the taskfailedwith an opaque message; the traceback is logged for operators only, not an unhandled crash). This also exposed a real bug in the monitor itself:buildFindingswas cutting its app-log classification sample to the first 400 bytes of the message, so the one line that names the fault in a multi-frame traceback (always last) never reachedclassify(), and every such hit fell through toinvestigateinstead of this already-trackedownersignature. Fixed 2026-10-15 with aheadTail()helper that keeps both ends of an over-budget message. Regression-covered in tests/gcp-triage.test.js ("classifies a GPU worker R2 .r2.dev 403 traceback as the r2-downstream family").
🟢 HTTP 503 /api/trenches: pulse-not-deployed-503 (§pulse-not-deployed)
- Source: api/trenches.js, a read-only proxy to
the
services/pulsemarket-intelligence service overPULSE_URL. - What it means:
pulsehas never been deployed (gcloud run services listhas nopulseentry,PULSE_URLis unset), so every request answers the designed 503pulse_offline("Pulse is not configured on this deployment"). Nothing in code is broken. - Resolve (owner, deploy): stand up the
pulseCloud Run service (its own Cloud SQLDATABASE_URL, dedicatedSOLANA_WS_URLS) per the "Deploy (owner-gated)" section of docs/trenches.md. Until then, 🟢 nothing required. - Monitor signature:
pulse-not-deployed-503in scripts/gcp-triage.mjs, classifiedowner.
🟡 HTTP 503 /api/health: wallet-rollup-503 (§wallet-rollup)
- Source: api/health.js via
api/_lib/health-probes.js's
gatherHealth(), which folds exactly four probes (database,rpc,worker,wallet) and answers 503 on the worst one. - What it means: read the body
(
curl -s https://three.ws/api/health) to see which probe isdown. If it iswalletwiththree_token_raildown, that isTHREE_TREASURY_WALLET/THREE_REWARDS_WALLETstaying unset by design (row 23: a wallet-naming decision only the owner can make, because the rewards candidate wallet is also the buyback signer). If it isrpc, that is the paid Solana RPC lane exhaustion (row 24, see "Every paid Solana RPC lane exhausted at once" above). - Resolve: nothing to fix in code unless a probe other than
wallet/rpcis the one reportingdown; those two are both pre-existing owner decisions. - Monitor signature:
wallet-rollup-503in scripts/gcp-triage.mjs, classifiedowner.
🔴 5xx on /api/vision, /api/tts/synthesize, /api/v1/ai/text-to-3d: three-ws-ai-providers-unavailable-5xx (§three-ws-ai-providers-unavailable)
- Source:
/api/vision,/api/tts/synthesizeand/api/v1/ai/text-to-3deach walk a provider chain (NVIDIA, then Vertex Gemini, then OpenAI) and answer an honest 502/503 when every rung fails. Mirrorsccv-ai-providers-unavailable-503above, for this repository's own AI routes./api/v1/ai/text-to-3dpins the NVIDIA NVCF TRELLIS rung specifically (added 2026-10-15 after a deep sweep found 3lane_error502s with no app-level crash log, matching this family). - What it means: the GCP billing hold ("Lightning dunning decision is
deny") on
aerial-vehicle-466722-p5blocks the Vertex/Gemini rung (row 20) and the OpenAI key is dead (row 6), leaving only the NVIDIA rung, which theforge_generationhealth sensor shows intermittently failing on free-tier NVCF capacity. - Resolve (owner, billing/credential): restoring either billing-held
provider clears this family. Nothing to fix in code: same pattern as
ccv-ai-providers-unavailable-503, just a different repository's routes. - Monitor signature:
three-ws-ai-providers-unavailable-5xxin scripts/gcp-triage.mjs, classifiedowner.
🔴 HTTP 503 /api/chat: chat-fallback-chain-exhausted-503 (§chat-fallback-chain-exhausted)
- Source:
/api/chat's failover chain (groq, then every openrouter sibling, then nvidia, then thevertex-geminianchor) answers 503rate_limitedonly once every rung including the anchor has failed (api/chat.js, the final branch around line 936). Thevertex-geminianchor is deliberately exempt from the fallback cooldown and theMAX_FALLBACK_ATTEMPTScap specifically so a healthy anchor should never let chat 503 (seebuildFallbackChainin the same file). - What it means: added 2026-10-24 after a deep sweep found 21
/api/chat503s in 6h. Traced one occurrence end to end vianpm run logs: groq 429, every openrouter sibling 429, nvidia 404 ("Function ... Not found for account"), then the vertex-gemini anchor 403 on the same GCP billing dunning hold ("Lightning dunning decision is deny" onaerial-vehicle-466722-p5, row 20) that blocks the anchor forthree-ws-ai-providers-unavailable-5xxabove. With the anchor itself billing-blocked there is nothing left to fail over to. The nvidia 404 is a separate, narrower signal: thatNVIDIA_API_KEYaccount has lost access to the configured NIM function server-side (not a stale model id in this repo; the model string is unchanged and groq's prior 429 already shows free-tier demand, not a routing bug, is the bottleneck). - Resolve (owner, billing/credential): restoring the Vertex billing hold
(row 20) clears this family the same way it clears
three-ws-ai-providers-unavailable-5xx. Separately, check the NVIDIA NIM account behindNVIDIA_API_KEYat the build.nvidia.com console if the 404 "Not found for account" line persists; that is an upstream account-state check, not a code fix. Nothing to fix in code unless a new rung (not groq/openrouter/nvidia/vertex-gemini) is the one failing. - Monitor signature:
chat-fallback-chain-exhausted-503in scripts/gcp-triage.mjs, classifiedowner.
🔴 HTTP 502 /api/3d/generate: forge-image-ladder-exhausted-502 (§forge-image-ladder-exhausted)
- Source:
/api/3d/generateself-calls/api/forge, whose reference-image step (api/_lib/forge-reference-image.js) tries Vertex, then falls through to the text-to-image ladder (api/_mcp3d/text-to-image.js): NIM FLUX, HF fal-ai, HF nscale, keyless Pollinations, Replicate. A 502 means every rung the request reached failed. - What it means: root-caused on 2026-10-12. Vertex 403s on the GCP billing
dunning hold (row 20); HF fal-ai and HF nscale both 402 "no remaining
credits" (a separate credit pool from the LLM lanes'
HF_TOKENusage in row 6, needs its own funding, see OWNER-ACTIONS). Before the 2026-10-12 fix ("stop the Vertex billing hold and HF credit exhaustion from re-paying their own timeout on every image request"), none of those three dead lanes had a circuit breaker, so every request re-paid their full timeouts before ever reaching Pollinations, often blowing the shared ladder budget before Pollinations ran at all, which is what turned an owner-gated billing/credit issue into a request-killing 502 storm instead of a quiet fallback to the free lane. - Resolve (owner, billing/credential): fund the Vertex billing hold (row
- and/or top up the HF Inference Providers credit balance (fal-ai,
nscale). Either one clears most of this family; both together clear it.
Nothing further to fix in code unless a new rung fails, or Pollinations
itself starts failing with no paid Replicate token configured as a backstop:
check
npm run logs -- -s three-ws-api --app --since 1h --grep "falling back"for which rung is actually down before assuming it is the same two causes above.
- and/or top up the HF Inference Providers credit balance (fal-ai,
nscale). Either one clears most of this family; both together clear it.
Nothing further to fix in code unless a new rung fails, or Pollinations
itself starts failing with no paid Replicate token configured as a backstop:
check
- Monitor signature:
forge-image-ladder-exhausted-502in scripts/gcp-triage.mjs, classifiedowner.
🟢 HTTP 502 /api/kol/trades: kol-trades-upstream-502 (§solana-rpc)
api/kol/trades.js answers a designed 502 with Retry-After: 15 when fetchKolTrades() fails and no last-good copy exists for the mint. It fires in the same bursts as the holders 503 while every Solana RPC lane cools down (healthz rpc_lanes). No action; investigate only if it persists after rpc_lanes reads ok.
🟢 HTTP 502 /api/social/sentiment-pulse: sentiment-pulse-pumpfun-502 (§pump-trending-upstreams)
By design (api/social/sentiment-pulse.js): a pump.fun outage with no caller-supplied texts answers 502 upstream_unavailable. Recovers when pump.fun answers.
🟢 HTTP 5xx on three-ws-redis-proxy: redis-proxy-upstash-timeout (§redis-proxy)
2026-10-23 22:33Z burst: the proxy's GenServer.call to Upstash timed out (SET rpccool:v1). The API's cache and rate limiter fall back to Postgres/in-memory and healthz cache stays ok, so nothing user-facing breaks. Investigate only if it outlasts a burst. See also redis-proxy-srh-crash and the optional dedicated cache store in the owner checklist.
🟢 HTTP 503 /api/agents/<id>/solana/networth: agent-networth-rpc-unavailable-503 (§solana-rpc)
- Source:
handleNetWorth()in api/agents/solana-wallet.js. Its hold-last-state contract serves a stale last-known-good snapshot when a live balance read fails; this 503 fires only when there is no snapshot to fall back to yet, meaning every RPC path is down for a wallet this endpoint has never read successfully. - What it means: same root cause as
holders-rpc-exhausted-503above (see "Every paid Solana RPC lane exhausted at once"), a different route hitting the same quota exhaustion (row 24). The handler returns 503 withRetry-After: 15rather than inventing a balance. - Resolve: check healthz
rpc_lanes/heliusfirst; this fires in the same bursts asholders-rpc-exhausted-503while every paid Solana RPC lane cools down. Investigate only if it persists afterrpc_lanesreadsok. - Monitor signature:
agent-networth-rpc-unavailable-503in scripts/gcp-triage.mjs,self-healing.
🟡 HTTP 5xx cryptocurrency-cv /api/whales: ccv-whales-500 (§cryptocurrency-cv-separate-repo)
- Source:
cryptocurrency-cv's/api/whales. Its source lives in its own repository (nirholas/cryptocurrency.cv), not this one, so a crash there cannot be fixed from this checkout regardless of root cause. - What it means: root-cause with
npm run logs -- -s cryptocurrency-cv --errors --app --since 6h. If the cause matches the already-tracked AI-provider billing hold (see §ccv-ai-providers-unavailable above), no further action here; otherwise file it against thecryptocurrency.cvrepository. - Resolve: out of scope for this repository by design. Scoped to
status >= 500only; a 429 on the same route falls back to the defaultself-healingclassification, not this signature. - Monitor signature:
ccv-whales-500in scripts/gcp-triage.mjs, classifiedowner.
🟢 WARNING invalid_permit2_spender on sperax-x402-facilitator: sperax-facilitator-invalid-permit2-spender
- Source:
sperax-x402-facilitator, a third-party-operated Cloud Run service in this GCP project.gcloud run services describe sperax-x402-facilitatorshowscreator/lastModifier[email protected]and an image built outsidethree-ws-build@; its source is not in this repository. - What it means: a caller presented a Permit2 spender address the facilitator does not recognize for the asset/network pair, a client-side integration mismatch. The facilitator answers the caller with the verify-failed result itself; this is not a crash and needs no retry on our side.
- Resolve: 🟢 nothing required here. If x402 settle volume through this
facilitator drops, forward the sample payload to
[email protected]. - Monitor signature:
sperax-facilitator-invalid-permit2-spenderin scripts/gcp-triage.mjs, classifiedowner, scoped tosperax-x402-facilitator.
The owner runbook — every fix as an exact command
Everything red/yellow above, condensed to the actions only the owner can take,
each reduced to a copy-paste command. Nothing here is a code change — the code
paths behind every line are already hardened and covered by tests
(tests/x402-ring-invariants.test.js,
tests/cache-circuit-breaker.test.js,
tests/cache-store-routing.test.js,
tests/cron-storage-backoff.test.js).
Apply the env writes to the three-ws-api Cloud Run service with
gcloud run services update; each write rolls out a new revision, so they take
effect as soon as that revision is serving traffic.
1. 🔴 World — stop every visitor having build rights (do this first)
The fail-closed patch (deploy/world/patches/0003-fail-closed-without-admin-code.patch)
is already in the repo; it just isn't live on the running revision. One script
generates the secret, rebuilds, redeploys, and polls /status until it reports
protected:true (needs Cloud Run / Secret Manager / Cloud Build on project
aerial-vehicle-466722-p5):
bash deploy/world/apply-hardening.sh # prints the admin code once — store it in a password manager
2. 🔴 x402 ring — pause cleanly, or finish arming
# Pause quietly (recommended unless you're actively going live) — kills the hourly guard alert:
gcloud run services update three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 --update-env-vars X402_AUTONOMOUS_ENABLED=false
# …or finish arming (moves real USDC) per docs/x402-ring-economy.md guard-env section.
3. 🟡 Neon — stop the write-crons preflight-skipping at the storage cap
DB sat at 593 MB vs the 470 MB high-water in the 2026-07-05 export, so
launcher-tick, coin-intel-observe, smart-money-rollup, recompute-reputation,
and intel-learn were skipping. Pick the lever that matches your Neon plan:
# (a) If your branch's real cap is well above 470 MB, raise the high-water to match
# (must stay under the real Neon cap):
gcloud run services update three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 --update-env-vars DB_RETENTION_HIGH_WATER_MB=900
# (b) …or shed the pump.fun firehose faster (keeps the branch smaller):
gcloud run services update three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 \
--update-env-vars PUMP_INTEL_RETENTION_DAYS=7,PUMP_INTEL_MIN_RETENTION_DAYS=2
# (c) Best durable fix: bump the Neon compute/storage plan in the Neon dashboard.
4. 🟡 Replicate — restore the paid forge lane (clears the two 502s)
Add credit at replicate.com/account/billing. The free NVIDIA NIM + HF lanes
keep serving meanwhile; paid credit brings back the high-fidelity TRELLIS lane.
5. 🟡 Upstash / Helius — kill the redis-timeout and 429 warnings (optional)
# A same-region dedicated cache store ends the 'redis SET failed' flood:
gcloud run services update three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 \
--update-env-vars UPSTASH_CACHE_REST_URL=<url>,UPSTASH_CACHE_REST_TOKEN=<token>
# …or just give a distant store more headroom:
gcloud run services update three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 --update-env-vars CACHE_REDIS_CMD_TIMEOUT_MS=5000
# Helius 429s: raise the plan/quota in the Helius dashboard (public-RPC fallback covers the gap).