Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(ops): make deep GCP triage reliable

nirholasb4c5fb463bf8
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index be04cccc6..f00a96fcf 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -97,6 +97,18 @@ sentence, `[cdn-object] signed read failed, serving public bucket domain`,
storage rejected the generation`, `[register/prep] object storage rejected the
manifest`, or `object_storage: down` in `/api/healthz`.
+The request-log-only form is `HTTP 503 POST /api/forge-upload`. The monitor
+classifies that as `r2-upload-unavailable` (owner) because the route has only
+two 503 outcomes: storage is absent, or its signing credential is rejected.
+Confirm which one from `healthz.object_storage`; do not spend a code-debugging
+cycle on a deliberately honest storage preflight response.
+
+The deep monitor runs its log, HTTP/TLS, fleet, page, cron, database and wallet
+probes concurrently. Keep child processes asynchronous: a synchronous Cloud
+SDK read blocks the Node event loop and turns healthy network probes into a
+cluster of false timeouts. Cloud SDK inventory calls also get a three-minute
+ceiling because concurrent credential/config reads may serialize.
+
**What it is.** Cloudflare R2 is refusing our signed requests, and the SECRET is
wrong. Every signed operation fails at once, read and write.
@@ -718,6 +730,12 @@ ws error: Unexpected server response: 301
matrix (which free node refuses which method, and why PublicNode as primary
silently breaks holder gating), the rotate-vs-fail classification contract,
and the recovery commands.
+- **Bonding-curve symptom:** `HTTP 502 GET /api/pump/curve` (and the v1 alias)
+ answers `upstream_error` only after the read-only RPC/Jupiter fallback is
+ exhausted. The monitor classifies an isolated occurrence as
+ `pump-curve-rpc-unavailable` (self-healing). Escalate it only when it persists
+ after `healthz.rpc_lanes` shows at least one healthy lane; no transaction was
+ broadcast and retrying cannot spend user funds.
---
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index 363ae1d69..d5c91c857 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -219,6 +219,19 @@ const KNOWN_HTTP_SIGNATURES = [
class: 'env-action',
action: `502 on a /api/coin/* route means every CoinGecko rung failed and no cached last-good existed. The usual cause is the DEMO KEY, not the upstream: the demo tier caps at 10,000 calls per MONTH, and once exhausted every request carrying COINGECKO_API_KEY gets 429 while the identical keyless request is still served (2026-07-28: /coin/detail, /tickers and /exchange all 502'd for hours on an exhausted key). Confirm in one call: curl -s -H "x-cg-demo-api-key: $KEY" https://api.coingecko.com/api/v3/key — error_code 10006 is the cap. geckoFetch now benches a rejected key for 15 min and retries keyless on its own, so a lingering 502 means the keyless tier is ALSO throttled (the Cloud Run egress IP is shared). Fix: gcloud run services update three-ws-api --region us-central1 --remove-env-vars COINGECKO_API_KEY (config-only, pre-approved) to stop paying the round trip, and tell the owner the key needs a paid tier or a monthly reset. ${RUNBOOK} §coingecko-quota-exhausted.`,
},
+ {
+ id: 'r2-upload-unavailable',
+ test: (g) => g.service === 'three-ws-api' && g.status === 503 && g.path === '/api/forge-upload',
+ class: 'owner',
+ action: `The upload preflight intentionally returns 503 when object storage is unconfigured or rejects its credential. Confirm healthz object_storage; SignatureDoesNotMatch means the owner must rotate the R2 credential and add a new s3-secret-access-key Secret Manager version. No code or retry fixes a rejected signing secret. ${RUNBOOK} §r2-credential.`,
+ },
+ {
+ id: 'pump-curve-rpc-unavailable',
+ test: (g) => g.service === 'three-ws-api' && g.status === 502
+ && (g.path === '/api/pump/curve' || g.path === '/api/v1/pump/curve'),
+ class: 'self-healing',
+ action: `The read-only bonding-curve route exhausted its RPC/Jupiter fallback and returned an honest no-store upstream_error. Check healthz rpc_lanes: short cooling windows recover automatically. Investigate only if the same route persists across sweeps after at least one RPC lane is healthy; then inspect api/_lib/pump-curve-view.js rather than retrying a transaction. ${RUNBOOK} §solana-rpc.`,
+ },
{
id: 'cc-unconfigured-503',
// Every /api/community/* and /api/clash route wraps the same client and
@@ -297,17 +310,20 @@ async function fetchHealthz() {
}
}
-function readLogs(opts) {
+async function readLogs(opts) {
const query = 'resource.type="cloud_run_revision" severity>=WARNING';
- const res = spawnSync('gcloud', [
+ // Keep this asynchronous. The deep sweep deliberately runs its HTTP/TLS
+ // probes alongside the log read; spawnSync blocked Node's event loop for
+ // 40+ seconds and made healthy 10-15 second probes time out in a batch.
+ const res = await runCommand(['gcloud',
'logging', 'read', query,
`--project=${opts.project}`,
`--freshness=${opts.since}`,
`--limit=${opts.limit}`,
'--format=json',
- ], { encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });
- if (res.status !== 0) {
- const why = res.error?.message || (res.stderr || '').trim() || 'unknown error';
+ ], { timeoutMs: 180000 });
+ if (res.code !== 0) {
+ const why = (res.stderr || '').trim() || (res.timedOut ? 'timed out' : 'unknown error');
console.error(`gcloud logging read failed: ${why}`);
if (/ENOENT/.test(why)) console.error('gcloud is not installed at any known path; see scripts/lib/gcloud-path.mjs.');
if (/reauth|invalid_grant|credential/i.test(why)) console.error('gcloud auth has lapsed; only the owner can re-run `gcloud auth login` here.');
@@ -635,7 +651,12 @@ async function probeTls() {
}
async function probeFleet(opts) {
- const r = await runCommand(['gcloud', 'run', 'services', 'list', `--project=${opts.project}`, '--region=us-central1', '--format=json'], { timeoutMs: 90000 });
+ // Deep mode launches the Scheduler and logging audits at the same time. The
+ // Cloud SDK can serialize credential/config reads under that load, so give
+ // the fleet inventory the same ceiling as the log read. A healthy 47-service
+ // inventory has exceeded 90 seconds during a concurrent sweep even though
+ // the identical standalone command completed successfully.
+ const r = await runCommand(['gcloud', 'run', 'services', 'list', `--project=${opts.project}`, '--region=us-central1', '--format=json'], { timeoutMs: 180000 });
if (r.code !== 0) return { status: 'error', note: tail(r.stderr, 3) };
const services = JSON.parse(r.stdout || '[]');
const notReady = [];
@@ -659,14 +680,26 @@ async function probePages() {
const m = out.match(/all (\d+) declared pages reachable/);
return { status: 'ok', note: m ? `all ${m[1]} advertised pages reachable` : 'all advertised pages reachable' };
}
- const lines = out.split('\n').filter((l) => l.includes('[check-pages]') && !l.includes(' OK ')).slice(0, 14);
- const countMatch = out.match(/(\d+) declared page/);
- return { status: 'findings', note: `${countMatch ? countMatch[1] : 'some'} page(s) failing`, findings: [deepFinding('pages', 'investigate', 'advertised pages failing on the live site', {
- count: countMatch ? Number(countMatch[1]) : 1, services: ['three-ws-api'], sample: lines.join('\n'),
+ const summary = summarizePageCheckFailure(out);
+ return { status: 'findings', note: `${summary.count} page(s) failing`, findings: [deepFinding('pages', 'investigate', 'advertised pages failing on the live site', {
+ count: summary.count, services: ['three-ws-api'], sample: summary.sample,
action: 'Each failing path is advertised in the sitemap and llms.txt. The sweep output states whether it is a routing bug or deploy lag (route landed after the running image). Routing bug: add the vercel.json rewrite and vite input, commit. Deploy lag: note it for the next owner-approved deploy.',
})] };
}
+export function summarizePageCheckFailure(output) {
+ const out = String(output || '');
+ const unreachable = Number(out.match(/(\d+) unreachable page\(s\)/)?.[1] || 0);
+ const wrong = Number(out.match(/(\d+) page\(s\) answered 200 with another page's content/)?.[1] || 0);
+ const count = unreachable + wrong || 1;
+ const lines = out.split('\n').filter((line) => {
+ if (!line.includes('[check-pages]')) return false;
+ if (/\d+\/\d+…/.test(line) || line.includes(' sweeping ')) return false;
+ return true;
+ }).slice(-20);
+ return { count, sample: lines.join('\n').slice(0, 1400) };
+}
+
async function probeCronDrift() {
const r = await runCommand(['node', 'scripts/check-cron-drift.mjs', '--json'], { timeoutMs: 120000 });
let report;
@@ -824,7 +857,7 @@ async function main() {
const deepPromise = opts.deep ? runDeepSweep(opts) : Promise.resolve(null);
const [healthz, entries] = await Promise.all([
fetchHealthz(),
- Promise.resolve().then(() => readLogs(opts)),
+ readLogs(opts),
]);
const findings = buildFindings(entries);
findings.push(...gpuCapacityFindings(findings));
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index ec7fd39f7..5c0b478ae 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -2,7 +2,7 @@
// added to KNOWN_SIGNATURES should get a case here: the monitor learning a
// benign pattern must never silently swallow a genuine fault that shares text.
import { describe, expect, it } from 'vitest';
-import { classify } from '../scripts/gcp-triage.mjs';
+import { buildFindings, classify, summarizePageCheckFailure } from '../scripts/gcp-triage.mjs';
describe('gcp-triage classify', () => {
it('classifies the Colyseus stale-seat refusal as self-healing', () => {
@@ -37,3 +37,43 @@ describe('gcp-triage classify', () => {
expect(classify('some brand new failure nobody has seen', ['three-ws-api'])).toBeNull();
});
});
+
+describe('gcp-triage HTTP classification', () => {
+ const request = ({ service = 'three-ws-api', status, path }) => ({
+ resource: { labels: { service_name: service } },
+ timestamp: '2026-09-15T00:00:00Z',
+ httpRequest: {
+ status,
+ requestMethod: status === 503 ? 'POST' : 'GET',
+ requestUrl: `https://three.ws${path}`,
+ userAgent: 'browser',
+ },
+ });
+
+ it('classifies Forge upload storage rejection as owner-gated', () => {
+ const [finding] = buildFindings([request({ status: 503, path: '/api/forge-upload' })]);
+ expect(finding.signature).toBe('r2-upload-unavailable');
+ expect(finding.class).toBe('owner');
+ });
+
+ it('classifies pump curve fallback exhaustion as self-healing', () => {
+ const [finding] = buildFindings([request({ status: 502, path: '/api/pump/curve' })]);
+ expect(finding.signature).toBe('pump-curve-rpc-unavailable');
+ expect(finding.class).toBe('self-healing');
+ });
+});
+
+describe('gcp-triage page failure summary', () => {
+ it('reports actual failures instead of the declared page total', () => {
+ const summary = summarizePageCheckFailure(`
+[check-pages] sweeping 822 declared pages against https://three.ws
+[check-pages] 50/822…
+[check-pages] 1 unreachable page(s) on https://three.ws:
+[check-pages] 404 /contributors
+[check-pages] The running revision is behind this checkout.
+`);
+ expect(summary.count).toBe(1);
+ expect(summary.sample).toContain('/contributors');
+ expect(summary.sample).not.toContain('50/822');
+ });
+});