Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(ops): make deep GCP triage reliable
nirholasb4c5fb463bf8
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.mdindex be04cccc6..f00a96fcf 100644--- a/docs/ops/production-log-triage.md+++ b/docs/ops/production-log-triage.md@@ -97,6 +97,18 @@ sentence, `[cdn-object] signed read failed, serving public bucket domain`,storage rejected the generation`, `[register/prep] object storage rejected themanifest`, or `object_storage: down` in `/api/healthz`.+The request-log-only form is `HTTP 503 POST /api/forge-upload`. The monitor+classifies that as `r2-upload-unavailable` (owner) because the route has only+two 503 outcomes: storage is absent, or its signing credential is rejected.+Confirm which one from `healthz.object_storage`; do not spend a code-debugging+cycle on a deliberately honest storage preflight response.++The deep monitor runs its log, HTTP/TLS, fleet, page, cron, database and wallet+probes concurrently. Keep child processes asynchronous: a synchronous Cloud+SDK read blocks the Node event loop and turns healthy network probes into a+cluster of false timeouts. Cloud SDK inventory calls also get a three-minute+ceiling because concurrent credential/config reads may serialize.+**What it is.** Cloudflare R2 is refusing our signed requests, and the SECRET iswrong. Every signed operation fails at once, read and write.@@ -718,6 +730,12 @@ ws error: Unexpected server response: 301matrix (which free node refuses which method, and why PublicNode as primarysilently breaks holder gating), the rotate-vs-fail classification contract,and the recovery commands.+- **Bonding-curve symptom:** `HTTP 502 GET /api/pump/curve` (and the v1 alias)+ answers `upstream_error` only after the read-only RPC/Jupiter fallback is+ exhausted. The monitor classifies an isolated occurrence as+ `pump-curve-rpc-unavailable` (self-healing). Escalate it only when it persists+ after `healthz.rpc_lanes` shows at least one healthy lane; no transaction was+ broadcast and retrying cannot spend user funds.---diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjsindex 363ae1d69..d5c91c857 100644--- a/scripts/gcp-triage.mjs+++ b/scripts/gcp-triage.mjs@@ -219,6 +219,19 @@ const KNOWN_HTTP_SIGNATURES = [class: 'env-action',action: `502 on a /api/coin/* route means every CoinGecko rung failed and no cached last-good existed. The usual cause is the DEMO KEY, not the upstream: the demo tier caps at 10,000 calls per MONTH, and once exhausted every request carrying COINGECKO_API_KEY gets 429 while the identical keyless request is still served (2026-07-28: /coin/detail, /tickers and /exchange all 502'd for hours on an exhausted key). Confirm in one call: curl -s -H "x-cg-demo-api-key: $KEY" https://api.coingecko.com/api/v3/key — error_code 10006 is the cap. geckoFetch now benches a rejected key for 15 min and retries keyless on its own, so a lingering 502 means the keyless tier is ALSO throttled (the Cloud Run egress IP is shared). Fix: gcloud run services update three-ws-api --region us-central1 --remove-env-vars COINGECKO_API_KEY (config-only, pre-approved) to stop paying the round trip, and tell the owner the key needs a paid tier or a monthly reset. ${RUNBOOK} §coingecko-quota-exhausted.`,},+ {+ id: 'r2-upload-unavailable',+ test: (g) => g.service === 'three-ws-api' && g.status === 503 && g.path === '/api/forge-upload',+ class: 'owner',+ action: `The upload preflight intentionally returns 503 when object storage is unconfigured or rejects its credential. Confirm healthz object_storage; SignatureDoesNotMatch means the owner must rotate the R2 credential and add a new s3-secret-access-key Secret Manager version. No code or retry fixes a rejected signing secret. ${RUNBOOK} §r2-credential.`,+ },+ {+ id: 'pump-curve-rpc-unavailable',+ test: (g) => g.service === 'three-ws-api' && g.status === 502+ && (g.path === '/api/pump/curve' || g.path === '/api/v1/pump/curve'),+ class: 'self-healing',+ action: `The read-only bonding-curve route exhausted its RPC/Jupiter fallback and returned an honest no-store upstream_error. Check healthz rpc_lanes: short cooling windows recover automatically. Investigate only if the same route persists across sweeps after at least one RPC lane is healthy; then inspect api/_lib/pump-curve-view.js rather than retrying a transaction. ${RUNBOOK} §solana-rpc.`,+ },{id: 'cc-unconfigured-503',// Every /api/community/* and /api/clash route wraps the same client and@@ -297,17 +310,20 @@ async function fetchHealthz() {}}-function readLogs(opts) {+async function readLogs(opts) {const query = 'resource.type="cloud_run_revision" severity>=WARNING';- const res = spawnSync('gcloud', [+ // Keep this asynchronous. The deep sweep deliberately runs its HTTP/TLS+ // probes alongside the log read; spawnSync blocked Node's event loop for+ // 40+ seconds and made healthy 10-15 second probes time out in a batch.+ const res = await runCommand(['gcloud','logging', 'read', query,`--project=${opts.project}`,`--freshness=${opts.since}`,`--limit=${opts.limit}`,'--format=json',- ], { encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });- if (res.status !== 0) {- const why = res.error?.message || (res.stderr || '').trim() || 'unknown error';+ ], { timeoutMs: 180000 });+ if (res.code !== 0) {+ const why = (res.stderr || '').trim() || (res.timedOut ? 'timed out' : 'unknown error');console.error(`gcloud logging read failed: ${why}`);if (/ENOENT/.test(why)) console.error('gcloud is not installed at any known path; see scripts/lib/gcloud-path.mjs.');if (/reauth|invalid_grant|credential/i.test(why)) console.error('gcloud auth has lapsed; only the owner can re-run `gcloud auth login` here.');@@ -635,7 +651,12 @@ async function probeTls() {}async function probeFleet(opts) {- const r = await runCommand(['gcloud', 'run', 'services', 'list', `--project=${opts.project}`, '--region=us-central1', '--format=json'], { timeoutMs: 90000 });+ // Deep mode launches the Scheduler and logging audits at the same time. The+ // Cloud SDK can serialize credential/config reads under that load, so give+ // the fleet inventory the same ceiling as the log read. A healthy 47-service+ // inventory has exceeded 90 seconds during a concurrent sweep even though+ // the identical standalone command completed successfully.+ const r = await runCommand(['gcloud', 'run', 'services', 'list', `--project=${opts.project}`, '--region=us-central1', '--format=json'], { timeoutMs: 180000 });if (r.code !== 0) return { status: 'error', note: tail(r.stderr, 3) };const services = JSON.parse(r.stdout || '[]');const notReady = [];@@ -659,14 +680,26 @@ async function probePages() {const m = out.match(/all (\d+) declared pages reachable/);return { status: 'ok', note: m ? `all ${m[1]} advertised pages reachable` : 'all advertised pages reachable' };}- const lines = out.split('\n').filter((l) => l.includes('[check-pages]') && !l.includes(' OK ')).slice(0, 14);- const countMatch = out.match(/(\d+) declared page/);- return { status: 'findings', note: `${countMatch ? countMatch[1] : 'some'} page(s) failing`, findings: [deepFinding('pages', 'investigate', 'advertised pages failing on the live site', {- count: countMatch ? Number(countMatch[1]) : 1, services: ['three-ws-api'], sample: lines.join('\n'),+ const summary = summarizePageCheckFailure(out);+ return { status: 'findings', note: `${summary.count} page(s) failing`, findings: [deepFinding('pages', 'investigate', 'advertised pages failing on the live site', {+ count: summary.count, services: ['three-ws-api'], sample: summary.sample,action: 'Each failing path is advertised in the sitemap and llms.txt. The sweep output states whether it is a routing bug or deploy lag (route landed after the running image). Routing bug: add the vercel.json rewrite and vite input, commit. Deploy lag: note it for the next owner-approved deploy.',})] };}+export function summarizePageCheckFailure(output) {+ const out = String(output || '');+ const unreachable = Number(out.match(/(\d+) unreachable page\(s\)/)?.[1] || 0);+ const wrong = Number(out.match(/(\d+) page\(s\) answered 200 with another page's content/)?.[1] || 0);+ const count = unreachable + wrong || 1;+ const lines = out.split('\n').filter((line) => {+ if (!line.includes('[check-pages]')) return false;+ if (/\d+\/\d+…/.test(line) || line.includes(' sweeping ')) return false;+ return true;+ }).slice(-20);+ return { count, sample: lines.join('\n').slice(0, 1400) };+}+async function probeCronDrift() {const r = await runCommand(['node', 'scripts/check-cron-drift.mjs', '--json'], { timeoutMs: 120000 });let report;@@ -824,7 +857,7 @@ async function main() {const deepPromise = opts.deep ? runDeepSweep(opts) : Promise.resolve(null);const [healthz, entries] = await Promise.all([fetchHealthz(),- Promise.resolve().then(() => readLogs(opts)),+ readLogs(opts),]);const findings = buildFindings(entries);findings.push(...gpuCapacityFindings(findings));diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.jsindex ec7fd39f7..5c0b478ae 100644--- a/tests/gcp-triage.test.js+++ b/tests/gcp-triage.test.js@@ -2,7 +2,7 @@// added to KNOWN_SIGNATURES should get a case here: the monitor learning a// benign pattern must never silently swallow a genuine fault that shares text.import { describe, expect, it } from 'vitest';-import { classify } from '../scripts/gcp-triage.mjs';+import { buildFindings, classify, summarizePageCheckFailure } from '../scripts/gcp-triage.mjs';describe('gcp-triage classify', () => {it('classifies the Colyseus stale-seat refusal as self-healing', () => {@@ -37,3 +37,43 @@ describe('gcp-triage classify', () => {expect(classify('some brand new failure nobody has seen', ['three-ws-api'])).toBeNull();});});++describe('gcp-triage HTTP classification', () => {+ const request = ({ service = 'three-ws-api', status, path }) => ({+ resource: { labels: { service_name: service } },+ timestamp: '2026-09-15T00:00:00Z',+ httpRequest: {+ status,+ requestMethod: status === 503 ? 'POST' : 'GET',+ requestUrl: `https://three.ws${path}`,+ userAgent: 'browser',+ },+ });++ it('classifies Forge upload storage rejection as owner-gated', () => {+ const [finding] = buildFindings([request({ status: 503, path: '/api/forge-upload' })]);+ expect(finding.signature).toBe('r2-upload-unavailable');+ expect(finding.class).toBe('owner');+ });++ it('classifies pump curve fallback exhaustion as self-healing', () => {+ const [finding] = buildFindings([request({ status: 502, path: '/api/pump/curve' })]);+ expect(finding.signature).toBe('pump-curve-rpc-unavailable');+ expect(finding.class).toBe('self-healing');+ });+});++describe('gcp-triage page failure summary', () => {+ it('reports actual failures instead of the declared page total', () => {+ const summary = summarizePageCheckFailure(`+[check-pages] sweeping 822 declared pages against https://three.ws+[check-pages] 50/822…+[check-pages] 1 unreachable page(s) on https://three.ws:+[check-pages] 404 /contributors+[check-pages] The running revision is behind this checkout.+`);+ expect(summary.count).toBe(1);+ expect(summary.sample).toContain('/contributors');+ expect(summary.sample).not.toContain('50/822');+ });+});