Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(triage): widen r2-downstream-5xx to cover avatar-render routes
nirholas5afb045cfa71
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.mdindex 5df0e0bcd..7e2cff456 100644--- a/docs/ops/production-log-triage.md+++ b/docs/ops/production-log-triage.md@@ -1237,34 +1237,47 @@ ERROR 502 POST /api/pump/launch-agent----## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`: `r2-downstream-5xx` (§r2-downstream)+## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`, `/api/render/*`, `/api/animations/clips/<id>`: `r2-downstream-5xx` (§r2-downstream)- **Source:** every route that reads or writes through[api/_lib/r2.js](../../api/_lib/r2.js) (`api/forge.js`, `api/gpt-forge.js`,`api/objects/library.js`, `api/avatars/[id].js`), the entire `/cdn/*`namespace (routed by `vercel.json`'s single `/cdn/(.+)` catch-all to[api/cdn-object.js](../../api/cdn-object.js), which always reads R2; no- subpath is an exception), plus two generic caller-supplied-URL fetch tools+ subpath is an exception), two generic caller-supplied-URL fetch tools(`api/print/quote.js`, `api/3d/inspect.js`, 502 `fetch_failed`) that happento be fetching R2-hosted GLBs in practice because forge/gpt-forge persist- generated models there.+ generated models there, and (added 2026-10-12) the avatar-render family that+ fetches an avatar's own `model_url` through `fetchModel`/`getObjectBuffer`+ instead of `api/_lib/r2.js`: `api/render/animate.js` (502 `render_failed`),+ `api/render/glb.js` and `api/render/avatar-clip.js` (502, same lib), and+ GET `api/animations/[id].js` (`/api/animations/clips/<id>`, 502+ `storage_error` from `getObjectBuffer`).- **What it means:** Cloudflare R2 flapping (healthz `object_storage``down`/`degraded`, tracked at[production-100-OWNER-ACTIONS.md](../../prompts/finish/_context/production-100-OWNER-ACTIONS.md)row 36) fans out into 5xx on this whole family at once with zero matchingapp-level crash logs, because each handler returns a clean upstream-failurestatus rather than throwing. Triage-lane sweep 2026-10-11 saw all nine- routes fire together inside the same window.+ original routes fire together inside the same window; the 2026-10-12 sweep+ reproduced it live for `/api/render/animate` by curling the failing+ avatar's own `model_url` (`https://pub-<account>.r2.dev/u/<owner>/<slug>.glb`)+ directly and getting the same `403` Cloudflare answers for every other+ route in this family.- **Resolve:** check healthz first:`curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'`.`down`/`degraded` confirms this family and the fix is the owner-sideCloudflare dashboard check in row 36. If `object_storage` reads `ok`,stop treating this as one signature and read the specific route's handler:- the four R2-direct routes point at a real credential/bucket problem, the two- fetch-tool routes point at whatever host the sample URL actually names- (confirm it is really `.r2.dev` before blaming R2).+ the R2-direct routes point at a real credential/bucket problem, the+ fetch-tool and avatar-render routes point at whatever host the sample URL+ actually names (confirm it is really `.r2.dev` before blaming R2).- **Monitor signature:** `r2-downstream-5xx` in[scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), classified `owner`.+ Regression-covered in+ [tests/gcp-triage.test.js](../../tests/gcp-triage.test.js) ("classifies an+ animated-render R2 fetch failure..." / "...a clip-detail R2 fetch+ failure...").---diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjsindex d56c6abb8..fa9030997 100644--- a/scripts/gcp-triage.mjs+++ b/scripts/gcp-triage.mjs@@ -342,15 +342,24 @@ const KNOWN_HTTP_SIGNATURES = [// which always reads R2 (api/cdn-object.js's own header comment): widen// the match to the whole /cdn/ namespace instead of enumerating every// subpath one at a time (2026-10-12: /cdn/avatars/mixamo/glb/*.glb hit- // the same R2 flap and was missing from the list).+ // the same R2 flap and was missing from the list). Also 2026-10-12+ // triage-lane sweep: /api/render/animate, /api/render/glb,+ // /api/render/avatar-clip and GET /api/animations/clips/<id> each fetch+ // an avatar's own model_url (a pub-*.r2.dev URL) through fetchModel /+ // getObjectBuffer rather than api/_lib/r2.js, but it is the identical+ // Cloudflare R2 flap: curl of one failing avatar's pub-*.r2.dev model_url+ // reproduced the exact 403 healthz already reports for object_storage.test: (g) => g.service === 'three-ws-api' && g.status >= 500 && (g.path === '/api/forge' || g.path === '/api/gpt-forge' ||g.path === '/api/glb' || g.path === '/api/avatars/upload' ||g.path === '/api/3d/inspect' || g.path === '/api/print/quote' ||- /^\/cdn\//.test(g.path) || /^\/api\/avatars\/[^/]+\/glb$/.test(g.path)+ g.path === '/api/render/animate' || g.path === '/api/render/glb' ||+ g.path === '/api/render/avatar-clip' ||+ /^\/cdn\//.test(g.path) || /^\/api\/avatars\/[^/]+\/glb$/.test(g.path) ||+ /^\/api\/animations\/clips\/[^/]+$/.test(g.path)),class: 'owner',- action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js and api/3d/inspect.js are generic caller-URL fetch tools (502 fetch_failed) that happen to be fetching R2-hosted models in the sample traffic, so confirm the sample URL's host before blaming R2 on those two. ${RUNBOOK} §r2-downstream.`,+ action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js, api/3d/inspect.js, api/render/animate.js, api/render/glb.js, api/render/avatar-clip.js and api/animations/[id].js fetch a caller- or avatar-supplied URL directly (502 render_failed / storage_error) that happens to be R2-hosted in the sample traffic, so confirm the sample URL's host before blaming R2 on those. ${RUNBOOK} §r2-downstream.`,},{id: 'pulse-not-deployed-503',diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.jsindex 6daea71e1..9c502d5a0 100644--- a/tests/gcp-triage.test.js+++ b/tests/gcp-triage.test.js@@ -120,6 +120,28 @@ describe('gcp-triage HTTP classification', () => {expect(finding.signature).toBe('holders-rpc-exhausted-503');expect(finding.class).toBe('self-healing');});++ // 2026-10-12 triage-lane sweep: confirmed live against production that+ // /api/render/animate?avatar=<id> and GET /api/animations/clips/<id> both+ // 502 with the exact same root cause as the rest of the r2-downstream-5xx+ // family (curl of the avatar's own pub-*.r2.dev model_url returned 403,+ // matching healthz object_storage "down"). Both routes fetch an R2-hosted+ // GLB outside api/_lib/r2.js (fetchModel / getObjectBuffer), so neither+ // matched the existing path list and both fell through to `investigate`+ // every sweep even though nothing in the handler is broken.+ it('classifies an animated-render R2 fetch failure as the r2-downstream family', () => {+ const [finding] = buildFindings([request({ status: 502, path: '/api/render/animate' })]);+ expect(finding.signature).toBe('r2-downstream-5xx');+ expect(finding.class).toBe('owner');+ });++ it('classifies a clip-detail R2 fetch failure as the r2-downstream family', () => {+ const [finding] = buildFindings([+ request({ status: 502, path: '/api/animations/clips/345b05d0-f81e-4f6c-9df8-c3f73572716c' }),+ ]);+ expect(finding.signature).toBe('r2-downstream-5xx');+ expect(finding.class).toBe('owner');+ });});describe('gcp-triage page failure summary', () => {