Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(triage): classify /api/avatar/render and /api/avatar/capabilities 502s as the r2-downstream family
nirholascdbaa3a7dc00
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.mdindex 44ebb9f7b..de3cd47f8 100644--- a/docs/ops/production-log-triage.md+++ b/docs/ops/production-log-triage.md@@ -1301,7 +1301,7 @@ ERROR 502 POST /api/pump/launch-agent----## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`, `/api/render/*`, `/api/animations/clips/<id>`: `r2-downstream-5xx` (§r2-downstream)+## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`, `/api/render/*`, `/api/animations/clips/<id>`, `/api/avatar/render`, `/api/avatar/capabilities`: `r2-downstream-5xx` (§r2-downstream)- **Source:** every route that reads or writes through[api/_lib/r2.js](../../api/_lib/r2.js) (`api/forge.js`, `api/gpt-forge.js`,@@ -1316,7 +1316,11 @@ ERROR 502 POST /api/pump/launch-agentinstead of `api/_lib/r2.js`: `api/render/animate.js` (502 `render_failed`),`api/render/glb.js` and `api/render/avatar-clip.js` (502, same lib), andGET `api/animations/[id].js` (`/api/animations/clips/<id>`, 502- `storage_error` from `getObjectBuffer`).+ `storage_error` from `getObjectBuffer`). Added 2026-10-25: `api/avatar/render.js`+ (502 `render_failed` from `avatar-render.js`'s `renderAvatarImage`) and+ `api/avatar/capabilities.js` (502 `fetch_failed` from+ `avatar-capabilities.js`'s `inspectGlbCapabilities`), the same+ avatar.model_url fetch pattern under two more public route names.- **What it means:** Cloudflare R2 unreachable (healthz `object_storage``down`/`degraded`, tracked at[production-100-OWNER-ACTIONS.md](../../prompts/finish/_context/production-100-OWNER-ACTIONS.md)@@ -1354,7 +1358,8 @@ ERROR 502 POST /api/pump/launch-agentRegression-covered in[tests/gcp-triage.test.js](../../tests/gcp-triage.test.js) ("classifies ananimated-render R2 fetch failure..." / "...a clip-detail R2 fetch- failure...").+ failure..." / "...an avatar-render R2 fetch failure..." / "...an+ avatar-capabilities R2 fetch failure...").- **Worker-side app-log variant (same outage, different shape):** a GPU/meshworker that routes a caller-supplied model URL through`worker_security.py`'s `fetch_remote_bytes` (`remesh-service` confirmeddiff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjsindex 4dac08cbb..105802ee0 100644--- a/scripts/gcp-triage.mjs+++ b/scripts/gcp-triage.mjs@@ -399,17 +399,23 @@ const KNOWN_HTTP_SIGNATURES = [// getObjectBuffer rather than api/_lib/r2.js, but it is the identical// Cloudflare R2 flap: curl of one failing avatar's pub-*.r2.dev model_url// reproduced the exact 403 healthz already reports for object_storage.+ // 2026-10-25 deep sweep: /api/avatar/render (api/_lib/avatar-render.js's+ // renderAvatarImage) and /api/avatar/capabilities+ // (api/_lib/avatar-capabilities.js's inspectGlbCapabilities) fetch the+ // same avatar.model_url the same way and were missing from this list,+ // so both fell through to `investigate` every sweep during the flap.test: (g) => g.service === 'three-ws-api' && g.status >= 500 && (g.path === '/api/forge' || g.path === '/api/gpt-forge' ||g.path === '/api/glb' || g.path === '/api/avatars/upload' ||g.path === '/api/3d/inspect' || g.path === '/api/print/quote' ||g.path === '/api/render/animate' || g.path === '/api/render/glb' ||g.path === '/api/render/avatar-clip' ||+ g.path === '/api/avatar/render' || g.path === '/api/avatar/capabilities' ||/^\/cdn\//.test(g.path) || /^\/api\/avatars\/[^/]+\/glb$/.test(g.path) ||/^\/api\/animations\/clips\/[^/]+$/.test(g.path)),class: 'owner',- action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js, api/3d/inspect.js, api/render/animate.js, api/render/glb.js, api/render/avatar-clip.js and api/animations/[id].js fetch a caller- or avatar-supplied URL directly (502 render_failed / storage_error) that happens to be R2-hosted in the sample traffic, so confirm the sample URL's host before blaming R2 on those. ${RUNBOOK} §r2-downstream.`,+ action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js, api/3d/inspect.js, api/render/animate.js, api/render/glb.js, api/render/avatar-clip.js, api/animations/[id].js, api/avatar/render.js and api/avatar/capabilities.js fetch a caller- or avatar-supplied URL directly (502 render_failed / fetch_failed) that happens to be R2-hosted in the sample traffic, so confirm the sample URL's host before blaming R2 on those. ${RUNBOOK} §r2-downstream.`,},{id: 'pulse-not-deployed-503',diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.jsindex 53fe68306..dea44bea4 100644--- a/tests/gcp-triage.test.js+++ b/tests/gcp-triage.test.js@@ -237,6 +237,24 @@ describe('gcp-triage HTTP classification', () => {expect(finding.signature).toBe('r2-downstream-5xx');expect(finding.class).toBe('owner');});++ // 2026-10-25 deep sweep: /api/avatar/render and /api/avatar/capabilities+ // both fetch an avatar's own model_url (api/_lib/avatar-render.js's+ // renderAvatarImage, api/_lib/avatar-capabilities.js's inspectGlbCapabilities)+ // the same way /api/render/animate already does, but neither path was in+ // the r2-downstream-5xx list, so both fell through to `investigate` every+ // sweep during the R2 flap even though nothing in either handler is broken.+ it('classifies an avatar-render R2 fetch failure as the r2-downstream family', () => {+ const [finding] = buildFindings([request({ status: 502, path: '/api/avatar/render' })]);+ expect(finding.signature).toBe('r2-downstream-5xx');+ expect(finding.class).toBe('owner');+ });++ it('classifies an avatar-capabilities R2 fetch failure as the r2-downstream family', () => {+ const [finding] = buildFindings([request({ status: 502, path: '/api/avatar/capabilities' })]);+ expect(finding.signature).toBe('r2-downstream-5xx');+ expect(finding.class).toBe('owner');+ });});describe('gcp-triage page failure summary', () => {