Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): widen r2-downstream-5xx to cover avatar-render routes

nirholas5afb045cfa71
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 5df0e0bcd..7e2cff456 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -1237,34 +1237,47 @@ ERROR 502 POST /api/pump/launch-agent
---
-## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`: `r2-downstream-5xx` (§r2-downstream)
+## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`, `/api/render/*`, `/api/animations/clips/<id>`: `r2-downstream-5xx` (§r2-downstream)
- **Source:** every route that reads or writes through
[api/_lib/r2.js](../../api/_lib/r2.js) (`api/forge.js`, `api/gpt-forge.js`,
`api/objects/library.js`, `api/avatars/[id].js`), the entire `/cdn/*`
namespace (routed by `vercel.json`'s single `/cdn/(.+)` catch-all to
[api/cdn-object.js](../../api/cdn-object.js), which always reads R2; no
- subpath is an exception), plus two generic caller-supplied-URL fetch tools
+ subpath is an exception), two generic caller-supplied-URL fetch tools
(`api/print/quote.js`, `api/3d/inspect.js`, 502 `fetch_failed`) that happen
to be fetching R2-hosted GLBs in practice because forge/gpt-forge persist
- generated models there.
+ generated models there, and (added 2026-10-12) the avatar-render family that
+ fetches an avatar's own `model_url` through `fetchModel`/`getObjectBuffer`
+ instead of `api/_lib/r2.js`: `api/render/animate.js` (502 `render_failed`),
+ `api/render/glb.js` and `api/render/avatar-clip.js` (502, same lib), and
+ GET `api/animations/[id].js` (`/api/animations/clips/<id>`, 502
+ `storage_error` from `getObjectBuffer`).
- **What it means:** Cloudflare R2 flapping (healthz `object_storage`
`down`/`degraded`, tracked at
[production-100-OWNER-ACTIONS.md](../../prompts/finish/_context/production-100-OWNER-ACTIONS.md)
row 36) fans out into 5xx on this whole family at once with zero matching
app-level crash logs, because each handler returns a clean upstream-failure
status rather than throwing. Triage-lane sweep 2026-10-11 saw all nine
- routes fire together inside the same window.
+ original routes fire together inside the same window; the 2026-10-12 sweep
+ reproduced it live for `/api/render/animate` by curling the failing
+ avatar's own `model_url` (`https://pub-<account>.r2.dev/u/<owner>/<slug>.glb`)
+ directly and getting the same `403` Cloudflare answers for every other
+ route in this family.
- **Resolve:** check healthz first:
`curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'`.
`down`/`degraded` confirms this family and the fix is the owner-side
Cloudflare dashboard check in row 36. If `object_storage` reads `ok`,
stop treating this as one signature and read the specific route's handler:
- the four R2-direct routes point at a real credential/bucket problem, the two
- fetch-tool routes point at whatever host the sample URL actually names
- (confirm it is really `.r2.dev` before blaming R2).
+ the R2-direct routes point at a real credential/bucket problem, the
+ fetch-tool and avatar-render routes point at whatever host the sample URL
+ actually names (confirm it is really `.r2.dev` before blaming R2).
- **Monitor signature:** `r2-downstream-5xx` in
[scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), classified `owner`.
+ Regression-covered in
+ [tests/gcp-triage.test.js](../../tests/gcp-triage.test.js) ("classifies an
+ animated-render R2 fetch failure..." / "...a clip-detail R2 fetch
+ failure...").
---
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index d56c6abb8..fa9030997 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -342,15 +342,24 @@ const KNOWN_HTTP_SIGNATURES = [
// which always reads R2 (api/cdn-object.js's own header comment): widen
// the match to the whole /cdn/ namespace instead of enumerating every
// subpath one at a time (2026-10-12: /cdn/avatars/mixamo/glb/*.glb hit
- // the same R2 flap and was missing from the list).
+ // the same R2 flap and was missing from the list). Also 2026-10-12
+ // triage-lane sweep: /api/render/animate, /api/render/glb,
+ // /api/render/avatar-clip and GET /api/animations/clips/<id> each fetch
+ // an avatar's own model_url (a pub-*.r2.dev URL) through fetchModel /
+ // getObjectBuffer rather than api/_lib/r2.js, but it is the identical
+ // Cloudflare R2 flap: curl of one failing avatar's pub-*.r2.dev model_url
+ // reproduced the exact 403 healthz already reports for object_storage.
test: (g) => g.service === 'three-ws-api' && g.status >= 500 && (
g.path === '/api/forge' || g.path === '/api/gpt-forge' ||
g.path === '/api/glb' || g.path === '/api/avatars/upload' ||
g.path === '/api/3d/inspect' || g.path === '/api/print/quote' ||
- /^\/cdn\//.test(g.path) || /^\/api\/avatars\/[^/]+\/glb$/.test(g.path)
+ g.path === '/api/render/animate' || g.path === '/api/render/glb' ||
+ g.path === '/api/render/avatar-clip' ||
+ /^\/cdn\//.test(g.path) || /^\/api\/avatars\/[^/]+\/glb$/.test(g.path) ||
+ /^\/api\/animations\/clips\/[^/]+$/.test(g.path)
),
class: 'owner',
- action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js and api/3d/inspect.js are generic caller-URL fetch tools (502 fetch_failed) that happen to be fetching R2-hosted models in the sample traffic, so confirm the sample URL's host before blaming R2 on those two. ${RUNBOOK} §r2-downstream.`,
+ action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js, api/3d/inspect.js, api/render/animate.js, api/render/glb.js, api/render/avatar-clip.js and api/animations/[id].js fetch a caller- or avatar-supplied URL directly (502 render_failed / storage_error) that happens to be R2-hosted in the sample traffic, so confirm the sample URL's host before blaming R2 on those. ${RUNBOOK} §r2-downstream.`,
},
{
id: 'pulse-not-deployed-503',
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index 6daea71e1..9c502d5a0 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -120,6 +120,28 @@ describe('gcp-triage HTTP classification', () => {
expect(finding.signature).toBe('holders-rpc-exhausted-503');
expect(finding.class).toBe('self-healing');
});
+
+ // 2026-10-12 triage-lane sweep: confirmed live against production that
+ // /api/render/animate?avatar=<id> and GET /api/animations/clips/<id> both
+ // 502 with the exact same root cause as the rest of the r2-downstream-5xx
+ // family (curl of the avatar's own pub-*.r2.dev model_url returned 403,
+ // matching healthz object_storage "down"). Both routes fetch an R2-hosted
+ // GLB outside api/_lib/r2.js (fetchModel / getObjectBuffer), so neither
+ // matched the existing path list and both fell through to `investigate`
+ // every sweep even though nothing in the handler is broken.
+ it('classifies an animated-render R2 fetch failure as the r2-downstream family', () => {
+ const [finding] = buildFindings([request({ status: 502, path: '/api/render/animate' })]);
+ expect(finding.signature).toBe('r2-downstream-5xx');
+ expect(finding.class).toBe('owner');
+ });
+
+ it('classifies a clip-detail R2 fetch failure as the r2-downstream family', () => {
+ const [finding] = buildFindings([
+ request({ status: 502, path: '/api/animations/clips/345b05d0-f81e-4f6c-9df8-c3f73572716c' }),
+ ]);
+ expect(finding.signature).toBe('r2-downstream-5xx');
+ expect(finding.class).toBe('owner');
+ });
});
describe('gcp-triage page failure summary', () => {