Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): widen r2-downstream-5xx to the whole /cdn/* namespace, not just /cdn/forge/*

nirholas0f7f2030e157
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index a9f9b57c6..5df0e0bcd 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -1237,14 +1237,17 @@ ERROR 502 POST /api/pump/launch-agent
---
-## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`: `r2-downstream-5xx` (§r2-downstream)
+## 🔴 5xx fan-out across `/api/forge`, `/api/gpt-forge`, `/api/glb`, avatars, `/api/3d/inspect`, `/api/print/quote`, `/cdn/*`: `r2-downstream-5xx` (§r2-downstream)
- **Source:** every route that reads or writes through
[api/_lib/r2.js](../../api/_lib/r2.js) (`api/forge.js`, `api/gpt-forge.js`,
- `api/objects/library.js`, `api/avatars/[id].js`), plus two generic
- caller-supplied-URL fetch tools (`api/print/quote.js`, `api/3d/inspect.js`,
- 502 `fetch_failed`) that happen to be fetching R2-hosted GLBs in practice
- because forge/gpt-forge persist generated models there.
+ `api/objects/library.js`, `api/avatars/[id].js`), the entire `/cdn/*`
+ namespace (routed by `vercel.json`'s single `/cdn/(.+)` catch-all to
+ [api/cdn-object.js](../../api/cdn-object.js), which always reads R2; no
+ subpath is an exception), plus two generic caller-supplied-URL fetch tools
+ (`api/print/quote.js`, `api/3d/inspect.js`, 502 `fetch_failed`) that happen
+ to be fetching R2-hosted GLBs in practice because forge/gpt-forge persist
+ generated models there.
- **What it means:** Cloudflare R2 flapping (healthz `object_storage`
`down`/`degraded`, tracked at
[production-100-OWNER-ACTIONS.md](../../prompts/finish/_context/production-100-OWNER-ACTIONS.md)
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index f3599572d..d56c6abb8 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -336,13 +336,18 @@ const KNOWN_HTTP_SIGNATURES = [
// avatars/*/glb, avatars/upload, api/glb, api/3d/inspect,
// api/print/quote, cdn/objects/library/manifest.json all fired together
// with zero matching app-level error log lines, i.e. each is a
- // handler-returned 5xx, not a crash).
+ // handler-returned 5xx, not a crash). ANY /cdn/<key> path (not just
+ // /cdn/forge/* and the one objects/library manifest listed above) maps
+ // through vercel.json's single `/cdn/(.+)` -> api/cdn-object.js catch-all,
+ // which always reads R2 (api/cdn-object.js's own header comment): widen
+ // the match to the whole /cdn/ namespace instead of enumerating every
+ // subpath one at a time (2026-10-12: /cdn/avatars/mixamo/glb/*.glb hit
+ // the same R2 flap and was missing from the list).
test: (g) => g.service === 'three-ws-api' && g.status >= 500 && (
g.path === '/api/forge' || g.path === '/api/gpt-forge' ||
g.path === '/api/glb' || g.path === '/api/avatars/upload' ||
g.path === '/api/3d/inspect' || g.path === '/api/print/quote' ||
- /^\/cdn\/forge\//.test(g.path) || /^\/api\/avatars\/<uuid>\/glb$/.test(g.path) ||
- g.path === '/cdn/objects/library/manifest.json'
+ /^\/cdn\//.test(g.path) || /^\/api\/avatars\/[^/]+\/glb$/.test(g.path)
),
class: 'owner',
action: `Check healthz object_storage first (curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'): "down" or "degraded" means this is the known Cloudflare R2 flap, not a new bug (production-100-OWNER-ACTIONS.md row 36 tracks the owner-side Cloudflare dashboard check). If object_storage reads ok, read the specific route's handler instead; api/forge.js, api/gpt-forge.js, api/objects/library.js and api/avatars/[id].js import from api/_lib/r2.js directly, while api/print/quote.js and api/3d/inspect.js are generic caller-URL fetch tools (502 fetch_failed) that happen to be fetching R2-hosted models in the sample traffic, so confirm the sample URL's host before blaming R2 on those two. ${RUNBOOK} §r2-downstream.`,