Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(cdn): stream the r2.dev fallback instead of redirecting to it

nirholas2d29ab72b274
diff --git a/ISSUES.md b/ISSUES.md
index 0826ff3d3..2dc91696d 100644
--- a/ISSUES.md
+++ b/ISSUES.md
@@ -183,20 +183,21 @@ here; the code-quality items from that pass are not production issues.
9. **Live R2 CORS does not match `scripts/set-r2-cors.mjs`** (owner action:
one credential). CONFIRMED by measurement, not inference, and RE-CONFIRMED
- unchanged on 2026-09-04: `node scripts/set-r2-cors.mjs --probe` reads the
+ unchanged on 2026-09-09: `node scripts/set-r2-cors.mjs --probe` reads the
enforced policy from outside and exits 1 on this bucket. The probe needs no
credentials of any kind (it discovers the public host from a live listing
- endpoint and the upload host from the auth-free `/api/forge-upload`), so
- anyone can re-check this in one command.
+ endpoint, and the upload host from an auth-free presign route or from an
+ explicit `--endpoint=`/`--bucket=`, both non-secret), so anyone can re-check
+ this in one command.
- Measured again 2026-09-04, every row from raw `curl` as well as the probe:
+ Measured again 2026-09-09, every row from raw `curl` as well as the probe:
| Surface | Result |
|---|---|
| Site edge, `three.ws/avatars/*.glb`, foreign origin | PASS, `access-control-allow-origin: *` with `access-control-allow-methods: GET, HEAD, OPTIONS`. Not affected. |
| Site edge, first-party `three.ws/cdn/<key>`, foreign origin | PASS, `access-control-allow-origin: *`. Serves the same bucket objects, so it is a working route around the row below for any caller that has the object key. |
- | Public bucket host `pub-*.r2.dev`, foreign origin GET/HEAD | FAIL. Body returns `200`, but with no `access-control-allow-origin`, so the browser discards it. Allowlisted origins DO get their origin echoed (`Vary: Origin`), which is how the live read rule is known to still be the old allowlist rather than the world-open `*`. Its `OPTIONS` preflight is a bare `403`. |
- | Presigned `PUT` preflight on the S3 endpoint | Mixed, unchanged. `204` for `three.ws`, `*.vercel.app` (wildcard confirmed with a synthetic subdomain), `localhost:3000`; `403` for `www.three.ws`, `*.app.github.dev`, `localhost:5173`. |
+ | Public bucket host `pub-*.r2.dev`, foreign origin GET/HEAD | FAIL. On a real `200` object, `https://example.org` gets the body with no `access-control-allow-origin`, so the browser discards it. Allowlisted origins DO get their origin echoed (`Vary: Origin`), which is how the live read rule is known to still be the old allowlist rather than the world-open `*`. Its `OPTIONS` preflight is a bare `403`. |
+ | Presigned `PUT` preflight on the S3 endpoint | Mixed, unchanged. `204` for `three.ws`, `*.vercel.app` (wildcard confirmed with a synthetic subdomain), `localhost:3000`; `403` for `www.three.ws`, `*.app.github.dev`, `localhost:5173`, `example.org`. |
The live policy is one allowlist rule serving both reads and writes
(`GET, PUT, HEAD, POST, DELETE`), predating the script's split into a
@@ -210,14 +211,18 @@ here; the code-quality items from that pass are not production issues.
applies where (docs/media-api.md, docs/character-library.md, both embed
tutorials).
- Fixed on the way past, 2026-09-04: the site edge advertised
- `access-control-allow-methods: GET, HEAD, OPTIONS` on the media routes but
- answered `OPTIONS` with the 404 page, because the filesystem phase of
- [server/index.mjs](server/index.mjs) serves GET/HEAD only. A simple
- cross-origin GET was unaffected (it sends no preflight), but any fetch
- carrying a non-safelisted header was blocked. Those routes now answer the
- preflight `204` from the headers the route already collected
- (tests/server-media-cors-preflight.test.js). Ships with the next deploy.
+ Fixed on the way past, 2026-09-04, and now VERIFIED LIVE (2026-09-09): the
+ site edge advertised `access-control-allow-methods: GET, HEAD, OPTIONS` on
+ the media routes but answered `OPTIONS` with the 404 page, because the
+ filesystem phase of [server/index.mjs](server/index.mjs) serves GET/HEAD
+ only. A simple cross-origin GET was unaffected (it sends no preflight), but
+ any fetch carrying a non-safelisted header was blocked. Those routes now
+ answer the preflight `204` from the headers the route already collected
+ (tests/server-media-cors-preflight.test.js). Confirmed on the live site:
+ `curl -I -X OPTIONS -H 'Origin: https://example.org'
+ -H 'Access-Control-Request-Method: GET' -H 'Access-Control-Request-Headers: range'
+ https://three.ws/avatars/cesium-man.glb` returns `204` with
+ `access-control-allow-origin: *` and `access-control-allow-headers: range`.
Blocked on one credential, not on code. The only R2 token reachable from
this machine (`S3_*` in `.env`, identical to the Cloud Run service env,
@@ -235,6 +240,36 @@ here; the code-quality items from that pass are not production issues.
of crashing), drop it in `.env.local` as `R2_ACCESS_KEY_ID`/`R2_SECRET_ACCESS_KEY`,
then `node scripts/set-r2-cors.mjs` and confirm with `--probe`.
+10. **Object storage is DOWN in production: the bucket rejects our credential**
+ (owner action: one env var). Found 2026-09-09 while measuring item 9.
+ `/api/healthz` reports the `object_storage` subsystem `down` with
+ `credentialFault: true` and the bucket's own words: `The request signature
+ we calculated does not match the signature you provided.` The access key id
+ is accepted, so the SECRET is what is wrong, not the code and not the key.
+
+ Live symptoms, all measured 2026-09-09:
+
+ | Surface | Result |
+ |---|---|
+ | `POST /api/forge-upload` | `503 storage_unavailable`, so no reference image can be parked and no upload can land. |
+ | `GET /api/avatars/library`, `GET /api/objects/library` | `{"total":0}`, empty manifests where the catalog should be. |
+ | `forge_generation` subsystem | `down`, 0/30 finished over 6 hours, every failure on the image path. |
+
+ No lane failover routes around it: `/cdn/<key>` falls back to the
+ rate-limited public bucket domain, and the forge cannot stage inputs at all.
+ On Cloudflare R2 the secret access key is the API token's SHA-256 digest,
+ not the token value, and a trailing newline fails identically (`env.js`
+ trims, so a padded value can only come from a store read elsewhere).
+
+ `S3_SECRET_ACCESS_KEY` on `three-ws-api` is a Secret Manager reference
+ (`secret:s3-secret-access-key:latest`), and reading or rotating it is
+ owner-gated from this machine. Owner: re-set that secret to the token's
+ SHA-256 digest, then confirm with
+ `curl -s https://three.ws/api/healthz | grep -o '"name":"object_storage"[^}]*'`.
+ Nothing else needs redeploying. Note this must be fixed BEFORE item 9's
+ admin-token step is worth doing: a correct CORS policy on a bucket we
+ cannot authenticate to changes nothing users can see.
+
---
## Recently closed (do not re-open without new evidence)
diff --git a/scripts/set-r2-cors.mjs b/scripts/set-r2-cors.mjs
index fcb5da5fb..6dbc21acd 100644
--- a/scripts/set-r2-cors.mjs
+++ b/scripts/set-r2-cors.mjs
@@ -16,6 +16,8 @@
* node scripts/set-r2-cors.mjs --probe # measure the LIVE policy from outside (no credentials at all)
* node scripts/set-r2-cors.mjs --probe --site=https://staging.example # probe another deployment
* node scripts/set-r2-cors.mjs --probe --key=thumb/x.png # read a specific object
+ * node scripts/set-r2-cors.mjs --probe --endpoint=https://<account>.r2.cloudflarestorage.com --bucket=<name>
+ * # measure the write rule when the presign routes are down
* node scripts/set-r2-cors.mjs --get # read the live policy (needs an admin token)
* node scripts/set-r2-cors.mjs --dry-run # print the policy, don't push
* node scripts/set-r2-cors.mjs # apply (idempotent, needs an admin token)
@@ -243,32 +245,42 @@ async function probe() {
return false;
}
+ // A missing write host does NOT abort the probe. The `public-read` rule is
+ // the one third-party embeds actually hit, it is measurable on its own, and
+ // the presign routes that reveal the write host return 503 exactly when
+ // object storage is unhealthy. Reporting the half we can measure beats
+ // exiting with nothing measured at all.
const writeUrl = await probeWriteUrl(site);
if (!writeUrl) {
- console.error('--probe could not determine the presigned-upload host.');
- console.error(`Set S3_ENDPOINT + S3_BUCKET, or check that ${site}/api/forge-upload is reachable.`);
- return false;
+ console.warn('Note: the presigned-upload host could not be determined, so the');
+ console.warn('`browser-upload` (write) rule is NOT measured below and shows as "skip".');
+ console.warn(`Pass it explicitly to measure that rule too, no credentials needed:`);
+ console.warn(' node scripts/set-r2-cors.mjs --probe --endpoint=https://<account>.r2.cloudflarestorage.com --bucket=<name>');
+ console.warn('(both values are non-secret; read them with `node scripts/read-service-env.mjs \'^S3_ENDPOINT$\' --raw`)\n');
}
- console.log(`Probing ${Bucket || hostLabel(writeUrl)} (read: ${readUrl})\n`);
+ console.log(`Probing ${Bucket || hostLabel(writeUrl || readUrl)} (read: ${readUrl})\n`);
console.log(`${'ORIGIN'.padEnd(38)} ${'READ'.padEnd(6)} ${'WRITE'.padEnd(6)} EXPECTED`);
let ok = true;
for (const origin of PROBE_ORIGINS) {
const [read, write] = await Promise.all([
corsAllowed(readUrl, origin, 'GET'),
- corsAllowed(writeUrl, origin, 'PUT'),
+ writeUrl ? corsAllowed(writeUrl, origin, 'PUT') : Promise.resolve(null),
]);
const wantRead = ruleAllows('public-read', origin);
const wantWrite = ruleAllows('browser-upload', origin);
- const bad = read !== wantRead || write !== wantWrite;
+ const bad = read !== wantRead || (write !== null && write !== wantWrite);
if (bad) ok = false;
const want = `read=${wantRead ? 'yes' : 'no'} write=${wantWrite ? 'yes' : 'no'}${bad ? ' <- DRIFT' : ''}`;
- console.log(`${origin.padEnd(38)} ${(read ? 'yes' : 'no').padEnd(6)} ${(write ? 'yes' : 'no').padEnd(6)} ${want}`);
+ const writeCell = write === null ? 'skip' : write ? 'yes' : 'no';
+ console.log(`${origin.padEnd(38)} ${(read ? 'yes' : 'no').padEnd(6)} ${writeCell.padEnd(6)} ${want}`);
}
console.log('');
- if (ok) {
+ if (ok && !writeUrl) {
+ console.log('Live READ policy matches POLICY in this script. The write rule was not measured.');
+ } else if (ok) {
console.log('Live policy matches POLICY in this script.');
} else {
console.log('Live policy DIFFERS from POLICY in this script. Apply it with an admin token:');
@@ -326,28 +338,46 @@ async function discoverPublicAssetUrl(site, publicHost) {
return null;
}
-// The presigned-upload host. From env when credentials are present, else from
-// the site's own auth-free presign endpoint, which returns a real upload URL on
-// the exact host the browser PUTs to. Only its OPTIONS preflight is ever sent,
-// so no object is created.
+// The presigned-upload host. Three sources, best first: an explicit
+// --endpoint (the S3 endpoint and bucket name are not secrets, so anyone can
+// pass them), the same values from env, and finally the site's own auth-free
+// presign endpoints, which return a real upload URL on the exact host the
+// browser PUTs to. Only an OPTIONS preflight is ever sent, so no object is
+// created and no presigned signature is ever used.
+//
+// Every presign endpoint is tried in turn rather than just the first: they all
+// hand back a URL on the one host we want, so any one of them answering is
+// enough. That matters because a presign route returns 503 while object storage
+// itself is unhealthy, and the write host is still perfectly measurable then.
async function probeWriteUrl(site) {
- if (process.env.S3_ENDPOINT && Bucket) {
- return `${process.env.S3_ENDPOINT.replace(/\/$/, '')}/${Bucket}/${PROBE_WRITE_KEY}`;
+ const endpoint = argValue('--endpoint') || process.env.S3_ENDPOINT;
+ const bucket = argValue('--bucket') || Bucket;
+ if (endpoint && bucket) {
+ return `${endpoint.replace(/\/$/, '')}/${bucket}/${PROBE_WRITE_KEY}`;
}
- try {
- const res = await fetch(`${site}/api/forge-upload`, {
- method: 'POST',
- headers: { 'content-type': 'application/json', 'x-forge-client': 'set-r2-cors-probe' },
- body: JSON.stringify({ content_type: 'image/png', size_bytes: 1024 }),
- });
- if (!res.ok) return null;
- const { upload_url: uploadUrl } = await res.json();
- if (!uploadUrl) return null;
- const u = new URL(uploadUrl);
- return `${u.origin}${u.pathname}`;
- } catch {
- return null;
+
+ const presignRoutes = [
+ ['/api/forge-upload', { content_type: 'image/png', size_bytes: 1024 }],
+ ['/api/avatar/presign-glb', { content_type: 'model/gltf-binary', size_bytes: 1024 }],
+ ['/api/avatar/presign-audio', { content_type: 'audio/mpeg', size_bytes: 1024 }],
+ ];
+ for (const [path, body] of presignRoutes) {
+ try {
+ const res = await fetch(`${site}${path}`, {
+ method: 'POST',
+ headers: { 'content-type': 'application/json', 'x-forge-client': 'set-r2-cors-probe' },
+ body: JSON.stringify(body),
+ });
+ if (!res.ok) continue;
+ const { upload_url: uploadUrl } = await res.json();
+ if (!uploadUrl) continue;
+ const u = new URL(uploadUrl);
+ return `${u.origin}${u.pathname}`;
+ } catch {
+ continue;
+ }
}
+ return null;
}
function hostLabel(url) {