Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(agents): answer a storage outage during registration with a designed 503, and teach triage to name it

nirholas6b53ff04a5da
diff --git a/api/agents/register/[action].js b/api/agents/register/[action].js
index ac3c865fb..a0f5a622e 100644
--- a/api/agents/register/[action].js
+++ b/api/agents/register/[action].js
@@ -9,7 +9,7 @@ import { parse } from '../../_lib/validate.js';
import { limits, clientIp } from '../../_lib/rate-limit.js';
import { env } from '../../_lib/env.js';
import { erc8004RegistryFields } from '../../_lib/three-brand.js';
-import { r2, publicUrl } from '../../_lib/r2.js';
+import { r2, publicUrl, isStorageInfrastructureError } from '../../_lib/r2.js';
import { SERVER_CHAIN_META } from '../../_lib/onchain.js';
import { evmRpcEndpoints } from '../../_lib/evm/rpc.js';
import { publishFeedEvent } from '../../_lib/feed.js';
@@ -128,7 +128,28 @@ async function handlePrep(req, res) {
...(body.embedPolicy && { embedPolicy: body.embedPolicy }),
...(body.demoSlug && { demoSlug: body.demoSlug }),
};
- const { cid, metadataURI } = await pinRegistrationJson(registrationJson);
+ // The manifest has to be stored somewhere resolvable before an on-chain
+ // registration can point at it, and the last-resort store is our own bucket:
+ // when the bucket rejects our credential there is nowhere to put it and the
+ // registration genuinely cannot proceed. Say that, instead of throwing the
+ // store's signing complaint at someone who was creating an agent (on
+ // 2026-09-07 a rejected R2 secret failed this path and the forge together, so
+ // an agent could be neither given an avatar nor registered).
+ let pinned;
+ try {
+ pinned = await pinRegistrationJson(registrationJson);
+ } catch (err) {
+ if (!isStorageInfrastructureError(err)) throw err;
+ console.error(`[register/prep] object storage rejected the manifest: ${err?.message || err}`);
+ res.setHeader('retry-after', '60');
+ return error(
+ res,
+ 503,
+ 'storage_unavailable',
+ 'Agent registration is temporarily unavailable while our asset storage recovers. Your avatar is safe: try again shortly.',
+ );
+ }
+ const { cid, metadataURI } = pinned;
const expiresAt = new Date(Date.now() + 60 * 60 * 1000);
// payload stores the pinned manifest plus a _local block confirm needs to
// bind the identity row (never pinned: added after the pin above).
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 2ef651cae..f97d0df56 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -37,8 +37,8 @@ var or add quota · 🟢 self-healing, no action needed.
> renders it with a plain-language fix for each degradation, and
> **`/api/healthz`** carries a machine-readable `subsystems` block (database, cache,
> rate limiter, Helius RPC, the whole Solana RPC lane tier (`rpc_lanes`), x402 ring,
-> **x402 settlement success**, **Forge 3D generation**, world, sniper, the OKX chat
-> bot, x402 config). The uptime cron
+> **x402 settlement success**, **Forge 3D generation**, **object storage**, world,
+> sniper, the OKX chat bot, x402 config). The uptime cron
> ([api/cron/uptime-check.js](../../api/cron/uptime-check.js)) parks a snapshot
> each tick and re-pages a degradation that persists. Source of the roll-up:
> [api/_lib/ops/subsystem-health.js](../../api/_lib/ops/subsystem-health.js). This
@@ -89,6 +89,56 @@ guard env violated:
---
+## 🔴 `The request signature we calculated does not match the signature you provided` (r2-credential)
+
+**Signature.** Any of: a 502 from `POST /api/forge` or `/api/gpt-forge` carrying that
+sentence, `[cdn-object] signed read failed, serving public bucket domain`,
+`[forge] object storage rejected the generation`, `[register/prep] object storage
+rejected the manifest`, or `object_storage: down` in `/api/healthz`.
+
+**What it is.** Cloudflare R2 is refusing our signed requests. The access key id is
+recognized (that would be `InvalidAccessKeyId`); the SECRET is wrong. Every signed
+operation fails at once, read and write.
+
+**Blast radius, which is the whole product.** 3D generation dies before it writes a
+row: the reference image cannot be parked, so text→3D fails on the website AND on
+the ChatGPT surfaces (they share the bucket, not the endpoint). Uploads cannot land.
+Agent registration cannot store its manifest. `/cdn/*` cannot read an object, so
+every avatar, thumbnail and GLB on the site falls back to the public bucket domain.
+
+**First seen 2026-09-07**, ~00:19 UTC: generation stopped dead for nearly five hours
+and was reported by users in Telegram, not by us, because nothing here had a signal
+for it. Two gaps were closed in the same change and both are worth knowing about:
+`forge_generation` read `ok, 89% success` throughout (its ledger only contains
+generations that got far enough to write a row, so a total stop looked like a quiet
+hour), and `isStorageInfrastructureError` matched only the compact
+`SignatureDoesNotMatch` code, which appears in `err.name` and never in the sentence
+above. `object_storage` now probes the credential directly and the forge sensor now
+reports a stall.
+
+**Resolution (owner).** Re-set the secret on the Cloud Run service:
+
+```bash
+gcloud run services update three-ws-api --region us-central1 \
+ --update-env-vars S3_SECRET_ACCESS_KEY=<secret> # or --update-secrets for a Secret Manager ref
+```
+
+Two traps that produce this exact error and look like a correct value:
+- On R2 the **Secret Access Key is the SHA-256 digest of the API token**, not the
+ token value. Pasting the token itself signs into `SignatureDoesNotMatch`.
+- A **trailing newline** in the stored secret fails identically. `env.js` trims
+ credential-class values now, so a padded value can only bite a consumer that
+ reads `process.env` directly.
+
+Verify with `curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[]
+| select(.name=="object_storage")'`, then one `POST /api/forge {"prompt":"cube"}`.
+
+**No code change routes around it.** The read path fails over to the public bucket
+domain (rate-limited, so it is a degradation, not a fix) and writes cannot fail over
+at all.
+
+---
+
## 🔴 `[text-to-image] replicate billing/credit failure: insufficient credit`
- **Source:** the forge text→image lane.
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index 5084ff3a4..363ae1d69 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -55,6 +55,12 @@ const KNOWN_SIGNATURES = [
class: 'owner',
action: `world.three.ws serving without ADMIN_CODE: every visitor has build rights. Owner runs deploy/world/apply-hardening.sh. ${RUNBOOK} §world.`,
},
+ {
+ id: 'r2-credential-rejected',
+ match: /does not match the signature you provided|SignatureDoesNotMatch|InvalidAccessKeyId|object storage rejected/i,
+ class: 'owner',
+ action: `Object storage is rejecting our credential: 3D generation, uploads and agent registration all fail, and /cdn falls back to the rate-limited public bucket domain. NO code path routes around it. Owner re-sets S3_SECRET_ACCESS_KEY on three-ws-api (on R2 the secret is the token's SHA-256 digest, not the token value). ${RUNBOOK} §r2-credential.`,
+ },
{
id: 'replicate-credit',
match: /replicate billing\/credit failure|insufficient credit/i,