Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(agents): answer a storage outage during registration with a designed 503, and teach triage to name it
nirholas6b53ff04a5da
diff --git a/api/agents/register/[action].js b/api/agents/register/[action].jsindex ac3c865fb..a0f5a622e 100644--- a/api/agents/register/[action].js+++ b/api/agents/register/[action].js@@ -9,7 +9,7 @@ import { parse } from '../../_lib/validate.js';import { limits, clientIp } from '../../_lib/rate-limit.js';import { env } from '../../_lib/env.js';import { erc8004RegistryFields } from '../../_lib/three-brand.js';-import { r2, publicUrl } from '../../_lib/r2.js';+import { r2, publicUrl, isStorageInfrastructureError } from '../../_lib/r2.js';import { SERVER_CHAIN_META } from '../../_lib/onchain.js';import { evmRpcEndpoints } from '../../_lib/evm/rpc.js';import { publishFeedEvent } from '../../_lib/feed.js';@@ -128,7 +128,28 @@ async function handlePrep(req, res) {...(body.embedPolicy && { embedPolicy: body.embedPolicy }),...(body.demoSlug && { demoSlug: body.demoSlug }),};- const { cid, metadataURI } = await pinRegistrationJson(registrationJson);+ // The manifest has to be stored somewhere resolvable before an on-chain+ // registration can point at it, and the last-resort store is our own bucket:+ // when the bucket rejects our credential there is nowhere to put it and the+ // registration genuinely cannot proceed. Say that, instead of throwing the+ // store's signing complaint at someone who was creating an agent (on+ // 2026-09-07 a rejected R2 secret failed this path and the forge together, so+ // an agent could be neither given an avatar nor registered).+ let pinned;+ try {+ pinned = await pinRegistrationJson(registrationJson);+ } catch (err) {+ if (!isStorageInfrastructureError(err)) throw err;+ console.error(`[register/prep] object storage rejected the manifest: ${err?.message || err}`);+ res.setHeader('retry-after', '60');+ return error(+ res,+ 503,+ 'storage_unavailable',+ 'Agent registration is temporarily unavailable while our asset storage recovers. Your avatar is safe: try again shortly.',+ );+ }+ const { cid, metadataURI } = pinned;const expiresAt = new Date(Date.now() + 60 * 60 * 1000);// payload stores the pinned manifest plus a _local block confirm needs to// bind the identity row (never pinned: added after the pin above).diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.mdindex 2ef651cae..f97d0df56 100644--- a/docs/ops/production-log-triage.md+++ b/docs/ops/production-log-triage.md@@ -37,8 +37,8 @@ var or add quota · 🟢 self-healing, no action needed.> renders it with a plain-language fix for each degradation, and> **`/api/healthz`** carries a machine-readable `subsystems` block (database, cache,> rate limiter, Helius RPC, the whole Solana RPC lane tier (`rpc_lanes`), x402 ring,-> **x402 settlement success**, **Forge 3D generation**, world, sniper, the OKX chat-> bot, x402 config). The uptime cron+> **x402 settlement success**, **Forge 3D generation**, **object storage**, world,+> sniper, the OKX chat bot, x402 config). The uptime cron> ([api/cron/uptime-check.js](../../api/cron/uptime-check.js)) parks a snapshot> each tick and re-pages a degradation that persists. Source of the roll-up:> [api/_lib/ops/subsystem-health.js](../../api/_lib/ops/subsystem-health.js). This@@ -89,6 +89,56 @@ guard env violated:---+## 🔴 `The request signature we calculated does not match the signature you provided` (r2-credential)++**Signature.** Any of: a 502 from `POST /api/forge` or `/api/gpt-forge` carrying that+sentence, `[cdn-object] signed read failed, serving public bucket domain`,+`[forge] object storage rejected the generation`, `[register/prep] object storage+rejected the manifest`, or `object_storage: down` in `/api/healthz`.++**What it is.** Cloudflare R2 is refusing our signed requests. The access key id is+recognized (that would be `InvalidAccessKeyId`); the SECRET is wrong. Every signed+operation fails at once, read and write.++**Blast radius, which is the whole product.** 3D generation dies before it writes a+row: the reference image cannot be parked, so text→3D fails on the website AND on+the ChatGPT surfaces (they share the bucket, not the endpoint). Uploads cannot land.+Agent registration cannot store its manifest. `/cdn/*` cannot read an object, so+every avatar, thumbnail and GLB on the site falls back to the public bucket domain.++**First seen 2026-09-07**, ~00:19 UTC: generation stopped dead for nearly five hours+and was reported by users in Telegram, not by us, because nothing here had a signal+for it. Two gaps were closed in the same change and both are worth knowing about:+`forge_generation` read `ok, 89% success` throughout (its ledger only contains+generations that got far enough to write a row, so a total stop looked like a quiet+hour), and `isStorageInfrastructureError` matched only the compact+`SignatureDoesNotMatch` code, which appears in `err.name` and never in the sentence+above. `object_storage` now probes the credential directly and the forge sensor now+reports a stall.++**Resolution (owner).** Re-set the secret on the Cloud Run service:++```bash+gcloud run services update three-ws-api --region us-central1 \+ --update-env-vars S3_SECRET_ACCESS_KEY=<secret> # or --update-secrets for a Secret Manager ref+```++Two traps that produce this exact error and look like a correct value:+- On R2 the **Secret Access Key is the SHA-256 digest of the API token**, not the+ token value. Pasting the token itself signs into `SignatureDoesNotMatch`.+- A **trailing newline** in the stored secret fails identically. `env.js` trims+ credential-class values now, so a padded value can only bite a consumer that+ reads `process.env` directly.++Verify with `curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[]+| select(.name=="object_storage")'`, then one `POST /api/forge {"prompt":"cube"}`.++**No code change routes around it.** The read path fails over to the public bucket+domain (rate-limited, so it is a degradation, not a fix) and writes cannot fail over+at all.++---+## 🔴 `[text-to-image] replicate billing/credit failure: insufficient credit`- **Source:** the forge text→image lane.diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjsindex 5084ff3a4..363ae1d69 100644--- a/scripts/gcp-triage.mjs+++ b/scripts/gcp-triage.mjs@@ -55,6 +55,12 @@ const KNOWN_SIGNATURES = [class: 'owner',action: `world.three.ws serving without ADMIN_CODE: every visitor has build rights. Owner runs deploy/world/apply-hardening.sh. ${RUNBOOK} §world.`,},+ {+ id: 'r2-credential-rejected',+ match: /does not match the signature you provided|SignatureDoesNotMatch|InvalidAccessKeyId|object storage rejected/i,+ class: 'owner',+ action: `Object storage is rejecting our credential: 3D generation, uploads and agent registration all fail, and /cdn falls back to the rate-limited public bucket domain. NO code path routes around it. Owner re-sets S3_SECRET_ACCESS_KEY on three-ws-api (on R2 the secret is the token's SHA-256 digest, not the token value). ${RUNBOOK} §r2-credential.`,+ },{id: 'replicate-credit',match: /replicate billing\/credit failure|insufficient credit/i,