Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(ops): record why one bucket write could take all of text to 3D down
nirholasbcada0600438
diff --git a/docs/ops/forge-storage-failover.md b/docs/ops/forge-storage-failover.mdnew file mode 100644index 000000000..e001e5018--- /dev/null+++ b/docs/ops/forge-storage-failover.md@@ -0,0 +1,122 @@+# Forge storage failover and the generation probe++Two things that exist because of the same incident: object storage was allowed to+be a single point of failure for the platform's flagship flow, and nothing we ran+proved otherwise until users said so.++## What happened++On 2026-09-07 the Cloudflare R2 credential stopped verifying. Every signed+operation began answering `SignatureDoesNotMatch`, and text to 3D returned a 502+to every caller on every surface. The message users saw was the storage vendor's+own sentence, verbatim: *"The request signature we calculated does not match the+signature you provided. Check your secret access key and signing method."*++Two things made it worse than it needed to be:++- **Image to 3D kept working the whole time.** It supplies its own reference view+ URLs and never touches the write path, so half the product looked healthy.+- **`/api/healthz` reported `forge_generation: unknown, 0 finished generations`.**+ The failure happens *before* a `forge_creations` row is written, so a total+ outage and a quiet afternoon produce the identical signal.++## Why one write could do that++Text to 3D synthesizes a reference view (FLUX, Vertex Imagen, or Gemini) and then+has to hand that image to a reconstructor. Reconstructors take a URL, not bytes,+so the image is parked in the bucket first+([api/_lib/image-persist.js](../../api/_lib/image-persist.js)). That one+`putObject` sat on the critical path of every text prompt with no failover of any+kind: if it threw, the generation was over before a GPU was ever asked for+anything.++## The failover++Our own GPU workers never needed the bucket. Every `workers/model-*` service+declares its input as `images: [data-uri|url, ...]` and base64-decodes an inline+payload directly (see `_decode_image` in+[workers/model-trellis/main.py](../../workers/model-trellis/main.py), and the+same contract in `model-hunyuan3d` and `model-triposg`).++So `persistImageBytes` no longer throws on a storage fault. It returns the view+inline as a `data:` URI, and the generation continues:++| Condition | Result |+|---|---|+| Bucket healthy | Durable `https://` URL, exactly as before |+| Storage infrastructure fault, payload under 4 MB | Inline `data:` URI, generation proceeds |+| Storage infrastructure fault, payload over 4 MB | Rethrows (never smuggle megabytes into a request body) |+| Any other error | Rethrows (a programming bug must not hide behind a fallback) |++"Storage infrastructure fault" is `isStorageInfrastructureError` in+[api/_lib/r2.js](../../api/_lib/r2.js): a rejected or revoked credential, a+missing bucket, an unreachable endpoint. Object-level faults are not included,+because those blame the object and the fallback would be wrong.++### Which lanes may receive an inline view++Only ours. `backendAcceptsInlineViews` in+[api/_lib/forge-tiers.js](../../api/_lib/forge-tiers.js) is true exactly for the+backends whose `provider` is `gcp` (`trellis_selfhost`, `hunyuan3d`, `triposg`),+and it is derived from the registry rather than a hand-written list, so a backend+added later is classified by what it is.++A third-party reconstructor (NVIDIA NIM, Hugging Face Spaces, Replicate, and the+BYOK vendors) fetches a URL it is given. Handing one a data URI would fail+opaquely, so [api/forge.js](../../api/forge.js) refuses it with a designed 503+that names the real cause and suggests the lanes that can still run:++```json+{+ "error": "storage_unavailable",+ "message": "Our image storage is not accepting writes right now, so this engine cannot be given the reference view. Try again on the free built-in engine, or retry shortly.",+ "retryable": true,+ "retry_backends": ["trellis_selfhost", "hunyuan3d"]+}+```++That path also closes the information leak: a caller is no longer told to check+our secret access key.++An inline view is a transport detail and never becomes an artifact. It is not+echoed as `preview_image_url` and not written to `forge_creations`, where a+multi-megabyte base64 string would bloat every poll payload and the row. The+field is `null` for that generation, which is the truth: nothing was parked. The+turnaround multi-view lane is skipped for the same reason (it edits an image it+fetches by URL), so those generations reconstruct from the single view.++**This is degradation, not a fix.** A generation that runs on the fallback has no+durable copy of its reference view, cannot use multi-view fusion, and cannot run+on a vendor lane. Restore the credential; the failover only buys the time to do+it without the product being down.++## Proving it, instead of assuming it++```bash+npm run probe:generation # every row against production+node scripts/probe-generation-suite.mjs --only text-to-3d,rig --verbose+node scripts/probe-generation-suite.mjs --base http://localhost:3000 --json out.json+```++[scripts/probe-generation-suite.mjs](../../scripts/probe-generation-suite.mjs)+submits real work to the real endpoints, polls to completion, downloads the+result and asserts the bytes are what they claim to be: glTF magic and a+non-empty primitive for a mesh, a PNG signature for a cutout, a parseable clip+for a motion. A `configured: true` in the catalog counts for nothing here, which+is the entire point, because that is what read green throughout the outage.++It exits non-zero when any selected row fails, so it works as a gate. Rows that+need a signed-in user (region retexture) read `AUDIT_EMAIL` / `AUDIT_PASSWORD`+from `.env`; without them that row reports the missing session rather than+passing quietly.++Paid rows are asserted at the challenge, never by paying: the x402 leg checks+that the 402 carries usable Solana accepts and that the generation surfaces are+listed in `/.well-known/x402.json`.++## Related++- Daily automated version of the same bar: [api/cron/forge-smoke.js](../../api/cron/forge-smoke.js), 09:30 UTC, alerts to ops Telegram.+- Which failure classes recur, from the outcome ledger: [forge-error-triage.md](./forge-error-triage.md).+- The credential's own health signal: [api/_lib/ops/object-storage-health.js](../../api/_lib/ops/object-storage-health.js).+- Worker URLs and env: [gcp-model-workers.md](./gcp-model-workers.md).