Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(selfie): keep the avatar when object storage refuses the final write
nirholas8b426bc58755
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e8fd68b86..f6050d8f2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,7 @@ Public history for [three.ws](https://three.ws), newest first. New pages come fr## 2026-09-11- **Selfie to avatar is unstuck: a scan that used to wait forever now finishes in seconds**: Uploading a photo on the selfie page could sit on the same message indefinitely and never start the scan, with no error and nothing to retry. One copy of the face-tracking runtime held at our CDN was telling browsers to expect 11 MB and then sending a third of that, so the page waited for bytes that were never coming. That copy is gone, and three changes make the failure impossible to repeat silently: the face-tracking runtime now loads from three.ws first instead of a public CDN, any wait on it gives up after fifteen seconds and builds your avatar from the photo as it is, and a photo is never rejected for having no face in it just because the detector could not load. A new check also walks every large file on the site the way a browser asks for it and fails loudly when the bytes do not add up. Measured after the fix: a selfie became a rigged, animation-ready avatar in about fifteen seconds, and still finished in under a minute with the face-tracking runtime completely unreachable. (`/create/selfie`) `[fix, infra]`+- **Your selfie avatar now survives a storage outage instead of vanishing at the last step**: For part of today, a selfie could finish building and then be thrown away. The 3D model was generated correctly, but the step that files it into your library could not reach our object storage, so the page said the avatar finished and could not be saved, and trying again hit the same wall. The avatar was never actually lost: a complete copy sits with the service that built it. The last step now recognises a storage outage for what it is and keeps your avatar, serving it from that copy and flagging it to be filed away properly once storage is back, so a problem on our side costs a housekeeping step instead of the model you waited for. Anything that is not a storage outage still fails loudly rather than quietly papering over a real bug. The image API for agents also now answers with a clear temporarily unavailable instead of a generic failure, so it no longer reads as a bad prompt when nothing was wrong with the prompt. (`/create/selfie`) `[fix, infra]`## 2026-09-10diff --git a/api/_lib/reconstruct-finalize.js b/api/_lib/reconstruct-finalize.jsindex ffeeb3f4d..827df8b01 100644--- a/api/_lib/reconstruct-finalize.js+++ b/api/_lib/reconstruct-finalize.js@@ -16,7 +16,7 @@// deployments keep their exact current behavior until rigging is wired.import { sql } from './db.js';-import { putObject, publicUrl } from './r2.js';+import { putObject, publicUrl, isStorageInfrastructureError } from './r2.js';import { storageKeyFor, createAvatar } from './avatars.js';import { inspectGlb, isValidGlbHeader } from './glb-inspect.js';import { dispatchWebhooks } from './webhook-dispatch.js';@@ -28,6 +28,46 @@ import { getRegenProviderForMode } from './regen-provider.js';import { fetchProviderGlbBuffer } from './provider-result-url.js';import { registerReconstructionCreation } from './forge-store.js';+// Park the finished mesh in our bucket, and keep the reconstruction when the+// bucket refuses the write.+//+// This one PUT sat between a user and the avatar they had already waited for.+// When the R2 credential stopped verifying at 03:15 UTC on 2026-09-11 (the+// second such outage in three days), every selfie reconstruction completed on+// the GPU and was then thrown away here: materialize aborted, no avatar row was+// written, and the page could only say "Avatar finished but could not be saved."+// The mesh was never actually lost - the reconstruction worker had already+// parked it in a public, lifecycle-free GCS bucket, and that URL was sitting in+// the job row the whole time.+//+// So a storage-INFRASTRUCTURE fault (credential rejected or revoked, bucket+// missing, endpoint unreachable) now costs the durable copy in our bucket, not+// the user's avatar: the provider's own public URL becomes the storage key.+// `publicUrl()` passes an absolute key straight through, `copyObject()` already+// declines to copy one, and `defaultStorageMode()` records r2.present = false,+// so the avatar renders and animates while staying findable for a later+// re-copy. This is the same trade image-persist.js makes for reference images,+// and for the same reason: storage must never hold the flagship flow hostage.+//+// Anything else (a programming error, an oversized body) still throws - only a+// fault we can name is allowed to degrade.+//+// @returns {Promise<string>} the storage key to record: the bucket key on a+// successful write, else the provider's absolute URL.+async function storeGlbOrKeepProviderUrl({ key, body, metadata, providerUrl }) {+ try {+ await putObject({ key, body, contentType: 'model/gltf-binary', metadata });+ return key;+ } catch (err) {+ if (!isStorageInfrastructureError(err)) throw err;+ if (!/^https:\/\//i.test(providerUrl || '')) throw err;+ console.warn(+ `[reconstruct] object storage rejected the mesh (${String(err?.name || err?.Code || 'storage error')}); serving the provider copy at ${providerUrl} so the reconstruction survives`,+ );+ return providerUrl;+ }+}+function glbMetaFrom(info) {return info? {@@ -72,6 +112,7 @@ async function materializeReconstructAvatar({glbInfo,storageKey,slug,+ providerUrl = null,extraTags = [],sourceMetaExtra = {},}) {@@ -99,12 +140,17 @@ async function materializeReconstructAvatar({console.warn('[reconstruct] canonicalize skipped:', err?.message);}- await putObject({+ const finalKey = await storeGlbOrKeepProviderUrl({key: storageKey,body: glbBuf,- contentType: 'model/gltf-binary',metadata: { source: 'reconstruct', job_id: jobId },+ providerUrl,});+ // The provider copy predates the bone canonicalization above, so an avatar+ // served from it carries the model's original rig names. The browser+ // canonicalizes at load time (src/glb-canonicalize.js), so it still animates;+ // recording the pending copy keeps the difference visible rather than silent.+ const servedFromProvider = finalKey !== storageKey;const tags = [baseTag, ...extraTags];if (glbInfo && !glbInfo.isRigged && !tags.includes('unrigged')) tags.push('unrigged');@@ -115,7 +161,7 @@ async function materializeReconstructAvatar({const avatar = await createAvatar({userId,- storageKey,+ storageKey: finalKey,input: {slug,name,@@ -123,7 +169,14 @@ async function materializeReconstructAvatar({size_bytes: glbBuf.length,content_type: 'model/gltf-binary',source: 'reconstruct',- source_meta: { jobId, provider: job.provider, ...glbMetaFrom(glbInfo), ...promptMeta, ...sourceMetaExtra },+ source_meta: {+ jobId,+ provider: job.provider,+ ...glbMetaFrom(glbInfo),+ ...promptMeta,+ ...sourceMetaExtra,+ ...(servedFromProvider ? { servedFrom: 'provider', pendingBucketKey: storageKey } : {}),+ },visibility,tags,checksum_sha256: null,@@ -158,8 +211,8 @@ async function materializeReconstructAvatar({jobId,provider: job.provider,prompt: fromPrompt && params.prompt ? String(params.prompt) : name,- glbKey: storageKey,- glbUrl: publicUrl(storageKey),+ glbKey: finalKey,+ glbUrl: publicUrl(finalKey),sizeBytes: glbBuf.length,visibility,previewImageUrl: referenceUrl,@@ -212,19 +265,21 @@ export async function finalizeReconstructStage({ userId, jobId, job, glbUrl }) {const canRig = !!(info && !info.isRigged && provider?.instance);if (!canRig) {- const avatar = await materializeReconstructAvatar({ userId, jobId, job, glbBuf, glbInfo: info, storageKey, slug });+ const avatar = await materializeReconstructAvatar({ userId, jobId, job, glbBuf, glbInfo: info, storageKey, slug, providerUrl: glbUrl });return { status: 'done', resultAvatarId: avatar.id };}// Store the bare mesh durably first: it gives the rig model a stable URL to- // fetch and guarantees a fallback if rigging fails.- await putObject({+ // fetch and guarantees a fallback if rigging fails. When the bucket refuses+ // the write, the provider's own public copy already satisfies both roles, so+ // rigging proceeds instead of stranding a finished reconstruction.+ const unriggedKey = await storeGlbOrKeepProviderUrl({key: storageKey,body: glbBuf,- contentType: 'model/gltf-binary',metadata: { source: 'reconstruct', job_id: jobId, stage: 'unrigged' },+ providerUrl: glbUrl,});- const unriggedUrl = publicUrl(storageKey);+ const unriggedUrl = publicUrl(unriggedKey);let rigSubmission;try {@@ -246,6 +301,7 @@ export async function finalizeReconstructStage({ userId, jobId, job, glbUrl }) {glbInfo: info,storageKey,slug,+ providerUrl: glbUrl,extraTags: ['unrigged'],sourceMetaExtra: { rigError: String(rigErr?.message || rigErr) },});@@ -291,6 +347,7 @@ export async function pollRiggingStage({ userId, jobId, job }) {const info = isValidGlbHeader(glbBuf) ? inspectGlb(glbBuf) : null;const avatar = await materializeReconstructAvatar({userId, jobId, job, glbBuf, glbInfo: info, storageKey, slug,+ providerUrl: rig.unriggedUrl,extraTags: ['unrigged'],sourceMetaExtra: { rigError: 'rig job not pollable' },});@@ -304,6 +361,7 @@ export async function pollRiggingStage({ userId, jobId, job }) {const info = isValidGlbHeader(glbBuf) ? inspectGlb(glbBuf) : null;const avatar = await materializeReconstructAvatar({userId, jobId, job, glbBuf, glbInfo: info, storageKey, slug,+ providerUrl: update.resultGlbUrl,sourceMetaExtra: { rigged: true, rigJobId: rig.extJobId, reconstructGlb: rig.unriggedUrl },});return { status: 'done', resultAvatarId: avatar.id };@@ -315,6 +373,7 @@ export async function pollRiggingStage({ userId, jobId, job }) {const info = isValidGlbHeader(glbBuf) ? inspectGlb(glbBuf) : null;const avatar = await materializeReconstructAvatar({userId, jobId, job, glbBuf, glbInfo: info, storageKey, slug,+ providerUrl: rig.unriggedUrl,extraTags: ['unrigged'],sourceMetaExtra: { rigFailed: true, rigError: update.error || null },});diff --git a/api/_lib/storage-mode.js b/api/_lib/storage-mode.jsindex 16308ed70..3931430cc 100644--- a/api/_lib/storage-mode.js+++ b/api/_lib/storage-mode.js@@ -33,11 +33,21 @@ export const storageModeSchema = z.object({}),});+// An avatar whose storage_key is an absolute URL does NOT live in our bucket:+// it is served straight from the origin that produced it (a reconstruction+// parked in GCS when the R2 write could not be made). Reporting r2.present for+// one of those told every reader the bytes were in the bucket when no object+// existed, so a re-copy sweep could never find the ones that still need one.+function livesInBucket(storageKey) {+ return typeof storageKey === 'string' && storageKey !== '' && !/^https?:\/\//i.test(storageKey);+}+export function defaultStorageMode(avatarRow = {}) {+ const inBucket = livesInBucket(avatarRow.storage_key);return {version: STORAGE_MODE_VERSION,primary: 'r2',- r2: { present: true, key: avatarRow.storage_key ?? null },+ r2: { present: inBucket, key: inBucket ? avatarRow.storage_key : null },ipfs: { pinned: false, cid: null, pinned_at: null },attestation: {hash: avatarRow.checksum_sha256 ?? null,diff --git a/api/v1/ai/image.js b/api/v1/ai/image.jsindex 1c86a3b46..1a2220ad6 100644--- a/api/v1/ai/image.js+++ b/api/v1/ai/image.js@@ -39,6 +39,7 @@ import {SUPPORTED_ASPECT_RATIOS,} from '../../_lib/ai-image-lanes.js';import { peekFreeQuota, consumeFreeQuota, freePerDay } from '../../_lib/ai-image-quota.js';+import { isInlineImageRef } from '../../_lib/image-persist.js';const ROUTE = '/api/v1/ai/image';const PRICE_ATOMICS = priceFor('ai-image', '20000'); // $0.02 USDC per image above quota@@ -225,6 +226,23 @@ async function generateAndRespond({ prompt, aspectRatio, seed }, { free, ip }) {}const url = result?.imageUrl;+ // The lane drew the image but object storage refused to keep it, so the bytes+ // came back inline (see api/_lib/image-persist.js). Our own workers read an+ // inline reference happily; a third-party caller of this endpoint cannot,+ // because the contract here is a durable https URL it can fetch later. So the+ // honest answer is a transient storage fault, not "generation_failed". That+ // told the caller to retry a prompt that was never the problem, and told the+ // operator nothing at all while the R2 credential sat rejected for hours+ // (2026-09-09, and again from 03:15 UTC on 2026-09-11).+ if (isInlineImageRef(url)) {+ throw Object.assign(+ new Error('The image was generated but could not be stored: object storage is rejecting writes. Retry shortly.'),+ // 5xx descriptions are redacted to a support ref on this route, so the+ // machine-readable code is what a caller reacts to and the sentence is+ // what the operator reads next to that ref in the logs.+ { status: 503, code: 'storage_unavailable' },+ );+ }if (!url || !/^https?:\/\//.test(url)) {throw Object.assign(new Error('Image lane returned no usable image URL.'), {status: 502,diff --git a/data/changelog.json b/data/changelog.jsonindex 130295338..65c0fbc44 100644--- a/data/changelog.json+++ b/data/changelog.json@@ -1,6 +1,16 @@{"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security. `link` is optional and must be a live three.ws path.","entries": [+ {+ "date": "2026-09-11",+ "title": "Your selfie avatar now survives a storage outage instead of vanishing at the last step",+ "summary": "For part of today, a selfie could finish building and then be thrown away. The 3D model was generated correctly, but the step that files it into your library could not reach our object storage, so the page said the avatar finished and could not be saved, and trying again hit the same wall. The avatar was never actually lost: a complete copy sits with the service that built it. The last step now recognises a storage outage for what it is and keeps your avatar, serving it from that copy and flagging it to be filed away properly once storage is back, so a problem on our side costs a housekeeping step instead of the model you waited for. Anything that is not a storage outage still fails loudly rather than quietly papering over a real bug. The image API for agents also now answers with a clear temporarily unavailable instead of a generic failure, so it no longer reads as a bad prompt when nothing was wrong with the prompt.",+ "tags": [+ "fix",+ "infra"+ ],+ "link": "/create/selfie"+ },{"date": "2026-09-11","title": "Selfie to avatar is unstuck: a scan that used to wait forever now finishes in seconds",diff --git a/docs/ops/gcp-production.md b/docs/ops/gcp-production.mdindex b73deade4..cf17977a2 100644--- a/docs/ops/gcp-production.md+++ b/docs/ops/gcp-production.md@@ -731,6 +731,87 @@ across all callers, and bounds every vision-model load with15 seconds instead of hanging the flow. `npm run verify:selfie -- --stall-vision`reproduces the hostile case end to end.+### A rejected object-storage credential silently eats finished avatars (2026-09-11)++The R2 credential stopped verifying at **03:15 UTC on 2026-09-11**, the second+such outage in three days (the first ran 05:04-05:19 on 2026-09-09 and was+fixed by rotating the secret). Every signed operation answered+`SignatureDoesNotMatch`, for reads and writes alike.++Confirm it in one command, against whatever the service is actually running+(the secret is a Secret Manager reference, so `describe` shows `valueFrom`):++```bash+export S3_ENDPOINT=$(node scripts/read-service-env.mjs '^S3_ENDPOINT$' --raw)+export S3_BUCKET=$(node scripts/read-service-env.mjs '^S3_BUCKET$' --raw)+export S3_PUBLIC_DOMAIN=$(node scripts/read-service-env.mjs '^S3_PUBLIC_DOMAIN$' --raw)+export S3_ACCESS_KEY_ID=$(node scripts/read-service-env.mjs '^S3_ACCESS_KEY_ID$' --raw)+export S3_SECRET_ACCESS_KEY=$(node scripts/read-service-env.mjs '^S3_SECRET_ACCESS_KEY$' --raw)+node -e "const {putObject}=await import('./api/_lib/r2.js');+await putObject({key:'forge/refs/probe.png',body:Buffer.from('89504e470d0a1a0a','hex'),contentType:'image/png'});+console.log('write ok')"+```++Count the blast radius, which is what tells a credential fault from a code bug:++```bash+gcloud logging read 'resource.type="cloud_run_revision"+ resource.labels.service_name="three-ws-api" textPayload:"SignatureDoesNotMatch"' \+ --freshness=12h --limit=1000 --format='value(timestamp)' \+ --project aerial-vehicle-466722-p5 | cut -c1-13 | sort | uniq -c+```++A clean start hour with nothing before it is a credential that died at that+minute. Failures spread evenly across many hours are a different fault.++**Symptom at the product level:** `/create/selfie` reconstructed the mesh on+the GPU, took ~35 s, and then said **"Avatar finished but could not be saved.+Try again."** Retrying could not work, because the prompt was never the problem.+The job row is the ground truth, and it logs nothing:++```sql+select job_id, status, result_avatar_id, error from avatar_regen_jobs+order by created_at desc limit 5;+-- status 'done', result_avatar_id NULL, error NULL+```++`error` stays NULL **on purpose**: `/api/cron/reconstruct-sweep` recovers exactly+the rows matching `status = 'done' and error is null`, so writing the failure+there would strand the job instead of retrying it. Do not "fix" that NULL.++**Why it is no longer fatal.** `api/_lib/reconstruct-finalize.js` treats a+storage-infrastructure fault (`isStorageInfrastructureError`: credential+rejected or revoked, bucket missing, endpoint unreachable) as a reason to lose+the durable copy, never the reconstruction. The mesh is already public and+lifecycle-free in `gs://three-ws-avatar-reconstructions`, so that URL becomes+the avatar's storage key: `publicUrl()` passes an absolute key through,+`defaultStorageMode()` records `r2.present = false`, and `source_meta` carries+`servedFrom: 'provider'` plus the `pendingBucketKey` a later re-copy needs. Any+other fault still throws. Same trade `api/_lib/image-persist.js` makes for+reference images.++`POST /api/v1/ai/image` cannot make that trade (its contract is a durable https+URL a third party can fetch), so it answers **503 `storage_unavailable`** rather+than the old opaque 502 `generation_failed`, which told callers to retry a good+prompt and told operators nothing.++**Real recovery** is a new R2 API token, and it is owner-gated: no Cloudflare+API token exists on this machine or on the Cloud Run service, so a token cannot+be minted from here. Mint one in the Cloudflare dashboard (R2 > Manage API+tokens, Object Read & Write on the bucket in `S3_BUCKET`), then:++```bash+printf %s "<new secret>" | gcloud secrets versions add s3-secret-access-key \+ --data-file=- --project aerial-vehicle-466722-p5+# only if the access key id changed too (--update-env-vars MERGES; --set-env-vars REPLACES)+gcloud run services update three-ws-api --region us-central1 \+ --update-env-vars S3_ACCESS_KEY_ID=<new id> --project aerial-vehicle-466722-p5+```++`api/_lib/r2.js` re-probes a rejected credential every 15 s, so a fixed token+recovers without a redeploy. Avatars written during the outage keep serving from+GCS; find them with `storage_mode->'r2'->>'present' = 'false'`.+### Client geo header (analytics country column)`api/_lib/client-geo.js` resolves a visitor's country from an edge geo headerdiff --git a/docs/selfie-to-avatar.md b/docs/selfie-to-avatar.mdindex 0f37c37ca..9f008e2ea 100644--- a/docs/selfie-to-avatar.md+++ b/docs/selfie-to-avatar.md@@ -79,6 +79,7 @@ Request body: `name` (1-120, required), `photos` (1-6 image URLs or data URIs) *- **Quality gating.** Only a frontal with no detectable face hard-blocks. Blur, illustrations, or multiple faces are surfaced as warnings you can proceed past. If the face detector itself cannot load, the gate steps aside rather than blocking: an unavailable model is not evidence about your photo, and the worker runs its own face pass regardless.- **Vision models are on a deadline.** The client-side refinement pass (subject matte, reframe, face box) needs MediaPipe's 11 MB runtime plus a model file. Every wait on one is capped at 15 seconds (`withVisionDeadline` in [src/shared/mediapipe-assets.js](../src/shared/mediapipe-assets.js)); past that the raw photo is submitted and the reconstruction proceeds. Before that cap existed, an asset that never finished downloading left the page on "Processing your face..." indefinitely, with the reconstruction request never sent. The runtime is served from three.ws (`public/vendor/mediapipe/`) and only falls back to a public CDN when our own origin definitively lacks the file.- **Error handling.** Camera denial distinguishes `NotAllowedError` ("Camera access was denied... use Upload instead") from other failures. Job failures map to friendly messages (no face, content-safety flag, blur, service unavailable, timeout, out-of-memory). Poll tolerates up to 5 consecutive network errors with backoff before "Lost connection to the avatar engine." Rate limits (`429`) start a 60-second cooldown with a live countdown. Pending jobs resume across reloads.+- **A storage outage costs the filing, not the avatar.** The last step copies the finished mesh into our object storage before the avatar row is written. When that write fails for an infrastructure reason (credential rejected or revoked, bucket missing, endpoint unreachable), the avatar is still created and points at the reconstruction worker's own public copy in Cloud Storage, which is durable and has no expiry. `storage_mode.r2.present` is `false` on such a row and `source_meta` carries `servedFrom: 'provider'` with the `pendingBucketKey` a later re-copy needs, so these are findable rather than silent. Any other failure in that step still fails the job. Before this, a rejected storage credential threw away a mesh the GPU had already finished and left the page saying "Avatar finished but could not be saved" against a retry that could not succeed (2026-09-11; see [docs/ops/gcp-production.md](ops/gcp-production.md)).- **Output.** glTF 2.0 GLB with a Mixamo-standard humanoid rig, 52 ARKit blendshapes, and 15 visemes. Private avatars return a short-lived signed `url`; `model_url` is the public CDN URL once you make it public.- **Error copy.** The worker classifies a failure as either the caller's input or a service fault. An input rejection ("no face detected in any of the provided photos") or a plan refusal is relayed verbatim, and the status response carries `errorKind: 'input'` so the client shows that text as-is instead of rewriting an unrecognised message into generic "try a clearer photo" advice; a service fault is masked into neutral retry copy, because its raw text can carry a vendor name or an operator-only correlation id. Your visibility choice rides along to the Forge store, so a private capture never surfaces in a public gallery.diff --git a/public/changelog-recent.json b/public/changelog-recent.jsonindex eb1f2781a..1fcc3b924 100644--- a/public/changelog-recent.json+++ b/public/changelog-recent.json@@ -1,11 +1,11 @@{- "generated_at": "2026-09-11T01:21:22.349Z",+ "generated_at": "2026-09-11T05:05:14.644Z","generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json","site": {"name": "three.ws","url": "https://three.ws"},- "total": 3671,+ "total": 3672,"complete": false,"full_feed": "/changelog.json","entries": [@@ -20,6 +20,17 @@"infra"]},+ {+ "date": "2026-09-11",+ "type": "update",+ "title": "Your selfie avatar now survives a storage outage instead of vanishing at the last step",+ "summary": "For part of today, a selfie could finish building and then be thrown away. The 3D model was generated correctly, but the step that files it into your library could not reach our object storage, so the page said the avatar finished and could not be saved, and trying again hit the same wall. The avatar was never actually lost: a complete copy sits with the service that built it. The last step now recognises a storage outage for what it is and keeps your avatar, serving it from that copy and flagging it to be filed away properly once storage is back, so a problem on our side costs a housekeeping step instead of the model you waited for. Anything that is not a storage outage still fails loudly rather than quietly papering over a real bug. The image API for agents also now answers with a clear temporarily unavailable instead of a generic failure, so it no longer reads as a bad prompt when nothing was wrong with the prompt.",+ "link": "/create/selfie",+ "tags": [+ "fix",+ "infra"+ ]+ },{"date": "2026-09-10","type": "launch",@@ -1608,17 +1619,6 @@"security","infra"]- },- {- "date": "2026-09-07",- "type": "update",- "title": "Shared avatar and agent links now show a real page to Google, not a loading spinner",- "summary": "Search engines were shown a different version of every /avatars/... and /agents/... page than people were, and that version had nothing on it: the name and description were hidden in a fallback block that a crawler removes before reading, the rest was a spinner, and the last line of the page sent the crawler straight back to the URL it was already on. Tens of thousands of avatar and agent pages therefore looked identical to Google and were filed as duplicates instead of being listed. Crawlers now get the same content a visitor sees: the name, the description, the rendered card, the creator, the tags, the skills, the on-chain identity, and links onward into the gallery and the directory. A link to an avatar or agent that no longer exists now says so honestly instead of quietly bouncing to a listing page, an id that belongs to the other side of the platform redirects to where it actually lives, and a private agent is no longer readable by anything sending a crawler's name. Signed-in surfaces were also unblocked in robots.txt so their existing no-index instruction can finally be read and obeyed.",- "link": null,- "tags": [- "fix",- "infra"- ]}]}diff --git a/public/changelog.json b/public/changelog.jsonindex 487bd24a8..424a4d400 100644--- a/public/changelog.json+++ b/public/changelog.json@@ -1,5 +1,5 @@{- "generated_at": "2026-09-11T01:21:22.327Z",+ "generated_at": "2026-09-11T05:05:14.623Z","generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json","site": {"name": "three.ws",@@ -17,6 +17,17 @@"infra"]},+ {+ "date": "2026-09-11",+ "type": "update",+ "title": "Your selfie avatar now survives a storage outage instead of vanishing at the last step",+ "summary": "For part of today, a selfie could finish building and then be thrown away. The 3D model was generated correctly, but the step that files it into your library could not reach our object storage, so the page said the avatar finished and could not be saved, and trying again hit the same wall. The avatar was never actually lost: a complete copy sits with the service that built it. The last step now recognises a storage outage for what it is and keeps your avatar, serving it from that copy and flagging it to be filed away properly once storage is back, so a problem on our side costs a housekeeping step instead of the model you waited for. Anything that is not a storage outage still fails loudly rather than quietly papering over a real bug. The image API for agents also now answers with a clear temporarily unavailable instead of a generic failure, so it no longer reads as a bad prompt when nothing was wrong with the prompt.",+ "link": "/create/selfie",+ "tags": [+ "fix",+ "infra"+ ]+ },{"date": "2026-09-10","type": "launch",diff --git a/scripts/verify-selfie-flow.mjs b/scripts/verify-selfie-flow.mjsindex 705dcd07f..6faae6511 100644--- a/scripts/verify-selfie-flow.mjs+++ b/scripts/verify-selfie-flow.mjs@@ -156,6 +156,11 @@ page.on('console', (msg) => {page.on('requestfailed', (req) => {failedRequests.push(`${req.method()} ${req.url().slice(0, 160)}: ${req.failure()?.errorText}`);});+// A refusal from the submit call itself is terminal: the job never starts, so+// no pipeline event will ever fire and waiting out the full build timeout only+// delays the answer. A full avatar library (402 plan_limit) cost one run ten+// minutes to report something the first response already said.+let submitRefusal = null;page.on('response', async (res) => {const url = res.url();if (!url.includes('/api/avatars/')) return;@@ -163,6 +168,15 @@ page.on('response', async (res) => {if (!res.ok()) entry.body = (await res.text().catch(() => '')).slice(0, 300);apiCalls.push(entry);log(` ${entry.status} ${entry.url}${entry.body ? ` ${entry.body}` : ''}`);+ if (url.includes('/api/avatars/reconstruct') && res.status() >= 400) {+ let parsed = null;+ try { parsed = JSON.parse(entry.body || ''); } catch { parsed = null; }+ submitRefusal = {+ status: res.status(),+ code: parsed?.error || 'error',+ message: parsed?.error_description || entry.body || '(no body)',+ };+ }});const started = Date.now();@@ -203,7 +217,14 @@ await submit.click();// id the job produced; selfie:build-error carries the message the user is// shown. Anything else at the deadline is a stall, and the last progress label// says where it stalled.-const outcome = await page.evaluate(async (timeout) => {+const outcome = await Promise.race([+ // The submit call was refused outright, so stop instead of waiting for a+ // pipeline that will never run.+ (async () => {+ while (!submitRefusal) await new Promise((r) => setTimeout(r, 250));+ return { state: 'refused', detail: `${submitRefusal.status} ${submitRefusal.code}: ${submitRefusal.message}`, ms: 0, events: [] };+ })(),+ page.evaluate(async (timeout) => {const started = Date.now();const events = () => window.__selfieEvents || [];const last = (name) => [...events()].reverse().find((e) => e.name === name);@@ -250,7 +271,8 @@ const outcome = await page.evaluate(async (timeout) => {}}, 1000);});-}, buildTimeoutMs);+}, buildTimeoutMs),+]);const trail = (outcome.events || []).map((e) => e.name.replace('selfie:', '')).join(' -> ');log(`pipeline: ${trail || '(no events fired)'}`);@@ -274,5 +296,8 @@ if (outcome.state === 'built') {console.log(`\n built in ${elapsed}s: ${avatar}\n evidence in ${outDir}\n`);process.exit(0);}+if (outcome.state === 'refused') {+ bail(1, `the reconstruct call was refused after ${elapsed}s: ${outcome.detail}\n evidence in ${outDir}`);+}if (outcome.state === 'error') bail(1, `the flow failed after ${elapsed}s: ${outcome.detail}\n evidence in ${outDir}`);bail(1, `no avatar after ${elapsed}s. Last status on screen: ${outcome.detail || '(blank)'}\n evidence in ${outDir}`);diff --git a/tests/api/reconstruct-finalize.test.js b/tests/api/reconstruct-finalize.test.jsindex a56df8299..e1f0ef623 100644--- a/tests/api/reconstruct-finalize.test.js+++ b/tests/api/reconstruct-finalize.test.js@@ -9,12 +9,26 @@ const sqlMock = vi.fn(async () => []);vi.mock('../../api/_lib/db.js', () => ({ sql: (...args) => sqlMock(...args), isDbUnavailableError: () => false, isDbCapacityError: () => false }));const putObjectMock = vi.fn(async () => undefined);-const publicUrlMock = vi.fn((key) => `https://cdn.test/${key}`);+// Mirrors the real publicUrl(): an absolute key is passed straight through, so a+// mesh served from the provider's own bucket resolves to itself.+const publicUrlMock = vi.fn((key) => (/^https?:\/\//i.test(key) ? key : `https://cdn.test/${key}`));vi.mock('../../api/_lib/r2.js', () => ({putObject: (...a) => putObjectMock(...a),publicUrl: (...a) => publicUrlMock(...a),+ // The real classifier, narrowed to the shapes these tests raise.+ isStorageInfrastructureError: (err) =>+ /signaturedoesnotmatch|unauthorized|nosuchbucket|access denied|econnrefused/i.test(+ [err?.name, err?.Code, err?.message].filter(Boolean).join(' '),+ ),}));+// The fault a rejected R2 token actually raises.+function signatureError() {+ return Object.assign(new Error('The request signature we calculated does not match the signature you provided.'), {+ name: 'SignatureDoesNotMatch',+ });+}+const createAvatarMock = vi.fn(async ({ input }) => ({ id: 'avatar-1', name: input.name, slug: input.slug }));vi.mock('../../api/_lib/avatars.js', () => ({storageKeyFor: ({ userId, slug }) => `u/${userId}/${slug}/m.glb`,@@ -229,3 +243,84 @@ describe('pollRiggingStage', () => {expect(createAvatarMock).not.toHaveBeenCalled();});});++// A rejected object-storage credential took the whole selfie flow down twice in+// three days (2026-09-09, then 2026-09-11 03:15 UTC): the mesh finished on the+// GPU and was discarded at the bucket write, so the user was told their avatar+// "could not be saved" while a perfectly good copy sat in the provider's own+// public bucket. Storage is allowed to lose the durable copy; it is not allowed+// to lose the reconstruction.+describe('object storage outage', () => {+ it('keeps the avatar, served from the provider copy, when the bucket rejects the write', async () => {+ inspectGlbMock.mockReturnValue(RIGGED);+ putObjectMock.mockRejectedValueOnce(signatureError());++ const out = await finalizeReconstructStage({+ userId: 'u1',+ jobId: 'j1',+ job: baseJob,+ glbUrl: 'https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/abc.glb',+ });++ expect(out).toEqual({ status: 'done', resultAvatarId: 'avatar-1' });+ const created = createAvatarMock.mock.calls[0][0];+ expect(created.storageKey).toBe('https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/abc.glb');+ // Flagged for a later re-copy rather than silently diverging from the bucket.+ expect(created.input.source_meta.servedFrom).toBe('provider');+ expect(created.input.source_meta.pendingBucketKey).toBe('u/u1/' + created.input.slug + '/m.glb');+ });++ it('registers the Forge creation against the URL actually serving the mesh', async () => {+ inspectGlbMock.mockReturnValue(RIGGED);+ putObjectMock.mockRejectedValueOnce(signatureError());++ await finalizeReconstructStage({+ userId: 'u1',+ jobId: 'j1',+ job: baseJob,+ glbUrl: 'https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/abc.glb',+ });++ const reg = registerReconstructionCreationMock.mock.calls[0][0];+ expect(reg.glbUrl).toBe('https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/abc.glb');+ expect(reg.glbKey).toBe(reg.glbUrl);+ });++ it('still rigs an unrigged mesh, handing the rig model the provider copy', async () => {+ inspectGlbMock.mockReturnValue(UNRIGGED);+ putObjectMock.mockRejectedValueOnce(signatureError());+ const submit = vi.fn(async () => ({ extJobId: 'rig-ext-1' }));+ providerMock.instance = { supportsMode: (m) => m === 'rerig', submit };++ const out = await finalizeReconstructStage({+ userId: 'u1',+ jobId: 'j1',+ job: baseJob,+ glbUrl: 'https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/abc.glb',+ });++ expect(out).toEqual({ status: 'rigging' });+ expect(submit.mock.calls[0][0].sourceUrl).toBe(+ 'https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/abc.glb',+ );+ });++ it('still throws on a fault that is not storage infrastructure', async () => {+ inspectGlbMock.mockReturnValue(RIGGED);+ putObjectMock.mockRejectedValueOnce(new TypeError('body is not a Buffer'));++ await expect(+ finalizeReconstructStage({ userId: 'u1', jobId: 'j1', job: baseJob, glbUrl: 'https://x/m.glb' }),+ ).rejects.toThrow('body is not a Buffer');+ expect(createAvatarMock).not.toHaveBeenCalled();+ });++ it('refuses to degrade when there is no provider URL to fall back to', async () => {+ inspectGlbMock.mockReturnValue(RIGGED);+ putObjectMock.mockRejectedValueOnce(signatureError());++ await expect(+ finalizeReconstructStage({ userId: 'u1', jobId: 'j1', job: baseJob, glbUrl: 'http://insecure/m.glb' }),+ ).rejects.toThrow(/signature/i);+ });+});diff --git a/tests/api/v1-ai-image.test.js b/tests/api/v1-ai-image.test.jsindex 4e4cd1c8a..e3954e00e 100644--- a/tests/api/v1-ai-image.test.js+++ b/tests/api/v1-ai-image.test.js@@ -261,3 +261,48 @@ describe('GET surfaces', () => {expect(body.lanes.replicate.status).toBe('ok');});});++// When object storage rejects the write, the lane still draws the image and+// image-persist.js hands back an inline data URI so our own 3D workers keep+// running. This endpoint's contract is a durable https URL a third party can+// fetch, so it cannot pass that through, but it also must not report it as a+// failed generation. That opaque 502 is what a rejected R2 credential looked+// like for hours on 2026-09-09 and again from 03:15 UTC on 2026-09-11.+describe('object storage outage', () => {+ const INLINE = 'data:image/jpeg;base64,/9j/4AAQSkZJRg==';++ it('answers 503 storage_unavailable when the image could only be stored inline', async () => {+ laneState.impl = async () => ({ imageUrl: INLINE, model: 'black-forest-labs/flux.1-schnell' });+ const res = makeRes();+ await handler(makeReq({ body: { prompt: 'a brass owl figurine' } }), res);+ expect(res.statusCode).toBe(503);+ const body = JSON.parse(res.body);+ // The code is the contract a caller reacts to: retry later, and do not+ // rewrite the prompt. (5xx descriptions on this route are redacted to a+ // support ref by design; the sentence reaches the operator via the log.)+ expect(body.error).toBe('storage_unavailable');+ expect(body.ref).toBeTruthy();+ });++ it('never leaks the inline bytes into the response', async () => {+ laneState.impl = async () => ({ imageUrl: INLINE, model: 'black-forest-labs/flux.1-schnell' });+ const res = makeRes();+ await handler(makeReq({ body: { prompt: 'a brass owl figurine' } }), res);+ expect(res.body).not.toContain('data:image/');+ });++ it('does not burn a free-quota slot on a storage failure', async () => {+ laneState.impl = async () => ({ imageUrl: INLINE, model: 'black-forest-labs/flux.1-schnell' });+ const res = makeRes();+ await handler(makeReq({ body: { prompt: 'a brass owl figurine' } }), res);+ expect(quotaState.consumed).toBe(0);+ });++ it('still reports a genuinely empty lane result as generation_failed', async () => {+ laneState.impl = async () => ({ imageUrl: null, model: 'black-forest-labs/flux.1-schnell' });+ const res = makeRes();+ await handler(makeReq({ body: { prompt: 'a brass owl figurine' } }), res);+ expect(res.statusCode).toBe(502);+ expect(JSON.parse(res.body).error).toBe('generation_failed');+ });+});