Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Avatar thumbnails
Every avatar on three.ws should have a small, fast-loading image the galleries can show instead of downloading a multi-megabyte GLB. This page explains where those images come from, the one rule every code path must obey, and how to operate the backfill.
The rule
A
thumbnail_keyis only ever persisted after the object behind it has been confirmed to exist in R2.
This is not style advice. avatars.thumbnail_key is turned into a public URL by
publicUrl() and handed to the browser inside an <img>. If the key points at an
object that was never written, R2 answers 404 with a text/plain body. Chrome's
Opaque Response Blocking then refuses the response, because a text/plain body is
not a valid image, and logs:
GET https://<bucket>.r2.dev/thumb/<avatarId>.png net::ERR_BLOCKED_BY_ORB
The homepage shipped exactly that bug: it guessed thumb/<avatarId>.png whenever
the API honestly reported "no thumbnail", producing five blocked requests on every
visit. The APIs were already correct, /api/explore returns image: null and
/api/marketplace returns thumbnail_url: null, so the fix was to stop guessing
and render the designed initial-letter placeholder instead.
Never synthesise a thumbnail URL on the client. If the API gives you null,
that avatar has no thumbnail; show a placeholder.
The read side: always thumbnailUrl(), never bare publicUrl()
The write paths guarantee that a key we store has an object behind it. The read
paths have to survive the keys that predate that guarantee. publicUrl() passes an
absolute value straight through, so a legacy *_og.png key becomes a URL pointing
at an origin that holds no object, a 404 the browser blocks as ORB.
So every read path goes through one helper in
api/_lib/r2.js:
import { thumbnailUrl } from './_lib/r2.js';
thumbnailUrl(null) // → null
thumbnailUrl('https://three.ws/avatars/x_og.png') // → null (legacy, always 404s)
thumbnailUrl('thumb/abc.png') // → https://three.ws/cdn/thumb/abc.png
The URL is a same-origin /cdn/<key> address (cdnUrlOrNull() in the same file),
not the raw public r2.dev domain. The first-party proxy in api/cdn-object.js
streams the object and fails over by itself when the public domain is blocked or
storage is unhealthy, so every API consumer gets that for free. Upload-side code
that needs the raw bucket address still uses publicUrl().
The helper is built on the OrNull variants rather than bare cdnUrl() or
publicUrl() because building a URL can throw on a deployment without object
storage configured, and a single un-renderable key must not take down a whole list
response.
It did: /api/pulse answered 502 pulse_failed and /api/search answered
503 not_configured for every caller until both were pointed at the read-path
helper. Upload paths keep calling publicUrl(), where the throw is the correct
signal that there is nowhere to put the bytes.
Return null and let the surface render its designed placeholder. There were 43
bare publicUrl(<thumbnail key>) call sites across 32 files, agent cards, the
marketplace, explore, user profiles, leaderboards, and the image baked into on-chain
token metadata. All now route through thumbnailUrl(), and
tests/thumbnail-url-guard.test.js walks
api/ to fail the build if a new one appears. Three files are allow-listed there,
each for a stated reason (the helper itself, a key published right after its
putObject in the same function, and a HEAD-checked vision fetch).
Where a thumbnail comes from
avatars.thumbnail_key holds a relative R2 key. An absolute URL in that column
is treated as missing everywhere (api/explore.js drops it; the backfill replaces
it), because publicUrl() passes absolute values through untouched and they resolve
against an origin where no object lives.
There are three writers, cheapest first.
1. Forge preview adoption (free)
An avatar forged from a forge_creations row can point straight at that creation's
already-uploaded preview image (forge_creations.preview_key). The object exists,
its Content-Type is already correct, and /forge's own gallery has always
rendered it. Adoption costs zero bytes and zero render time.
This happens automatically at insert time in
api/cron/forge-seed-cron.js, and retroactively
via adoptForgePreviews() in the backfill.
2. Client capture (POST /api/avatars/thumbnail)
The browser renders a PNG and uploads it. Stored at thumb/<avatarId>.png with
Content-Type: image/png. Owner or admin only. The account save flow
(src/account.js) and the /app agent editor (src/app.js) render that poster
offscreen with glbFileToThumbnail() in
src/erc8004/thumbnail.js, which poses the
avatar exactly like the server does (see The rest pose).
3. Server render (CPU rasterizer, chromium failover)
Everything else (studio avatars, uploads, forge rows older than preview capture)
is rendered server-side through renderGlbToPng() in
api/_lib/render-glb.js: the GLB is presigned and
rendered to a 768x768 PNG, which is uploaded to thumb/<avatarId>.png. The
renderer tries the in-process software rasterizer first
(api/_lib/render-cpu.js, the @three-ws/render
package, roughly 200-900 ms and no subprocess) and falls back to headless chromium
running a three.js viewer only for the models the CPU lane cannot decode on its
own (Draco-compressed geometry, KTX2/Basis textures). A CPU miss is logged as
[render] cpu lane fell back to chromium and costs latency, never a failed
thumbnail. Set RENDER_CPU_LANE=off on the service to pin every render back onto
chromium without a deploy. A chromium render costs 3-15 s per model, so rendering
always runs in bounded batches.
All of this lives in api/_lib/avatar-thumbs.js,
the single owner of the invariant above.
The rest pose
A thumbnail never shows an avatar in its raw bind pose (a T-pose or A-pose with
the arms straight out). Every renderer that turns a GLB into a picture runs
src/thumbnail-pose.js before framing the model:
the CPU rasterizer (api/_lib/render-cpu.js), the chromium failover
(api/_lib/render-glb.js, which ships the module into its render page as
data: URL modules built by thumbnailPoseModules() in
api/_lib/pose-runtime.js), and the browser's
offscreen capture. That covers the OG card (/api/avatar/:id/og), the backfill
and x402 regeneration crons, forge previews, and /api/render/glb.
How it poses, in order:
- Not a humanoid (no skin, or fewer canonical bones than
AnimationManager.supportsCanonicalClips()accepts): left as authored. Props, vehicles and creatures render exactly as before. - Arms already hanging in the bind pose (
armsAlreadyHanging()): left as authored. Reposing a stylized rig that already stands naturally only risks folding its arms. - Otherwise the avatar takes one held frame of the motion library's
standing idle (
av-idle-male, sampled once bysampleClipPose()) throughAnimationManager, the same retargeting path the live viewer animates with: canonical bone names fromsrc/glb-canonicalize.js, rest-frame correction fromsrc/animation-retarget.js, and the fallen-pose guard. Only joint rotations are applied; the avatar keeps its own hip height. - The clip cannot drive the rig (too few bones map, for example a rig with
no finger bones), or it drove the arms somewhere wrong: the rig goes back
to its bind pose and the arms are swung down geometrically by
relaxUndrivenArms(). - Every posed result is checked (
armsPointBelow()): both upper arms must end at least 30 degrees below horizontal. A retarget that leaves the arms out, or swings them up over the head (seen on a real production rig), fails it and falls to step 4. If the arm swing fails it too, the rig renders as authored and reportspose-rejected, so the regeneration script skips it rather than replacing a T-pose with something worse.
Renderers report the outcome through an onPose({ posed, mode }) callback, and
/api/render/glb through its x-render-pose header. Pass pose: 'bind' to any
of them to render the skeleton exactly as authored.
A skeleton convention the canonicalizer does not recognise renders as authored.
Add its bone names to src/glb-canonicalize.js (with a case in
tests/glb-canonicalize.test.js) and every renderer picks it up; never add a
bone table to a renderer.
Regenerating thumbnails after a pose change
Posters rendered before the rest pose existed stay in the bucket until something
re-renders them.
scripts/regenerate-avatar-thumbnails.mjs
does that for the keys the platform writes itself (thumb/<uuid>.png,
og/avatar/<uuid>.png, <storage key>_og.png, forge/thumb/<uuid>.png); a
creator's uploaded cover or an adopted forge reference image is never touched.
Work is planned per distinct key, most-visible first, so an agent body cloned
from a gallery avatar is fixed by the same single render as its source.
It is a dry run unless --apply is passed. --apply overwrites each object in
place under the same key, stamped x-amz-meta-thumbnail-pose: <THUMBNAIL_POSE_VERSION>, and changes no database row. A model that comes back
unposed (a prop, or a rig already standing at rest) is skipped, and an object
already carrying the current stamp is skipped, so a run can be stopped and
resumed.
eval "$(node scripts/read-service-env.mjs '^S3_')"
# What would be visited, in order. Renders nothing.
node --env-file=.env.local scripts/regenerate-avatar-thumbnails.mjs --plan-only --limit=50
# Render 25 locally and save <avatar>-before.png / <avatar>-after.png to review.
node --env-file=.env.local scripts/regenerate-avatar-thumbnails.mjs --limit=25 --out=/tmp/thumbs
# Overwrite the stale posters, 500 keys at a time, resuming with --offset.
node --env-file=.env.local scripts/regenerate-avatar-thumbnails.mjs --apply --limit=500 --concurrency=3
node --env-file=.env.local scripts/regenerate-avatar-thumbnails.mjs --apply --limit=500 --offset=500 --concurrency=3
Every surface derives the image URL from thumbnail_key at read time, so no
URL changes. The public bucket domain sends no Cache-Control header for these
objects, so a fresh request sees the new bytes as soon as the overwrite lands,
and a browser holding a heuristically cached copy revalidates against the ETag,
which the overwrite changes. When the pose itself changes, bump
THUMBNAIL_POSE_VERSION in src/thumbnail-pose.js and run the script again. Each re-render gets a two-minute budget; a model that never finishes is counted as failed and the pass moves on, so one stuck GLB cannot stall the run.
The three crons
| Cron | Schedule | Job |
|---|---|---|
avatar-thumbnail-render |
*/10 * * * * |
Re-renders stale thumbnails for marketplace listings, driven by the x402 spend loop. |
avatar-thumbnail-backfill |
*/5 * * * * |
Fills in absent thumbnails: adopts forge previews, then renders whatever is left. |
agent-avatar-backfill |
*/10 * * * * |
Assigns a 3D body to any agent with no avatar (agent_identities.avatar_id IS NULL or dangling): clones a random public, thumbnailed humanoid from the gallery into the agent owner's account (api/_lib/agent-avatars.js, reusing circulation's cloneAvatarFor). Pure DB work: the clone shares the source's storage_key and thumbnail_key, so the agent card has a preview immediately. Batch via AGENT_AVATAR_BACKFILL_BATCH (default 100). |
Together the last two make "every agent card shows a real preview" an invariant:
one guarantees the agent has an avatar, the other guarantees the avatar has a
thumbnail. Creation paths keep the gap from reopening: createAvatar accepts an
internal thumbnail_key seed, auto-rig siblings inherit their source's thumbnail,
and the forge/avatar seed crons adopt the forge preview at insert.
They share the thumb/<avatarId>.png key space and each is a no-op on the other's
rows. The backfill drains most-visible-first, featured, then public, then
view_count, then newest, so the surfaces users actually look at heal first.
Each cron handler is wrapped in wrapCron() with a 120 s budget (its maxDuration). A tick that overruns answers 503 { reason: "cron_timeout" } and records a failed heartbeat instead of hanging until Cloud Run's 900 s kill, and the chromium render in render-glb.js bounds its own browser launch (30 s), page load (20 s) and screenshot (15 s).
Tuning (env, on the Cloud Run service):
| Var | Default | Meaning |
|---|---|---|
THUMBNAIL_BACKFILL_RENDER_BATCH |
8 |
Models rendered per tick. |
THUMBNAIL_BACKFILL_ADOPT_BATCH |
200 |
Forge previews adopted per tick. |
THUMBNAIL_BACKFILL_CONCURRENCY |
2 |
Parallel renders (one shared chromium). |
Bounded retries
A GLB that cannot be rendered, corrupt bytes, over the 25 MB cap, or a
storage_key whose object has been deleted, would otherwise sit at the head of the
priority order and burn every tick forever. The avatar_thumbnail_backfill table is
a claim + retry ledger:
- One row per attempted avatar.
claimed_atis a 15-minute lease, so a run that dies mid-render releases its claim. - Rows are deleted on success (the avatar now has a
thumbnail_key, so it drops out of the candidate set on its own). - On failure
attemptsis bumped andlast_errorrecorded. After 3 failures the avatar is retired and never claimed again.
Claim selection and the claim write are a single SQL statement using
FOR UPDATE … SKIP LOCKED, so the cron and an operator's bulk run can execute at
the same time without ever claiming the same avatar.
Blame the browser, not the model
Retiring an avatar after 3 failures is only safe if those failures are the model's
fault. Chromium is the first process the OOM killer reaps on a memory-tight
container, and it dies exactly when a long batch render is under way. Once it does,
every remaining render fails in milliseconds with Connection closed.
So the runner distinguishes the two:
- Model failure (
glb fetch failed: …,render failed: …): charge the attempt, recordlast_error, keep going. - Infrastructure failure (
Connection closed.,Target closed,Protocol error, aspawn EFAULT/ENOMEM/EAGAINlaunch refusal, or object storage answeringMissing required env var: S3_…,InvalidAccessKeyId,NoSuchBucket,ECONNREFUSED, …): the model is blameless. Roll the attempt back, roll back every claim the aborted batch never reached, and stop the batch.renderBatch()returnsaborted: "<reason>"; the cron logsbackfill_browser_diedand the next tick retries the same avatars on a fresh container.
Two classifiers decide which side a failure lands on. isBrowserInfrastructureError()
in api/_lib/render-glb.js covers the browser (its
INFRA_ERROR_PATTERN), and isStorageInfrastructureError() in
api/_lib/r2.js covers object storage (STORAGE_ERROR_PATTERN).
Both are exported as pattern strings so the ledger repair below asks the same
question in SQL with ~* instead of maintaining a drifting copy, and
tests/avatar-thumbs.test.js imports the real
classifiers rather than a stand-in for the same reason. The cached browser evicts
itself on disconnected so the next render relaunches instead of reusing a corpse.
Storage is also checked before any claim is made: both thumbnail crons return
{ ok: false, reason: "object_storage_unconfigured" } and touch nothing when
objectStorageConfigured() in r2.js reports the S3_* set incomplete, because
three ticks of identical S3_BUCKET failures at */5 would otherwise retire every
remaining avatar in under fifteen minutes.
This is not hypothetical: before the classifier existed, one OOM-killed chromium retired 1,283 perfectly renderable avatars in a single run. If you ever suspect that happened again:
node --env-file=.env.local scripts/backfill-avatar-thumbnails.mjs --reset-infra
It deletes every ledger row whose last_error matches either infrastructure
pattern (browser or storage), returning those avatars to the candidate set. Rows
recording a model-attributable error are left retired. Safe to run at any time.
Add --repair-only to run the repair (and any --restyle) and stop before anything
touches R2: that mode needs only DATABASE_URL, so the undo stays reachable while
object storage is exactly the thing that is broken.
Running more than one bulk backfill at once is what causes the OOM in the first
place. The claim ledger makes it correct, but not free, one runner at
--concurrency=2..3 beats three runners fighting for RAM.
Operating the backfill
scripts/backfill-avatar-thumbnails.mjs
is the bulk counterpart. It talks to Postgres and R2 directly, no admin token, no
running server, and shares the same claim ledger, so it is safe to run while the
cron is live.
# How much coverage do we have?
node --env-file=.env.local scripts/backfill-avatar-thumbnails.mjs --status
# Free phase only: adopt every forge preview, never boot chromium.
node --env-file=.env.local scripts/backfill-avatar-thumbnails.mjs --adopt-only
# Render 50 avatars, 3 at a time.
node --env-file=.env.local scripts/backfill-avatar-thumbnails.mjs --limit=50 --concurrency=3
# Clear a large backlog: keep refilling the budget until nothing is left to claim.
node --env-file=.env.local scripts/backfill-avatar-thumbnails.mjs --limit=2000 --concurrency=4 --loop
# After deploying a renderer change: re-queue up to N posters baked by the older
# renderer (clears only server-rendered thumb/<uuid>.png keys via queueRestyle();
# client snapshots and adopted forge previews are never touched).
node --env-file=.env.local scripts/backfill-avatar-thumbnails.mjs --restyle=200 --limit=200
Requires DATABASE_URL plus the S3_* credentials (S3_BUCKET,
S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_ENDPOINT, S3_PUBLIC_DOMAIN);
--status and --repair-only need DATABASE_URL alone. Only DATABASE_URL is in
.env.local; the authoritative copy of the S3_* set lives on the Cloud Run
service, so --env-file=.env.local alone exits with S3_BUCKET is unset for any
mode that renders or adopts. Export them into the shell first:
eval "$(node scripts/read-service-env.mjs '^S3_')"
Then any of the commands above runs as written.
Rendering a thumbnail yourself
import { renderThumbnail, coverage } from './api/_lib/avatar-thumbs.js';
const { url, bytes, ms } = await renderThumbnail({
id: 'a4bad2f5-8a07-43cf-82e5-b6ba1314441e',
storage_key: 'u/<ownerId>/model.glb',
});
console.log(url, bytes, ms); // https://<cdn>/thumb/<id>.png 104356 5182
console.log(await coverage());
// { total: 26398, covered: 26361, missing: 37, exhausted: 33 }
renderThumbnail() uploads the PNG before it writes the key, so a failed upload
can never leave a thumbnail_key pointing at nothing. That ordering is pinned by
tests/avatar-thumbs.test.js, alongside the rule
that adoption HEAD-checks the preview object before persisting it.
Related
tests/home-thumbnail-orb.test.js: guards the homepage against ever re-introducing a fabricated thumbnail URL.api/_lib/r2.js:publicUrl(),cdnUrl(),headObject(), andisLegacyOgThumbnailKey().- STRUCTURE.md: where every product surface lives.