Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
The Forge pipeline: architecture deep dive
This is the engineering guide to the Forge, the text/image/sketch to 3D generation system behind /forge, the free MCP tools, and every agent-facing 3D endpoint. It maps the whole path a generation takes: entry points, the engine grid, free-first routing, the job lifecycle, failover, persistence, GPU workers, post-processing, and payments.
If you want the product-level overview, read Forge. If you want the plain-language version, read How the Forge works. This document is for people changing the pipeline.
Bird's-eye view
/forge page MCP tools x402 buyers SDK / agents
home mini-forge (paid stdio + /api/x402/forge /api/3d/generate
/forge-studio free hosted) /api/v1/ai/text-to-3d
│ │ │ │
└────────────────────┴──────────┬─────────┴──────────────────┘
▼
POST /api/forge (api/forge.js, the orchestrator)
│
resolve (path, tier, backend) via api/_lib/forge-tiers.js
health-aware free-first routing + circuit breakers
│
┌─────────────┬───────────┼───────────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
NVIDIA NIM HF Spaces self-host GPU BYOK vendors Replicate
(TRELLIS, (Hunyuan3D workers on (Meshy, Tripo, (paid last
text only) chain) Cloud Run Rodin, ...) resort)
│ │ │ │ │
└─────────────┴───────────┼───────────────┴──────────────┘
▼
job handle (HMAC token) ── poll GET /api/forge?job=<id>
poll-time failover (forge-failover.js) if a lane dies mid-job
│
▼
forge_creations row (Neon Postgres) + GLB mirrored to R2 CDN
│
▼
viewer / AR / rig / refine / restyle / remesh / gallery / remix
Three properties define the design:
- One orchestrator, many front doors. Every entry point is a thin client over
POST /api/forgeandGET /api/forge?job=<id>. No generation logic is duplicated. - Free-first routing over a declared engine grid. Engines are data, not code paths: each is a registry entry declaring what it can do, what it costs, and what env it needs. Routing walks that registry.
- Failure is a routing event, not an error. A rejected submit walks to the next lane at submit time; a lane that accepts and then dies is silently re-dispatched at poll time. The client sees the same job id throughout.
1. Entry points
| Surface | Route / tool | Notes |
|---|---|---|
| Forge page | /forge, src/forge.js |
Text, photos (up to 6 views), sketch. One shared polling loop. /forge-max deep-links straight into the maximum-quality lane. |
| Homepage mini-forge | src/home-forge.js |
Health-aware default lane, never pins a backend. |
| Forge Studio | /forge-studio, src/forge-studio/ |
Create, refine, stylize, Game-Ready export, remix. |
| Core API | POST /api/forge (api/forge.js) |
The orchestrator. Auth-free. Text, image, and sketch bodies. |
| Job poll | GET /api/forge?job=<id> |
Free to poll regardless of how the job was paid for. |
| Cost preview | GET /api/forge-quote (api/forge-quote.js) |
Public, spend-free price for a (path, tier, backend, options) combination, or ?table=1 for all three tiers. Reads the same tables the generation endpoint enforces, so a quote cannot drift from the charge. |
| Pay-to-keep | GET/POST /api/forge-keep (api/forge-keep.js) |
Off unless FORGE_PAY_TO_KEEP=on (flip with npm run forge:pay-to-keep -- on|off|status). When on, generating and previewing on a free lane stays free and the original file is released once per creation to a $THREE holder or against one settled $THREE payment; with it off every generation is keepable. |
| Image to splat | POST/GET /api/forge-splat (api/forge-splat.js) |
TripoSplat: one public https image to a 3D Gaussian splat (num_gaussians 8192 to 262144), opened in the Splat Viewer. 503 unconfigured until MODEL_TRIPOSPLAT_URL is set. |
| Photo to scene | POST/GET /api/forge-scene-objects (api/forge-scene-objects.js) |
SAM 3D Objects: one photo to a scene GLB with one named node per object, laid out like the photo. 503 backend_unconfigured until MODEL_SAM3D_URL points at a deployed worker (32 GB+ GPU, gated weights). |
| Tripo webhook | POST /api/forge/webhook/tripo (api/forge/webhook/tripo.js) |
Optional fast-completion path for Tripo jobs: HMAC-signed (TRIPO_WEBHOOK_SECRET, 5 minute replay window) and idempotent per delivery (forge_webhook_deliveries). Polling stays the fallback. |
| Catalog | GET /api/forge?catalog=1 |
The tier/backend/cost matrix the UI renders. |
| Health | GET /api/forge?health=1 |
Live per-lane upstream status. |
| Rig | POST /api/forge?action=rig |
Auto-rig a finished GLB (see post-processing). |
| Paid x402 twin | POST /api/x402/forge (api/x402/forge.js) |
Pay-per-call USDC on Solana, no account. Prices come from the tier table. |
| Agent REST | POST /api/3d/generate (api/3d/generate.js) |
Keyless agent shape wrapping the free draft lane. |
| Flagship free API | POST /api/v1/ai/text-to-3d (api/v1/ai/text-to-3d.js) |
NVIDIA NIM TRELLIS draft lane, 10 generations per IP per day, 429 + x402 upsell above quota. |
| Refinement | POST /api/forge-iterate (api/forge-iterate.js) |
Conversational refinement with version lineage for the signed-in Studio. |
| MCP (paid stdio) | mesh_forge, forge_avatar, forge_free, text_to_avatar, rig_mesh, refine_model, restyle_material (mcp-server/src/tools/) |
Shared cores in mcp-server/src/tools/_studio-core.js; thin clients over /api/forge. |
| ChatGPT clone | POST /api/gpt-forge (api/gpt-forge.js) |
Clone of the orchestrator dedicated to the ChatGPT surfaces (the Apps SDK MCP connector api/mcp-studio and the custom-GPT Actions endpoint api/3d/studio, via api/_mcp-studio/gpt-forge-client.js), so the ChatGPT pipeline can evolve without touching /api/forge. Same lanes, tiers, job tokens, and forge_creations rows. First divergence: its poll responses also carry prompt, preview_image_url (the painted concept view), and text_to_image_model, so the ChatGPT surfaces can show the concept image while the mesh generates. It also remembers a finished job's poll frame (api/_lib/forge-done-cache.js), so later polls of a done job answer in milliseconds without re-reading the provider. |
| MCP (hosted HTTP) | text_to_3d, image_to_3d, auto_rig_model, plus the wider studio toolset (generation_status, preview_3d, remove_background, remesh_model, stylize_model, segment_model, retexture_model, pose_model, save_avatar, persona tools; api/mcp-3d.js, tools in api/_mcp3d/tools/studio.js) |
Streamable HTTP server registered as io.github.nirholas/three-ws-3d-studio. |
| DCC plugins | integrations/blender/, integrations/comfyui/, integrations/_pyclient/ |
First-party clients driving the same API. |
The rule when adding a surface: never talk to a provider directly. Submit through /api/forge (or the shared MCP cores) so routing, failover, persistence, and payments stay in one place.
2. The request axes: path and tier
Every request resolves to a (path, tier, backend) triple. api/_lib/forge-tiers.js is the single source of truth for all three; the endpoint, the catalog, the x402 pricing, and every provider param builder read from it.
path is how geometry is produced:
image: image-intermediate, the default. Text is painted into a reference view (FLUX/Imagen) and reconstructed to a mesh, or the user's own photos are reconstructed directly.geometry: geometry-first. A native 3D model (Meshy, Tripo, Rodin) emits mesh directly from the prompt, so detail is not capped by what one synthesized image implies.sketch: sketch-conditioned. A drawing plus a prompt naming what it depicts drives TripoSG-scribble straight to geometry.
tier is how much budget to spend (TIERS in forge-tiers.js):
| Tier | Poly target | Textures | ETA multiplier | Price (USDC atomics) |
|---|---|---|---|---|
draft |
12,000 | none | 0.6 | 50,000 ($0.05) |
standard |
30,000 | 2K | 1.0 | 150,000 ($0.15) |
high |
200,000 | PBR + HD | 2.2 | 500,000 ($0.50) |
Prices live only here, as priceUsdcAtomics. priceAtomicsForTier() feeds the x402 402 challenge; priceUsdcForTier() feeds the catalog and docs. Never hardcode a second copy.
Self-hosted TRELLIS additionally maps tiers to sampler and texture budgets via SELFHOST_TRELLIS_QUALITY (draft: 12/12 steps, 1024px textures; standard: 35/35, 2048px; high: 50/50, 4096px). Texture sizes must be powers of two: nvdiffrast hard-fails on mip generation otherwise (live incident, 2026-07-16).
3. The engine grid
BACKENDS in api/_lib/forge-tiers.js declares every lane. Each entry names the provider client (api/_providers/*.js), which paths it serves, whether it is free, whether it needs a BYOK key, the env vars that must be present (requiresEnv), and its latency estimate (baseEta, plus coldStart for scale-to-zero workers).
| id | Engine | Hosting | Paths | Cost model | Env |
|---|---|---|---|---|---|
nvidia |
Microsoft TRELLIS | NVIDIA NIM (hosted) | image (text only, rejects user photos) | free | NVIDIA_API_KEY |
huggingface |
Hunyuan3D 2.1 → Hunyuan3D 2 → TRELLIS → TripoSR (internal chain) | HF Spaces | image | free, blocking submit | HF_TOKEN |
trellis2 |
Microsoft TRELLIS.2 (MIT, PBR with alpha) | Our Cloud Run L4 worker (workers/model-trellis2/) |
image (native single-hop image to 3D) | free (our GPU) | MODEL_TRELLIS2_URL + GCP_RECONSTRUCTION_KEY |
trellis_selfhost |
Microsoft TRELLIS | Our Cloud Run L4 worker (workers/model-trellis/) |
image (native single-hop image to 3D) | free (our GPU) | MODEL_TRELLIS_URL + GCP_RECONSTRUCTION_KEY |
hunyuan3d |
Tencent Hunyuan3D-2.1 | Our Cloud Run L4 worker (workers/model-hunyuan3d/) |
image | free (our GPU) | GCP_HUNYUAN3D_URL + key |
modly |
VAST AI TripoSG via Modly (MIT, untextured geometry) | Our Cloud Run L4 worker (workers/modly/) |
image (single photo) | free (our GPU) | MODEL_MODLY_URL + GCP_RECONSTRUCTION_KEY |
modly_hunyuan |
Tencent Hunyuan3D 2 Mini Turbo via Modly (untextured geometry) | Same worker as modly |
image (single photo) | free (our GPU) | MODEL_MODLY_URL + GCP_RECONSTRUCTION_KEY |
pixal3d |
TencentARC Pixal3D ("Match my photo", MIT) | Our Cloud Run worker (workers/model-pixal3d/) |
image (1 to 4 labeled photos fused pixel-aligned; opt-in only) | free (our GPU) | MODEL_PIXAL3D_URL + GCP_RECONSTRUCTION_KEY |
anigen |
VAST AI AniGen (MIT) | Our Cloud Run worker (workers/model-anigen/) |
image (mesh, skeleton and skin weights in one pass, already rigged) | free (our GPU) | MODEL_ANIGEN_URL + GCP_RECONSTRUCTION_KEY |
partcrafter |
PartCrafter (MIT) | Our Cloud Run worker (workers/model-partcrafter/) |
image (2 to 16 named part meshes) | free (our GPU) | MODEL_PARTCRAFTER_URL + GCP_RECONSTRUCTION_KEY |
triposg |
VAST AI TripoSG-scribble | Our Cloud Run L4 worker (workers/model-triposg/) |
sketch only | free (our GPU) | GCP_TRIPOSG_URL + key |
trellis |
Microsoft TRELLIS | Replicate | image | paid platform, last resort | REPLICATE_API_TOKEN |
meshy |
Meshy 6 | vendor API | geometry, image | BYOK, credits | user key |
tripo |
Tripo v3.1 | vendor API | geometry, image | BYOK, or house lane when TRIPO_API_KEY is set |
user key, or TRIPO_API_KEY |
rodin |
Rodin (Hyper3D) | vendor API | geometry, image | BYOK | user key |
stability |
Stable Fast 3D | vendor API | image, synchronous | BYOK | user key |
replicate_byok |
Replicate models | caller's Replicate account | image | BYOK | user key |
The two Modly lanes run Modly's backend headless on one worker (workers/modly/README.md): TripoSG and Hunyuan3D 2 Mini Turbo through Modly's own extensions, then a mesh repair pass and a decimate to the tier's face budget (target_polycount). They return geometry without textures, so they rank below every textured free lane and are meant as an explicit pick, or for retexture and stylize afterwards.
pixal3d, anigen and partcrafter are opt-in lanes: none is a named default or a failover target, so a plain request never lands on one by chance. TripoSplat (/api/forge-splat) and SAM 3D Objects (/api/forge-scene-objects) are not BACKENDS entries at all: they are standalone endpoints over the same gcp provider (modes triposplat and sam3d), because their outputs (a splat, a multi-object scene) are not a single mesh.
Two lanes deserve a warning. hunyuan3d must point at the general Hunyuan3D worker URL, never at the avatar-pipeline controller: the controller fronts a face pipeline and rejects general object images. And nvidia is text-only; every routing list that might carry a user photo filters it out.
4. Routing: free-first, subject-aware, health-tempered
Routing answers one question: given (path, tier, userImages, prompt), which configured lane runs first, and who is next when it fails.
Static preference order. freeLaneCandidates() builds the ordered list both resolvers walk:
- The tier's named free default (
FREE_DEFAULT_FOR_TIERS): every image tier (draft, standard, high) namestrellis2, the only default lane that serves every region. Hunyuan3D is no longer a named default; it is an explicit lane and a fallback outside the restricted territories (see Territory enforcement below). The unconfigured case matters: untilMODEL_TRELLIS2_URLis set the lane is skipped and the chain behind it serves the request. The workers are our own GPUs; NVIDIA NIM is no longer a named default (its text-only, no-reference-image conditioning is the realism hole this table closes) but stays explicitly selectable and remains the final fallthrough in the per-path chain. - The per-path free fallback chain (
FREE_FALLBACK_FOR_PATH). Forimagethat istrellis2,trellis_selfhost,hunyuan3d,huggingface,modly,modly_hunyuan,nvidia: our own textured GPU workers first (zero vendor cost), then external free lanes, then the untextured Modly lanes, with the text-only NIM lane trailing as a safety net. - A configured house-key lane (
houseDefaultForPath()): on the High tier only, whenTRIPO_API_KEYis set, Tripo becomes the default geometry engine and bills through the same $THREE hold-or-pay gate as every platform lane. A caller's own inline or saved Tripo key still takes precedence and runs exempt. Draft and standard are untouched, and with the env var unset behavior is unchanged. The catalog reportshouse_key_availableso the picker can offer it without a key prompt. - The paid platform default (
DEFAULT_BACKEND_FOR_PATH: image →trellison Replicate, geometry →meshy, sketch →triposg) only when nothing free is configured and healthy.
Generation controls (api/_lib/forge-options.js). target_polycount (alias target_faces) is honored by every lane that has polygon control, including the BYOK providers (Meshy, Tripo, Rodin), which used to hardcode the tier default. bbox ([x, y, z] meters) is lane-agnostic: after generation api/_lib/glb-bbox-fit.js uniformly scales the delivered mesh to fit the box, centered on X/Z and floored at Y=0, on every sync and async lane. pose (t or a) asks for a rest pose where the lane can honor it (poseControlFor(): text-conditioned requests only, since a caller photo fixes the pose, and never anigen, whose rigging pass re-normalizes the pose), and the response reports pose_applied.
Subject-aware reorder (high tier). classifyForgeSubject() regex-classifies the prompt as hard-surface or organic. Hard-surface prompts (vehicles, machines, architecture) hoist trellis_selfhost above hunyuan3d; organic subjects (people, creatures) keep Hunyuan3D first. This exists because the two engines genuinely differ on those subject classes at high fidelity.
Operator levers. FORGE_PREFER_FREE (default on) prefers free reconstruction before paid Replicate. FORGE_SELFHOST_PRIMARY (default off) hoists self-host GPU lanes above hosted free lanes across the board.
Health tempering. resolveBackendIdWithHealth() walks the candidate list against the snapshot from api/_lib/forge-lane-health.js: lanes reporting ok or unknown are preferred, lanes confirmed down are skipped, and paid lanes are considered only when every free lane is down. The UI consumes the same data via GET /api/forge?health=1 and disables down lanes with the real upstream reason before the user clicks Generate.
Circuit breakers. A failing lane is sidelined, not retried in a loop: per-lane cooldowns (forge-lane:<id>, 90s) plus dedicated NIM cooldowns (120s model, 30s gateway) short-circuit routing away from a degraded upstream. Cooldowns are shared cross-instance via api/_lib/provider-health.js and expire on their own, so a recovered lane is retried promptly. Every entry is env-gated, so partial deployments degrade to a shorter list instead of erroring.
The HuggingFace Spaces pause. The free HF lane fails as a whole (every Space in its chain GPU-quota-exhausted or errored), and each attempt still costs the caller 3-15 s of Space round trips. So one chain-wide failure records forge-lane:huggingface for HF_LANE_COOLDOWN_SECONDS (300 s, api/_lib/forge-lane-health.js). While it holds, runHfImageLane reports "not served" at once, the health-aware router reads the lane as down, and GET /api/forge?health=1 reports huggingface as down (whoami alone only proves the token), so the engine picker disables it. A caller who explicitly picked the HF engine gets 503 provider_busy with Retry-After: 30 and retry_backends naming the configured image-capable engines; the pick is never swapped silently, and /forge hops to the first suggestion visibly through the same lane-failed path a mid-job failure uses. The first request after the window re-tests the Spaces for real.
5. Job lifecycle
Submit
POST /api/forge validates input, resolves the triple, runs the payment gate if the tier requires one (section 9), inserts a forge_creations row (status: generating), and submits to the resolved lane. Synchronous lanes (NVIDIA NIM, HF Spaces, BYOK-sync like Stability) complete inside the request and return status: done with the GLB URL directly. Async lanes return status: queued plus a job_id.
Submit-time failover. If a lane rejects the submit outright (down, quota, cooldown), startJob walks the free-first chain to the next candidate in the same request. The client never sees the rejected lane.
Backdrop pre-matting. A synthesized reference view is on a plain background, which is not the same as a transparent one, and the reconstructor fuses that backdrop in as geometry: six text personas through the live router returned four figures standing on a full-footprint slab and two that were a bare plane with no figure at all. Non-draft text jobs therefore get the same rembg cutout that image jobs already got (matte: tier.id !== 'draft'), while draft stays single-view and fast. It is best-effort worker-side: a rembg miss falls back to the original image rather than failing the generation.
Floor-slab strip. Pre-matting is not enough on its own: TRELLIS and Hunyuan3D read "an object resting on the ground" into even a clean alpha cutout and still fuse a thin, wide plate under the subject (measured on 2026-09-30: a teapot on a black marble square, a telescope on a grey sheet 65 times its footprint, both from spotless cutouts). materializeCreation therefore runs api/_lib/glb-floor-slab.js right after geometry cleanup and before scoring, on geometry alone, so it never depends on a vision provider answering. A bottom band counts as a fused slab only when it is a solid sheet seen from above (it fills at least 90% of its bounding rectangle; feet, legs and soles fill 3-42%, a round lamp foot about 79%) and is wider than the subject standing on it. A flat subject, a thick plinth, a round foot and a tripod are all left alone. The cut keeps the plate only directly under where the subject touches down, so the subject keeps its own underside. The creation's cleanup stats record floor_slab_removed_tris when it fires.
Reference-image budget. On the text path the reference-image step (the Vertex lane in api/_lib/forge-reference-image.js, its QA score and corrective retry, and the textToImage fallthrough ladder, Livepeer included) runs under one shared budget, TEXT_TO_IMAGE_BUDGET_MS (default 60 s). A stalled paint provider can no longer hold a submit open for minutes before a job exists; once the budget is spent the caller gets a fast, retryable failure instead.
The job handle
Async jobs get an opaque HMAC-signed token (f1.<payload>.<sig>, api/_lib/forge-job-token.js) that records provider, kind, and upstream task id, signed with JWT_SECRET. The legacy Replicate path keeps the bare prediction id. Tokens make the poll endpoint stateless: any instance can decode the handle and ask the right upstream.
Poll
GET /api/forge?job=<id> returns status: queued, running, done (with glb_url, viewer_url, and the path/tier/backend that actually produced it), or failed. Polling is free and unauthenticated.
Every non-terminal poll also carries honest timing: elapsed_seconds (the age of the job, from created_at, not of the tab watching it), eta_seconds (the lane's typical total) and eta_remaining_seconds. The remaining figure is floored at 5, because the estimate is a typical duration rather than a deadline and a countdown that goes negative reads as "stuck" on a job that is merely slower than average. A queued poll on a scale-to-zero self-host lane additionally carries cold_start and cold_start_seconds (that lane's spin-up budget). laneColdStart() is module-scope for exactly this reason: it used to be a closure inside the submit handler, so only the submit response could say a worker was booting, and a client that never saw it (a coalesced job, a page resumed after a reload, a failover successor) was told "queued" with no explanation for a wait that is a container boot rather than a stall. forgeStageNarration() and coldStartState() in src/shared/forge-frames.js turn those fields into the same copy on every surface, off the payload and never off a local clock, and the flag clears on the first real running poll rather than on a timer. The in-flight job id is also written to localStorage, so an interrupted browser session resumes polling for up to 30 minutes; background completion persists the result to the user's gallery either way (see Background generation).
Territory enforcement (api/_lib/forge-territory.js). The Hunyuan3D license excludes the European Union, the United Kingdom and South Korea for the model and its output. The Forge resolves the requester's country on the server from the edge geo header (clientCountry), never from the request body, and never routes a restricted request to a Tencent-model lane. Those lanes are hunyuan3d, huggingface (the Spaces chain runs Hunyuan3D-2 and 2.1) and modly_hunyuan (Hunyuan3D 2 Mini Turbo, under the Hunyuan3D 2.0 licence with the same exclusion). An unknown or malformed country is treated as restricted, so a request with no geo signal fails closed to the MIT and non-Tencent lanes. The filter sits in every place a lane is chosen: auto routing and its health-aware variant, the HuggingFace failure hop, poll-time redispatch, the retry_backends suggestions, and the prompt-to-avatar plan. An explicit backend=hunyuan3d, backend=huggingface or backend=modly_hunyuan from a restricted country answers 403 region_restricted and lists the open lanes in retry_backends. The public catalog is CDN-cached and country-agnostic, so it advertises the fail-closed defaults.
Resolution option. resolution (512, 1024 or 1536) is a request option honoured by the trellis2 lane and part of the dedup and cache key, so two resolutions of one prompt never share a result. It is rejected with a clear error when invalid, and a worker that cannot hold the requested size steps down and reports resolution_served.
Poll-time failover (api/_lib/forge-failover.js) closes the nastier gap: a self-host worker can accept a task and fail it minutes later, in a poll handled by a different instance that no longer holds the request. On a failed poll the module recovers the original inputs (prompt plus reference image) from the creation row, resubmits to the next configured lane, and binds old handle to new handle in Redis (2 hour TTL). The client keeps polling the same job id and simply sees status: running with a new backend. Hops are capped by MAX_FAILOVER_HOPS = 3 (one primary plus up to three backups). The redispatch order is trellis2, trellis_selfhost, hunyuan3d, trellis, modly, modly_hunyuan, then triposg as the final rung (Tencent lanes are skipped for a restricted or unknown territory). TripoSG runs its photo pipeline (gcp provider mode triposg) and returns untextured geometry, so it only serves a job once every textured lane has failed or is marked down; the client's lane note says the result will have no textures. It was added after the 2026-09-18 to 09-28 outage, when Hunyuan3D and TRELLIS self-host failed together, Replicate was unconfigured, and roughly 400 photo jobs a day ended in a hard failure while the TripoSG worker was healthy. TripoSG stays out of the retry_backends suggestions because its public lane is sketch-only; the two Modly lanes are image lanes and close that list. A Modly lane that rejects the submit itself (cold, restarting, throttled) is cooled and the same request retries on trellis2, then trellis_selfhost, inside the submit. HuggingFace is excluded because its provider blocks through the whole generation (cannot be ridden from a poll), and NVIDIA is excluded because poll-time redispatch always reconstructs from a stored reference image, which NIM rejects. Both failover paths (this attended one and the unattended finalizer sweep) link the failed attempt to its successor row (forge_creations.superseded_by, migration 20260814200000_forge_failover_supersede.sql), so the outcome ledger, the forge health sensor and npm run forge:errors count a recovered attempt as a recovery, not a loss.
Stalled NVIDIA jobs. The unattended finalizer (api/cron/forge-finalize.js) is the only consumer of a poll-time failure on the free NIM text lane once the browser stops polling. A stalled text-to-3D job there gets one resubmit (decideNvidiaStall), and both give-up points (nvcf_expired and the bare TTL timeout) now mark the forge-nim-trellis cooldown, so new submissions skip to the next lane within the 120 s window instead of feeding a broken lane for up to 45 minutes.
Designed terminal failure. When no successor is possible, the failed response carries retryable: true plus retry_backends: [...], the ordered configured lanes a client can one-click resubmit to. A failure is never a bare string. The whole module is fail-open: no Redis disables automatic redispatch but keeps suggestions; no DB disables recovery but never turns a clean failure into a hang.
Quality, caching, and scale
- Quality gate + best-of retry.
scoreQualityGateflags degenerate results (near-empty geometry, texture failures); synchronous lanes retry once inline before returning. - Result cache (
api/_lib/forge-cache.js). Identical text prompts on platform-keyed non-high lanes are served from cache withcached: true. The key covers path, tier, backend, the normalized prompt, and the output-affecting options (seed included), with a 7-day TTL that a read never refreshes.force_regenerate: truein the request body skips the cache READ while still writing the fresh result back. Any caller that must prove the pipeline is alive rather than that a mesh exists has to send it: the daily smoke cron (api/cron/forge-smoke.js) submits a constant prompt with no seed, so without the flag its key never changes and it verifies a replay. Automation that seeds the gallery takes the other route and varies the seed per submit (api/cron/forge-seed-cron.js), so a repeated prompt yields a genuinely new asset instead of a second catalog entry pointing at one GLB. - Scale controls (
api/_lib/forge-scale.js). In-flight coalescing, blocking-lane slots, and daily paid caps keep a traffic spike from stampeding the GPU fleet or the paid last resort. Its circuit breaker (circuitRecordFailure) widens the open window linearly per consecutive failure but caps it at one hour (CIRCUIT_MAX_OPEN_MS), so a provider outage parks the seed cron for at most an hour before it re-probes the lane.
6. Persistence and storage
forge_creations (Neon Postgres, migration api/_lib/migrations/20260604000000_forge_creations.sql) is the ground truth for every generation. Key columns:
- Identity and attribution:
id(uuid),client_key(sha256 of the anonymous browser id),ip_hash,user_id. - Inputs:
prompt,aspect,preview_image_url(the reference view),text_to_image_model,views_requested/views_used/multiview. - Routing outcome:
backend,tier,path,replicate_job_id(poll correlation for any provider, despite the name),model_category(avatar, accessory, item, scene, creature, vehicle, other). - Result:
glb_key,glb_url,size_bytes,status(generating,done,failed),error. - Feedback flywheel:
outcome(generated,accepted,rejected),rating,note,downloaded,feedback_at. The outcome partial index feeds training-signal export. - Funnel instrumentation (migration
20260921120000_forge_destination.sql):destination(what the maker is building for:game,web,avatar,simulation,print,ar,play; sent on the request or answered afterwards in the composer, null when never answered),completed_at(when the row reacheddoneorfailed;updated_atcannot stand in because later votes and feedback move it), andinternal(true for rows the catalog seeder or the quality benchmark generated, so they stay out of conversion numbers). The handler statesdestinationandinternalonce per request in a creation context thatcreateCreationreads, so no insert branch can omit them, and both failover paths copy them to the successor row. Read by the Forge funnel board. - Lineage and remix (migration
20260625000000_forge_refine_lineage.sql):parent_creation_id,refine_instruction,lineage_index,remixable,remix_royalty_bps(0 to 2000, default 1000),creator_wallet_solana,remix_settlement_ref. - Forge-Off (migration
20260625120000_forge_board.sql):vote_count, plus theforge_votesandforge_board_winnerstables. - Failover and cleanup:
superseded_by(migration20260814200000_forge_failover_supersede.sql, the successor row's id when this attempt was re-dispatched; a plain uuid, not a foreign key, so retention pruning stays free) andsource_image_keys(migration20260813160000_forge_source_image_keys.sql, the storage key of every uploaded reference photo, so a delete can erase them).
api/_lib/forge-store.js owns all writes: createCreation on submit (also records the funnel start event), materializeCreation on completion, markFailed, markSupersededBy when a failed attempt is re-dispatched, verdict capture, and deleteCreation (behind DELETE /api/forge-creation?id=), which removes the mesh, preview and source uploads from storage before the row and is scoped to the owning client key. It is fail-soft: without a DB or storage config it no-ops rather than blocking generation.
Storage is two-layer. GPU workers write raw output to GCS (gs://$GCS_BUCKET/raw-meshes/...), and provider URLs generally expire within about an hour. materializeCreation therefore mirrors the mesh and reference image into Cloudflare R2 (api/_lib/r2.js) and stores the durable same-origin /cdn/ URL (cdnUrl()) in glb_url. Stored and returned URLs go through the /cdn/ proxy (api/cdn-object.js), never the raw *.r2.dev domain, which browsers block on cross-origin reads (ORB/CORS); normalizeStoredUrl() rewrites legacy raw URLs on read. Anything user-facing must reference the R2 URL, never the raw provider URL.
When object storage is the thing that is down
A rejected S3 secret takes 3D generation and every avatar, thumbnail and GLB read down at once, and on 2026-09-07 it did so with no health signature at all, because the forge sensor only sees generations that got far enough to write a row. Four rules now bound that blast radius:
- It has its own sensor. api/_lib/ops/object-storage-health.js probes the bucket with a signed one-key list (retried once after a 300 ms backoff, so a one-off blip stays
okwhile two consecutive failures still classifydown) and reportsok/degraded/downwith the R2-specific hint (the secret is the token digest, not the token). - The reference-view write is no longer a single point of failure.
persistImageBytesruns before the reconstructor is ever called, so a refusing bucket returned 502 on 100% of text-to-3D while image-to-3D, which supplies its own view URLs, kept working. A storage-infrastructure fault now falls back to an inlinedata:URI under 4 MB (api/_lib/image-persist.js).backendAcceptsInlineViews()names the lanes that can read one, and it is true for our own GPU workers and only them, because everyworkers/model-*declaresimages: [data-uri|url]while a third-party reconstructor takes a URL it fetches itself. A programming error or an oversized payload still throws. - A refusal is reported as a refusal.
/api/forge-uploadused to check only that the S3 env vars were present, so a rotated token still minted a well-formed presigned URL, the bucket answered the browser's PUT with a403 SignatureDoesNotMatchcarrying noAccess-Control-Allow-Origin, and the page fell into its catch-all "network error" branch for a fault that had nothing to do with the user's connection.objectStorageUsable()inr2.jsruns the same probe, cached and single-flighted so a six-slot burst costs one, and both forge entry points answer503 storage_unavailablerather than forwarding a raw signing complaint. - Reads fail over to the public domain. When the signed read fails on infrastructure rather than on the object, api/cdn-object.js streams the same key from the public bucket domain. It must stream and never redirect:
r2.devsends noaccess-control-allow-origin, a browser re-runs the CORS check on a redirect's target, and the wildcard grant this handler sets is discarded the moment the hop is followed (measured on 2026-09-09, every cross-origin/cdnread threw "Failed to fetch" while the same URL curled 200). Streaming keeps the response on three.ws so the CORS, CORP and content-type headers survive the degraded path, and a short 60 s edge cache absorbs a thumbnail burst instead of pointing every browser at the rate-limited public domain the route exists to dodge. When both lanes fail on a storage-infrastructure fault, an image key gets a bundled 64x64 placeholder PNG with a 30 s cache instead of a raw 502, so real content resumes within 30 s of recovery; non-image keys (a.glb) still fail, because a static image cannot stand in for a model.
Anonymous presigned uploads cannot sign a content length, so api/cron/anon-upload-sweep.js (hourly) deletes objects that exceed their prefix ceiling in api/_lib/anon-upload-limits.js (for example 8 MB under forge/uploads/).
Note the asymmetry with rule 3: objectStorageUsable() fails closed only on a rejected credential, which never recovers on retry, and stays open through a timeout, so a flaky probe cannot take uploads down on its own.
The delivery variant: what a phone downloads is not what a download gives you
A raw generation is 1.25 to 3.5 MB of unquantized vertex data and embedded PNG skins. On a mid-tier phone over slow 4G (1.6 Mbps) that is ten to twenty seconds before the first frame, which is most of what made the model pages and the marketplace grid unusable on a handset.
So every finished mesh gets a second object, the web-delivery variant: meshopt geometry plus WebP textures capped at 2048 px on the long edge, produced by deliveryCompressionOptions() in api/_lib/glb-compress.js. Six real production forge outputs sampled on 2026-09-04 compressed like this:
| source | original | variant | saved |
|---|---|---|---|
| geometry-only mesh | 3.00 MB | 0.52 MB | 83% |
| geometry-only mesh | 1.25 MB | 0.38 MB | 70% |
| textured mesh | 2.94 MB | 0.51 MB | 83% |
| textured mesh | 1.70 MB | 0.11 MB | 94% |
| textured mesh | 2.64 MB | 0.22 MB | 92% |
| textured mesh | 3.34 MB | 0.67 MB | 80% |
Three rules govern it, and none of them are optional:
- It is a second object, never a replacement.
glb_urlstays exactly the bytes the caller'soutput_formatasked for.web_glb_url(migration20260904120000_forge_web_glb.sql, withweb_glb_keyandweb_size_bytes) is the variant. Viewers takeweb_glb_url; every download link and every third-party API consumer takesglb_url, so nobody with a bareGLTFLoaderis handed anEXT_meshopt_compressionfile they cannot decode, and nobody downloading an asset they will edit gets a texture-capped copy. - Null means "serve the original". A creation forged before this existed, one whose compression pass has not run, and one already small enough that a second object would not pay for itself all carry
web_glb_url = null. Every reader treats that asglb_url, so there is no separate "not ready" state to handle. - It is built after the row is already
done.buildWebVariant()is fire-and-forget, exactly like the physics grade beside it. The pass costs 0.6 to 5.6 s of CPU and the person waiting on their generation must never wait on it, nor lose a finished model to a compression bug or an instance recycle.
Meshes that predate the variant are drained by api/cron/forge-web-variants.js (every 10 minutes, FORGE_WEB_VARIANT_BATCH rows per tick, newest first, because the newest models are the ones being opened on a phone). It reads the stored original, writes the variant to a second key, and never rewrites the original object; a row that yields no gain has web_glb_key pointed at the original key so it leaves the pending index while web_glb_url stays null.
The decoder ships from our own origin. EXT_meshopt_compression needs a decoder that <model-viewer> loads as a classic script, and that URL used to be a public CDN. It is now /vendor/meshopt_decoder.js, vendored from the meshoptimizer package by npm run vendor:meshopt and pinned byte-for-byte by tests/meshopt-vendor.test.js. A hard dependency of the format we serve by default cannot have worse availability than the site itself, and the extra DNS plus TLS handshake sat on the critical path of the first 3D frame. The viewer also listens for a decode failure and retries once against glb_url, so a browser that genuinely cannot decode the variant gets a slower load rather than a dead page.
Texture dimensions come from the encoder, not from a header parse. @gltf-transform's own textureCompress({ resize }) derives its target from a minimal PNG header reader, and three of the six meshes above carry a PNG it reads as 65536x4292542531, which made the whole pass throw Expected positive integer for width but received 0 and lost the geometry win with it. recompressTextures() reads dimensions from sharp, which is the thing about to decode the bytes, and isolates every texture in its own try/catch so one unreadable skin costs that skin and nothing else.
The LOD variant: opt-in, because decimation is real work
Send "lods": true on the generation request (api/_lib/forge-options.js) and materializeCreation builds a third object alongside glb_url and web_glb_url: a single combined GLB tagged with the archived MSFT_lod extension, carrying LOD0 (the delivered mesh, untouched), LOD1 and LOD2 (decimated with meshoptimizer's simplifier through packages/glb-tools' buildLodChain). A real-time engine that understands MSFT_lod switches levels from this one file at runtime; one that does not simply renders LOD0, because the lower-detail levels are merged in as orphan nodes outside the scene's child graph.
It is off by default for the same reason the delivery variant is not: building three decimated meshes costs real CPU (meshoptimizer's simplifier, run three times), and most callers never render in an engine that benefits from a LOD chain. The same real pipeline backs the Engine QA panel's own "Build & grade LOD chain" / "Check LOD & collider fitness" upgrade action (public/engine-qa-panel.js, GET/POST /api/3d/engine-qa?lods=1, see engine-qa.md), so a caller who only wants to grade fitness, not keep the chain, can do that on demand without ever setting lods on the generation itself.
Two rules, mirroring the delivery variant:
- It is a second object, never a replacement.
lod_glb_url(migration20261015020000_forge_lod_variant.sql, withlod_glb_keyandlod_levels) is the combinedMSFT_lodGLB.glb_urlstays exactly the caller's requestedoutput_format, untouched. - Null means "no chain was requested, or the build failed."
buildLodVariant()inapi/_lib/forge-store.jsis fire-and-forget, built after the row is alreadydone, exactly likebuildWebVariant: the person waiting on their generation never waits on it, and a build failure (logged, never thrown) simply leaveslod_glb_urlnull.lod_levelsis a JSON array of{ name, ratio, triangleCount, vertexCount }per level, so a reader can show the chain's shape without fetching and parsing the GLB.
7. GPU workers
Workers are Python FastAPI services on Cloud Run (scale-to-zero L4 GPUs unless noted), documented worker-by-worker in workers/README.md. All share one contract: POST /infer returns {task_id}, GET /tasks/:id returns progress and finally a result_gcs_url, authenticated by the shared bearer GCP_RECONSTRUCTION_KEY. api/_providers/gcp.js maps modes to worker URLs.
| Worker | Role |
|---|---|
workers/model-trellis2/ |
Self-hosted TRELLIS.2, the default trellis2 lane. resolution 512, 1024 or 1536, PBR GLB with alpha, meshopt packed. |
workers/modly/ |
Modly's backend headless, the modly (TripoSG) and modly_hunyuan (Hunyuan3D 2 Mini Turbo) lanes. Untextured GLB, repaired and decimated to the tier's face budget. |
workers/model-trellis/ |
Self-hosted TRELLIS, the trellis_selfhost lane. Quality-tiered up to 4096px textures. |
workers/model-hunyuan3d/ |
Hunyuan3D-2.1 single image to textured mesh, the hunyuan3d lane. |
workers/model-pixal3d/ |
Pixal3D, the pixal3d "Match my photo" lane: 1 to 4 photos fused into one PBR GLB. |
workers/model-anigen/ |
AniGen, the anigen lane: mesh, skeleton and skin weights predicted together. |
workers/model-partcrafter/ |
PartCrafter, the partcrafter lane: named part meshes. workers/model-holopart/ is the companion that completes occluded geometry on meshes segmented after the fact. |
workers/model-triposplat/ |
TripoSplat behind /api/forge-splat: image to Gaussian splat. |
workers/model-sam3d-objects/ |
SAM 3D Objects behind /api/forge-scene-objects: photo to laid-out scene. 32 GB+ GPU, off until the gated weights are staged (npm run stage:sam3d-objects). |
workers/model-triposg/ |
TripoSG 1.5B. mode: image for the avatar pool, mode: scribble for the sketch path. |
workers/model-triposr/ |
TripoSR fast path and fallback (5 to 15s). |
workers/rig/ |
Make-It-Animatable rigging: 52-bone Mixamo skeleton, skin weights, ARKit-52 blendshapes, sub-second GPU predict, grafted into the original GLB bytes. Replaced the retired workers/unirig/ (2026-07-17); cutover was the GCP_UNIRIG_URL env change. |
workers/avatar-pipeline-controller/ |
CPU orchestrator for the avatar flow: POST /reconstruct picks a mesh backend by weighted random, then auto-rigs. |
workers/remesh/, workers/texture/, workers/segment/, workers/stylize/, workers/rembg/ |
Post-processing: retopology and Game-Ready, SDXL retexture, part splitting, geometric stylize, background removal. rembg also offers birefnet (birefnet-general-lite) for cutouts where hair and thin edges decide the mesh: roughly 6 s against the default's 1 s, so it is worth it on a hero image and not on a batch, and the default is deliberately unchanged. |
Model weights are staged from gs://three-ws-model-weights via workers/deploy/stage-weights.sh. Each worker has its own cloudbuild.yaml; every config pins the three-ws-build@ service account.
8. Post-processing
- Rigging.
POST /api/forge?action=rigprefers the self-host rig worker and falls back to Replicate; unconfigured deployments return a clean 501. Auto-rig-on-create (api/_lib/auto-rig.js) mints a sibling creation row rather than mutating the original (provenance and attestation depend on the original bytes staying put), gated by the humanoid prompt classifier (mcp-server/src/tools/_humanoid.js) andrigInfoIsRigged()(already-skinned GLBs are skipped). - Skeleton canonicalization.
src/glb-canonicalize.jsmaps any humanoid rig's bone names (Mixamo, Avaturn, Unreal, VRM/VRoid, Daz, MakeHuman, MMD/MikuMikuDance with its Japanese bone names, Blender.L, and more) onto the canonical bone set, repacking the GLB in place. There is no rig allowlist; a new convention means a new bone-name mapping plus a test case intests/glb-canonicalize.test.js. - Animation retargeting.
src/animation-retarget.jsapplies the pre-baked canonical clip library (public/animations/clips/*.json) to arbitrary rigs: rewrites track names, drops missing bones, rescales hip translation, and requires 50% bone coverage. Pure JS, shared between the browser and the Node MCP tool. - Refinement lineage.
api/forge-iterate.jsand the MCPrefine_modeltool share one core (mcp-server/src/tools/_lineage.js) for composing refinements and maintaining the revertable, branchable version tree overparent_creation_idandlineage_index. - Restyle.
api/material-studio.jsre-skins a GLB without regenerating the mesh: an LLM proposes PBR factors from an instruction, applied server-side. The paid MCPrestyle_materialis a thin client over it. - Mesh operations.
forge-remesh,forge-gameready(QuadriFlow retopology to a poly budget, GLB plus FBX, $0.10),forge-stylize,forge-segment,forge-rembg. Each is also sold as a standalone x402 stage; see 3D pipeline.
9. Payments and gating
The pipeline separates what a generation costs us from what it costs the user. Draft and standard route to free lanes; we charge for quality, not to recover vendor bills.
- Direct x402 (
api/x402/forge.js): per-tier USDC prices from the tier table, exact-settle on Solana mainnet. The order is verify, submit, settle: a failed submit never charges. The returned job token polls free on the standard endpoint. - High-tier gate (
api/_lib/forge-high-payment.js): the high tier is $THREE hold-or-pay gated regardless of which free engine serves it. Non-holders pay per generation through the token rail in three phases:assertForgePaymentvalidates at the gate,redeemForgePaymentatomically claims single-use (primary-key race-safe),releaseForgePaymentundoes the claim if dispatch fails. Settled consumption lands intoken_paymentswithref_type: 'forge'. - MCP pricing: the
paid()wrapper (mcp-server/src/payments.js) charges thepriceUsdeach tool module declares at its own call site (mesh_forge$0.25,rig_mesh$0.20,forge_avatar$0.45,refine_model$0.25,text_to_avatar$0.15,restyle_material$0.05, all undermcp-server/src/tools/);forge_freeneeds no payment. The hosted MCP server sums per-tool prices for batched x402 challenges. - Quota upsell: the free flagship API allows 10 generations per IP per day, then returns 429 with the x402 endpoint as the paid path.
- BYOK lanes bill the caller's own vendor key and are never charged to a platform account.
10. Operator reference
The env vars that shape a deployment (production values live on the Cloud Run service, not in any Vercel export):
| Var | Effect |
|---|---|
NVIDIA_API_KEY |
Enables the free NIM TRELLIS text lane. |
HF_TOKEN |
Enables the HF Spaces lane. |
MODEL_TRELLIS2_URL, MODEL_TRELLIS_URL, MODEL_MODLY_URL, MODEL_PIXAL3D_URL, MODEL_ANIGEN_URL, MODEL_PARTCRAFTER_URL, MODEL_TRIPOSPLAT_URL, MODEL_SAM3D_URL, GCP_HUNYUAN3D_URL, GCP_TRIPOSG_URL, GCP_UNIRIG_URL, GCP_RECONSTRUCTION_URL |
Self-host worker URLs (TRELLIS, Modly, Pixal3D, AniGen, PartCrafter, TripoSplat, SAM 3D Objects, Hunyuan3D, sketch, rig, avatar controller). |
TRIPO_API_KEY, TRIPO_WEBHOOK_SECRET |
Optional. The key makes Tripo the High-tier default geometry engine (house lane); the secret enables the signed Tripo webhook receiver. |
FORGE_PAY_TO_KEEP |
Default off; gates releasing the original file behind a $THREE hold or payment (api/forge-keep.js). |
GCP_RECONSTRUCTION_KEY |
Shared bearer for all self-host workers. |
REPLICATE_API_TOKEN |
Enables the paid Replicate last resort. |
FORGE_PREFER_FREE |
Default on; prefer free reconstruction before paid Replicate. |
FORGE_SELFHOST_PRIMARY |
Default off; hoist self-host lanes above hosted free lanes. |
JWT_SECRET |
Signs job handle tokens. |
DATABASE_URL, R2 S3_* config |
Persistence and durable storage; both fail soft. |
LIVEPEER_FEDERATION_ENABLED |
Default off; inserts the federated Livepeer text-to-image lane (api/_providers/livepeer.js) into the reference-image chain after the free lanes and before the paid Replicate backstop. See ops/livepeer-federation.md for the measured state before flipping it. |
LIVEPEER_API_KEY, LIVEPEER_GATEWAY_URL, LIVEPEER_T2I_MODEL |
Studio gateway key (absent today), gateway base-URL override, and pipeline model override (ByteDance/SDXL-Lightning default) for the federated lane. |
Fast diagnostics: forge_creations carries per-generation backend, status, error, and prompt, so it is the quickest ground truth for generation issues. GET /api/forge?health=1 shows the live lane picture.
Whether the output is any good to the people asking for it is a separate question with its own report: npm run forge:funnel (or the owner board at /forge-funnel) gives useful-output rate, repeat use, per-destination acceptance and attempts per kept model by engine. See ops/forge-funnel.md.
11. Extending the pipeline
Adding a generation lane:
- Add a provider client in
api/_providers/<name>.js(submit and poll, normalized errors). - Register the backend in
BACKENDS(api/_lib/forge-tiers.js): paths,freeorbyok,requiresEnv, ETA, credits. - Place it in the routing lists it belongs in:
FREE_FALLBACK_FOR_PATHand, if it should be a default,FREE_DEFAULT_FOR_TIERSorDEFAULT_BACKEND_FOR_PATH. - If it is async and can be re-dispatched from a poll, add it to
ASYNC_REDISPATCH_ORDERand the suggestion orders inapi/_lib/forge-failover.js. - Nothing else: the catalog, the UI engine picker, health checks, and x402 pricing all derive from the registry.
Adding a rig convention: add the bone-name mapping to src/glb-canonicalize.js and a case to tests/glb-canonicalize.test.js. Never introduce a curated rig allowlist.
The invariants to preserve:
- Free-first: a paid lane must never run while a healthy free lane can serve the request.
- One source of truth: prices, poly targets, and lane capabilities live only in
forge-tiers.js. - Fail-open persistence: the DB and storage layers may degrade, generation must not.
- Same job id across failover: the client never re-submits because a lane died.
- Durable URLs: user-facing results reference R2, never expiring provider URLs.
Related
- Forge, the product-level doc for the /forge surface.
- How the Forge works, the plain-language explainer.
- 3D pipeline, the pay-per-stage x402 pipeline (rig, remesh, gameready, stylize, rembg).
- 3D asset pipeline, formats and the animation/conversion flow.
- Avatar pipeline, the selfie-to-avatar reconstruction flow.
- Background generation, job resume and gallery persistence.
workers/README.md, the worker-by-worker operational reference.