Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
GCP model workers — the self-host generation lanes
three.ws runs open 3D models on our own Google Cloud Run GPU workers, against
the project's GCP credits. These self-host lanes are the resilient default for
/api/forge: every generation we can serve on our own hardware costs no vendor
money and carries no BYOK dependency. This runbook covers what is deployed, the
env that wires it in, how routing picks a lane, and how to (re)start a worker.
Scope: routing + operations for the forge generation lanes. The avatar reconstruction pipeline (controller + the
model-rigauto-rigger) is documented inworkers/deploy/README.md; only its shared bearer secret overlaps here.
The workers
Each model runs as its own Cloud Run service (NVIDIA L4 GPU, us-central1),
built and deployed from workers/<dir>/cloudbuild.yaml. They all speak the same
task shape — POST /<endpoint> → { task_id }, GET /tasks/:id → { status, result_* } — and authenticate with one shared bearer secret.
Lane (backend id) |
Worker dir | Cloud Run service | Path served | What it does |
|---|---|---|---|---|
trellis2 |
workers/model-trellis2 |
model-trellis2 |
image |
Default image→3D lane. TRELLIS.2 (MIT): PBR GLB with alpha, resolution 512/1024/1536, meshopt packed. Weights staged by workers/model-trellis2/stage_weights.sh. |
trellis_selfhost |
workers/model-trellis |
model-trellis |
image |
Native single-hop image→3D (Microsoft TRELLIS). Accepts user photos and the FLUX-synthesized view for text prompts. Textured GLB. |
hunyuan3d |
workers/model-hunyuan3d |
model-hunyuan3d |
image |
High-poly image-conditioned reconstruction (Tencent Hunyuan3D). Poly-budget aware. |
modly |
workers/modly |
model-modly |
image |
TripoSG through Modly's headless backend (MIT). Single photo to untextured GLB, repaired and decimated to the tier's face budget. Weights staged by workers/modly/stage_weights.sh. |
modly_hunyuan |
workers/modly |
model-modly |
image |
Hunyuan3D 2 Mini Turbo on the same worker. Untextured. A Tencent lane: never routed to a restricted or unknown territory (api/_lib/forge-territory.js). |
triposg |
workers/model-triposg |
model-triposg |
sketch |
Sketch→3D (TripoSG-scribble): a drawing + a prompt naming it → untextured geometry. |
All three are provider: 'gcp', free: true. model-trellis pins one
always-warm instance and bursts to three under load (_MIN_INSTANCES: "1",
_MAX_INSTANCES: "3"): a cold FUSE-mounted weight load can stall for many
minutes, and the GCP credit budget covers the standing GPU-hour.
model-hunyuan3d's cloudbuild default is also min 1, but the live service was
set to --min-instances=0 on 2026-07-26 to free the shared us-central1 L4 pool
(3 granted; see the gpu-quota-starved signature in
docs/ops/production-log-triage.md); model-triposg scales to zero. A
cold start on a scale-to-zero worker pays a one-time model-load before the
job runs. We surface that honestly (see Cold start), never as a fake timer,
and the gpu-keepwarm cron (api/cron/gpu-keepwarm.js, every 10 min during
peak hours via Cloud Scheduler) holds allowlisted scale-to-zero lanes resident
where warming them does not contend for that L4 pool (FORGE_KEEPWARM_LANES
overrides the set without a deploy). A lane id in that override that matches no
known lane comes back in the tick's unknown_lanes and drops its ok to false,
so a typo in the env var fails visibly instead of quietly warming nothing.
Every routed lane that scales to zero must appear in that registry even when it
is not warmed by default, because the registry is what the override can reach: a
lane missing from it is unwarmable at any quota, and its absence is invisible
(the tick's skipped_for_quota simply lists one fewer deferred lane and still
reads healthy). model-hunyuan3d sat outside it that way despite carrying the
fleet's longest spin-up. tests/cron-forge-lane-guards.test.js now pins the
registry against BACKENDS: every provider: 'gcp' lane with a
coldStartSeconds budget must be listed, under the same URL env var the router
itself resolves that worker from, so a typo'd urlEnv (which would report
unconfigured forever without ever throwing) fails the suite instead.
us-east4 has its own grant of 3 L4s and the same arithmetic applies there. It
runs standbys of model-trellis (pinned at min 1), model-hunyuan3d,
model-triposr, and model-text2motion, which is that region's production lane
(GCP_TEXT2MOTION_URL, max 2) and the only one of the four carrying user
traffic. The model-triposr standby was pinned at min 1 until 2026-08-11, which
left text2motion a single slot to burst into; it now scales to zero, as does the
model-triposr-warm Cloud Scheduler probe in us-central1 (paused the same
day, since each 10-minute ping cold-booted an L4 for a lane with no traffic).
Re-pinning either one is a quota decision: name the lane giving up the slot.
Wire shapes: model-trellis and model-hunyuan3d both speak the standard task
shape (POST /infer → { task_id }, GET /tasks/:id → result_gcs_url);
the gcp provider drives them with modes trellis and hunyuan respectively.
The Hunyuan worker also aliases /reconstruct + /jobs/:id (returning
glb_url) for callers still on the avatar-controller contract; new code should
use the hunyuan mode.
The remesh, stylize, texture, rembg, segment workers back
post-generation tools (Game-Ready export, retexture, etc.), not the primary
generation lanes; they follow the same task shape and bearer secret.
Environment
A self-host lane is configured only when its worker URL and the shared
key are present; a deployment missing either degrades cleanly — the lane drops
out of routing and the catalog reports it configured: false. Nothing is ever
faked.
| Env var | Used by | Notes |
|---|---|---|
MODEL_TRELLIS2_URL |
trellis2 |
Cloud Run URL of model-trellis2. Unset means the lane is skipped. |
MODEL_TRELLIS_URL |
trellis_selfhost |
Cloud Run URL of model-trellis. |
GCP_HUNYUAN3D_URL |
hunyuan3d |
Cloud Run URL of model-hunyuan3d. Not GCP_RECONSTRUCTION_URL — that is the avatar face pipeline, which rejects non-face images. |
MODEL_MODLY_URL |
modly, modly_hunyuan |
Cloud Run URL of model-modly. Unset means both Modly lanes are skipped. |
GCP_TRIPOSG_URL |
sketch, triposg |
Cloud Run URL of model-triposg. Serves the sketch lane and the last-resort, untextured photo rung of the poll-time failover. |
GCP_UNIRIG_URL |
auto-rig (rerig) |
Cloud Run URL of model-rig (the env name predates the engine swap that retired the old unirig service). Required for rigging: it is the only rerig backend, so without it the provider reports the mode unsupported rather than submitting anywhere. The provider speaks the worker's native schema (mesh_gcs_url in, rigged_gcs_url out) when this is set. |
GCP_RECONSTRUCTION_KEY |
all of the above | Shared bearer secret every worker checks (avatar-reconstruction-key in Secret Manager; model-rig's API_KEY references the same secret). |
GCP_REMESH_URL |
Game-Ready export | model/remesh worker (post-gen). |
FORGE_PREFER_FREE |
routing (optional) | Defaults on. Set false only to restore the paid-default ordering once the paid account is funded. |
The URL each worker registers on the three-ws-api Cloud Run service env is
printed at the end of workers/deploy/deploy-all.sh, or read it from
gcloud run services describe <service> --region us-central1 --format='value(status.url)'.
Secrets never appear in logs or the public catalog.
How routing picks a lane
Routing is health-aware and free-first. For a request with no explicitly chosen
backend, /api/forge builds the ordered list of free lanes that can serve the
(path, tier, userImages) request (freeLaneCandidates in
api/_lib/forge-tiers.js), then picks with a cached liveness snapshot
(api/_lib/forge-lane-health.js):
- Healthy self-host first. The first lane the liveness probe confirms
ok, and our own GPU workers lead the ordering — so a healthytrellis_selfhost(thenhunyuan3d) wins before any external free lane. - Then another healthy free lane. NVIDIA NIM (native text→3D, draft/standard) or HuggingFace Spaces (photos + high tier) when no self-host lane is healthy.
- Only then the paid default. The standing Replicate TRELLIS lane is the last resort, used solely when every free lane is confirmed down or none is configured. BYOK vendors (Meshy/Tripo/Rodin) are never auto-selected — they stay explicitly selectable in the catalog.
Liveness combines two signals, both fail-open (missing telemetry never blocks a lane — an unknown lane is treated as usable):
- A recent submit failure records a short cooldown in the shared cache, so every instance skips a just-failed lane until it expires.
- A cheap authenticated GET against the worker root (Cloud Run answers
<500once routable). Round-trip latency doubles as a warmth signal.
The snapshot is cached per instance (~20s), so a burst of generations shares one probe and the hot path usually pays no probe latency.
Submit-time failover
Health routing skips a lane known-bad before submit. As a safety net, if the
chosen self-host worker still errors or times out at submit (cold, restarting,
throttled), the handler cools that lane and transparently retries the next
configured lane for the path — trellis_selfhost → hunyuan3d → HuggingFace →
paid Replicate — before surfacing any error. The sketch path has a single lane,
so a TripoSG failure returns a designed, retryable 503 rather than a dead error.
Every response reports the backend that actually served it, so a failover is
never silent.
Poll-time failover
A worker can also accept a task and then fail it asynchronously — the failure
only surfaces on a later GET /api/forge?job=<id> poll. That poll
(api/_lib/forge-failover.js) cools the failed lane, recovers the original
inputs from the creation row (prompt + reference image), resubmits to the next
configured async lane (trellis_selfhost → hunyuan3d → paid Replicate, up to
3 backups per job), and binds old-handle → new-handle in Redis. The client
keeps polling the same id and sees status: "running" with the new backend
plus failover_from — the failure never reaches the user. When no automatic
lane remains, the terminal failed response names the still-configured lanes
in retry_backends so the forge UI offers one-click engine switches (and hops
to the first one automatically). HuggingFace can't be driven from a poll (its
provider blocks through the generation), so it appears only in the
suggestions, served by the client's fresh resubmit. Without Redis the
successor chain disables cleanly and only the suggestion layer applies.
Cold start
When the liveness probe reaches a self-host worker but it answers slowly (a
scale-to-zero container spinning up), the response carries cold_start: true and
the eta_seconds is widened by that worker's cold-start budget
(coldStartSeconds in the registry). This only widens the estimate — actual
progress still comes from real polling of GET /api/forge?job=<id>.
The realism pipeline (text→3D)
Everything below runs on GCP credits; no third-party API is involved.
- Prompt direction. The Granite art-director pass runs by default on text
prompts (
director: falseopts out): it rewrites the raw prompt into a single-subject spec with photoreal-by-default material cues (api/_lib/forge-director-prompts.js). Fail-soft: a dead LLM lane keeps the raw prompt. - Reference image.
textToImage()leads with the Vertex Gemini image model (gemini-2.5-flash-image) whenGOOGLE_CLOUD_PROJECTis set, materially more photoreal than 4-step distilled FLUX.VERTEX_IMAGEN_FIRST=0restores the legacy NIM-first order.enhanceFluxPrompt()appends realism + isolation cues unless the caller named an art style or composition. - Turnaround views. For the fusing self-host TRELLIS lane at standard/high
tier,
synthesizeTurnaroundViews()rotates the reference into side + back views via the Vertex Gemini edit lane (identity-preserving instructions). Best-effort: a failed view means fewer views, never a failed job. - Multi-view fusion.
workers/model-trellisfuses all supplied views of one subject (TrellisImageTo3DPipeline.run_multi_image) so unseen sides stop being hallucinated. Single view uses the plainrun()path. - Per-tier sampler/export budgets.
SELFHOST_TRELLIS_QUALITY(api/_lib/forge-tiers.js) scales diffusion steps, kept geometry and baked texture size per tier (draft 12 steps/1K, standard 35/2K, high 50/4K; texture sizes must be powers of two or the nvdiffrast bake hard-fails); the worker clamps every knob server-side (_clamped_quality, defaults 40 steps / 4K texture / 0.75 simplify). The Hunyuan3D worker mapstier+target_polycountto its own generation budgets.
Health & cost observability
- Live health:
GET /api/forge?healthprobes every lane (auth/quota gates, zero vendor spend), the limiter store, and recent real generation outcomes. - Free-vs-paid serve counts: the same payload's
metrics.by_costreports how many generations were servedfree(our GPU workers + free external lanes) vspaid(Replicate platform credits / BYOK), plusmetrics.free_share. This is the headline number proving the vendor-cost reduction. Counters are the rolling buckets inapi/_lib/forge-events.js; they need Redis (omitted otherwise). - Structured logs: every terminal generation logs one
evt:"forge_gen"JSON line withbackend, so a log drain can compute the same split with no Redis.
(Re)starting a GPU worker
The workers scale to zero; they do not need a manual "start" for normal traffic — the first request wakes a container (cold start). You restart/redeploy when shipping a new image or recovering a wedged service.
Redeploy a single worker (Cloud Build builds the image and deploys the service in one step):
gcloud builds submit --config workers/model-trellis/cloudbuild.yaml . \
--region us-central1 --project aerial-vehicle-466722-p5 \
--substitutions=SHORT_SHA=manual$(date +%s)
# swap in workers/model-hunyuan3d/… or workers/model-triposg/… as needed
SHORT_SHA is only populated for trigger-driven builds. A manual submit must
pass it, because these configs tag and deploy .../server:$SHORT_SHA, and an
empty tag fails the deploy step after the whole image has already built.
Provision/redeploy the whole pipeline (idempotent; prints the URLs to set on
the three-ws-api Cloud Run service env):
PROJECT_ID=<gcp-project> SERVICES="hunyuan3d trellis triposg rig" \
workers/deploy/deploy-all.sh
Force a fresh revision / clear a wedged container without a code change:
gcloud run services update model-trellis --region us-central1 \
--update-env-vars OPS_BUMP="$(git rev-parse --short HEAD)"
Keep a worker warm (eliminates cold starts at a standing GPU cost — use only under sustained load):
gcloud run services update model-trellis --region us-central1 --min-instances 1
Verify a worker is live (same check the router's liveness probe makes):
curl -sS -o /dev/null -w '%{http_code}\n' \
-H "authorization: Bearer $GCP_RECONSTRUCTION_KEY" "$MODEL_TRELLIS_URL"
# any <500 means up and routable
Note: GPU-backed Cloud Run requires L4 quota in the region (
us-central1). Quota approval can take hours — request it before a first deploy. The shared A100 workbench used for experiments is separate from these serving workers and may be stopped without affecting them.