Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
avatar-pipeline-controller
The CPU orchestrator behind Scan yourself to 3D. A FastAPI service that turns selfie photos into one rigged, animation-ready GLB by fanning out to the GPU model + rigging workers and surfacing the whole thing as a single job the site can poll.
When it is the wired reconstruct backend, the site talks only to this
controller: it never calls the mesh or rigging GPU services directly. The
controller picks a mesh backend, runs the raw mesh through the rig worker,
uploads the final GLB to Cloud Storage, and reports status. It holds no model
weights and runs no inference; all GPU work happens in the services it calls. It
keeps the same external contract the avatar-reconstruction service exposes, so
the backend provider (api/_providers/gcp.js,
env GCP_RECONSTRUCTION_URL) needs no changes to swap between them.
Not deployed today. No
avatar-pipeline-controllerCloud Run service exists inus-central1, andGCP_RECONSTRUCTION_URLonthree-ws-apipoints at the single-serviceavatar-reconstructionworker (face_texture_transfer_v2), which serves/scantoday. This controller is the multi-backend replacement lane: it builds and runs, andworkers/deploy/deploy-all.shprovisions it together with the GPU workers it fans out to. Standing it up is onedeploy-all.shrun plus repointingGCP_RECONSTRUCTION_URL; both are owner-gated production changes. Verify withgcloud run services listbefore trusting any deployment claim here.
/scan (browser)
└─ POST /api/avatars/reconstruct (three-ws-api on Cloud Run)
└─ controller POST /reconstruct (this service, Cloud Run, CPU)
├─ mesh model POST /infer (Cloud Run, L4 GPU) Hunyuan3D / TRELLIS / TripoSR / TripoSG
└─ model-rig POST /rig (Cloud Run, L4 GPU) skeleton + skinning + blendshapes
└─ rigged GLB → GCS → GET /jobs/:id returns glb_url
What it does
For each job the controller (main.py, _run_pipeline):
- Picks a mesh backend — weighted random over the backends whose URL env var
is set (
_pick_model), or honours an explicitmodelin the request when that backend is configured. - Generates the mesh —
POST {backend}/inferwith the images, then pollsGET {backend}/tasks/:id(3 s cadence, 300 s timeout) until the raw mesh GCS URL is ready. - Auto-rigs — if
UNIRIG_URLis set andSKIP_RIGGINGis off,POST {UNIRIG_URL}/rigwith the mesh URL (template: "wolf3d_neutral",blendshapes: true), polls to completion (3 s cadence, 180 s timeout). If the rig worker rejects the job, the pipeline falls back to the unrigged mesh rather than failing. - Finalizes — copies the resulting GLB into the output bucket under
avatars/{job_id}.glb(skipped if it is already there) and writes the finalstatus: "done"+glb_url+ timing to Firestore. The download is streamed with a 512 MiB ceiling (MAX_GLB_BYTES), so a runaway mesh fails the job instead of the instance.
Job state lives in the Firestore collection avatar_pipeline_jobs, so a job
survives an instance restart and any instance can answer a poll. The pipeline
itself runs as a FastAPI background task, so a job whose instance dies mid-run
stays at its last written stage; the platform-side backstop for that is
api/cron/reconstruct-sweep.js, which
re-polls quiet reconstruct jobs and finalizes them. All
inter-service calls carry the shared bearer secret; failures are recorded as
status: "failed" with an opaque, correlation-id-tagged error (safe_error in
worker_security.py).
This service exposes no
/rigendpoint of its own; it calls the rig worker's/rig.api/_providers/gcp.jsfalls back toGCP_RECONSTRUCTION_URLforrerigjobs only whenGCP_UNIRIG_URLis unset; in that fallback the rig submit hits this controller's non-existent/rigand 404s. SetGCP_UNIRIG_URLsorerigreaches the realmodel-rigworker.
API
All routes require an Authorization: Bearer <API_KEY> header (constant-time
check). /health is unauthenticated.
POST /reconstruct → 202
Request:
{
"images": ["https://…/selfie1.jpg", "https://…/selfie2.jpg"],
"body_type": "neutral",
"model": null,
"tier": null,
"path": "image",
"target_polycount": null,
"job_id": null
}
images— 1 to 6 image URLs (required).body_type— defaults to"neutral".model— optional; force a specific configured backend (hunyuan3d,trellis,triposr,triposg). Ignored if that backend isn't wired.tier/path/target_polycount— optional quality-tier provenance (api/_lib/forge-tiers.js).target_polycount(100…1,000,000) is forwarded to backends that honour a poly budget and recorded on the job.job_id— optional caller-supplied id; a UUID is generated if omitted.
Response:
{ "job_id": "6f1c…", "status": "queued", "model": "hunyuan3d" }
The pipeline runs in a background task; poll /jobs/:id for progress.
GET /jobs/{job_id} → 200 / 404
Returns the full Firestore job document, e.g.:
{
"job_id": "6f1c…",
"status": "done",
"stage": "done",
"model": "hunyuan3d",
"image_count": 2,
"body_type": "neutral",
"tier": null,
"path": "image",
"mesh_gcs_url": "https://storage.googleapis.com/…/mesh.glb",
"mesh_time_ms": 42010,
"rig_time_ms": 18300,
"glb_url": "https://storage.googleapis.com/three-ws-avatar-reconstructions/avatars/6f1c….glb",
"total_time_ms": 61550,
"created_at": "2026-07-10T12:00:00+00:00",
"updated_at": "2026-07-10T12:01:01+00:00"
}
status progresses queued → running → done (or failed); stage mirrors the
active step (queued, mesh_generation, rigging, finalizing, done,
failed). A failed job carries an opaque error string. 404 if the id is
unknown.
GET /health → 200
{
"ok": true,
"pipeline": "avatar_pipeline_controller_v2",
"backends": ["hunyuan3d", "trellis"],
"weights": { "hunyuan3d": 0.6, "trellis": 0.4 },
"unirig": true,
"skip_rigging": false
}
Reports which mesh backends are wired, their normalized routing weights, and whether rigging is active. It reflects configuration only — it does not probe the downstream services for liveness.
Env
| Var | Required | Default | Notes |
|---|---|---|---|
API_KEY |
yes | — | Shared bearer secret checked on every request and sent to every backend (Secret Manager: avatar-reconstruction-key). |
GCS_BUCKET |
yes | — | Output bucket for final GLBs (three-ws-avatar-reconstructions). |
FIRESTORE_PROJECT |
yes | — | Project holding the avatar_pipeline_jobs collection. |
MODEL_HUNYUAN3D_URL |
one+ | — | Cloud Run URL of the Hunyuan3D mesh worker. |
MODEL_TRELLIS_URL |
one+ | — | Cloud Run URL of the TRELLIS mesh worker. |
MODEL_TRIPOSR_URL |
one+ | — | Cloud Run URL of the TripoSR mesh worker. |
MODEL_TRIPOSG_URL |
one+ | — | Cloud Run URL of the TripoSG mesh worker. |
UNIRIG_URL |
no | (none) | Cloud Run URL of the rig worker, deployed as model-rig. The env name predates the engine swap that retired the old unirig worker. Unset (or SKIP_RIGGING=true) returns the unrigged mesh. |
MODEL_WEIGHTS |
no | equal | JSON routing weights, e.g. {"hunyuan3d":0.6,"trellis":0.4}; normalized to the configured backends. |
SKIP_RIGGING |
no | false |
true skips the rigging stage (testing). |
At least one MODEL_*_URL must be set — with zero backends, /reconstruct
returns 503 no model backends configured.
The backend wires to this service through two env vars on the three-ws-api
Cloud Run service: GCP_RECONSTRUCTION_URL (this controller's URL) and
GCP_RECONSTRUCTION_KEY (the same shared secret as API_KEY).
Run
Local
cd workers/avatar-pipeline-controller
pip install -r requirements.txt
API_KEY=dev-secret \
GCS_BUCKET=three-ws-avatar-reconstructions \
FIRESTORE_PROJECT=your-gcp-project \
MODEL_TRELLIS_URL=https://model-trellis-…run.app \
UNIRIG_URL=https://model-rig-…run.app \
uvicorn main:app --host 0.0.0.0 --port 8080
Firestore and Cloud Storage need Application Default Credentials
(gcloud auth application-default login). The downstream model services must be
reachable, or /reconstruct jobs will fail.
Tests
test_pipeline.py drives the real app end to end (submit → mesh → rig →
finalize → poll) against a threaded stdlib HTTP server speaking the workers'
task contract, with in-process stand-ins for Firestore and Cloud Storage. No
GPU, no GCP credentials, no network:
cd workers/avatar-pipeline-controller
pip install -r requirements.txt
python3 test_pipeline.py # → "all controller pipeline checks passed"
It covers the happy path, the copy-into-bucket and oversize-rejection branches of finalize, degradation when the rig worker rejects a job, an opaque failure when a mesh backend fails, bearer auth on every route, and model routing.
Container
docker build -t avatar-pipeline-controller workers/avatar-pipeline-controller
Production
When provisioned, it deploys as the avatar-pipeline-controller Cloud Run service in
us-central1, CPU-only (2 vCPU / 2 GiB, --min-instances 1,
--max-instances 4, 600 s timeout, no GPU, orchestration only), from
cloudbuild.yaml. Its build step is dir: workers/avatar-pipeline-controller,
so the submit must run from the repo root with . as the source, and a
manual submit must supply SHORT_SHA (only build triggers set it):
gcloud builds submit --config workers/avatar-pipeline-controller/cloudbuild.yaml \
--region us-central1 --project aerial-vehicle-466722-p5 \
--substitutions=SHORT_SHA=manual$(date +%s) .
That build sets GCS_BUCKET, FIRESTORE_PROJECT, and SKIP_RIGGING=false and
mounts API_KEY from the avatar-reconstruction-key secret, but does not
know the backend URLs (they don't exist until the GPU services deploy). The full
pipeline provisioner wires those in afterward and prints the controller URL:
PROJECT_ID=<gcp-project> SERVICES="hunyuan3d trellis triposr rig" \
workers/deploy/deploy-all.sh
Example
Submit a job and poll it:
CTRL=https://avatar-pipeline-controller-…run.app
KEY=$GCP_RECONSTRUCTION_KEY
JOB=$(curl -sS -X POST "$CTRL/reconstruct" \
-H "authorization: Bearer $KEY" \
-H 'content-type: application/json' \
-d '{"images":["https://storage.googleapis.com/three-ws-avatar-reconstructions/samples/selfie.jpg"],"body_type":"neutral"}' \
| python3 -c 'import sys,json;print(json.load(sys.stdin)["job_id"])')
curl -sS "$CTRL/jobs/$JOB" -H "authorization: Bearer $KEY"
# → {"status":"running","stage":"mesh_generation", …} then {"status":"done","glb_url":"…"}
Liveness check (any status < 500 means routable):
curl -sS "$CTRL/health"
Related
- Full pipeline runbook:
workers/deploy/README.md. - GPU mesh/rig lanes + shared secret:
docs/ops/gcp-model-workers.md. - Cloud Run production runbook:
docs/ops/gcp-production.md.