Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
model-video2motion
Video of a person to a retargetable animation clip + body-swap plates. A
CPU-only FastAPI worker (no GPU, deploys like the segment/rembg lanes) that
powers the Motion Swap page (/motion-swap): upload a video of yourself,
get it back with your avatar performing your motion. The same capture path
produces the lexical clips behind sign-language playback (src/sign-speech.js),
which is why hand articulation is solved as carefully as the body.
What it produces
For one input video, four artifacts land in GCS under motion-swap/<task_id>/:
| Artifact | What it is |
|---|---|
clip.json |
three.js AnimationClip.toJSON() on the canonical Wolf3D skeleton, the SAME document shape the animation library and model-text2motion serve, so it retargets onto any rigged avatar via src/animation-retarget.js |
video.mp4 |
the normalized source (long edge capped to 1280, capped fps/duration, audio kept, source metadata stripped) |
mask.mp4 |
grayscale per-frame person mask aligned with video.mp4 |
meta.json |
version, fps, frames, width, height, and per-frame screen anchors (hip position x/y, subject height h, visibility v) for pinning the avatar over the subject |
The browser compositor does the rest: original footage underneath, subject pixelated via the mask, avatar retargeted onto the clip and pinned per frame.
Models (all Apache-2.0, baked into the image)
- MediaPipe PoseLandmarker (heavy): 33 world landmarks per frame, VIDEO mode with tracking.
- MediaPipe HandLandmarker: 21 landmarks per hand, up to 2 hands, VIDEO mode with tracking.
- MediaPipe ImageSegmenter (selfie multiclass 256x256): the person mask is
1 - background confidence, which holds up far better than the single-class selfie model on wide framing (house tours, rooms).
The landmark to joint-rotation solve is ours: pose_solver.py builds global
orientations for pelvis/chest/head from body lines, solves each limb with its
bend-plane to fix twist, converts to locals against the parent chain via
hierarchical FK in the canonical rig convention, and emits
hemisphere-continuous quaternion tracks. Pure NumPy, unit-tested from synthetic
poses (test_pose_solver.py) without mediapipe installed.
When hands are visible, the solver also emits full hand articulation: detected
hands are assigned to the subject's left/right side by proximity to the pose
wrists (the model's handedness label is only a mirrored-convention fallback),
the wrist gets a palm-frame orientation (fixing twist the body-only solve
cannot see), and all 30 canonical finger bones (LeftHandIndex1 through
RightHandThumb3) are solved as hinge rotations per frame. A frame that loses
hand tracking holds the previous finger pose instead of snapping to rest. This
is what makes fine-gesture and sign-language capture work: a video of a signer
comes back as a clip whose handshapes retarget onto any rigged avatar.
API
POST /infer { video_url, fps?=24, max_seconds?, job_id? } → 202 { task_id, status }
GET /tasks/:id → { status, result_url?, meta_url?, video_url?, mask_url?, frames?, fps?, error? }
GET /health → { ok, models_loaded }
fps is clamped to 8..30 by request validation; max_seconds is clamped down
to MAX_SECONDS. job_id, when supplied, becomes the task id, which lets a
caller choose its own correlation id; the platform does not send one and simply
polls the task_id it got back. status moves queued to running to done
or failed; a failure carries an opaque error string with a correlation id
that matches the server log line.
Bearer auth on /infer and /tasks/:id (Authorization: Bearer $API_KEY);
/health is unauthenticated so uptime checks need no secret. Input is fetched
through the SSRF-hardened worker_security.fetch_remote_bytes (https-only,
private/loopback/metadata IPs rejected, redirect hops re-validated), capped at
256 MiB, and processing is capped at MAX_SECONDS (default 90).
The platform reaches this worker through the GCP provider's video2motion
mode (GCP_VIDEO2MOTION_URL), exposed as POST /api/motion-swap
(+ GET /api/motion-swap?job=<id>).
Environment
| Var | Default | Purpose |
|---|---|---|
API_KEY |
required | bearer secret for /infer and /tasks/:id |
GCS_BUCKET |
required | output bucket for the four artifacts |
MAX_CONCURRENT |
2 |
in-flight captures per instance |
MAX_SECONDS |
90 |
hard cap on processed duration |
MODELS_DIR |
/models |
directory holding the three model bundles |
Startup fails fast if any bundle is missing from MODELS_DIR.
Deploy
gcloud builds submit --config workers/model-video2motion/cloudbuild.yaml \
--region us-central1 --project aerial-vehicle-466722-p5 \
--substitutions=SHORT_SHA=manual$(date +%s)
Then point the API at it:
gcloud run services update three-ws-api --region us-central1 \
--update-env-vars GCP_VIDEO2MOTION_URL=$(gcloud run services describe \
model-video2motion --region us-central1 --format='value(status.url)')
Shares the avatar-reconstruction-key bearer secret (GCP_RECONSTRUCTION_KEY
on the API side) with the other reconstruction-family workers.
The instance count is load-bearing, not a cost setting. The service runs
--min-instances=1 --max-instances=1 --no-cpu-throttling. Capture happens in a
FastAPI background task after the 202 returns, so request-based billing starves
it (a 30 s clip took 3.5 h wall in production versus 3 min unthrottled, measured
2026-07-19); min 1 stops the instance being reaped between polls, and max 1
keeps the in-memory task store authoritative, so a poll always lands on the
instance running the job. Raising max-instances would make /tasks/:id
return 404 for live jobs.
Tests
The solver tests need only NumPy. The service tests additionally need ffmpeg and the worker's own requirements:
cd workers/model-video2motion
python -m pytest test_pose_solver.py test_service.py -q
test_service.py covers the paths around the solver with real ffmpeg and real
MediaPipe: the HTTP auth/validation contract, video normalization (fps,
duration, long-edge and metadata handling), mask-video encoding and decode
round-trip, the SSRF guard on the caller-supplied URL, the queue to
task-store failure path, and a real three-model pass over a generated video.
The MediaPipe test skips unless the bundles are staged; point MODELS_DIR at a
directory holding all three, or run the suite inside the image, which already
has them:
docker build -t model-video2motion:local workers/model-video2motion
docker run --rm -v "$PWD/workers/model-video2motion:/src" -w /src \
-e API_KEY=test -e GCS_BUCKET=test model-video2motion:local \
sh -c 'pip install -q pytest && python -m pytest test_pose_solver.py test_service.py -q'
Local development
local_solve.py runs the same pose and hand pipeline as the server (minus
segmentation, GCS and HTTP) straight to a clip.json, which is the fast loop
for changing pose_solver.py:
python local_solve.py input.mp4 out_clip.json --fps=24 --dump-landmarks=lm.npz
It reads bundles from $MODELS_DIR (default ./.models, where the pose and
hand bundles are checked in). Stage any missing bundle from the same public
URLs the Dockerfile uses:
curl -fsSL -o .models/selfie_multiclass_256x256.tflite \
https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_multiclass_256x256/float32/latest/selfie_multiclass_256x256.tflite
Verify a real capture
The pytest suite deliberately stops short of GCS uploads and real human
footage. To prove the whole path, run the image against a live video URL with
credentials mounted (any public https video of a person works; a previous
capture's normalized video.mp4 is a convenient one):
docker run --rm -p 8080:8080 \
-e API_KEY=local-key -e GCS_BUCKET=three-ws-avatar-reconstructions \
-e GOOGLE_APPLICATION_CREDENTIALS=/adc.json \
-v "$HOME/.config/gcloud/application_default_credentials.json:/adc.json:ro" \
model-video2motion:local
curl -s localhost:8080/infer -H 'Authorization: Bearer local-key' \
-H 'content-type: application/json' \
-d '{"video_url":"https://storage.googleapis.com/.../video.mp4","fps":12,"max_seconds":6}'
curl -s localhost:8080/tasks/<task_id> -H 'Authorization: Bearer local-key'
status: "done" carries the four artifact URLs. Capture is CPU-bound and runs
around 6x the clip's own length on the deployed 8-core instance (a 6 s clip at
12 fps, 72 frames, took 37.9 s end to end including fetch and upload, measured
against the live service 2026-08-11; the 30 s clip behind
MODE_ETA.video2motion = 120 in api/_providers/gcp.js lands in the same
ratio).