Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

4.3 KB
Raw Blame

Motion Swap — replace yourself in a video with your avatar

Page: /motion-swap · API: POST /api/motion-swap · Worker: workers/model-video2motion

Upload a video of yourself (a house tour, a product demo, a talking walk-through) and get it back with your 3D avatar performing your exact motion instead of you. You stay pixelated under the avatar; the motion, framing, and audio stay yours. Built for creators who want to publish video without publishing their face.

How to use it

  1. Open /motion-swap and drop in a video (MP4, MOV, or WebM, up to 256 MB; the first 90 seconds are processed). One person, full body in frame, tracks best.
  2. Wait for capture (about 2 minutes for a 30-second clip). The server tracks your pose in every frame and masks you out.
  3. Pick an avatar: a stock one, or paste the GLB URL of any rigged humanoid — one you generated, captured from a selfie, or rigged from a mesh. Swap avatars any time; the motion re-binds instantly.
  4. Press Export video to record the composite (with the original audio) to a downloadable .webm.

The "Pixelate me under the avatar" toggle controls the privacy treatment of the original subject beneath the avatar.

How it works

Three stages, split between one CPU worker and your browser:

  1. Track (server). MediaPipe PoseLandmarker follows 33 body landmarks through every frame, MediaPipe HandLandmarker follows 21 landmarks on each hand, and MediaPipe ImageSegmenter produces a per-frame person mask. All three models are Apache-2.0 and run without a GPU.
  2. Solve (server). An in-house solver (workers/model-video2motion/pose_solver.py) converts the landmark sequence into local joint rotations on the canonical Wolf3D skeleton, including wrist orientation taken from the palm and all 30 finger bones when your hands are in frame (a frame that loses hand tracking holds the previous handshape rather than snapping flat), and emits a standard three.js AnimationClip JSON — the same format the animation library serves, so the clip retargets onto any rigged avatar with the platform's existing engine.
  3. Composite (browser). One WebGL scene: the normalized footage on a fullscreen quad (subject pixelated under the mask in the shader), the avatar driven by the clip and pinned to the subject's on-screen position and size each frame via the capture's anchor track, and a MediaRecorder export of canvas + source audio.

Privacy properties: the motion clip contains no imagery (skeleton rotations only), and processing artifacts live under an unguessable job id.

API

POST /api/motion-swap   { action: "upload", content_type, size_bytes }
  → { upload_url, public_url, method, headers, expires_in }
                                           # presigned direct-to-storage PUT

POST /api/motion-swap   { video_url, fps?, max_seconds? }
  → 202 { job_id, status, eta_seconds }

GET  /api/motion-swap?job=<id>
  → { status, clip_url, meta_url, video_url, mask_url, frames, fps }

content_type must be video/mp4, video/quicktime, or video/webm, and size_bytes must be between 1 byte and 256 MB. Start the PUT within expires_in seconds of getting the URL: the signature is minted for exactly that window, and storage rejects a late upload with a bare 403.

clip_url is a three.js AnimationClip.toJSON() document you can use anywhere the platform accepts a clip. meta_url carries fps, dimensions, and per-frame anchors ({x, y, h, v}: hip centre in normalized image coordinates, subject height as a fraction of frame height, visibility).

Operations

Deploy and wiring: see workers/model-video2motion/README.md. The API routes through the GCP provider's video2motion mode; unset GCP_VIDEO2MOTION_URL degrades the endpoint to a clean 503 and the page shows its error state.

Related