Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
model-video2scene
Streaming video to 3D point-cloud reconstruction on Cloud Run GPU.
Wraps LingBot-Map (Apache-2.0), a feed-forward "Geometric Context Transformer" that reconstructs a dense world-space point cloud from a monocular video or image sequence using paged-KV-cache attention (FlashInfer).
The worker drives the upstream repo's own entry points (demo.load_images,
demo.load_model, demo.postprocess), unprojects the predicted depth with the
predicted camera to get world-space points, optionally removes sky pixels with
LingBot-Map's sky segmentation, fuses those points with their per-pixel RGB into
a single coloured cloud, writes a binary PLY, and uploads it to Cloud Storage. The
three.ws Scene Capture page (/capture) renders that PLY directly in a
WebGL point-cloud viewer (src/pointcloud-viewer.js).
API
POST /infer { video_url | images[], mode?, fps?, keyframe_interval?, ... } -> 202 { task_id }
GET /tasks/:id -> { status, result_gcs_url?, num_points?, frames?, bytes?, error? }
GET /health -> { ok, gpu_available, model_loaded, weights_present, max_frames }
Bearer auth on /infer and /tasks/:id (Authorization: Bearer $API_KEY). The
three.ws API (api/scene-capture.js) holds the
matching GCP_RECONSTRUCTION_KEY and points at GCP_VIDEO2SCENE_URL.
Request fields
| Field | Default | Meaning |
|---|---|---|
video_url |
none | Public https URL to an .mp4 / .mov / .webm. One of video_url or images[] is required. |
images[] |
none | Frames as https URLs or data:image URIs, in order. |
mode |
streaming |
streaming (one pass, KV cache) or windowed (independent windows, aligned across overlaps). Each mode is a different upstream model class, so switching modes reloads the checkpoint. |
fps |
8 |
Frames per second sampled from the video. |
keyframe_interval |
4 |
Every N-th frame after the scale frames keeps its KV in cache. Higher means less memory, less detail. |
num_scale_frames |
8 |
Leading frames processed bidirectionally to fix scale. Baked into the model build, so changing it also reloads the checkpoint. |
window_size / overlap_size |
128 / 16 |
Windowed mode only. |
mask_sky |
true |
Runs LingBot-Map sky segmentation and drops sky pixels. |
conf_percentile |
30 |
Drops the lowest N percent of surviving points by model confidence. |
max_points |
1500000 |
Hard cap after fusion (deterministic stride subsample). |
voxel_size |
0 |
World-unit voxel edge. Above 0, points sharing a cell are merged and colour-averaged before the cap, which de-duplicates re-observed surfaces. |
job_id |
none | Caller correlation id, echoed in logs. |
A finished task also reports frames_truncated and sky_points_removed.
Where the geometry comes from
The released checkpoints have no point head: GCTStream is built with
enable_point=False, so inference returns depth and depth_conf and never a
world_points map. The cloud is therefore built by unprojecting each depth map
with the camera the model predicted for that frame, which is what LingBot-Map's
own viewer and GLB export do. The unprojector documents its extrinsic as
world-to-camera while postprocess hands back camera-to-world, so the worker
inverts before unprojecting; feeding it the wrong one puts the cloud somewhere
other than in front of the cameras. If a future checkpoint does ship a point
head, the worker uses its world_points directly.
Frame budget
MAX_FRAMES (default 512) caps a single job. Fusion holds world points,
confidences and RGB for every pixel of every frame at once (about 12 MB per
518x518 frame), so 512 frames peaks near 10 GB of the instance's 32 GiB. At the
default fps: 8 that is 64 seconds of video; longer clips should lower fps
rather than raise the cap. An input longer than the cap is truncated and the
task reports frames_truncated: true. MAX_FRAME_NUM (default 1024) is the
separate 3D RoPE budget the model is built with and bounds MAX_FRAMES.
Model weights
Mounted read-only from the shared weights bucket at WEIGHTS_DIR/MODEL_FILE
(default /weights/lingbot-map/lingbot-map-long.pt), plus skyseg.onnx in the
same directory for sky masking. Both are already staged in
gs://three-ws-model-weights/lingbot-map/. To restage from scratch:
curl -L -o lingbot-map-long.pt https://huggingface.co/robbyant/lingbot-map/resolve/main/lingbot-map-long.pt
curl -L -o skyseg.onnx https://huggingface.co/JianyuanWang/skyseg/resolve/main/skyseg.onnx
gsutil -m cp lingbot-map-long.pt skyseg.onnx gs://three-ws-model-weights/lingbot-map/
The checkpoint is 4.6 GB and the sky model 168 MB. If skyseg.onnx is absent
from the mount the worker downloads it once into SKYSEG_CACHE_DIR
(/tmp/skyseg) so sky masking still works; the checkpoint has no such fallback
and /infer answers 503 until it is mounted.
Environment
| Variable | Default | Purpose |
|---|---|---|
API_KEY |
required | Shared bearer secret (Cloud Run secret avatar-reconstruction-key). |
GCS_BUCKET |
required | Bucket for output point clouds (three-ws-avatar-reconstructions). |
WEIGHTS_DIR |
/weights/lingbot-map |
Mounted weights directory. |
MODEL_FILE |
lingbot-map-long.pt |
Checkpoint filename. |
SKYSEG_FILE |
skyseg.onnx |
Sky-segmentation model filename. |
LINGBOT_DIR |
/opt/lingbot-map |
Upstream checkout on PYTHONPATH. |
MAX_CONCURRENT |
1 |
Parallel reconstructions per instance. |
MAX_FRAMES |
512 |
Frame cap per job. |
MAX_FRAME_NUM |
1024 |
3D RoPE frame budget used to build the model. |
USE_SDPA |
unset | Set to 1 to force PyTorch SDPA attention instead of FlashInfer. |
Build and test
# Fusion + PLY core path, no GPU and no checkpoint needed.
cd workers/model-video2scene && python -m unittest test_scene_fusion
# The image the deploy ships. The build reruns those tests and imports the
# service module, so a broken fusion path or import fails the build.
docker build -t model-video2scene:local workers/model-video2scene
Run it locally against real weights (CPU works but is slow; FlashInfer needs a GPU, so the worker falls back to SDPA attention automatically):
docker run --rm -p 8080:8080 \
-e API_KEY=local -e GCS_BUCKET=three-ws-avatar-reconstructions \
-v /path/to/weights:/weights/lingbot-map:ro \
model-video2scene:local
curl -s localhost:8080/health
The Cloud Storage client binds on first upload, not at startup, so the service
boots, loads the checkpoint and answers /health without credentials. Only the
upload at the end of a job needs them: add
-v $HOME/.config/gcloud:/root/.config/gcloud:ro after
gcloud auth application-default login to finish a job locally.
Deploy
Not currently deployed as its own Cloud Run service. Deploying it is one command, and it must be run by the repo owner:
gcloud builds submit --config workers/model-video2scene/cloudbuild.yaml . \
--region us-central1 --project aerial-vehicle-466722-p5 \
--substitutions=SHORT_SHA=manual$(date +%s)
The config builds, pushes, and deploys the model-video2scene service on an
NVIDIA L4 with the weights bucket mounted at /weights. Afterwards set
GCP_VIDEO2SCENE_URL to the service URL on the three.ws API service
(gcloud run services update three-ws-api --region us-central1 --update-env-vars GCP_VIDEO2SCENE_URL=...). GCP_RECONSTRUCTION_KEY is already
shared across the model workers. Until then /capture returns its designed
503 unconfigured state and renders its built-in sample cloud.
Hardware
NVIDIA L4 (CUDA 12.8), 8 vCPU / 32 GiB. Frames stay in host RAM and are moved to
the GPU one window at a time, so peak VRAM tracks the window, not the clip. For
long clips, lower fps, raise keyframe_interval, or use mode: "windowed"
with a smaller window_size.