Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

feat(forge): scene from a photo with SAM 3D Objects, built end to end and off until the gated weights are staged

nirholase96d1eae95b1
diff --git a/STRUCTURE.md b/STRUCTURE.md
index 81fd8efef..d7e135abe 100644
--- a/STRUCTURE.md
+++ b/STRUCTURE.md
@@ -221,6 +221,7 @@ packages, a generated standalone GitHub mirror). See
| Animation pipeline | [public/animations/](public/animations) + [scripts/build-animations.mjs](scripts/build-animations.mjs) | In-repo | Mixamo source FBX + GLB retargeting pipeline · see [docs/3d-asset-pipeline.md](docs/3d-asset-pipeline.md) |
| Procedural animation (runtime IK layers) | [src/procedural/](src/procedural) — analytic two-bone IK [two-bone-ik.js](src/procedural/two-bone-ik.js), gaze [look-at.js](src/procedural/look-at.js), terrain foot planting [foot-plant.js](src/procedural/foot-plant.js), drift-free re-application [pose-baseline.js](src/procedural/pose-baseline.js) | Live | Post-mixer layers that do what no baked clip can: the chest/neck/head chain turns toward a world point, and each foot lands on its own patch of uneven ground. Rig-agnostic via `canonicalNodeMapFromObject`; an unsupported rig reports `enabled === false` and no-ops. Wired into the corner companion ([walk-sdk/src/companion.js](walk-sdk/src/companion.js) `_updateGaze`), the walkaround world ([src/walk.js](src/walk.js) `rebuildFootPlant`), and agent avatars ([src/agent-avatar.js](src/agent-avatar.js) `_applyLookTarget` — which is what finally makes `setLookTarget()` and the `LOOK_AT` action move the rig). Ambient idle motion (breathing/blink/saccade/weight-shift) stays in [src/idle-animation.js](src/idle-animation.js). Docs: [docs/procedural-animation.md](docs/procedural-animation.md). Tests: [tests/procedural-anim.test.js](tests/procedural-anim.test.js), [tests/agent-avatar-look-target.test.js](tests/agent-avatar-look-target.test.js) |
| Self-hosted 3D GPU fleet (the engines behind the Forge) | [workers/model-trellis2/](workers/model-trellis2) (TRELLIS.2, MIT, the default image-to-3D lane `trellis2`: PBR GLB with alpha, `resolution` 512/1024/1536; built and tested, deploy owner-gated), [workers/model-pixal3d/](workers/model-pixal3d) (TencentARC Pixal3D, MIT, on TRELLIS.2's backbone: the `pixal3d` fidelity lane labeled "Match my photo", 1 to 4 pixel-aligned input photos with optional `view_labels` and one-photo `fill_other_sides` view synthesis via MV-Adapter (Apache-2.0); built and tested, deploy owner-gated), territory router [api/_lib/forge-territory.js](api/_lib/forge-territory.js) (EU, UK and KR never reach a Tencent-model lane; unknown country fails closed), [workers/model-trellis/](workers/model-trellis), [workers/model-hunyuan3d/](workers/model-hunyuan3d) (2.0 + 2.1 PBR + 2.1 RTX), [workers/model-triposg/](workers/model-triposg), [workers/modly/](workers/modly) (Modly v0.4.3 running its TripoSG and Hunyuan3D 2 Mini Turbo extensions as the untextured Forge lanes `modly` and `modly_hunyuan`, behind `MODEL_MODLY_URL`; weights staged, built and tested, deploy owner-gated), [workers/model-triposr/](workers/model-triposr), [workers/model-text2motion/](workers/model-text2motion), [workers/rig/](workers/rig) (auto-rigging, deployed as `model-rig`), [workers/rig-nonhuman/](workers/rig-nonhuman) (UniRig + SkinTokens, MIT, the non-humanoid rigging fallback `model-rig` reaches for when its own `mesh_classify.py` heuristic rules out a Mixamo-shaped humanoid, behind `RIG_NONHUMAN_URL`; built and tested, deploy owner-gated), [workers/model-anigen/](workers/model-anigen) (VAST-AI-Research AniGen, MIT, the rig-on-generation Forge lane `anigen` labeled "Rig on generate": predicts mesh, joint skeleton and skin weights together from a single reference image, so the GLB it returns needs no separate rerig call; opt-in only, never a default or failover target; built and tested, deploy owner-gated), [workers/model-partcrafter/](workers/model-partcrafter) (PartCrafter, MIT, LICENSE read 2026-10-11, the `partcrafter` Forge lane labeled "Editable parts": decodes a reference photo straight into N individually named part meshes in one diffusion-transformer pass; built and tested, deploy owner-gated), [workers/model-holopart/](workers/model-holopart) (HoloPart, MIT, LICENSE read 2026-10-11, the "Complete hidden parts" lane behind `/api/forge-holopart`: takes an already part-segmented mesh (from `workers/segment` or `model-partcrafter`) and fills in the occluded geometry of every part, closing each into a standalone shape; built and tested, deploy owner-gated), [workers/model-triposplat/](workers/model-triposplat) (VAST-AI-Research TripoSplat, MIT, LICENSE read 2026-10-12: the image-to-Gaussian-splat lane behind `POST /api/forge-splat`, output opens in the Splat Viewer; behind `MODEL_TRIPOSPLAT_URL`; built and tested, deploy owner-gated), [workers/avatar-reconstruction/](workers/avatar-reconstruction), CPU stages [workers/rembg/](workers/rembg) · [workers/remesh/](workers/remesh) · [workers/segment/](workers/segment) · [workers/stylize/](workers/stylize), talking-avatar video [workers/longcat/](workers/longcat) (LongCat-Video-Avatar-1.5, gates the `/create/video` lane via `LONGCAT_WORKER_URL`; the one fleet member that is not a Cloud Run service, because its ~45 GB resident weight set needs an 80 GB GPU, so it is built and tested but undeployed), streaming video→3D [workers/model-video2scene/](workers/model-video2scene), selfie-pipeline CPU orchestrator [workers/avatar-pipeline-controller/](workers/avatar-pipeline-controller) | Live · Cloud Run GPU (us-central1, plus us-east4 where trellis/hunyuan/triposr hold standbys and `model-text2motion` runs as its production lane) | Every generation engine the Forge dispatches to, each with its own `cloudbuild.yaml`. The L4 quota is **per region** and both us-central1 and us-east4 are granted 3, so a warm pin in one region only starves lanes in that same region (increase to 16 filed 2026-07-16, still reconciling); the Hunyuan3D-2.1 PBR realism lane also ships an RTX PRO 6000 build (`model-hunyuan3d-21-rtx`, quota granted 1000) as the warm primary. `model-text2motion` (MDM default + MoMask, both MIT, selectable per request via `model: "mdm" | "momask"`) is the text→motion lane behind `GCP_TEXT2MOTION_URL`, the `text_to_animation` MCP tool, and `POST /api/forge-motion`: prompt in, a retargetable three.js `AnimationClip` out, applied with the same engine (`src/animation-retarget.js`) as the curated animation library. It is the one GPU service whose production URL points at us-east4, and its `cloudbuild.yaml` deploys both regions from one image so the standby never lags. Ops map: [docs/ops/gcp-credits-plan.md](docs/ops/gcp-credits-plan.md) |
+| Scene from a photo (SAM 3D Objects) | [workers/model-sam3d-objects/](workers/model-sam3d-objects) (FastAPI worker, 32 GB+ GPU, pinned upstream), platform [api/forge-scene-objects.js](api/forge-scene-objects.js) + [api/_lib/forge-objects.js](api/_lib/forge-objects.js), provider mode `sam3d` in [api/_providers/gcp.js](api/_providers/gcp.js), AR Studio tab "From a photo" in [src/ar-studio.js](src/ar-studio.js), weights staging [scripts/stage-sam3d-objects.mjs](scripts/stage-sam3d-objects.mjs) (`npm run stage:sam3d-objects`) | Built, off until the owner stages the gated weights and deploys | One photo becomes one GLB with a named node per object, laid out like the picture. `503 backend_unconfigured` until `MODEL_SAM3D_URL` is set. Weights are gated by the SAM License on Hugging Face (manual approval); the endpoint refuses comprehensively sanctioned territories. Tests: [tests/forge-objects.test.js](tests/forge-objects.test.js), [tests/api/forge-scene-objects.test.js](tests/api/forge-scene-objects.test.js), [tests/api/gcp-sam3d-provider.test.js](tests/api/gcp-sam3d-provider.test.js), [tests/stage-sam3d-objects.test.js](tests/stage-sam3d-objects.test.js), worker `pytest`. |
| Self-hosted Granite Guardian (governance lane behind watsonx) | [workers/granite-guardian/](workers/granite-guardian/README.md) (Cloud Run `granite-guardian`, europe-west4, vLLM on one L4), client lane in [api/_lib/granite-guardian.js](api/_lib/granite-guardian.js) | Live · Cloud Run GPU (europe-west4) | IBM's open-weight `ibm-granite/granite-guardian-3.3-8b` served by the stock vLLM OpenAI image, so `/api/guardian/assess`, the identity integrity check, and the Granite Twin/Oracle/Attest/IRL screens return real Granite Guardian verdicts on a deployment with no IBM Cloud account. watsonx.ai stays the lead lane when configured; every verdict names the `model` and `provider` that produced it. Placed in europe-west4 because that region's L4 grant is otherwise unused, so its warm pin never starves the 3D lanes. Tests: [tests/api-guardian.test.js](tests/api-guardian.test.js), [tests/api/identity-integrity.test.js](tests/api/identity-integrity.test.js) |
| Reconstruct completion backstop (abandoned selfie jobs) | [api/cron/reconstruct-sweep.js](api/cron/reconstruct-sweep.js), every 5 min via `vercel.json` crons → Cloud Scheduler | Live 2026-07-25 | A reconstruct job used to be advanced **only by the user's browser**: `/create/selfie` polls `regenerate-status`, and each poll runs the finalize stages. Close the tab and the worker still finished the GLB and stored it durably, but nothing ever collected it — 26 jobs were stranded that way, oldest from 2026-05-31, invisible to `db-retention` (which prunes only terminal rows). This sweep is the server-side driver: it picks up reconstruct jobs quiet >3 min and runs **the same shared stages** in [reconstruct-finalize.js](api/_lib/reconstruct-finalize.js), so an abandoned job still lands in the user's library. 30-day rescue window (the worker's GCS blobs + Firestore job state are durable, so a weeks-old job is still deliverable); past that, or when a job's provider is no longer pollable from platform credentials, rows fail out with an honest reason so the open set can't grow unbounded. BYOK (meshy/tripo) jobs stay browser-driven — a cron has no user context to resolve their keys. Sibling of [auto-rig-sweep.js](api/cron/auto-rig-sweep.js), which plays the same role for the rerig lane. Docs: [docs/avatar-reconstruction.md](docs/avatar-reconstruction.md#who-finishes-the-job) |
| Avatar fidelity evaluation (selfie → avatar, scored) | [workers/avatar-reconstruction/eval/](workers/avatar-reconstruction/eval) | In-repo · gates every reconstruction change | **Identity Shape Error (ISE)** — does the reconstructed avatar have *this person's* face shape? Texture-blind, pose- and scale-invariant, dimensionless: MediaPipe's 468 landmarks and the avatar's corresponding head vertices are both Umeyama-aligned onto the neutral canonical face over a stable identity subset (no eyes/lips, so a smile is not scored as identity), then compared. A pipeline cannot score better by resizing or tilting the head — only by matching shape, a property pinned in [eval/test_identity_eval.py](workers/avatar-reconstruction/eval/test_identity_eval.py) alongside monotonicity. [`run_eval.py`](workers/avatar-reconstruction/eval/run_eval.py) scores a whole reference set against a live worker and diffs two revisions, surfacing per-face regressions a mean would hide; [`tune_morph.py`](workers/avatar-reconstruction/eval/tune_morph.py) sweeps the morph parameters against ISE offline; [`robustness.py`](workers/avatar-reconstruction/eval/robustness.py) measures the opposing axis (does a blurred / turned / occluded / dim / low-res photo of the same person still yield the same head?), and the shipped `max_displacement_frac` is the knee of those two curves. The 40-face reference set is **synthesised** by [`make_refs.py`](workers/avatar-reconstruction/eval/make_refs.py) over a deterministic grid of the platform's diversity matrix — no real biometrics, no licence terms, rebuildable by any agent — gitignored and mirrored to `gs://three-ws-avatar-reconstructions/eval-refs/`. Docs: [eval/README.md](workers/avatar-reconstruction/eval/README.md), program plan [docs/avatar-fidelity-program.md](docs/avatar-fidelity-program.md) |
diff --git a/api/_lib/forge-objects.js b/api/_lib/forge-objects.js
new file mode 100644
index 000000000..ef10bbca6
--- /dev/null
+++ b/api/_lib/forge-objects.js
@@ -0,0 +1,126 @@
+/**
+ * Helpers for the "scene from a photo" lane (api/forge-scene-objects.js).
+ *
+ * The SAM 3D Objects worker reconstructs the objects a caller names in a photo.
+ * Two pure functions live here so they can be tested without a network:
+ *
+ * - normalizeDetectedBoxes: turn a vision model's object list into the clean,
+ * bounded, de-duplicated boxes the worker accepts.
+ * - isSanctionedTerritory: the SAM License refuses comprehensively sanctioned
+ * jurisdictions, so those requests never reach the worker.
+ *
+ * And one that talks to the vision chain: detectObjectBoxes.
+ */
+
+import { describeImageJson } from './vision.js';
+
+export const MAX_SCENE_OBJECTS = 12;
+export const DEFAULT_SCENE_OBJECTS = 8;
+export const MIN_BOX_SIDE = 0.04;
+export const MIN_BOX_AREA = 0.002;
+const DUPLICATE_IOU = 0.7;
+
+/**
+ * Comprehensively sanctioned jurisdictions (ISO 3166-1 alpha-2) the SAM License
+ * and Hugging Face both refuse. Only a country we positively resolved as one of
+ * these is blocked: an unknown country is not, because the guard exists to honour
+ * the licence, not to turn away every visitor behind a proxy.
+ */
+export const SANCTIONED_COUNTRIES = Object.freeze(['CU', 'IR', 'KP', 'SY']);
+
+export function isSanctionedTerritory(country) {
+ return typeof country === 'string' && SANCTIONED_COUNTRIES.includes(country.toUpperCase());
+}
+
+function iou(a, b) {
+ const ix = Math.max(0, Math.min(a[2], b[2]) - Math.max(a[0], b[0]));
+ const iy = Math.max(0, Math.min(a[3], b[3]) - Math.max(a[1], b[1]));
+ const inter = ix * iy;
+ const union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter;
+ return union > 0 ? inter / union : 0;
+}
+
+/**
+ * Accepts the vision model's `{ objects: [{ label, box_2d: [ymin, xmin, ymax, xmax] }] }`
+ * (0..1000, the convention Gemini is trained on) and returns
+ * `[{ label, box: [x0, y0, x1, y1] }]` in 0..1 fractions of the frame, largest
+ * first, with slivers, out-of-order boxes and near-duplicates dropped.
+ */
+export function normalizeDetectedBoxes(json, maxObjects = DEFAULT_SCENE_OBJECTS) {
+ const list = Array.isArray(json) ? json : Array.isArray(json?.objects) ? json.objects : [];
+ const cap = Math.min(Math.max(1, Math.floor(maxObjects) || DEFAULT_SCENE_OBJECTS), MAX_SCENE_OBJECTS);
+ const candidates = [];
+ for (const item of list) {
+ const raw = item?.box_2d ?? item?.box;
+ if (!Array.isArray(raw) || raw.length !== 4) continue;
+ const nums = raw.map(Number);
+ if (!nums.every(Number.isFinite)) continue;
+ const [ymin, xmin, ymax, xmax] = nums.map((v) => Math.min(Math.max(v / 1000, 0), 1));
+ if (!(xmin < xmax && ymin < ymax)) continue;
+ if (xmax - xmin < MIN_BOX_SIDE || ymax - ymin < MIN_BOX_SIDE) continue;
+ if ((xmax - xmin) * (ymax - ymin) < MIN_BOX_AREA) continue;
+ const label = typeof item.label === 'string' ? item.label.replace(/\s+/g, ' ').trim().slice(0, 60) : '';
+ candidates.push({ label, box: [xmin, ymin, xmax, ymax].map((v) => Math.round(v * 10000) / 10000) });
+ }
+ candidates.sort(
+ (a, b) => (b.box[2] - b.box[0]) * (b.box[3] - b.box[1]) - (a.box[2] - a.box[0]) * (a.box[3] - a.box[1]),
+ );
+ const kept = [];
+ for (const c of candidates) {
+ if (kept.some((k) => iou(k.box, c.box) > DUPLICATE_IOU)) continue;
+ kept.push(c);
+ if (kept.length >= cap) break;
+ }
+ return kept;
+}
+
+/**
+ * Strictly validate caller-supplied boxes (`[x0, y0, x1, y1]`, fractions of the
+ * frame). Unlike detected boxes these are never clamped or reordered: a caller
+ * who sent a bad box is told which one, not silently given a different object.
+ * Returns `{ boxes }` or `{ error }`.
+ */
+export function parseUserBoxes(raw, maxObjects = MAX_SCENE_OBJECTS) {
+ if (!Array.isArray(raw) || raw.length === 0) return { error: 'boxes must be a non-empty list.' };
+ if (raw.length > maxObjects) return { error: `${raw.length} boxes sent but max_objects is ${maxObjects}.` };
+ const boxes = [];
+ for (let i = 0; i < raw.length; i++) {
+ const b = raw[i];
+ if (!Array.isArray(b) || b.length !== 4 || !b.every((v) => typeof v === 'number' && Number.isFinite(v))) {
+ return { error: `boxes[${i}] must be [x0, y0, x1, y1] as four numbers.` };
+ }
+ const [x0, y0, x1, y1] = b;
+ if (!(x0 >= 0 && y0 >= 0 && x1 <= 1 && y1 <= 1 && x0 < x1 && y0 < y1)) {
+ return { error: `boxes[${i}] must satisfy 0 <= x0 < x1 <= 1 and 0 <= y0 < y1 <= 1 (fractions of the frame).` };
+ }
+ if (x1 - x0 < MIN_BOX_SIDE || y1 - y0 < MIN_BOX_SIDE || (x1 - x0) * (y1 - y0) < MIN_BOX_AREA) {
+ return { error: `boxes[${i}] is too small to reconstruct; it must cover at least 4% of the frame width and height.` };
+ }
+ for (let j = 0; j < i; j++) {
+ if (iou(boxes[j], b) > DUPLICATE_IOU) {
+ return { error: `boxes[${i}] overlaps boxes[${j}] by more than 70%; they name the same object.` };
+ }
+ }
+ boxes.push([x0, y0, x1, y1]);
+ }
+ return { boxes };
+}
+
+const DETECT_PROMPT = (max) =>
+ `List the distinct, separable physical objects in this photo, up to ${max}, most prominent first. ` +
+ 'Skip walls, floors, ceilings, sky, ground, and anything cut off by more than half the frame edge. ' +
+ 'Reply ONLY with JSON: {"objects":[{"label":"short name","box_2d":[ymin,xmin,ymax,xmax]}]} ' +
+ 'where box_2d is the tight bounding box in whole numbers from 0 to 1000 (fractions of the image height ' +
+ 'and width times 1000, origin top-left).';
+
+/** Ask the vision chain to box the objects in a photo. Throws on a vision outage. */
+export async function detectObjectBoxes(imageUrl, maxObjects = DEFAULT_SCENE_OBJECTS) {
+ const result = await describeImageJson({
+ prompt: DETECT_PROMPT(maxObjects),
+ imageUrl,
+ maxTokens: 1024,
+ timeoutMs: 25_000,
+ deadlineMs: 45_000,
+ });
+ return normalizeDetectedBoxes(result.json, maxObjects);
+}
diff --git a/api/_providers/gcp.js b/api/_providers/gcp.js
index 17f1af862..1a274c109 100644
--- a/api/_providers/gcp.js
+++ b/api/_providers/gcp.js
@@ -33,6 +33,10 @@
// triposplat → MODEL_TRIPOSPLAT_URL (workers/model-triposplat, MIT)
// One image → 3D Gaussian splat (.splat + .ply). Standard
// /infer + /tasks/:id shape; the result is a splat, not a GLB.
+// sam3d → MODEL_SAM3D_URL (workers/model-sam3d-objects, SAM License)
+// One photo → one GLB with a named node per object, laid out
+// like the photo. Needs a 32 GB+ GPU. Standard /infer +
+// /tasks/:id shape.
// remesh → GCP_REMESH_URL (workers/remesh)
// stylize → GCP_STYLIZE_URL (workers/stylize)
// retex → GCP_TEXTURE_URL (workers/texture)
@@ -139,6 +143,9 @@ function serviceUrlForMode(mode) {
// output of `segment`), preserving part node names. Same
// /infer + /tasks/:id task shape.
return readEnv('MODEL_HOLOPART_URL');
+ case 'sam3d':
+ // Self-hosted SAM 3D Objects worker (workers/model-sam3d-objects).
+ return readEnv('MODEL_SAM3D_URL');
case 'triposplat':
// Self-hosted TripoSplat worker (workers/model-triposplat): image to
// Gaussian splat. Same /infer + /tasks/:id task shape.
@@ -309,6 +316,22 @@ function buildWorkerRequest(request) {
};
}
+ if (mode === 'sam3d') {
+ // Self-hosted SAM 3D Objects: one photo plus either per-object masks or
+ // boxes (never both). The worker rebuilds each object and lays them out
+ // like the photo in a single scene GLB.
+ const body = { image_url: sourceUrl };
+ if (Array.isArray(params?.maskUrls) && params.maskUrls.length) body.mask_urls = params.maskUrls;
+ else if (Array.isArray(params?.boxes) && params.boxes.length) body.boxes = params.boxes;
+ if (Number.isFinite(Number(params?.maxObjects))) body.max_objects = Math.round(Number(params.maxObjects));
+ if (Number.isFinite(Number(params?.seed))) body.seed = Math.floor(Number(params.seed));
+ return {
+ path: '/infer',
+ resultKey: 'result_gcs_url',
+ body,
+ };
+ }
+
if (mode === 'triposplat') {
// Self-hosted TripoSplat (workers/model-triposplat): the worker
// conditions on the first reference image only.
@@ -633,6 +656,8 @@ const MODE_ETA = {
holopart: 100,
// Single-pass flow-matching denoise plus a decoder replay on the L4.
triposplat: 45,
+ // Two diffusion stages per object, several objects, on a 96 GB-class GPU.
+ sam3d: 150,
// TripoSG's 20 to 50 flow steps plus repair and decimate on the L4.
modly: 60,
// Hunyuan3D 2 Mini Turbo is a few-step distilled sampler.
@@ -743,7 +768,7 @@ export function createRegenProvider({ reconstructUrl } = {}) {
// on the primary view only (report 1 so a multi-view request is never
// silently claimed as fused); other modes are single-source.
const viewsUsed =
- mode === 'hunyuan' || mode === 'triposplat' || mode === 'trellis2' || mode === 'modly' || mode === 'modly_hunyuan' || mode === 'anigen' || mode === 'partcrafter'
+ mode === 'hunyuan' || mode === 'triposplat' || mode === 'trellis2' || mode === 'modly' || mode === 'modly_hunyuan' || mode === 'anigen' || mode === 'partcrafter' || mode === 'sam3d'
? 1
: (mode === 'reconstruct' || mode === 'trellis' || mode === 'pixal3d') && Array.isArray(workerReq.body.images)
? workerReq.body.images.length
@@ -825,6 +850,12 @@ export function createRegenProvider({ reconstructUrl } = {}) {
result.resultSplatUrl = url;
if (data.ply_url) result.plyUrl = data.ply_url;
if (typeof data.num_gaussians === 'number') result.numGaussians = data.num_gaussians;
+ } else if (mode === 'sam3d') {
+ // A laid-out scene GLB plus the per-object manifest.
+ result.resultGlbUrl = url;
+ if (Array.isArray(data.objects)) result.objects = data.objects;
+ if (typeof data.object_count === 'number') result.objectCount = data.object_count;
+ if (Array.isArray(data.skipped) && data.skipped.length) result.skipped = data.skipped;
} else if (mode === 'video2scene') {
// The result is a .ply point cloud, not a GLB mesh. Surface the
// reconstruction telemetry so the client can show point/frame counts.
diff --git a/api/forge-scene-objects.js b/api/forge-scene-objects.js
new file mode 100644
index 000000000..fdfe2109a
--- /dev/null
+++ b/api/forge-scene-objects.js
@@ -0,0 +1,206 @@
+/**
+ * /api/forge-scene-objects: turn one photo into a 3D scene of the objects in it
+ * (SAM 3D Objects, workers/model-sam3d-objects).
+ *
+ * POST /api/forge-scene-objects
+ * { image_url: string,
+ * boxes?: [[x0, y0, x1, y1], ...] fractions of the frame, origin top-left
+ * mask_urls?: [string, ...] one https PNG mask per object (white = object)
+ * detect?: true find the objects automatically (default when
+ * neither boxes nor mask_urls is sent)
+ * max_objects?: 1..12, seed?: int }
+ * → 202 { job_id, status, eta_seconds, source: 'boxes'|'masks'|'detected', objects: [{ label, box }] }
+ *
+ * GET /api/forge-scene-objects?job=<id>
+ * → { job_id, status, result_url?, objects?, object_count?, skipped?, error? }
+ *
+ * The scene GLB has one named node per object (object_01, ...) laid out like the
+ * photo, and `objects` carries each node's position and size.
+ *
+ * Until MODEL_SAM3D_URL points at a deployed worker this answers 503
+ * backend_unconfigured (the worker needs a 32 GB+ GPU). The SAM License refuses
+ * comprehensively sanctioned jurisdictions, so those requests are turned away
+ * before any work is queued.
+ */
+
+import { cors, json, method, readJson, wrap, rateLimited } from './_lib/http.js';
+import { limits, clientIp } from './_lib/rate-limit.js';
+import { assertPublicHttpsUrl, SsrfError } from './_lib/ssrf.js';
+import { createRegenProvider } from './_providers/gcp.js';
+import { requestTerritory } from './_lib/forge-territory.js';
+import {
+ DEFAULT_SCENE_OBJECTS,
+ MAX_SCENE_OBJECTS,
+ detectObjectBoxes,
+ isSanctionedTerritory,
+ parseUserBoxes,
+} from './_lib/forge-objects.js';
+
+const JOB_ID_RE = /^[A-Za-z0-9_-]{20,600}$/;
+const SEED_MAX = 4_294_967_295;
+
+function unconfigured(res) {
+ return json(res, 503, {
+ error: 'backend_unconfigured',
+ message:
+ 'Scene-from-a-photo is not switched on yet. It needs the SAM 3D Objects worker (a 32 GB+ GPU): ' +
+ 'deploy workers/model-sam3d-objects and set MODEL_SAM3D_URL and GCP_RECONSTRUCTION_KEY.',
+ });
+}
+
+async function publicUrl(value, field, res) {
+ try {
+ return await assertPublicHttpsUrl(typeof value === 'string' ? value.trim() : '');
+ } catch (err) {
+ json(res, 400, {
+ error: `invalid_${field}`,
+ message: err instanceof SsrfError ? `${field} rejected: ${err.message}` : `${field} must be a public https URL.`,
+ });
+ return null;
+ }
+}
+
+async function startJob(req, res) {
+ if (isSanctionedTerritory(requestTerritory(req))) {
+ return json(res, 451, {
+ error: 'unavailable_in_region',
+ message: 'This model is licensed to exclude comprehensively sanctioned jurisdictions, so it is not available from your region.',
+ });
+ }
+
+ const rl = await limits.mcp3dGenerate(clientIp(req));
+ if (!rl.success) return rateLimited(res, rl);
+
+ const body = await readJson(req, 16_000).catch(() => null);
+ if (!body || typeof body !== 'object') {
+ return json(res, 400, { error: 'invalid_body', message: 'Send a JSON body with image_url.' });
+ }
+
+ let provider;
+ try {
+ provider = createRegenProvider();
+ if (!provider.supportsMode('sam3d')) return unconfigured(res);
+ } catch {
+ return unconfigured(res);
+ }
+
+ const imageUrl = await publicUrl(body.image_url, 'image_url', res);
+ if (!imageUrl) return undefined;
+
+ let maxObjects = DEFAULT_SCENE_OBJECTS;
+ if (body.max_objects !== undefined && body.max_objects !== null) {
+ const n = Number(body.max_objects);
+ if (!Number.isInteger(n) || n < 1 || n > MAX_SCENE_OBJECTS) {
+ return json(res, 400, { error: 'invalid_max_objects', message: `max_objects must be an integer from 1 to ${MAX_SCENE_OBJECTS}.` });
+ }
+ maxObjects = n;
+ }
+
+ let seed;
+ if (body.seed !== undefined && body.seed !== null) {
+ const n = Number(body.seed);
+ if (!Number.isInteger(n) || n < 0) {
+ return json(res, 400, { error: 'invalid_seed', message: 'seed must be a non-negative integer.' });
+ }
+ seed = Math.min(n, SEED_MAX);
+ }
+
+ const hasMasks = Array.isArray(body.mask_urls) && body.mask_urls.length > 0;
+ const hasBoxes = Array.isArray(body.boxes) && body.boxes.length > 0;
+ if (hasMasks && hasBoxes) {
+ return json(res, 400, { error: 'ambiguous_objects', message: 'Send mask_urls or boxes, not both.' });
+ }
+
+ const params = { seed, maxObjects };
+ let source;
+ let objects;
+ if (hasMasks) {
+ if (body.mask_urls.length > maxObjects) {
+ return json(res, 400, { error: 'too_many_objects', message: `${body.mask_urls.length} masks sent but max_objects is ${maxObjects}.` });
+ }
+ const masks = [];
+ for (const raw of body.mask_urls) {
+ const url = await publicUrl(raw, 'mask_urls', res);
+ if (!url) return undefined;
+ masks.push(url);
+ }
+ params.maskUrls = masks;
+ source = 'masks';
+ objects = masks.map((_, i) => ({ label: `object ${i + 1}`, box: null }));
+ } else {
+ let boxes;
+ if (hasBoxes) {
+ const parsed = parseUserBoxes(body.boxes, maxObjects);
+ if (parsed.error) return json(res, 400, { error: 'invalid_boxes', message: parsed.error });
+ boxes = parsed.boxes.map((box, i) => ({ label: `object ${i + 1}`, box }));
+ source = 'boxes';
+ } else {
+ try {
+ boxes = await detectObjectBoxes(imageUrl, maxObjects);
+ } catch (err) {
+ return json(res, 502, {
+ error: 'detect_failed',
+ message: `Could not find objects in the photo automatically (${err?.message || 'vision unavailable'}). Send boxes or mask_urls to choose them yourself.`,
+ });
+ }
+ if (!boxes.length) {
+ return json(res, 422, {
+ error: 'no_objects_found',
+ message: 'No separate objects were found in the photo. Try a photo with clearer, whole objects, or send boxes to choose them yourself.',
+ });
+ }
+ source = 'detected';
+ }
+ params.boxes = boxes.map((b) => b.box);
+ objects = boxes;
+ }
+
+ try {
+ const job = await provider.submit({ mode: 'sam3d', sourceUrl: imageUrl, params });
+ return json(res, 202, { job_id: job.extJobId, status: 'queued', eta_seconds: job.eta, source, objects });
+ } catch (err) {
+ return json(res, 502, { error: 'sam3d_failed', message: err?.message || 'Scene reconstruction could not start.' });
+ }
+}
+
+async function pollJob(req, res, jobId) {
+ if (!JOB_ID_RE.test(jobId)) return json(res, 400, { error: 'invalid_job', message: 'Malformed job id.' });
+
+ const rl = await limits.mcp3dStatus(clientIp(req));
+ if (!rl.success) return rateLimited(res, rl);
+
+ let provider;
+ try {
+ provider = createRegenProvider();
+ } catch {
+ return unconfigured(res);
+ }
+
+ let result;
+ try {
+ result = await provider.status(jobId);
+ } catch (err) {
+ return json(res, 502, { error: 'sam3d_status_failed', message: err?.message || 'Could not read the scene job.' });
+ }
+ return json(res, 200, {
+ job_id: jobId,
+ status: result.status,
+ result_url: result.resultGlbUrl || null,
+ objects: result.objects || null,
+ object_count: result.objectCount ?? null,
+ skipped: result.skipped || null,
+ error: result.error || null,
+ });
+}
+
+export default wrap(async (req, res) => {
+ if (cors(req, res, { methods: 'GET,POST,OPTIONS' })) return;
+ if (!method(req, res, ['GET', 'POST'])) return;
+
+ if (req.method === 'POST') return startJob(req, res);
+
+ const url = new URL(req.url, 'http://localhost');
+ const jobId = (url.searchParams.get('job') || '').trim();
+ if (!jobId) return json(res, 400, { error: 'missing_job', message: 'Pass ?job=<id> to poll.' });
+ return pollJob(req, res, jobId);
+});
diff --git a/data/changelog.json b/data/changelog.json
index dc5a400eb..55dcc8184 100644
--- a/data/changelog.json
+++ b/data/changelog.json
@@ -1,6 +1,13 @@
{
"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security, desktop (desktop entries become the desktop app's release notes). `link` is optional and must be a live three.ws path.",
"entries": [
+ {
+ "date": "2026-10-24",
+ "title": "Scene from a photo, built and waiting on weights",
+ "summary": "POST /api/forge-scene-objects rebuilds the objects in one photo as a laid-out 3D scene with SAM 3D Objects, and AR Studio gains a From a photo tab that drops the result into the room. The 32 GB+ GPU worker, its Cloud Build config and a one-command weights staging script are in the repo. It answers backend_unconfigured until the gated weights are staged and the worker is deployed.",
+ "tags": ["feature", "docs"],
+ "link": "/ar/studio"
+ },
{
"date": "2026-10-24",
"title": "Pay for a model request with a wallet, no account needed",
diff --git a/data/open-source-credits.json b/data/open-source-credits.json
index ca2a7ae05..db7ef09c2 100644
--- a/data/open-source-credits.json
+++ b/data/open-source-credits.json
@@ -395,6 +395,18 @@
"powers": "workers/rembg: background removal ahead of image-to-3D generation.",
"org": "danielgatis, ZhengPeng7"
},
+ {
+ "id": "model:sam3d-objects",
+ "kind": "model",
+ "name": "SAM 3D Objects",
+ "area": "generation",
+ "license": "SAM License (Meta, last updated November 19, 2025; use allowed subject to Trade Controls, gated weights)",
+ "repo": "https://github.com/facebookresearch/sam-3d-objects",
+ "funding": null,
+ "version": null,
+ "powers": "workers/model-sam3d-objects: rebuilds the objects in one photo as a laid-out 3D scene (Forge and AR Studio \"From a photo\"), a 32 GB+ GPU lane that stays off until the owner stages the weights.",
+ "org": "Meta"
+ },
{
"id": "model:hunyuan3d",
"kind": "model",
diff --git a/docs/api-reference.md b/docs/api-reference.md
index 0dfff959e..592a7e7f0 100644
--- a/docs/api-reference.md
+++ b/docs/api-reference.md
@@ -10658,6 +10658,8 @@ a delivery from the log or the `/dashboard/developers` page. The signing secret
| `GET /api/forge-quote?path=text&tier=standard` | none | What a generation would cost. Optional `backend`, `images`, `own_key`, `gameready`; `table=1` returns every tier. No spend, `cache-control: private, max-age=30`. |
| `GET /api/forge-keep?id=<creation>` | none | Whether the creation is kept, and whether pay-to-keep is enabled. |
| `POST /api/forge-keep` | optional $THREE wallet or payment | Release a creation for download. With `FORGE_PAY_TO_KEEP` off it always returns `kept: true`. Non-holders get `402 three_hold_required` with the `pay_per_use` action. A reused payment returns `409 payment_already_used`. |
+| `POST /api/forge-scene-objects` | none | One photo to a laid-out 3D scene (SAM 3D Objects). Body `{image_url, boxes? \| mask_urls?, max_objects?, seed?}`; auto-detects objects when neither is sent. `202 {job_id}`, `451` in sanctioned territories, `503 backend_unconfigured` until the worker is deployed. |
+| `GET /api/forge-scene-objects?job=<id>` | none | Job status with `result_url`, `objects`, `object_count`, `skipped`. |
Details and the enable command: [forge.md](./forge.md#cost-preview-and-pay-to-keep).
diff --git a/docs/forge.md b/docs/forge.md
index f00fea939..9e7a44416 100644
--- a/docs/forge.md
+++ b/docs/forge.md
@@ -649,3 +649,30 @@ dashboard at [/quality-bench](https://three.ws/quality-bench) (internal).
- [Diorama](./diorama.md) forges a whole scene of Forge objects into one explorable world.
- The 3D generation architecture: [3D pipeline](./3d-pipeline.md), [3D asset pipeline](./3d-asset-pipeline.md), [3D API](./3d-api.md).
- Pages: [/create](https://three.ws/create) (the avatar creator), [/scene](https://three.ws/scene) (Scene Studio), [/restyle](https://three.ws/restyle) (re-skin a GLB), [/gallery](https://three.ws/gallery).
+
+### Scene from a photo (SAM 3D Objects)
+
+`/api/forge-scene-objects` rebuilds every object in one photo as a single GLB with
+a named node per object (`object_01`, ...), laid out the way the objects sit in
+the picture (relative layout, not metric). It runs
+[SAM 3D Objects](https://github.com/facebookresearch/sam-3d-objects) on
+`workers/model-sam3d-objects` and is also the AR Studio tray tab "From a photo".
+
+```bash
+curl -sX POST https://three.ws/api/forge-scene-objects \
+ -H 'content-type: application/json' \
+ -d '{"image_url":"https://.../desk.jpg","max_objects":8}'
+# → 202 { "job_id": "...", "status": "queued", "eta_seconds": 150,
+# "source": "detected", "objects": [{ "label": "mug", "box": [...] }] }
+
+curl -s 'https://three.ws/api/forge-scene-objects?job=<job_id>'
+# → { "status": "done", "result_url": "https://.../scene.glb", "object_count": 3, "skipped": [] }
+```
+
+Choose the objects with `boxes` (`[x0,y0,x1,y1]` fractions, top-left origin),
+`mask_urls` (one https PNG per object), or neither and the vision chain finds
+them. Requests from comprehensively sanctioned territories (the SAM License's
+Trade Controls clause) get `451` before any work is queued. Returns `503
+backend_unconfigured` until `MODEL_SAM3D_URL` and `GCP_RECONSTRUCTION_KEY` point
+at the deployed worker; staging the gated weights is `npm run stage:sam3d-objects`
+(see the worker README).
diff --git a/package.json b/package.json
index e71d64c34..0749d4bba 100644
--- a/package.json
+++ b/package.json
@@ -582,6 +582,7 @@
"check:mcp-catalog": "node scripts/check-mcp-catalog.mjs",
"vet:mcp-catalog": "node scripts/vet-mcp-integrations.mjs",
"forge:pay-to-keep": "node scripts/forge-pay-to-keep.mjs",
+ "stage:sam3d-objects": "node scripts/stage-sam3d-objects.mjs",
"rescan:skills": "node --env-file=.env.local scripts/rescan-marketplace-skills.mjs"
},
"dependencies": {
diff --git a/pages/ar-studio.html b/pages/ar-studio.html
index 2fe91498c..8e0633370 100644
--- a/pages/ar-studio.html
+++ b/pages/ar-studio.html
@@ -329,6 +329,8 @@
border-radius: 10px; padding: 10px 12px;
}
.ars-link-row input:focus { outline: none; border-color: rgba(139, 124, 248, 0.6); }
+ .ars-photo-progress { display: flex; align-items: center; gap: 8px; font-size: 13px; color: var(--ink-faint); }
+ .ars-photo-error { margin: 0; font-size: 12.5px; color: var(--warn, #f59e0b); line-height: 1.5; }
.ars-link-hint { margin: 0; font-size: 12.5px; color: var(--ink-faint); line-height: 1.5; }
.ars-objects-head { display: flex; flex-direction: column; gap: 7px; padding: 2px 2px 10px; }
.ars-objects-more { padding: 10px 0 2px; display: flex; justify-content: center; }
@@ -462,6 +464,7 @@
<button class="ars-tab" type="button" role="tab" id="ars-tab-community" data-tab="community" aria-selected="false" aria-controls="ars-tray-body" data-i18n="ar_studio.community">Community</button>
<button class="ars-tab" type="button" role="tab" id="ars-tab-objects" data-tab="objects" aria-selected="false" aria-controls="ars-tray-body" data-i18n="ar_studio.objects">Objects</button>
<button class="ars-tab" type="button" role="tab" id="ars-tab-link" data-tab="link" aria-selected="false" aria-controls="ars-tray-body" data-i18n="ar_studio.glb_link">GLB link</button>
+ <button class="ars-tab" type="button" role="tab" id="ars-tab-photo" data-tab="photo" aria-selected="false" aria-controls="ars-tray-body">From a photo</button>
</div>
<div class="ars-tray-body" id="ars-tray-body" role="tabpanel" tabindex="0" aria-labelledby="ars-tab-recent"></div>
</div>
diff --git a/scripts/lib/open-source-credits.mjs b/scripts/lib/open-source-credits.mjs
index 10975a211..5979ab1ed 100644
--- a/scripts/lib/open-source-credits.mjs
+++ b/scripts/lib/open-source-credits.mjs
@@ -202,6 +202,15 @@ export const WORKER_MODELS = [
area: 'generation',
powers: "workers/model-trellis2: the default image-to-3D PBR engine behind Forge's primary lane.",
},
+ {
+ id: 'model:sam3d-objects',
+ name: 'SAM 3D Objects',
+ org: 'Meta',
+ license: 'SAM License (Meta, last updated November 19, 2025; use allowed subject to Trade Controls, gated weights)',
+ repo: 'https://github.com/facebookresearch/sam-3d-objects',
+ area: 'generation',
+ powers: 'workers/model-sam3d-objects: rebuilds the objects in one photo as a laid-out 3D scene (Forge and AR Studio "From a photo"), a 32 GB+ GPU lane that stays off until the owner stages the weights.',
+ },
{
id: 'model:hunyuan3d',
name: 'Tencent Hunyuan3D 2.1',
diff --git a/scripts/stage-sam3d-objects.mjs b/scripts/stage-sam3d-objects.mjs
new file mode 100644
index 000000000..2b9cf2d40
--- /dev/null
+++ b/scripts/stage-sam3d-objects.mjs
@@ -0,0 +1,193 @@
+#!/usr/bin/env node
+// Stage the SAM 3D Objects checkpoints into the model-weights bucket, after the
+// account owner has accepted the SAM License. One command:
+//
+// HF_TOKEN=hf_... npm run stage:sam3d-objects -- --accept-sam-license
+//
+// What it does, in order:
+// 1. Refuses to run without --accept-sam-license. That flag is the owner
+// saying "I have read the SAM License and I accept it for this account".
+// It is never implied, never defaulted, and never set by CI.
+// 2. Checks the Hugging Face token can read facebook/sam-3d-objects (the repo
+// is gated by manual approval). If not, it prints exactly where to request
+// access and exits.
+// 3. Streams every file under checkpoints/ from Hugging Face straight into
+// gs://<bucket>/sam3d-objects/hf/ (no local disk), skipping objects that
+// already exist at the right size, and verifies sizes after the copy.
+// 4. Copies the upstream LICENSE next to the weights, because the licence
+// requires a copy of the agreement to travel with the SAM Materials.
+// 5. Creates the Artifact Registry repository the worker image is pushed to
+// when it does not exist yet.
+//
+// The token reaches curl through the environment, never argv. Options:
+// --dry-run print what would be copied, change nothing
+// --bucket X default three-ws-model-weights
+// --prefix X default sam3d-objects
+// --project X default aerial-vehicle-466722-p5
+// --region X default us-central1
+
+import { spawnSync } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+
+export const REPO = 'facebook/sam-3d-objects';
+export const LICENSE_URL = 'https://github.com/facebookresearch/sam-3d-objects/blob/main/LICENSE';
+export const ACCESS_URL = `https://huggingface.co/${REPO}`;
+
+export const LICENSE_SUMMARY = [
+ 'SAM License (Meta, last updated November 19, 2025). Read the full text: ' + LICENSE_URL,
+ ' - Use, reproduce, modify and distribute are allowed, royalty-free, subject to the agreement.',
+ ' - Anyone you give the SAM Materials to receives them under this same agreement, with a copy of it.',
+ ' - Use must comply with Trade Controls (sanctions and export controls). You must not be a target of',
+ ' them. No ITAR, military, warfare, nuclear, espionage or weapons end uses.',
+ ' - No reverse engineering, decompiling or discovering the underlying components of the SAM Materials.',
+ ' - Research publications that use the SAM Materials must acknowledge it.',
+ ' - Provided as is, no warranty, no support, no liability.',
+ ' - The weights are gated on Hugging Face by manual approval and are refused in comprehensively',
+ ' sanctioned jurisdictions.',
+];
+
+/** Parse argv into options. Pure, so it is unit-tested. */
+export function parseArgs(argv, env = {}) {
+ const opts = {
+ accept: false,
+ dryRun: false,
+ bucket: 'three-ws-model-weights',
+ prefix: 'sam3d-objects',
+ project: 'aerial-vehicle-466722-p5',
+ region: 'us-central1',
+ token: env.HF_TOKEN || '',
+ };
+ for (let i = 0; i < argv.length; i++) {
+ const a = argv[i];
+ if (a === '--accept-sam-license') opts.accept = true;
+ else if (a === '--dry-run') opts.dryRun = true;
+ else if (['--bucket', '--prefix', '--project', '--region'].includes(a)) {
+ const value = argv[++i];
+ if (!value || value.startsWith('--')) throw new Error(`${a} needs a value`);
+ opts[a.slice(2)] = value;
+ } else throw new Error(`unknown option ${a}`);
+ }
+ for (const key of ['bucket', 'prefix', 'project', 'region']) {
+ if (!/^[a-z0-9][a-z0-9._-]*$/i.test(opts[key])) throw new Error(`--${key} has unexpected characters`);
+ }
+ return opts;
+}
+
+/**
+ * Turn the Hugging Face file listing into the copy plan. Only files under
+ * checkpoints/ are weights; the plan keeps their relative layout under hf/ and
+ * adds the upstream LICENSE. Pure, so it is unit-tested.
+ */
+export function planCopies(siblings, bucket, prefix) {
+ const dest = `gs://${bucket}/${prefix}`;
+ const plan = [];
+ for (const s of siblings) {
+ const path = s.rfilename;
+ if (path === 'LICENSE') plan.push({ path, size: s.size || 0, dest: `${dest}/LICENSE` });
+ else if (path.startsWith('checkpoints/') && !path.endsWith('.gitattributes'))
+ plan.push({ path, size: s.size || 0, dest: `${dest}/hf/${path.slice('checkpoints/'.length)}` });
+ }
+ if (!plan.some((p) => p.path === 'checkpoints/pipeline.yaml'))
+ throw new Error('checkpoints/pipeline.yaml is not in the repository listing; upstream layout changed');
+ return plan;
+}
+
+function run(cmd, args, options = {}) {
+ return spawnSync(cmd, args, { encoding: 'utf8', ...options });
+}
+
+async function main() {
+ const opts = parseArgs(process.argv.slice(2), process.env);
+ console.log(LICENSE_SUMMARY.join('\n') + '\n');
+ if (!opts.accept) {
+ console.error(
+ 'Not staging: the SAM License has not been accepted.\n' +
+ `1. Read it: ${LICENSE_URL}\n` +
+ `2. Request access with your Hugging Face account (manual approval): ${ACCESS_URL}\n` +
+ '3. Re-run with --accept-sam-license once you accept it for this account.',
+ );
+ process.exit(2);
+ }
+ if (!opts.token) {
+ console.error(`HF_TOKEN is not set. Create a read token for the account that was approved for ${REPO} and re-run.`);
+ process.exit(2);
+ }
+ const auth = { Authorization: `Bearer ${opts.token}` };
+
+ const probe = await fetch(`https://huggingface.co/${REPO}/resolve/main/LICENSE`, { headers: auth, redirect: 'follow' });
+ if (probe.status === 401 || probe.status === 403) {
+ console.error(`The token cannot read ${REPO} (HTTP ${probe.status}). Request access and wait for approval: ${ACCESS_URL}`);
+ process.exit(2);
+ }
+ if (!probe.ok) {
+ console.error(`Hugging Face answered HTTP ${probe.status} for ${REPO}. Retry shortly.`);
+ process.exit(1);
+ }
+
+ const listing = await fetch(`https://huggingface.co/api/models/${REPO}?blobs=true`, { headers: auth });
+ if (!listing.ok) {
+ console.error(`Could not list ${REPO} (HTTP ${listing.status}).`);
+ process.exit(1);
+ }
+ const plan = planCopies((await listing.json()).siblings || [], opts.bucket, opts.prefix);
+ const totalGb = plan.reduce((n, p) => n + p.size, 0) / 2 ** 30;
+ console.log(`${plan.length} files, ${totalGb.toFixed(1)} GiB -> gs://${opts.bucket}/${opts.prefix}/`);
+ if (opts.dryRun) {
+ for (const p of plan) console.log(`would copy ${p.path} -> ${p.dest} (${p.size} bytes)`);
+ return;
+ }
+
+ const gcloud = run('gcloud', ['--version']);
+ if (gcloud.status !== 0) {
+ console.error('gcloud is not installed or not on PATH. Install the Google Cloud CLI and run `gcloud auth login`.');
+ process.exit(1);
+ }
+
+ const repoCheck = run('gcloud', ['artifacts', 'repositories', 'describe', 'model-sam3d-objects', `--location=${opts.region}`, `--project=${opts.project}`]);
+ if (repoCheck.status !== 0) {
+ console.log('creating Artifact Registry repository model-sam3d-objects');
+ const created = run('gcloud', ['artifacts', 'repositories', 'create', 'model-sam3d-objects', '--repository-format=docker', `--location=${opts.region}`, `--project=${opts.project}`], { stdio: 'inherit' });
+ if (created.status !== 0) process.exit(1);
+ }
+
+ const remoteSize = (uri) => {
+ const r = run('gcloud', ['storage', 'objects', 'describe', uri, '--format=value(size)']);
+ return r.status === 0 ? r.stdout.trim() : '';
+ };
+ for (const p of plan) {
+ if (p.size && remoteSize(p.dest) === String(p.size)) {
+ console.log(`skip ${p.dest}`);
+ continue;
+ }
+ console.log(`copy ${p.path} -> ${p.dest} (${p.size} bytes)`);
+ const url = `https://huggingface.co/${REPO}/resolve/main/${p.path.split('/').map(encodeURIComponent).join('/')}`;
+ const piped = run('bash', ['-o', 'pipefail', '-c', 'curl -fsSL -H "Authorization: Bearer $HF_TOKEN" "$SRC" | gcloud storage cp - "$DEST" --quiet'], {
+ stdio: ['ignore', 'inherit', 'inherit'],
+ env: { ...process.env, HF_TOKEN: opts.token, SRC: url, DEST: p.dest },
+ });
+ if (piped.status !== 0) {
+ console.error(`FAIL ${p.dest}: copy exited ${piped.status}`);
+ process.exit(1);
+ }
+ const got = remoteSize(p.dest);
+ if (p.size && got !== String(p.size)) {
+ console.error(`FAIL ${p.dest}: expected ${p.size} bytes, bucket has ${got || 'none'}`);
+ process.exit(1);
+ }
+ }
+
+ console.log(
+ `\nSAM 3D Objects checkpoints staged at gs://${opts.bucket}/${opts.prefix}/hf/.\n` +
+ 'Next: build and deploy the worker (needs a 32 GB+ GPU class, see workers/model-sam3d-objects/README.md):\n' +
+ ' gcloud builds submit --config workers/model-sam3d-objects/cloudbuild.yaml --substitutions=SHORT_SHA=manual$(date +%s) .\n' +
+ 'then point the platform at it:\n' +
+ ' gcloud run services update three-ws-api --region us-central1 --update-env-vars MODEL_SAM3D_URL=<service url>',
+ );
+}
+
+if (process.argv[1] === fileURLToPath(import.meta.url)) {
+ main().catch((err) => {
+ console.error(err.message || err);
+ process.exit(1);
+ });
+}
diff --git a/src/ar-studio.js b/src/ar-studio.js
index d4dd119b3..a19bd1ce0 100644
--- a/src/ar-studio.js
+++ b/src/ar-studio.js
@@ -1812,6 +1812,82 @@ async function fetchTrayItems(tab) {
return items;
}
+
+// "From a photo": SAM 3D Objects (POST /api/forge-scene-objects) turns one photo
+// into a single GLB with a named node per object, laid out like the photo. The
+// scene lands in the room as one model. The endpoint answers 503
+// backend_unconfigured until the GPU worker is deployed, and the tab says so.
+const PHOTO_POLL_MS = 4000;
+const PHOTO_MAX_POLLS = 150;
+let photoJobRunning = false;
+
+async function runPhotoScene(imageUrl, ui) {
+ const res = await fetch('/api/forge-scene-objects', {
+ method: 'POST',
+ credentials: 'include',
+ headers: { 'content-type': 'application/json', 'x-forge-client': CLIENT_ID },
+ body: JSON.stringify({ image_url: imageUrl }),
+ });
+ const start = await res.json().catch(() => ({}));
+ if (!res.ok) throw new Error(start.message || start.error_description || `Request failed (${res.status}).`);
+ ui.progress(`Reading the photo: ${start.objects?.length || 'some'} object(s) found. Rebuilding in 3D…`);
+ for (let i = 0; i < PHOTO_MAX_POLLS; i++) {
+ await new Promise((r) => setTimeout(r, PHOTO_POLL_MS));
+ const poll = await fetch(`/api/forge-scene-objects?job=${encodeURIComponent(start.job_id)}`, { credentials: 'include' });
+ const body = await poll.json().catch(() => ({}));
+ if (!poll.ok) throw new Error(body.message || `Status check failed (${poll.status}).`);
+ if (body.status === 'failed' || body.error) throw new Error(body.error || 'The reconstruction failed.');
+ if (body.result_url) return body;
+ }
+ throw new Error('The reconstruction is taking too long. Try again in a few minutes.');
+}
+
+function renderPhotoTab() {
+ trayBody.innerHTML = `
+ <form class="ars-link-form" id="ars-photo-form">
+ <label class="ars-link-label" for="ars-photo-input">Paste a photo link</label>
+ <div class="ars-link-row">
+ <input id="ars-photo-input" type="url" inputmode="url" required
+ placeholder="https://example.com/desk.jpg" />
+ <button type="submit" class="ars-btn ars-btn-primary" id="ars-photo-go">Build scene</button>
+ </div>
+ <p class="ars-link-hint">Every object SAM 3D finds in the photo becomes a 3D object, laid out the way it sits in the picture. Takes a couple of minutes.</p>
+ <div id="ars-photo-state" role="status" aria-live="polite"></div>
+ </form>`;
+ const form = $('ars-photo-form');
+ const state = $('ars-photo-state');
+ const go = $('ars-photo-go');
+ form?.addEventListener('submit', async (e) => {
+ e.preventDefault();
+ if (photoJobRunning) return;
+ const url = String($('ars-photo-input')?.value || '').trim();
+ if (!/^https:\/\//i.test(url)) {
+ state.innerHTML = '<p class="ars-photo-error">That needs to be a public https link to a photo.</p>';
+ return;
+ }
+ photoJobRunning = true;
+ go.disabled = true;
+ const ui = {
+ progress: (msg) => {
+ state.innerHTML = '<div class="ars-photo-progress"><span class="ars-spinner" aria-hidden="true"></span><span></span></div>';
+ state.querySelector('span:last-child').textContent = msg;
+ },
+ };
+ ui.progress('Finding the objects in your photo…');
+ try {
+ const done = await runPhotoScene(url, ui);
+ closeTray();
+ await addModel({ src: done.result_url, title: 'Photo scene' });
+ } catch (err) {
+ state.innerHTML = '<p class="ars-photo-error"></p>';
+ state.firstChild.textContent = err.message;
+ } finally {
+ photoJobRunning = false;
+ go.disabled = false;
+ }
+ });
+}
+
async function renderTray() {
if (!trayBody) return;
const tab = trayTab;
@@ -1842,6 +1918,11 @@ async function renderTray() {
return;
}
+ if (tab === 'photo') {
+ renderPhotoTab();
+ return;
+ }
+
trayBody.innerHTML = '<div class="ars-tray-loading"><span class="ars-spinner" aria-hidden="true"></span> Loading models…</div>';
let items;
try {
diff --git a/tests/api/forge-scene-objects.test.js b/tests/api/forge-scene-objects.test.js
new file mode 100644
index 000000000..e2538144b
--- /dev/null
+++ b/tests/api/forge-scene-objects.test.js
@@ -0,0 +1,207 @@
+/**
+ * Tests for the scene-from-a-photo endpoint (api/forge-scene-objects.js).
+ *
+ * The SAM 3D Objects worker and the vision chain are stubbed at their module
+ * boundaries, so this exercises the endpoint's own logic: the sanctions guard,
+ * SSRF checks, object selection (boxes, masks, automatic detection), the
+ * backend_unconfigured answer, and the poll passthrough.
+ */
+
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+import { Readable } from 'node:stream';
+
+const submit = vi.fn(async () => ({ extJobId: `task-sam3d-${'x'.repeat(20)}`, eta: 150 }));
+const status = vi.fn(async () => ({
+ status: 'done',
+ resultGlbUrl: 'https://storage.googleapis.com/bucket/sam3d/task.glb',
+ objects: [{ id: 'object_01', centroid: [0, 0, -3] }],
+ objectCount: 1,
+}));
+let supportsSam3d = true;
+
+vi.mock('../../api/_providers/gcp.js', () => ({
+ createRegenProvider: () => ({
+ supportsMode: (m) => m === 'sam3d' && supportsSam3d,
+ submit,
+ status,
+ }),
+}));
+
+const detect = vi.fn();
+vi.mock('../../api/_lib/vision.js', () => ({
+ describeImageJson: (...a) => detect(...a),
+}));
+
+vi.mock('../../api/_lib/ssrf.js', async (importOriginal) => {
+ const mod = await importOriginal();
+ return {
+ ...mod,
+ assertPublicHttpsUrl: vi.fn(async (url) => {
+ const parsed = new URL(url);
+ if (parsed.protocol !== 'https:') throw new mod.SsrfError('https required');
+ return url;
+ }),
+ };
+});
+
+vi.mock('../../api/_lib/rate-limit.js', async (importActual) => {
+ const actual = await importActual();
+ return {
+ ...actual,
+ clientIp: () => '203.0.113.9',
+ limits: {
+ ...actual.limits,
+ mcp3dGenerate: vi.fn(async () => ({ success: true, reset: Date.now() + 1000 })),
+ mcp3dStatus: vi.fn(async () => ({ success: true, reset: Date.now() + 1000 })),
+ },
+ };
+});
+
+const { default: handler } = await import('../../api/forge-scene-objects.js');
+
+function makeRes() {
+ return {
+ statusCode: 200,
+ headers: {},
+ body: null,
+ setHeader(n, v) {
+ this.headers[String(n).toLowerCase()] = v;
+ },
+ end(b) {
+ this.body = b ?? null;
+ },
+ };
+}
+
+function makeReq({ method = 'POST', url = '/api/forge-scene-objects', headers = {}, body = null } = {}) {
+ const payload = body == null ? '' : JSON.stringify(body);
+ const stream = Readable.from(payload ? [Buffer.from(payload, 'utf8')] : []);
+ stream.method = method;
+ stream.url = url;
+ stream.headers = { 'content-type': 'application/json', 'x-forwarded-for': '203.0.113.9', ...headers };
+ return stream;
+}
+
+const PHOTO = 'https://storage.googleapis.com/bucket/room.jpg';
+
+beforeEach(() => {
+ submit.mockClear();
+ status.mockClear();
+ detect.mockReset();
+ supportsSam3d = true;
+});
+
+describe('POST /api/forge-scene-objects', () => {
+ it('queues a job with the caller\'s boxes in order', async () => {
+ const res = makeRes();
+ await handler(makeReq({ body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]], seed: 3 } }), res);
+ expect(res.statusCode).toBe(202);
+ const body = JSON.parse(res.body);
+ expect(body.source).toBe('boxes');
+ expect(body.eta_seconds).toBe(150);
+ const call = submit.mock.calls[0][0];
+ expect(call.mode).toBe('sam3d');
+ expect(call.sourceUrl).toBe(PHOTO);
+ expect(call.params.boxes).toEqual([[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]]);
+ expect(call.params.seed).toBe(3);
+ expect(detect).not.toHaveBeenCalled();
+ });
+
+ it('queues a job with mask URLs after the SSRF check', async () => {
+ const res = makeRes();
+ await handler(makeReq({ body: { image_url: PHOTO, mask_urls: ['https://storage.googleapis.com/bucket/m1.png'] } }), res);
+ expect(res.statusCode).toBe(202);
+ expect(JSON.parse(res.body).source).toBe('masks');
+ expect(submit.mock.calls[0][0].params.maskUrls).toEqual(['https://storage.googleapis.com/bucket/m1.png']);
+ });
+
+ it('finds the objects itself when none are named', async () => {
+ detect.mockResolvedValue({ json: { objects: [{ label: 'lamp', box_2d: [100, 100, 600, 400] }] } });
+ const res = makeRes();
+ await handler(makeReq({ body: { image_url: PHOTO } }), res);
+ expect(res.statusCode).toBe(202);
+ const body = JSON.parse(res.body);
+ expect(body.source).toBe('detected');
+ expect(body.objects).toEqual([{ label: 'lamp', box: [0.1, 0.1, 0.4, 0.6] }]);
+ expect(submit.mock.calls[0][0].params.boxes).toEqual([[0.1, 0.1, 0.4, 0.6]]);
+ });
+
+ it('explains a vision outage and an empty photo instead of failing blind', async () => {
+ detect.mockRejectedValue(new Error('no vision provider'));
+ let res = makeRes();
+ await handler(makeReq({ body: { image_url: PHOTO } }), res);
+ expect(res.statusCode).toBe(502);
+ expect(JSON.parse(res.body).error).toBe('detect_failed');
+
+ detect.mockResolvedValue({ json: { objects: [] } });
+ res = makeRes();
+ await handler(makeReq({ body: { image_url: PHOTO } }), res);
+ expect(res.statusCode).toBe(422);
+ expect(JSON.parse(res.body).error).toBe('no_objects_found');
+ expect(submit).not.toHaveBeenCalled();
+ });
+
+ it('turns away a comprehensively sanctioned jurisdiction before any work', async () => {
+ const res = makeRes();
+ await handler(makeReq({ headers: { 'x-client-geo-location': 'IR,Tehran' }, body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]] } }), res);
+ expect(res.statusCode).toBe(451);
+ expect(JSON.parse(res.body).error).toBe('unavailable_in_region');
+ expect(submit).not.toHaveBeenCalled();
+ });
+
+ it('does not block an unresolved country or an ordinary one', async () => {
+ for (const headers of [{}, { 'x-client-geo-location': 'DE,Berlin' }]) {
+ const res = makeRes();
+ await handler(makeReq({ headers, body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]] } }), res);
+ expect(res.statusCode).toBe(202);
+ }
+ });
+
+ it('answers 503 backend_unconfigured until the worker is set', async () => {
+ supportsSam3d = false;
+ const res = makeRes();
+ await handler(makeReq({ body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]] } }), res);
+ expect(res.statusCode).toBe(503);
+ expect(JSON.parse(res.body).error).toBe('backend_unconfigured');
+ expect(submit).not.toHaveBeenCalled();
+ });
+
+ it.each([
+ [{ image_url: 'http://169.254.169.254/x.jpg', boxes: [[0.1, 0.1, 0.4, 0.5]] }, 'invalid_image_url'],
+ [{ image_url: PHOTO, mask_urls: ['http://10.0.0.1/m.png'] }, 'invalid_mask_urls'],
+ [{ image_url: PHOTO, boxes: [[0.5, 0.5, 0.4, 0.4]] }, 'invalid_boxes'],
+ [{ image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]], mask_urls: ['https://x.test/m.png'] }, 'ambiguous_objects'],
+ [{ image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]], max_objects: 99 }, 'invalid_max_objects'],
+ [{ image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]], seed: -2 }, 'invalid_seed'],
+ ])('rejects %j', async (body, error) => {
+ const res = makeRes();
+ await handler(makeReq({ body }), res);
+ expect(res.statusCode).toBe(400);
+ expect(JSON.parse(res.body).error).toBe(error);
+ expect(submit).not.toHaveBeenCalled();
+ });
+});
+
+describe('GET /api/forge-scene-objects?job=<id>', () => {
+ it('polls the job and surfaces the object manifest', async () => {
+ const jobId = `task-sam3d-${'x'.repeat(20)}`;
+ const res = makeRes();
+ await handler(makeReq({ method: 'GET', url: `/api/forge-scene-objects?job=${jobId}` }), res);
+ expect(res.statusCode).toBe(200);
+ const body = JSON.parse(res.body);
+ expect(body.status).toBe('done');
+ expect(body.result_url).toBe('https://storage.googleapis.com/bucket/sam3d/task.glb');
+ expect(body.objects).toEqual([{ id: 'object_01', centroid: [0, 0, -3] }]);
+ expect(body.object_count).toBe(1);
+ });
+
+ it('rejects a malformed or missing job id', async () => {
+ let res = makeRes();
+ await handler(makeReq({ method: 'GET', url: '/api/forge-scene-objects?job=short' }), res);
+ expect(JSON.parse(res.body).error).toBe('invalid_job');
+ res = makeRes();
+ await handler(makeReq({ method: 'GET', url: '/api/forge-scene-objects' }), res);
+ expect(JSON.parse(res.body).error).toBe('missing_job');
+ expect(status).not.toHaveBeenCalled();
+ });
+});
diff --git a/tests/api/gcp-sam3d-provider.test.js b/tests/api/gcp-sam3d-provider.test.js
new file mode 100644
index 000000000..533052662
--- /dev/null
+++ b/tests/api/gcp-sam3d-provider.test.js
@@ -0,0 +1,88 @@
+/**
+ * The gcp provider's `sam3d` mode (api/_providers/gcp.js): the request shape
+ * sent to workers/model-sam3d-objects and how a finished task maps to a scene
+ * result. fetch is mocked at the network boundary only.
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+
+const realFetch = globalThis.fetch;
+const WORKER = 'https://sam3d.example.run.app';
+
+beforeEach(() => {
+ process.env.GCP_RECONSTRUCTION_KEY = 'test-key';
+ process.env.MODEL_SAM3D_URL = WORKER;
+});
+afterEach(() => {
+ globalThis.fetch = realFetch;
+ delete process.env.MODEL_SAM3D_URL;
+});
+
+const { createRegenProvider } = await import('../../api/_providers/gcp.js');
+
+describe('gcp provider: sam3d', () => {
+ it('is supported only when MODEL_SAM3D_URL is set', () => {
+ expect(createRegenProvider().supportsMode('sam3d')).toBe(true);
+ delete process.env.MODEL_SAM3D_URL;
+ expect(createRegenProvider().supportsMode('sam3d')).toBe(false);
+ });
+
+ it('posts the photo and boxes to /infer', async () => {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ task_id: 't-1', status: 'queued' }), { status: 202 }));
+ globalThis.fetch = fetchMock;
+ const job = await createRegenProvider().submit({
+ mode: 'sam3d',
+ sourceUrl: 'https://example.com/room.jpg',
+ params: { boxes: [[0.1, 0.1, 0.5, 0.5]], maxObjects: 4, seed: 9 },
+ });
+ const [url, init] = fetchMock.mock.calls[0];
+ expect(url).toBe(`${WORKER}/infer`);
+ expect(JSON.parse(init.body)).toEqual({
+ image_url: 'https://example.com/room.jpg',
+ boxes: [[0.1, 0.1, 0.5, 0.5]],
+ max_objects: 4,
+ seed: 9,
+ });
+ expect(job.eta).toBe(150);
+ });
+
+ it('prefers mask URLs and never sends both', async () => {
+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ task_id: 't-1' }), { status: 202 }));
+ globalThis.fetch = fetchMock;
+ await createRegenProvider().submit({
+ mode: 'sam3d',
+ sourceUrl: 'https://example.com/room.jpg',
+ params: { maskUrls: ['https://example.com/m.png'], boxes: [[0.1, 0.1, 0.5, 0.5]] },
+ });
+ const body = JSON.parse(fetchMock.mock.calls[0][1].body);
+ expect(body.mask_urls).toEqual(['https://example.com/m.png']);
+ expect(body.boxes).toBeUndefined();
+ });
+
+ it('maps a done task to the scene GLB and its objects', async () => {
+ globalThis.fetch = vi.fn(async () => new Response(JSON.stringify({ task_id: 't-2' }), { status: 202 }));
+ const provider = createRegenProvider();
+ const { extJobId } = await provider.submit({ mode: 'sam3d', sourceUrl: 'https://example.com/room.jpg', params: { boxes: [[0.1, 0.1, 0.5, 0.5]] } });
+ globalThis.fetch = vi.fn(async () =>
+ new Response(
+ JSON.stringify({
+ task_id: 't-2',
+ status: 'done',
+ result_gcs_url: 'https://storage.googleapis.com/b/scene.glb',
+ objects: [{ id: 'object_01', centroid: [0, 0, -2] }],
+ object_count: 1,
+ skipped: [{ index: 1, reason: 'no mesh' }],
+ }),
+ { status: 200 },
+ ),
+ );
+ const out = await provider.status(extJobId);
+ expect(out).toMatchObject({
+ status: 'done',
+ resultGlbUrl: 'https://storage.googleapis.com/b/scene.glb',
+ objects: [{ id: 'object_01', centroid: [0, 0, -2] }],
+ objectCount: 1,
+ skipped: [{ index: 1, reason: 'no mesh' }],
+ });
+ });
+});
diff --git a/tests/forge-objects.test.js b/tests/forge-objects.test.js
new file mode 100644
index 000000000..3aab57c8e
--- /dev/null
+++ b/tests/forge-objects.test.js
@@ -0,0 +1,71 @@
+import { describe, it, expect } from 'vitest';
+import {
+ SANCTIONED_COUNTRIES,
+ isSanctionedTerritory,
+ normalizeDetectedBoxes,
+ parseUserBoxes,
+} from '../api/_lib/forge-objects.js';
+
+describe('isSanctionedTerritory', () => {
+ it('blocks only positively resolved comprehensively sanctioned countries', () => {
+ for (const c of SANCTIONED_COUNTRIES) expect(isSanctionedTerritory(c)).toBe(true);
+ expect(isSanctionedTerritory('ir')).toBe(true);
+ expect(isSanctionedTerritory('US')).toBe(false);
+ expect(isSanctionedTerritory('DE')).toBe(false);
+ });
+ it('does not turn away an unresolved country', () => {
+ expect(isSanctionedTerritory(null)).toBe(false);
+ expect(isSanctionedTerritory(undefined)).toBe(false);
+ expect(isSanctionedTerritory('')).toBe(false);
+ });
+});
+
+describe('normalizeDetectedBoxes', () => {
+ it('converts [ymin, xmin, ymax, xmax] on 0..1000 to [x0, y0, x1, y1] fractions', () => {
+ const out = normalizeDetectedBoxes({ objects: [{ label: 'chair', box_2d: [200, 100, 800, 500] }] });
+ expect(out).toEqual([{ label: 'chair', box: [0.1, 0.2, 0.5, 0.8] }]);
+ });
+ it('orders largest first, drops slivers and near-duplicates, and honours the cap', () => {
+ const out = normalizeDetectedBoxes(
+ {
+ objects: [
+ { label: 'small', box_2d: [0, 0, 100, 100] },
+ { label: 'big', box_2d: [0, 0, 900, 900] },
+ { label: 'big again', box_2d: [10, 10, 900, 900] },
+ { label: 'sliver', box_2d: [0, 0, 500, 10] },
+ { label: 'inverted', box_2d: [500, 500, 400, 400] },
+ ],
+ },
+ 5,
+ );
+ expect(out.map((o) => o.label)).toEqual(['big', 'small']);
+ expect(normalizeDetectedBoxes({ objects: [{ box_2d: [0, 0, 500, 500] }, { box_2d: [500, 500, 1000, 1000] }] }, 1)).toHaveLength(1);
+ });
+ it('clamps to the frame and tolerates junk', () => {
+ expect(normalizeDetectedBoxes({ objects: [{ label: 'x', box_2d: [-50, -50, 1200, 1200] }] })[0].box).toEqual([0, 0, 1, 1]);
+ expect(normalizeDetectedBoxes(null)).toEqual([]);
+ expect(normalizeDetectedBoxes({ objects: [{ box_2d: ['a', 1, 2, 3] }, { box_2d: [1, 2] }, null, 7] })).toEqual([]);
+ });
+ it('accepts a bare array and trims labels', () => {
+ const out = normalizeDetectedBoxes([{ label: ' red mug ', box_2d: [100, 100, 400, 400] }]);
+ expect(out[0].label).toBe('red mug');
+ });
+});
+
+describe('parseUserBoxes', () => {
+ it('accepts valid boxes untouched and in order', () => {
+ const r = parseUserBoxes([[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]]);
+ expect(r.boxes).toEqual([[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]]);
+ });
+ it('names the offending box', () => {
+ expect(parseUserBoxes([[0.1, 0.1, 0.4, 0.5], [0.5, 0.5, 0.4, 0.4]]).error).toMatch(/boxes\[1\]/);
+ expect(parseUserBoxes([[0, 0, 2, 1]]).error).toMatch(/boxes\[0\]/);
+ expect(parseUserBoxes([['0', 0, 1, 1]]).error).toMatch(/four numbers/);
+ expect(parseUserBoxes([[0.1, 0.1, 0.12, 0.9]]).error).toMatch(/too small/);
+ });
+ it('rejects two boxes that name the same object, an empty list and too many boxes', () => {
+ expect(parseUserBoxes([[0.1, 0.1, 0.5, 0.5], [0.11, 0.11, 0.5, 0.5]]).error).toMatch(/same object/);
+ expect(parseUserBoxes([]).error).toBeTruthy();
+ expect(parseUserBoxes([[0, 0, 0.5, 0.5], [0.5, 0.5, 1, 1]], 1).error).toMatch(/max_objects/);
+ });
+});
diff --git a/tests/stage-sam3d-objects.test.js b/tests/stage-sam3d-objects.test.js
new file mode 100644
index 000000000..eae2a5708
--- /dev/null
+++ b/tests/stage-sam3d-objects.test.js
@@ -0,0 +1,82 @@
+import { describe, it, expect } from 'vitest';
+import { spawnSync } from 'node:child_process';
+import { readFileSync } from 'node:fs';
+import { resolve } from 'node:path';
+import { parseArgs, planCopies } from '../scripts/stage-sam3d-objects.mjs';
+
+const ROOT = resolve(import.meta.dirname, '..');
+const SCRIPT = resolve(ROOT, 'scripts/stage-sam3d-objects.mjs');
+
+describe('stage-sam3d-objects', () => {
+ it('refuses to stage until the SAM License is explicitly accepted', () => {
+ const r = spawnSync(process.execPath, [SCRIPT], { encoding: 'utf8', env: { PATH: process.env.PATH, HF_TOKEN: 'hf_x' } });
+ expect(r.status).toBe(2);
+ expect(r.stderr).toMatch(/SAM License has not been accepted/);
+ expect(r.stderr).toMatch(/huggingface\.co\/facebook\/sam-3d-objects/);
+ expect(r.stdout).toMatch(/Trade Controls/);
+ });
+
+ it('refuses without a token even when the licence is accepted', () => {
+ const r = spawnSync(process.execPath, [SCRIPT, '--accept-sam-license'], { encoding: 'utf8', env: { PATH: process.env.PATH } });
+ expect(r.status).toBe(2);
+ expect(r.stderr).toMatch(/HF_TOKEN is not set/);
+ });
+
+ it('parses options and rejects unknown or malformed ones', () => {
+ const o = parseArgs(['--accept-sam-license', '--dry-run', '--bucket', 'b1'], { HF_TOKEN: 't' });
+ expect(o).toMatchObject({ accept: true, dryRun: true, bucket: 'b1', prefix: 'sam3d-objects', token: 't' });
+ expect(parseArgs([], {}).accept).toBe(false);
+ expect(() => parseArgs(['--nope'])).toThrow(/unknown option/);
+ expect(() => parseArgs(['--bucket'])).toThrow(/needs a value/);
+ expect(() => parseArgs(['--bucket', 'a;rm -rf /'])).toThrow(/unexpected characters/);
+ });
+
+ it('plans only checkpoints and the licence, under hf/', () => {
+ const plan = planCopies(
+ [
+ { rfilename: '.gitattributes' },
+ { rfilename: 'README.md', size: 5 },
+ { rfilename: 'LICENSE', size: 10 },
+ { rfilename: 'doc/intro.png', size: 7 },
+ { rfilename: 'checkpoints/pipeline.yaml', size: 100 },
+ { rfilename: 'checkpoints/slat_generator.ckpt', size: 4000 },
+ ],
+ 'bk',
+ 'sam3d-objects',
+ );
+ expect(plan.map((p) => p.dest)).toEqual([
+ 'gs://bk/sam3d-objects/LICENSE',
+ 'gs://bk/sam3d-objects/hf/pipeline.yaml',
+ 'gs://bk/sam3d-objects/hf/slat_generator.ckpt',
+ ]);
+ });
+
+ it('fails loudly when upstream drops pipeline.yaml', () => {
+ expect(() => planCopies([{ rfilename: 'checkpoints/a.ckpt', size: 1 }], 'bk', 'p')).toThrow(/pipeline\.yaml/);
+ });
+});
+
+describe('workers/model-sam3d-objects build files', () => {
+ const read = (f) => readFileSync(resolve(ROOT, 'workers/model-sam3d-objects', f), 'utf8');
+
+ it('pins the build service account and a 32 GB+ GPU class', () => {
+ const cb = read('cloudbuild.yaml');
+ expect(cb).toMatch(/serviceAccount: projects\/aerial-vehicle-466722-p5\/serviceAccounts\/three-ws-build@/);
+ expect(cb).toMatch(/_GPU_TYPE: nvidia-rtx-pro-6000/);
+ expect(cb).not.toMatch(/nvidia-l4/);
+ });
+
+ it('pins upstream to a commit and gates the image on the tests', () => {
+ const df = read('Dockerfile');
+ expect(df).toMatch(/ARG SAM3D_COMMIT=[0-9a-f]{40}/);
+ expect(df).toMatch(/pytest .*test_request_policy\.py.*test_scene_layout\.py/s);
+ expect(df).toMatch(/test_app_contract\.py/);
+ });
+
+ it('copies every module the app imports into the image', () => {
+ const df = read('Dockerfile');
+ for (const f of ['main.py', 'pipeline.py', 'scene_layout.py', 'request_policy.py', 'worker_security.py', 'storage_backend.py']) {
+ expect(df).toContain(f);
+ }
+ });
+});
diff --git a/workers/model-sam3d-objects/Dockerfile b/workers/model-sam3d-objects/Dockerfile
new file mode 100644
index 000000000..4f5fecfab
--- /dev/null
+++ b/workers/model-sam3d-objects/Dockerfile
@@ -0,0 +1,75 @@
+# SAM 3D Objects (facebookresearch/sam-3d-objects, SAM License): one photo in, a
+# posed 3D scene of its objects out. Upstream needs a Linux x86-64 host and a
+# 32 GB+ NVIDIA GPU, so this image targets the Cloud Run RTX Pro 6000 class, not
+# the L4 the other workers use (see cloudbuild.yaml).
+#
+# Pinned stack, read from the upstream commit below on 2026-10-24: Python 3.11,
+# CUDA 12.1, torch 2.5.1. The official PyTorch devel image carries exactly that
+# plus nvcc, which pytorch3d, kaolin and gsplat compile against.
+#
+# The SAM License is NOT accepted by building this image: the checkpoints are not
+# in it. They are staged separately by the account owner with
+# `npm run stage:sam3d-objects -- --accept-sam-license`.
+FROM pytorch/pytorch:2.5.1-cuda12.1-cudnn9-devel
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+ PYTHONUNBUFFERED=1 \
+ PIP_NO_CACHE_DIR=1 \
+ DEBIAN_FRONTEND=noninteractive \
+ HF_HUB_DISABLE_TELEMETRY=1 \
+ DO_NOT_TRACK=1 \
+ CUDA_HOME=/usr/local/cuda \
+ CONDA_PREFIX=/usr/local/cuda \
+ LIDRA_SKIP_INIT=true \
+ TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0;12.0" \
+ SAM3D_SRC=/app/sam3d-objects \
+ SAM3D_CHECKPOINT_DIR=/app/checkpoints/hf \
+ PIP_EXTRA_INDEX_URL="https://pypi.ngc.nvidia.com https://download.pytorch.org/whl/cu121" \
+ PIP_FIND_LINKS="https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.5.1_cu121.html"
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+ git curl build-essential ninja-build \
+ libgl1 libglib2.0-0 libgomp1 libegl1 \
+ && rm -rf /var/lib/apt/lists/*
+
+# 1. SAM 3D Objects, pinned to the commit read on 2026-10-24
+# (https://github.com/facebookresearch/sam-3d-objects/commit/f91db411c50efee93d8db7aeb323885650f6f722).
+ARG SAM3D_COMMIT=f91db411c50efee93d8db7aeb323885650f6f722
+RUN git clone https://github.com/facebookresearch/sam-3d-objects.git ${SAM3D_SRC} \
+ && git -C ${SAM3D_SRC} checkout ${SAM3D_COMMIT} \
+ && rm -rf ${SAM3D_SRC}/.git
+
+# 2. Upstream's own install sequence (its doc/setup.md), in the same order: the
+# core stack, then pytorch3d (its pip metadata is broken without the 2-step),
+# then the inference extras.
+WORKDIR ${SAM3D_SRC}
+RUN pip install -e '.[dev]' \
+ && pip install -e '.[p3d]' \
+ && pip install -e '.[inference]'
+
+# 3. Upstream's hydra patch (patching/hydra), applied from the same pinned fork
+# commit it downloads, with the file's checksum verified first.
+ARG HYDRA_PATCH_SHA256=b5799ea99626593e7650cd6ab7e15639e32d495d1a16f71d627811f86a2a6fba
+RUN curl -fsSL -o /tmp/hydra_utils.py \
+ https://raw.githubusercontent.com/gleize/hydra/78f00766b5f37672aa7232ebbf01bdd74246bd60/hydra/core/utils.py \
+ && echo "${HYDRA_PATCH_SHA256} /tmp/hydra_utils.py" | sha256sum -c - \
+ && python -c 'import hydra, os, shutil; assert hydra.__version__ == "1.3.2", hydra.__version__; shutil.copy("/tmp/hydra_utils.py", os.path.join(os.path.dirname(hydra.__file__), "core", "utils.py"))' \
+ && rm /tmp/hydra_utils.py
+
+WORKDIR /app
+COPY requirements.txt /app/requirements.txt
+RUN pip install -r /app/requirements.txt
+
+COPY worker_security.py storage_backend.py request_policy.py scene_layout.py pipeline.py main.py entrypoint.sh /app/
+COPY test_request_policy.py test_scene_layout.py test_app_contract.py /app/
+RUN chmod +x /app/entrypoint.sh
+
+# Build-time guard: the pure-logic tests and the served-contract test need no
+# GPU and no checkpoints, so they run here and the image does not ship if request
+# validation, the scene geometry or the import chain broke.
+RUN python -m pytest -q /app/test_request_policy.py /app/test_scene_layout.py \
+ && python /app/test_app_contract.py
+
+EXPOSE 8080
+
+CMD ["/app/entrypoint.sh"]
diff --git a/workers/model-sam3d-objects/README.md b/workers/model-sam3d-objects/README.md
new file mode 100644
index 000000000..8b208578e
--- /dev/null
+++ b/workers/model-sam3d-objects/README.md
@@ -0,0 +1,89 @@
+# model-sam3d-objects: one photo to a laid-out 3D scene (SAM 3D Objects)
+
+FastAPI inference service that rebuilds the **objects in a single photo as one 3D
+scene GLB** with [SAM 3D Objects](https://github.com/facebookresearch/sam-3d-objects)
+(Meta). Each object becomes a named node (`object_01`, `object_02`, ...) placed,
+rotated and scaled the way it sits in the picture. The layout is relative, not
+metric. It backs `POST /api/forge-scene-objects` and the AR Studio tray tab
+"From a photo".
+
+Upstream is pinned to commit `f91db411c50efee93d8db7aeb323885650f6f722`
+(`SAM3D_COMMIT` in the Dockerfile).
+
+## Status: off until the owner stages weights
+
+The weights are gated on Hugging Face by manual approval under the SAM License
+and the model needs a **32 GB+ GPU**, so the platform answers
+`503 backend_unconfigured` until all of this is done:
+
+1. Accept the SAM License and get access: <https://huggingface.co/facebook/sam-3d-objects>.
+2. `HF_TOKEN=... npm run stage:sam3d-objects -- --accept-sam-license` (add `--dry-run` first to see the plan). It copies the checkpoints and the LICENSE to `gs://three-ws-model-weights/sam3d-objects/`.
+3. Make sure the project has quota for the GPU class below, then
+ `gcloud builds submit --config workers/model-sam3d-objects/cloudbuild.yaml --substitutions=SHORT_SHA=manual$(date +%s) .`
+4. `gcloud run services update three-ws-api --region us-central1 --update-env-vars MODEL_SAM3D_URL=<service url>` (use `--update-env-vars`, never `--set-env-vars`). `GCP_RECONSTRUCTION_KEY` is shared with the other model workers.
+
+## GPU class
+
+The 24 GB L4 the other workers use cannot hold the model. `cloudbuild.yaml`
+defaults to `nvidia-rtx-pro-6000` with 20 CPU and 80Gi memory (Cloud Run's floor
+for that class). Override `_GPU_TYPE`, `_CPU`, `_MEMORY` together if the class
+changes. Scale-to-zero is the default; a warm floor is a spend decision.
+
+## Endpoints
+
+`POST /infer`, `GET /tasks/{id}` and `GET /results/{id}.glb` need
+`Authorization: Bearer $API_KEY`. `GET /` and `GET /health` are open (`/health`
+returns 503 once a model load failure latches).
+
+`POST /infer` returns `202 {task_id}`:
+
+```json
+{ "image_url": "https://example.com/desk.jpg",
+ "boxes": [[0.1, 0.2, 0.5, 0.9]],
+ "max_objects": 8, "seed": 42 }
+```
+
+| Field | Meaning |
+|---|---|
+| `image_url` | Public https photo (SSRF-checked). |
+| `mask_urls` or `boxes` | Exactly one. Masks are https PNGs, white is the object. Boxes are `[x0, y0, x1, y1]` fractions of the frame, origin top-left, turned into masks with GrabCut. |
+| `max_objects` | 1 to 12, default 8. Slivers (under 4% of a side or 0.2% of the frame) are rejected. |
+| `seed` | 0 to 4294967295. |
+
+Bad requests are `422` from `request_policy.py`. A task that finishes holds
+`result_gcs_url`, `objects` (name, position, size per node) and `skipped`
+(objects that failed individually). It fails only if every object fails.
+
+## Environment
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `API_KEY` | required | Bearer secret, from Secret Manager `avatar-reconstruction-key`. |
+| `GCS_BUCKET` | unset | Upload to `raw-meshes/sam3d-objects/{task}.glb`; otherwise serve from `OUTPUT_DIR`. |
+| `WEIGHTS_ROOT` | `/weights/sam3d-objects` | Mounted weights; `entrypoint.sh` copies `hf/` to `/app/checkpoints/hf`. |
+| `SAM3D_CHECKPOINT_DIR`, `SAM3D_SRC` | image paths | Checkpoints and the pinned upstream checkout. |
+| `MAX_CONCURRENT` | `1` | Simultaneous reconstructions per instance. |
+| `MODEL_LOAD_ATTEMPTS`, `MODEL_LOAD_RETRY_BASE_S`, `MODEL_LOAD_RETRY_CAP_S` | `4`, `15`, `120` | Load retry policy. |
+| `IMAGE_FETCH_TIMEOUT_S` | `60` | Photo and mask fetch timeout. |
+
+## Tests
+
+`python -m pytest test_request_policy.py test_scene_layout.py test_app_contract.py`
+(the Dockerfile runs them as a build gate). They cover request validation, the pose
+math, mask handling, scene assembly and the HTTP contract with a stub pipeline.
+`worker_security.py` is vendored; `npm run check:vendored` keeps it identical.
+
+## Not verified without a GPU and the weights
+
+The image build, a real model run, and the pose orientation (checked against
+upstream source only) have not been exercised. Whether `pipeline.yaml` pulls an
+auxiliary depth model from Hugging Face at first load is not visible without the
+gated file, so the container is not run offline and the first load may need HF
+egress. Cloud Run availability of the GPU class in the chosen region is also
+unconfirmed.
+
+## License
+
+SAM License, see the LICENSE copied beside the weights. Use must comply with
+Trade Controls; the platform endpoint refuses requests from comprehensively
+sanctioned territories before queueing.
diff --git a/workers/model-sam3d-objects/cloudbuild.yaml b/workers/model-sam3d-objects/cloudbuild.yaml
new file mode 100644
index 000000000..4fd290f72
--- /dev/null
+++ b/workers/model-sam3d-objects/cloudbuild.yaml
@@ -0,0 +1,89 @@
+# Build + deploy the SAM 3D Objects "scene from a photo" worker.
+# Manual submit (SHORT_SHA is only auto-set by triggers):
+# gcloud builds submit --config workers/model-sam3d-objects/cloudbuild.yaml \
+# --substitutions=SHORT_SHA=manual$(date +%s) .
+#
+# Prerequisites, owner-run once:
+# HF_TOKEN=... npm run stage:sam3d-objects -- --accept-sam-license
+# GPU quota for the class named in _GPU_TYPE (see README: GPU class).
+serviceAccount: projects/aerial-vehicle-466722-p5/serviceAccounts/[email protected]
+
+substitutions:
+ _REGION: us-central1
+ _RUN_SA: avatar-reconstruction-sa@aerial-vehicle-466722-p5.iam.gserviceaccount.com
+ _SERVICE: model-sam3d-objects
+ _REPO: model-sam3d-objects
+ _GCS_BUCKET: three-ws-avatar-reconstructions
+ _WEIGHTS_BUCKET: three-ws-model-weights
+ # Upstream requires 32 GB+ of VRAM. The L4 (24 GB) the other workers use cannot
+ # run it. The 96 GB RTX Pro 6000 class can, and Cloud Run requires its larger
+ # CPU/memory floor, hence the values below. Override both together if the GPU
+ # type changes.
+ _GPU_TYPE: nvidia-rtx-pro-6000
+ _CPU: "20"
+ _MEMORY: 80Gi
+ # Scale-to-zero by default: a warm floor on a 96 GB GPU is a quota and spend
+ # decision for the owner, not a default.
+ _MIN_INSTANCES: "0"
+ _MAX_INSTANCES: "1"
+
+steps:
+ - id: build
+ name: gcr.io/cloud-builders/docker
+ dir: workers/model-sam3d-objects
+ args:
+ - build
+ - --tag
+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA
+ - --tag
+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:latest
+ - --cache-from
+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:latest
+ - .
+
+ - id: push
+ name: gcr.io/cloud-builders/docker
+ args:
+ - push
+ - --all-tags
+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server
+
+ - id: deploy
+ name: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
+ entrypoint: gcloud
+ args:
+ - run
+ - deploy
+ - ${_SERVICE}
+ - --image=${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA
+ - --region=${_REGION}
+ - --service-account=${_RUN_SA}
+ - --platform=managed
+ - --port=8080
+ - --gpu=1
+ - --gpu-type=${_GPU_TYPE}
+ - --no-gpu-zonal-redundancy
+ - --no-cpu-throttling
+ - --cpu=${_CPU}
+ - --memory=${_MEMORY}
+ - --min-instances=${_MIN_INSTANCES}
+ - --max-instances=${_MAX_INSTANCES}
+ - --timeout=900
+ - --startup-probe=tcpSocket.port=8080,initialDelaySeconds=0,periodSeconds=60,timeoutSeconds=60,failureThreshold=15
+ - --cpu-boost
+ - --allow-unauthenticated
+ - --add-volume=name=weights,type=cloud-storage,bucket=${_WEIGHTS_BUCKET}
+ - --add-volume-mount=volume=weights,mount-path=/weights
+ - --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},MAX_CONCURRENT=1,WEIGHTS_ROOT=/weights/sam3d-objects
+ - --set-secrets=API_KEY=avatar-reconstruction-key:latest
+ - --liveness-probe=httpGet.path=/health,initialDelaySeconds=60,periodSeconds=60,timeoutSeconds=10,failureThreshold=3
+
+images:
+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA
+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:latest
+
+options:
+ machineType: E2_HIGHCPU_32
+ logging: CLOUD_LOGGING_ONLY
+
+timeout: 3600s
diff --git a/workers/model-sam3d-objects/entrypoint.sh b/workers/model-sam3d-objects/entrypoint.sh
new file mode 100755
index 000000000..84343a4d8
--- /dev/null
+++ b/workers/model-sam3d-objects/entrypoint.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+# Wire staged SAM 3D Objects checkpoints into the path pipeline.py loads from
+# (SAM3D_CHECKPOINT_DIR), then serve.
+#
+# Checkpoints are COPIED from the FUSE-mounted bucket to local disk rather than
+# symlinked: torch.load over gcsfuse does slow random reads, and a sequential
+# cp is fast and deterministic.
+set -eu
+
+WEIGHTS_ROOT="${WEIGHTS_ROOT:-/weights/sam3d-objects}"
+SAM3D_CHECKPOINT_DIR="${SAM3D_CHECKPOINT_DIR:-/app/checkpoints/hf}"
+
+if [ ! -e "$WEIGHTS_ROOT/hf/pipeline.yaml" ]; then
+ echo "fatal: expected staged checkpoints at $WEIGHTS_ROOT/hf (run: HF_TOKEN=... npm run stage:sam3d-objects -- --accept-sam-license)" >&2
+ exit 1
+fi
+
+mkdir -p "$(dirname "$SAM3D_CHECKPOINT_DIR")"
+rm -rf "$SAM3D_CHECKPOINT_DIR"
+echo "staging $WEIGHTS_ROOT/hf -> $SAM3D_CHECKPOINT_DIR"
+cp -r "$WEIGHTS_ROOT/hf" "$SAM3D_CHECKPOINT_DIR"
+[ -f "$WEIGHTS_ROOT/LICENSE" ] && cp "$WEIGHTS_ROOT/LICENSE" "$SAM3D_CHECKPOINT_DIR/../SAM_LICENSE" || true
+
+exec uvicorn main:app --host 0.0.0.0 --port 8080 --workers 1 --timeout-keep-alive 300
diff --git a/workers/model-sam3d-objects/main.py b/workers/model-sam3d-objects/main.py
new file mode 100644
index 000000000..152de5506
--- /dev/null
+++ b/workers/model-sam3d-objects/main.py
@@ -0,0 +1,432 @@
+"""
+SAM 3D Objects inference service: one photo in, a 3D scene of its objects out
+(facebookresearch/sam-3d-objects, SAM License; generative single-image 3D
+object reconstruction with scene layout).
+
+This is the Forge / AR Studio "From a photo" lane: the caller names the objects
+in the photo (one mask PNG or one normalized box per object) and gets back one
+GLB laid out like the photo, with a named node per object and a manifest of
+each object's position and size.
+
+API contract (same task shape as the other GPU workers):
+ POST /infer { image_url: https URL to the photo,
+ mask_urls?: [https URL of a PNG mask per object, white = object],
+ boxes?: [[x0, y0, x1, y1] per object, fractions of the frame],
+ max_objects?: int (default 8, max 12), seed?: int, job_id?: str }
+ -> 202 { task_id, status: "queued" }
+
+ GET /tasks/:id -> { task_id, status, result_gcs_url?, result_url?, objects?,
+ object_count?, skipped?, error? }
+ GET /results/:id.glb -> the scene GLB when OUTPUT_DIR local storage is active
+ GET /health -> { ok, model, gpu_available, load_error, load_attempts };
+ 503 once the model load has spent its retry budget.
+ GET / -> unauthenticated descriptor for the platform warmth probe.
+
+Weights: gs://three-ws-model-weights/sam3d-objects (npm run stage:sam3d-objects).
+
+Environment variables (README.md carries the full table):
+ API_KEY, GCS_BUCKET, OUTPUT_DIR, MAX_CONCURRENT (default 1),
+ SAM3D_CHECKPOINT_DIR (default /app/checkpoints/hf), SAM3D_SRC (default /app/sam3d-objects),
+ IMAGE_FETCH_TIMEOUT_S, MODEL_LOAD_ATTEMPTS, MODEL_LOAD_RETRY_BASE_S / _CAP_S
+"""
+
+from __future__ import annotations
+
+import asyncio
+import gc
+import io
+import json
+import logging
+import os
+import time
+import uuid
+from contextlib import asynccontextmanager
+from pathlib import Path
+from typing import Optional
+from urllib.parse import urlsplit
+
+import httpx
+import numpy as np
+from fastapi import BackgroundTasks, FastAPI, Header, HTTPException, Response
+from fastapi.responses import FileResponse
+from google.api_core.exceptions import NotFound
+from google.cloud import storage
+from pydantic import BaseModel, Field
+
+import scene_layout as layout
+from request_policy import RequestRejected, validate_request
+from storage_backend import LocalStorage
+from worker_security import UnsafeUrlError, fetch_remote_bytes, require_api_key, safe_error
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
+log = logging.getLogger("model-sam3d-objects")
+
+API_KEY = os.environ["API_KEY"]
+GCS_BUCKET = os.environ.get("GCS_BUCKET", "").strip()
+OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/output")
+CHECKPOINT_DIR = os.environ.get("SAM3D_CHECKPOINT_DIR", "/app/checkpoints/hf")
+SAM3D_SRC = os.environ.get("SAM3D_SRC", "/app/sam3d-objects")
+MAX_CONCURRENT = int(os.environ.get("MAX_CONCURRENT", "1"))
+MODEL_LOAD_ATTEMPTS = int(os.environ.get("MODEL_LOAD_ATTEMPTS", "4"))
+MODEL_LOAD_RETRY_BASE_S = float(os.environ.get("MODEL_LOAD_RETRY_BASE_S", "15"))
+MODEL_LOAD_RETRY_CAP_S = float(os.environ.get("MODEL_LOAD_RETRY_CAP_S", "120"))
+IMAGE_FETCH_TIMEOUT_S = float(os.environ.get("IMAGE_FETCH_TIMEOUT_S", "60"))
+FETCH_ATTEMPTS = 3
+MAX_IMAGE_BYTES = 32 * 1024 * 1024
+MAX_MASK_BYTES = 16 * 1024 * 1024
+# The model works at a fixed internal resolution; a larger photo only costs memory.
+MAX_PHOTO_SIDE = 2048
+MAX_PHOTO_PIXELS = 40_000_000
+
+_bucket: Optional[storage.Bucket] = None
+_local_storage: Optional[LocalStorage] = None
+_sem: Optional[asyncio.Semaphore] = None
+_ready: Optional[asyncio.Event] = None
+_load_error: Optional[str] = None
+_load_attempts = 0
+_pipeline_loaded = False
+_tasks: dict[str, dict] = {}
+_task_locks: dict[str, asyncio.Lock] = {}
+
+
+def _torch():
+ try:
+ import torch
+
+ return torch
+ except ImportError:
+ return None
+
+
+def _cuda_available() -> bool:
+ torch = _torch()
+ return bool(torch and torch.cuda.is_available())
+
+
+def _load_pipeline_sync() -> None:
+ global _pipeline_loaded
+ from pipeline import load_pipeline
+
+ log.info("Loading SAM 3D Objects: checkpoints=%s", CHECKPOINT_DIR)
+ load_pipeline(CHECKPOINT_DIR, SAM3D_SRC)
+ _pipeline_loaded = True
+ log.info("SAM 3D Objects loaded")
+
+
+async def _load_pipeline_bg() -> None:
+ """Load off the request path; retry with backoff before latching a failure."""
+ global _load_error, _load_attempts
+ loop = asyncio.get_event_loop()
+ attempts = max(1, MODEL_LOAD_ATTEMPTS)
+ for attempt in range(1, attempts + 1):
+ _load_attempts = attempt
+ try:
+ await loop.run_in_executor(None, _load_pipeline_sync)
+ _load_error = None
+ _ready.set()
+ return
+ except Exception as exc: # noqa: BLE001 - surfaced via /health and task status
+ if attempt >= attempts:
+ _load_error = safe_error(exc, context="model load")
+ log.error("SAM 3D Objects load FAILED after %d attempt(s): %s", attempt, exc)
+ return
+ delay = min(MODEL_LOAD_RETRY_BASE_S * (2 ** (attempt - 1)), MODEL_LOAD_RETRY_CAP_S)
+ log.warning("SAM 3D Objects load attempt %d/%d failed (%s); retrying in %.0fs", attempt, attempts, exc, delay)
+ await asyncio.sleep(delay)
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+ global _bucket, _local_storage, _sem, _ready
+ if GCS_BUCKET:
+ _bucket = storage.Client().bucket(GCS_BUCKET)
+ _local_storage = None
+ else:
+ _bucket = None
+ _local_storage = LocalStorage(OUTPUT_DIR)
+ _sem = asyncio.Semaphore(MAX_CONCURRENT)
+ _ready = asyncio.Event()
+ asyncio.create_task(_load_pipeline_bg())
+ log.info("Service starting, pipeline loading in background (max_concurrent=%d)", MAX_CONCURRENT)
+ yield
+
+
+app = FastAPI(title="model-sam3d-objects", lifespan=lifespan)
+
+
+def _require_api_key(authorization: str) -> None:
+ try:
+ require_api_key(authorization, API_KEY)
+ except PermissionError as exc:
+ raise HTTPException(status_code=401, detail=str(exc)) from exc
+
+
+class ImageSourceError(ValueError):
+ """A caller-supplied photo or mask could not be read."""
+
+
+def _source_label(src: str) -> str:
+ return urlsplit(src).hostname or src[:60]
+
+
+def _fetch_bytes(url: str, what: str, max_bytes: int) -> bytes:
+ label = _source_label(url)
+ last_exc: Optional[BaseException] = None
+ for attempt in range(1, FETCH_ATTEMPTS + 1):
+ try:
+ return fetch_remote_bytes(url, timeout=IMAGE_FETCH_TIMEOUT_S, max_bytes=max_bytes)
+ except UnsafeUrlError as exc:
+ raise ImageSourceError(f"refused to fetch {what} ({label}): {exc}") from exc
+ except httpx.HTTPStatusError as exc:
+ raise ImageSourceError(f"{what} source {label} returned HTTP {exc.response.status_code}") from exc
+ except httpx.HTTPError as exc:
+ last_exc = exc
+ if attempt < FETCH_ATTEMPTS:
+ time.sleep(min(2 ** attempt, 5))
+ except ValueError as exc:
+ raise ImageSourceError(f"{what} source {label} rejected: {exc}") from exc
+ raise ImageSourceError(
+ f"{what} source {label} unreachable after {FETCH_ATTEMPTS} attempts "
+ f"({type(last_exc).__name__}); check the URL is publicly readable"
+ )
+
+
+def _load_photo(url: str) -> np.ndarray:
+ from PIL import Image, ImageOps
+
+ data = _fetch_bytes(url, "photo", MAX_IMAGE_BYTES)
+ try:
+ Image.MAX_IMAGE_PIXELS = MAX_PHOTO_PIXELS
+ img = Image.open(io.BytesIO(data))
+ img.load()
+ except Exception as exc: # noqa: BLE001 - PIL raises many types, including decompression bombs
+ raise ImageSourceError("photo is not a readable image (use PNG, JPEG or WebP)") from exc
+ img = ImageOps.exif_transpose(img).convert("RGB")
+ if max(img.size) > MAX_PHOTO_SIDE:
+ scale = MAX_PHOTO_SIDE / max(img.size)
+ img = img.resize((max(1, round(img.width * scale)), max(1, round(img.height * scale))), Image.LANCZOS)
+ return np.asarray(img)
+
+
+def _build_masks(request: dict, photo: np.ndarray) -> tuple[list[np.ndarray], list[str]]:
+ height, width = photo.shape[:2]
+ masks: list[np.ndarray] = []
+ if request["mask_urls"]:
+ for i, url in enumerate(request["mask_urls"]):
+ data = _fetch_bytes(url, f"mask {i + 1}", MAX_MASK_BYTES)
+ try:
+ masks.append(layout.mask_from_png_bytes(data, width, height))
+ except ValueError as exc:
+ raise ImageSourceError(f"mask {i + 1}: {exc}") from exc
+ return masks, ["mask"] * len(masks)
+ for box in request["boxes"]:
+ masks.append(layout.grabcut_mask(photo, layout.box_to_pixels(box, width, height)))
+ return masks, ["box"] * len(masks)
+
+
+def _task_blob(task_id: str, name: str | None = None):
+ if _bucket is None:
+ raise RuntimeError("GCS task storage is not active")
+ return _bucket.blob(name or f"tasks/{task_id}.json")
+
+
+def _using_gcs() -> bool:
+ return bool(GCS_BUCKET)
+
+
+def _get_local_storage() -> LocalStorage:
+ global _local_storage
+ if _local_storage is None:
+ _local_storage = LocalStorage(OUTPUT_DIR)
+ return _local_storage
+
+
+async def _update_task(task_id: str, **fields) -> dict:
+ lock = _task_locks.setdefault(task_id, asyncio.Lock())
+ async with lock:
+ task = _tasks.setdefault(task_id, {"task_id": task_id})
+ task.update(fields)
+ task["updated_at"] = time.time()
+ snapshot = task.copy()
+ loop = asyncio.get_event_loop()
+ if _using_gcs():
+ payload = json.dumps(snapshot)
+ await loop.run_in_executor(
+ None, lambda: _task_blob(task_id).upload_from_string(payload, content_type="application/json")
+ )
+ else:
+ await loop.run_in_executor(None, _get_local_storage().write_task, snapshot)
+ return task
+
+
+_TERMINAL_STATUSES = frozenset({"done", "failed"})
+
+
+async def _resolve_task(task_id: str) -> dict:
+ task = _tasks.get(task_id)
+ if task is not None and task.get("status") in _TERMINAL_STATUSES:
+ return task
+ loop = asyncio.get_event_loop()
+ try:
+ if _using_gcs():
+ data = await loop.run_in_executor(None, _task_blob(task_id).download_as_bytes)
+ task = json.loads(data)
+ else:
+ task = await loop.run_in_executor(None, _get_local_storage().read_task, task_id)
+ except (NotFound, FileNotFoundError, ValueError):
+ if task is not None:
+ return task
+ raise HTTPException(status_code=404, detail="task not found")
+ except Exception as exc:
+ raise HTTPException(status_code=502, detail=safe_error(exc, context="task lookup")) from exc
+ if task.get("status") in _TERMINAL_STATUSES:
+ _tasks[task_id] = task
+ return task
+
+
+def _generate_sync(request: dict) -> dict:
+ from pipeline import run_objects
+
+ photo = _load_photo(request["image_url"])
+ masks, sources = _build_masks(request, photo)
+ return run_objects(photo, masks, sources, request["seed"] if request["seed"] is not None else 42)
+
+
+def _release_gpu_memory() -> None:
+ gc.collect()
+ torch = _torch()
+ if torch and torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ torch.cuda.ipc_collect()
+
+
+async def _run_job(task_id: str, request: dict) -> None:
+ from pipeline import ObjectReconstructionError
+
+ if _load_error:
+ await _update_task(task_id, status="failed", error=f"pipeline unavailable: {_load_error}")
+ return
+ try:
+ await asyncio.wait_for(_ready.wait(), timeout=900)
+ except asyncio.TimeoutError:
+ await _update_task(task_id, status="failed", error="pipeline not ready (model load timed out)")
+ return
+ if _load_error:
+ await _update_task(task_id, status="failed", error=f"pipeline unavailable: {_load_error}")
+ return
+
+ async with _sem:
+ await _update_task(task_id, status="running")
+ loop = asyncio.get_event_loop()
+ t0 = time.time()
+ try:
+ result = await loop.run_in_executor(None, _generate_sync, request)
+ glb_bytes = result["scene_glb"]
+ if _using_gcs():
+ blob_name = f"raw-meshes/sam3d-objects/{task_id}.glb"
+ blob = _bucket.blob(blob_name)
+ await loop.run_in_executor(
+ None, lambda: blob.upload_from_string(glb_bytes, content_type="model/gltf-binary")
+ )
+ result_fields = {"result_gcs_url": f"https://storage.googleapis.com/{GCS_BUCKET}/{blob_name}"}
+ else:
+ result_path = await loop.run_in_executor(None, _get_local_storage().write_result, task_id, glb_bytes)
+ result_fields = {"result_url": f"/results/{task_id}.glb", "result_path": str(result_path)}
+ elapsed = time.time() - t0
+ await _update_task(
+ task_id,
+ status="done",
+ **result_fields,
+ objects=result["objects"],
+ object_count=len(result["objects"]),
+ skipped=result["skipped"],
+ bytes=len(glb_bytes),
+ elapsed_ms=int(elapsed * 1000),
+ )
+ log.info("[%s] done in %.1fs (objects=%d) %d bytes", task_id, elapsed, len(result["objects"]), len(glb_bytes))
+ except (ImageSourceError, ObjectReconstructionError) as exc:
+ log.warning("[%s] rejected: %s", task_id, exc)
+ await _update_task(task_id, status="failed", error=str(exc), elapsed_ms=int((time.time() - t0) * 1000))
+ except Exception as exc:
+ await _update_task(
+ task_id, status="failed", error=safe_error(exc, context=f"[{task_id}] inference"),
+ elapsed_ms=int((time.time() - t0) * 1000),
+ )
+ finally:
+ await loop.run_in_executor(None, _release_gpu_memory)
+
+
+class InferRequest(BaseModel):
+ image_url: str = Field(..., min_length=1, max_length=4096)
+ mask_urls: list[str] | None = None
+ boxes: list[list[float]] | None = None
+ max_objects: int | None = None
+ seed: int | None = None
+ job_id: str | None = None
+
+
[email protected]("/infer", status_code=202)
+async def infer(body: InferRequest, background_tasks: BackgroundTasks, authorization: str = Header(...)) -> dict:
+ _require_api_key(authorization)
+ if _load_error:
+ raise HTTPException(status_code=503, detail=f"pipeline unavailable: {_load_error}")
+ payload = body.model_dump(exclude_none=True)
+ try:
+ request = validate_request(payload)
+ except RequestRejected as exc:
+ raise HTTPException(status_code=422, detail=str(exc)) from exc
+ task_id = str(uuid.uuid4())
+ await _update_task(task_id, status="queued", model="sam3d-objects")
+ background_tasks.add_task(_run_job, task_id, request)
+ return {"task_id": task_id, "status": "queued"}
+
+
[email protected]("/tasks/{task_id}")
+async def get_task(task_id: str, authorization: str = Header(...)) -> dict:
+ _require_api_key(authorization)
+ return await _resolve_task(task_id)
+
+
[email protected]("/results/{task_id}.glb")
+async def get_result(task_id: str, authorization: str = Header(...)):
+ _require_api_key(authorization)
+ if _using_gcs():
+ raise HTTPException(status_code=404, detail="local result storage is not active")
+ try:
+ path = _get_local_storage().result_path(task_id)
+ except ValueError as exc:
+ raise HTTPException(status_code=404, detail="result not found") from exc
+ if not path.is_file():
+ raise HTTPException(status_code=404, detail="result not found")
+ return FileResponse(path, media_type="model/gltf-binary", filename=path.name)
+
+
+async def root() -> dict:
+ return {
+ "service": "model-sam3d-objects",
+ "model": "sam3d-objects",
+ "ready": bool(_ready and _ready.is_set()),
+ "endpoints": ["POST /infer", "GET /tasks/{task_id}", "GET /results/{task_id}.glb", "GET /health"],
+ }
+
+
+async def health(response: Response) -> dict:
+ dead = _load_error is not None
+ if dead:
+ response.status_code = 503
+ torch = _torch()
+ cuda = _cuda_available()
+ return {
+ "ok": not dead,
+ "model": "sam3d-objects",
+ "gpu_available": cuda,
+ "gpu_name": torch.cuda.get_device_name(0) if cuda else None,
+ "gpu_memory_gb": round(torch.cuda.get_device_properties(0).total_memory / 2**30, 1) if cuda else None,
+ "pipeline_loaded": _pipeline_loaded,
+ "ready": bool(_ready and _ready.is_set()),
+ "load_error": _load_error,
+ "load_attempts": _load_attempts,
+ "output_backend": "gcs" if _using_gcs() else "local",
+ }
diff --git a/workers/model-sam3d-objects/pipeline.py b/workers/model-sam3d-objects/pipeline.py
new file mode 100644
index 000000000..1f4e256a4
--- /dev/null
+++ b/workers/model-sam3d-objects/pipeline.py
@@ -0,0 +1,114 @@
+"""SAM 3D Objects (facebookresearch/sam-3d-objects, SAM License) pipeline wrapper.
+
+One photo plus one mask per object in, one posed mesh per object out. Loading
+and running the model needs torch, pytorch3d, kaolin and a 32 GB+ GPU, so every
+heavy import lives inside a function: this module imports on a laptop and the
+pure logic (masks, boxes, layout) is covered by test_scene_layout.py.
+
+Upstream exposes its inference class only as notebook/inference.py in the
+repository (it is not part of the pip package), so the Dockerfile leaves the
+checkout at SAM3D_SRC and this module puts SAM3D_SRC/notebook on sys.path,
+exactly as upstream's README does.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import sys
+import threading
+from pathlib import Path
+
+import numpy as np
+
+import scene_layout as layout
+
+log = logging.getLogger("model-sam3d-objects.pipeline")
+
+_lock = threading.Lock()
+_inference = None
+
+MIN_OBJECT_FACES = 8
+
+
+class ObjectReconstructionError(RuntimeError):
+ """The model ran but produced no usable mesh for any requested object."""
+
+
+def load_pipeline(checkpoint_dir: str, sam3d_src: str):
+ """Load (once) and return upstream's Inference singleton."""
+ global _inference
+ with _lock:
+ if _inference is None:
+ config = Path(checkpoint_dir) / "pipeline.yaml"
+ if not config.is_file():
+ raise FileNotFoundError(
+ f"{config} is missing: stage the SAM 3D Objects checkpoints with "
+ "`npm run stage:sam3d-objects -- --accept-sam-license`"
+ )
+ # upstream's inference.py reads these at import time
+ os.environ.setdefault("CONDA_PREFIX", os.environ.get("CUDA_HOME", "/usr/local/cuda"))
+ os.environ.setdefault("LIDRA_SKIP_INIT", "true")
+ notebook = str(Path(sam3d_src) / "notebook")
+ if notebook not in sys.path:
+ sys.path.append(notebook)
+ from inference import Inference # type: ignore[import-not-found]
+
+ log.info("loading SAM 3D Objects: config=%s", config)
+ _inference = Inference(str(config), compile=False)
+ return _inference
+
+
+def _to_numpy(value) -> np.ndarray:
+ if hasattr(value, "detach"):
+ value = value.detach().float().cpu().numpy()
+ return np.asarray(value, dtype=np.float64)
+
+
+def run_objects(image_rgb: np.ndarray, masks: list[np.ndarray], sources: list[str], seed: int) -> dict:
+ """Reconstruct every masked object and lay the results out like the photo.
+
+ Returns {"scene_glb": bytes, "objects": [manifest entries], "skipped": [...]}.
+ The scene GLB has one named node per object (object_01, object_02, ...).
+ A single object that fails is skipped and reported; the call fails only when
+ none succeeds.
+ """
+ inference = _inference
+ if inference is None:
+ raise RuntimeError("pipeline is not loaded")
+
+ posed_items: list[dict] = []
+ manifest: list[dict] = []
+ skipped: list[dict] = []
+
+ for i, (mask, source) in enumerate(zip(masks, sources)):
+ try:
+ out = inference(image_rgb, mask, seed=seed)
+ mesh = out.get("glb")
+ if mesh is None or len(mesh.faces) < MIN_OBJECT_FACES:
+ raise ObjectReconstructionError("the model returned no mesh for this object")
+ verts = np.asarray(mesh.vertices, dtype=np.float64)
+ faces = np.asarray(mesh.faces, dtype=np.int64)
+ colors = None
+ vis = getattr(mesh, "visual", None)
+ if vis is not None and getattr(vis, "kind", None) == "vertex":
+ colors = np.asarray(vis.vertex_colors)
+ quat = _to_numpy(out["rotation"]).reshape(-1)[:4]
+ trans = _to_numpy(out["translation"]).reshape(-1)[:3]
+ scale = _to_numpy(out["scale"]).reshape(-1)
+ posed = layout.pose_vertices(verts, quat, trans, scale)
+ solo = layout.standalone_vertices(verts, scale)
+ except Exception as exc: # noqa: BLE001 - one bad object must not sink the scene
+ log.warning("object %d failed: %s", i + 1, exc)
+ skipped.append({"index": i, "reason": str(exc)[:200]})
+ continue
+ n = len(posed_items)
+ posed_items.append({"vertices": posed, "faces": faces, "vertex_colors": colors})
+ manifest.append(layout.manifest_entry(n, posed, faces, solo, quat, trans, scale, sources[i]))
+
+ if not posed_items:
+ reasons = "; ".join(s["reason"] for s in skipped) or "no objects"
+ raise ObjectReconstructionError(f"no object could be reconstructed ({reasons})")
+
+ scene_glb = layout.export_glb(layout.build_scene(posed_items))
+ return {"scene_glb": scene_glb, "objects": manifest, "skipped": skipped}
diff --git a/workers/model-sam3d-objects/request_policy.py b/workers/model-sam3d-objects/request_policy.py
new file mode 100644
index 000000000..8936004ed
--- /dev/null
+++ b/workers/model-sam3d-objects/request_policy.py
@@ -0,0 +1,107 @@
+"""Request validation for the SAM 3D Objects worker.
+
+Pure Python (stdlib only) so it unit-tests without a GPU, torch or FastAPI.
+Everything a caller can send is bounded here, before any URL is fetched or any
+GPU time is spent, and every rejection is a caller-facing sentence.
+
+A request names ONE photo and the objects to lift out of it. SAM 3D Objects
+reconstructs an object from the photo plus a mask that says which pixels are
+that object, so each object is given as either:
+
+ * ``mask_urls``: one public https PNG per object (white = the object), or
+ * ``boxes``: one normalized ``[x0, y0, x1, y1]`` box per object (0..1, origin
+ top-left); the worker turns each box into a mask with GrabCut.
+
+Masks win when both are present for the same request (a mask is exact, a box is
+an estimate), and the two are never mixed within one request.
+"""
+
+from __future__ import annotations
+
+import math
+from typing import Any
+
+# One photo can hold a roomful, but each object is a full diffusion pass and the
+# scene is one GLB a phone has to open. Twelve is a generous room, not a stress test.
+MAX_OBJECTS = 12
+DEFAULT_MAX_OBJECTS = 8
+MIN_BOX_SIDE = 0.04 # 4% of the frame: smaller is a speck the model cannot reconstruct
+MIN_BOX_AREA = 0.002
+MAX_URL_CHARS = 4096
+SEED_MAX = 4_294_967_295
+
+
+class RequestRejected(ValueError):
+ """The request is malformed; the message is safe to show the caller."""
+
+
+def _url(value: Any, field: str) -> str:
+ if not isinstance(value, str) or not value.strip():
+ raise RequestRejected(f"{field} must be a non-empty https URL")
+ value = value.strip()
+ if len(value) > MAX_URL_CHARS:
+ raise RequestRejected(f"{field} is longer than {MAX_URL_CHARS} characters")
+ return value
+
+
+def _box(raw: Any, index: int) -> tuple[float, float, float, float]:
+ label = f"boxes[{index}]"
+ if not isinstance(raw, (list, tuple)) or len(raw) != 4:
+ raise RequestRejected(f"{label} must be [x0, y0, x1, y1]")
+ try:
+ x0, y0, x1, y1 = (float(v) for v in raw)
+ except (TypeError, ValueError) as exc:
+ raise RequestRejected(f"{label} must contain numbers") from exc
+ if not all(math.isfinite(v) for v in (x0, y0, x1, y1)):
+ raise RequestRejected(f"{label} must contain finite numbers")
+ if not (0.0 <= x0 < x1 <= 1.0 and 0.0 <= y0 < y1 <= 1.0):
+ raise RequestRejected(f"{label} must satisfy 0 <= x0 < x1 <= 1 and 0 <= y0 < y1 <= 1 (fractions of the frame)")
+ if (x1 - x0) < MIN_BOX_SIDE or (y1 - y0) < MIN_BOX_SIDE or (x1 - x0) * (y1 - y0) < MIN_BOX_AREA:
+ raise RequestRejected(f"{label} is too small to reconstruct; choose an object that fills at least 4% of the frame width and height")
+ return (x0, y0, x1, y1)
+
+
+def validate_request(payload: dict) -> dict:
+ """Return a normalized request or raise RequestRejected."""
+ if not isinstance(payload, dict):
+ raise RequestRejected("request body must be a JSON object")
+
+ image_url = _url(payload.get("image_url"), "image_url")
+
+ max_objects = payload.get("max_objects", DEFAULT_MAX_OBJECTS)
+ if isinstance(max_objects, bool) or not isinstance(max_objects, int) or not 1 <= max_objects <= MAX_OBJECTS:
+ raise RequestRejected(f"max_objects must be an integer from 1 to {MAX_OBJECTS}")
+
+ seed = payload.get("seed")
+ if seed is not None and (isinstance(seed, bool) or not isinstance(seed, int) or not 0 <= seed <= SEED_MAX):
+ raise RequestRejected("seed must be a non-negative integer")
+
+ raw_masks = payload.get("mask_urls")
+ raw_boxes = payload.get("boxes")
+ if raw_masks is not None and raw_boxes is not None and raw_masks and raw_boxes:
+ raise RequestRejected("send mask_urls or boxes, not both")
+
+ mask_urls: list[str] = []
+ boxes: list[tuple[float, float, float, float]] = []
+ if raw_masks:
+ if not isinstance(raw_masks, list):
+ raise RequestRejected("mask_urls must be a list of https URLs")
+ mask_urls = [_url(u, f"mask_urls[{i}]") for i, u in enumerate(raw_masks)]
+ elif raw_boxes:
+ if not isinstance(raw_boxes, list):
+ raise RequestRejected("boxes must be a list of [x0, y0, x1, y1] boxes")
+ boxes = [_box(b, i) for i, b in enumerate(raw_boxes)]
+ else:
+ raise RequestRejected("name the objects to reconstruct: send mask_urls (one PNG mask per object) or boxes (one [x0, y0, x1, y1] per object)")
+
+ count = len(mask_urls) or len(boxes)
+ if count > max_objects:
+ raise RequestRejected(f"{count} objects requested but max_objects is {max_objects}")
+
+ return {
+ "image_url": image_url,
+ "mask_urls": mask_urls,
+ "boxes": boxes,
+ "max_objects": max_objects,
+ "seed": seed,
+ }
diff --git a/workers/model-sam3d-objects/requirements.txt b/workers/model-sam3d-objects/requirements.txt
new file mode 100644
index 000000000..9b908acb5
--- /dev/null
+++ b/workers/model-sam3d-objects/requirements.txt
@@ -0,0 +1,15 @@
+# This worker's own FastAPI/GCS layer. torch, pytorch3d, kaolin, gsplat and the
+# rest of SAM 3D Objects' stack install in their own Dockerfile layers BEFORE
+# this file (upstream is installed from its pinned checkout).
+trimesh>=4.4.0,<5
+numpy>=1.24,<2
+Pillow>=10,<12
+
+fastapi>=0.115,<1
+uvicorn[standard]>=0.32,<1
+pydantic>=2.0.0
+httpx>=0.27.0
+
+google-cloud-storage==2.18.0
+
+pytest>=8,<9
diff --git a/workers/model-sam3d-objects/scene_layout.py b/workers/model-sam3d-objects/scene_layout.py
new file mode 100644
index 000000000..de4aa9d7a
--- /dev/null
+++ b/workers/model-sam3d-objects/scene_layout.py
@@ -0,0 +1,189 @@
+"""Pure geometry for the SAM 3D Objects worker: masks, poses and the scene GLB.
+
+numpy, PIL and trimesh only (cv2 is imported lazily for GrabCut), so it
+unit-tests without torch or a GPU. The model-facing code lives in pipeline.py.
+
+Conventions, from facebookresearch/sam-3d-objects at the pinned commit:
+
+ * ``output["glb"]`` is a trimesh in a Y-up frame. Upstream's ``to_glb`` makes
+ it from the decoder's native Z-up vertices with ``v @ DECODER_TO_GLB``.
+ * ``output["rotation"|"translation"|"scale"]`` place the object in the camera
+ frame of the photo, applied to the decoder-native vertices (upstream's
+ ``make_scene`` applies them to the gaussians' native xyz). The rotation is a
+ PyTorch3D quaternion, real part first.
+ * The camera frame is PyTorch3D's: +X left, +Y up, +Z away from the camera.
+ glTF viewers look down -Z with +X right, so the final conversion flips X and Z.
+
+The returned scene therefore reads like the photo: the camera sits at the origin
+looking down -Z, objects stand where the photo shows them, at the model's own
+relative scale (the layout is relative, not metric: no tape measure was in frame).
+"""
+
+from __future__ import annotations
+
+import io
+from typing import Sequence
+
+import numpy as np
+
+# upstream to_glb: vertices @ [[1, 0, 0], [0, 0, -1], [0, 1, 0]] => (x, y, z) -> (x, z, -y)
+_DECODER_TO_GLB = np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]], dtype=np.float64)
+_GLB_TO_DECODER = _DECODER_TO_GLB.T # orthonormal: the inverse is the transpose
+_CAMERA_TO_GLTF = np.diag([-1.0, 1.0, -1.0])
+
+
+def glb_to_decoder_vertices(vertices: np.ndarray) -> np.ndarray:
+ """Undo upstream's decoder-to-GLB axis change."""
+ return np.asarray(vertices, dtype=np.float64) @ _GLB_TO_DECODER
+
+
+def quat_wxyz_to_matrix(q: Sequence[float]) -> np.ndarray:
+ """Rotation matrix for a real-first quaternion (PyTorch3D convention)."""
+ w, x, y, z = (float(v) for v in q)
+ n = w * w + x * x + y * y + z * z
+ if not np.isfinite(n) or n < 1e-12:
+ raise ValueError("quaternion has no rotation")
+ s = 2.0 / n
+ return np.array(
+ [
+ [1 - s * (y * y + z * z), s * (x * y - z * w), s * (x * z + y * w)],
+ [s * (x * y + z * w), 1 - s * (x * x + z * z), s * (y * z - x * w)],
+ [s * (x * z - y * w), s * (y * z + x * w), 1 - s * (x * x + y * y)],
+ ],
+ dtype=np.float64,
+ )
+
+
+def pose_vertices(
+ glb_vertices: np.ndarray,
+ quat_wxyz: Sequence[float],
+ translation: Sequence[float],
+ scale: Sequence[float] | float,
+) -> np.ndarray:
+ """Place an object's GLB-frame vertices in the glTF scene frame of the photo."""
+ native = glb_to_decoder_vertices(glb_vertices)
+ scale_vec = np.broadcast_to(np.asarray(scale, dtype=np.float64).reshape(-1), (3,)) if np.ndim(scale) else np.full(3, float(scale))
+ posed = (native * scale_vec) @ quat_wxyz_to_matrix(quat_wxyz).T + np.asarray(translation, dtype=np.float64).reshape(3)
+ return posed @ _CAMERA_TO_GLTF.T
+
+
+def standalone_vertices(glb_vertices: np.ndarray, scale: Sequence[float] | float) -> np.ndarray:
+ """The object on its own: scaled, centred on X/Z and resting on y = 0."""
+ verts = np.asarray(glb_vertices, dtype=np.float64)
+ scale_vec = np.broadcast_to(np.asarray(scale, dtype=np.float64).reshape(-1), (3,)) if np.ndim(scale) else np.full(3, float(scale))
+ # Scale in the decoder frame, where the model's axes mean what the scale means.
+ native = glb_to_decoder_vertices(verts) * scale_vec
+ out = native @ _DECODER_TO_GLB
+ lo, hi = out.min(axis=0), out.max(axis=0)
+ out[:, 0] -= (lo[0] + hi[0]) / 2.0
+ out[:, 2] -= (lo[2] + hi[2]) / 2.0
+ out[:, 1] -= lo[1]
+ return out
+
+
+def box_to_pixels(box: Sequence[float], width: int, height: int) -> tuple[int, int, int, int]:
+ """Normalized [x0, y0, x1, y1] to an integer pixel rectangle inside the image."""
+ x0, y0, x1, y1 = box
+ px0 = min(max(int(round(x0 * width)), 0), width - 1)
+ py0 = min(max(int(round(y0 * height)), 0), height - 1)
+ px1 = min(max(int(round(x1 * width)), px0 + 1), width)
+ py1 = min(max(int(round(y1 * height)), py0 + 1), height)
+ return px0, py0, px1, py1
+
+
+def mask_from_png_bytes(data: bytes, width: int, height: int) -> np.ndarray:
+ """Decode a caller's mask PNG to a boolean (height, width) array.
+
+ An RGBA mask uses its alpha channel when the alpha is not constant,
+ otherwise its luminance. The mask is resized to the photo with nearest
+ neighbour so a slightly different export size still lines up.
+ """
+ from PIL import Image
+
+ try:
+ img = Image.open(io.BytesIO(data))
+ img.load()
+ except Exception as exc: # noqa: BLE001 - PIL raises many types for a bad file
+ raise ValueError("mask is not a readable image") from exc
+ if img.mode in ("RGBA", "LA"):
+ alpha = np.asarray(img.getchannel("A"))
+ channel = alpha if alpha.min() != alpha.max() else np.asarray(img.convert("L"))
+ else:
+ channel = np.asarray(img.convert("L"))
+ if channel.shape != (height, width):
+ channel = np.asarray(Image.fromarray(channel).resize((width, height), Image.NEAREST))
+ mask = channel > 127
+ if not mask.any():
+ raise ValueError("mask is empty (no white pixels)")
+ return mask
+
+
+def grabcut_mask(image_rgb: np.ndarray, box_px: tuple[int, int, int, int], iterations: int = 5) -> np.ndarray:
+ """Estimate an object mask from a box with GrabCut (OpenCV, Apache-2.0)."""
+ import cv2
+
+ x0, y0, x1, y1 = box_px
+ h, w = image_rgb.shape[:2]
+ mask = np.zeros((h, w), np.uint8)
+ rect = (x0, y0, x1 - x0, y1 - y0)
+ bgd = np.zeros((1, 65), np.float64)
+ fgd = np.zeros((1, 65), np.float64)
+ cv2.grabCut(cv2.cvtColor(image_rgb, cv2.COLOR_RGB2BGR), mask, rect, bgd, fgd, iterations, cv2.GC_INIT_WITH_RECT)
+ out = (mask == cv2.GC_FGD) | (mask == cv2.GC_PR_FGD)
+ box_area = max((x1 - x0) * (y1 - y0), 1)
+ # GrabCut can collapse to almost nothing on a low-contrast box; the box itself
+ # is then the honest answer, because an empty mask would fail the model.
+ if out.sum() < 0.05 * box_area:
+ out = np.zeros((h, w), bool)
+ out[y0:y1, x0:x1] = True
+ return out
+
+
+def object_name(index: int) -> str:
+ return f"object_{index + 1:02d}"
+
+
+def build_scene(objects: Sequence[dict]):
+ """One trimesh.Scene, one named node per object, laid out like the photo.
+
+ Each item needs ``vertices`` (already posed), ``faces`` and optional
+ ``vertex_colors``.
+ """
+ import trimesh
+
+ scene = trimesh.Scene()
+ for i, obj in enumerate(objects):
+ name = object_name(i)
+ mesh = trimesh.Trimesh(vertices=obj["vertices"], faces=obj["faces"], process=False)
+ if obj.get("vertex_colors") is not None:
+ mesh.visual.vertex_colors = obj["vertex_colors"]
+ scene.add_geometry(mesh, node_name=name, geom_name=name)
+ return scene
+
+
+def export_glb(scene_or_mesh) -> bytes:
+ buf = io.BytesIO()
+ scene_or_mesh.export(buf, file_type="glb")
+ return buf.getvalue()
+
+
+def _round(values, n=4):
+ return [round(float(v), n) for v in np.asarray(values).reshape(-1)]
+
+
+def manifest_entry(index: int, posed: np.ndarray, faces: np.ndarray, standalone: np.ndarray,
+ quat_wxyz, translation, scale, source: str) -> dict:
+ """Machine-readable record of one reconstructed object."""
+ lo, hi = posed.min(axis=0), posed.max(axis=0)
+ size = standalone.max(axis=0) - standalone.min(axis=0)
+ return {
+ "id": object_name(index),
+ "name": object_name(index),
+ "source": source, # "mask" or "box"
+ "face_count": int(len(faces)),
+ "vertex_count": int(len(posed)),
+ "bbox": {"min": _round(lo), "max": _round(hi)},
+ "centroid": _round((lo + hi) / 2.0),
+ "size": _round(size), # width, height, depth of the object on its own, layout units
+ "pose": {"translation": _round(translation), "rotation_wxyz": _round(quat_wxyz, 6), "scale": _round(scale)},
+ }
diff --git a/workers/model-sam3d-objects/storage_backend.py b/workers/model-sam3d-objects/storage_backend.py
new file mode 100644
index 000000000..e90e2752d
--- /dev/null
+++ b/workers/model-sam3d-objects/storage_backend.py
@@ -0,0 +1,63 @@
+"""Storage primitives for the SAM 3D Objects worker.
+
+Production persists tasks and results in GCS because requests can move between
+Cloud Run instances. An NGC user runs one container with a mounted output
+volume, so the same service needs a local backend with atomic writes and strict
+task-id validation.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import uuid
+from pathlib import Path
+
+
+def normalize_task_id(task_id: str) -> str:
+ """Return a canonical UUID or reject a value that could escape the volume."""
+ parsed = uuid.UUID(task_id)
+ canonical = str(parsed)
+ if task_id.lower() != canonical:
+ raise ValueError("task_id must be a canonical UUID")
+ return canonical
+
+
+class LocalStorage:
+ """Durable task metadata and GLB results on a mounted local volume."""
+
+ def __init__(self, output_dir: str | os.PathLike[str]):
+ self.output_dir = Path(output_dir).expanduser().resolve()
+ self.tasks_dir = self.output_dir / "tasks"
+ self.results_dir = self.output_dir / "results"
+ self.tasks_dir.mkdir(parents=True, exist_ok=True)
+ self.results_dir.mkdir(parents=True, exist_ok=True)
+
+ def task_path(self, task_id: str) -> Path:
+ return self.tasks_dir / f"{normalize_task_id(task_id)}.json"
+
+ def result_path(self, task_id: str) -> Path:
+ return self.results_dir / f"{normalize_task_id(task_id)}.glb"
+
+ @staticmethod
+ def _atomic_write(path: Path, data: bytes) -> None:
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ with temporary.open("wb") as handle:
+ handle.write(data)
+ handle.flush()
+ os.fsync(handle.fileno())
+ temporary.replace(path)
+
+ def write_task(self, task: dict) -> Path:
+ path = self.task_path(str(task["task_id"]))
+ payload = json.dumps(task, separators=(",", ":"), sort_keys=True).encode("utf-8")
+ self._atomic_write(path, payload)
+ return path
+
+ def read_task(self, task_id: str) -> dict:
+ return json.loads(self.task_path(task_id).read_text(encoding="utf-8"))
+
+ def write_result(self, task_id: str, glb_bytes: bytes) -> Path:
+ path = self.result_path(task_id)
+ self._atomic_write(path, glb_bytes)
+ return path
diff --git a/workers/model-sam3d-objects/test_app_contract.py b/workers/model-sam3d-objects/test_app_contract.py
new file mode 100644
index 000000000..4740cf389
--- /dev/null
+++ b/workers/model-sam3d-objects/test_app_contract.py
@@ -0,0 +1,93 @@
+"""Smoke test for the served contract. Runs INSIDE the built image (and on any
+machine with fastapi, httpx, Pillow, trimesh and google-cloud-storage):
+
+ docker build -t model-sam3d-objects workers/model-sam3d-objects
+ docker run --rm model-sam3d-objects python3 test_app_contract.py
+
+It needs no GPU, no weights and no GCP credentials: the ASGI lifespan (which
+opens the GCS client and starts the model load) is deliberately not run, so
+every assertion is about request handling, not inference.
+
+What it proves:
+ 1. The whole app module imports.
+ 2. The auth boundary rejects a wrong bearer on both authenticated routes.
+ 3. Malformed requests are refused with 422 and a sentence, before a task exists.
+ 4. The unauthenticated probes answer, and a dead pipeline says so on /health
+ and /infer so the caller fails over.
+ 5. The photo and mask loaders refuse everything the SSRF guard should refuse.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+
+os.environ.setdefault("API_KEY", "smoke-test-key")
+os.environ.setdefault("GCS_BUCKET", "smoke-test-bucket")
+
+KEY = os.environ["API_KEY"]
+PASSED = 0
+
+
+def check(name: str, condition: bool, detail: str = "") -> None:
+ global PASSED
+ if not condition:
+ print(f"FAIL {name} {detail}")
+ sys.exit(1)
+ PASSED += 1
+ print(f"ok {name}")
+
+
+def main() -> int:
+ import main as app_module
+ from fastapi.testclient import TestClient
+
+ client = TestClient(app_module.app) # no `with`: lifespan stays off
+ good = {"Authorization": f"Bearer {KEY}"}
+ bad = {"Authorization": "Bearer wrong"}
+ photo = "https://example.com/room.jpg"
+
+ check("imports", app_module.app.title == "model-sam3d-objects")
+
+ r = client.post("/infer", json={"image_url": photo, "boxes": [[0.1, 0.1, 0.5, 0.5]]}, headers=bad)
+ check("infer rejects a wrong key", r.status_code == 401, str(r.status_code))
+ r = client.get("/tasks/00000000-0000-4000-8000-000000000000", headers=bad)
+ check("tasks rejects a wrong key", r.status_code == 401, str(r.status_code))
+
+ r = client.post("/infer", json={"image_url": photo}, headers=good)
+ check("infer without objects is 422", r.status_code == 422 and "name the objects" in r.text, r.text)
+ r = client.post("/infer", json={"image_url": photo, "boxes": [[0.5, 0.5, 0.4, 0.4]]}, headers=good)
+ check("infer with an inverted box is 422", r.status_code == 422 and "boxes[0]" in r.text, r.text)
+ r = client.post(
+ "/infer",
+ json={"image_url": photo, "boxes": [[0.1, 0.1, 0.5, 0.5]], "mask_urls": ["https://example.com/m.png"]},
+ headers=good,
+ )
+ check("infer with masks and boxes is 422", r.status_code == 422, r.text)
+
+ r = client.get("/")
+ check("descriptor answers", r.status_code == 200 and r.json()["model"] == "sam3d-objects")
+ r = client.get("/health")
+ check("health answers while loading", r.status_code == 200 and r.json()["ok"] is True)
+
+ for url in ("http://169.254.169.254/latest/meta-data", "https://localhost/x.png", "file:///etc/passwd"):
+ try:
+ app_module._load_photo(url)
+ except app_module.ImageSourceError:
+ check(f"photo loader refuses {url}", True)
+ else:
+ check(f"photo loader refuses {url}", False)
+
+ app_module._load_error = "weights missing"
+ r = client.get("/health")
+ check("dead pipeline drains on /health", r.status_code == 503 and r.json()["ok"] is False, r.text)
+ r = client.post("/infer", json={"image_url": photo, "boxes": [[0.1, 0.1, 0.5, 0.5]]}, headers=good)
+ check("dead pipeline fails /infer fast", r.status_code == 503, str(r.status_code))
+ app_module._load_error = None
+
+ print(f"\n{PASSED} checks passed")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/workers/model-sam3d-objects/test_request_policy.py b/workers/model-sam3d-objects/test_request_policy.py
new file mode 100644
index 000000000..252179dc5
--- /dev/null
+++ b/workers/model-sam3d-objects/test_request_policy.py
@@ -0,0 +1,84 @@
+import pytest
+
+from request_policy import MAX_OBJECTS, RequestRejected, validate_request
+
+PHOTO = "https://example.com/room.jpg"
+BOX = [0.1, 0.2, 0.5, 0.8]
+
+
+def test_boxes_request_is_normalized():
+ out = validate_request({"image_url": PHOTO, "boxes": [BOX, [0.5, 0.5, 0.9, 0.9]]})
+ assert out["image_url"] == PHOTO
+ assert out["mask_urls"] == []
+ assert out["boxes"][0] == (0.1, 0.2, 0.5, 0.8)
+ assert out["max_objects"] == 8
+ assert out["seed"] is None
+
+
+def test_masks_request():
+ out = validate_request({"image_url": PHOTO, "mask_urls": ["https://example.com/a.png"], "seed": 7})
+ assert out["mask_urls"] == ["https://example.com/a.png"]
+ assert out["boxes"] == []
+ assert out["seed"] == 7
+
+
[email protected]("payload", [None, [], "x", 3])
+def test_body_must_be_an_object(payload):
+ with pytest.raises(RequestRejected):
+ validate_request(payload)
+
+
+def test_photo_required():
+ with pytest.raises(RequestRejected, match="image_url"):
+ validate_request({"boxes": [BOX]})
+
+
+def test_must_name_objects():
+ with pytest.raises(RequestRejected, match="name the objects"):
+ validate_request({"image_url": PHOTO})
+ with pytest.raises(RequestRejected, match="name the objects"):
+ validate_request({"image_url": PHOTO, "boxes": [], "mask_urls": []})
+
+
+def test_masks_and_boxes_are_exclusive():
+ with pytest.raises(RequestRejected, match="not both"):
+ validate_request({"image_url": PHOTO, "boxes": [BOX], "mask_urls": ["https://example.com/a.png"]})
+
+
+ "box",
+ [
+ [0.5, 0.2, 0.4, 0.8], # x1 < x0
+ [-0.1, 0.2, 0.5, 0.8], # outside the frame
+ [0.1, 0.2, 0.5, 1.2],
+ [0.1, 0.2, 0.12, 0.8], # sliver
+ [0.1, 0.2, 0.5], # wrong arity
+ [0.1, 0.2, "a", 0.8],
+ [0.1, 0.2, float("nan"), 0.8],
+ ],
+)
+def test_bad_boxes_are_rejected(box):
+ with pytest.raises(RequestRejected, match=r"boxes\[0\]"):
+ validate_request({"image_url": PHOTO, "boxes": [box]})
+
+
+def test_object_cap():
+ with pytest.raises(RequestRejected, match="max_objects"):
+ validate_request({"image_url": PHOTO, "boxes": [BOX] * 3, "max_objects": 2})
+ with pytest.raises(RequestRejected, match="max_objects"):
+ validate_request({"image_url": PHOTO, "boxes": [BOX], "max_objects": MAX_OBJECTS + 1})
+ with pytest.raises(RequestRejected, match="max_objects"):
+ validate_request({"image_url": PHOTO, "boxes": [BOX], "max_objects": True})
+
+
[email protected]("seed", [-1, 1.5, True, "3"])
+def test_bad_seed(seed):
+ with pytest.raises(RequestRejected, match="seed"):
+ validate_request({"image_url": PHOTO, "boxes": [BOX], "seed": seed})
+
+
+def test_url_limits():
+ with pytest.raises(RequestRejected, match="image_url"):
+ validate_request({"image_url": "x" * 5000, "boxes": [BOX]})
+ with pytest.raises(RequestRejected, match=r"mask_urls\[0\]"):
+ validate_request({"image_url": PHOTO, "mask_urls": [""]})
diff --git a/workers/model-sam3d-objects/test_scene_layout.py b/workers/model-sam3d-objects/test_scene_layout.py
new file mode 100644
index 000000000..76b8ede28
--- /dev/null
+++ b/workers/model-sam3d-objects/test_scene_layout.py
@@ -0,0 +1,126 @@
+import io
+
+import numpy as np
+import pytest
+import trimesh
+from PIL import Image
+
+import scene_layout as sl
+
+
+def test_identity_quaternion_is_identity():
+ assert np.allclose(sl.quat_wxyz_to_matrix([1, 0, 0, 0]), np.eye(3))
+
+
+def test_quaternion_is_normalized_and_rotates_correctly():
+ # 90 degrees about +Z, real part first, deliberately not unit length
+ q = [2 * np.cos(np.pi / 4), 0, 0, 2 * np.sin(np.pi / 4)]
+ m = sl.quat_wxyz_to_matrix(q)
+ assert np.allclose(m @ [1, 0, 0], [0, 1, 0], atol=1e-9)
+ assert np.allclose(m @ m.T, np.eye(3), atol=1e-9)
+
+
+def test_zero_quaternion_rejected():
+ with pytest.raises(ValueError):
+ sl.quat_wxyz_to_matrix([0, 0, 0, 0])
+
+
+def test_decoder_axis_change_round_trips():
+ v = np.random.default_rng(0).normal(size=(50, 3))
+ glb = v @ sl._DECODER_TO_GLB
+ assert np.allclose(sl.glb_to_decoder_vertices(glb), v)
+
+
+def test_pose_puts_object_in_front_of_the_camera():
+ # An object centred in the decoder frame, pushed 3 units away from the camera
+ # (+Z in the model's camera frame) must end up at -Z in a glTF viewer.
+ box = trimesh.creation.box(extents=(1, 1, 1))
+ glb_vertices = np.asarray(box.vertices) @ sl._DECODER_TO_GLB
+ posed = sl.pose_vertices(glb_vertices, [1, 0, 0, 0], [0, 0, 3], 1.0)
+ centre = posed.mean(axis=0)
+ assert np.allclose(centre, [0, 0, -3], atol=1e-9)
+
+
+def test_pose_flips_left_right_into_gltf_x():
+ box = trimesh.creation.box(extents=(1, 1, 1))
+ glb_vertices = np.asarray(box.vertices) @ sl._DECODER_TO_GLB
+ # +X in the PyTorch3D camera frame is LEFT; in glTF +X is right.
+ posed = sl.pose_vertices(glb_vertices, [1, 0, 0, 0], [2, 0, 5], 1.0)
+ assert posed.mean(axis=0)[0] == pytest.approx(-2.0)
+
+
+def test_pose_scale_is_uniform_or_per_axis():
+ v = np.array([[1.0, 1.0, 1.0], [-1.0, -1.0, -1.0]]) @ sl._DECODER_TO_GLB
+ a = sl.pose_vertices(v, [1, 0, 0, 0], [0, 0, 0], 2.0)
+ b = sl.pose_vertices(v, [1, 0, 0, 0], [0, 0, 0], [2.0, 2.0, 2.0])
+ assert np.allclose(a, b)
+ assert np.abs(a).max() == pytest.approx(2.0)
+
+
+def test_standalone_rests_on_the_floor_and_is_centred():
+ box = trimesh.creation.box(extents=(2, 4, 6))
+ glb_vertices = np.asarray(box.vertices) @ sl._DECODER_TO_GLB
+ solo = sl.standalone_vertices(glb_vertices, 0.5)
+ assert solo[:, 1].min() == pytest.approx(0.0)
+ assert (solo[:, 0].min() + solo[:, 0].max()) / 2 == pytest.approx(0.0)
+ assert (solo[:, 2].min() + solo[:, 2].max()) / 2 == pytest.approx(0.0)
+ assert solo.max(axis=0)[1] - solo.min(axis=0)[1] == pytest.approx(3.0) # 6 * 0.5 along decoder Z = glb Y
+
+
+def test_box_to_pixels_clamps_and_keeps_area():
+ assert sl.box_to_pixels((0.0, 0.0, 1.0, 1.0), 200, 100) == (0, 0, 200, 100)
+ x0, y0, x1, y1 = sl.box_to_pixels((0.999, 0.999, 1.0, 1.0), 200, 100)
+ assert x1 > x0 and y1 > y0 and x1 <= 200 and y1 <= 100
+
+
+def _png(array, mode):
+ buf = io.BytesIO()
+ Image.fromarray(array, mode).save(buf, format="PNG")
+ return buf.getvalue()
+
+
+def test_mask_decodes_luminance_and_resizes_to_the_photo():
+ arr = np.zeros((10, 10), np.uint8)
+ arr[2:8, 2:8] = 255
+ mask = sl.mask_from_png_bytes(_png(arr, "L"), 20, 20)
+ assert mask.shape == (20, 20)
+ assert mask[10, 10] and not mask[0, 0]
+
+
+def test_mask_prefers_alpha_when_it_varies():
+ rgba = np.zeros((8, 8, 4), np.uint8)
+ rgba[..., :3] = 255 # all-white colour, so luminance alone would select everything
+ rgba[2:4, 2:4, 3] = 255
+ mask = sl.mask_from_png_bytes(_png(rgba, "RGBA"), 8, 8)
+ assert mask.sum() == 4
+
+
+def test_empty_and_corrupt_masks_are_rejected():
+ with pytest.raises(ValueError, match="empty"):
+ sl.mask_from_png_bytes(_png(np.zeros((4, 4), np.uint8), "L"), 4, 4)
+ with pytest.raises(ValueError, match="readable"):
+ sl.mask_from_png_bytes(b"not an image", 4, 4)
+
+
+def test_scene_has_one_named_node_per_object_and_round_trips():
+ items = []
+ for i in range(3):
+ box = trimesh.creation.box(extents=(1, 1, 1))
+ items.append({"vertices": box.vertices + [i * 2, 0, -3], "faces": box.faces,
+ "vertex_colors": np.tile([200, 40, 40, 255], (len(box.vertices), 1)).astype(np.uint8)})
+ glb = sl.export_glb(sl.build_scene(items))
+ loaded = trimesh.load(io.BytesIO(glb), file_type="glb")
+ assert set(loaded.graph.nodes_geometry) == {"object_01", "object_02", "object_03"}
+ assert len(loaded.geometry) == 3
+
+
+def test_manifest_entry_reports_position_and_size():
+ box = trimesh.creation.box(extents=(1, 2, 3))
+ posed = box.vertices + [4, 0, -5]
+ solo = sl.standalone_vertices(np.asarray(box.vertices), 1.0)
+ entry = sl.manifest_entry(1, posed, box.faces, solo, [1, 0, 0, 0], [0, 0, 5], [1, 1, 1], "box")
+ assert entry["id"] == "object_02"
+ assert entry["source"] == "box"
+ assert entry["centroid"] == pytest.approx([4, 0, -5])
+ assert entry["face_count"] == len(box.faces)
+ assert entry["pose"]["rotation_wxyz"] == [1.0, 0.0, 0.0, 0.0]
diff --git a/workers/model-sam3d-objects/worker_security.py b/workers/model-sam3d-objects/worker_security.py
new file mode 100644
index 000000000..802e1c32d
--- /dev/null
+++ b/workers/model-sam3d-objects/worker_security.py
@@ -0,0 +1,320 @@
+"""
+Shared security helpers for the three.ws GPU/CPU workers.
+
+This module is intentionally dependency-light (stdlib only) so it can be
+COPYed verbatim into every worker's Docker build context. The same canonical
+file lives in each worker directory; keep them byte-identical when editing.
+
+It provides three primitives that every worker needs:
+
+ 1. ``require_api_key`` - constant-time bearer-token check (timing-safe).
+ 2. ``fetch_remote_bytes`` - SSRF-hardened HTTP(S) fetch of caller-supplied
+ URLs: https-only, DNS resolution with private/loopback/link-local/metadata
+ IP rejection, per-hop redirect re-validation, and a bounded response size.
+ 3. ``safe_error`` - turns an exception into an opaque, correlation-id-tagged
+ message safe to persist/return to callers, after logging the full
+ traceback server-side only.
+"""
+
+from __future__ import annotations
+
+import hmac
+import ipaddress
+import logging
+import asyncio
+import socket
+import time
+import uuid
+from typing import Iterable, Optional
+from urllib.parse import urlsplit
+
+import httpx
+
+log = logging.getLogger("worker_security")
+
+# ── auth ────────────────────────────────────────────────────────────────────────
+
+
+def require_api_key(authorization: Optional[str], api_key: str) -> None:
+ """Validate a ``Bearer <token>`` header in constant time.
+
+ Raises ``PermissionError`` on any failure so callers can translate it into
+ the framework's 401 (FastAPI workers map this to HTTPException(401)).
+ """
+ # Fail closed on an empty configured key: with api_key == "" a bare
+ # "Bearer " header would otherwise strip to "" and compare equal.
+ if not api_key:
+ raise PermissionError("worker api key not configured")
+ if not authorization or not authorization.startswith("Bearer "):
+ raise PermissionError("missing bearer token")
+ token = authorization[len("Bearer "):].strip()
+ if not token:
+ raise PermissionError("missing bearer token")
+ # Compare bytes: hmac.compare_digest is timing-safe, and on str operands it
+ # raises TypeError for non-ASCII input, which would surface as a 500.
+ if not hmac.compare_digest(token.encode("utf-8"), api_key.encode("utf-8")):
+ raise PermissionError("invalid api key")
+
+
+# ── SSRF-hardened fetch ─────────────────────────────────────────────────────────
+
+# Default cap on a fetched response. Avatar selfies / driving audio / raw meshes
+# are comfortably under this; anything larger is almost certainly abuse.
+DEFAULT_MAX_BYTES = 64 * 1024 * 1024 # 64 MiB
+_MAX_REDIRECTS = 4
+
+# Transient fetch failures are retried: a dropped connection, a timeout, or a
+# 429/5xx from the image host. The public R2 dev endpoint that serves every
+# forge reference image drops connections under load ("Server disconnected
+# without sending a response"), and before this retry each drop ended a
+# generation as an opaque internal error on TripoSG and Hunyuan3D (about 20 a
+# week measured 2026-10-09). Refusals (UnsafeUrlError), oversize bodies, and
+# other 4xx answers are final and never retried.
+_FETCH_ATTEMPTS = 3
+_FETCH_BACKOFF_SECS = (0.5, 1.5)
+
+# Hosts that enforce a User-Agent policy refuse httpx's default
+# "python-httpx/<version>": upload.wikimedia.org answers it with 403 Forbidden
+# and serves the same image to a descriptive agent (measured 2026-10-09, when
+# model-triposg failed every job whose reference photo was a Wikimedia link).
+# Callers may still pass their own User-Agent; it replaces this one.
+DEFAULT_USER_AGENT = "three.ws-worker/1.0 (+https://three.ws)"
+
+
+def _fetch_headers(headers: Optional[dict]) -> dict:
+ """Caller headers over the default User-Agent, matched case-insensitively."""
+ merged = dict(headers or {})
+ if not any(str(k).lower() == "user-agent" for k in merged):
+ merged["User-Agent"] = DEFAULT_USER_AGENT
+ return merged
+
+
+class UnsafeUrlError(ValueError):
+ """Raised when a URL targets a disallowed scheme or a private/internal IP."""
+
+
+def _is_blocked_ip(ip: ipaddress._BaseAddress) -> bool:
+ """Reject any address that could reach internal infrastructure.
+
+ Covers loopback, RFC1918 private ranges, link-local (incl. the cloud
+ metadata server at 169.254.169.254), unique-local IPv6 (fc00::/7),
+ multicast, reserved, unspecified, and any other non-global address (CGNAT
+ 100.64.0.0/10 included). IPv4-mapped IPv6 addresses
+ are unwrapped first so a mapped private v4 can't slip through.
+ """
+ if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
+ ip = ip.ipv4_mapped
+ return (
+ ip.is_private
+ or ip.is_loopback
+ or ip.is_link_local
+ or ip.is_multicast
+ or ip.is_reserved
+ or ip.is_unspecified
+ # Anything not globally routable, which also covers the CGNAT shared
+ # space 100.64.0.0/10 that is_private does not flag.
+ or not ip.is_global
+ )
+
+
+def _resolve_safe_host(host: str) -> None:
+ """DNS-resolve ``host`` and raise UnsafeUrlError if ANY address is internal.
+
+ Resolving and checking every returned address closes the gap where a name
+ resolves to both a public and a private address.
+ """
+ if not host:
+ raise UnsafeUrlError("missing host")
+ try:
+ infos = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
+ except socket.gaierror as exc:
+ raise UnsafeUrlError(f"could not resolve host: {host}") from exc
+
+ addrs = {info[4][0] for info in infos}
+ if not addrs:
+ raise UnsafeUrlError(f"no addresses for host: {host}")
+ for addr in addrs:
+ ip = ipaddress.ip_address(addr)
+ if _is_blocked_ip(ip):
+ raise UnsafeUrlError(f"host resolves to a disallowed address: {host}")
+
+
+def assert_safe_url(url: str, *, allow_http: bool = False) -> str:
+ """Validate scheme + resolved IPs for a single URL. Returns it unchanged.
+
+ https is required by default; ``allow_http`` exists only for callers that
+ legitimately need cleartext (not used by the workers).
+ """
+ parts = urlsplit(url)
+ scheme = parts.scheme.lower()
+ allowed = ("https", "http") if allow_http else ("https",)
+ if scheme not in allowed:
+ raise UnsafeUrlError(f"scheme not allowed: {scheme or '(none)'}")
+ if not parts.hostname:
+ raise UnsafeUrlError("missing host")
+ _resolve_safe_host(parts.hostname)
+ return url
+
+
+def fetch_remote_bytes(
+ url: str,
+ *,
+ timeout: float = 30.0,
+ max_bytes: int = DEFAULT_MAX_BYTES,
+ allow_http: bool = False,
+ headers: Optional[dict] = None,
+) -> bytes:
+ """Fetch ``url`` safely and return its body bytes.
+
+ Hardening applied:
+ * https-only (unless ``allow_http``).
+ * every URL - including each redirect target - is DNS-resolved and
+ rejected if it maps to a private/loopback/link-local/metadata address.
+ * redirects are followed manually (max ``_MAX_REDIRECTS``) so each hop is
+ re-validated; httpx's own redirect following is disabled.
+ * the response body is streamed and aborted once it exceeds ``max_bytes``.
+ * transient failures (dropped connection, timeout, 429/5xx) are retried
+ with a short backoff; see ``_FETCH_ATTEMPTS``.
+
+ Raises ``UnsafeUrlError`` for disallowed targets and ``httpx.HTTPError`` /
+ ``ValueError`` for transport or size failures.
+ """
+ for attempt in range(_FETCH_ATTEMPTS - 1):
+ try:
+ return _fetch_once(
+ url, timeout=timeout, max_bytes=max_bytes,
+ allow_http=allow_http, headers=headers,
+ )
+ except httpx.HTTPError as exc:
+ if not _is_transient(exc):
+ raise
+ log.warning("transient fetch failure (attempt %d): %s", attempt + 1, type(exc).__name__)
+ time.sleep(_FETCH_BACKOFF_SECS[attempt])
+ return _fetch_once(
+ url, timeout=timeout, max_bytes=max_bytes,
+ allow_http=allow_http, headers=headers,
+ )
+
+
+def _is_transient(exc: httpx.HTTPError) -> bool:
+ if isinstance(exc, httpx.HTTPStatusError):
+ code = exc.response.status_code
+ return code == 429 or code >= 500
+ return isinstance(exc, httpx.TransportError)
+
+
+def _fetch_once(
+ url: str,
+ *,
+ timeout: float,
+ max_bytes: int,
+ allow_http: bool,
+ headers: Optional[dict],
+) -> bytes:
+ current = assert_safe_url(url, allow_http=allow_http)
+ with httpx.Client(follow_redirects=False, timeout=timeout) as client:
+ for _ in range(_MAX_REDIRECTS + 1):
+ with client.stream("GET", current, headers=_fetch_headers(headers)) as resp:
+ if resp.is_redirect:
+ location = resp.headers.get("location")
+ if not location:
+ raise UnsafeUrlError("redirect without Location header")
+ current = assert_safe_url(
+ str(resp.url.join(location)), allow_http=allow_http
+ )
+ continue
+ resp.raise_for_status()
+ declared = resp.headers.get("content-length")
+ if declared is not None and declared.isdigit() and int(declared) > max_bytes:
+ raise ValueError(f"response too large: {declared} bytes")
+ chunks: list[bytes] = []
+ total = 0
+ for chunk in resp.iter_bytes(65536):
+ total += len(chunk)
+ if total > max_bytes:
+ raise ValueError(f"response exceeded {max_bytes} bytes")
+ chunks.append(chunk)
+ return b"".join(chunks)
+ raise UnsafeUrlError(f"too many redirects (>{_MAX_REDIRECTS})")
+
+
+async def fetch_remote_bytes_async(
+ client: "httpx.AsyncClient",
+ url: str,
+ *,
+ max_bytes: int = DEFAULT_MAX_BYTES,
+ allow_http: bool = False,
+ headers: Optional[dict] = None,
+) -> bytes:
+ """Async variant of ``fetch_remote_bytes`` using a caller-provided client.
+
+ The provided ``client`` MUST be constructed with ``follow_redirects=False``
+ so this function can re-validate each redirect hop itself. Transient
+ failures are retried exactly as in the sync variant.
+ """
+ for attempt in range(_FETCH_ATTEMPTS - 1):
+ try:
+ return await _fetch_once_async(
+ client, url, max_bytes=max_bytes,
+ allow_http=allow_http, headers=headers,
+ )
+ except httpx.HTTPError as exc:
+ if not _is_transient(exc):
+ raise
+ log.warning("transient fetch failure (attempt %d): %s", attempt + 1, type(exc).__name__)
+ await asyncio.sleep(_FETCH_BACKOFF_SECS[attempt])
+ return await _fetch_once_async(
+ client, url, max_bytes=max_bytes,
+ allow_http=allow_http, headers=headers,
+ )
+
+
+async def _fetch_once_async(
+ client: "httpx.AsyncClient",
+ url: str,
+ *,
+ max_bytes: int,
+ allow_http: bool,
+ headers: Optional[dict],
+) -> bytes:
+ current = assert_safe_url(url, allow_http=allow_http)
+ for _ in range(_MAX_REDIRECTS + 1):
+ async with client.stream("GET", current, headers=_fetch_headers(headers)) as resp:
+ if resp.is_redirect:
+ location = resp.headers.get("location")
+ if not location:
+ raise UnsafeUrlError("redirect without Location header")
+ current = assert_safe_url(
+ str(resp.url.join(location)), allow_http=allow_http
+ )
+ continue
+ resp.raise_for_status()
+ declared = resp.headers.get("content-length")
+ if declared is not None and declared.isdigit() and int(declared) > max_bytes:
+ raise ValueError(f"response too large: {declared} bytes")
+ chunks: list[bytes] = []
+ total = 0
+ async for chunk in resp.aiter_bytes(65536):
+ total += len(chunk)
+ if total > max_bytes:
+ raise ValueError(f"response exceeded {max_bytes} bytes")
+ chunks.append(chunk)
+ return b"".join(chunks)
+ raise UnsafeUrlError(f"too many redirects (>{_MAX_REDIRECTS})")
+
+
+# ── opaque error handling ───────────────────────────────────────────────────────
+
+
+def safe_error(exc: BaseException, *, context: str = "") -> str:
+ """Log the full traceback server-side and return an opaque client message.
+
+ Never leak ``traceback.format_exc()`` (absolute paths, library versions,
+ internal hostnames, fragments of upstream responses) to callers. A short
+ correlation id is embedded in both the log line and the returned string so
+ operators can join a client-reported failure to the server log.
+ """
+ correlation_id = uuid.uuid4().hex[:12]
+ label = f"{context} " if context else ""
+ log.exception("%serror [correlation_id=%s]", label, correlation_id)
+ return f"internal error (ref {correlation_id})"