Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
feat(forge): scene from a photo with SAM 3D Objects, built end to end and off until the gated weights are staged
nirholase96d1eae95b1
diff --git a/STRUCTURE.md b/STRUCTURE.mdindex 81fd8efef..d7e135abe 100644--- a/STRUCTURE.md+++ b/STRUCTURE.md@@ -221,6 +221,7 @@ packages, a generated standalone GitHub mirror). See| Animation pipeline | [public/animations/](public/animations) + [scripts/build-animations.mjs](scripts/build-animations.mjs) | In-repo | Mixamo source FBX + GLB retargeting pipeline · see [docs/3d-asset-pipeline.md](docs/3d-asset-pipeline.md) || Procedural animation (runtime IK layers) | [src/procedural/](src/procedural) — analytic two-bone IK [two-bone-ik.js](src/procedural/two-bone-ik.js), gaze [look-at.js](src/procedural/look-at.js), terrain foot planting [foot-plant.js](src/procedural/foot-plant.js), drift-free re-application [pose-baseline.js](src/procedural/pose-baseline.js) | Live | Post-mixer layers that do what no baked clip can: the chest/neck/head chain turns toward a world point, and each foot lands on its own patch of uneven ground. Rig-agnostic via `canonicalNodeMapFromObject`; an unsupported rig reports `enabled === false` and no-ops. Wired into the corner companion ([walk-sdk/src/companion.js](walk-sdk/src/companion.js) `_updateGaze`), the walkaround world ([src/walk.js](src/walk.js) `rebuildFootPlant`), and agent avatars ([src/agent-avatar.js](src/agent-avatar.js) `_applyLookTarget` — which is what finally makes `setLookTarget()` and the `LOOK_AT` action move the rig). Ambient idle motion (breathing/blink/saccade/weight-shift) stays in [src/idle-animation.js](src/idle-animation.js). Docs: [docs/procedural-animation.md](docs/procedural-animation.md). Tests: [tests/procedural-anim.test.js](tests/procedural-anim.test.js), [tests/agent-avatar-look-target.test.js](tests/agent-avatar-look-target.test.js) || Self-hosted 3D GPU fleet (the engines behind the Forge) | [workers/model-trellis2/](workers/model-trellis2) (TRELLIS.2, MIT, the default image-to-3D lane `trellis2`: PBR GLB with alpha, `resolution` 512/1024/1536; built and tested, deploy owner-gated), [workers/model-pixal3d/](workers/model-pixal3d) (TencentARC Pixal3D, MIT, on TRELLIS.2's backbone: the `pixal3d` fidelity lane labeled "Match my photo", 1 to 4 pixel-aligned input photos with optional `view_labels` and one-photo `fill_other_sides` view synthesis via MV-Adapter (Apache-2.0); built and tested, deploy owner-gated), territory router [api/_lib/forge-territory.js](api/_lib/forge-territory.js) (EU, UK and KR never reach a Tencent-model lane; unknown country fails closed), [workers/model-trellis/](workers/model-trellis), [workers/model-hunyuan3d/](workers/model-hunyuan3d) (2.0 + 2.1 PBR + 2.1 RTX), [workers/model-triposg/](workers/model-triposg), [workers/modly/](workers/modly) (Modly v0.4.3 running its TripoSG and Hunyuan3D 2 Mini Turbo extensions as the untextured Forge lanes `modly` and `modly_hunyuan`, behind `MODEL_MODLY_URL`; weights staged, built and tested, deploy owner-gated), [workers/model-triposr/](workers/model-triposr), [workers/model-text2motion/](workers/model-text2motion), [workers/rig/](workers/rig) (auto-rigging, deployed as `model-rig`), [workers/rig-nonhuman/](workers/rig-nonhuman) (UniRig + SkinTokens, MIT, the non-humanoid rigging fallback `model-rig` reaches for when its own `mesh_classify.py` heuristic rules out a Mixamo-shaped humanoid, behind `RIG_NONHUMAN_URL`; built and tested, deploy owner-gated), [workers/model-anigen/](workers/model-anigen) (VAST-AI-Research AniGen, MIT, the rig-on-generation Forge lane `anigen` labeled "Rig on generate": predicts mesh, joint skeleton and skin weights together from a single reference image, so the GLB it returns needs no separate rerig call; opt-in only, never a default or failover target; built and tested, deploy owner-gated), [workers/model-partcrafter/](workers/model-partcrafter) (PartCrafter, MIT, LICENSE read 2026-10-11, the `partcrafter` Forge lane labeled "Editable parts": decodes a reference photo straight into N individually named part meshes in one diffusion-transformer pass; built and tested, deploy owner-gated), [workers/model-holopart/](workers/model-holopart) (HoloPart, MIT, LICENSE read 2026-10-11, the "Complete hidden parts" lane behind `/api/forge-holopart`: takes an already part-segmented mesh (from `workers/segment` or `model-partcrafter`) and fills in the occluded geometry of every part, closing each into a standalone shape; built and tested, deploy owner-gated), [workers/model-triposplat/](workers/model-triposplat) (VAST-AI-Research TripoSplat, MIT, LICENSE read 2026-10-12: the image-to-Gaussian-splat lane behind `POST /api/forge-splat`, output opens in the Splat Viewer; behind `MODEL_TRIPOSPLAT_URL`; built and tested, deploy owner-gated), [workers/avatar-reconstruction/](workers/avatar-reconstruction), CPU stages [workers/rembg/](workers/rembg) · [workers/remesh/](workers/remesh) · [workers/segment/](workers/segment) · [workers/stylize/](workers/stylize), talking-avatar video [workers/longcat/](workers/longcat) (LongCat-Video-Avatar-1.5, gates the `/create/video` lane via `LONGCAT_WORKER_URL`; the one fleet member that is not a Cloud Run service, because its ~45 GB resident weight set needs an 80 GB GPU, so it is built and tested but undeployed), streaming video→3D [workers/model-video2scene/](workers/model-video2scene), selfie-pipeline CPU orchestrator [workers/avatar-pipeline-controller/](workers/avatar-pipeline-controller) | Live · Cloud Run GPU (us-central1, plus us-east4 where trellis/hunyuan/triposr hold standbys and `model-text2motion` runs as its production lane) | Every generation engine the Forge dispatches to, each with its own `cloudbuild.yaml`. The L4 quota is **per region** and both us-central1 and us-east4 are granted 3, so a warm pin in one region only starves lanes in that same region (increase to 16 filed 2026-07-16, still reconciling); the Hunyuan3D-2.1 PBR realism lane also ships an RTX PRO 6000 build (`model-hunyuan3d-21-rtx`, quota granted 1000) as the warm primary. `model-text2motion` (MDM default + MoMask, both MIT, selectable per request via `model: "mdm" | "momask"`) is the text→motion lane behind `GCP_TEXT2MOTION_URL`, the `text_to_animation` MCP tool, and `POST /api/forge-motion`: prompt in, a retargetable three.js `AnimationClip` out, applied with the same engine (`src/animation-retarget.js`) as the curated animation library. It is the one GPU service whose production URL points at us-east4, and its `cloudbuild.yaml` deploys both regions from one image so the standby never lags. Ops map: [docs/ops/gcp-credits-plan.md](docs/ops/gcp-credits-plan.md) |+| Scene from a photo (SAM 3D Objects) | [workers/model-sam3d-objects/](workers/model-sam3d-objects) (FastAPI worker, 32 GB+ GPU, pinned upstream), platform [api/forge-scene-objects.js](api/forge-scene-objects.js) + [api/_lib/forge-objects.js](api/_lib/forge-objects.js), provider mode `sam3d` in [api/_providers/gcp.js](api/_providers/gcp.js), AR Studio tab "From a photo" in [src/ar-studio.js](src/ar-studio.js), weights staging [scripts/stage-sam3d-objects.mjs](scripts/stage-sam3d-objects.mjs) (`npm run stage:sam3d-objects`) | Built, off until the owner stages the gated weights and deploys | One photo becomes one GLB with a named node per object, laid out like the picture. `503 backend_unconfigured` until `MODEL_SAM3D_URL` is set. Weights are gated by the SAM License on Hugging Face (manual approval); the endpoint refuses comprehensively sanctioned territories. Tests: [tests/forge-objects.test.js](tests/forge-objects.test.js), [tests/api/forge-scene-objects.test.js](tests/api/forge-scene-objects.test.js), [tests/api/gcp-sam3d-provider.test.js](tests/api/gcp-sam3d-provider.test.js), [tests/stage-sam3d-objects.test.js](tests/stage-sam3d-objects.test.js), worker `pytest`. || Self-hosted Granite Guardian (governance lane behind watsonx) | [workers/granite-guardian/](workers/granite-guardian/README.md) (Cloud Run `granite-guardian`, europe-west4, vLLM on one L4), client lane in [api/_lib/granite-guardian.js](api/_lib/granite-guardian.js) | Live · Cloud Run GPU (europe-west4) | IBM's open-weight `ibm-granite/granite-guardian-3.3-8b` served by the stock vLLM OpenAI image, so `/api/guardian/assess`, the identity integrity check, and the Granite Twin/Oracle/Attest/IRL screens return real Granite Guardian verdicts on a deployment with no IBM Cloud account. watsonx.ai stays the lead lane when configured; every verdict names the `model` and `provider` that produced it. Placed in europe-west4 because that region's L4 grant is otherwise unused, so its warm pin never starves the 3D lanes. Tests: [tests/api-guardian.test.js](tests/api-guardian.test.js), [tests/api/identity-integrity.test.js](tests/api/identity-integrity.test.js) || Reconstruct completion backstop (abandoned selfie jobs) | [api/cron/reconstruct-sweep.js](api/cron/reconstruct-sweep.js), every 5 min via `vercel.json` crons → Cloud Scheduler | Live 2026-07-25 | A reconstruct job used to be advanced **only by the user's browser**: `/create/selfie` polls `regenerate-status`, and each poll runs the finalize stages. Close the tab and the worker still finished the GLB and stored it durably, but nothing ever collected it — 26 jobs were stranded that way, oldest from 2026-05-31, invisible to `db-retention` (which prunes only terminal rows). This sweep is the server-side driver: it picks up reconstruct jobs quiet >3 min and runs **the same shared stages** in [reconstruct-finalize.js](api/_lib/reconstruct-finalize.js), so an abandoned job still lands in the user's library. 30-day rescue window (the worker's GCS blobs + Firestore job state are durable, so a weeks-old job is still deliverable); past that, or when a job's provider is no longer pollable from platform credentials, rows fail out with an honest reason so the open set can't grow unbounded. BYOK (meshy/tripo) jobs stay browser-driven — a cron has no user context to resolve their keys. Sibling of [auto-rig-sweep.js](api/cron/auto-rig-sweep.js), which plays the same role for the rerig lane. Docs: [docs/avatar-reconstruction.md](docs/avatar-reconstruction.md#who-finishes-the-job) || Avatar fidelity evaluation (selfie → avatar, scored) | [workers/avatar-reconstruction/eval/](workers/avatar-reconstruction/eval) | In-repo · gates every reconstruction change | **Identity Shape Error (ISE)** — does the reconstructed avatar have *this person's* face shape? Texture-blind, pose- and scale-invariant, dimensionless: MediaPipe's 468 landmarks and the avatar's corresponding head vertices are both Umeyama-aligned onto the neutral canonical face over a stable identity subset (no eyes/lips, so a smile is not scored as identity), then compared. A pipeline cannot score better by resizing or tilting the head — only by matching shape, a property pinned in [eval/test_identity_eval.py](workers/avatar-reconstruction/eval/test_identity_eval.py) alongside monotonicity. [`run_eval.py`](workers/avatar-reconstruction/eval/run_eval.py) scores a whole reference set against a live worker and diffs two revisions, surfacing per-face regressions a mean would hide; [`tune_morph.py`](workers/avatar-reconstruction/eval/tune_morph.py) sweeps the morph parameters against ISE offline; [`robustness.py`](workers/avatar-reconstruction/eval/robustness.py) measures the opposing axis (does a blurred / turned / occluded / dim / low-res photo of the same person still yield the same head?), and the shipped `max_displacement_frac` is the knee of those two curves. The 40-face reference set is **synthesised** by [`make_refs.py`](workers/avatar-reconstruction/eval/make_refs.py) over a deterministic grid of the platform's diversity matrix — no real biometrics, no licence terms, rebuildable by any agent — gitignored and mirrored to `gs://three-ws-avatar-reconstructions/eval-refs/`. Docs: [eval/README.md](workers/avatar-reconstruction/eval/README.md), program plan [docs/avatar-fidelity-program.md](docs/avatar-fidelity-program.md) |diff --git a/api/_lib/forge-objects.js b/api/_lib/forge-objects.jsnew file mode 100644index 000000000..ef10bbca6--- /dev/null+++ b/api/_lib/forge-objects.js@@ -0,0 +1,126 @@+/**+ * Helpers for the "scene from a photo" lane (api/forge-scene-objects.js).+ *+ * The SAM 3D Objects worker reconstructs the objects a caller names in a photo.+ * Two pure functions live here so they can be tested without a network:+ *+ * - normalizeDetectedBoxes: turn a vision model's object list into the clean,+ * bounded, de-duplicated boxes the worker accepts.+ * - isSanctionedTerritory: the SAM License refuses comprehensively sanctioned+ * jurisdictions, so those requests never reach the worker.+ *+ * And one that talks to the vision chain: detectObjectBoxes.+ */++import { describeImageJson } from './vision.js';++export const MAX_SCENE_OBJECTS = 12;+export const DEFAULT_SCENE_OBJECTS = 8;+export const MIN_BOX_SIDE = 0.04;+export const MIN_BOX_AREA = 0.002;+const DUPLICATE_IOU = 0.7;++/**+ * Comprehensively sanctioned jurisdictions (ISO 3166-1 alpha-2) the SAM License+ * and Hugging Face both refuse. Only a country we positively resolved as one of+ * these is blocked: an unknown country is not, because the guard exists to honour+ * the licence, not to turn away every visitor behind a proxy.+ */+export const SANCTIONED_COUNTRIES = Object.freeze(['CU', 'IR', 'KP', 'SY']);++export function isSanctionedTerritory(country) {+ return typeof country === 'string' && SANCTIONED_COUNTRIES.includes(country.toUpperCase());+}++function iou(a, b) {+ const ix = Math.max(0, Math.min(a[2], b[2]) - Math.max(a[0], b[0]));+ const iy = Math.max(0, Math.min(a[3], b[3]) - Math.max(a[1], b[1]));+ const inter = ix * iy;+ const union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter;+ return union > 0 ? inter / union : 0;+}++/**+ * Accepts the vision model's `{ objects: [{ label, box_2d: [ymin, xmin, ymax, xmax] }] }`+ * (0..1000, the convention Gemini is trained on) and returns+ * `[{ label, box: [x0, y0, x1, y1] }]` in 0..1 fractions of the frame, largest+ * first, with slivers, out-of-order boxes and near-duplicates dropped.+ */+export function normalizeDetectedBoxes(json, maxObjects = DEFAULT_SCENE_OBJECTS) {+ const list = Array.isArray(json) ? json : Array.isArray(json?.objects) ? json.objects : [];+ const cap = Math.min(Math.max(1, Math.floor(maxObjects) || DEFAULT_SCENE_OBJECTS), MAX_SCENE_OBJECTS);+ const candidates = [];+ for (const item of list) {+ const raw = item?.box_2d ?? item?.box;+ if (!Array.isArray(raw) || raw.length !== 4) continue;+ const nums = raw.map(Number);+ if (!nums.every(Number.isFinite)) continue;+ const [ymin, xmin, ymax, xmax] = nums.map((v) => Math.min(Math.max(v / 1000, 0), 1));+ if (!(xmin < xmax && ymin < ymax)) continue;+ if (xmax - xmin < MIN_BOX_SIDE || ymax - ymin < MIN_BOX_SIDE) continue;+ if ((xmax - xmin) * (ymax - ymin) < MIN_BOX_AREA) continue;+ const label = typeof item.label === 'string' ? item.label.replace(/\s+/g, ' ').trim().slice(0, 60) : '';+ candidates.push({ label, box: [xmin, ymin, xmax, ymax].map((v) => Math.round(v * 10000) / 10000) });+ }+ candidates.sort(+ (a, b) => (b.box[2] - b.box[0]) * (b.box[3] - b.box[1]) - (a.box[2] - a.box[0]) * (a.box[3] - a.box[1]),+ );+ const kept = [];+ for (const c of candidates) {+ if (kept.some((k) => iou(k.box, c.box) > DUPLICATE_IOU)) continue;+ kept.push(c);+ if (kept.length >= cap) break;+ }+ return kept;+}++/**+ * Strictly validate caller-supplied boxes (`[x0, y0, x1, y1]`, fractions of the+ * frame). Unlike detected boxes these are never clamped or reordered: a caller+ * who sent a bad box is told which one, not silently given a different object.+ * Returns `{ boxes }` or `{ error }`.+ */+export function parseUserBoxes(raw, maxObjects = MAX_SCENE_OBJECTS) {+ if (!Array.isArray(raw) || raw.length === 0) return { error: 'boxes must be a non-empty list.' };+ if (raw.length > maxObjects) return { error: `${raw.length} boxes sent but max_objects is ${maxObjects}.` };+ const boxes = [];+ for (let i = 0; i < raw.length; i++) {+ const b = raw[i];+ if (!Array.isArray(b) || b.length !== 4 || !b.every((v) => typeof v === 'number' && Number.isFinite(v))) {+ return { error: `boxes[${i}] must be [x0, y0, x1, y1] as four numbers.` };+ }+ const [x0, y0, x1, y1] = b;+ if (!(x0 >= 0 && y0 >= 0 && x1 <= 1 && y1 <= 1 && x0 < x1 && y0 < y1)) {+ return { error: `boxes[${i}] must satisfy 0 <= x0 < x1 <= 1 and 0 <= y0 < y1 <= 1 (fractions of the frame).` };+ }+ if (x1 - x0 < MIN_BOX_SIDE || y1 - y0 < MIN_BOX_SIDE || (x1 - x0) * (y1 - y0) < MIN_BOX_AREA) {+ return { error: `boxes[${i}] is too small to reconstruct; it must cover at least 4% of the frame width and height.` };+ }+ for (let j = 0; j < i; j++) {+ if (iou(boxes[j], b) > DUPLICATE_IOU) {+ return { error: `boxes[${i}] overlaps boxes[${j}] by more than 70%; they name the same object.` };+ }+ }+ boxes.push([x0, y0, x1, y1]);+ }+ return { boxes };+}++const DETECT_PROMPT = (max) =>+ `List the distinct, separable physical objects in this photo, up to ${max}, most prominent first. ` ++ 'Skip walls, floors, ceilings, sky, ground, and anything cut off by more than half the frame edge. ' ++ 'Reply ONLY with JSON: {"objects":[{"label":"short name","box_2d":[ymin,xmin,ymax,xmax]}]} ' ++ 'where box_2d is the tight bounding box in whole numbers from 0 to 1000 (fractions of the image height ' ++ 'and width times 1000, origin top-left).';++/** Ask the vision chain to box the objects in a photo. Throws on a vision outage. */+export async function detectObjectBoxes(imageUrl, maxObjects = DEFAULT_SCENE_OBJECTS) {+ const result = await describeImageJson({+ prompt: DETECT_PROMPT(maxObjects),+ imageUrl,+ maxTokens: 1024,+ timeoutMs: 25_000,+ deadlineMs: 45_000,+ });+ return normalizeDetectedBoxes(result.json, maxObjects);+}diff --git a/api/_providers/gcp.js b/api/_providers/gcp.jsindex 17f1af862..1a274c109 100644--- a/api/_providers/gcp.js+++ b/api/_providers/gcp.js@@ -33,6 +33,10 @@// triposplat → MODEL_TRIPOSPLAT_URL (workers/model-triposplat, MIT)// One image → 3D Gaussian splat (.splat + .ply). Standard// /infer + /tasks/:id shape; the result is a splat, not a GLB.+// sam3d → MODEL_SAM3D_URL (workers/model-sam3d-objects, SAM License)+// One photo → one GLB with a named node per object, laid out+// like the photo. Needs a 32 GB+ GPU. Standard /infer ++// /tasks/:id shape.// remesh → GCP_REMESH_URL (workers/remesh)// stylize → GCP_STYLIZE_URL (workers/stylize)// retex → GCP_TEXTURE_URL (workers/texture)@@ -139,6 +143,9 @@ function serviceUrlForMode(mode) {// output of `segment`), preserving part node names. Same// /infer + /tasks/:id task shape.return readEnv('MODEL_HOLOPART_URL');+ case 'sam3d':+ // Self-hosted SAM 3D Objects worker (workers/model-sam3d-objects).+ return readEnv('MODEL_SAM3D_URL');case 'triposplat':// Self-hosted TripoSplat worker (workers/model-triposplat): image to// Gaussian splat. Same /infer + /tasks/:id task shape.@@ -309,6 +316,22 @@ function buildWorkerRequest(request) {};}+ if (mode === 'sam3d') {+ // Self-hosted SAM 3D Objects: one photo plus either per-object masks or+ // boxes (never both). The worker rebuilds each object and lays them out+ // like the photo in a single scene GLB.+ const body = { image_url: sourceUrl };+ if (Array.isArray(params?.maskUrls) && params.maskUrls.length) body.mask_urls = params.maskUrls;+ else if (Array.isArray(params?.boxes) && params.boxes.length) body.boxes = params.boxes;+ if (Number.isFinite(Number(params?.maxObjects))) body.max_objects = Math.round(Number(params.maxObjects));+ if (Number.isFinite(Number(params?.seed))) body.seed = Math.floor(Number(params.seed));+ return {+ path: '/infer',+ resultKey: 'result_gcs_url',+ body,+ };+ }+if (mode === 'triposplat') {// Self-hosted TripoSplat (workers/model-triposplat): the worker// conditions on the first reference image only.@@ -633,6 +656,8 @@ const MODE_ETA = {holopart: 100,// Single-pass flow-matching denoise plus a decoder replay on the L4.triposplat: 45,+ // Two diffusion stages per object, several objects, on a 96 GB-class GPU.+ sam3d: 150,// TripoSG's 20 to 50 flow steps plus repair and decimate on the L4.modly: 60,// Hunyuan3D 2 Mini Turbo is a few-step distilled sampler.@@ -743,7 +768,7 @@ export function createRegenProvider({ reconstructUrl } = {}) {// on the primary view only (report 1 so a multi-view request is never// silently claimed as fused); other modes are single-source.const viewsUsed =- mode === 'hunyuan' || mode === 'triposplat' || mode === 'trellis2' || mode === 'modly' || mode === 'modly_hunyuan' || mode === 'anigen' || mode === 'partcrafter'+ mode === 'hunyuan' || mode === 'triposplat' || mode === 'trellis2' || mode === 'modly' || mode === 'modly_hunyuan' || mode === 'anigen' || mode === 'partcrafter' || mode === 'sam3d'? 1: (mode === 'reconstruct' || mode === 'trellis' || mode === 'pixal3d') && Array.isArray(workerReq.body.images)? workerReq.body.images.length@@ -825,6 +850,12 @@ export function createRegenProvider({ reconstructUrl } = {}) {result.resultSplatUrl = url;if (data.ply_url) result.plyUrl = data.ply_url;if (typeof data.num_gaussians === 'number') result.numGaussians = data.num_gaussians;+ } else if (mode === 'sam3d') {+ // A laid-out scene GLB plus the per-object manifest.+ result.resultGlbUrl = url;+ if (Array.isArray(data.objects)) result.objects = data.objects;+ if (typeof data.object_count === 'number') result.objectCount = data.object_count;+ if (Array.isArray(data.skipped) && data.skipped.length) result.skipped = data.skipped;} else if (mode === 'video2scene') {// The result is a .ply point cloud, not a GLB mesh. Surface the// reconstruction telemetry so the client can show point/frame counts.diff --git a/api/forge-scene-objects.js b/api/forge-scene-objects.jsnew file mode 100644index 000000000..fdfe2109a--- /dev/null+++ b/api/forge-scene-objects.js@@ -0,0 +1,206 @@+/**+ * /api/forge-scene-objects: turn one photo into a 3D scene of the objects in it+ * (SAM 3D Objects, workers/model-sam3d-objects).+ *+ * POST /api/forge-scene-objects+ * { image_url: string,+ * boxes?: [[x0, y0, x1, y1], ...] fractions of the frame, origin top-left+ * mask_urls?: [string, ...] one https PNG mask per object (white = object)+ * detect?: true find the objects automatically (default when+ * neither boxes nor mask_urls is sent)+ * max_objects?: 1..12, seed?: int }+ * → 202 { job_id, status, eta_seconds, source: 'boxes'|'masks'|'detected', objects: [{ label, box }] }+ *+ * GET /api/forge-scene-objects?job=<id>+ * → { job_id, status, result_url?, objects?, object_count?, skipped?, error? }+ *+ * The scene GLB has one named node per object (object_01, ...) laid out like the+ * photo, and `objects` carries each node's position and size.+ *+ * Until MODEL_SAM3D_URL points at a deployed worker this answers 503+ * backend_unconfigured (the worker needs a 32 GB+ GPU). The SAM License refuses+ * comprehensively sanctioned jurisdictions, so those requests are turned away+ * before any work is queued.+ */++import { cors, json, method, readJson, wrap, rateLimited } from './_lib/http.js';+import { limits, clientIp } from './_lib/rate-limit.js';+import { assertPublicHttpsUrl, SsrfError } from './_lib/ssrf.js';+import { createRegenProvider } from './_providers/gcp.js';+import { requestTerritory } from './_lib/forge-territory.js';+import {+ DEFAULT_SCENE_OBJECTS,+ MAX_SCENE_OBJECTS,+ detectObjectBoxes,+ isSanctionedTerritory,+ parseUserBoxes,+} from './_lib/forge-objects.js';++const JOB_ID_RE = /^[A-Za-z0-9_-]{20,600}$/;+const SEED_MAX = 4_294_967_295;++function unconfigured(res) {+ return json(res, 503, {+ error: 'backend_unconfigured',+ message:+ 'Scene-from-a-photo is not switched on yet. It needs the SAM 3D Objects worker (a 32 GB+ GPU): ' ++ 'deploy workers/model-sam3d-objects and set MODEL_SAM3D_URL and GCP_RECONSTRUCTION_KEY.',+ });+}++async function publicUrl(value, field, res) {+ try {+ return await assertPublicHttpsUrl(typeof value === 'string' ? value.trim() : '');+ } catch (err) {+ json(res, 400, {+ error: `invalid_${field}`,+ message: err instanceof SsrfError ? `${field} rejected: ${err.message}` : `${field} must be a public https URL.`,+ });+ return null;+ }+}++async function startJob(req, res) {+ if (isSanctionedTerritory(requestTerritory(req))) {+ return json(res, 451, {+ error: 'unavailable_in_region',+ message: 'This model is licensed to exclude comprehensively sanctioned jurisdictions, so it is not available from your region.',+ });+ }++ const rl = await limits.mcp3dGenerate(clientIp(req));+ if (!rl.success) return rateLimited(res, rl);++ const body = await readJson(req, 16_000).catch(() => null);+ if (!body || typeof body !== 'object') {+ return json(res, 400, { error: 'invalid_body', message: 'Send a JSON body with image_url.' });+ }++ let provider;+ try {+ provider = createRegenProvider();+ if (!provider.supportsMode('sam3d')) return unconfigured(res);+ } catch {+ return unconfigured(res);+ }++ const imageUrl = await publicUrl(body.image_url, 'image_url', res);+ if (!imageUrl) return undefined;++ let maxObjects = DEFAULT_SCENE_OBJECTS;+ if (body.max_objects !== undefined && body.max_objects !== null) {+ const n = Number(body.max_objects);+ if (!Number.isInteger(n) || n < 1 || n > MAX_SCENE_OBJECTS) {+ return json(res, 400, { error: 'invalid_max_objects', message: `max_objects must be an integer from 1 to ${MAX_SCENE_OBJECTS}.` });+ }+ maxObjects = n;+ }++ let seed;+ if (body.seed !== undefined && body.seed !== null) {+ const n = Number(body.seed);+ if (!Number.isInteger(n) || n < 0) {+ return json(res, 400, { error: 'invalid_seed', message: 'seed must be a non-negative integer.' });+ }+ seed = Math.min(n, SEED_MAX);+ }++ const hasMasks = Array.isArray(body.mask_urls) && body.mask_urls.length > 0;+ const hasBoxes = Array.isArray(body.boxes) && body.boxes.length > 0;+ if (hasMasks && hasBoxes) {+ return json(res, 400, { error: 'ambiguous_objects', message: 'Send mask_urls or boxes, not both.' });+ }++ const params = { seed, maxObjects };+ let source;+ let objects;+ if (hasMasks) {+ if (body.mask_urls.length > maxObjects) {+ return json(res, 400, { error: 'too_many_objects', message: `${body.mask_urls.length} masks sent but max_objects is ${maxObjects}.` });+ }+ const masks = [];+ for (const raw of body.mask_urls) {+ const url = await publicUrl(raw, 'mask_urls', res);+ if (!url) return undefined;+ masks.push(url);+ }+ params.maskUrls = masks;+ source = 'masks';+ objects = masks.map((_, i) => ({ label: `object ${i + 1}`, box: null }));+ } else {+ let boxes;+ if (hasBoxes) {+ const parsed = parseUserBoxes(body.boxes, maxObjects);+ if (parsed.error) return json(res, 400, { error: 'invalid_boxes', message: parsed.error });+ boxes = parsed.boxes.map((box, i) => ({ label: `object ${i + 1}`, box }));+ source = 'boxes';+ } else {+ try {+ boxes = await detectObjectBoxes(imageUrl, maxObjects);+ } catch (err) {+ return json(res, 502, {+ error: 'detect_failed',+ message: `Could not find objects in the photo automatically (${err?.message || 'vision unavailable'}). Send boxes or mask_urls to choose them yourself.`,+ });+ }+ if (!boxes.length) {+ return json(res, 422, {+ error: 'no_objects_found',+ message: 'No separate objects were found in the photo. Try a photo with clearer, whole objects, or send boxes to choose them yourself.',+ });+ }+ source = 'detected';+ }+ params.boxes = boxes.map((b) => b.box);+ objects = boxes;+ }++ try {+ const job = await provider.submit({ mode: 'sam3d', sourceUrl: imageUrl, params });+ return json(res, 202, { job_id: job.extJobId, status: 'queued', eta_seconds: job.eta, source, objects });+ } catch (err) {+ return json(res, 502, { error: 'sam3d_failed', message: err?.message || 'Scene reconstruction could not start.' });+ }+}++async function pollJob(req, res, jobId) {+ if (!JOB_ID_RE.test(jobId)) return json(res, 400, { error: 'invalid_job', message: 'Malformed job id.' });++ const rl = await limits.mcp3dStatus(clientIp(req));+ if (!rl.success) return rateLimited(res, rl);++ let provider;+ try {+ provider = createRegenProvider();+ } catch {+ return unconfigured(res);+ }++ let result;+ try {+ result = await provider.status(jobId);+ } catch (err) {+ return json(res, 502, { error: 'sam3d_status_failed', message: err?.message || 'Could not read the scene job.' });+ }+ return json(res, 200, {+ job_id: jobId,+ status: result.status,+ result_url: result.resultGlbUrl || null,+ objects: result.objects || null,+ object_count: result.objectCount ?? null,+ skipped: result.skipped || null,+ error: result.error || null,+ });+}++export default wrap(async (req, res) => {+ if (cors(req, res, { methods: 'GET,POST,OPTIONS' })) return;+ if (!method(req, res, ['GET', 'POST'])) return;++ if (req.method === 'POST') return startJob(req, res);++ const url = new URL(req.url, 'http://localhost');+ const jobId = (url.searchParams.get('job') || '').trim();+ if (!jobId) return json(res, 400, { error: 'missing_job', message: 'Pass ?job=<id> to poll.' });+ return pollJob(req, res, jobId);+});diff --git a/data/changelog.json b/data/changelog.jsonindex dc5a400eb..55dcc8184 100644--- a/data/changelog.json+++ b/data/changelog.json@@ -1,6 +1,13 @@{"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security, desktop (desktop entries become the desktop app's release notes). `link` is optional and must be a live three.ws path.","entries": [+ {+ "date": "2026-10-24",+ "title": "Scene from a photo, built and waiting on weights",+ "summary": "POST /api/forge-scene-objects rebuilds the objects in one photo as a laid-out 3D scene with SAM 3D Objects, and AR Studio gains a From a photo tab that drops the result into the room. The 32 GB+ GPU worker, its Cloud Build config and a one-command weights staging script are in the repo. It answers backend_unconfigured until the gated weights are staged and the worker is deployed.",+ "tags": ["feature", "docs"],+ "link": "/ar/studio"+ },{"date": "2026-10-24","title": "Pay for a model request with a wallet, no account needed",diff --git a/data/open-source-credits.json b/data/open-source-credits.jsonindex ca2a7ae05..db7ef09c2 100644--- a/data/open-source-credits.json+++ b/data/open-source-credits.json@@ -395,6 +395,18 @@"powers": "workers/rembg: background removal ahead of image-to-3D generation.","org": "danielgatis, ZhengPeng7"},+ {+ "id": "model:sam3d-objects",+ "kind": "model",+ "name": "SAM 3D Objects",+ "area": "generation",+ "license": "SAM License (Meta, last updated November 19, 2025; use allowed subject to Trade Controls, gated weights)",+ "repo": "https://github.com/facebookresearch/sam-3d-objects",+ "funding": null,+ "version": null,+ "powers": "workers/model-sam3d-objects: rebuilds the objects in one photo as a laid-out 3D scene (Forge and AR Studio \"From a photo\"), a 32 GB+ GPU lane that stays off until the owner stages the weights.",+ "org": "Meta"+ },{"id": "model:hunyuan3d","kind": "model",diff --git a/docs/api-reference.md b/docs/api-reference.mdindex 0dfff959e..592a7e7f0 100644--- a/docs/api-reference.md+++ b/docs/api-reference.md@@ -10658,6 +10658,8 @@ a delivery from the log or the `/dashboard/developers` page. The signing secret| `GET /api/forge-quote?path=text&tier=standard` | none | What a generation would cost. Optional `backend`, `images`, `own_key`, `gameready`; `table=1` returns every tier. No spend, `cache-control: private, max-age=30`. || `GET /api/forge-keep?id=<creation>` | none | Whether the creation is kept, and whether pay-to-keep is enabled. || `POST /api/forge-keep` | optional $THREE wallet or payment | Release a creation for download. With `FORGE_PAY_TO_KEEP` off it always returns `kept: true`. Non-holders get `402 three_hold_required` with the `pay_per_use` action. A reused payment returns `409 payment_already_used`. |+| `POST /api/forge-scene-objects` | none | One photo to a laid-out 3D scene (SAM 3D Objects). Body `{image_url, boxes? \| mask_urls?, max_objects?, seed?}`; auto-detects objects when neither is sent. `202 {job_id}`, `451` in sanctioned territories, `503 backend_unconfigured` until the worker is deployed. |+| `GET /api/forge-scene-objects?job=<id>` | none | Job status with `result_url`, `objects`, `object_count`, `skipped`. |Details and the enable command: [forge.md](./forge.md#cost-preview-and-pay-to-keep).diff --git a/docs/forge.md b/docs/forge.mdindex f00fea939..9e7a44416 100644--- a/docs/forge.md+++ b/docs/forge.md@@ -649,3 +649,30 @@ dashboard at [/quality-bench](https://three.ws/quality-bench) (internal).- [Diorama](./diorama.md) forges a whole scene of Forge objects into one explorable world.- The 3D generation architecture: [3D pipeline](./3d-pipeline.md), [3D asset pipeline](./3d-asset-pipeline.md), [3D API](./3d-api.md).- Pages: [/create](https://three.ws/create) (the avatar creator), [/scene](https://three.ws/scene) (Scene Studio), [/restyle](https://three.ws/restyle) (re-skin a GLB), [/gallery](https://three.ws/gallery).++### Scene from a photo (SAM 3D Objects)++`/api/forge-scene-objects` rebuilds every object in one photo as a single GLB with+a named node per object (`object_01`, ...), laid out the way the objects sit in+the picture (relative layout, not metric). It runs+[SAM 3D Objects](https://github.com/facebookresearch/sam-3d-objects) on+`workers/model-sam3d-objects` and is also the AR Studio tray tab "From a photo".++```bash+curl -sX POST https://three.ws/api/forge-scene-objects \+ -H 'content-type: application/json' \+ -d '{"image_url":"https://.../desk.jpg","max_objects":8}'+# → 202 { "job_id": "...", "status": "queued", "eta_seconds": 150,+# "source": "detected", "objects": [{ "label": "mug", "box": [...] }] }++curl -s 'https://three.ws/api/forge-scene-objects?job=<job_id>'+# → { "status": "done", "result_url": "https://.../scene.glb", "object_count": 3, "skipped": [] }+```++Choose the objects with `boxes` (`[x0,y0,x1,y1]` fractions, top-left origin),+`mask_urls` (one https PNG per object), or neither and the vision chain finds+them. Requests from comprehensively sanctioned territories (the SAM License's+Trade Controls clause) get `451` before any work is queued. Returns `503+backend_unconfigured` until `MODEL_SAM3D_URL` and `GCP_RECONSTRUCTION_KEY` point+at the deployed worker; staging the gated weights is `npm run stage:sam3d-objects`+(see the worker README).diff --git a/package.json b/package.jsonindex e71d64c34..0749d4bba 100644--- a/package.json+++ b/package.json@@ -582,6 +582,7 @@"check:mcp-catalog": "node scripts/check-mcp-catalog.mjs","vet:mcp-catalog": "node scripts/vet-mcp-integrations.mjs","forge:pay-to-keep": "node scripts/forge-pay-to-keep.mjs",+ "stage:sam3d-objects": "node scripts/stage-sam3d-objects.mjs","rescan:skills": "node --env-file=.env.local scripts/rescan-marketplace-skills.mjs"},"dependencies": {diff --git a/pages/ar-studio.html b/pages/ar-studio.htmlindex 2fe91498c..8e0633370 100644--- a/pages/ar-studio.html+++ b/pages/ar-studio.html@@ -329,6 +329,8 @@border-radius: 10px; padding: 10px 12px;}.ars-link-row input:focus { outline: none; border-color: rgba(139, 124, 248, 0.6); }+ .ars-photo-progress { display: flex; align-items: center; gap: 8px; font-size: 13px; color: var(--ink-faint); }+ .ars-photo-error { margin: 0; font-size: 12.5px; color: var(--warn, #f59e0b); line-height: 1.5; }.ars-link-hint { margin: 0; font-size: 12.5px; color: var(--ink-faint); line-height: 1.5; }.ars-objects-head { display: flex; flex-direction: column; gap: 7px; padding: 2px 2px 10px; }.ars-objects-more { padding: 10px 0 2px; display: flex; justify-content: center; }@@ -462,6 +464,7 @@<button class="ars-tab" type="button" role="tab" id="ars-tab-community" data-tab="community" aria-selected="false" aria-controls="ars-tray-body" data-i18n="ar_studio.community">Community</button><button class="ars-tab" type="button" role="tab" id="ars-tab-objects" data-tab="objects" aria-selected="false" aria-controls="ars-tray-body" data-i18n="ar_studio.objects">Objects</button><button class="ars-tab" type="button" role="tab" id="ars-tab-link" data-tab="link" aria-selected="false" aria-controls="ars-tray-body" data-i18n="ar_studio.glb_link">GLB link</button>+ <button class="ars-tab" type="button" role="tab" id="ars-tab-photo" data-tab="photo" aria-selected="false" aria-controls="ars-tray-body">From a photo</button></div><div class="ars-tray-body" id="ars-tray-body" role="tabpanel" tabindex="0" aria-labelledby="ars-tab-recent"></div></div>diff --git a/scripts/lib/open-source-credits.mjs b/scripts/lib/open-source-credits.mjsindex 10975a211..5979ab1ed 100644--- a/scripts/lib/open-source-credits.mjs+++ b/scripts/lib/open-source-credits.mjs@@ -202,6 +202,15 @@ export const WORKER_MODELS = [area: 'generation',powers: "workers/model-trellis2: the default image-to-3D PBR engine behind Forge's primary lane.",},+ {+ id: 'model:sam3d-objects',+ name: 'SAM 3D Objects',+ org: 'Meta',+ license: 'SAM License (Meta, last updated November 19, 2025; use allowed subject to Trade Controls, gated weights)',+ repo: 'https://github.com/facebookresearch/sam-3d-objects',+ area: 'generation',+ powers: 'workers/model-sam3d-objects: rebuilds the objects in one photo as a laid-out 3D scene (Forge and AR Studio "From a photo"), a 32 GB+ GPU lane that stays off until the owner stages the weights.',+ },{id: 'model:hunyuan3d',name: 'Tencent Hunyuan3D 2.1',diff --git a/scripts/stage-sam3d-objects.mjs b/scripts/stage-sam3d-objects.mjsnew file mode 100644index 000000000..2b9cf2d40--- /dev/null+++ b/scripts/stage-sam3d-objects.mjs@@ -0,0 +1,193 @@+#!/usr/bin/env node+// Stage the SAM 3D Objects checkpoints into the model-weights bucket, after the+// account owner has accepted the SAM License. One command:+//+// HF_TOKEN=hf_... npm run stage:sam3d-objects -- --accept-sam-license+//+// What it does, in order:+// 1. Refuses to run without --accept-sam-license. That flag is the owner+// saying "I have read the SAM License and I accept it for this account".+// It is never implied, never defaulted, and never set by CI.+// 2. Checks the Hugging Face token can read facebook/sam-3d-objects (the repo+// is gated by manual approval). If not, it prints exactly where to request+// access and exits.+// 3. Streams every file under checkpoints/ from Hugging Face straight into+// gs://<bucket>/sam3d-objects/hf/ (no local disk), skipping objects that+// already exist at the right size, and verifies sizes after the copy.+// 4. Copies the upstream LICENSE next to the weights, because the licence+// requires a copy of the agreement to travel with the SAM Materials.+// 5. Creates the Artifact Registry repository the worker image is pushed to+// when it does not exist yet.+//+// The token reaches curl through the environment, never argv. Options:+// --dry-run print what would be copied, change nothing+// --bucket X default three-ws-model-weights+// --prefix X default sam3d-objects+// --project X default aerial-vehicle-466722-p5+// --region X default us-central1++import { spawnSync } from 'node:child_process';+import { fileURLToPath } from 'node:url';++export const REPO = 'facebook/sam-3d-objects';+export const LICENSE_URL = 'https://github.com/facebookresearch/sam-3d-objects/blob/main/LICENSE';+export const ACCESS_URL = `https://huggingface.co/${REPO}`;++export const LICENSE_SUMMARY = [+ 'SAM License (Meta, last updated November 19, 2025). Read the full text: ' + LICENSE_URL,+ ' - Use, reproduce, modify and distribute are allowed, royalty-free, subject to the agreement.',+ ' - Anyone you give the SAM Materials to receives them under this same agreement, with a copy of it.',+ ' - Use must comply with Trade Controls (sanctions and export controls). You must not be a target of',+ ' them. No ITAR, military, warfare, nuclear, espionage or weapons end uses.',+ ' - No reverse engineering, decompiling or discovering the underlying components of the SAM Materials.',+ ' - Research publications that use the SAM Materials must acknowledge it.',+ ' - Provided as is, no warranty, no support, no liability.',+ ' - The weights are gated on Hugging Face by manual approval and are refused in comprehensively',+ ' sanctioned jurisdictions.',+];++/** Parse argv into options. Pure, so it is unit-tested. */+export function parseArgs(argv, env = {}) {+ const opts = {+ accept: false,+ dryRun: false,+ bucket: 'three-ws-model-weights',+ prefix: 'sam3d-objects',+ project: 'aerial-vehicle-466722-p5',+ region: 'us-central1',+ token: env.HF_TOKEN || '',+ };+ for (let i = 0; i < argv.length; i++) {+ const a = argv[i];+ if (a === '--accept-sam-license') opts.accept = true;+ else if (a === '--dry-run') opts.dryRun = true;+ else if (['--bucket', '--prefix', '--project', '--region'].includes(a)) {+ const value = argv[++i];+ if (!value || value.startsWith('--')) throw new Error(`${a} needs a value`);+ opts[a.slice(2)] = value;+ } else throw new Error(`unknown option ${a}`);+ }+ for (const key of ['bucket', 'prefix', 'project', 'region']) {+ if (!/^[a-z0-9][a-z0-9._-]*$/i.test(opts[key])) throw new Error(`--${key} has unexpected characters`);+ }+ return opts;+}++/**+ * Turn the Hugging Face file listing into the copy plan. Only files under+ * checkpoints/ are weights; the plan keeps their relative layout under hf/ and+ * adds the upstream LICENSE. Pure, so it is unit-tested.+ */+export function planCopies(siblings, bucket, prefix) {+ const dest = `gs://${bucket}/${prefix}`;+ const plan = [];+ for (const s of siblings) {+ const path = s.rfilename;+ if (path === 'LICENSE') plan.push({ path, size: s.size || 0, dest: `${dest}/LICENSE` });+ else if (path.startsWith('checkpoints/') && !path.endsWith('.gitattributes'))+ plan.push({ path, size: s.size || 0, dest: `${dest}/hf/${path.slice('checkpoints/'.length)}` });+ }+ if (!plan.some((p) => p.path === 'checkpoints/pipeline.yaml'))+ throw new Error('checkpoints/pipeline.yaml is not in the repository listing; upstream layout changed');+ return plan;+}++function run(cmd, args, options = {}) {+ return spawnSync(cmd, args, { encoding: 'utf8', ...options });+}++async function main() {+ const opts = parseArgs(process.argv.slice(2), process.env);+ console.log(LICENSE_SUMMARY.join('\n') + '\n');+ if (!opts.accept) {+ console.error(+ 'Not staging: the SAM License has not been accepted.\n' ++ `1. Read it: ${LICENSE_URL}\n` ++ `2. Request access with your Hugging Face account (manual approval): ${ACCESS_URL}\n` ++ '3. Re-run with --accept-sam-license once you accept it for this account.',+ );+ process.exit(2);+ }+ if (!opts.token) {+ console.error(`HF_TOKEN is not set. Create a read token for the account that was approved for ${REPO} and re-run.`);+ process.exit(2);+ }+ const auth = { Authorization: `Bearer ${opts.token}` };++ const probe = await fetch(`https://huggingface.co/${REPO}/resolve/main/LICENSE`, { headers: auth, redirect: 'follow' });+ if (probe.status === 401 || probe.status === 403) {+ console.error(`The token cannot read ${REPO} (HTTP ${probe.status}). Request access and wait for approval: ${ACCESS_URL}`);+ process.exit(2);+ }+ if (!probe.ok) {+ console.error(`Hugging Face answered HTTP ${probe.status} for ${REPO}. Retry shortly.`);+ process.exit(1);+ }++ const listing = await fetch(`https://huggingface.co/api/models/${REPO}?blobs=true`, { headers: auth });+ if (!listing.ok) {+ console.error(`Could not list ${REPO} (HTTP ${listing.status}).`);+ process.exit(1);+ }+ const plan = planCopies((await listing.json()).siblings || [], opts.bucket, opts.prefix);+ const totalGb = plan.reduce((n, p) => n + p.size, 0) / 2 ** 30;+ console.log(`${plan.length} files, ${totalGb.toFixed(1)} GiB -> gs://${opts.bucket}/${opts.prefix}/`);+ if (opts.dryRun) {+ for (const p of plan) console.log(`would copy ${p.path} -> ${p.dest} (${p.size} bytes)`);+ return;+ }++ const gcloud = run('gcloud', ['--version']);+ if (gcloud.status !== 0) {+ console.error('gcloud is not installed or not on PATH. Install the Google Cloud CLI and run `gcloud auth login`.');+ process.exit(1);+ }++ const repoCheck = run('gcloud', ['artifacts', 'repositories', 'describe', 'model-sam3d-objects', `--location=${opts.region}`, `--project=${opts.project}`]);+ if (repoCheck.status !== 0) {+ console.log('creating Artifact Registry repository model-sam3d-objects');+ const created = run('gcloud', ['artifacts', 'repositories', 'create', 'model-sam3d-objects', '--repository-format=docker', `--location=${opts.region}`, `--project=${opts.project}`], { stdio: 'inherit' });+ if (created.status !== 0) process.exit(1);+ }++ const remoteSize = (uri) => {+ const r = run('gcloud', ['storage', 'objects', 'describe', uri, '--format=value(size)']);+ return r.status === 0 ? r.stdout.trim() : '';+ };+ for (const p of plan) {+ if (p.size && remoteSize(p.dest) === String(p.size)) {+ console.log(`skip ${p.dest}`);+ continue;+ }+ console.log(`copy ${p.path} -> ${p.dest} (${p.size} bytes)`);+ const url = `https://huggingface.co/${REPO}/resolve/main/${p.path.split('/').map(encodeURIComponent).join('/')}`;+ const piped = run('bash', ['-o', 'pipefail', '-c', 'curl -fsSL -H "Authorization: Bearer $HF_TOKEN" "$SRC" | gcloud storage cp - "$DEST" --quiet'], {+ stdio: ['ignore', 'inherit', 'inherit'],+ env: { ...process.env, HF_TOKEN: opts.token, SRC: url, DEST: p.dest },+ });+ if (piped.status !== 0) {+ console.error(`FAIL ${p.dest}: copy exited ${piped.status}`);+ process.exit(1);+ }+ const got = remoteSize(p.dest);+ if (p.size && got !== String(p.size)) {+ console.error(`FAIL ${p.dest}: expected ${p.size} bytes, bucket has ${got || 'none'}`);+ process.exit(1);+ }+ }++ console.log(+ `\nSAM 3D Objects checkpoints staged at gs://${opts.bucket}/${opts.prefix}/hf/.\n` ++ 'Next: build and deploy the worker (needs a 32 GB+ GPU class, see workers/model-sam3d-objects/README.md):\n' ++ ' gcloud builds submit --config workers/model-sam3d-objects/cloudbuild.yaml --substitutions=SHORT_SHA=manual$(date +%s) .\n' ++ 'then point the platform at it:\n' ++ ' gcloud run services update three-ws-api --region us-central1 --update-env-vars MODEL_SAM3D_URL=<service url>',+ );+}++if (process.argv[1] === fileURLToPath(import.meta.url)) {+ main().catch((err) => {+ console.error(err.message || err);+ process.exit(1);+ });+}diff --git a/src/ar-studio.js b/src/ar-studio.jsindex d4dd119b3..a19bd1ce0 100644--- a/src/ar-studio.js+++ b/src/ar-studio.js@@ -1812,6 +1812,82 @@ async function fetchTrayItems(tab) {return items;}++// "From a photo": SAM 3D Objects (POST /api/forge-scene-objects) turns one photo+// into a single GLB with a named node per object, laid out like the photo. The+// scene lands in the room as one model. The endpoint answers 503+// backend_unconfigured until the GPU worker is deployed, and the tab says so.+const PHOTO_POLL_MS = 4000;+const PHOTO_MAX_POLLS = 150;+let photoJobRunning = false;++async function runPhotoScene(imageUrl, ui) {+ const res = await fetch('/api/forge-scene-objects', {+ method: 'POST',+ credentials: 'include',+ headers: { 'content-type': 'application/json', 'x-forge-client': CLIENT_ID },+ body: JSON.stringify({ image_url: imageUrl }),+ });+ const start = await res.json().catch(() => ({}));+ if (!res.ok) throw new Error(start.message || start.error_description || `Request failed (${res.status}).`);+ ui.progress(`Reading the photo: ${start.objects?.length || 'some'} object(s) found. Rebuilding in 3D…`);+ for (let i = 0; i < PHOTO_MAX_POLLS; i++) {+ await new Promise((r) => setTimeout(r, PHOTO_POLL_MS));+ const poll = await fetch(`/api/forge-scene-objects?job=${encodeURIComponent(start.job_id)}`, { credentials: 'include' });+ const body = await poll.json().catch(() => ({}));+ if (!poll.ok) throw new Error(body.message || `Status check failed (${poll.status}).`);+ if (body.status === 'failed' || body.error) throw new Error(body.error || 'The reconstruction failed.');+ if (body.result_url) return body;+ }+ throw new Error('The reconstruction is taking too long. Try again in a few minutes.');+}++function renderPhotoTab() {+ trayBody.innerHTML = `+ <form class="ars-link-form" id="ars-photo-form">+ <label class="ars-link-label" for="ars-photo-input">Paste a photo link</label>+ <div class="ars-link-row">+ <input id="ars-photo-input" type="url" inputmode="url" required+ placeholder="https://example.com/desk.jpg" />+ <button type="submit" class="ars-btn ars-btn-primary" id="ars-photo-go">Build scene</button>+ </div>+ <p class="ars-link-hint">Every object SAM 3D finds in the photo becomes a 3D object, laid out the way it sits in the picture. Takes a couple of minutes.</p>+ <div id="ars-photo-state" role="status" aria-live="polite"></div>+ </form>`;+ const form = $('ars-photo-form');+ const state = $('ars-photo-state');+ const go = $('ars-photo-go');+ form?.addEventListener('submit', async (e) => {+ e.preventDefault();+ if (photoJobRunning) return;+ const url = String($('ars-photo-input')?.value || '').trim();+ if (!/^https:\/\//i.test(url)) {+ state.innerHTML = '<p class="ars-photo-error">That needs to be a public https link to a photo.</p>';+ return;+ }+ photoJobRunning = true;+ go.disabled = true;+ const ui = {+ progress: (msg) => {+ state.innerHTML = '<div class="ars-photo-progress"><span class="ars-spinner" aria-hidden="true"></span><span></span></div>';+ state.querySelector('span:last-child').textContent = msg;+ },+ };+ ui.progress('Finding the objects in your photo…');+ try {+ const done = await runPhotoScene(url, ui);+ closeTray();+ await addModel({ src: done.result_url, title: 'Photo scene' });+ } catch (err) {+ state.innerHTML = '<p class="ars-photo-error"></p>';+ state.firstChild.textContent = err.message;+ } finally {+ photoJobRunning = false;+ go.disabled = false;+ }+ });+}+async function renderTray() {if (!trayBody) return;const tab = trayTab;@@ -1842,6 +1918,11 @@ async function renderTray() {return;}+ if (tab === 'photo') {+ renderPhotoTab();+ return;+ }+trayBody.innerHTML = '<div class="ars-tray-loading"><span class="ars-spinner" aria-hidden="true"></span> Loading models…</div>';let items;try {diff --git a/tests/api/forge-scene-objects.test.js b/tests/api/forge-scene-objects.test.jsnew file mode 100644index 000000000..e2538144b--- /dev/null+++ b/tests/api/forge-scene-objects.test.js@@ -0,0 +1,207 @@+/**+ * Tests for the scene-from-a-photo endpoint (api/forge-scene-objects.js).+ *+ * The SAM 3D Objects worker and the vision chain are stubbed at their module+ * boundaries, so this exercises the endpoint's own logic: the sanctions guard,+ * SSRF checks, object selection (boxes, masks, automatic detection), the+ * backend_unconfigured answer, and the poll passthrough.+ */++import { describe, it, expect, vi, beforeEach } from 'vitest';+import { Readable } from 'node:stream';++const submit = vi.fn(async () => ({ extJobId: `task-sam3d-${'x'.repeat(20)}`, eta: 150 }));+const status = vi.fn(async () => ({+ status: 'done',+ resultGlbUrl: 'https://storage.googleapis.com/bucket/sam3d/task.glb',+ objects: [{ id: 'object_01', centroid: [0, 0, -3] }],+ objectCount: 1,+}));+let supportsSam3d = true;++vi.mock('../../api/_providers/gcp.js', () => ({+ createRegenProvider: () => ({+ supportsMode: (m) => m === 'sam3d' && supportsSam3d,+ submit,+ status,+ }),+}));++const detect = vi.fn();+vi.mock('../../api/_lib/vision.js', () => ({+ describeImageJson: (...a) => detect(...a),+}));++vi.mock('../../api/_lib/ssrf.js', async (importOriginal) => {+ const mod = await importOriginal();+ return {+ ...mod,+ assertPublicHttpsUrl: vi.fn(async (url) => {+ const parsed = new URL(url);+ if (parsed.protocol !== 'https:') throw new mod.SsrfError('https required');+ return url;+ }),+ };+});++vi.mock('../../api/_lib/rate-limit.js', async (importActual) => {+ const actual = await importActual();+ return {+ ...actual,+ clientIp: () => '203.0.113.9',+ limits: {+ ...actual.limits,+ mcp3dGenerate: vi.fn(async () => ({ success: true, reset: Date.now() + 1000 })),+ mcp3dStatus: vi.fn(async () => ({ success: true, reset: Date.now() + 1000 })),+ },+ };+});++const { default: handler } = await import('../../api/forge-scene-objects.js');++function makeRes() {+ return {+ statusCode: 200,+ headers: {},+ body: null,+ setHeader(n, v) {+ this.headers[String(n).toLowerCase()] = v;+ },+ end(b) {+ this.body = b ?? null;+ },+ };+}++function makeReq({ method = 'POST', url = '/api/forge-scene-objects', headers = {}, body = null } = {}) {+ const payload = body == null ? '' : JSON.stringify(body);+ const stream = Readable.from(payload ? [Buffer.from(payload, 'utf8')] : []);+ stream.method = method;+ stream.url = url;+ stream.headers = { 'content-type': 'application/json', 'x-forwarded-for': '203.0.113.9', ...headers };+ return stream;+}++const PHOTO = 'https://storage.googleapis.com/bucket/room.jpg';++beforeEach(() => {+ submit.mockClear();+ status.mockClear();+ detect.mockReset();+ supportsSam3d = true;+});++describe('POST /api/forge-scene-objects', () => {+ it('queues a job with the caller\'s boxes in order', async () => {+ const res = makeRes();+ await handler(makeReq({ body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]], seed: 3 } }), res);+ expect(res.statusCode).toBe(202);+ const body = JSON.parse(res.body);+ expect(body.source).toBe('boxes');+ expect(body.eta_seconds).toBe(150);+ const call = submit.mock.calls[0][0];+ expect(call.mode).toBe('sam3d');+ expect(call.sourceUrl).toBe(PHOTO);+ expect(call.params.boxes).toEqual([[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]]);+ expect(call.params.seed).toBe(3);+ expect(detect).not.toHaveBeenCalled();+ });++ it('queues a job with mask URLs after the SSRF check', async () => {+ const res = makeRes();+ await handler(makeReq({ body: { image_url: PHOTO, mask_urls: ['https://storage.googleapis.com/bucket/m1.png'] } }), res);+ expect(res.statusCode).toBe(202);+ expect(JSON.parse(res.body).source).toBe('masks');+ expect(submit.mock.calls[0][0].params.maskUrls).toEqual(['https://storage.googleapis.com/bucket/m1.png']);+ });++ it('finds the objects itself when none are named', async () => {+ detect.mockResolvedValue({ json: { objects: [{ label: 'lamp', box_2d: [100, 100, 600, 400] }] } });+ const res = makeRes();+ await handler(makeReq({ body: { image_url: PHOTO } }), res);+ expect(res.statusCode).toBe(202);+ const body = JSON.parse(res.body);+ expect(body.source).toBe('detected');+ expect(body.objects).toEqual([{ label: 'lamp', box: [0.1, 0.1, 0.4, 0.6] }]);+ expect(submit.mock.calls[0][0].params.boxes).toEqual([[0.1, 0.1, 0.4, 0.6]]);+ });++ it('explains a vision outage and an empty photo instead of failing blind', async () => {+ detect.mockRejectedValue(new Error('no vision provider'));+ let res = makeRes();+ await handler(makeReq({ body: { image_url: PHOTO } }), res);+ expect(res.statusCode).toBe(502);+ expect(JSON.parse(res.body).error).toBe('detect_failed');++ detect.mockResolvedValue({ json: { objects: [] } });+ res = makeRes();+ await handler(makeReq({ body: { image_url: PHOTO } }), res);+ expect(res.statusCode).toBe(422);+ expect(JSON.parse(res.body).error).toBe('no_objects_found');+ expect(submit).not.toHaveBeenCalled();+ });++ it('turns away a comprehensively sanctioned jurisdiction before any work', async () => {+ const res = makeRes();+ await handler(makeReq({ headers: { 'x-client-geo-location': 'IR,Tehran' }, body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]] } }), res);+ expect(res.statusCode).toBe(451);+ expect(JSON.parse(res.body).error).toBe('unavailable_in_region');+ expect(submit).not.toHaveBeenCalled();+ });++ it('does not block an unresolved country or an ordinary one', async () => {+ for (const headers of [{}, { 'x-client-geo-location': 'DE,Berlin' }]) {+ const res = makeRes();+ await handler(makeReq({ headers, body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]] } }), res);+ expect(res.statusCode).toBe(202);+ }+ });++ it('answers 503 backend_unconfigured until the worker is set', async () => {+ supportsSam3d = false;+ const res = makeRes();+ await handler(makeReq({ body: { image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]] } }), res);+ expect(res.statusCode).toBe(503);+ expect(JSON.parse(res.body).error).toBe('backend_unconfigured');+ expect(submit).not.toHaveBeenCalled();+ });++ it.each([+ [{ image_url: 'http://169.254.169.254/x.jpg', boxes: [[0.1, 0.1, 0.4, 0.5]] }, 'invalid_image_url'],+ [{ image_url: PHOTO, mask_urls: ['http://10.0.0.1/m.png'] }, 'invalid_mask_urls'],+ [{ image_url: PHOTO, boxes: [[0.5, 0.5, 0.4, 0.4]] }, 'invalid_boxes'],+ [{ image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]], mask_urls: ['https://x.test/m.png'] }, 'ambiguous_objects'],+ [{ image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]], max_objects: 99 }, 'invalid_max_objects'],+ [{ image_url: PHOTO, boxes: [[0.1, 0.1, 0.4, 0.5]], seed: -2 }, 'invalid_seed'],+ ])('rejects %j', async (body, error) => {+ const res = makeRes();+ await handler(makeReq({ body }), res);+ expect(res.statusCode).toBe(400);+ expect(JSON.parse(res.body).error).toBe(error);+ expect(submit).not.toHaveBeenCalled();+ });+});++describe('GET /api/forge-scene-objects?job=<id>', () => {+ it('polls the job and surfaces the object manifest', async () => {+ const jobId = `task-sam3d-${'x'.repeat(20)}`;+ const res = makeRes();+ await handler(makeReq({ method: 'GET', url: `/api/forge-scene-objects?job=${jobId}` }), res);+ expect(res.statusCode).toBe(200);+ const body = JSON.parse(res.body);+ expect(body.status).toBe('done');+ expect(body.result_url).toBe('https://storage.googleapis.com/bucket/sam3d/task.glb');+ expect(body.objects).toEqual([{ id: 'object_01', centroid: [0, 0, -3] }]);+ expect(body.object_count).toBe(1);+ });++ it('rejects a malformed or missing job id', async () => {+ let res = makeRes();+ await handler(makeReq({ method: 'GET', url: '/api/forge-scene-objects?job=short' }), res);+ expect(JSON.parse(res.body).error).toBe('invalid_job');+ res = makeRes();+ await handler(makeReq({ method: 'GET', url: '/api/forge-scene-objects' }), res);+ expect(JSON.parse(res.body).error).toBe('missing_job');+ expect(status).not.toHaveBeenCalled();+ });+});diff --git a/tests/api/gcp-sam3d-provider.test.js b/tests/api/gcp-sam3d-provider.test.jsnew file mode 100644index 000000000..533052662--- /dev/null+++ b/tests/api/gcp-sam3d-provider.test.js@@ -0,0 +1,88 @@+/**+ * The gcp provider's `sam3d` mode (api/_providers/gcp.js): the request shape+ * sent to workers/model-sam3d-objects and how a finished task maps to a scene+ * result. fetch is mocked at the network boundary only.+ */++import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';++const realFetch = globalThis.fetch;+const WORKER = 'https://sam3d.example.run.app';++beforeEach(() => {+ process.env.GCP_RECONSTRUCTION_KEY = 'test-key';+ process.env.MODEL_SAM3D_URL = WORKER;+});+afterEach(() => {+ globalThis.fetch = realFetch;+ delete process.env.MODEL_SAM3D_URL;+});++const { createRegenProvider } = await import('../../api/_providers/gcp.js');++describe('gcp provider: sam3d', () => {+ it('is supported only when MODEL_SAM3D_URL is set', () => {+ expect(createRegenProvider().supportsMode('sam3d')).toBe(true);+ delete process.env.MODEL_SAM3D_URL;+ expect(createRegenProvider().supportsMode('sam3d')).toBe(false);+ });++ it('posts the photo and boxes to /infer', async () => {+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ task_id: 't-1', status: 'queued' }), { status: 202 }));+ globalThis.fetch = fetchMock;+ const job = await createRegenProvider().submit({+ mode: 'sam3d',+ sourceUrl: 'https://example.com/room.jpg',+ params: { boxes: [[0.1, 0.1, 0.5, 0.5]], maxObjects: 4, seed: 9 },+ });+ const [url, init] = fetchMock.mock.calls[0];+ expect(url).toBe(`${WORKER}/infer`);+ expect(JSON.parse(init.body)).toEqual({+ image_url: 'https://example.com/room.jpg',+ boxes: [[0.1, 0.1, 0.5, 0.5]],+ max_objects: 4,+ seed: 9,+ });+ expect(job.eta).toBe(150);+ });++ it('prefers mask URLs and never sends both', async () => {+ const fetchMock = vi.fn(async () => new Response(JSON.stringify({ task_id: 't-1' }), { status: 202 }));+ globalThis.fetch = fetchMock;+ await createRegenProvider().submit({+ mode: 'sam3d',+ sourceUrl: 'https://example.com/room.jpg',+ params: { maskUrls: ['https://example.com/m.png'], boxes: [[0.1, 0.1, 0.5, 0.5]] },+ });+ const body = JSON.parse(fetchMock.mock.calls[0][1].body);+ expect(body.mask_urls).toEqual(['https://example.com/m.png']);+ expect(body.boxes).toBeUndefined();+ });++ it('maps a done task to the scene GLB and its objects', async () => {+ globalThis.fetch = vi.fn(async () => new Response(JSON.stringify({ task_id: 't-2' }), { status: 202 }));+ const provider = createRegenProvider();+ const { extJobId } = await provider.submit({ mode: 'sam3d', sourceUrl: 'https://example.com/room.jpg', params: { boxes: [[0.1, 0.1, 0.5, 0.5]] } });+ globalThis.fetch = vi.fn(async () =>+ new Response(+ JSON.stringify({+ task_id: 't-2',+ status: 'done',+ result_gcs_url: 'https://storage.googleapis.com/b/scene.glb',+ objects: [{ id: 'object_01', centroid: [0, 0, -2] }],+ object_count: 1,+ skipped: [{ index: 1, reason: 'no mesh' }],+ }),+ { status: 200 },+ ),+ );+ const out = await provider.status(extJobId);+ expect(out).toMatchObject({+ status: 'done',+ resultGlbUrl: 'https://storage.googleapis.com/b/scene.glb',+ objects: [{ id: 'object_01', centroid: [0, 0, -2] }],+ objectCount: 1,+ skipped: [{ index: 1, reason: 'no mesh' }],+ });+ });+});diff --git a/tests/forge-objects.test.js b/tests/forge-objects.test.jsnew file mode 100644index 000000000..3aab57c8e--- /dev/null+++ b/tests/forge-objects.test.js@@ -0,0 +1,71 @@+import { describe, it, expect } from 'vitest';+import {+ SANCTIONED_COUNTRIES,+ isSanctionedTerritory,+ normalizeDetectedBoxes,+ parseUserBoxes,+} from '../api/_lib/forge-objects.js';++describe('isSanctionedTerritory', () => {+ it('blocks only positively resolved comprehensively sanctioned countries', () => {+ for (const c of SANCTIONED_COUNTRIES) expect(isSanctionedTerritory(c)).toBe(true);+ expect(isSanctionedTerritory('ir')).toBe(true);+ expect(isSanctionedTerritory('US')).toBe(false);+ expect(isSanctionedTerritory('DE')).toBe(false);+ });+ it('does not turn away an unresolved country', () => {+ expect(isSanctionedTerritory(null)).toBe(false);+ expect(isSanctionedTerritory(undefined)).toBe(false);+ expect(isSanctionedTerritory('')).toBe(false);+ });+});++describe('normalizeDetectedBoxes', () => {+ it('converts [ymin, xmin, ymax, xmax] on 0..1000 to [x0, y0, x1, y1] fractions', () => {+ const out = normalizeDetectedBoxes({ objects: [{ label: 'chair', box_2d: [200, 100, 800, 500] }] });+ expect(out).toEqual([{ label: 'chair', box: [0.1, 0.2, 0.5, 0.8] }]);+ });+ it('orders largest first, drops slivers and near-duplicates, and honours the cap', () => {+ const out = normalizeDetectedBoxes(+ {+ objects: [+ { label: 'small', box_2d: [0, 0, 100, 100] },+ { label: 'big', box_2d: [0, 0, 900, 900] },+ { label: 'big again', box_2d: [10, 10, 900, 900] },+ { label: 'sliver', box_2d: [0, 0, 500, 10] },+ { label: 'inverted', box_2d: [500, 500, 400, 400] },+ ],+ },+ 5,+ );+ expect(out.map((o) => o.label)).toEqual(['big', 'small']);+ expect(normalizeDetectedBoxes({ objects: [{ box_2d: [0, 0, 500, 500] }, { box_2d: [500, 500, 1000, 1000] }] }, 1)).toHaveLength(1);+ });+ it('clamps to the frame and tolerates junk', () => {+ expect(normalizeDetectedBoxes({ objects: [{ label: 'x', box_2d: [-50, -50, 1200, 1200] }] })[0].box).toEqual([0, 0, 1, 1]);+ expect(normalizeDetectedBoxes(null)).toEqual([]);+ expect(normalizeDetectedBoxes({ objects: [{ box_2d: ['a', 1, 2, 3] }, { box_2d: [1, 2] }, null, 7] })).toEqual([]);+ });+ it('accepts a bare array and trims labels', () => {+ const out = normalizeDetectedBoxes([{ label: ' red mug ', box_2d: [100, 100, 400, 400] }]);+ expect(out[0].label).toBe('red mug');+ });+});++describe('parseUserBoxes', () => {+ it('accepts valid boxes untouched and in order', () => {+ const r = parseUserBoxes([[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]]);+ expect(r.boxes).toEqual([[0.1, 0.1, 0.4, 0.5], [0.6, 0.6, 0.9, 0.9]]);+ });+ it('names the offending box', () => {+ expect(parseUserBoxes([[0.1, 0.1, 0.4, 0.5], [0.5, 0.5, 0.4, 0.4]]).error).toMatch(/boxes\[1\]/);+ expect(parseUserBoxes([[0, 0, 2, 1]]).error).toMatch(/boxes\[0\]/);+ expect(parseUserBoxes([['0', 0, 1, 1]]).error).toMatch(/four numbers/);+ expect(parseUserBoxes([[0.1, 0.1, 0.12, 0.9]]).error).toMatch(/too small/);+ });+ it('rejects two boxes that name the same object, an empty list and too many boxes', () => {+ expect(parseUserBoxes([[0.1, 0.1, 0.5, 0.5], [0.11, 0.11, 0.5, 0.5]]).error).toMatch(/same object/);+ expect(parseUserBoxes([]).error).toBeTruthy();+ expect(parseUserBoxes([[0, 0, 0.5, 0.5], [0.5, 0.5, 1, 1]], 1).error).toMatch(/max_objects/);+ });+});diff --git a/tests/stage-sam3d-objects.test.js b/tests/stage-sam3d-objects.test.jsnew file mode 100644index 000000000..eae2a5708--- /dev/null+++ b/tests/stage-sam3d-objects.test.js@@ -0,0 +1,82 @@+import { describe, it, expect } from 'vitest';+import { spawnSync } from 'node:child_process';+import { readFileSync } from 'node:fs';+import { resolve } from 'node:path';+import { parseArgs, planCopies } from '../scripts/stage-sam3d-objects.mjs';++const ROOT = resolve(import.meta.dirname, '..');+const SCRIPT = resolve(ROOT, 'scripts/stage-sam3d-objects.mjs');++describe('stage-sam3d-objects', () => {+ it('refuses to stage until the SAM License is explicitly accepted', () => {+ const r = spawnSync(process.execPath, [SCRIPT], { encoding: 'utf8', env: { PATH: process.env.PATH, HF_TOKEN: 'hf_x' } });+ expect(r.status).toBe(2);+ expect(r.stderr).toMatch(/SAM License has not been accepted/);+ expect(r.stderr).toMatch(/huggingface\.co\/facebook\/sam-3d-objects/);+ expect(r.stdout).toMatch(/Trade Controls/);+ });++ it('refuses without a token even when the licence is accepted', () => {+ const r = spawnSync(process.execPath, [SCRIPT, '--accept-sam-license'], { encoding: 'utf8', env: { PATH: process.env.PATH } });+ expect(r.status).toBe(2);+ expect(r.stderr).toMatch(/HF_TOKEN is not set/);+ });++ it('parses options and rejects unknown or malformed ones', () => {+ const o = parseArgs(['--accept-sam-license', '--dry-run', '--bucket', 'b1'], { HF_TOKEN: 't' });+ expect(o).toMatchObject({ accept: true, dryRun: true, bucket: 'b1', prefix: 'sam3d-objects', token: 't' });+ expect(parseArgs([], {}).accept).toBe(false);+ expect(() => parseArgs(['--nope'])).toThrow(/unknown option/);+ expect(() => parseArgs(['--bucket'])).toThrow(/needs a value/);+ expect(() => parseArgs(['--bucket', 'a;rm -rf /'])).toThrow(/unexpected characters/);+ });++ it('plans only checkpoints and the licence, under hf/', () => {+ const plan = planCopies(+ [+ { rfilename: '.gitattributes' },+ { rfilename: 'README.md', size: 5 },+ { rfilename: 'LICENSE', size: 10 },+ { rfilename: 'doc/intro.png', size: 7 },+ { rfilename: 'checkpoints/pipeline.yaml', size: 100 },+ { rfilename: 'checkpoints/slat_generator.ckpt', size: 4000 },+ ],+ 'bk',+ 'sam3d-objects',+ );+ expect(plan.map((p) => p.dest)).toEqual([+ 'gs://bk/sam3d-objects/LICENSE',+ 'gs://bk/sam3d-objects/hf/pipeline.yaml',+ 'gs://bk/sam3d-objects/hf/slat_generator.ckpt',+ ]);+ });++ it('fails loudly when upstream drops pipeline.yaml', () => {+ expect(() => planCopies([{ rfilename: 'checkpoints/a.ckpt', size: 1 }], 'bk', 'p')).toThrow(/pipeline\.yaml/);+ });+});++describe('workers/model-sam3d-objects build files', () => {+ const read = (f) => readFileSync(resolve(ROOT, 'workers/model-sam3d-objects', f), 'utf8');++ it('pins the build service account and a 32 GB+ GPU class', () => {+ const cb = read('cloudbuild.yaml');+ expect(cb).toMatch(/serviceAccount: projects\/aerial-vehicle-466722-p5\/serviceAccounts\/three-ws-build@/);+ expect(cb).toMatch(/_GPU_TYPE: nvidia-rtx-pro-6000/);+ expect(cb).not.toMatch(/nvidia-l4/);+ });++ it('pins upstream to a commit and gates the image on the tests', () => {+ const df = read('Dockerfile');+ expect(df).toMatch(/ARG SAM3D_COMMIT=[0-9a-f]{40}/);+ expect(df).toMatch(/pytest .*test_request_policy\.py.*test_scene_layout\.py/s);+ expect(df).toMatch(/test_app_contract\.py/);+ });++ it('copies every module the app imports into the image', () => {+ const df = read('Dockerfile');+ for (const f of ['main.py', 'pipeline.py', 'scene_layout.py', 'request_policy.py', 'worker_security.py', 'storage_backend.py']) {+ expect(df).toContain(f);+ }+ });+});diff --git a/workers/model-sam3d-objects/Dockerfile b/workers/model-sam3d-objects/Dockerfilenew file mode 100644index 000000000..4f5fecfab--- /dev/null+++ b/workers/model-sam3d-objects/Dockerfile@@ -0,0 +1,75 @@+# SAM 3D Objects (facebookresearch/sam-3d-objects, SAM License): one photo in, a+# posed 3D scene of its objects out. Upstream needs a Linux x86-64 host and a+# 32 GB+ NVIDIA GPU, so this image targets the Cloud Run RTX Pro 6000 class, not+# the L4 the other workers use (see cloudbuild.yaml).+#+# Pinned stack, read from the upstream commit below on 2026-10-24: Python 3.11,+# CUDA 12.1, torch 2.5.1. The official PyTorch devel image carries exactly that+# plus nvcc, which pytorch3d, kaolin and gsplat compile against.+#+# The SAM License is NOT accepted by building this image: the checkpoints are not+# in it. They are staged separately by the account owner with+# `npm run stage:sam3d-objects -- --accept-sam-license`.+FROM pytorch/pytorch:2.5.1-cuda12.1-cudnn9-devel++ENV PYTHONDONTWRITEBYTECODE=1 \+ PYTHONUNBUFFERED=1 \+ PIP_NO_CACHE_DIR=1 \+ DEBIAN_FRONTEND=noninteractive \+ HF_HUB_DISABLE_TELEMETRY=1 \+ DO_NOT_TRACK=1 \+ CUDA_HOME=/usr/local/cuda \+ CONDA_PREFIX=/usr/local/cuda \+ LIDRA_SKIP_INIT=true \+ TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0;12.0" \+ SAM3D_SRC=/app/sam3d-objects \+ SAM3D_CHECKPOINT_DIR=/app/checkpoints/hf \+ PIP_EXTRA_INDEX_URL="https://pypi.ngc.nvidia.com https://download.pytorch.org/whl/cu121" \+ PIP_FIND_LINKS="https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.5.1_cu121.html"++RUN apt-get update && apt-get install -y --no-install-recommends \+ git curl build-essential ninja-build \+ libgl1 libglib2.0-0 libgomp1 libegl1 \+ && rm -rf /var/lib/apt/lists/*++# 1. SAM 3D Objects, pinned to the commit read on 2026-10-24+# (https://github.com/facebookresearch/sam-3d-objects/commit/f91db411c50efee93d8db7aeb323885650f6f722).+ARG SAM3D_COMMIT=f91db411c50efee93d8db7aeb323885650f6f722+RUN git clone https://github.com/facebookresearch/sam-3d-objects.git ${SAM3D_SRC} \+ && git -C ${SAM3D_SRC} checkout ${SAM3D_COMMIT} \+ && rm -rf ${SAM3D_SRC}/.git++# 2. Upstream's own install sequence (its doc/setup.md), in the same order: the+# core stack, then pytorch3d (its pip metadata is broken without the 2-step),+# then the inference extras.+WORKDIR ${SAM3D_SRC}+RUN pip install -e '.[dev]' \+ && pip install -e '.[p3d]' \+ && pip install -e '.[inference]'++# 3. Upstream's hydra patch (patching/hydra), applied from the same pinned fork+# commit it downloads, with the file's checksum verified first.+ARG HYDRA_PATCH_SHA256=b5799ea99626593e7650cd6ab7e15639e32d495d1a16f71d627811f86a2a6fba+RUN curl -fsSL -o /tmp/hydra_utils.py \+ https://raw.githubusercontent.com/gleize/hydra/78f00766b5f37672aa7232ebbf01bdd74246bd60/hydra/core/utils.py \+ && echo "${HYDRA_PATCH_SHA256} /tmp/hydra_utils.py" | sha256sum -c - \+ && python -c 'import hydra, os, shutil; assert hydra.__version__ == "1.3.2", hydra.__version__; shutil.copy("/tmp/hydra_utils.py", os.path.join(os.path.dirname(hydra.__file__), "core", "utils.py"))' \+ && rm /tmp/hydra_utils.py++WORKDIR /app+COPY requirements.txt /app/requirements.txt+RUN pip install -r /app/requirements.txt++COPY worker_security.py storage_backend.py request_policy.py scene_layout.py pipeline.py main.py entrypoint.sh /app/+COPY test_request_policy.py test_scene_layout.py test_app_contract.py /app/+RUN chmod +x /app/entrypoint.sh++# Build-time guard: the pure-logic tests and the served-contract test need no+# GPU and no checkpoints, so they run here and the image does not ship if request+# validation, the scene geometry or the import chain broke.+RUN python -m pytest -q /app/test_request_policy.py /app/test_scene_layout.py \+ && python /app/test_app_contract.py++EXPOSE 8080++CMD ["/app/entrypoint.sh"]diff --git a/workers/model-sam3d-objects/README.md b/workers/model-sam3d-objects/README.mdnew file mode 100644index 000000000..8b208578e--- /dev/null+++ b/workers/model-sam3d-objects/README.md@@ -0,0 +1,89 @@+# model-sam3d-objects: one photo to a laid-out 3D scene (SAM 3D Objects)++FastAPI inference service that rebuilds the **objects in a single photo as one 3D+scene GLB** with [SAM 3D Objects](https://github.com/facebookresearch/sam-3d-objects)+(Meta). Each object becomes a named node (`object_01`, `object_02`, ...) placed,+rotated and scaled the way it sits in the picture. The layout is relative, not+metric. It backs `POST /api/forge-scene-objects` and the AR Studio tray tab+"From a photo".++Upstream is pinned to commit `f91db411c50efee93d8db7aeb323885650f6f722`+(`SAM3D_COMMIT` in the Dockerfile).++## Status: off until the owner stages weights++The weights are gated on Hugging Face by manual approval under the SAM License+and the model needs a **32 GB+ GPU**, so the platform answers+`503 backend_unconfigured` until all of this is done:++1. Accept the SAM License and get access: <https://huggingface.co/facebook/sam-3d-objects>.+2. `HF_TOKEN=... npm run stage:sam3d-objects -- --accept-sam-license` (add `--dry-run` first to see the plan). It copies the checkpoints and the LICENSE to `gs://three-ws-model-weights/sam3d-objects/`.+3. Make sure the project has quota for the GPU class below, then+ `gcloud builds submit --config workers/model-sam3d-objects/cloudbuild.yaml --substitutions=SHORT_SHA=manual$(date +%s) .`+4. `gcloud run services update three-ws-api --region us-central1 --update-env-vars MODEL_SAM3D_URL=<service url>` (use `--update-env-vars`, never `--set-env-vars`). `GCP_RECONSTRUCTION_KEY` is shared with the other model workers.++## GPU class++The 24 GB L4 the other workers use cannot hold the model. `cloudbuild.yaml`+defaults to `nvidia-rtx-pro-6000` with 20 CPU and 80Gi memory (Cloud Run's floor+for that class). Override `_GPU_TYPE`, `_CPU`, `_MEMORY` together if the class+changes. Scale-to-zero is the default; a warm floor is a spend decision.++## Endpoints++`POST /infer`, `GET /tasks/{id}` and `GET /results/{id}.glb` need+`Authorization: Bearer $API_KEY`. `GET /` and `GET /health` are open (`/health`+returns 503 once a model load failure latches).++`POST /infer` returns `202 {task_id}`:++```json+{ "image_url": "https://example.com/desk.jpg",+ "boxes": [[0.1, 0.2, 0.5, 0.9]],+ "max_objects": 8, "seed": 42 }+```++| Field | Meaning |+|---|---|+| `image_url` | Public https photo (SSRF-checked). |+| `mask_urls` or `boxes` | Exactly one. Masks are https PNGs, white is the object. Boxes are `[x0, y0, x1, y1]` fractions of the frame, origin top-left, turned into masks with GrabCut. |+| `max_objects` | 1 to 12, default 8. Slivers (under 4% of a side or 0.2% of the frame) are rejected. |+| `seed` | 0 to 4294967295. |++Bad requests are `422` from `request_policy.py`. A task that finishes holds+`result_gcs_url`, `objects` (name, position, size per node) and `skipped`+(objects that failed individually). It fails only if every object fails.++## Environment++| Variable | Default | Meaning |+|---|---|---|+| `API_KEY` | required | Bearer secret, from Secret Manager `avatar-reconstruction-key`. |+| `GCS_BUCKET` | unset | Upload to `raw-meshes/sam3d-objects/{task}.glb`; otherwise serve from `OUTPUT_DIR`. |+| `WEIGHTS_ROOT` | `/weights/sam3d-objects` | Mounted weights; `entrypoint.sh` copies `hf/` to `/app/checkpoints/hf`. |+| `SAM3D_CHECKPOINT_DIR`, `SAM3D_SRC` | image paths | Checkpoints and the pinned upstream checkout. |+| `MAX_CONCURRENT` | `1` | Simultaneous reconstructions per instance. |+| `MODEL_LOAD_ATTEMPTS`, `MODEL_LOAD_RETRY_BASE_S`, `MODEL_LOAD_RETRY_CAP_S` | `4`, `15`, `120` | Load retry policy. |+| `IMAGE_FETCH_TIMEOUT_S` | `60` | Photo and mask fetch timeout. |++## Tests++`python -m pytest test_request_policy.py test_scene_layout.py test_app_contract.py`+(the Dockerfile runs them as a build gate). They cover request validation, the pose+math, mask handling, scene assembly and the HTTP contract with a stub pipeline.+`worker_security.py` is vendored; `npm run check:vendored` keeps it identical.++## Not verified without a GPU and the weights++The image build, a real model run, and the pose orientation (checked against+upstream source only) have not been exercised. Whether `pipeline.yaml` pulls an+auxiliary depth model from Hugging Face at first load is not visible without the+gated file, so the container is not run offline and the first load may need HF+egress. Cloud Run availability of the GPU class in the chosen region is also+unconfirmed.++## License++SAM License, see the LICENSE copied beside the weights. Use must comply with+Trade Controls; the platform endpoint refuses requests from comprehensively+sanctioned territories before queueing.diff --git a/workers/model-sam3d-objects/cloudbuild.yaml b/workers/model-sam3d-objects/cloudbuild.yamlnew file mode 100644index 000000000..4fd290f72--- /dev/null+++ b/workers/model-sam3d-objects/cloudbuild.yaml@@ -0,0 +1,89 @@+# Build + deploy the SAM 3D Objects "scene from a photo" worker.+# Manual submit (SHORT_SHA is only auto-set by triggers):+# gcloud builds submit --config workers/model-sam3d-objects/cloudbuild.yaml \+# --substitutions=SHORT_SHA=manual$(date +%s) .+#+# Prerequisites, owner-run once:+# HF_TOKEN=... npm run stage:sam3d-objects -- --accept-sam-license+# GPU quota for the class named in _GPU_TYPE (see README: GPU class).+serviceAccount: projects/aerial-vehicle-466722-p5/serviceAccounts/[email protected]++substitutions:+ _REGION: us-central1+ _RUN_SA: avatar-reconstruction-sa@aerial-vehicle-466722-p5.iam.gserviceaccount.com+ _SERVICE: model-sam3d-objects+ _REPO: model-sam3d-objects+ _GCS_BUCKET: three-ws-avatar-reconstructions+ _WEIGHTS_BUCKET: three-ws-model-weights+ # Upstream requires 32 GB+ of VRAM. The L4 (24 GB) the other workers use cannot+ # run it. The 96 GB RTX Pro 6000 class can, and Cloud Run requires its larger+ # CPU/memory floor, hence the values below. Override both together if the GPU+ # type changes.+ _GPU_TYPE: nvidia-rtx-pro-6000+ _CPU: "20"+ _MEMORY: 80Gi+ # Scale-to-zero by default: a warm floor on a 96 GB GPU is a quota and spend+ # decision for the owner, not a default.+ _MIN_INSTANCES: "0"+ _MAX_INSTANCES: "1"++steps:+ - id: build+ name: gcr.io/cloud-builders/docker+ dir: workers/model-sam3d-objects+ args:+ - build+ - --tag+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA+ - --tag+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:latest+ - --cache-from+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:latest+ - .++ - id: push+ name: gcr.io/cloud-builders/docker+ args:+ - push+ - --all-tags+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server++ - id: deploy+ name: gcr.io/google.com/cloudsdktool/cloud-sdk:slim+ entrypoint: gcloud+ args:+ - run+ - deploy+ - ${_SERVICE}+ - --image=${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA+ - --region=${_REGION}+ - --service-account=${_RUN_SA}+ - --platform=managed+ - --port=8080+ - --gpu=1+ - --gpu-type=${_GPU_TYPE}+ - --no-gpu-zonal-redundancy+ - --no-cpu-throttling+ - --cpu=${_CPU}+ - --memory=${_MEMORY}+ - --min-instances=${_MIN_INSTANCES}+ - --max-instances=${_MAX_INSTANCES}+ - --timeout=900+ - --startup-probe=tcpSocket.port=8080,initialDelaySeconds=0,periodSeconds=60,timeoutSeconds=60,failureThreshold=15+ - --cpu-boost+ - --allow-unauthenticated+ - --add-volume=name=weights,type=cloud-storage,bucket=${_WEIGHTS_BUCKET}+ - --add-volume-mount=volume=weights,mount-path=/weights+ - --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},MAX_CONCURRENT=1,WEIGHTS_ROOT=/weights/sam3d-objects+ - --set-secrets=API_KEY=avatar-reconstruction-key:latest+ - --liveness-probe=httpGet.path=/health,initialDelaySeconds=60,periodSeconds=60,timeoutSeconds=10,failureThreshold=3++images:+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA+ - ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:latest++options:+ machineType: E2_HIGHCPU_32+ logging: CLOUD_LOGGING_ONLY++timeout: 3600sdiff --git a/workers/model-sam3d-objects/entrypoint.sh b/workers/model-sam3d-objects/entrypoint.shnew file mode 100755index 000000000..84343a4d8--- /dev/null+++ b/workers/model-sam3d-objects/entrypoint.sh@@ -0,0 +1,24 @@+#!/bin/sh+# Wire staged SAM 3D Objects checkpoints into the path pipeline.py loads from+# (SAM3D_CHECKPOINT_DIR), then serve.+#+# Checkpoints are COPIED from the FUSE-mounted bucket to local disk rather than+# symlinked: torch.load over gcsfuse does slow random reads, and a sequential+# cp is fast and deterministic.+set -eu++WEIGHTS_ROOT="${WEIGHTS_ROOT:-/weights/sam3d-objects}"+SAM3D_CHECKPOINT_DIR="${SAM3D_CHECKPOINT_DIR:-/app/checkpoints/hf}"++if [ ! -e "$WEIGHTS_ROOT/hf/pipeline.yaml" ]; then+ echo "fatal: expected staged checkpoints at $WEIGHTS_ROOT/hf (run: HF_TOKEN=... npm run stage:sam3d-objects -- --accept-sam-license)" >&2+ exit 1+fi++mkdir -p "$(dirname "$SAM3D_CHECKPOINT_DIR")"+rm -rf "$SAM3D_CHECKPOINT_DIR"+echo "staging $WEIGHTS_ROOT/hf -> $SAM3D_CHECKPOINT_DIR"+cp -r "$WEIGHTS_ROOT/hf" "$SAM3D_CHECKPOINT_DIR"+[ -f "$WEIGHTS_ROOT/LICENSE" ] && cp "$WEIGHTS_ROOT/LICENSE" "$SAM3D_CHECKPOINT_DIR/../SAM_LICENSE" || true++exec uvicorn main:app --host 0.0.0.0 --port 8080 --workers 1 --timeout-keep-alive 300diff --git a/workers/model-sam3d-objects/main.py b/workers/model-sam3d-objects/main.pynew file mode 100644index 000000000..152de5506--- /dev/null+++ b/workers/model-sam3d-objects/main.py@@ -0,0 +1,432 @@+"""+SAM 3D Objects inference service: one photo in, a 3D scene of its objects out+(facebookresearch/sam-3d-objects, SAM License; generative single-image 3D+object reconstruction with scene layout).++This is the Forge / AR Studio "From a photo" lane: the caller names the objects+in the photo (one mask PNG or one normalized box per object) and gets back one+GLB laid out like the photo, with a named node per object and a manifest of+each object's position and size.++API contract (same task shape as the other GPU workers):+ POST /infer { image_url: https URL to the photo,+ mask_urls?: [https URL of a PNG mask per object, white = object],+ boxes?: [[x0, y0, x1, y1] per object, fractions of the frame],+ max_objects?: int (default 8, max 12), seed?: int, job_id?: str }+ -> 202 { task_id, status: "queued" }++ GET /tasks/:id -> { task_id, status, result_gcs_url?, result_url?, objects?,+ object_count?, skipped?, error? }+ GET /results/:id.glb -> the scene GLB when OUTPUT_DIR local storage is active+ GET /health -> { ok, model, gpu_available, load_error, load_attempts };+ 503 once the model load has spent its retry budget.+ GET / -> unauthenticated descriptor for the platform warmth probe.++Weights: gs://three-ws-model-weights/sam3d-objects (npm run stage:sam3d-objects).++Environment variables (README.md carries the full table):+ API_KEY, GCS_BUCKET, OUTPUT_DIR, MAX_CONCURRENT (default 1),+ SAM3D_CHECKPOINT_DIR (default /app/checkpoints/hf), SAM3D_SRC (default /app/sam3d-objects),+ IMAGE_FETCH_TIMEOUT_S, MODEL_LOAD_ATTEMPTS, MODEL_LOAD_RETRY_BASE_S / _CAP_S+"""++from __future__ import annotations++import asyncio+import gc+import io+import json+import logging+import os+import time+import uuid+from contextlib import asynccontextmanager+from pathlib import Path+from typing import Optional+from urllib.parse import urlsplit++import httpx+import numpy as np+from fastapi import BackgroundTasks, FastAPI, Header, HTTPException, Response+from fastapi.responses import FileResponse+from google.api_core.exceptions import NotFound+from google.cloud import storage+from pydantic import BaseModel, Field++import scene_layout as layout+from request_policy import RequestRejected, validate_request+from storage_backend import LocalStorage+from worker_security import UnsafeUrlError, fetch_remote_bytes, require_api_key, safe_error++logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")+log = logging.getLogger("model-sam3d-objects")++API_KEY = os.environ["API_KEY"]+GCS_BUCKET = os.environ.get("GCS_BUCKET", "").strip()+OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "/output")+CHECKPOINT_DIR = os.environ.get("SAM3D_CHECKPOINT_DIR", "/app/checkpoints/hf")+SAM3D_SRC = os.environ.get("SAM3D_SRC", "/app/sam3d-objects")+MAX_CONCURRENT = int(os.environ.get("MAX_CONCURRENT", "1"))+MODEL_LOAD_ATTEMPTS = int(os.environ.get("MODEL_LOAD_ATTEMPTS", "4"))+MODEL_LOAD_RETRY_BASE_S = float(os.environ.get("MODEL_LOAD_RETRY_BASE_S", "15"))+MODEL_LOAD_RETRY_CAP_S = float(os.environ.get("MODEL_LOAD_RETRY_CAP_S", "120"))+IMAGE_FETCH_TIMEOUT_S = float(os.environ.get("IMAGE_FETCH_TIMEOUT_S", "60"))+FETCH_ATTEMPTS = 3+MAX_IMAGE_BYTES = 32 * 1024 * 1024+MAX_MASK_BYTES = 16 * 1024 * 1024+# The model works at a fixed internal resolution; a larger photo only costs memory.+MAX_PHOTO_SIDE = 2048+MAX_PHOTO_PIXELS = 40_000_000++_bucket: Optional[storage.Bucket] = None+_local_storage: Optional[LocalStorage] = None+_sem: Optional[asyncio.Semaphore] = None+_ready: Optional[asyncio.Event] = None+_load_error: Optional[str] = None+_load_attempts = 0+_pipeline_loaded = False+_tasks: dict[str, dict] = {}+_task_locks: dict[str, asyncio.Lock] = {}+++def _torch():+ try:+ import torch++ return torch+ except ImportError:+ return None+++def _cuda_available() -> bool:+ torch = _torch()+ return bool(torch and torch.cuda.is_available())+++def _load_pipeline_sync() -> None:+ global _pipeline_loaded+ from pipeline import load_pipeline++ log.info("Loading SAM 3D Objects: checkpoints=%s", CHECKPOINT_DIR)+ load_pipeline(CHECKPOINT_DIR, SAM3D_SRC)+ _pipeline_loaded = True+ log.info("SAM 3D Objects loaded")+++async def _load_pipeline_bg() -> None:+ """Load off the request path; retry with backoff before latching a failure."""+ global _load_error, _load_attempts+ loop = asyncio.get_event_loop()+ attempts = max(1, MODEL_LOAD_ATTEMPTS)+ for attempt in range(1, attempts + 1):+ _load_attempts = attempt+ try:+ await loop.run_in_executor(None, _load_pipeline_sync)+ _load_error = None+ _ready.set()+ return+ except Exception as exc: # noqa: BLE001 - surfaced via /health and task status+ if attempt >= attempts:+ _load_error = safe_error(exc, context="model load")+ log.error("SAM 3D Objects load FAILED after %d attempt(s): %s", attempt, exc)+ return+ delay = min(MODEL_LOAD_RETRY_BASE_S * (2 ** (attempt - 1)), MODEL_LOAD_RETRY_CAP_S)+ log.warning("SAM 3D Objects load attempt %d/%d failed (%s); retrying in %.0fs", attempt, attempts, exc, delay)+ await asyncio.sleep(delay)+++@asynccontextmanager+async def lifespan(app: FastAPI):+ global _bucket, _local_storage, _sem, _ready+ if GCS_BUCKET:+ _bucket = storage.Client().bucket(GCS_BUCKET)+ _local_storage = None+ else:+ _bucket = None+ _local_storage = LocalStorage(OUTPUT_DIR)+ _sem = asyncio.Semaphore(MAX_CONCURRENT)+ _ready = asyncio.Event()+ asyncio.create_task(_load_pipeline_bg())+ log.info("Service starting, pipeline loading in background (max_concurrent=%d)", MAX_CONCURRENT)+ yield+++app = FastAPI(title="model-sam3d-objects", lifespan=lifespan)+++def _require_api_key(authorization: str) -> None:+ try:+ require_api_key(authorization, API_KEY)+ except PermissionError as exc:+ raise HTTPException(status_code=401, detail=str(exc)) from exc+++class ImageSourceError(ValueError):+ """A caller-supplied photo or mask could not be read."""+++def _source_label(src: str) -> str:+ return urlsplit(src).hostname or src[:60]+++def _fetch_bytes(url: str, what: str, max_bytes: int) -> bytes:+ label = _source_label(url)+ last_exc: Optional[BaseException] = None+ for attempt in range(1, FETCH_ATTEMPTS + 1):+ try:+ return fetch_remote_bytes(url, timeout=IMAGE_FETCH_TIMEOUT_S, max_bytes=max_bytes)+ except UnsafeUrlError as exc:+ raise ImageSourceError(f"refused to fetch {what} ({label}): {exc}") from exc+ except httpx.HTTPStatusError as exc:+ raise ImageSourceError(f"{what} source {label} returned HTTP {exc.response.status_code}") from exc+ except httpx.HTTPError as exc:+ last_exc = exc+ if attempt < FETCH_ATTEMPTS:+ time.sleep(min(2 ** attempt, 5))+ except ValueError as exc:+ raise ImageSourceError(f"{what} source {label} rejected: {exc}") from exc+ raise ImageSourceError(+ f"{what} source {label} unreachable after {FETCH_ATTEMPTS} attempts "+ f"({type(last_exc).__name__}); check the URL is publicly readable"+ )+++def _load_photo(url: str) -> np.ndarray:+ from PIL import Image, ImageOps++ data = _fetch_bytes(url, "photo", MAX_IMAGE_BYTES)+ try:+ Image.MAX_IMAGE_PIXELS = MAX_PHOTO_PIXELS+ img = Image.open(io.BytesIO(data))+ img.load()+ except Exception as exc: # noqa: BLE001 - PIL raises many types, including decompression bombs+ raise ImageSourceError("photo is not a readable image (use PNG, JPEG or WebP)") from exc+ img = ImageOps.exif_transpose(img).convert("RGB")+ if max(img.size) > MAX_PHOTO_SIDE:+ scale = MAX_PHOTO_SIDE / max(img.size)+ img = img.resize((max(1, round(img.width * scale)), max(1, round(img.height * scale))), Image.LANCZOS)+ return np.asarray(img)+++def _build_masks(request: dict, photo: np.ndarray) -> tuple[list[np.ndarray], list[str]]:+ height, width = photo.shape[:2]+ masks: list[np.ndarray] = []+ if request["mask_urls"]:+ for i, url in enumerate(request["mask_urls"]):+ data = _fetch_bytes(url, f"mask {i + 1}", MAX_MASK_BYTES)+ try:+ masks.append(layout.mask_from_png_bytes(data, width, height))+ except ValueError as exc:+ raise ImageSourceError(f"mask {i + 1}: {exc}") from exc+ return masks, ["mask"] * len(masks)+ for box in request["boxes"]:+ masks.append(layout.grabcut_mask(photo, layout.box_to_pixels(box, width, height)))+ return masks, ["box"] * len(masks)+++def _task_blob(task_id: str, name: str | None = None):+ if _bucket is None:+ raise RuntimeError("GCS task storage is not active")+ return _bucket.blob(name or f"tasks/{task_id}.json")+++def _using_gcs() -> bool:+ return bool(GCS_BUCKET)+++def _get_local_storage() -> LocalStorage:+ global _local_storage+ if _local_storage is None:+ _local_storage = LocalStorage(OUTPUT_DIR)+ return _local_storage+++async def _update_task(task_id: str, **fields) -> dict:+ lock = _task_locks.setdefault(task_id, asyncio.Lock())+ async with lock:+ task = _tasks.setdefault(task_id, {"task_id": task_id})+ task.update(fields)+ task["updated_at"] = time.time()+ snapshot = task.copy()+ loop = asyncio.get_event_loop()+ if _using_gcs():+ payload = json.dumps(snapshot)+ await loop.run_in_executor(+ None, lambda: _task_blob(task_id).upload_from_string(payload, content_type="application/json")+ )+ else:+ await loop.run_in_executor(None, _get_local_storage().write_task, snapshot)+ return task+++_TERMINAL_STATUSES = frozenset({"done", "failed"})+++async def _resolve_task(task_id: str) -> dict:+ task = _tasks.get(task_id)+ if task is not None and task.get("status") in _TERMINAL_STATUSES:+ return task+ loop = asyncio.get_event_loop()+ try:+ if _using_gcs():+ data = await loop.run_in_executor(None, _task_blob(task_id).download_as_bytes)+ task = json.loads(data)+ else:+ task = await loop.run_in_executor(None, _get_local_storage().read_task, task_id)+ except (NotFound, FileNotFoundError, ValueError):+ if task is not None:+ return task+ raise HTTPException(status_code=404, detail="task not found")+ except Exception as exc:+ raise HTTPException(status_code=502, detail=safe_error(exc, context="task lookup")) from exc+ if task.get("status") in _TERMINAL_STATUSES:+ _tasks[task_id] = task+ return task+++def _generate_sync(request: dict) -> dict:+ from pipeline import run_objects++ photo = _load_photo(request["image_url"])+ masks, sources = _build_masks(request, photo)+ return run_objects(photo, masks, sources, request["seed"] if request["seed"] is not None else 42)+++def _release_gpu_memory() -> None:+ gc.collect()+ torch = _torch()+ if torch and torch.cuda.is_available():+ torch.cuda.empty_cache()+ torch.cuda.ipc_collect()+++async def _run_job(task_id: str, request: dict) -> None:+ from pipeline import ObjectReconstructionError++ if _load_error:+ await _update_task(task_id, status="failed", error=f"pipeline unavailable: {_load_error}")+ return+ try:+ await asyncio.wait_for(_ready.wait(), timeout=900)+ except asyncio.TimeoutError:+ await _update_task(task_id, status="failed", error="pipeline not ready (model load timed out)")+ return+ if _load_error:+ await _update_task(task_id, status="failed", error=f"pipeline unavailable: {_load_error}")+ return++ async with _sem:+ await _update_task(task_id, status="running")+ loop = asyncio.get_event_loop()+ t0 = time.time()+ try:+ result = await loop.run_in_executor(None, _generate_sync, request)+ glb_bytes = result["scene_glb"]+ if _using_gcs():+ blob_name = f"raw-meshes/sam3d-objects/{task_id}.glb"+ blob = _bucket.blob(blob_name)+ await loop.run_in_executor(+ None, lambda: blob.upload_from_string(glb_bytes, content_type="model/gltf-binary")+ )+ result_fields = {"result_gcs_url": f"https://storage.googleapis.com/{GCS_BUCKET}/{blob_name}"}+ else:+ result_path = await loop.run_in_executor(None, _get_local_storage().write_result, task_id, glb_bytes)+ result_fields = {"result_url": f"/results/{task_id}.glb", "result_path": str(result_path)}+ elapsed = time.time() - t0+ await _update_task(+ task_id,+ status="done",+ **result_fields,+ objects=result["objects"],+ object_count=len(result["objects"]),+ skipped=result["skipped"],+ bytes=len(glb_bytes),+ elapsed_ms=int(elapsed * 1000),+ )+ log.info("[%s] done in %.1fs (objects=%d) %d bytes", task_id, elapsed, len(result["objects"]), len(glb_bytes))+ except (ImageSourceError, ObjectReconstructionError) as exc:+ log.warning("[%s] rejected: %s", task_id, exc)+ await _update_task(task_id, status="failed", error=str(exc), elapsed_ms=int((time.time() - t0) * 1000))+ except Exception as exc:+ await _update_task(+ task_id, status="failed", error=safe_error(exc, context=f"[{task_id}] inference"),+ elapsed_ms=int((time.time() - t0) * 1000),+ )+ finally:+ await loop.run_in_executor(None, _release_gpu_memory)+++class InferRequest(BaseModel):+ image_url: str = Field(..., min_length=1, max_length=4096)+ mask_urls: list[str] | None = None+ boxes: list[list[float]] | None = None+ max_objects: int | None = None+ seed: int | None = None+ job_id: str | None = None++[email protected]("/infer", status_code=202)+async def infer(body: InferRequest, background_tasks: BackgroundTasks, authorization: str = Header(...)) -> dict:+ _require_api_key(authorization)+ if _load_error:+ raise HTTPException(status_code=503, detail=f"pipeline unavailable: {_load_error}")+ payload = body.model_dump(exclude_none=True)+ try:+ request = validate_request(payload)+ except RequestRejected as exc:+ raise HTTPException(status_code=422, detail=str(exc)) from exc+ task_id = str(uuid.uuid4())+ await _update_task(task_id, status="queued", model="sam3d-objects")+ background_tasks.add_task(_run_job, task_id, request)+ return {"task_id": task_id, "status": "queued"}++[email protected]("/tasks/{task_id}")+async def get_task(task_id: str, authorization: str = Header(...)) -> dict:+ _require_api_key(authorization)+ return await _resolve_task(task_id)++[email protected]("/results/{task_id}.glb")+async def get_result(task_id: str, authorization: str = Header(...)):+ _require_api_key(authorization)+ if _using_gcs():+ raise HTTPException(status_code=404, detail="local result storage is not active")+ try:+ path = _get_local_storage().result_path(task_id)+ except ValueError as exc:+ raise HTTPException(status_code=404, detail="result not found") from exc+ if not path.is_file():+ raise HTTPException(status_code=404, detail="result not found")+ return FileResponse(path, media_type="model/gltf-binary", filename=path.name)++[email protected]("/")+async def root() -> dict:+ return {+ "service": "model-sam3d-objects",+ "model": "sam3d-objects",+ "ready": bool(_ready and _ready.is_set()),+ "endpoints": ["POST /infer", "GET /tasks/{task_id}", "GET /results/{task_id}.glb", "GET /health"],+ }++[email protected]("/health")+async def health(response: Response) -> dict:+ dead = _load_error is not None+ if dead:+ response.status_code = 503+ torch = _torch()+ cuda = _cuda_available()+ return {+ "ok": not dead,+ "model": "sam3d-objects",+ "gpu_available": cuda,+ "gpu_name": torch.cuda.get_device_name(0) if cuda else None,+ "gpu_memory_gb": round(torch.cuda.get_device_properties(0).total_memory / 2**30, 1) if cuda else None,+ "pipeline_loaded": _pipeline_loaded,+ "ready": bool(_ready and _ready.is_set()),+ "load_error": _load_error,+ "load_attempts": _load_attempts,+ "output_backend": "gcs" if _using_gcs() else "local",+ }diff --git a/workers/model-sam3d-objects/pipeline.py b/workers/model-sam3d-objects/pipeline.pynew file mode 100644index 000000000..1f4e256a4--- /dev/null+++ b/workers/model-sam3d-objects/pipeline.py@@ -0,0 +1,114 @@+"""SAM 3D Objects (facebookresearch/sam-3d-objects, SAM License) pipeline wrapper.++One photo plus one mask per object in, one posed mesh per object out. Loading+and running the model needs torch, pytorch3d, kaolin and a 32 GB+ GPU, so every+heavy import lives inside a function: this module imports on a laptop and the+pure logic (masks, boxes, layout) is covered by test_scene_layout.py.++Upstream exposes its inference class only as notebook/inference.py in the+repository (it is not part of the pip package), so the Dockerfile leaves the+checkout at SAM3D_SRC and this module puts SAM3D_SRC/notebook on sys.path,+exactly as upstream's README does.+"""++from __future__ import annotations++import logging+import os+import sys+import threading+from pathlib import Path++import numpy as np++import scene_layout as layout++log = logging.getLogger("model-sam3d-objects.pipeline")++_lock = threading.Lock()+_inference = None++MIN_OBJECT_FACES = 8+++class ObjectReconstructionError(RuntimeError):+ """The model ran but produced no usable mesh for any requested object."""+++def load_pipeline(checkpoint_dir: str, sam3d_src: str):+ """Load (once) and return upstream's Inference singleton."""+ global _inference+ with _lock:+ if _inference is None:+ config = Path(checkpoint_dir) / "pipeline.yaml"+ if not config.is_file():+ raise FileNotFoundError(+ f"{config} is missing: stage the SAM 3D Objects checkpoints with "+ "`npm run stage:sam3d-objects -- --accept-sam-license`"+ )+ # upstream's inference.py reads these at import time+ os.environ.setdefault("CONDA_PREFIX", os.environ.get("CUDA_HOME", "/usr/local/cuda"))+ os.environ.setdefault("LIDRA_SKIP_INIT", "true")+ notebook = str(Path(sam3d_src) / "notebook")+ if notebook not in sys.path:+ sys.path.append(notebook)+ from inference import Inference # type: ignore[import-not-found]++ log.info("loading SAM 3D Objects: config=%s", config)+ _inference = Inference(str(config), compile=False)+ return _inference+++def _to_numpy(value) -> np.ndarray:+ if hasattr(value, "detach"):+ value = value.detach().float().cpu().numpy()+ return np.asarray(value, dtype=np.float64)+++def run_objects(image_rgb: np.ndarray, masks: list[np.ndarray], sources: list[str], seed: int) -> dict:+ """Reconstruct every masked object and lay the results out like the photo.++ Returns {"scene_glb": bytes, "objects": [manifest entries], "skipped": [...]}.+ The scene GLB has one named node per object (object_01, object_02, ...).+ A single object that fails is skipped and reported; the call fails only when+ none succeeds.+ """+ inference = _inference+ if inference is None:+ raise RuntimeError("pipeline is not loaded")++ posed_items: list[dict] = []+ manifest: list[dict] = []+ skipped: list[dict] = []++ for i, (mask, source) in enumerate(zip(masks, sources)):+ try:+ out = inference(image_rgb, mask, seed=seed)+ mesh = out.get("glb")+ if mesh is None or len(mesh.faces) < MIN_OBJECT_FACES:+ raise ObjectReconstructionError("the model returned no mesh for this object")+ verts = np.asarray(mesh.vertices, dtype=np.float64)+ faces = np.asarray(mesh.faces, dtype=np.int64)+ colors = None+ vis = getattr(mesh, "visual", None)+ if vis is not None and getattr(vis, "kind", None) == "vertex":+ colors = np.asarray(vis.vertex_colors)+ quat = _to_numpy(out["rotation"]).reshape(-1)[:4]+ trans = _to_numpy(out["translation"]).reshape(-1)[:3]+ scale = _to_numpy(out["scale"]).reshape(-1)+ posed = layout.pose_vertices(verts, quat, trans, scale)+ solo = layout.standalone_vertices(verts, scale)+ except Exception as exc: # noqa: BLE001 - one bad object must not sink the scene+ log.warning("object %d failed: %s", i + 1, exc)+ skipped.append({"index": i, "reason": str(exc)[:200]})+ continue+ n = len(posed_items)+ posed_items.append({"vertices": posed, "faces": faces, "vertex_colors": colors})+ manifest.append(layout.manifest_entry(n, posed, faces, solo, quat, trans, scale, sources[i]))++ if not posed_items:+ reasons = "; ".join(s["reason"] for s in skipped) or "no objects"+ raise ObjectReconstructionError(f"no object could be reconstructed ({reasons})")++ scene_glb = layout.export_glb(layout.build_scene(posed_items))+ return {"scene_glb": scene_glb, "objects": manifest, "skipped": skipped}diff --git a/workers/model-sam3d-objects/request_policy.py b/workers/model-sam3d-objects/request_policy.pynew file mode 100644index 000000000..8936004ed--- /dev/null+++ b/workers/model-sam3d-objects/request_policy.py@@ -0,0 +1,107 @@+"""Request validation for the SAM 3D Objects worker.++Pure Python (stdlib only) so it unit-tests without a GPU, torch or FastAPI.+Everything a caller can send is bounded here, before any URL is fetched or any+GPU time is spent, and every rejection is a caller-facing sentence.++A request names ONE photo and the objects to lift out of it. SAM 3D Objects+reconstructs an object from the photo plus a mask that says which pixels are+that object, so each object is given as either:++ * ``mask_urls``: one public https PNG per object (white = the object), or+ * ``boxes``: one normalized ``[x0, y0, x1, y1]`` box per object (0..1, origin+ top-left); the worker turns each box into a mask with GrabCut.++Masks win when both are present for the same request (a mask is exact, a box is+an estimate), and the two are never mixed within one request.+"""++from __future__ import annotations++import math+from typing import Any++# One photo can hold a roomful, but each object is a full diffusion pass and the+# scene is one GLB a phone has to open. Twelve is a generous room, not a stress test.+MAX_OBJECTS = 12+DEFAULT_MAX_OBJECTS = 8+MIN_BOX_SIDE = 0.04 # 4% of the frame: smaller is a speck the model cannot reconstruct+MIN_BOX_AREA = 0.002+MAX_URL_CHARS = 4096+SEED_MAX = 4_294_967_295+++class RequestRejected(ValueError):+ """The request is malformed; the message is safe to show the caller."""+++def _url(value: Any, field: str) -> str:+ if not isinstance(value, str) or not value.strip():+ raise RequestRejected(f"{field} must be a non-empty https URL")+ value = value.strip()+ if len(value) > MAX_URL_CHARS:+ raise RequestRejected(f"{field} is longer than {MAX_URL_CHARS} characters")+ return value+++def _box(raw: Any, index: int) -> tuple[float, float, float, float]:+ label = f"boxes[{index}]"+ if not isinstance(raw, (list, tuple)) or len(raw) != 4:+ raise RequestRejected(f"{label} must be [x0, y0, x1, y1]")+ try:+ x0, y0, x1, y1 = (float(v) for v in raw)+ except (TypeError, ValueError) as exc:+ raise RequestRejected(f"{label} must contain numbers") from exc+ if not all(math.isfinite(v) for v in (x0, y0, x1, y1)):+ raise RequestRejected(f"{label} must contain finite numbers")+ if not (0.0 <= x0 < x1 <= 1.0 and 0.0 <= y0 < y1 <= 1.0):+ raise RequestRejected(f"{label} must satisfy 0 <= x0 < x1 <= 1 and 0 <= y0 < y1 <= 1 (fractions of the frame)")+ if (x1 - x0) < MIN_BOX_SIDE or (y1 - y0) < MIN_BOX_SIDE or (x1 - x0) * (y1 - y0) < MIN_BOX_AREA:+ raise RequestRejected(f"{label} is too small to reconstruct; choose an object that fills at least 4% of the frame width and height")+ return (x0, y0, x1, y1)+++def validate_request(payload: dict) -> dict:+ """Return a normalized request or raise RequestRejected."""+ if not isinstance(payload, dict):+ raise RequestRejected("request body must be a JSON object")++ image_url = _url(payload.get("image_url"), "image_url")++ max_objects = payload.get("max_objects", DEFAULT_MAX_OBJECTS)+ if isinstance(max_objects, bool) or not isinstance(max_objects, int) or not 1 <= max_objects <= MAX_OBJECTS:+ raise RequestRejected(f"max_objects must be an integer from 1 to {MAX_OBJECTS}")++ seed = payload.get("seed")+ if seed is not None and (isinstance(seed, bool) or not isinstance(seed, int) or not 0 <= seed <= SEED_MAX):+ raise RequestRejected("seed must be a non-negative integer")++ raw_masks = payload.get("mask_urls")+ raw_boxes = payload.get("boxes")+ if raw_masks is not None and raw_boxes is not None and raw_masks and raw_boxes:+ raise RequestRejected("send mask_urls or boxes, not both")++ mask_urls: list[str] = []+ boxes: list[tuple[float, float, float, float]] = []+ if raw_masks:+ if not isinstance(raw_masks, list):+ raise RequestRejected("mask_urls must be a list of https URLs")+ mask_urls = [_url(u, f"mask_urls[{i}]") for i, u in enumerate(raw_masks)]+ elif raw_boxes:+ if not isinstance(raw_boxes, list):+ raise RequestRejected("boxes must be a list of [x0, y0, x1, y1] boxes")+ boxes = [_box(b, i) for i, b in enumerate(raw_boxes)]+ else:+ raise RequestRejected("name the objects to reconstruct: send mask_urls (one PNG mask per object) or boxes (one [x0, y0, x1, y1] per object)")++ count = len(mask_urls) or len(boxes)+ if count > max_objects:+ raise RequestRejected(f"{count} objects requested but max_objects is {max_objects}")++ return {+ "image_url": image_url,+ "mask_urls": mask_urls,+ "boxes": boxes,+ "max_objects": max_objects,+ "seed": seed,+ }diff --git a/workers/model-sam3d-objects/requirements.txt b/workers/model-sam3d-objects/requirements.txtnew file mode 100644index 000000000..9b908acb5--- /dev/null+++ b/workers/model-sam3d-objects/requirements.txt@@ -0,0 +1,15 @@+# This worker's own FastAPI/GCS layer. torch, pytorch3d, kaolin, gsplat and the+# rest of SAM 3D Objects' stack install in their own Dockerfile layers BEFORE+# this file (upstream is installed from its pinned checkout).+trimesh>=4.4.0,<5+numpy>=1.24,<2+Pillow>=10,<12++fastapi>=0.115,<1+uvicorn[standard]>=0.32,<1+pydantic>=2.0.0+httpx>=0.27.0++google-cloud-storage==2.18.0++pytest>=8,<9diff --git a/workers/model-sam3d-objects/scene_layout.py b/workers/model-sam3d-objects/scene_layout.pynew file mode 100644index 000000000..de4aa9d7a--- /dev/null+++ b/workers/model-sam3d-objects/scene_layout.py@@ -0,0 +1,189 @@+"""Pure geometry for the SAM 3D Objects worker: masks, poses and the scene GLB.++numpy, PIL and trimesh only (cv2 is imported lazily for GrabCut), so it+unit-tests without torch or a GPU. The model-facing code lives in pipeline.py.++Conventions, from facebookresearch/sam-3d-objects at the pinned commit:++ * ``output["glb"]`` is a trimesh in a Y-up frame. Upstream's ``to_glb`` makes+ it from the decoder's native Z-up vertices with ``v @ DECODER_TO_GLB``.+ * ``output["rotation"|"translation"|"scale"]`` place the object in the camera+ frame of the photo, applied to the decoder-native vertices (upstream's+ ``make_scene`` applies them to the gaussians' native xyz). The rotation is a+ PyTorch3D quaternion, real part first.+ * The camera frame is PyTorch3D's: +X left, +Y up, +Z away from the camera.+ glTF viewers look down -Z with +X right, so the final conversion flips X and Z.++The returned scene therefore reads like the photo: the camera sits at the origin+looking down -Z, objects stand where the photo shows them, at the model's own+relative scale (the layout is relative, not metric: no tape measure was in frame).+"""++from __future__ import annotations++import io+from typing import Sequence++import numpy as np++# upstream to_glb: vertices @ [[1, 0, 0], [0, 0, -1], [0, 1, 0]] => (x, y, z) -> (x, z, -y)+_DECODER_TO_GLB = np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]], dtype=np.float64)+_GLB_TO_DECODER = _DECODER_TO_GLB.T # orthonormal: the inverse is the transpose+_CAMERA_TO_GLTF = np.diag([-1.0, 1.0, -1.0])+++def glb_to_decoder_vertices(vertices: np.ndarray) -> np.ndarray:+ """Undo upstream's decoder-to-GLB axis change."""+ return np.asarray(vertices, dtype=np.float64) @ _GLB_TO_DECODER+++def quat_wxyz_to_matrix(q: Sequence[float]) -> np.ndarray:+ """Rotation matrix for a real-first quaternion (PyTorch3D convention)."""+ w, x, y, z = (float(v) for v in q)+ n = w * w + x * x + y * y + z * z+ if not np.isfinite(n) or n < 1e-12:+ raise ValueError("quaternion has no rotation")+ s = 2.0 / n+ return np.array(+ [+ [1 - s * (y * y + z * z), s * (x * y - z * w), s * (x * z + y * w)],+ [s * (x * y + z * w), 1 - s * (x * x + z * z), s * (y * z - x * w)],+ [s * (x * z - y * w), s * (y * z + x * w), 1 - s * (x * x + y * y)],+ ],+ dtype=np.float64,+ )+++def pose_vertices(+ glb_vertices: np.ndarray,+ quat_wxyz: Sequence[float],+ translation: Sequence[float],+ scale: Sequence[float] | float,+) -> np.ndarray:+ """Place an object's GLB-frame vertices in the glTF scene frame of the photo."""+ native = glb_to_decoder_vertices(glb_vertices)+ scale_vec = np.broadcast_to(np.asarray(scale, dtype=np.float64).reshape(-1), (3,)) if np.ndim(scale) else np.full(3, float(scale))+ posed = (native * scale_vec) @ quat_wxyz_to_matrix(quat_wxyz).T + np.asarray(translation, dtype=np.float64).reshape(3)+ return posed @ _CAMERA_TO_GLTF.T+++def standalone_vertices(glb_vertices: np.ndarray, scale: Sequence[float] | float) -> np.ndarray:+ """The object on its own: scaled, centred on X/Z and resting on y = 0."""+ verts = np.asarray(glb_vertices, dtype=np.float64)+ scale_vec = np.broadcast_to(np.asarray(scale, dtype=np.float64).reshape(-1), (3,)) if np.ndim(scale) else np.full(3, float(scale))+ # Scale in the decoder frame, where the model's axes mean what the scale means.+ native = glb_to_decoder_vertices(verts) * scale_vec+ out = native @ _DECODER_TO_GLB+ lo, hi = out.min(axis=0), out.max(axis=0)+ out[:, 0] -= (lo[0] + hi[0]) / 2.0+ out[:, 2] -= (lo[2] + hi[2]) / 2.0+ out[:, 1] -= lo[1]+ return out+++def box_to_pixels(box: Sequence[float], width: int, height: int) -> tuple[int, int, int, int]:+ """Normalized [x0, y0, x1, y1] to an integer pixel rectangle inside the image."""+ x0, y0, x1, y1 = box+ px0 = min(max(int(round(x0 * width)), 0), width - 1)+ py0 = min(max(int(round(y0 * height)), 0), height - 1)+ px1 = min(max(int(round(x1 * width)), px0 + 1), width)+ py1 = min(max(int(round(y1 * height)), py0 + 1), height)+ return px0, py0, px1, py1+++def mask_from_png_bytes(data: bytes, width: int, height: int) -> np.ndarray:+ """Decode a caller's mask PNG to a boolean (height, width) array.++ An RGBA mask uses its alpha channel when the alpha is not constant,+ otherwise its luminance. The mask is resized to the photo with nearest+ neighbour so a slightly different export size still lines up.+ """+ from PIL import Image++ try:+ img = Image.open(io.BytesIO(data))+ img.load()+ except Exception as exc: # noqa: BLE001 - PIL raises many types for a bad file+ raise ValueError("mask is not a readable image") from exc+ if img.mode in ("RGBA", "LA"):+ alpha = np.asarray(img.getchannel("A"))+ channel = alpha if alpha.min() != alpha.max() else np.asarray(img.convert("L"))+ else:+ channel = np.asarray(img.convert("L"))+ if channel.shape != (height, width):+ channel = np.asarray(Image.fromarray(channel).resize((width, height), Image.NEAREST))+ mask = channel > 127+ if not mask.any():+ raise ValueError("mask is empty (no white pixels)")+ return mask+++def grabcut_mask(image_rgb: np.ndarray, box_px: tuple[int, int, int, int], iterations: int = 5) -> np.ndarray:+ """Estimate an object mask from a box with GrabCut (OpenCV, Apache-2.0)."""+ import cv2++ x0, y0, x1, y1 = box_px+ h, w = image_rgb.shape[:2]+ mask = np.zeros((h, w), np.uint8)+ rect = (x0, y0, x1 - x0, y1 - y0)+ bgd = np.zeros((1, 65), np.float64)+ fgd = np.zeros((1, 65), np.float64)+ cv2.grabCut(cv2.cvtColor(image_rgb, cv2.COLOR_RGB2BGR), mask, rect, bgd, fgd, iterations, cv2.GC_INIT_WITH_RECT)+ out = (mask == cv2.GC_FGD) | (mask == cv2.GC_PR_FGD)+ box_area = max((x1 - x0) * (y1 - y0), 1)+ # GrabCut can collapse to almost nothing on a low-contrast box; the box itself+ # is then the honest answer, because an empty mask would fail the model.+ if out.sum() < 0.05 * box_area:+ out = np.zeros((h, w), bool)+ out[y0:y1, x0:x1] = True+ return out+++def object_name(index: int) -> str:+ return f"object_{index + 1:02d}"+++def build_scene(objects: Sequence[dict]):+ """One trimesh.Scene, one named node per object, laid out like the photo.++ Each item needs ``vertices`` (already posed), ``faces`` and optional+ ``vertex_colors``.+ """+ import trimesh++ scene = trimesh.Scene()+ for i, obj in enumerate(objects):+ name = object_name(i)+ mesh = trimesh.Trimesh(vertices=obj["vertices"], faces=obj["faces"], process=False)+ if obj.get("vertex_colors") is not None:+ mesh.visual.vertex_colors = obj["vertex_colors"]+ scene.add_geometry(mesh, node_name=name, geom_name=name)+ return scene+++def export_glb(scene_or_mesh) -> bytes:+ buf = io.BytesIO()+ scene_or_mesh.export(buf, file_type="glb")+ return buf.getvalue()+++def _round(values, n=4):+ return [round(float(v), n) for v in np.asarray(values).reshape(-1)]+++def manifest_entry(index: int, posed: np.ndarray, faces: np.ndarray, standalone: np.ndarray,+ quat_wxyz, translation, scale, source: str) -> dict:+ """Machine-readable record of one reconstructed object."""+ lo, hi = posed.min(axis=0), posed.max(axis=0)+ size = standalone.max(axis=0) - standalone.min(axis=0)+ return {+ "id": object_name(index),+ "name": object_name(index),+ "source": source, # "mask" or "box"+ "face_count": int(len(faces)),+ "vertex_count": int(len(posed)),+ "bbox": {"min": _round(lo), "max": _round(hi)},+ "centroid": _round((lo + hi) / 2.0),+ "size": _round(size), # width, height, depth of the object on its own, layout units+ "pose": {"translation": _round(translation), "rotation_wxyz": _round(quat_wxyz, 6), "scale": _round(scale)},+ }diff --git a/workers/model-sam3d-objects/storage_backend.py b/workers/model-sam3d-objects/storage_backend.pynew file mode 100644index 000000000..e90e2752d--- /dev/null+++ b/workers/model-sam3d-objects/storage_backend.py@@ -0,0 +1,63 @@+"""Storage primitives for the SAM 3D Objects worker.++Production persists tasks and results in GCS because requests can move between+Cloud Run instances. An NGC user runs one container with a mounted output+volume, so the same service needs a local backend with atomic writes and strict+task-id validation.+"""++from __future__ import annotations++import json+import os+import uuid+from pathlib import Path+++def normalize_task_id(task_id: str) -> str:+ """Return a canonical UUID or reject a value that could escape the volume."""+ parsed = uuid.UUID(task_id)+ canonical = str(parsed)+ if task_id.lower() != canonical:+ raise ValueError("task_id must be a canonical UUID")+ return canonical+++class LocalStorage:+ """Durable task metadata and GLB results on a mounted local volume."""++ def __init__(self, output_dir: str | os.PathLike[str]):+ self.output_dir = Path(output_dir).expanduser().resolve()+ self.tasks_dir = self.output_dir / "tasks"+ self.results_dir = self.output_dir / "results"+ self.tasks_dir.mkdir(parents=True, exist_ok=True)+ self.results_dir.mkdir(parents=True, exist_ok=True)++ def task_path(self, task_id: str) -> Path:+ return self.tasks_dir / f"{normalize_task_id(task_id)}.json"++ def result_path(self, task_id: str) -> Path:+ return self.results_dir / f"{normalize_task_id(task_id)}.glb"++ @staticmethod+ def _atomic_write(path: Path, data: bytes) -> None:+ temporary = path.with_suffix(path.suffix + ".tmp")+ with temporary.open("wb") as handle:+ handle.write(data)+ handle.flush()+ os.fsync(handle.fileno())+ temporary.replace(path)++ def write_task(self, task: dict) -> Path:+ path = self.task_path(str(task["task_id"]))+ payload = json.dumps(task, separators=(",", ":"), sort_keys=True).encode("utf-8")+ self._atomic_write(path, payload)+ return path++ def read_task(self, task_id: str) -> dict:+ return json.loads(self.task_path(task_id).read_text(encoding="utf-8"))++ def write_result(self, task_id: str, glb_bytes: bytes) -> Path:+ path = self.result_path(task_id)+ self._atomic_write(path, glb_bytes)+ return pathdiff --git a/workers/model-sam3d-objects/test_app_contract.py b/workers/model-sam3d-objects/test_app_contract.pynew file mode 100644index 000000000..4740cf389--- /dev/null+++ b/workers/model-sam3d-objects/test_app_contract.py@@ -0,0 +1,93 @@+"""Smoke test for the served contract. Runs INSIDE the built image (and on any+machine with fastapi, httpx, Pillow, trimesh and google-cloud-storage):++ docker build -t model-sam3d-objects workers/model-sam3d-objects+ docker run --rm model-sam3d-objects python3 test_app_contract.py++It needs no GPU, no weights and no GCP credentials: the ASGI lifespan (which+opens the GCS client and starts the model load) is deliberately not run, so+every assertion is about request handling, not inference.++What it proves:+ 1. The whole app module imports.+ 2. The auth boundary rejects a wrong bearer on both authenticated routes.+ 3. Malformed requests are refused with 422 and a sentence, before a task exists.+ 4. The unauthenticated probes answer, and a dead pipeline says so on /health+ and /infer so the caller fails over.+ 5. The photo and mask loaders refuse everything the SSRF guard should refuse.+"""++from __future__ import annotations++import os+import sys++os.environ.setdefault("API_KEY", "smoke-test-key")+os.environ.setdefault("GCS_BUCKET", "smoke-test-bucket")++KEY = os.environ["API_KEY"]+PASSED = 0+++def check(name: str, condition: bool, detail: str = "") -> None:+ global PASSED+ if not condition:+ print(f"FAIL {name} {detail}")+ sys.exit(1)+ PASSED += 1+ print(f"ok {name}")+++def main() -> int:+ import main as app_module+ from fastapi.testclient import TestClient++ client = TestClient(app_module.app) # no `with`: lifespan stays off+ good = {"Authorization": f"Bearer {KEY}"}+ bad = {"Authorization": "Bearer wrong"}+ photo = "https://example.com/room.jpg"++ check("imports", app_module.app.title == "model-sam3d-objects")++ r = client.post("/infer", json={"image_url": photo, "boxes": [[0.1, 0.1, 0.5, 0.5]]}, headers=bad)+ check("infer rejects a wrong key", r.status_code == 401, str(r.status_code))+ r = client.get("/tasks/00000000-0000-4000-8000-000000000000", headers=bad)+ check("tasks rejects a wrong key", r.status_code == 401, str(r.status_code))++ r = client.post("/infer", json={"image_url": photo}, headers=good)+ check("infer without objects is 422", r.status_code == 422 and "name the objects" in r.text, r.text)+ r = client.post("/infer", json={"image_url": photo, "boxes": [[0.5, 0.5, 0.4, 0.4]]}, headers=good)+ check("infer with an inverted box is 422", r.status_code == 422 and "boxes[0]" in r.text, r.text)+ r = client.post(+ "/infer",+ json={"image_url": photo, "boxes": [[0.1, 0.1, 0.5, 0.5]], "mask_urls": ["https://example.com/m.png"]},+ headers=good,+ )+ check("infer with masks and boxes is 422", r.status_code == 422, r.text)++ r = client.get("/")+ check("descriptor answers", r.status_code == 200 and r.json()["model"] == "sam3d-objects")+ r = client.get("/health")+ check("health answers while loading", r.status_code == 200 and r.json()["ok"] is True)++ for url in ("http://169.254.169.254/latest/meta-data", "https://localhost/x.png", "file:///etc/passwd"):+ try:+ app_module._load_photo(url)+ except app_module.ImageSourceError:+ check(f"photo loader refuses {url}", True)+ else:+ check(f"photo loader refuses {url}", False)++ app_module._load_error = "weights missing"+ r = client.get("/health")+ check("dead pipeline drains on /health", r.status_code == 503 and r.json()["ok"] is False, r.text)+ r = client.post("/infer", json={"image_url": photo, "boxes": [[0.1, 0.1, 0.5, 0.5]]}, headers=good)+ check("dead pipeline fails /infer fast", r.status_code == 503, str(r.status_code))+ app_module._load_error = None++ print(f"\n{PASSED} checks passed")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/workers/model-sam3d-objects/test_request_policy.py b/workers/model-sam3d-objects/test_request_policy.pynew file mode 100644index 000000000..252179dc5--- /dev/null+++ b/workers/model-sam3d-objects/test_request_policy.py@@ -0,0 +1,84 @@+import pytest++from request_policy import MAX_OBJECTS, RequestRejected, validate_request++PHOTO = "https://example.com/room.jpg"+BOX = [0.1, 0.2, 0.5, 0.8]+++def test_boxes_request_is_normalized():+ out = validate_request({"image_url": PHOTO, "boxes": [BOX, [0.5, 0.5, 0.9, 0.9]]})+ assert out["image_url"] == PHOTO+ assert out["mask_urls"] == []+ assert out["boxes"][0] == (0.1, 0.2, 0.5, 0.8)+ assert out["max_objects"] == 8+ assert out["seed"] is None+++def test_masks_request():+ out = validate_request({"image_url": PHOTO, "mask_urls": ["https://example.com/a.png"], "seed": 7})+ assert out["mask_urls"] == ["https://example.com/a.png"]+ assert out["boxes"] == []+ assert out["seed"] == 7++[email protected]("payload", [None, [], "x", 3])+def test_body_must_be_an_object(payload):+ with pytest.raises(RequestRejected):+ validate_request(payload)+++def test_photo_required():+ with pytest.raises(RequestRejected, match="image_url"):+ validate_request({"boxes": [BOX]})+++def test_must_name_objects():+ with pytest.raises(RequestRejected, match="name the objects"):+ validate_request({"image_url": PHOTO})+ with pytest.raises(RequestRejected, match="name the objects"):+ validate_request({"image_url": PHOTO, "boxes": [], "mask_urls": []})+++def test_masks_and_boxes_are_exclusive():+ with pytest.raises(RequestRejected, match="not both"):+ validate_request({"image_url": PHOTO, "boxes": [BOX], "mask_urls": ["https://example.com/a.png"]})+++ "box",+ [+ [0.5, 0.2, 0.4, 0.8], # x1 < x0+ [-0.1, 0.2, 0.5, 0.8], # outside the frame+ [0.1, 0.2, 0.5, 1.2],+ [0.1, 0.2, 0.12, 0.8], # sliver+ [0.1, 0.2, 0.5], # wrong arity+ [0.1, 0.2, "a", 0.8],+ [0.1, 0.2, float("nan"), 0.8],+ ],+)+def test_bad_boxes_are_rejected(box):+ with pytest.raises(RequestRejected, match=r"boxes\[0\]"):+ validate_request({"image_url": PHOTO, "boxes": [box]})+++def test_object_cap():+ with pytest.raises(RequestRejected, match="max_objects"):+ validate_request({"image_url": PHOTO, "boxes": [BOX] * 3, "max_objects": 2})+ with pytest.raises(RequestRejected, match="max_objects"):+ validate_request({"image_url": PHOTO, "boxes": [BOX], "max_objects": MAX_OBJECTS + 1})+ with pytest.raises(RequestRejected, match="max_objects"):+ validate_request({"image_url": PHOTO, "boxes": [BOX], "max_objects": True})++[email protected]("seed", [-1, 1.5, True, "3"])+def test_bad_seed(seed):+ with pytest.raises(RequestRejected, match="seed"):+ validate_request({"image_url": PHOTO, "boxes": [BOX], "seed": seed})+++def test_url_limits():+ with pytest.raises(RequestRejected, match="image_url"):+ validate_request({"image_url": "x" * 5000, "boxes": [BOX]})+ with pytest.raises(RequestRejected, match=r"mask_urls\[0\]"):+ validate_request({"image_url": PHOTO, "mask_urls": [""]})diff --git a/workers/model-sam3d-objects/test_scene_layout.py b/workers/model-sam3d-objects/test_scene_layout.pynew file mode 100644index 000000000..76b8ede28--- /dev/null+++ b/workers/model-sam3d-objects/test_scene_layout.py@@ -0,0 +1,126 @@+import io++import numpy as np+import pytest+import trimesh+from PIL import Image++import scene_layout as sl+++def test_identity_quaternion_is_identity():+ assert np.allclose(sl.quat_wxyz_to_matrix([1, 0, 0, 0]), np.eye(3))+++def test_quaternion_is_normalized_and_rotates_correctly():+ # 90 degrees about +Z, real part first, deliberately not unit length+ q = [2 * np.cos(np.pi / 4), 0, 0, 2 * np.sin(np.pi / 4)]+ m = sl.quat_wxyz_to_matrix(q)+ assert np.allclose(m @ [1, 0, 0], [0, 1, 0], atol=1e-9)+ assert np.allclose(m @ m.T, np.eye(3), atol=1e-9)+++def test_zero_quaternion_rejected():+ with pytest.raises(ValueError):+ sl.quat_wxyz_to_matrix([0, 0, 0, 0])+++def test_decoder_axis_change_round_trips():+ v = np.random.default_rng(0).normal(size=(50, 3))+ glb = v @ sl._DECODER_TO_GLB+ assert np.allclose(sl.glb_to_decoder_vertices(glb), v)+++def test_pose_puts_object_in_front_of_the_camera():+ # An object centred in the decoder frame, pushed 3 units away from the camera+ # (+Z in the model's camera frame) must end up at -Z in a glTF viewer.+ box = trimesh.creation.box(extents=(1, 1, 1))+ glb_vertices = np.asarray(box.vertices) @ sl._DECODER_TO_GLB+ posed = sl.pose_vertices(glb_vertices, [1, 0, 0, 0], [0, 0, 3], 1.0)+ centre = posed.mean(axis=0)+ assert np.allclose(centre, [0, 0, -3], atol=1e-9)+++def test_pose_flips_left_right_into_gltf_x():+ box = trimesh.creation.box(extents=(1, 1, 1))+ glb_vertices = np.asarray(box.vertices) @ sl._DECODER_TO_GLB+ # +X in the PyTorch3D camera frame is LEFT; in glTF +X is right.+ posed = sl.pose_vertices(glb_vertices, [1, 0, 0, 0], [2, 0, 5], 1.0)+ assert posed.mean(axis=0)[0] == pytest.approx(-2.0)+++def test_pose_scale_is_uniform_or_per_axis():+ v = np.array([[1.0, 1.0, 1.0], [-1.0, -1.0, -1.0]]) @ sl._DECODER_TO_GLB+ a = sl.pose_vertices(v, [1, 0, 0, 0], [0, 0, 0], 2.0)+ b = sl.pose_vertices(v, [1, 0, 0, 0], [0, 0, 0], [2.0, 2.0, 2.0])+ assert np.allclose(a, b)+ assert np.abs(a).max() == pytest.approx(2.0)+++def test_standalone_rests_on_the_floor_and_is_centred():+ box = trimesh.creation.box(extents=(2, 4, 6))+ glb_vertices = np.asarray(box.vertices) @ sl._DECODER_TO_GLB+ solo = sl.standalone_vertices(glb_vertices, 0.5)+ assert solo[:, 1].min() == pytest.approx(0.0)+ assert (solo[:, 0].min() + solo[:, 0].max()) / 2 == pytest.approx(0.0)+ assert (solo[:, 2].min() + solo[:, 2].max()) / 2 == pytest.approx(0.0)+ assert solo.max(axis=0)[1] - solo.min(axis=0)[1] == pytest.approx(3.0) # 6 * 0.5 along decoder Z = glb Y+++def test_box_to_pixels_clamps_and_keeps_area():+ assert sl.box_to_pixels((0.0, 0.0, 1.0, 1.0), 200, 100) == (0, 0, 200, 100)+ x0, y0, x1, y1 = sl.box_to_pixels((0.999, 0.999, 1.0, 1.0), 200, 100)+ assert x1 > x0 and y1 > y0 and x1 <= 200 and y1 <= 100+++def _png(array, mode):+ buf = io.BytesIO()+ Image.fromarray(array, mode).save(buf, format="PNG")+ return buf.getvalue()+++def test_mask_decodes_luminance_and_resizes_to_the_photo():+ arr = np.zeros((10, 10), np.uint8)+ arr[2:8, 2:8] = 255+ mask = sl.mask_from_png_bytes(_png(arr, "L"), 20, 20)+ assert mask.shape == (20, 20)+ assert mask[10, 10] and not mask[0, 0]+++def test_mask_prefers_alpha_when_it_varies():+ rgba = np.zeros((8, 8, 4), np.uint8)+ rgba[..., :3] = 255 # all-white colour, so luminance alone would select everything+ rgba[2:4, 2:4, 3] = 255+ mask = sl.mask_from_png_bytes(_png(rgba, "RGBA"), 8, 8)+ assert mask.sum() == 4+++def test_empty_and_corrupt_masks_are_rejected():+ with pytest.raises(ValueError, match="empty"):+ sl.mask_from_png_bytes(_png(np.zeros((4, 4), np.uint8), "L"), 4, 4)+ with pytest.raises(ValueError, match="readable"):+ sl.mask_from_png_bytes(b"not an image", 4, 4)+++def test_scene_has_one_named_node_per_object_and_round_trips():+ items = []+ for i in range(3):+ box = trimesh.creation.box(extents=(1, 1, 1))+ items.append({"vertices": box.vertices + [i * 2, 0, -3], "faces": box.faces,+ "vertex_colors": np.tile([200, 40, 40, 255], (len(box.vertices), 1)).astype(np.uint8)})+ glb = sl.export_glb(sl.build_scene(items))+ loaded = trimesh.load(io.BytesIO(glb), file_type="glb")+ assert set(loaded.graph.nodes_geometry) == {"object_01", "object_02", "object_03"}+ assert len(loaded.geometry) == 3+++def test_manifest_entry_reports_position_and_size():+ box = trimesh.creation.box(extents=(1, 2, 3))+ posed = box.vertices + [4, 0, -5]+ solo = sl.standalone_vertices(np.asarray(box.vertices), 1.0)+ entry = sl.manifest_entry(1, posed, box.faces, solo, [1, 0, 0, 0], [0, 0, 5], [1, 1, 1], "box")+ assert entry["id"] == "object_02"+ assert entry["source"] == "box"+ assert entry["centroid"] == pytest.approx([4, 0, -5])+ assert entry["face_count"] == len(box.faces)+ assert entry["pose"]["rotation_wxyz"] == [1.0, 0.0, 0.0, 0.0]diff --git a/workers/model-sam3d-objects/worker_security.py b/workers/model-sam3d-objects/worker_security.pynew file mode 100644index 000000000..802e1c32d--- /dev/null+++ b/workers/model-sam3d-objects/worker_security.py@@ -0,0 +1,320 @@+"""+Shared security helpers for the three.ws GPU/CPU workers.++This module is intentionally dependency-light (stdlib only) so it can be+COPYed verbatim into every worker's Docker build context. The same canonical+file lives in each worker directory; keep them byte-identical when editing.++It provides three primitives that every worker needs:++ 1. ``require_api_key`` - constant-time bearer-token check (timing-safe).+ 2. ``fetch_remote_bytes`` - SSRF-hardened HTTP(S) fetch of caller-supplied+ URLs: https-only, DNS resolution with private/loopback/link-local/metadata+ IP rejection, per-hop redirect re-validation, and a bounded response size.+ 3. ``safe_error`` - turns an exception into an opaque, correlation-id-tagged+ message safe to persist/return to callers, after logging the full+ traceback server-side only.+"""++from __future__ import annotations++import hmac+import ipaddress+import logging+import asyncio+import socket+import time+import uuid+from typing import Iterable, Optional+from urllib.parse import urlsplit++import httpx++log = logging.getLogger("worker_security")++# ── auth ────────────────────────────────────────────────────────────────────────+++def require_api_key(authorization: Optional[str], api_key: str) -> None:+ """Validate a ``Bearer <token>`` header in constant time.++ Raises ``PermissionError`` on any failure so callers can translate it into+ the framework's 401 (FastAPI workers map this to HTTPException(401)).+ """+ # Fail closed on an empty configured key: with api_key == "" a bare+ # "Bearer " header would otherwise strip to "" and compare equal.+ if not api_key:+ raise PermissionError("worker api key not configured")+ if not authorization or not authorization.startswith("Bearer "):+ raise PermissionError("missing bearer token")+ token = authorization[len("Bearer "):].strip()+ if not token:+ raise PermissionError("missing bearer token")+ # Compare bytes: hmac.compare_digest is timing-safe, and on str operands it+ # raises TypeError for non-ASCII input, which would surface as a 500.+ if not hmac.compare_digest(token.encode("utf-8"), api_key.encode("utf-8")):+ raise PermissionError("invalid api key")+++# ── SSRF-hardened fetch ─────────────────────────────────────────────────────────++# Default cap on a fetched response. Avatar selfies / driving audio / raw meshes+# are comfortably under this; anything larger is almost certainly abuse.+DEFAULT_MAX_BYTES = 64 * 1024 * 1024 # 64 MiB+_MAX_REDIRECTS = 4++# Transient fetch failures are retried: a dropped connection, a timeout, or a+# 429/5xx from the image host. The public R2 dev endpoint that serves every+# forge reference image drops connections under load ("Server disconnected+# without sending a response"), and before this retry each drop ended a+# generation as an opaque internal error on TripoSG and Hunyuan3D (about 20 a+# week measured 2026-10-09). Refusals (UnsafeUrlError), oversize bodies, and+# other 4xx answers are final and never retried.+_FETCH_ATTEMPTS = 3+_FETCH_BACKOFF_SECS = (0.5, 1.5)++# Hosts that enforce a User-Agent policy refuse httpx's default+# "python-httpx/<version>": upload.wikimedia.org answers it with 403 Forbidden+# and serves the same image to a descriptive agent (measured 2026-10-09, when+# model-triposg failed every job whose reference photo was a Wikimedia link).+# Callers may still pass their own User-Agent; it replaces this one.+DEFAULT_USER_AGENT = "three.ws-worker/1.0 (+https://three.ws)"+++def _fetch_headers(headers: Optional[dict]) -> dict:+ """Caller headers over the default User-Agent, matched case-insensitively."""+ merged = dict(headers or {})+ if not any(str(k).lower() == "user-agent" for k in merged):+ merged["User-Agent"] = DEFAULT_USER_AGENT+ return merged+++class UnsafeUrlError(ValueError):+ """Raised when a URL targets a disallowed scheme or a private/internal IP."""+++def _is_blocked_ip(ip: ipaddress._BaseAddress) -> bool:+ """Reject any address that could reach internal infrastructure.++ Covers loopback, RFC1918 private ranges, link-local (incl. the cloud+ metadata server at 169.254.169.254), unique-local IPv6 (fc00::/7),+ multicast, reserved, unspecified, and any other non-global address (CGNAT+ 100.64.0.0/10 included). IPv4-mapped IPv6 addresses+ are unwrapped first so a mapped private v4 can't slip through.+ """+ if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:+ ip = ip.ipv4_mapped+ return (+ ip.is_private+ or ip.is_loopback+ or ip.is_link_local+ or ip.is_multicast+ or ip.is_reserved+ or ip.is_unspecified+ # Anything not globally routable, which also covers the CGNAT shared+ # space 100.64.0.0/10 that is_private does not flag.+ or not ip.is_global+ )+++def _resolve_safe_host(host: str) -> None:+ """DNS-resolve ``host`` and raise UnsafeUrlError if ANY address is internal.++ Resolving and checking every returned address closes the gap where a name+ resolves to both a public and a private address.+ """+ if not host:+ raise UnsafeUrlError("missing host")+ try:+ infos = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)+ except socket.gaierror as exc:+ raise UnsafeUrlError(f"could not resolve host: {host}") from exc++ addrs = {info[4][0] for info in infos}+ if not addrs:+ raise UnsafeUrlError(f"no addresses for host: {host}")+ for addr in addrs:+ ip = ipaddress.ip_address(addr)+ if _is_blocked_ip(ip):+ raise UnsafeUrlError(f"host resolves to a disallowed address: {host}")+++def assert_safe_url(url: str, *, allow_http: bool = False) -> str:+ """Validate scheme + resolved IPs for a single URL. Returns it unchanged.++ https is required by default; ``allow_http`` exists only for callers that+ legitimately need cleartext (not used by the workers).+ """+ parts = urlsplit(url)+ scheme = parts.scheme.lower()+ allowed = ("https", "http") if allow_http else ("https",)+ if scheme not in allowed:+ raise UnsafeUrlError(f"scheme not allowed: {scheme or '(none)'}")+ if not parts.hostname:+ raise UnsafeUrlError("missing host")+ _resolve_safe_host(parts.hostname)+ return url+++def fetch_remote_bytes(+ url: str,+ *,+ timeout: float = 30.0,+ max_bytes: int = DEFAULT_MAX_BYTES,+ allow_http: bool = False,+ headers: Optional[dict] = None,+) -> bytes:+ """Fetch ``url`` safely and return its body bytes.++ Hardening applied:+ * https-only (unless ``allow_http``).+ * every URL - including each redirect target - is DNS-resolved and+ rejected if it maps to a private/loopback/link-local/metadata address.+ * redirects are followed manually (max ``_MAX_REDIRECTS``) so each hop is+ re-validated; httpx's own redirect following is disabled.+ * the response body is streamed and aborted once it exceeds ``max_bytes``.+ * transient failures (dropped connection, timeout, 429/5xx) are retried+ with a short backoff; see ``_FETCH_ATTEMPTS``.++ Raises ``UnsafeUrlError`` for disallowed targets and ``httpx.HTTPError`` /+ ``ValueError`` for transport or size failures.+ """+ for attempt in range(_FETCH_ATTEMPTS - 1):+ try:+ return _fetch_once(+ url, timeout=timeout, max_bytes=max_bytes,+ allow_http=allow_http, headers=headers,+ )+ except httpx.HTTPError as exc:+ if not _is_transient(exc):+ raise+ log.warning("transient fetch failure (attempt %d): %s", attempt + 1, type(exc).__name__)+ time.sleep(_FETCH_BACKOFF_SECS[attempt])+ return _fetch_once(+ url, timeout=timeout, max_bytes=max_bytes,+ allow_http=allow_http, headers=headers,+ )+++def _is_transient(exc: httpx.HTTPError) -> bool:+ if isinstance(exc, httpx.HTTPStatusError):+ code = exc.response.status_code+ return code == 429 or code >= 500+ return isinstance(exc, httpx.TransportError)+++def _fetch_once(+ url: str,+ *,+ timeout: float,+ max_bytes: int,+ allow_http: bool,+ headers: Optional[dict],+) -> bytes:+ current = assert_safe_url(url, allow_http=allow_http)+ with httpx.Client(follow_redirects=False, timeout=timeout) as client:+ for _ in range(_MAX_REDIRECTS + 1):+ with client.stream("GET", current, headers=_fetch_headers(headers)) as resp:+ if resp.is_redirect:+ location = resp.headers.get("location")+ if not location:+ raise UnsafeUrlError("redirect without Location header")+ current = assert_safe_url(+ str(resp.url.join(location)), allow_http=allow_http+ )+ continue+ resp.raise_for_status()+ declared = resp.headers.get("content-length")+ if declared is not None and declared.isdigit() and int(declared) > max_bytes:+ raise ValueError(f"response too large: {declared} bytes")+ chunks: list[bytes] = []+ total = 0+ for chunk in resp.iter_bytes(65536):+ total += len(chunk)+ if total > max_bytes:+ raise ValueError(f"response exceeded {max_bytes} bytes")+ chunks.append(chunk)+ return b"".join(chunks)+ raise UnsafeUrlError(f"too many redirects (>{_MAX_REDIRECTS})")+++async def fetch_remote_bytes_async(+ client: "httpx.AsyncClient",+ url: str,+ *,+ max_bytes: int = DEFAULT_MAX_BYTES,+ allow_http: bool = False,+ headers: Optional[dict] = None,+) -> bytes:+ """Async variant of ``fetch_remote_bytes`` using a caller-provided client.++ The provided ``client`` MUST be constructed with ``follow_redirects=False``+ so this function can re-validate each redirect hop itself. Transient+ failures are retried exactly as in the sync variant.+ """+ for attempt in range(_FETCH_ATTEMPTS - 1):+ try:+ return await _fetch_once_async(+ client, url, max_bytes=max_bytes,+ allow_http=allow_http, headers=headers,+ )+ except httpx.HTTPError as exc:+ if not _is_transient(exc):+ raise+ log.warning("transient fetch failure (attempt %d): %s", attempt + 1, type(exc).__name__)+ await asyncio.sleep(_FETCH_BACKOFF_SECS[attempt])+ return await _fetch_once_async(+ client, url, max_bytes=max_bytes,+ allow_http=allow_http, headers=headers,+ )+++async def _fetch_once_async(+ client: "httpx.AsyncClient",+ url: str,+ *,+ max_bytes: int,+ allow_http: bool,+ headers: Optional[dict],+) -> bytes:+ current = assert_safe_url(url, allow_http=allow_http)+ for _ in range(_MAX_REDIRECTS + 1):+ async with client.stream("GET", current, headers=_fetch_headers(headers)) as resp:+ if resp.is_redirect:+ location = resp.headers.get("location")+ if not location:+ raise UnsafeUrlError("redirect without Location header")+ current = assert_safe_url(+ str(resp.url.join(location)), allow_http=allow_http+ )+ continue+ resp.raise_for_status()+ declared = resp.headers.get("content-length")+ if declared is not None and declared.isdigit() and int(declared) > max_bytes:+ raise ValueError(f"response too large: {declared} bytes")+ chunks: list[bytes] = []+ total = 0+ async for chunk in resp.aiter_bytes(65536):+ total += len(chunk)+ if total > max_bytes:+ raise ValueError(f"response exceeded {max_bytes} bytes")+ chunks.append(chunk)+ return b"".join(chunks)+ raise UnsafeUrlError(f"too many redirects (>{_MAX_REDIRECTS})")+++# ── opaque error handling ───────────────────────────────────────────────────────+++def safe_error(exc: BaseException, *, context: str = "") -> str:+ """Log the full traceback server-side and return an opaque client message.++ Never leak ``traceback.format_exc()`` (absolute paths, library versions,+ internal hostnames, fragments of upstream responses) to callers. A short+ correlation id is embedded in both the log line and the returned string so+ operators can join a client-reported failure to the server log.+ """+ correlation_id = uuid.uuid4().hex[:12]+ label = f"{context} " if context else ""+ log.exception("%serror [correlation_id=%s]", label, correlation_id)+ return f"internal error (ref {correlation_id})"