Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(trellis): retry a failed model load and refuse work once it is terminal
nirholas84bc3bb38954
diff --git a/workers/model-trellis/README.md b/workers/model-trellis/README.mdindex f60254093..01ea1e1d7 100644--- a/workers/model-trellis/README.md+++ b/workers/model-trellis/README.md@@ -140,6 +140,7 @@ forever behind a dead runner. Unknown ids return `404`."pipeline_loaded": true,"ready": true,"load_error": null,+ "load_attempts": 1,"tiers": ["draft", "standard", "high", "max"],"default_quality": { "ss_steps": 40, "slat_steps": 40, "ss_cfg": 7.5, "slat_cfg": 3.0, "simplify": 0.75, "texture_size": 4096 },"rembg_matte": true@@ -150,6 +151,23 @@ The ~3 GB pipeline loads in the background after the port opens, so a coldinstance reports `ready: false` briefly; `/infer` tasks submitted during thatwindow wait for `ready` (up to 600 s) rather than failing.+**A dead instance says so.** The load is retried (`MODEL_LOAD_ATTEMPTS`) before+its error is treated as terminal, because the failures that reach it are+transient upstream ones rather than code faults. While retries remain,+`load_error` stays `null` and jobs keep waiting on `ready`. Once the budget is+spent, `load_error` latches and this endpoint answers **`503` with+`"ok": false`**, `/infer` refuses new work with `503`, and the Cloud Run+liveness probe recycles the container.++That behaviour is the fix for the 2026-09-02 outage: a single transient+`HTTP Error 403: rate limit exceeded` during the load latched forever, `/health`+went on answering `ok: true`, `/infer` went on accepting jobs it could never+run, and `min-instances=1` kept the dead instance resident. The default free+image lane failed 70 of 219 generations over 12 hours while every status surface+read green. Callers rely on the `503` from `/infer`: it is what trips the lane+failover in `api/forge.js`, which reroutes the request to another backend+instead of letting it die at poll time.+### `GET /`Unauthenticated service descriptor. It exists so a root ping stops recording a@@ -181,6 +199,11 @@ against two real error events, which is how a genuine failure goes unnoticed.| `WEIGHTS_GCS_URI` | no | - | `gs://` weight tree staged to local disk at startup with the storage client. Set in production; see the note below || `WEIGHTS_LOCAL_DIR` | no | `/tmp/trellis-weights` | Where that staging lands, and what the pipeline then loads from || `IMAGE_FETCH_TIMEOUT_S` | no | `30` | Per-attempt timeout for fetching a caller-supplied `https://` image (3 attempts) |+| `MODEL_LOAD_ATTEMPTS` | no | `4` | Model-load attempts before `load_error` latches and the instance reports itself dead |+| `MODEL_LOAD_RETRY_BASE_S` | no | `15` | First load-retry backoff, doubling per attempt |+| `MODEL_LOAD_RETRY_CAP_S` | no | `120` | Ceiling on that backoff |+| `DINOV2_LOCAL_DIR` | no | `/opt/dinov2` | Image-baked dinov2 checkout used for TRELLIS's image conditioner, so the load never calls GitHub |+| `TORCH_HOME` | no | `/opt/torch` | Set in the image; holds the pre-baked `dinov2_vitl14_reg` checkpoint so `torch.hub` finds it cached |Weights are **not** baked into the image — the `three-ws-model-weights` bucketis mounted at `/weights`, so refreshing weights needs no rebuild.diff --git a/workers/model-trellis/cloudbuild.yaml b/workers/model-trellis/cloudbuild.yamlindex df0167d8d..af4b5d92f 100644--- a/workers/model-trellis/cloudbuild.yaml+++ b/workers/model-trellis/cloudbuild.yaml@@ -68,6 +68,17 @@ steps:- --add-volume-mount=volume=weights,mount-path=/weights- --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},WEIGHTS_DIR=/weights/trellis-large,MAX_CONCURRENT=1,WEIGHTS_GCS_URI=gs://${_WEIGHTS_BUCKET}/${_WEIGHTS_BUCKET_PREFIX},WEIGHTS_LOCAL_DIR=/tmp/trellis-weights,REMBG_SERVICE_URL=${_REMBG_SERVICE_URL}- --set-secrets=API_KEY=avatar-reconstruction-key:latest+ # Recycle an instance whose model load is permanently dead. The startup+ # probe is a TCP check, and the port opens immediately because the ~3 GB+ # load is backgrounded, so nothing here ever noticed a load that failed:+ # with min-instances=1 the failed instance stayed resident and answered+ # every job with "pipeline unavailable" until somebody redeployed. That is+ # the 2026-09-02 outage. main.py answers /health with 503 only once the+ # whole retry budget is spent, so an ordinary cold load (200 while+ # loading) is never killed mid-load, and a genuinely dead container is+ # replaced within ~3 minutes. Inference runs in a thread executor, so+ # /health stays responsive while the GPU is busy.+ - --liveness-probe=httpGet.path=/health,initialDelaySeconds=60,periodSeconds=60,timeoutSeconds=10,failureThreshold=3images:- ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHAdiff --git a/workers/model-trellis/main.py b/workers/model-trellis/main.pyindex ba583fdbf..b96c012ec 100644--- a/workers/model-trellis/main.py+++ b/workers/model-trellis/main.py@@ -21,7 +21,12 @@ API contract (consumed by the Pipeline Controller):GET /tasks/:id → { task_id, status, result_gcs_url?, tier?, matted_views?,quality?, error? }- GET /health → { ok, model, gpu_available, tiers, rembg_matte }+ GET /health → { ok, model, gpu_available, tiers, rembg_matte, load_error,+ load_attempts }. Answers 503 with ok:false once the model+ load has spent its retry budget, so a permanently dead+ instance is drained by the platform health report and+ recycled by the Cloud Run liveness probe instead of+ staying resident and failing every job it accepts.GET / → { service, model, ready, endpoints }. Unauthenticatedservice descriptor for the platform's warmth probe.@@ -42,6 +47,11 @@ Environment variables (README.md carries the full table):REMBG_MODEL default rembg model (default: isnet-general-use)REMBG_TIMEOUT_S matte round-trip budget (default: 90)IMAGE_FETCH_TIMEOUT_S per-attempt image fetch timeout (default: 30)+ MODEL_LOAD_ATTEMPTS model-load attempts before the error latches (default: 4)+ MODEL_LOAD_RETRY_BASE_S first load-retry backoff, doubling (default: 15)+ MODEL_LOAD_RETRY_CAP_S ceiling on that backoff (default: 120)+ DINOV2_LOCAL_DIR baked dinov2 checkout for the image conditioner+ (default: /opt/dinov2), so the load never calls GitHub"""from __future__ import annotationsdiff --git a/workers/model-trellis/test_app_contract.py b/workers/model-trellis/test_app_contract.pyindex 71cf35f39..5ccaf1b06 100644--- a/workers/model-trellis/test_app_contract.py+++ b/workers/model-trellis/test_app_contract.py@@ -110,6 +110,80 @@ def main() -> int:check("root names the service", body.get("service") == "model-trellis", str(body)[:120])check("root lists the working endpoints", "POST /infer" in body.get("endpoints", []), str(body)[:160])+ # 4b. A dead instance must SAY it is dead, on both surfaces.+ #+ # This is the regression that took the default free image lane down for+ # 12 hours on 2026-09-02. A transient rate limit failed the model load;+ # the error latched in memory; /health went on answering ok:true, so the+ # platform health report and Cloud Run both still called the instance+ # healthy; and /infer went on accepting jobs it could never run, which+ # bypassed the caller's lane failover entirely and made every one of+ # those 70 generations terminal. Both surfaces are asserted here, in the+ # build, because neither failure is visible from outside until real user+ # traffic is already being dropped.+ app_module._load_error = "internal error (ref deadbeefcafe)"+ try:+ res = client.get("/health")+ body = res.json()+ check("a dead pipeline makes /health answer 503", res.status_code == 503, f"got {res.status_code}")+ check("a dead pipeline makes /health report ok:false", body.get("ok") is False, str(body)[:160])+ check("health surfaces the load error", "deadbeefcafe" in str(body.get("load_error")), str(body)[:160])+ res = client.post("/infer", headers={"Authorization": f"Bearer {KEY}"}, json={"images": [png_data_uri()]})+ check(+ "a dead pipeline makes /infer refuse with 503 so the caller fails over",+ res.status_code == 503,+ f"got {res.status_code}",+ )+ finally:+ app_module._load_error = None+ res = client.get("/health")+ check("a live pipeline still answers /health 200", res.status_code == 200, f"got {res.status_code}")+ check("a live pipeline reports ok:true", res.json().get("ok") is True, str(res.json())[:160])++ # 4c. The image conditioner resolves locally, so the model load never depends+ # on GitHub's rate limit for an anonymous shared Cloud Run egress IP.+ # torch.hub.load() is given no branch by TRELLIS, so it calls github.com+ # on every load and re-raises any non-404 before consulting its cache;+ # that is the exact call that 403'd and started the outage above.+ check(+ "the dinov2 checkout is baked into the image",+ os.path.isfile(os.path.join(app_module.DINOV2_LOCAL_DIR, "hubconf.py")),+ f"missing hubconf.py under {app_module.DINOV2_LOCAL_DIR}",+ )+ import torch++ original_hub_load = torch.hub.load+ try:+ # Install the pin over a recorder, so the assertion is about what the+ # wrapper forwards rather than about building a ViT on a GPU-less builder.+ forwarded = {}++ def _record(repo_or_dir, model, *args, **kwargs):+ forwarded["repo"] = repo_or_dir+ forwarded["source"] = kwargs.get("source")+ return "recorded"++ app_module._dinov2_pinned = False+ torch.hub.load = _record+ check("the dinov2 pin installs", app_module._pin_dinov2_to_local_checkout() is True)+ torch.hub.load("facebookresearch/dinov2", "dinov2_vitl14_reg", pretrained=True)+ check(+ "torch.hub resolves dinov2 from the local checkout, never over the network",+ forwarded.get("repo") == app_module.DINOV2_LOCAL_DIR and forwarded.get("source") == "local",+ str(forwarded),+ )+ # Everything else must still reach the real loader untouched.+ forwarded.clear()+ torch.hub.load("pytorch/vision", "resnet18")+ check(+ "an unrelated torch.hub repo is left alone",+ forwarded.get("repo") == "pytorch/vision" and forwarded.get("source") is None,+ str(forwarded),+ )+ finally:+ torch.hub.load = original_hub_load+ app_module._dinov2_pinned = False+# 5. Request validation, before anything reaches the GPU.good = {"Authorization": f"Bearer {KEY}"}res = client.post("/infer", headers=good, json={"images": []})