Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(trellis): retry a failed model load and refuse work once it is terminal

nirholas84bc3bb38954
diff --git a/workers/model-trellis/README.md b/workers/model-trellis/README.md
index f60254093..01ea1e1d7 100644
--- a/workers/model-trellis/README.md
+++ b/workers/model-trellis/README.md
@@ -140,6 +140,7 @@ forever behind a dead runner. Unknown ids return `404`.
"pipeline_loaded": true,
"ready": true,
"load_error": null,
+ "load_attempts": 1,
"tiers": ["draft", "standard", "high", "max"],
"default_quality": { "ss_steps": 40, "slat_steps": 40, "ss_cfg": 7.5, "slat_cfg": 3.0, "simplify": 0.75, "texture_size": 4096 },
"rembg_matte": true
@@ -150,6 +151,23 @@ The ~3 GB pipeline loads in the background after the port opens, so a cold
instance reports `ready: false` briefly; `/infer` tasks submitted during that
window wait for `ready` (up to 600 s) rather than failing.
+**A dead instance says so.** The load is retried (`MODEL_LOAD_ATTEMPTS`) before
+its error is treated as terminal, because the failures that reach it are
+transient upstream ones rather than code faults. While retries remain,
+`load_error` stays `null` and jobs keep waiting on `ready`. Once the budget is
+spent, `load_error` latches and this endpoint answers **`503` with
+`"ok": false`**, `/infer` refuses new work with `503`, and the Cloud Run
+liveness probe recycles the container.
+
+That behaviour is the fix for the 2026-09-02 outage: a single transient
+`HTTP Error 403: rate limit exceeded` during the load latched forever, `/health`
+went on answering `ok: true`, `/infer` went on accepting jobs it could never
+run, and `min-instances=1` kept the dead instance resident. The default free
+image lane failed 70 of 219 generations over 12 hours while every status surface
+read green. Callers rely on the `503` from `/infer`: it is what trips the lane
+failover in `api/forge.js`, which reroutes the request to another backend
+instead of letting it die at poll time.
+
### `GET /`
Unauthenticated service descriptor. It exists so a root ping stops recording a
@@ -181,6 +199,11 @@ against two real error events, which is how a genuine failure goes unnoticed.
| `WEIGHTS_GCS_URI` | no | - | `gs://` weight tree staged to local disk at startup with the storage client. Set in production; see the note below |
| `WEIGHTS_LOCAL_DIR` | no | `/tmp/trellis-weights` | Where that staging lands, and what the pipeline then loads from |
| `IMAGE_FETCH_TIMEOUT_S` | no | `30` | Per-attempt timeout for fetching a caller-supplied `https://` image (3 attempts) |
+| `MODEL_LOAD_ATTEMPTS` | no | `4` | Model-load attempts before `load_error` latches and the instance reports itself dead |
+| `MODEL_LOAD_RETRY_BASE_S` | no | `15` | First load-retry backoff, doubling per attempt |
+| `MODEL_LOAD_RETRY_CAP_S` | no | `120` | Ceiling on that backoff |
+| `DINOV2_LOCAL_DIR` | no | `/opt/dinov2` | Image-baked dinov2 checkout used for TRELLIS's image conditioner, so the load never calls GitHub |
+| `TORCH_HOME` | no | `/opt/torch` | Set in the image; holds the pre-baked `dinov2_vitl14_reg` checkpoint so `torch.hub` finds it cached |
Weights are **not** baked into the image — the `three-ws-model-weights` bucket
is mounted at `/weights`, so refreshing weights needs no rebuild.
diff --git a/workers/model-trellis/cloudbuild.yaml b/workers/model-trellis/cloudbuild.yaml
index df0167d8d..af4b5d92f 100644
--- a/workers/model-trellis/cloudbuild.yaml
+++ b/workers/model-trellis/cloudbuild.yaml
@@ -68,6 +68,17 @@ steps:
- --add-volume-mount=volume=weights,mount-path=/weights
- --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},WEIGHTS_DIR=/weights/trellis-large,MAX_CONCURRENT=1,WEIGHTS_GCS_URI=gs://${_WEIGHTS_BUCKET}/${_WEIGHTS_BUCKET_PREFIX},WEIGHTS_LOCAL_DIR=/tmp/trellis-weights,REMBG_SERVICE_URL=${_REMBG_SERVICE_URL}
- --set-secrets=API_KEY=avatar-reconstruction-key:latest
+ # Recycle an instance whose model load is permanently dead. The startup
+ # probe is a TCP check, and the port opens immediately because the ~3 GB
+ # load is backgrounded, so nothing here ever noticed a load that failed:
+ # with min-instances=1 the failed instance stayed resident and answered
+ # every job with "pipeline unavailable" until somebody redeployed. That is
+ # the 2026-09-02 outage. main.py answers /health with 503 only once the
+ # whole retry budget is spent, so an ordinary cold load (200 while
+ # loading) is never killed mid-load, and a genuinely dead container is
+ # replaced within ~3 minutes. Inference runs in a thread executor, so
+ # /health stays responsive while the GPU is busy.
+ - --liveness-probe=httpGet.path=/health,initialDelaySeconds=60,periodSeconds=60,timeoutSeconds=10,failureThreshold=3
images:
- ${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPO}/server:$SHORT_SHA
diff --git a/workers/model-trellis/main.py b/workers/model-trellis/main.py
index ba583fdbf..b96c012ec 100644
--- a/workers/model-trellis/main.py
+++ b/workers/model-trellis/main.py
@@ -21,7 +21,12 @@ API contract (consumed by the Pipeline Controller):
GET /tasks/:id → { task_id, status, result_gcs_url?, tier?, matted_views?,
quality?, error? }
- GET /health → { ok, model, gpu_available, tiers, rembg_matte }
+ GET /health → { ok, model, gpu_available, tiers, rembg_matte, load_error,
+ load_attempts }. Answers 503 with ok:false once the model
+ load has spent its retry budget, so a permanently dead
+ instance is drained by the platform health report and
+ recycled by the Cloud Run liveness probe instead of
+ staying resident and failing every job it accepts.
GET / → { service, model, ready, endpoints }. Unauthenticated
service descriptor for the platform's warmth probe.
@@ -42,6 +47,11 @@ Environment variables (README.md carries the full table):
REMBG_MODEL default rembg model (default: isnet-general-use)
REMBG_TIMEOUT_S matte round-trip budget (default: 90)
IMAGE_FETCH_TIMEOUT_S per-attempt image fetch timeout (default: 30)
+ MODEL_LOAD_ATTEMPTS model-load attempts before the error latches (default: 4)
+ MODEL_LOAD_RETRY_BASE_S first load-retry backoff, doubling (default: 15)
+ MODEL_LOAD_RETRY_CAP_S ceiling on that backoff (default: 120)
+ DINOV2_LOCAL_DIR baked dinov2 checkout for the image conditioner
+ (default: /opt/dinov2), so the load never calls GitHub
"""
from __future__ import annotations
diff --git a/workers/model-trellis/test_app_contract.py b/workers/model-trellis/test_app_contract.py
index 71cf35f39..5ccaf1b06 100644
--- a/workers/model-trellis/test_app_contract.py
+++ b/workers/model-trellis/test_app_contract.py
@@ -110,6 +110,80 @@ def main() -> int:
check("root names the service", body.get("service") == "model-trellis", str(body)[:120])
check("root lists the working endpoints", "POST /infer" in body.get("endpoints", []), str(body)[:160])
+ # 4b. A dead instance must SAY it is dead, on both surfaces.
+ #
+ # This is the regression that took the default free image lane down for
+ # 12 hours on 2026-09-02. A transient rate limit failed the model load;
+ # the error latched in memory; /health went on answering ok:true, so the
+ # platform health report and Cloud Run both still called the instance
+ # healthy; and /infer went on accepting jobs it could never run, which
+ # bypassed the caller's lane failover entirely and made every one of
+ # those 70 generations terminal. Both surfaces are asserted here, in the
+ # build, because neither failure is visible from outside until real user
+ # traffic is already being dropped.
+ app_module._load_error = "internal error (ref deadbeefcafe)"
+ try:
+ res = client.get("/health")
+ body = res.json()
+ check("a dead pipeline makes /health answer 503", res.status_code == 503, f"got {res.status_code}")
+ check("a dead pipeline makes /health report ok:false", body.get("ok") is False, str(body)[:160])
+ check("health surfaces the load error", "deadbeefcafe" in str(body.get("load_error")), str(body)[:160])
+ res = client.post("/infer", headers={"Authorization": f"Bearer {KEY}"}, json={"images": [png_data_uri()]})
+ check(
+ "a dead pipeline makes /infer refuse with 503 so the caller fails over",
+ res.status_code == 503,
+ f"got {res.status_code}",
+ )
+ finally:
+ app_module._load_error = None
+ res = client.get("/health")
+ check("a live pipeline still answers /health 200", res.status_code == 200, f"got {res.status_code}")
+ check("a live pipeline reports ok:true", res.json().get("ok") is True, str(res.json())[:160])
+
+ # 4c. The image conditioner resolves locally, so the model load never depends
+ # on GitHub's rate limit for an anonymous shared Cloud Run egress IP.
+ # torch.hub.load() is given no branch by TRELLIS, so it calls github.com
+ # on every load and re-raises any non-404 before consulting its cache;
+ # that is the exact call that 403'd and started the outage above.
+ check(
+ "the dinov2 checkout is baked into the image",
+ os.path.isfile(os.path.join(app_module.DINOV2_LOCAL_DIR, "hubconf.py")),
+ f"missing hubconf.py under {app_module.DINOV2_LOCAL_DIR}",
+ )
+ import torch
+
+ original_hub_load = torch.hub.load
+ try:
+ # Install the pin over a recorder, so the assertion is about what the
+ # wrapper forwards rather than about building a ViT on a GPU-less builder.
+ forwarded = {}
+
+ def _record(repo_or_dir, model, *args, **kwargs):
+ forwarded["repo"] = repo_or_dir
+ forwarded["source"] = kwargs.get("source")
+ return "recorded"
+
+ app_module._dinov2_pinned = False
+ torch.hub.load = _record
+ check("the dinov2 pin installs", app_module._pin_dinov2_to_local_checkout() is True)
+ torch.hub.load("facebookresearch/dinov2", "dinov2_vitl14_reg", pretrained=True)
+ check(
+ "torch.hub resolves dinov2 from the local checkout, never over the network",
+ forwarded.get("repo") == app_module.DINOV2_LOCAL_DIR and forwarded.get("source") == "local",
+ str(forwarded),
+ )
+ # Everything else must still reach the real loader untouched.
+ forwarded.clear()
+ torch.hub.load("pytorch/vision", "resnet18")
+ check(
+ "an unrelated torch.hub repo is left alone",
+ forwarded.get("repo") == "pytorch/vision" and forwarded.get("source") is None,
+ str(forwarded),
+ )
+ finally:
+ torch.hub.load = original_hub_load
+ app_module._dinov2_pinned = False
+
# 5. Request validation, before anything reaches the GPU.
good = {"Authorization": f"Bearer {KEY}"}
res = client.post("/infer", headers=good, json={"images": []})