Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): classify a bare SIGKILL on model-trellis as an OOM

nirholasfeab48674078
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 6b101a0ad..34ef9d3fe 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -655,6 +655,89 @@ HTTP 503 GET /health (model-* service) textPayload: The request failed becau
---
+## 🟡 `Container terminated on signal 9.` on `model-trellis` (gpu-worker-sigkill)
+
+```
+WARNING [model-trellis] The request failed because either the HTTP response was malformed or connection to the instance had an error.
+WARNING [model-trellis] Container terminated on signal 9.
+```
+
+- **Source:** an out-of-memory kill in the middle of a job. The L4 instance is
+ already at Cloud Run's 32 GiB ceiling for that GPU, and Cloud Run does not
+ print its usual "Memory limit of N MiB exceeded" line for these instances, so
+ the bare SIGKILL is the only trace (the generic `run-oom` signature never
+ fires). The 5xx just before it is whatever poll was in flight when the
+ container died; the caller treats a poll error as "still running".
+- **How it was proven (2026-10-08):** twelve kills in five days, every one on
+ the single warm instance. The revision's memory utilization climbed in steps
+ after heavy jobs and never came back (0.46, 0.56, 0.71, 0.81, 0.88, 0.91), and
+ each kill landed seconds after the job log printed
+ `After remove invisible faces`, i.e. inside the UV unwrap and texture bake.
+ Read the curve yourself:
+ ```sh
+ TOKEN=$(gcloud auth print-access-token)
+ curl -s -G -H "Authorization: Bearer $TOKEN" \
+ "https://monitoring.googleapis.com/v3/projects/aerial-vehicle-466722-p5/timeSeries" \
+ --data-urlencode 'filter=metric.type="run.googleapis.com/container/memory/utilizations" AND resource.labels.service_name="model-trellis" AND resource.labels.location="us-central1"' \
+ --data-urlencode "interval.startTime=$(date -u -d '-24 hours' +%FT%TZ)" \
+ --data-urlencode "interval.endTime=$(date -u +%FT%TZ)" \
+ --data-urlencode "aggregation.alignmentPeriod=1200s" \
+ --data-urlencode "aggregation.perSeriesAligner=ALIGN_PERCENTILE_50"
+ ```
+- **User impact before the fix:** the killed job and the one queued behind it
+ kept a durable `running` record for 30 minutes, long after the forge client
+ stopped polling, so lane failover never redispatched them
+ (`forge_creations` rows ended as `task orphaned` or stuck in `generating`).
+- **Fix (code, in `workers/model-trellis`):** `MALLOC_ARENA_MAX=2` plus
+ `malloc_trim` after every job stop glibc hoarding freed heap per executor
+ thread; the 3 GiB of staged weights in the in-memory `/tmp` are deleted once
+ the model is on the GPU; an instance that has still grown past 80% of its
+ limit drains and restarts between jobs ("recycling instance" in its log)
+ instead of dying inside one; and every live task heartbeats its record, so an
+ orphan now fails within ~3 minutes while the client is still polling and the
+ forge fails it over. See `workers/model-trellis/instance_health.py`.
+- **If it recurs after that deploy:** it is a regression or a new leak. Each job
+ now logs `memory after job: …`; find where the step happens.
+- **Monitor signature:** `gpu-worker-sigkill` in
+ [scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), scoped to
+ `model-trellis`, classified `investigate`.
+
+---
+
+## 🟡 Revision never Ready on a single-GPU service: "Startup probes timed out after 4m" with no container logs
+
+```
+Ready condition status changed to False for Revision model-hunyuan3d-21-rtx-000NN-xxx with message:
+Container failed to become healthy. Startup probes timed out after 4m (1 attempts with a period of 4m each).
+```
+
+- **Source:** not the probe budget and not the image. `model-hunyuan3d-21-rtx`
+ runs on the RTX PRO 6000 quota, which Cloud Run enforces at **one** GPU in
+ us-central1, and the serving revision holds that GPU with `min-instances=1`.
+ A rolling update needs a second GPU instance for the new revision before
+ traffic can move, so that instance never starts: its log shows only
+ `Starting new instance` once a minute, with no container output at all, until
+ the 4 minute startup window expires. Traffic stays on the old revision, so
+ users are unaffected, but the change never ships. The 2026-09-18 attempt
+ (revision 00002, adding `U2NET_HOME`) sat in this state for three weeks.
+- **Resolve (config-only, pre-approved; ~10 minutes without this lane while the
+ new instance loads, forge fails over meanwhile):** release the GPU, roll, then
+ restore autoscaling. Manual scaling is a service-level setting, so neither the
+ first nor the last step creates a revision:
+ ```sh
+ gcloud run services update model-hunyuan3d-21-rtx --region us-central1 --scaling=0
+ gcloud run services update model-hunyuan3d-21-rtx --region us-central1 --update-env-vars KEY=VALUE # or deploy the new image
+ gcloud run services update model-hunyuan3d-21-rtx --region us-central1 --scaling=auto --max=1
+ ```
+ `workers/model-hunyuan3d/cloudbuild.hunyuan21rtx.yaml` runs this same
+ sequence around its deploy, and restores autoscaling even when the deploy
+ fails. Applied 2026-10-08: revision `model-hunyuan3d-21-rtx-00003-jms` became
+ Ready in 35 s.
+- **Monitor:** the deep sweep's fleet readiness probe (`npm run triage:gcp:deep`)
+ reports the service as not Ready.
+
+---
+
## 🟡 `[x402-audit] insert failed … db query exceeded 3000ms deadline`
- **Source:** [api/_lib/x402/audit-log.js](../../api/_lib/x402/audit-log.js) `logPaymentEvent`.
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index 497bffdc5..296aba2a5 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -85,6 +85,18 @@ const KNOWN_SIGNATURES = [
class: 'env-action',
action: 'Container OOM. Raise the service memory: gcloud run services update <service> --region us-central1 --memory <2x-current> (config-only, pre-approved).',
},
+ {
+ // Scoped: Cloud Run never prints its "memory limit exceeded" line for
+ // these GPU instances, so a bare SIGKILL is the only trace an OOM leaves.
+ // It stays `investigate` because the code fix (heap trim, freed staged
+ // weights, recycle-between-jobs) should have ended it; a recurrence is a
+ // regression or a new leak, not noise.
+ id: 'gpu-worker-sigkill',
+ match: /Container terminated on signal 9/i,
+ services: ['model-trellis'],
+ class: 'investigate',
+ action: `model-trellis was SIGKILLed, almost always an out-of-memory kill mid-job: its L4 instance is already at Cloud Run's 32 GiB ceiling, so --memory cannot be raised. Confirm with the revision's memory utilization (a step ratchet toward 0.9 before the kill; query in the runbook) and the job log stopping inside mesh postprocess. The in-flight job's record goes orphaned and expires within ~3 min so forge fails it over. A clean "recycling instance" restart is the designed alternative and is not this signature. ${RUNBOOK} §gpu-worker-sigkill.`,
+ },
{
id: 'run-no-instance',
match: /no available instance|The request was aborted because there was no available instance/i,
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index 2b8babc1a..c6e41e3dd 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -41,6 +41,15 @@ describe('gcp-triage classify', () => {
expect(classify(line, ['three-ws-api'])?.id).not.toBe('ccv-ai-providers-unavailable');
});
+ it('classifies a bare SIGKILL as the trellis OOM only on model-trellis', () => {
+ const line = 'Container terminated on signal 9.';
+ const onTrellis = classify(line, ['model-trellis']);
+ expect(onTrellis?.id).toBe('gpu-worker-sigkill');
+ expect(onTrellis?.class).toBe('investigate');
+ expect(classify(line, ['three-ws-api'])).toBeNull();
+ expect(classify(line, ['model-hunyuan3d-21-rtx'])).toBeNull();
+ });
+
it('returns null for an unknown message', () => {
expect(classify('some brand new failure nobody has seen', ['three-ws-api'])).toBeNull();
});