Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(ops): explain why a failed-over generation is not a lost one in the triage runbook
nirholas9346c88dade8
diff --git a/docs/ops/forge-error-triage.md b/docs/ops/forge-error-triage.mdindex 4b383c762..9623dfed0 100644--- a/docs/ops/forge-error-triage.md+++ b/docs/ops/forge-error-triage.md@@ -4,10 +4,11 @@ Answering "what is actually failing in 3D generation, and how often?" from theoutcome ledger, without hand-writing SQL against production.```bash-npm run forge:errors # last 7 days, ranked by class+npm run forge:errors # last 7 days, ranked by classnpm run forge:errors -- --days 30-npm run forge:errors -- --days 7 --json # machine-readable, same numbers-npm run forge:errors -- --class timeout # every message in one class+npm run forge:errors -- --days 7 --json # machine-readable, same numbers+npm run forge:errors -- --class lost_task # every message in one class+npm run forge:errors -- --include-recovered # count failed-over attempts too```Read-only: the script ([scripts/forge-error-report.mjs](../../scripts/forge-error-report.mjs))@@ -65,19 +66,61 @@ honest fallback, no invented buckets. Covered by[tests/forge-error-class.test.js](../../tests/forge-error-class.test.js); whenyou add a class, add its case there.+## A failed attempt is not always a lost generation++When a lane fails a job, the platform re-dispatches the original inputs to the+next healthy lane: on the attended poll ([api/forge.js](../../api/forge.js)) and,+long after every browser has stopped polling, on the finalizer cron+([api/cron/forge-finalize.js](../../api/cron/forge-finalize.js)). The successor+gets its own creation row and carries the request to completion, so the user+still gets a model.++The failed attempt keeps its own row, which is correct bookkeeping and used to be+actively misleading: every reader counted it as a user-visible failure. Measured+against production on 2026-08-14, over the prior 7 days: **3642 generations, 23+failures, and 13 of the 14 `trellis_selfhost` orphan failures had already been+recovered on `hunyuan3d` seconds later.** The top "failure class" was a class the+platform already handles.++So both failover paths now write `forge_creations.superseded_by` (the successor's+id) on the attempt they replaced:++- **This report** excludes recovered attempts from the ranking and the success+ rate, and prints how many it excluded. `--include-recovered` ranks them too,+ which is what you want when asking "which lane keeps needing a failover?"+- **The health sensor** excludes them from its window, so the failover machinery+ working can no longer degrade or page the `forge_generation` subsystem.++The column is forward-looking: it is set when a failover happens, and historical+rows were deliberately not backfilled, because reconstructing the link from+matching prompts and timestamps is a guess, and this ledger is the platform's+ground truth. Expect `recovered 0` on windows that predate it.+## Reading the output+Real output, production, 2026-08-14:+```- generations 812 done 731 failed 74 still running 7- success rate 90.8%+ generations 3642 done 3619 failed 23 still running 0+ of those failures, 0 were re-dispatched to another lane and finished there; 23 were lost+ success rate 99.4% (recovered attempts are not counted against it)rank n share class worst lane message- 1 31 41.9% timeout selfhost×29 generation timed out after 41 minutes- 2 18 24.3% lost_task selfhost×18 task not found: 7f3a2b19c4d5e6f7+ 1 19 82.6% lost_task trellis_selfhost×14 NVCF request not found or expired+ 2 3 13.0% timeout nvidia×3 generation timed out after 45 minutes+ 3 1 4.3% other trellis_selfhost×1 internal error (ref 16485b338fc5)- Top class: timeout (timed out before the lane returned)+ Top class: lost_task (lane lost the task (poll found nothing))+ 19 of 23 failures (82.6%), lanes trellis_selfhost×14, nvidia×5```+One class is 82.6% of all forge failures, and it is the same failure on two+unrelated lanes: the lane accepted the job and later could not find it. On+`trellis_selfhost` that is the worker's own orphan reaper (a Cloud Run instance+restarted mid-job and nothing resumes persisted tasks); on `nvidia` it is NVCF+expiring the request. Both are recoverable by failover, which is why the+recovery accounting above matters more than the raw count.+- **worst lane** is the `backend` column: which generation lane produced most ofthat class. A class concentrated on one lane is a lane problem; one spreadevenly across lanes is a pipeline problem.@@ -96,7 +139,12 @@ you add a class, add its case there.worker's own logs for the same period ([gcp-logs.md](gcp-logs.md)).- **`lost_task`** means the poll asked the lane for a job it does not know about.On a self-hosted worker that is a restarted or scaled-to-zero instance losing- in-memory task state.+ in-memory task state: the worker's reaper writes+ `task orphaned: no progress within 30 minutes` after+ `_PENDING_TTL_SECS` ([workers/model-trellis/main.py](../../workers/model-trellis/main.py),+ and the same reaper in `model-hunyuan3d` and `model-triposg`). On NVIDIA it is+ `NVCF request not found or expired`. Check the recovered count before treating+ a spike as user-visible.- **`upstream_5xx`, `rate_limited`, `unauthorized`, `payment_required`** are allvendor-side and resolve on the vendor's dashboard or in the env, not in code.See [production-log-triage.md](production-log-triage.md) for the same