Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(ops): explain why a failed-over generation is not a lost one in the triage runbook

nirholas9346c88dade8
diff --git a/docs/ops/forge-error-triage.md b/docs/ops/forge-error-triage.md
index 4b383c762..9623dfed0 100644
--- a/docs/ops/forge-error-triage.md
+++ b/docs/ops/forge-error-triage.md
@@ -4,10 +4,11 @@ Answering "what is actually failing in 3D generation, and how often?" from the
outcome ledger, without hand-writing SQL against production.
```bash
-npm run forge:errors # last 7 days, ranked by class
+npm run forge:errors # last 7 days, ranked by class
npm run forge:errors -- --days 30
-npm run forge:errors -- --days 7 --json # machine-readable, same numbers
-npm run forge:errors -- --class timeout # every message in one class
+npm run forge:errors -- --days 7 --json # machine-readable, same numbers
+npm run forge:errors -- --class lost_task # every message in one class
+npm run forge:errors -- --include-recovered # count failed-over attempts too
```
Read-only: the script ([scripts/forge-error-report.mjs](../../scripts/forge-error-report.mjs))
@@ -65,19 +66,61 @@ honest fallback, no invented buckets. Covered by
[tests/forge-error-class.test.js](../../tests/forge-error-class.test.js); when
you add a class, add its case there.
+## A failed attempt is not always a lost generation
+
+When a lane fails a job, the platform re-dispatches the original inputs to the
+next healthy lane: on the attended poll ([api/forge.js](../../api/forge.js)) and,
+long after every browser has stopped polling, on the finalizer cron
+([api/cron/forge-finalize.js](../../api/cron/forge-finalize.js)). The successor
+gets its own creation row and carries the request to completion, so the user
+still gets a model.
+
+The failed attempt keeps its own row, which is correct bookkeeping and used to be
+actively misleading: every reader counted it as a user-visible failure. Measured
+against production on 2026-08-14, over the prior 7 days: **3642 generations, 23
+failures, and 13 of the 14 `trellis_selfhost` orphan failures had already been
+recovered on `hunyuan3d` seconds later.** The top "failure class" was a class the
+platform already handles.
+
+So both failover paths now write `forge_creations.superseded_by` (the successor's
+id) on the attempt they replaced:
+
+- **This report** excludes recovered attempts from the ranking and the success
+ rate, and prints how many it excluded. `--include-recovered` ranks them too,
+ which is what you want when asking "which lane keeps needing a failover?"
+- **The health sensor** excludes them from its window, so the failover machinery
+ working can no longer degrade or page the `forge_generation` subsystem.
+
+The column is forward-looking: it is set when a failover happens, and historical
+rows were deliberately not backfilled, because reconstructing the link from
+matching prompts and timestamps is a guess, and this ledger is the platform's
+ground truth. Expect `recovered 0` on windows that predate it.
+
## Reading the output
+Real output, production, 2026-08-14:
+
```
- generations 812 done 731 failed 74 still running 7
- success rate 90.8%
+ generations 3642 done 3619 failed 23 still running 0
+ of those failures, 0 were re-dispatched to another lane and finished there; 23 were lost
+ success rate 99.4% (recovered attempts are not counted against it)
rank n share class worst lane message
- 1 31 41.9% timeout selfhost×29 generation timed out after 41 minutes
- 2 18 24.3% lost_task selfhost×18 task not found: 7f3a2b19c4d5e6f7
+ 1 19 82.6% lost_task trellis_selfhost×14 NVCF request not found or expired
+ 2 3 13.0% timeout nvidia×3 generation timed out after 45 minutes
+ 3 1 4.3% other trellis_selfhost×1 internal error (ref 16485b338fc5)
- Top class: timeout (timed out before the lane returned)
+ Top class: lost_task (lane lost the task (poll found nothing))
+ 19 of 23 failures (82.6%), lanes trellis_selfhost×14, nvidia×5
```
+One class is 82.6% of all forge failures, and it is the same failure on two
+unrelated lanes: the lane accepted the job and later could not find it. On
+`trellis_selfhost` that is the worker's own orphan reaper (a Cloud Run instance
+restarted mid-job and nothing resumes persisted tasks); on `nvidia` it is NVCF
+expiring the request. Both are recoverable by failover, which is why the
+recovery accounting above matters more than the raw count.
+
- **worst lane** is the `backend` column: which generation lane produced most of
that class. A class concentrated on one lane is a lane problem; one spread
evenly across lanes is a pipeline problem.
@@ -96,7 +139,12 @@ you add a class, add its case there.
worker's own logs for the same period ([gcp-logs.md](gcp-logs.md)).
- **`lost_task`** means the poll asked the lane for a job it does not know about.
On a self-hosted worker that is a restarted or scaled-to-zero instance losing
- in-memory task state.
+ in-memory task state: the worker's reaper writes
+ `task orphaned: no progress within 30 minutes` after
+ `_PENDING_TTL_SECS` ([workers/model-trellis/main.py](../../workers/model-trellis/main.py),
+ and the same reaper in `model-hunyuan3d` and `model-triposg`). On NVIDIA it is
+ `NVCF request not found or expired`. Check the recovered count before treating
+ a spike as user-visible.
- **`upstream_5xx`, `rate_limited`, `unauthorized`, `payment_required`** are all
vendor-side and resolve on the vendor's dashboard or in the env, not in code.
See [production-log-triage.md](production-log-triage.md) for the same