Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Background generation and completion notifications
Forge generations are server-tracked to completion. You can start a generation on /forge, close the tab, and the platform finishes the job, saves the model to your gallery, and tells you when it's ready.
What the user experiences
- Leave any time. The generating panel says so explicitly. Navigating away or closing the tab does not lose the job.
- Come back, pick up where you left off. Returning to
/forgewithin 30 minutes resumes the live progress view for the in-flight job (honest elapsed timer included). The finished model also appears in the on-page gallery on any later visit. - Get notified. Signed-in creators receive a notification when a generation finishes unattended: the bell inbox and Web Push fire with a link straight to the model, and an email goes out with a preview image. Failures notify too (bell and push only). Every channel has its own switch under the Creations category in the notification preference center. Anonymous users aren't left out: their browser resumes the job from local storage and the finished model is waiting in their gallery.
Attended completions never notify. If the result appeared on your screen, the platform does not also push, email, or badge you about it.
How it works
Three cooperating pieces:
Client resume (src/forge.js): after a job is submitted, its signed job token persists in
localStorage(forge:inflight). On page load the forge resumes polling that token; all real state lives server-side, so a reload costs nothing. The record clears on completion, terminal failure, or cancel. The poll loop itself is built on src/shared/resilient-poll.js (retry, visibility-wake, attempt-based budget), because a generation runs for minutes and on a phone the tab is backgrounded, the radio drops, and the edge answers the odd 502, none of which mean the job died. A connection we finally gave up waiting on throws withkind = 'transport_lost', and that one case deliberately keeps the inflight record rather than clearing it, which is what makes the error's own "reopen /forge to pick it back up" true.Server-side finalizer (api/cron/forge-finalize.js, every minute via Cloud Scheduler): sweeps
forge_creationsrows still atstatus='generating'after a 2-minute grace period. Rows whose job id is a GCP worker envelope (the async self-host lanes: TRELLIS, Hunyuan3D, TripoSG sketch) are polled directly on the worker, and NVIDIA NIM rows are polled too (the x402 lane stores the signed job token; browser free-lane rows store the bare NVCF request id). Finished jobs are materialized into durable storage with the exact same writer the browser poll uses (materializeCreation). Failures do not dead-end: a failed self-host job past the attended polling window (13 minutes, so a live browser's own failover is never raced) is redispatched unattended to the next healthy lane under the same hop cap as the attended path, and an expired NVCF request is resubmitted from the inputs stored on the row. A redispatched attempt is linked to its successor row (forge_creations.superseded_by, migration20260814200000_forge_failover_supersede.sql), so the outcome ledger records a recovery rather than a loss. Only a job with nothing left to try is marked failed, and the finalizer logs that it declined the failover (every alternative lane already attempted or cooling) rather than failing silently. Anything non-terminal after 45 minutes is marked failed as timed out, so no row can be orphaned atgeneratingagain.Notifications (api/_lib/forge-notify.js): the finalizer, and only the finalizer, notifies.
forge_completeandforge_failedare standard notification types in thecreationspreference category (defaults: in-app, push, email and the on-page companion avatar on; Telegram off; email only for completions). Theavatarchannel is delivered client-side by the corner companion (src/notification-herald.js), which walks on and says it out loud while the user is on the site; it can be muted per browser from its bubble. Email delivery uses the platform Resend pipeline and skips undeliverable placeholder addresses.Developer webhooks are the one channel that ignores the attended / unattended split.
forge.completedfires frommaterializeCreation, the universal completion writer, so an API integrator learns every outcome whether or not a browser was watching.forge.failedfires fromnotifyForgeFailedinstead, because this is the only place that can tell a terminal failure from a failover that is still running:markFailedalso runs on the redispatch path, and telling a subscriber the job died while it is still running would be a lie. See api-reference.md.
Self-host poll recovery (the "task not found" grace)
A self-host worker poll (/tasks/:id or /jobs/:id) can 404 when the durable
task record is not yet visible to the instance the poll load-balanced to: the
brief post-submit cross-instance window (workers run at high concurrency with no
session affinity), or a completion write racing the poll. The worker persists
each task's queued state to GCS before returning its 202 and re-reads GCS for
non-terminal records, so a 404 is almost never a real loss.
The router used to treat that first 404 as terminal, which made
task not found on gcp service the platform's single largest generation-failure
class (image→3D). Recovery now mirrors the NVCF path (never dead-end on a
recoverable signal): the GCP provider tags a 404 with code: 'gcp_task_missing'
(api/_providers/gcp.js) and the poll handler runs a
pure decision (api/_lib/forge-selfhost-recovery.js):
- Store re-check: if the creation row already materialized (
done+ a glb), a racing poll or the completion write beat us; resolve as complete. - Grace window: a job younger than
GCP_TASK_MISSING_GRACE_MS(90s) reportsrunningso the client keeps polling; the durable record resolves within seconds. - Terminal: past the window a still-missing task is genuinely orphaned; the failure surfaces and the existing lane failover redispatches to a healthy lane. Nothing here disables failover, it just stops firing it on transient 404s.
Generation success-rate health
Lane liveness ("can a worker serve right now?") is not the same as outcome
health ("are generations succeeding?"). A warm lane can still fail half its
jobs: the 404 burst above ran ~48% failure for days while every liveness probe
read healthy. The Forge 3D generation subsystem
(api/_lib/ops/forge-health-sensor.js)
closes that: it reads forge_creations outcomes over a 6h window and reports the
success rate (done / (done + failed); running/queued excluded, and so are
attempts that were re-dispatched to another lane, superseded_by set, so the
failover machinery working is never counted as loss), naming the
worst backend/path and top error class so a page is actionable. It rolls into
/api/healthz, /status, and the uptime-cron escalation. States: ok ≥ 85%,
degraded 60 to 85%, down < 60%, unknown below 15 finished generations
in-window. This is the forge twin of the x402 settlement-success sensor.
The rate can only judge generations that got far enough to write a
forge_creations row, so classifyForgeStall answers the other question: has
the forge stopped starting generations? A failure upstream of that row (on
2026-09-07, object storage rejecting the reference-image upload every lane
needs) writes no rows at all, and no rows otherwise reads as a quiet hour. A
lane averaging at least 3 starts an hour that has started nothing for 45 minutes
reports down with the hint to check object_storage in the same payload
first, because that shape is a shared dependency rather than a lane fault.
Operational notes
- The finalizer needs
CRON_SECRET(the shared fail-closed cron gate,api/_lib/cron-auth.js),DATABASE_URL, the R2S3_*storage vars, andGCP_RECONSTRUCTION_KEYto poll workers. With no worker key it still reaps timed-out rows. - Batch size is 25 per tick (
FORGE_FINALIZE_BATCHoverrides). Ticks are idempotent and safe to overlap. - The response body reports
{ swept, done, failed, failed_over, resubmitted, timed_out, still_running, unpollable }per tick, so a quick manual hit shows queue health at a glance. - A finished mesh also gets a phone-sized delivery variant (
buildWebVariant, meshopt geometry plus WebP textures capped at 2048 px) written straight after materialization, to a second key, so the stored original is never touched anddownloadand every API consumer keep the full-resolution mesh. Meshes forged before 2026-09-04 are backfilled newest-first by api/cron/forge-web-variants.js (every 10 minutes,FORGE_WEB_VARIANT_BATCHmeshes per tick, default 6). A row whose variant would not be smaller keepsweb_glb_urlnull, and every reader already treats null as "serveglb_url".
Related: 3d-pipeline.md for the generation lanes themselves, and the notification preference model in api/_lib/notify-prefs.js.