Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(forge): bound the text-to-image paint ladder to one shared budget so a stalled provider cannot hang a submit

nirholas8cca11cf0a37
diff --git a/.env.example b/.env.example
index d5281dfde..9d9f6a234 100644
--- a/.env.example
+++ b/.env.example
@@ -102,6 +102,12 @@ GROQ_API_KEY=
# NOTE: this is the hosted-API var. NGC_API_KEY is only for self-hosted NIM Docker
# containers (nvcr.io) — not used here. Same secret value, different name/path.
NVIDIA_API_KEY=
+# Total time the text-to-image paint ladder (Vertex -> NIM FLUX -> Livepeer ->
+# Replicate) may spend before the forge answers a fast, retryable 429. Each lane
+# keeps its own ceiling; this bounds the ladder as a whole so a stalled leading
+# lane cannot hold a submit open past the caller's socket. Default 60000.
+# TEXT_TO_IMAGE_BUDGET_MS=60000
+
# SambaNova Cloud (free tier, no card): Llama 3.3 70B on its own free quota
# pool (about 20 req/min, 200K tokens/day per model). Optional rung in the
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2b1f428cb..8ec2568fa 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@ Public history for [three.ws](https://three.ws), newest first. New pages come fr
- **Docs · The $THREE thesis** (`/docs/three-thesis`): The long-form case for $THREE: what three.ws is, every utility and sink the token has, how platform revenue routes into buybacks, where it trades and is verified, the ecosystem footprint, the roadmap, and every risk the platform documents about itself.
- **Docs · Weekly Report #1** (`/docs/weekly-report-1`): The first three.ws weekly report: a week-by-week roundup of everything shipped from the first commit on 2026-04-14 through 2026-08-25, with every launch, partnership, announcement, and where $THREE fits.
+- **Text-to-3D no longer hangs when the concept painter stalls**: Every text-to-3D generation starts by painting a concept image, and that step tried its image providers one after another with no overall time limit. When the first provider stalled, a request could sit for two and a half minutes before answering, and anything with a shorter patience (ChatGPT, agent clients) simply saw nothing. The painter now works to one shared budget: a slow provider hands off with real time left for the next one, and if the whole chain runs out you get a quick, honest retry-in-a-moment answer instead of a hang. Image-to-3D was never affected. (`/forge`) `[fix]`
- **The first-contribution guide now hands you real open work, and the docs sidebar can find it**: The 15-minute contributor guide walked through adding MikuMikuDance rig support as its example, but that support had already shipped, so anyone following it literally would have re-done merged work and been told a bone was unmapped when it was not. It now shows the MMD change as the finished reference, exactly as it landed with its tests, and points you at two rig conventions that are genuinely still open, each with the bones that fail today. The community, triage, and first-contribution pages also appear in the docs sidebar under Guides instead of being reachable only from the README, and the triage page now explains every label the repo actually uses. (`/docs/first-contribution`) `[docs, fix]`
- **The full $THREE thesis is now a single document**: A long-form research thesis on $THREE now lives in the docs: what three.ws is, every way the token is used (hold-to-access tiers, plan and pass discounts, marketplace and labor escrow, game cosmetics, gated embeds, deploy-fee waivers), how platform revenue is routed into buybacks, where the token trades and is verified, the ecosystem footprint, the roadmap, and every risk the platform documents about itself, each claim sourced to the code. (`/three-token`) `[docs]`
- **Weekly Report #1: everything shipped since day one, week by week**: The first three.ws weekly report walks every week from the first commit on April 14 to today: what launched, what was announced, which partnerships landed, and where $THREE fits, with the totals (commits, changelog entries, pages, packages, MCP servers, listings) up front. It is written for the timeline, so no tables, and ships with a screenshot kit of the live surfaces. (`/docs/weekly-report-1`) `[docs]`
diff --git a/api/_mcp3d/text-to-image.js b/api/_mcp3d/text-to-image.js
index 67858729c..529e0e7cd 100644
--- a/api/_mcp3d/text-to-image.js
+++ b/api/_mcp3d/text-to-image.js
@@ -44,8 +44,8 @@ const REPLICATE_TERMINAL_STATUSES = new Set(['succeeded', 'failed', 'canceled'])
// prediction; returns the last seen object (caller surfaces a clear error) when
// the poll budget is exhausted. A transient poll blip is retried within budget,
// never fatal — the prediction keeps running upstream regardless.
-async function pollReplicatePrediction(getUrl, token) {
- const deadline = Date.now() + REPLICATE_POLL_TIMEOUT_MS;
+async function pollReplicatePrediction(getUrl, token, { timeoutMs = REPLICATE_POLL_TIMEOUT_MS } = {}) {
+ const deadline = Date.now() + Math.min(REPLICATE_POLL_TIMEOUT_MS, Math.max(1_000, timeoutMs));
let last = null;
while (Date.now() < deadline) {
await sleep(REPLICATE_POLL_INTERVAL_MS);
@@ -108,6 +108,45 @@ const NIM_TIMEOUT_MS = 60_000;
// second attempt would just double the wait before failover (the same reasoning
// that makes the TRELLIS submit timeout terminal). Bounded to one extra attempt
// so a genuinely-down gateway still hands off fast.
+// ── Ladder budget ─────────────────────────────────────────────────────────
+// Every lane below is bounded on its own (Vertex 90s, NIM 60s, Replicate 45s
+// of polling), but the LADDER never was: a stalled leading lane burned its
+// full window, then the next lane burned its own, and the caller's socket
+// (ChatGPT Actions allow ~45s, the OKX/MCP clients 90s) was long gone before a
+// job even existed. Measured on 2026-08-25: text submits hung 95s+ with no
+// answer, or took 156s to say "busy", while image submits answered in 3.7s.
+//
+// The ladder now shares ONE budget (TEXT_TO_IMAGE_BUDGET_MS, default 60s, or
+// the caller's `budgetMs`). Each lane gets min(its own ceiling, what is left),
+// and while a fallback lane still remains the current lane is capped at
+// max(a quarter of the budget, 60% of what is left) so a stalled leader hands
+// off with real time to spare. A lane with under a tenth of the budget left is
+// skipped, and an exhausted ladder surfaces a retryable rate_limited error the
+// forge boundary already maps to a fast 429 with a Retry-After.
+// Vertex's own ceiling lives in vertex-imagen.js (90s); mirrored here so the
+// ladder math can cap it without importing the lane eagerly.
+const VERTEX_LANE_CEILING_MS = 90_000;
+const DEFAULT_BUDGET_MS = 60_000;
+export function ladderBudgetMs(override) {
+ const env = Number(readEnv('TEXT_TO_IMAGE_BUDGET_MS'));
+ const chosen = Number(override) || (Number.isFinite(env) && env > 0 ? env : DEFAULT_BUDGET_MS);
+ return Math.max(5_000, chosen);
+}
+// Pure: how long the current lane may run. `budgetMs` is the ladder total,
+// `remainingMs` what is left of it, `laneRemains` whether a fallback follows.
+export function laneTimeoutMs(ownCeilingMs, { budgetMs, remainingMs, laneRemains }) {
+ if (remainingMs <= 0) return 0;
+ let cap = remainingMs;
+ if (laneRemains) cap = Math.min(remainingMs, Math.max(budgetMs * 0.25, remainingMs * 0.6));
+ return Math.max(0, Math.min(ownCeilingMs, Math.floor(cap)));
+}
+function budgetExhausted(budgetMs) {
+ return Object.assign(
+ new Error('Image generation is taking longer than usual, please retry in a few seconds.'),
+ { code: 'rate_limited', queued: true, retryAfter: 15, budgetMs },
+ );
+}
+
const NIM_GATEWAY_RETRY_STATUSES = new Set([502, 503, 504]);
const NIM_MAX_ATTEMPTS = 2;
const NIM_RETRY_DELAY_MS = 1_200;
@@ -195,14 +234,15 @@ async function persistDataUriImage(result) {
// comes back inline as base64, no poll. Caller guarantees NVIDIA_API_KEY is set.
// Throws on any failure (timeout, throttle, malformed body) so the caller can
// degrade to the paid lanes; never returns a half-result.
-async function nimFluxImage(prompt, aspectRatio, seed = 0) {
+async function nimFluxImage(prompt, aspectRatio, seed = 0, { timeoutMs = NIM_TIMEOUT_MS } = {}) {
const key = readEnv('NVIDIA_API_KEY');
const [width, height] = NIM_DIMENSIONS[aspectRatio] || NIM_DIMENSIONS['1:1'];
+ const laneTimeout = Math.max(1_000, Math.min(NIM_TIMEOUT_MS, Number(timeoutMs) || NIM_TIMEOUT_MS));
let lastErr = null;
for (let attempt = 1; attempt <= NIM_MAX_ATTEMPTS; attempt++) {
const controller = new AbortController();
- const timer = setTimeout(() => controller.abort(), NIM_TIMEOUT_MS);
+ const timer = setTimeout(() => controller.abort(), laneTimeout);
let res;
try {
res = await fetch(NIM_FLUX_URL, {
@@ -348,8 +388,15 @@ export function enhanceFluxPrompt(raw) {
// paid Replicate backstop on any failure — a broken or throttled preferred
// provider must hand off, never take down the whole text→3D pipeline. The last
// configured lane's error is surfaced only when nothing is left to try.
-export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false, seed } = {}) {
+export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false, seed, budgetMs } = {}) {
prompt = enhanceFluxPrompt(prompt);
+ const budget = ladderBudgetMs(budgetMs);
+ const deadline = Date.now() + budget;
+ const remaining = () => deadline - Date.now();
+ // A lane with under a tenth of the budget left cannot finish; skip it.
+ const canTry = () => remaining() >= budget * 0.1;
+ const laneMs = (ownCeilingMs, laneRemains) =>
+ laneTimeoutMs(ownCeilingMs, { budgetMs: budget, remainingMs: remaining(), laneRemains });
// Optional deterministic seed. Honored on the lanes that expose one (NIM FLUX,
// Replicate flux); the Vertex/Gemini image API has no seed parameter, so a seed
// is silently ignored there. Undefined preserves the prior default (seed 0).
@@ -385,7 +432,15 @@ export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false
try {
const { generateImage, isConfigured } = await import('./vertex-imagen.js');
if (!isConfigured()) return null;
- return logImageProvider(await persistDataUriImage(await generateImage(prompt, { aspectRatio })));
+ if (!canTry()) {
+ if (!laneRemains) throw budgetExhausted(budget);
+ return null;
+ }
+ return logImageProvider(
+ await persistDataUriImage(
+ await generateImage(prompt, { aspectRatio, timeoutMs: laneMs(VERTEX_LANE_CEILING_MS, laneRemains) }),
+ ),
+ );
} catch (err) {
if (!laneRemains) throw err;
console.warn(`vertex imagen failed, falling back: ${err?.message}`);
@@ -407,9 +462,15 @@ export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false
const nimCooling =
hasFallback &&
(skipNim || (await providersInCooldown([NIM_FLUX_COOLDOWN_KEY])).has(NIM_FLUX_COOLDOWN_KEY));
- if (readEnv('NVIDIA_API_KEY') && !nimCooling) {
+ const nimWanted = readEnv('NVIDIA_API_KEY') && !nimCooling;
+ if (nimWanted && !canTry() && !hasFallback) throw budgetExhausted(budget);
+ if (nimWanted && canTry()) {
try {
- return logImageProvider(await nimFluxImage(prompt, aspectRatio, hasSeed ? seed : 0));
+ return logImageProvider(
+ await nimFluxImage(prompt, aspectRatio, hasSeed ? seed : 0, {
+ timeoutMs: laneMs(NIM_TIMEOUT_MS, hasFallback),
+ }),
+ );
} catch (err) {
// A degraded lane (timeout / unreachable / throttle / 5xx) cools down so the
// next caller skips it; a clean 4xx (bad input) is not a lane-health fault.
@@ -437,9 +498,12 @@ export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false
// Replicate backstop (it costs real money per image): a successful federated
// call is strictly cheaper than every remaining option. Off by default; the
// measured case for flipping it is in docs/ops/livepeer-federation.md.
- if (hasLivepeer) {
+ if (hasLivepeer && !canTry() && !token) throw budgetExhausted(budget);
+ if (hasLivepeer && canTry()) {
try {
- return logImageProvider(await livepeerTextToImage(prompt, { aspectRatio, seed }));
+ return logImageProvider(
+ await livepeerTextToImage(prompt, { aspectRatio, seed, timeoutMs: laneMs(REPLICATE_POLL_TIMEOUT_MS, !!token) }),
+ );
} catch (err) {
// Nothing downstream to fall through to → surface the Livepeer error.
if (!token) throw err;
@@ -457,6 +521,8 @@ export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false
);
}
+ if (!canTry()) throw budgetExhausted(budget);
+
const modelRef = readEnv('REPLICATE_TXT2IMG_MODEL') || DEFAULT_TXT2IMG_MODEL;
const isVersionHash = /^[a-f0-9]{40,64}$/i.test(modelRef);
const slug = modelRef.match(/^([a-z0-9-]+)\/([a-z0-9._-]+)(?::([a-f0-9]+))?$/i);
@@ -517,6 +583,9 @@ export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false
prefer: 'wait',
},
body,
+ // The create call previously had NO timeout, so a `prefer: wait` that
+ // never returned held the ladder open indefinitely.
+ signal: AbortSignal.timeout(Math.max(1_000, laneMs(REPLICATE_POLL_TIMEOUT_MS, false))),
});
} catch (err) {
throw Object.assign(new Error(`text-to-image provider unreachable: ${err?.message}`), {
@@ -575,7 +644,7 @@ export async function textToImage(prompt, { aspectRatio = '1:1', skipNim = false
const getUrl = data?.urls?.get;
const nonTerminal = data.status && !REPLICATE_TERMINAL_STATUSES.has(data.status);
if (getUrl && nonTerminal) {
- const finished = await pollReplicatePrediction(getUrl, token);
+ const finished = await pollReplicatePrediction(getUrl, token, { timeoutMs: laneMs(REPLICATE_POLL_TIMEOUT_MS, false) });
url = extractImageUrl(finished?.output);
if (url) {
return logImageProvider({ imageUrl: url, predictionId: finished?.id || data.id, model: modelRef });
diff --git a/api/_mcp3d/vertex-imagen.js b/api/_mcp3d/vertex-imagen.js
index 9bc8d1cbd..35c53b71e 100644
--- a/api/_mcp3d/vertex-imagen.js
+++ b/api/_mcp3d/vertex-imagen.js
@@ -138,7 +138,10 @@ function imageLaneLocation() {
);
}
-export async function generateImage(prompt, { aspectRatio = '1:1' } = {}) {
+// `timeoutMs` lets the paint ladder (text-to-image.js) cap this lane to its
+// remaining budget; the module default is the ceiling, never the floor.
+export async function generateImage(prompt, { aspectRatio = '1:1', timeoutMs = VERTEX_TIMEOUT_MS } = {}) {
+ const laneTimeoutMs = Math.max(1_000, Math.min(VERTEX_TIMEOUT_MS, Number(timeoutMs) || VERTEX_TIMEOUT_MS));
const project = readEnv('GOOGLE_CLOUD_PROJECT');
if (!project) {
throw Object.assign(
@@ -238,12 +241,12 @@ async function postJson(endpoint, body, token, label) {
'content-type': 'application/json',
},
body: JSON.stringify(body),
- signal: AbortSignal.timeout(VERTEX_TIMEOUT_MS),
+ signal: AbortSignal.timeout(laneTimeoutMs),
});
} catch (err) {
// A blown VERTEX_TIMEOUT_MS aborts the fetch and lands here too, so a stalled
// socket degrades to the same fallback path as a refused connection.
- const detail = err?.name === 'TimeoutError' ? `no response in ${VERTEX_TIMEOUT_MS}ms` : err?.message;
+ const detail = err?.name === 'TimeoutError' ? `no response in ${laneTimeoutMs}ms` : err?.message;
throw Object.assign(
new Error(`Vertex AI ${label} unreachable: ${detail}`),
{ code: 'provider_unreachable' },
diff --git a/api/_providers/livepeer.js b/api/_providers/livepeer.js
index 80e1f6ba4..c86a3b4a6 100644
--- a/api/_providers/livepeer.js
+++ b/api/_providers/livepeer.js
@@ -114,7 +114,8 @@ function resolveImageUrl(raw, base) {
// "hand off to the next lane"): provider_unreachable for socket/TLS failures,
// rate_limited for 429, verification_failed for a well-formed 200 whose
// payload is not a usable image, and upstream_error otherwise.
-export async function livepeerTextToImage(prompt, { aspectRatio = '1:1', seed, model } = {}) {
+export async function livepeerTextToImage(prompt, { aspectRatio = '1:1', seed, model, timeoutMs = T2I_TIMEOUT_MS } = {}) {
+ const laneTimeoutMs = Math.max(1_000, Math.min(T2I_TIMEOUT_MS, Number(timeoutMs) || T2I_TIMEOUT_MS));
const { base, gateway, key } = livepeerGatewayConfig();
const [width, height] = DIMENSIONS[aspectRatio] || DIMENSIONS['1:1'];
const modelId = model || process.env.LIVEPEER_T2I_MODEL || env.LIVEPEER_T2I_MODEL || DEFAULT_T2I_MODEL;
@@ -143,7 +144,7 @@ export async function livepeerTextToImage(prompt, { aspectRatio = '1:1', seed, m
method: 'POST',
headers,
body: JSON.stringify(body),
- signal: AbortSignal.timeout(T2I_TIMEOUT_MS),
+ signal: AbortSignal.timeout(laneTimeoutMs),
});
} catch (err) {
throw Object.assign(new Error(`livepeer gateway unreachable: ${describeTransportFailure(err)}`), {
@@ -177,7 +178,7 @@ export async function livepeerTextToImage(prompt, { aspectRatio = '1:1', seed, m
try {
imgRes = await fetch(imageUrl, {
headers: key ? { authorization: `Bearer ${key}` } : {},
- signal: AbortSignal.timeout(T2I_TIMEOUT_MS),
+ signal: AbortSignal.timeout(laneTimeoutMs),
});
} catch (err) {
throw Object.assign(new Error(`livepeer image fetch failed: ${describeTransportFailure(err)}`), {
diff --git a/data/changelog.json b/data/changelog.json
index 2ec2f8c31..3c0c48f59 100644
--- a/data/changelog.json
+++ b/data/changelog.json
@@ -1,6 +1,15 @@
{
"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security. `link` is optional and must be a live three.ws path.",
"entries": [
+ {
+ "date": "2026-08-25",
+ "title": "Text-to-3D no longer hangs when the concept painter stalls",
+ "summary": "Every text-to-3D generation starts by painting a concept image, and that step tried its image providers one after another with no overall time limit. When the first provider stalled, a request could sit for two and a half minutes before answering, and anything with a shorter patience (ChatGPT, agent clients) simply saw nothing. The painter now works to one shared budget: a slow provider hands off with real time left for the next one, and if the whole chain runs out you get a quick, honest retry-in-a-moment answer instead of a hang. Image-to-3D was never affected.",
+ "tags": [
+ "fix"
+ ],
+ "link": "/forge"
+ },
{
"date": "2026-08-25",
"title": "The first-contribution guide now hands you real open work, and the docs sidebar can find it",
diff --git a/prompts/okx-ai/PROGRESS.md b/prompts/okx-ai/PROGRESS.md
index 3c9a24cd8..9430db40b 100644
--- a/prompts/okx-ai/PROGRESS.md
+++ b/prompts/okx-ai/PROGRESS.md
@@ -106,8 +106,16 @@ real one (two amounts for one rail). `handleSse` now takes the list price.
0.8 s. The health probe reads the generator's static tier matrix, not its acceptance
latency, so it cannot see a saturated lane. From inside `forge_3d` that call would have hit
the 90 s client timeout and surfaced as `timeout` with `retry_after`, which is honest but
-slow. Worth a follow-up: an acceptance-latency reading in the health probe, and a shorter
-submit budget on the OKX surface.
+slow. **Root-caused and fixed the same day.** Image-to-3D (no paint step) answered in 3.7 s, so
+the hang was the text-to-image paint ladder: Vertex Imagen leads it with a 90 s timeout,
+NIM follows with 60 s, and nothing bounded the ladder as a whole (90 + 60 + throttle = the
+156 s → 429). None of the hung submits ever created a `forge_creations` row, which is what
+placed the stall before job creation. `api/_mcp3d/text-to-image.js` now runs the ladder on
+one shared budget (`TEXT_TO_IMAGE_BUDGET_MS`, default 60 s): a lane with a fallback behind
+it is capped at max(25% of budget, 60% of what is left), a lane with under 10% left is
+skipped, and exhaustion is a fast retryable `rate_limited`. Regression test:
+`tests/text-to-image-budget.test.js`. Still open: the health probe cannot see a saturated
+lane because it reads the static tier matrix.
### Still owner-gated (unchanged, and unchanged by this rebuild)
diff --git a/public/changelog.json b/public/changelog.json
index 09e14c39d..7ecded9b8 100644
--- a/public/changelog.json
+++ b/public/changelog.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-08-25T15:09:27.024Z",
+ "generated_at": "2026-08-25T15:23:45.001Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
@@ -26,6 +26,16 @@
"launch"
]
},
+ {
+ "date": "2026-08-25",
+ "type": "update",
+ "title": "Text-to-3D no longer hangs when the concept painter stalls",
+ "summary": "Every text-to-3D generation starts by painting a concept image, and that step tried its image providers one after another with no overall time limit. When the first provider stalled, a request could sit for two and a half minutes before answering, and anything with a shorter patience (ChatGPT, agent clients) simply saw nothing. The painter now works to one shared budget: a slow provider hands off with real time left for the next one, and if the whole chain runs out you get a quick, honest retry-in-a-moment answer instead of a hang. Image-to-3D was never affected.",
+ "link": "/forge",
+ "tags": [
+ "fix"
+ ]
+ },
{
"date": "2026-08-25",
"type": "update",
diff --git a/public/changelog.xml b/public/changelog.xml
index 1db140db9..5f71aad88 100644
--- a/public/changelog.xml
+++ b/public/changelog.xml
@@ -22,6 +22,14 @@
<category>launch</category>
<description>The first three.ws weekly report: a week-by-week roundup of everything shipped from the first commit on 2026-04-14 through 2026-08-25, with every launch, partnership, announcement, and where $THREE fits.</description>
</item>
+ <item>
+ <title>Text-to-3D no longer hangs when the concept painter stalls</title>
+ <link>https://three.ws/forge</link>
+ <guid isPermaLink="false">2026-08-25:Text-to-3D no longer hangs when the concept painter stalls</guid>
+ <pubDate>Tue, 25 Aug 2026 12:00:00 GMT</pubDate>
+ <category>fix</category>
+ <description>Every text-to-3D generation starts by painting a concept image, and that step tried its image providers one after another with no overall time limit. When the first provider stalled, a request could sit for two and a half minutes before answering, and anything with a shorter patience (ChatGPT, agent clients) simply saw nothing. The painter now works to one shared budget: a slow provider hands off with real time left for the next one, and if the whole chain runs out you get a quick, honest retry-in-a-moment answer instead of a hang. Image-to-3D was never affected.</description>
+ </item>
<item>
<title>The first-contribution guide now hands you real open work, and the docs sidebar can find it</title>
<link>https://three.ws/docs/first-contribution</link>
diff --git a/public/features.json b/public/features.json
index 464991f31..4857fd470 100644
--- a/public/features.json
+++ b/public/features.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-08-25T15:09:27.003Z",
+ "generated_at": "2026-08-25T15:23:44.980Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json",
"site": {
"name": "three.ws",
diff --git a/tests/text-to-image-budget.test.js b/tests/text-to-image-budget.test.js
new file mode 100644
index 000000000..ebf68bf83
--- /dev/null
+++ b/tests/text-to-image-budget.test.js
@@ -0,0 +1,119 @@
+/**
+ * The paint ladder (api/_mcp3d/text-to-image.js) shares ONE time budget.
+ *
+ * Before this, every lane was bounded on its own (Vertex 90s, NIM 60s,
+ * Replicate 45s) and the ladder as a whole was not, so a stalled leading lane
+ * burned its full window, the next lane burned its own, and the caller's
+ * socket was gone before a forge job even existed. Measured live on
+ * 2026-08-25: text submits hung 95s+ or took 156s to answer 429, while image
+ * submits (no paint step) answered in 3.7s.
+ */
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+
+process.env.GOOGLE_CLOUD_PROJECT = 'test-project';
+process.env.NVIDIA_API_KEY = 'nim-test-key';
+delete process.env.REPLICATE_API_TOKEN;
+delete process.env.LIVEPEER_FEDERATION_ENABLED;
+delete process.env.VERTEX_IMAGEN_FIRST;
+delete process.env.TEXT_TO_IMAGE_BUDGET_MS;
+
+// The Vertex lane: configured, and it stalls for exactly the timeout it is
+// handed, which is how a stalled upstream presents (the fetch aborts).
+const vertex = vi.hoisted(() => ({ calls: [] }));
+vi.mock('../api/_mcp3d/vertex-imagen.js', () => ({
+ isConfigured: () => true,
+ generateImage: (prompt, opts) => {
+ vertex.calls.push(opts);
+ return new Promise((_, reject) =>
+ setTimeout(
+ () => reject(Object.assign(new Error(`Vertex AI unreachable: no response in ${opts.timeoutMs}ms`), { code: 'provider_unreachable' })),
+ opts.timeoutMs,
+ ),
+ );
+ },
+}));
+vi.mock('../api/_lib/provider-health.js', () => ({
+ providersInCooldown: async () => new Set(),
+ markProviderCooldown: async () => {},
+}));
+vi.mock('../api/_lib/forge-scale.js', () => ({
+ reserveProviderRateSlot: async () => ({ ok: true, waitMs: 0 }),
+ SCALE_LIMITS: { replicateRatePerMin: 6, replicateRateBurst: 1, replicateQueueMaxMs: 1000 },
+}));
+vi.mock('../api/_lib/image-persist.js', () => ({ persistImageBase64: async () => 'https://cdn.test/img.png' }));
+vi.mock('../api/_providers/livepeer.js', () => ({
+ livepeerFederationEnabled: () => false,
+ livepeerTextToImage: async () => {
+ throw new Error('unused');
+ },
+}));
+
+const { textToImage, laneTimeoutMs, ladderBudgetMs } = await import('../api/_mcp3d/text-to-image.js');
+
+// NIM: a fetch that honours the abort signal (so a lane cap really stops it),
+// or fails fast, per test.
+let nimMode = 'fail-fast';
+globalThis.fetch = vi.fn(
+ (url, init) =>
+ new Promise((resolve, reject) => {
+ if (nimMode === 'fail-fast') return reject(new TypeError('fetch failed'));
+ init?.signal?.addEventListener('abort', () => reject(Object.assign(new Error('aborted'), { name: 'AbortError' })));
+ }),
+);
+
+beforeEach(() => {
+ vertex.calls.length = 0;
+ nimMode = 'fail-fast';
+});
+
+describe('lane budget math', () => {
+ it('caps a lane that still has a fallback behind it so a stalled leader hands off with time to spare', () => {
+ // 60s budget, nothing spent yet: the leader may use 60% of it, not all.
+ expect(laneTimeoutMs(90_000, { budgetMs: 60_000, remainingMs: 60_000, laneRemains: true })).toBe(36_000);
+ // The last lane standing gets everything that is left.
+ expect(laneTimeoutMs(90_000, { budgetMs: 60_000, remainingMs: 60_000, laneRemains: false })).toBe(60_000);
+ // Never more than the lane's own ceiling.
+ expect(laneTimeoutMs(45_000, { budgetMs: 60_000, remainingMs: 60_000, laneRemains: false })).toBe(45_000);
+ // Never more than what is left, even when the floor would say otherwise.
+ expect(laneTimeoutMs(90_000, { budgetMs: 60_000, remainingMs: 10_000, laneRemains: true })).toBe(10_000);
+ expect(laneTimeoutMs(90_000, { budgetMs: 60_000, remainingMs: 0, laneRemains: true })).toBe(0);
+ });
+
+ it('defaults the ladder to 60s and honours the env override and a caller budget', () => {
+ expect(ladderBudgetMs()).toBe(60_000);
+ process.env.TEXT_TO_IMAGE_BUDGET_MS = '20000';
+ expect(ladderBudgetMs()).toBe(20_000);
+ expect(ladderBudgetMs(9_000)).toBe(9_000);
+ delete process.env.TEXT_TO_IMAGE_BUDGET_MS;
+ // A silly budget is floored so no lane is handed a sub-second window.
+ expect(ladderBudgetMs(10)).toBe(5_000);
+ });
+});
+
+describe('the ladder as a whole is bounded', () => {
+ it('a stalled Vertex lane is cut at its cap and NIM still gets its turn, all inside the budget', async () => {
+ const started = Date.now();
+ await expect(textToImage('a fox', { budgetMs: 5_000 })).rejects.toThrow(/fetch failed/);
+ const elapsed = Date.now() - started;
+ // Vertex was handed 60% of the budget, not its 90s ceiling.
+ expect(vertex.calls).toHaveLength(1);
+ // (a millisecond or two elapses between the deadline being set and the
+ // lane being handed its share, so this is a tight band, not an equality)
+ expect(vertex.calls[0].timeoutMs).toBeGreaterThanOrEqual(2_950);
+ expect(vertex.calls[0].timeoutMs).toBeLessThanOrEqual(3_000);
+ // NIM ran after it (one retry after 1.2s), and the whole thing stayed
+ // well inside the budget instead of the old 150s.
+ expect(globalThis.fetch).toHaveBeenCalled();
+ expect(elapsed).toBeLessThan(5_000 + 1_500);
+ }, 15_000);
+
+ it('a stalled NIM lane is cut at what is left of the budget, never its own 60s ceiling', async () => {
+ nimMode = 'hang';
+ const started = Date.now();
+ await expect(textToImage('a fox', { budgetMs: 5_000 })).rejects.toThrow();
+ const elapsed = Date.now() - started;
+ // Vertex 3s, then NIM gets the ~2s remainder and is aborted there.
+ expect(elapsed).toBeGreaterThanOrEqual(4_500);
+ expect(elapsed).toBeLessThan(5_000 + 1_500);
+ }, 15_000);
+});