Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(forge-quality): park a billing-denied Vertex so the gate scores on the free lane instead of failing open
nirholasf8c90123f372
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3ae11352e..aeeeb2a4e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -70,6 +70,7 @@ Public history for [three.ws](https://three.ws), newest first. New pages come fr- **The press kit reads in the three.ws typeface, fits a phone, and quotes every mark at its true size**: Every paragraph on the press kit was rendering in the browser default serif on a bare white-ish ground: the page styled its own cards and headings but never applied the site typeface, colour, or background to the document itself, so the one page a journalist opens looked nothing like the rest of three.ws. It now uses the platform design tokens throughout, in both light and dark mode. On a 320px phone the two OpenAI announcement cards were sliced off at the right edge, taking their Download PNG buttons with them, and because the page clips rather than scrolls sideways there was no way to reach what was cut; both card grids now fold to a single full-width column instead. The pixel dimensions printed under each mark were also all slightly wrong, off by a pixel or two from the artwork they labelled, which is the number a designer laying out a conference programme actually acts on. They are corrected, and the brand asset renderer now writes them back from the PNG headers on every build the way it already did for the archive size, so they cannot drift again. Last, the Copy buttons beside the boilerplate did nothing at all when the browser withheld clipboard access; they now select the passage in place and tell you which keys to press. (`/press`) `[fix, improvement]`- **The Pump Visualizer's live counters stopped freezing a few seconds after the page opened**: A full audit of /pump-visualizer, the 3D scene of live pump.fun mints. The status bar across the top counts the tokens in the scene, the mint rate per minute and the total market cap, and about fifteen seconds into every visit all three froze: the counter sat at 0 and the rate and total sat at a dash while the scene behind them kept filling with new coins. The cause was the page's own translation pass. Those three numbers were written inside blocks of translated copy, so the moment the language file arrived the browser rebuilt that copy and threw away the very elements the page was writing into. The numbers now live beside their labels instead of inside them, so the labels still translate into all 85 languages and the values keep counting for as long as the page is open. The per-minute and per-hour units now translate too, which they never did once the page switched between live mints and migrations. The page also gained a heading and a labelled search box for screen readers, and its browser tab now reads Pump.fun Visualizer, three.ws instead of a retired product name. (`/pump-visualizer`) `[fix, improvement]`- **The Pump.fun Stream tells you it is live, counts what it has seen, and has working Recent and Favorites tabs**: A full audit of /pumpfun, the live pump.fun feed with a 3D agent narrating it. The connection badge in the header said "connecting..." forever, even with events pouring in, and the counter on the "Powered by PumpPortal" pill stayed on zero no matter how many launches arrived. Both were the same fault: the page's translated-copy pass rewrote those two headings wholesale every time it ran, throwing away the very elements the page updates. Seven controls were affected, including the Share button's "Copied!" flash and the avatar picker's name. Each one now hands the page ownership of its live value. The avatar picker's Recent and Favorites tabs were listed but had nothing behind them: opening either showed a blank panel. Both work now, kept in your own browser: every avatar you narrate with lands in Recent, and a star on any card adds it to Favorites. Filtering the feed down to nothing used to leave a blank column; it now says how many events are hidden and offers one click to clear the filters. If the stream drops, the feed says so and offers a Reconnect button instead of sitting on "Waiting for events...", and it notices about twice as fast as before. Also: the page title now matches the page (it said "PumpFun migrations" while the page streamed new tokens, graduations and claims too), a "Launch a coin" link sits next to the status badge, the first-time-fee-claims panel shows skeleton rows during its load and an actionable message with a retry if it fails, and clicking a tab in the agent-config dialog no longer reaches into the avatar picker and deselects its tabs. (`/pumpfun`) `[fix, improvement]`+- **The quality check on a generated model keeps running when one AI provider goes down**: Every generated model can be looked at by a vision model before you see it, which is what catches a flat slab or a melted blob before it reaches you. That check asks one provider first and a free provider second, but when the first one started refusing every request it was still asked first every single time, and half the checks ended up returning no verdict at all, so models shipped unchecked. The check now notices a provider that is refusing, sets it aside for a few minutes, and goes straight to the one that is answering. The set-aside expires by itself, so the moment the first provider is healthy again it is used again with nothing to switch back on. `[fix, improvement]`- **The Recording Pipeline board stops showing mainnet numbers when you ask it about devnet**: A full audit of /pipeline, the one-glance health board for the loop that turns every pump.fun launch into a trade signal. The recorder worker writes a single liveness record for whichever network it is recording, and the board was reading that record no matter which network you selected: switching to devnet showed the recorder as LIVE and told you the loop was waiting on the next launch, when nothing had ever recorded there. Devnet now answers honestly, with the recorder marked off and the reason spelled out, and the whole board follows: idle, every stage at zero, and the deploy instruction on screen. Switching networks while a refresh was already running used to be dropped silently, so you got the previous network's numbers under the new label; the newer request now always wins. Five more fixes came with it. The board refreshes every eight seconds and used to rebuild all seven cards each time, which threw away keyboard focus mid-read and made screen readers re-announce the entire grid on every tick; it now updates the numbers in place and announces only the one-line health summary. A response missing a section no longer collapses the page into a false connection error. The refresh spinner turns for exactly as long as the request actually takes. The network switch and refresh controls have a visible focus ring. And the board now points at the public endpoint behind it, so an agent can read the same stages and the same recommended next action as JSON. (`/pipeline`) `[fix, improvement]`- **The retired Flappin UFO page is readable in light mode, and its two bottom-corner controls stopped sitting on top of each other**: A full audit of /play/ufo, the page that stands where the retired Flappin UFO arcade demo used to be. In light mode its headline was painted in a violet-to-cyan gradient that all but vanished against the white card, the Retired badge and the Still live label were too pale to read, and the keyboard focus ring was that same near-invisible cyan, so tabbing through the page showed you nothing. All four now clear the contrast bar in both themes. The three links to the experiences that are still live were 37 pixels tall and the three.ws logo link was 21, both under the 44 pixels a thumb needs; every link on the page now clears it. The language picker and the Ctrl+K search hint were both pinned to the bottom right corner and overlapped each other, because this page carries its own compact header instead of the site nav that normally arranges that corner; they now stack. Two fixes from this audit land site-wide: the search hint was drawing dark text on its dark default pill on every light-themed page that does not define the site's surface colors, and neither floating control could be reached by screen reader landmark navigation. The page copy also lost the long dash the house style bans, in English and in all 84 translations. (`/play/ufo`) `[fix, improvement]`- **The sitemap filter matches words in any order, opens the first hit on Enter, and stops leaving an empty bar behind**: A full audit of /sitemap, the human-readable index of all 924 pages on three.ws. Filtering used to test your whole query as one run of characters, so "avatar studio" found the Avatar Studio but "studio avatar" found nothing; every word is now matched independently and either order returns the same 23 pages. Typing a query that matched nothing left the section jump-list on screen as an empty bordered bar hovering above the no-results panel; it now steps aside with the sections it points at. Pressing Enter in the filter box did nothing at all, which is the one key anybody tries after typing a page name; it now opens the first match, and the hint at the right of the box switches from the / shortcut to a return arrow to say so. Under the surface: the address bar is updated once you stop typing rather than on every keystroke, which is what Safari's limit on that call actually allows, and each row's visibility is only rewritten when it changes rather than 924 times per key. A query of nothing but spaces is no longer treated as a search. (`/sitemap`) `[improvement, fix]`diff --git a/api/_lib/forge-quality-gate.js b/api/_lib/forge-quality-gate.jsindex 4c81e431d..d914b3309 100644--- a/api/_lib/forge-quality-gate.js+++ b/api/_lib/forge-quality-gate.js@@ -27,6 +27,17 @@// free lanes rather than silently disabling itself.// 3. Neither available → fail-open pass with qa_available:false.//+// Rung 1 is health-gated, which matters more than it sounds. A project-wide+// billing denial ("Lightning dunning decision is deny for project") makes Vertex+// answer 403 to every request until ops fixes billing, and without a breaker the+// gate spends attempt-0 of EVERY generation on a lane that cannot answer. The+// same circuit breaker the vision chain uses (provider-health) now covers this+// lane: an auth/billing fault parks Vertex for AUTH_COOLDOWN_SECONDS so the very+// next generation goes straight to the free non-GCP rungs, and the park expires+// on its own, so the lane recovers with no deploy the moment billing is+// restored. Measured on production 2026-09-09 under the live hold: 5 of 10+// calls fell open with no verdict while Vertex was re-probed on every call.+//// ── The router seam ───────────────────────────────────────────────────────────// The generation router (api/forge.js, owned by another agent) wires this in by:// const verdict = await runQualityGate({ glbUrl, prompt });@@ -38,6 +49,12 @@import { getGcpAccessToken } from './gcp-auth.js';import { describeImageJson, visionConfigured, parseJsonLoose } from './vision.js';+import {+ AUTH_COOLDOWN_SECONDS,+ clearProviderCooldown,+ markProviderCooldown,+ providersInCooldown,+} from './provider-health.js';import { renderAvatarScene, SCENE_PRESETS } from './avatar-render.js';import { validatePublicUrl, isPrivateAddress, SsrfError } from './ssrf.js';import { isIP } from 'node:net';@@ -73,6 +90,14 @@ export const QUALITY_GATE_DEFAULTS = Object.freeze({timeoutMs: intEnv('FORGE_QUALITY_TIMEOUT_MS', 25_000),});+// Circuit-breaker key for the Vertex scoring rung. Namespaced away from the+// vision chain's own lanes so parking the quality gate's Vertex call never+// disables Vertex for a different consumer.+const VERTEX_QUALITY_LANE = 'forge-quality:vertex';+// Transient-failure park (timeout, 5xx, network blip). Matches the vision+// chain's lane cooldown so the two breakers behave the same way.+const VISION_FALLBACK_COOLDOWN_SECONDS = 45;+// ── Subject awareness ─────────────────────────────────────────────────────────// The photoreal bar differs by subject: a person needs correct anatomy and skin;// food needs appetizing surface and no plastic sheen; a vehicle needs panel gaps@@ -439,10 +464,28 @@ export async function runQualityGate({let scored = null;const failures = [];if (vertexQualityConfigured()) {- try {- scored = await scoreViaVertex({ ...image, prompt, subject: subj, timeoutMs: QUALITY_GATE_DEFAULTS.timeoutMs });- } catch (e) {- failures.push(`vertex: ${e?.message || e}`);+ // Skip a lane already known to be refusing. Best-effort: an unreadable+ // cache reads as "not cooling", which is exactly the pre-breaker path.+ const cooling = await providersInCooldown([VERTEX_QUALITY_LANE]);+ if (cooling.has(VERTEX_QUALITY_LANE)) {+ failures.push(`vertex: skipped, lane cooling (${cooling.get(VERTEX_QUALITY_LANE)})`);+ } else {+ try {+ scored = await scoreViaVertex({ ...image, prompt, subject: subj, timeoutMs: QUALITY_GATE_DEFAULTS.timeoutMs });+ void clearProviderCooldown(VERTEX_QUALITY_LANE);+ } catch (e) {+ failures.push(`vertex: ${e?.message || e}`);+ // 401/402/403 is a key or billing fault: broken until ops acts, so+ // park it for the long window instead of re-probing every call. Any+ // other failure is treated as transient and parked briefly.+ const status = Number(e?.status);+ const authFault = status === 401 || status === 402 || status === 403;+ void markProviderCooldown(+ VERTEX_QUALITY_LANE,+ authFault ? AUTH_COOLDOWN_SECONDS : VISION_FALLBACK_COOLDOWN_SECONDS,+ authFault ? 'auth' : 'health',+ );+ }}}if (!scored && visionConfigured()) {diff --git a/data/changelog.json b/data/changelog.jsonindex 7117c31d5..d924dcc26 100644--- a/data/changelog.json+++ b/data/changelog.json@@ -27997,6 +27997,15 @@"sdk"],"link": "/docs/web-component"+ },+ {+ "date": "2026-09-09",+ "title": "The quality check on a generated model keeps running when one AI provider goes down",+ "summary": "Every generated model can be looked at by a vision model before you see it, which is what catches a flat slab or a melted blob before it reaches you. That check asks one provider first and a free provider second, but when the first one started refusing every request it was still asked first every single time, and half the checks ended up returning no verdict at all, so models shipped unchecked. The check now notices a provider that is refusing, sets it aside for a few minutes, and goes straight to the one that is answering. The set-aside expires by itself, so the moment the first provider is healthy again it is used again with nothing to switch back on.",+ "tags": [+ "fix",+ "improvement"+ ]}]}diff --git a/public/changelog-recent.json b/public/changelog-recent.jsonindex 0ec70d83a..8589c17c8 100644--- a/public/changelog-recent.json+++ b/public/changelog-recent.json@@ -1,11 +1,11 @@{- "generated_at": "2026-09-09T20:27:30.345Z",+ "generated_at": "2026-09-09T22:09:39.893Z","generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json","site": {"name": "three.ws","url": "https://three.ws"},- "total": 3656,+ "total": 3657,"complete": false,"full_feed": "/changelog.json","entries": [@@ -700,6 +700,17 @@"improvement"]},+ {+ "date": "2026-09-09",+ "type": "update",+ "title": "The quality check on a generated model keeps running when one AI provider goes down",+ "summary": "Every generated model can be looked at by a vision model before you see it, which is what catches a flat slab or a melted blob before it reaches you. That check asks one provider first and a free provider second, but when the first one started refusing every request it was still asked first every single time, and half the checks ended up returning no verdict at all, so models shipped unchecked. The check now notices a provider that is refusing, sets it aside for a few minutes, and goes straight to the one that is answering. The set-aside expires by itself, so the moment the first provider is healthy again it is used again with nothing to switch back on.",+ "link": null,+ "tags": [+ "fix",+ "improvement"+ ]+ },{"date": "2026-09-09","type": "update",@@ -1601,16 +1612,6 @@"tags": ["launch"]- },- {- "date": "2026-09-05",- "type": "launch",- "title": "Printable IRL sign",- "summary": "Print a sign for a placed IRL agent: its name, thumbnail and a QR of the visit link. Scan it at the spot to open IRL, walk up, and meet the 3D AI agent in AR. Carries no coordinate.",- "link": "/irl/sign",- "tags": [- "launch"- ]}]}diff --git a/public/changelog.json b/public/changelog.jsonindex 29246649d..ce34ce86e 100644--- a/public/changelog.json+++ b/public/changelog.json@@ -1,5 +1,5 @@{- "generated_at": "2026-09-09T20:27:30.268Z",+ "generated_at": "2026-09-09T22:09:39.871Z","generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json","site": {"name": "three.ws",@@ -697,6 +697,17 @@"improvement"]},+ {+ "date": "2026-09-09",+ "type": "update",+ "title": "The quality check on a generated model keeps running when one AI provider goes down",+ "summary": "Every generated model can be looked at by a vision model before you see it, which is what catches a flat slab or a melted blob before it reaches you. That check asks one provider first and a free provider second, but when the first one started refusing every request it was still asked first every single time, and half the checks ended up returning no verdict at all, so models shipped unchecked. The check now notices a provider that is refusing, sets it aside for a few minutes, and goes straight to the one that is answering. The set-aside expires by itself, so the moment the first provider is healthy again it is used again with nothing to switch back on.",+ "link": null,+ "tags": [+ "fix",+ "improvement"+ ]+ },{"date": "2026-09-09","type": "update",diff --git a/public/changelog.xml b/public/changelog.xmlindex 3fc4f9de0..384bf1051 100644--- a/public/changelog.xml+++ b/public/changelog.xml@@ -518,6 +518,14 @@<category>fix</category><description>A full audit of /pumpfun, the live pump.fun feed with a 3D agent narrating it. The connection badge in the header said "connecting..." forever, even with events pouring in, and the counter on the "Powered by PumpPortal" pill stayed on zero no matter how many launches arrived. Both were the same fault: the page's translated-copy pass rewrote those two headings wholesale every time it ran, throwing away the very elements the page updates. Seven controls were affected, including the Share button's "Copied!" flash and the avatar picker's name. Each one now hands the page ownership of its live value. The avatar picker's Recent and Favorites tabs were listed but had nothing behind them: opening either showed a blank panel. Both work now, kept in your own browser: every avatar you narrate with lands in Recent, and a star on any card adds it to Favorites. Filtering the feed down to nothing used to leave a blank column; it now says how many events are hidden and offers one click to clear the filters. If the stream drops, the feed says so and offers a Reconnect button instead of sitting on "Waiting for events...", and it notices about twice as fast as before. Also: the page title now matches the page (it said "PumpFun migrations" while the page streamed new tokens, graduations and claims too), a "Launch a coin" link sits next to the status badge, the first-time-fee-claims panel shows skeleton rows during its load and an actionable message with a retry if it fails, and clicking a tab in the agent-config dialog no longer reaches into the avatar picker and deselects its tabs.</description></item>+ <item>+ <title>The quality check on a generated model keeps running when one AI provider goes down</title>+ <link>https://three.ws/changelog</link>+ <guid isPermaLink="false">2026-09-09:The quality check on a generated model keeps running when one AI provider goes down</guid>+ <pubDate>Wed, 09 Sep 2026 12:00:00 GMT</pubDate>+ <category>fix</category>+ <description>Every generated model can be looked at by a vision model before you see it, which is what catches a flat slab or a melted blob before it reaches you. That check asks one provider first and a free provider second, but when the first one started refusing every request it was still asked first every single time, and half the checks ended up returning no verdict at all, so models shipped unchecked. The check now notices a provider that is refusing, sets it aside for a few minutes, and goes straight to the one that is answering. The set-aside expires by itself, so the moment the first provider is healthy again it is used again with nothing to switch back on.</description>+ </item><item><title>The Recording Pipeline board stops showing mainnet numbers when you ask it about devnet</title><link>https://three.ws/pipeline</link>diff --git a/public/features.json b/public/features.jsonindex 2c923c043..bcfd0da69 100644--- a/public/features.json+++ b/public/features.json@@ -1,5 +1,5 @@{- "generated_at": "2026-09-09T20:08:05.336Z",+ "generated_at": "2026-09-09T22:09:39.860Z","generated_by": "scripts/build-page-index.mjs from data/pages.json","site": {"name": "three.ws",diff --git a/tests/forge-quality-gate-vertex-breaker.test.js b/tests/forge-quality-gate-vertex-breaker.test.jsnew file mode 100644index 000000000..6e2fb1ff7--- /dev/null+++ b/tests/forge-quality-gate-vertex-breaker.test.js@@ -0,0 +1,116 @@+// The quality gate's Vertex rung is health-gated.+//+// Under a project-wide billing denial Vertex answers 403 to every request until+// ops restores billing. Without a breaker the gate spent attempt-0 of every+// single generation on that lane, and on production (2026-09-09, live hold)+// 5 of 10 scoring calls returned no verdict at all. These tests pin the fix:+// the first 403 parks the lane for the auth window, and the next call skips it+// and scores on the free non-GCP chain instead.+import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from 'vitest';++const VERDICT = {+ score: 78,+ realism: 70,+ completeness: 85,+ defects: [],+ reason: 'clean model',+};++const describeImageJson = vi.fn(async () => ({+ json: VERDICT,+ provider: 'nvidia',+ model: 'meta/llama-3.2-11b-vision-instruct',+}));++vi.mock('../api/_lib/vision.js', () => ({+ describeImageJson: (...args) => describeImageJson(...args),+ visionConfigured: () => true,+ parseJsonLoose: (text) => JSON.parse(text),+}));++vi.mock('../api/_lib/gcp-auth.js', () => ({+ getGcpAccessToken: async () => 'test-token',+}));++let runQualityGate;+let clearProviderCooldown;+const fetchSpy = vi.fn();+const realFetch = globalThis.fetch;+const PNG_1PX =+ 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==';++beforeAll(async () => {+ process.env.NODE_ENV = 'development';+ process.env.GOOGLE_CLOUD_PROJECT = 'test-project';+ globalThis.fetch = fetchSpy;+ ({ runQualityGate } = await import('../api/_lib/forge-quality-gate.js'));+ ({ clearProviderCooldown } = await import('../api/_lib/provider-health.js'));+});++afterAll(() => {+ globalThis.fetch = realFetch;+ delete process.env.GOOGLE_CLOUD_PROJECT;+});++beforeEach(async () => {+ fetchSpy.mockReset();+ describeImageJson.mockClear();+ await clearProviderCooldown('forge-quality:vertex');+});++function vertexDenied() {+ return {+ ok: false,+ status: 403,+ text: async () =>+ JSON.stringify({ error: { code: 403, message: 'Lightning dunning decision is deny for project', status: 'PERMISSION_DENIED' } }),+ };+}++function score() {+ return runQualityGate({ renderBase64: PNG_1PX, mimeType: 'image/png', prompt: 'a ceramic teapot' });+}++describe('quality gate, Vertex circuit breaker', () => {+ it('still scores through the free non-GCP chain when Vertex is denied', async () => {+ fetchSpy.mockResolvedValueOnce(vertexDenied());++ const verdict = await score();++ expect(fetchSpy).toHaveBeenCalledTimes(1);+ expect(verdict.qa_available).toBe(true);+ expect(verdict.provider).toBe('nvidia');+ expect(verdict.score).toBe(78);+ });++ it('skips the denied lane on the next call instead of re-probing it', async () => {+ fetchSpy.mockResolvedValueOnce(vertexDenied());+ await score();+ expect(fetchSpy).toHaveBeenCalledTimes(1);++ const second = await score();++ // The breaker parked Vertex, so no second request left the process.+ expect(fetchSpy).toHaveBeenCalledTimes(1);+ expect(describeImageJson).toHaveBeenCalledTimes(2);+ expect(second.qa_available).toBe(true);+ expect(second.provider).toBe('nvidia');+ });++ it('clears the park as soon as Vertex answers again', async () => {+ fetchSpy.mockResolvedValueOnce(vertexDenied());+ await score();++ await clearProviderCooldown('forge-quality:vertex');+ fetchSpy.mockResolvedValueOnce({+ ok: true,+ status: 200,+ json: async () => ({ candidates: [{ content: { parts: [{ text: JSON.stringify(VERDICT) }] } }] }),+ });++ const verdict = await score();++ expect(verdict.qa_available).toBe(true);+ expect(verdict.provider).toBe('vertex');+ });+});