Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(vision): make parseable JSON part of the failover test, not a filter after it

nirholase2074bd5821d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aeeeb2a4e..40e25a0e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,10 @@
Public history for [three.ws](https://three.ws), newest first. New pages come from `added` dates in data/pages.json; everything else is curated in data/changelog.json. Also available as [JSON](https://three.ws/changelog.json) and [RSS](https://three.ws/changelog.xml), live at [three.ws/changelog](https://three.ws/changelog).
+## 2026-09-10
+
+- **The image check now moves on when a model answers in the wrong format**: Generated models are inspected by a vision model before you see them, and that inspection asks several providers in turn so one being busy is never the end of it. The catch was that a provider only had to reply, not reply usefully: a model that narrated its thinking instead of returning the requested format counted as the answer, and the healthy providers waiting behind it were never asked. A reply the check cannot read is now treated as that provider failing, so it moves straight to the next one, and it only gives up once every provider has been tried. The practical effect is that the quality check returns a real verdict far more often, and when it cannot, the reason it reports names the provider that actually lost. `[fix, improvement]`
+
## 2026-09-09
- **Docs · Fade Radar** (`/docs/fade-radar`): How the Fade Radar scores a coin's buy side: what makes a wallet a reverse indicator, the 0 to 100 fade score, the verdict bands, and the out-of-sample measurement behind them.
diff --git a/api/_lib/vision.js b/api/_lib/vision.js
index ac31723af..66706dd09 100644
--- a/api/_lib/vision.js
+++ b/api/_lib/vision.js
@@ -413,6 +413,7 @@ export async function describeImage({
timeoutMs = 20_000,
deadlineMs = null,
track = null,
+ accept = null,
}) {
const chain = visionChain();
if (!chain.length) throw new VisionUnavailableError();
@@ -549,30 +550,67 @@ export async function describeImage({
}
const data = await upstream.json();
const usage = p.extractUsage(data);
+ // Meter it either way: the tokens were spent whether or not the reply is
+ // usable, and a ledger that hides rejected replies understates the cost of
+ // a lane that answers fluently in the wrong shape.
recordVisionSpend(p, usage, Date.now() - startedAt, track);
+ const text = (p.extractText(data) || '').trim();
+ // A 200 is not the same as an answer. When the caller requires a shape
+ // (describeImageJson requires parseable JSON), validating it HERE makes the
+ // shape part of the chain's success test instead of a filter bolted on
+ // after a winner is picked. That distinction is the whole bug: with the
+ // check outside the loop, the first lane to emit prose ended the request
+ // and every healthy rung behind it went untried, which is exactly how a
+ // reasoning model that narrates before it answers took the forge quality
+ // gate from intermittent to 0 verdicts in 10 (production, 2026-09-10).
+ //
+ // A rejected lane is NOT cooled. The cooldown key is shared with callers
+ // that want free-form prose, and that reply is perfectly good to them;
+ // benching it here would degrade alt-text to punish a JSON caller.
+ let accepted;
+ if (accept) {
+ try {
+ accepted = accept(text);
+ } catch (e) {
+ attempts.push({
+ provider: p.name,
+ model: p.model || null,
+ status: upstream.status,
+ detail: `reply rejected: ${String(e?.message || e).slice(0, 120)}`,
+ });
+ lastErr = Object.assign(
+ new Error(`${p.name} vision reply rejected: ${e?.message || e}`),
+ { status: 502, code: e?.code || 'vision_bad_reply' },
+ );
+ continue;
+ }
+ }
// A lane that just served a real request is healthy whatever an earlier
// window recorded; waiting out the rest of a disproved cooldown only keeps
// a recovered lane off the menu.
void clearProviderCooldown(laneKey(p));
return {
- text: (p.extractText(data) || '').trim(),
+ text,
provider: p.name,
model: p.model,
usage,
raw: data,
+ accepted,
};
}
throw Object.assign(lastErr || new VisionUnavailableError(), { lanes: attempts });
}
-// Convenience: describeImage + tolerant JSON parse of the reply. VLMs reliably
-// honor "reply ONLY JSON" (probes/vision.md) but may wrap it in a ```json fence
-// or a trailing newline; this strips both. Returns the parsed object plus the
-// provider metadata, or throws if the model returned unparseable text (the
-// caller's degraded path handles that exactly like a vision outage).
+// Convenience: describeImage with "parseable JSON" as the acceptance test. VLMs
+// mostly honor "reply ONLY JSON" (probes/vision.md) but may wrap it in a ```json
+// fence or a trailing newline, and a reasoning model may narrate around it;
+// parseJsonLoose strips what it can. A reply it cannot parse fails THAT LANE and
+// the chain moves to the next rung, so one chatty model no longer costs the
+// request every healthy provider behind it. Throws only when no rung produced
+// parseable JSON (the caller's degraded path handles that like a vision outage).
export async function describeImageJson(opts) {
- const result = await describeImage(opts);
- return { ...result, json: parseJsonLoose(result.text) };
+ const result = await describeImage({ ...opts, accept: parseJsonLoose });
+ return { ...result, json: result.accepted };
}
// Strip a ```json fence / stray prose and parse the first JSON object/array in
diff --git a/data/changelog.json b/data/changelog.json
index d924dcc26..0800e4c6b 100644
--- a/data/changelog.json
+++ b/data/changelog.json
@@ -28006,6 +28006,15 @@
"fix",
"improvement"
]
+ },
+ {
+ "date": "2026-09-10",
+ "title": "The image check now moves on when a model answers in the wrong format",
+ "summary": "Generated models are inspected by a vision model before you see them, and that inspection asks several providers in turn so one being busy is never the end of it. The catch was that a provider only had to reply, not reply usefully: a model that narrated its thinking instead of returning the requested format counted as the answer, and the healthy providers waiting behind it were never asked. A reply the check cannot read is now treated as that provider failing, so it moves straight to the next one, and it only gives up once every provider has been tried. The practical effect is that the quality check returns a real verdict far more often, and when it cannot, the reason it reports names the provider that actually lost.",
+ "tags": [
+ "fix",
+ "improvement"
+ ]
}
]
}
diff --git a/public/changelog-recent.json b/public/changelog-recent.json
index 8589c17c8..44b4dea1a 100644
--- a/public/changelog-recent.json
+++ b/public/changelog-recent.json
@@ -1,14 +1,25 @@
{
- "generated_at": "2026-09-09T22:09:39.893Z",
+ "generated_at": "2026-09-10T04:03:58.066Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
"url": "https://three.ws"
},
- "total": 3657,
+ "total": 3658,
"complete": false,
"full_feed": "/changelog.json",
"entries": [
+ {
+ "date": "2026-09-10",
+ "type": "update",
+ "title": "The image check now moves on when a model answers in the wrong format",
+ "summary": "Generated models are inspected by a vision model before you see them, and that inspection asks several providers in turn so one being busy is never the end of it. The catch was that a provider only had to reply, not reply usefully: a model that narrated its thinking instead of returning the requested format counted as the answer, and the healthy providers waiting behind it were never asked. A reply the check cannot read is now treated as that provider failing, so it moves straight to the next one, and it only gives up once every provider has been tried. The practical effect is that the quality check returns a real verdict far more often, and when it cannot, the reason it reports names the provider that actually lost.",
+ "link": null,
+ "tags": [
+ "fix",
+ "improvement"
+ ]
+ },
{
"date": "2026-09-09",
"type": "launch",
@@ -1602,16 +1613,6 @@
"tags": [
"launch"
]
- },
- {
- "date": "2026-09-05",
- "type": "launch",
- "title": "Pocket Console",
- "summary": "A handheld console with a live 3D agent inside the screen. Steer it with the D-pad, swap cartridges for any public agent, and embed the whole console anywhere with one line of HTML.",
- "link": "/pocket",
- "tags": [
- "launch"
- ]
}
]
}
diff --git a/public/changelog.json b/public/changelog.json
index ce34ce86e..cc25070bc 100644
--- a/public/changelog.json
+++ b/public/changelog.json
@@ -1,11 +1,22 @@
{
- "generated_at": "2026-09-09T22:09:39.871Z",
+ "generated_at": "2026-09-10T04:03:58.046Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
"url": "https://three.ws"
},
"entries": [
+ {
+ "date": "2026-09-10",
+ "type": "update",
+ "title": "The image check now moves on when a model answers in the wrong format",
+ "summary": "Generated models are inspected by a vision model before you see them, and that inspection asks several providers in turn so one being busy is never the end of it. The catch was that a provider only had to reply, not reply usefully: a model that narrated its thinking instead of returning the requested format counted as the answer, and the healthy providers waiting behind it were never asked. A reply the check cannot read is now treated as that provider failing, so it moves straight to the next one, and it only gives up once every provider has been tried. The practical effect is that the quality check returns a real verdict far more often, and when it cannot, the reason it reports names the provider that actually lost.",
+ "link": null,
+ "tags": [
+ "fix",
+ "improvement"
+ ]
+ },
{
"date": "2026-09-09",
"type": "launch",
diff --git a/public/changelog.xml b/public/changelog.xml
index 384bf1051..defd0cdde 100644
--- a/public/changelog.xml
+++ b/public/changelog.xml
@@ -6,6 +6,14 @@
<atom:link href="https://three.ws/changelog.xml" rel="self" type="application/rss+xml"/>
<description>What's new on three.ws: features, improvements, and releases, newest first.</description>
<language>en</language>
+ <item>
+ <title>The image check now moves on when a model answers in the wrong format</title>
+ <link>https://three.ws/changelog</link>
+ <guid isPermaLink="false">2026-09-10:The image check now moves on when a model answers in the wrong format</guid>
+ <pubDate>Thu, 10 Sep 2026 12:00:00 GMT</pubDate>
+ <category>fix</category>
+ <description>Generated models are inspected by a vision model before you see them, and that inspection asks several providers in turn so one being busy is never the end of it. The catch was that a provider only had to reply, not reply usefully: a model that narrated its thinking instead of returning the requested format counted as the answer, and the healthy providers waiting behind it were never asked. A reply the check cannot read is now treated as that provider failing, so it moves straight to the next one, and it only gives up once every provider has been tried. The practical effect is that the quality check returns a real verdict far more often, and when it cannot, the reason it reports names the provider that actually lost.</description>
+ </item>
<item>
<title>Docs · Fade Radar</title>
<link>https://three.ws/docs/fade-radar</link>
diff --git a/public/features.json b/public/features.json
index bcfd0da69..31e7a2461 100644
--- a/public/features.json
+++ b/public/features.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-09-09T22:09:39.860Z",
+ "generated_at": "2026-09-10T04:03:58.034Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json",
"site": {
"name": "three.ws",
diff --git a/tests/vision-json-lane-failover.test.js b/tests/vision-json-lane-failover.test.js
new file mode 100644
index 000000000..04389bc8c
--- /dev/null
+++ b/tests/vision-json-lane-failover.test.js
@@ -0,0 +1,93 @@
+// A 200 is not an answer: a lane whose reply is not parseable JSON must fail
+// THAT LANE, not the request.
+//
+// describeImageJson used to run parseJsonLoose outside the failover loop, so
+// describeImage returned the first HTTP-200 reply and the parse happened after a
+// winner had already been picked. One chatty model therefore cost the request
+// every healthy provider behind it. Measured on production 2026-09-10: the forge
+// quality gate returned 0 verdicts in 10 with "vision reply was not valid JSON",
+// while a rung that answers clean JSON sat untried in the same chain.
+import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from 'vitest';
+
+const VERDICT = { score: 71, realism: 60, completeness: 80, defects: [], reason: 'ok' };
+
+let describeImageJson;
+const fetchSpy = vi.fn();
+const realFetch = globalThis.fetch;
+const PNG_1PX =
+ 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==';
+
+// Two free lanes on one host, so the chain has a rung to fall through to.
+beforeAll(async () => {
+ process.env.NODE_ENV = 'development';
+ process.env.NVIDIA_API_KEY = 'test-nvidia-key';
+ process.env.OPENROUTER_API_KEY = 'test-openrouter-key';
+ delete process.env.OPENAI_API_KEY;
+ delete process.env.GOOGLE_CLOUD_PROJECT;
+ globalThis.fetch = fetchSpy;
+ ({ describeImageJson } = await import('../api/_lib/vision.js'));
+});
+
+afterAll(() => {
+ globalThis.fetch = realFetch;
+ delete process.env.NVIDIA_API_KEY;
+ delete process.env.OPENROUTER_API_KEY;
+});
+
+beforeEach(() => {
+ fetchSpy.mockReset();
+});
+
+function reply(content) {
+ return {
+ ok: true,
+ status: 200,
+ json: async () => ({
+ choices: [{ message: { content } }],
+ usage: { prompt_tokens: 10, completion_tokens: 10 },
+ }),
+ };
+}
+
+function call() {
+ return describeImageJson({
+ prompt: 'score this',
+ imageBase64: PNG_1PX,
+ mimeType: 'image/png',
+ timeoutMs: 5000,
+ deadlineMs: 9000,
+ });
+}
+
+describe('describeImageJson lane acceptance', () => {
+ it('parses a fenced JSON reply', async () => {
+ fetchSpy.mockResolvedValueOnce(reply('```json\n' + JSON.stringify(VERDICT) + '\n```'));
+ const out = await call();
+ expect(out.json).toEqual(VERDICT);
+ });
+
+ it('falls through to the next rung when a lane narrates instead of answering', async () => {
+ // A reasoning model narrating around braces: the outermost-span fallback
+ // grabs from the first brace in the thinking block and cannot parse. This
+ // is the regression that took the forge quality gate to 0/10 in prod.
+ const chatty = '<think>{the user wants a score} I will now decide</think> the score is high';
+ fetchSpy
+ .mockResolvedValueOnce(reply(chatty))
+ .mockResolvedValueOnce(reply(JSON.stringify(VERDICT)));
+
+ const out = await call();
+
+ // The chatty lane lost, the next rung answered, and the caller got JSON.
+ expect(fetchSpy).toHaveBeenCalledTimes(2);
+ expect(out.json).toEqual(VERDICT);
+ });
+
+ it('throws only once every rung has failed the shape test', async () => {
+ fetchSpy.mockResolvedValue(reply('not json at all'));
+
+ await expect(call()).rejects.toThrow();
+
+ // Every lane in the chain was tried, not just the first.
+ expect(fetchSpy.mock.calls.length).toBeGreaterThan(1);
+ });
+});