Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(backlog): supersede the mixed-verdict fact-check note with the publish run

nirholase791d9f982f4
diff --git a/prompts/finish/production-100-04-fact-check-mixed-verdicts.md b/prompts/finish/production-100-04-fact-check-mixed-verdicts.md
deleted file mode 100644
index 062cf5c3b..000000000
--- a/prompts/finish/production-100-04-fact-check-mixed-verdicts.md
+++ /dev/null
@@ -1,92 +0,0 @@
-# P100-04: The `mixed` verdict scores 0/10, and it is the biggest accuracy lever
-
-**How to run this:** paste this whole file into a fresh Claude Code chat opened in
-`/workspaces/three.ws`, or say "execute `prompts/finish/production-100-04-fact-check-mixed-verdicts.md`".
-Read [00-INDEX.md](production-100-00-INDEX.md) and `CLAUDE.md` first.
-
-## Binding operating clause
-
-1. Finish 100%. Never end with a question or an unexecuted plan. Delete this file when the
- definition of done is verified; follow-up files for any remainder first.
-2. **Never publish a number you would not defend.** The benchmark is public trust surface.
- The runner's guards exist because a total LLM outage once nearly published ~25% as the
- product's accuracy with zero errors. Respect the degraded-check guard and the error-rate
- ceiling; never weaken either to make a run publishable.
-3. Hard rules: no mocks, no fixture-fitting hacks, explicit-path commits, no em-dashes.
-
-## The problem, measured 2026-08-02 (re-measure in step 0)
-
-Two published runs of the 40-claim benchmark scored 50% and 55% exact-verdict. The `mixed`
-class went 0/10 both times. In the first run's confusion matrix the chain never emitted
-`mixed` at all (those claims resolved 6 contradicted, 4 insufficient); the second session
-confirmed the endpoint CAN emit `mixed`, so this is verdict calculus behavior, not a
-taxonomy bug. `computeVerdict` returns `mixed` only when neither side clears 70% of
-stance-bearing weight, a band real search evidence rarely lands in. Fixing this class alone
-is worth up to +25 points on the headline number.
-
-## Step 0: re-derive the state
-
-```bash
-curl -s https://three.ws/api/fact-check-benchmark | head -c 1500 # ran? source? per-class table
-grep -rn "computeVerdict\|mixed" api/_lib/fact-check-benchmark.js | head
-grep -rln "computeVerdict" api/ | head # find the real verdict module
-node -e "const f=require('./tests/fixtures/fact-check-benchmark.json');console.log(f.claims?.length ?? Object.keys(f))"
-ls docs/fact-check.md && grep -rn "fact-check-benchmark" vercel.json | head -3
-```
-
-If the live endpoint still answers `ran: false`, production predates the DB-first handler;
-that flips with the A-category ship and does not block this order's code work. If someone
-already lifted `mixed` above 0, re-read the whole per-class table and rewrite this order to
-whatever the weakest class now is (chunk protocol), or retire it if the lever is spent.
-
-## Tasks
-
-1. **Understand before tuning.** Read the verdict path end to end: how stance weights are
- accumulated, what "stance-bearing weight" excludes, and exactly why each of the 10 mixed
- fixtures resolved wrong (run them through the module locally and log per-claim evidence
- weights). Write the finding down in the report; a threshold nudged blind is not a fix.
-2. **Fix the calculus, not the fixture.** Plausible directions the diagnosis should choose
- between: a mixed band defined by BOTH sides holding meaningful support (for example each
- side over a minimum share) rather than neither clearing 70%; counting strong bidirectional
- evidence explicitly; or treating high-variance stances across sources as mixed rather
- than insufficient. Whatever you change must be principled and unit-tested on synthetic
- stance distributions, not shaped to the 10 fixtures.
-3. **Guard against seesaw.** The risk is trading `contradicted`/`supported` accuracy for
- `mixed` recall. Add unit tests pinning clear-cut distributions (unanimous support,
- unanimous contradiction, empty evidence) to their verdicts so the band cannot swallow
- them. Then run the full 40-claim suite locally through the module and compare per-class
- exact-match before/after; every class must hold or improve, `mixed` must leave zero.
-4. **Re-run the real benchmark and publish honestly.** Use the in-process runner path (the
- weekly cron handler logic, Redis verdict cache disabled) so you measure the chain, not
- the cache. Publish via the established `savePublishedRun` path only if the error-rate
- gate passes and the headline number improved; a run that regresses gets diagnosed, not
- published. Verify the benchmark page renders the new run (denominators, per-class and
- per-difficulty tables) in a real browser.
-5. **Docs and changelog.** Update `docs/fact-check.md` if the verdict semantics description
- changed. The benchmark page is public: one `data/changelog.json` entry (tags
- `improvement`) when a better run is published.
-
-## Definition of done
-
-- [ ] Written root-cause finding for the old 0/10 (per-claim, from real runs, in the report).
-- [ ] Verdict change unit-tested on synthetic distributions plus the clear-cut pins; all
- green; `npm run gate` no worse than baseline.
-- [ ] Full-suite per-class table before/after: `mixed` above zero, no class regressed.
-- [ ] A real, non-degraded benchmark run published to the DB with an improved headline
- number, and the page verified in a browser (locally if prod predates the handler).
-- [ ] Changelog entry written; outcome logged in [PROGRESS.md](production-100-PROGRESS.md); this file
- deleted, with a follow-up file if a class remains weak enough to be the next lever.
-
-## Never blocked
-
-| Blocker | Resolution |
-|---|---|
-| LLM lanes exhausted mid-run | Every lane has a failover chain; a degraded check counts as unreachable, never as a verdict. If the whole chain is down, fix the lane first (backlog 06 shipped the map in `docs/ops/llm-lanes.md`). |
-| `INTERNAL_API_KEY` missing locally | It is on the Cloud Run service env and mirrored in `.env` since 2026-08-02; the playbook order in CLAUDE.md finds it. |
-| The 7-day verdict cache poisons an HTTP re-run | Documented behavior: the in-process runner with cache disabled is the authoritative producer. Use it. |
-| Accuracy will not budge past the mixed fix | Publish the honest number anyway if improved, and write the follow-up order naming the next weakest class with its confusion rows. |
-
-## Report format
-
-Root cause in two sentences, the calculus change and why it is principled, before/after
-per-class table, the published run's headline and where it is visible, and the next lever.
diff --git a/prompts/finish/production-100-04b-fact-check-publish-run.md b/prompts/finish/production-100-04b-fact-check-publish-run.md
index 30d929d48..5e5c840ce 100644
--- a/prompts/finish/production-100-04b-fact-check-publish-run.md
+++ b/prompts/finish/production-100-04b-fact-check-publish-run.md
@@ -61,6 +61,15 @@ If `/api/fact-check-benchmark` already reports a run generated after the fix lan
aerial-vehicle-466722-p5 --format=yaml`). Search needs no key: Wikipedia and DuckDuckGo
are keyless rungs, though a `BRAVE_API_KEY` / `TAVILY_API_KEY` materially improves the
evidence the verdict sees.
+
+ **Do not plan on the keyless LLM floor.** It exists and it is real, but it is two rungs,
+ not three, and from this workspace's shared egress IP it answered 0 of 8 probes on
+ 2026-09-02: OVH 429 and Pollinations 429 on every attempt across 64 seconds, with one
+ isolated success after a long idle. LLM7 was the third keyless rung and is not one any
+ more (llm7.io retired its anonymous tier; every unauthenticated call is 401
+ `invalid_api_key`, the `unused` token its docs used to accept included), so it is gated
+ on `LLM7_API_KEY` now. A 40-claim run is 80 LLM turns; the keyless floor will not carry
+ it. Get a real key.
2. **Check the production chain is not itself degraded before spending a run.** On
2026-09-02 a live free-lane check returned
`query generation unavailable: openai 429 billing_not_active`, meaning every free rung
@@ -76,6 +85,17 @@ If `/api/fact-check-benchmark` already reports a run generated after the fix lan
`by_class` against the last published run (`mixed` must leave zero, no class may
regress) and read `/tmp/fc-detail.json` for the per-source stances behind any class that
moved the wrong way.
+
+ **Fidelity trap, and it decides whether step 4 is honest.** The published 2026-08-10 run
+ was produced by production, whose search chain leads with Vertex-grounded Google Search.
+ A local run without `GOOGLE_CLOUD_PROJECT` and ADC (or a Brave/Tavily/Exa/Serper key)
+ silently falls to the keyless rungs, and the evidence is visibly worse: probing the
+ `mixed` claims here on 2026-09-02 returned five Wikipedia pages per claim, among them
+ "Aunty Donna's Coffee Cafe" for "Coffee is bad for your health" and "Zootopia" for the
+ tongue map. That tier is fine for an A/B of the CALCULUS against itself, where both arms
+ see the same evidence. It is not the product's accuracy, and publishing it as the
+ headline number would understate a chain the public number is supposed to describe.
+ Before `--publish`, confirm the run actually used the grounded search rung.
4. **Publish only an improved, non-degraded run:** re-run with `--publish`. The live
`/api/fact-check-benchmark` reads the DB row first, so this reaches the public page
without a deploy, but the CHAIN itself only improves once the code is deployed, so
@@ -98,7 +118,8 @@ If `/api/fact-check-benchmark` already reports a run generated after the fix lan
| Blocker | Resolution |
|---|---|
-| No LLM key anywhere on the machine | Restore `gcloud` auth for the Vertex anchor, then read the service env. If a key genuinely exists nowhere, that is one line for OWNER-ACTIONS naming the single missing variable, not a reason to leave the rest undone. |
+| No LLM key anywhere on the machine | Restore `gcloud` auth for the Vertex anchor, then read the service env. If a key genuinely exists nowhere, that is one line for OWNER-ACTIONS naming the single missing variable, not a reason to leave the rest undone. Do not spend the session retrying the keyless rungs: they were 0/8 from this egress IP on 2026-09-02 (task 1). |
+| Tempted to publish a run measured on the keyless search tier | Do not. It measures a chain the public number does not describe (task 3). Report the A/B, leave the published run alone, and say what is missing. |
| The deploy is owner-gated | Prepare it so the ship is one command (`npm run deploy:gcp:full` after `npm run prep:worktree -- --apply`) and say so. Publishing the run does NOT need the deploy, but running it against the OLD chain measures the old code, so do not publish that as the fix's number. |
| Accuracy improves on `mixed` but a neighbour regresses | That is the seesaw the calculus was designed against. Read `/tmp/fc-detail.json` for the regressed claims' stances: a `partial` where the source plainly affirms or plainly refutes is a rubric problem, not a calculus one. |
| The run is refused as degraded | Working as designed. Fix the lane, then re-run. Never lower `MAX_ERROR_RATE`. |
diff --git a/prompts/finish/production-100-PROGRESS.md b/prompts/finish/production-100-PROGRESS.md
index 90a8846c9..c87956097 100644
--- a/prompts/finish/production-100-PROGRESS.md
+++ b/prompts/finish/production-100-PROGRESS.md
@@ -163,3 +163,112 @@ credit the customers (recommended), contact them first, or write the balance off
the second customer wallet needs one keyed `node scripts/audit-custodial-key-health.mjs
--json` run on a machine with `WALLET_ENCRYPTION_KEY` (row 15); the decision itself does
not wait on it. Order file deleted.
+
+## 2026-09-02: the `mixed` class was unreachable by construction, not by threshold
+
+P100-04 assumed the `mixed` verdict needed a threshold nudge. It did not. The verdict
+taxonomy has four classes and the stance vocabulary had three (`supports`, `contradicts`,
+`neutral`), and the stance rubric told the model outright that when content "affirms one
+[assertion] while refuting another", it must "choose the stance for the assertion the
+content speaks to most directly". Every mixed fixture is a partial truth ("Napoleon was
+unusually short", "carrots give significantly better night vision") whose sources AGREE
+with each other that the claim is half right. Collapsed onto one side, they produce a
+near-unanimous `contradicts` distribution, which `computeVerdict` read correctly as
+`contradicted`. Both published runs show it: mixed to contradicted, 7 of 10, twice. The
+`mixed` branch was defined as inter-source DISAGREEMENT while the class it had to predict
+is intra-source QUALIFICATION, so no threshold reaches it.
+
+Fixed by adding `partial` as a fourth stance and making `computeVerdict` count its weight
+as stance-bearing evidence that takes neither side. With zero `partial` sources the
+function's output is identical to the old one, which is the anti-seesaw guarantee, pinned
+by `tests/api/fact-check-verdict.test.js`.
+
+Found a second defect while measuring, and it is the one that matters for trust in the
+number. Two live free-lane checks on 2026-09-02 came back `insufficient` with every stance
+`neutral`, every excerpt the raw search snippet, real LLM tokens spent, and nothing marked
+degraded: "A tomato is a vegetable." resolved as unengaged evidence while holding
+*Nix v. Hedden*, the Supreme Court case that settles it. Both LLM stages extracted their
+JSON with a NON-GREEDY `/\[[\s\S]*?\]/`, so they stopped at the first `]` in the response.
+A reasoning block, a code fence, or a `[1]` citation truncated the match, the stage fell
+back to all-neutral, and it said nothing. That fabricated verdict was scored by the
+accuracy benchmark as real, was invisible to the degraded-run guard that exists precisely
+to catch this, and was written into the 7-day cache. The same fault collapsed the three
+search angles to one on every check. `extractJsonArray` now scans for a balanced,
+string-literal-aware array of the shape the caller asked for, and an unreadable answer
+reports `stance extraction unreadable` / `query generation unreadable` instead of a silent
+`insufficient`.
+
+Did: the stance vocabulary, the verdict weighting, the response reader, the degradation
+contract, `docs/fact-check.md`, `agents/fact-checker/README.md`, and two changelog
+entries. 70 tests green across the four fact-check suites, `check:rules` clean on the
+touched paths. The full `npx vitest run` is 26,965 passing with 17 failures in 13 files,
+none of them fact-check (3d-studio, branding, cron-scheduler-sync, x402-discovery-parity,
+glb-quality, oracle-calibrate-cron, rate-limit-buckets, asset-host-liveness,
+deploy-artifacts, no-nul-bytes), all in code other sessions were editing at the time.
+
+Left: nothing has MEASURED the fix, so there is no before/after table and no published
+run. The in-process runner needs an LLM lane and this machine has none: no provider key in
+`.env` or `.env.local`, and `gcloud` refuses every call with "Reauthentication failed.
+cannot prompt during non-interactive execution", which also takes out the Vertex Gemini
+anchor. Production's own chain is currently falling through to a dead paid backstop
+(`openai 429: billing_not_active` on a live check), so a run today would be refused by the
+error-rate ceiling anyway, correctly. `prompts/finish/production-100-04b-fact-check-publish-run.md`
+carries the remainder with the exact commands. Order file 04 deleted.
+
+## 2026-09-02: 04 mixed verdicts (calculus fix, lane fix, and what could not be measured)
+
+Measured first: the live endpoint still served the 2026-08-10 run, `source: database`,
+40% overall with `mixed` 0/10 and its confusion row reading `contradicted` 7,
+`insufficient` 2, `supported` 1.
+
+Root cause, and it is not a threshold. The ten `mixed` fixtures are all partial truths
+("a tomato is a vegetable", "Napoleon was unusually short", the tongue map), and on a
+partial truth the sources do not disagree with each other: every one reads the same
+nuance. The stance vocabulary was `supports | contradicts | neutral`, and the extraction
+rubric explicitly told the model to pick one side when a source affirmed part of a claim
+and refuted another. So every source projected the same way, the projection cleared the
+70% dominance bar, and the claim came back flat. `computeVerdict`'s `mixed` branch was
+reachable only from inter-source disagreement, which that evidence never produces. The
+class was unreachable by construction.
+
+Fix: a fourth stance, `partial`, for a source that engages the claim and finds it true in
+one respect and wrong, overstated, or only conditionally true in another. It is
+stance-bearing but takes neither side, so it dilutes dominance and pushes the result to
+`mixed`; the rubric is deliberately narrow (hedged prose, thin coverage and extractor
+uncertainty stay `neutral`). Mixed confidence was rewritten too: it was
+`max(supportRatio, contraRatio)`, which grew as a split became more lopsided. It is now a
+mixedness score, `partialRatio + 2 * min(supportRatio, contraRatio)`, clamped to 1.
+Wired through the extraction rubric, the image-evidence lane, the x402 response schema,
+the source pill on `/fact-checker`, `docs/fact-check.md` and the agent README.
+
+Anti-seesaw, proven mechanically rather than statistically: `tests/api/fact-check-verdict.test.js`
+runs 5000 seeded random distributions drawn only from the three legacy stances and asserts
+the new `computeVerdict` returns exactly what a transcription of the old rule returns. So
+no clear-cut claim can be pulled into the mixed band by the calculus; the only claims whose
+verdict can move are the ones a source actually reports as half-true. 23 tests green,
+alongside the clear-cut pins (unanimous, boundary-at-70%, empty, all-neutral, zero-weight,
+lone-source coverage floor).
+
+Could not be measured here, which is why 04b exists. A 40-claim before/after needs an LLM
+lane and this box has none: no provider key in `.env` or `.env.local`, `gcloud` auth dead,
+and the keyless floor answered 0 of 8 probes (OVH 429, Pollinations 429 on every attempt).
+A baseline worktree at `1407949cb` was staged for the A/B and torn down unused. Publishing
+was refused on top of that for a second reason worth keeping: without ADC the search chain
+falls to Wikipedia and DuckDuckGo, which returned "Aunty Donna's Coffee Cafe" for "coffee
+is bad for your health", so a local run would have understated a chain the public number
+is meant to describe. Both facts are now written into 04b with the trap named.
+
+Side fix, from diagnosing that floor: LLM7.io retired the anonymous tier its rung was added
+on, so every unauthenticated call is a 401 `invalid_api_key` (the `unused` token its docs
+used to accept included). The rung sat in the chain unconditionally, spending a guaranteed
+round trip at the tail of an already-exhausted chain. It is gated on `LLM7_API_KEY` now,
+and `docs/ops/llm-lanes.md`, `docs/free-llm-providers.md` and `.env.example` no longer
+claim three keyless rungs when there are two.
+
+Left: 04b, unchanged in scope. It needs one LLM key and, for a publishable number, the
+grounded search rung.
+
+Worth knowing for the next agent: this ran alongside another agent on the same order.
+Everything here reached `main` swept into that agent's commits (the shared worktree does
+`git add -A`), so the content landed even though almost none of it carries a commit of
+mine. Four commit attempts lost the `HEAD` ref race outright.