Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(ledgers): record the home liveness root cause and the retired-model sweep

nirholasef10b756b7d3
diff --git a/prompts/finish/_context/backlog-PROGRESS.md b/prompts/finish/_context/backlog-PROGRESS.md
index 325d7985e..13d469850 100644
--- a/prompts/finish/_context/backlog-PROGRESS.md
+++ b/prompts/finish/_context/backlog-PROGRESS.md
@@ -2388,3 +2388,34 @@ cheapest path for our own objects. `docs/model-diff.md` needed no change, its cl
arbitrary third-party CDNs. `npm run audit:docs` clean across 1,599 files; `docs/ALL.md` regenerated.
**ISSUES.md** item 9 moved to the closed section with the evidence above.
+
+## 2026-09-10 | cross-lane finding | the free LLM chain was full of retired model ids
+
+Found while looking for a translation lane that Vertex's billing hold does not cover. Checked
+every NVIDIA-routed model id in the tree against the live catalog
+(`GET https://integrate.api.nvidia.com/v1/models`). Most were dead, all answering `410 Gone`:
+`meta/llama-3.3-70b-instruct` (EOL 2026-08-26), `nvidia/nv-embedqa-e5-v5` (EOL 2026-08-25),
+`nemotron-3-nano-30b-a3b`, `llama-3.3-nemotron-super-49b-v1.5`, `nvidia-nemotron-nano-9b-v2`
+and `meta/llama-4-maverick-17b-128e-instruct`. Fixed in `4dbe02cc7`.
+
+**Why nobody noticed, and why it still cost something.** A fallback chain cannot distinguish
+`410 Gone` from a rate limit: it moves on. No outage is logged and every surface still answers.
+But each dead id consumed one of a deliberately small number of retry slots, so the chain
+reached its paid backstop, or gave up, sooner and more often than designed. This is the kind of
+rot that only a live-catalog audit finds; reading the code tells you nothing.
+
+The embedding half is the one with state. `NIM_EMBED_TAG` now points at
+`nvidia/nemotron-3-embed-1b` at 2048 dims (measured from a live call, not assumed), and the old
+1024-dim tag stays in the registry as `NIM_EMBED_TAG_RETIRED`, resolvable forever but reporting
+`configured: false` so it is never chosen for new work. Rows written before today keep resolving,
+which is the whole point of the tagging scheme. Vectors are jsonb, not pgvector, so the dimension
+change needed no migration.
+
+`api/_lib/rerank.js` is deliberately left pinned to a model verified dead: the stage is opt-in
+behind `KNOWLEDGE_RERANK_ENABLED`, which is unset on the service, and `rerank()` fails open by
+contract. A verified-dead pin with a comment beats an unverified guess.
+
+**Working lanes as of today:** NVIDIA (free, the one that works), Groq (key valid but capped at
+200k tokens/day, far too small for bulk), OpenAI (`billing_not_active`), Google Vertex AND the
+Generative Language API (both denied project-wide by the same dunning hold, so ISSUES item 1 and
+the fact-check order's row 20 share one cause).
diff --git a/prompts/finish/_context/home-PROGRESS.md b/prompts/finish/_context/home-PROGRESS.md
index 38d86c6da..f63c0116b 100644
--- a/prompts/finish/_context/home-PROGRESS.md
+++ b/prompts/finish/_context/home-PROGRESS.md
@@ -3001,3 +3001,36 @@ run them without hunting.
is closed.
**Commits:** `d063a6f86`, `efe9c3b3a`, `3f97e2078`, `b42d86c51` (the spec change, swept into a
peer's commit), plus this entry.
+
+## 2026-09-10 | order 20 blocking finding 1 | ROOT-CAUSED AND FIXED
+
+The "Live, updated moments ago" over a stopped house is fixed (`a9b725221`). It was not a
+timing or a pool-warmth problem in the way run 4 framed it; the warm pool was the condition
+that EXPOSED it, not the cause.
+
+**Cause.** `home_connections.status` had exactly two writers, `onConnectSuccess` and
+`onConnectFailure` in `api/_lib/home/runtime.js`, and both run only inside the initial
+`entry.ready` promise. A warm `acquire()` never re-enters that path: it finds the pooled entry,
+awaits an already-resolved `ready`, and returns the bridge. The bridge's liveness ping DID
+notice the house was gone and DID emit `disconnected`, and `wireEvents` handled it, but only in
+memory (`entry.stale`, `entry.status`) plus a `notify()` to live subscribers. Nothing wrote the
+row. `/api/home` and `isDegraded` in `src/home/manage.js` both read the row, so the card kept
+reporting `connected` with its old `last_ok_at` and no `last_error_at`, forever.
+
+That is exactly why it passed in isolation and failed in the full tier: a cold pool re-handshakes,
+`onConnectFailure` fires, and the row flips in seconds. It was never flaky.
+
+**Fix.** A `persistStatus(entry, status, detail)` helper writes transitions through from the
+bridge's own lifecycle events, both directions, skipping a write when the stored status already
+matches so a flapping house does not buy a round trip per event. Both drop paths are covered
+because the library's own `disconnected` event and the liveness ping's failure are the same
+`#emit('disconnected')` (`packages/home-bridge/src/bridge.js:174` and `#startLiveness`).
+
+**Tests.** Three in `tests/home-runtime.test.js`, and the first was confirmed to FAIL with the
+fix removed rather than assumed to cover it. The recovery test asserts a NEW write (length + 1)
+rather than the last row's status, because asserting only the status passes even with the
+recovery path missing: `onConnectSuccess` already stored `connected`.
+
+**Left open:** the campaign, still. This clears blocking finding 1 only. Findings 2 (order 17's
+translation) and 3 (orders 17 and 19 open) stand, and order 20 cannot be re-run until they are
+retired. The e2e spec has NOT been re-run against a live container from this session.