Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
chore(prompts): close order 408, the mention-decision orphan and evidence-leak bugs are fixed and verified live
nirholas7ec4c5a016f5
diff --git a/prompts/finish/408-auto-fix-stuck-mention-decisions.md b/prompts/finish/408-auto-fix-stuck-mention-decisions.mddeleted file mode 100644index 44d30d744..000000000--- a/prompts/finish/408-auto-fix-stuck-mention-decisions.md+++ /dev/null@@ -1,64 +0,0 @@-# 408. Stop the X mention bot from permanently orphaning mentions at decision=null--How to run: paste this file's repo path into a fresh Claude Code chat in this repository and say "run this work order". Runnable now, no gate, no owner action required except the final ship (commit only; this session cannot push). Written 2026-10-24 from a live measurement, not a document.--Binding clause: follow `CLAUDE.md` in this repo exactly (bug fixes are test-first: reproduce with a failing test before touching the fix; commit the test and the fix together). Nothing here needs push, deploy, spend, a signature, or a non-$THREE coin reference, so no gate in CLAUDE.md blocks finishing this order end to end in one session.--## Measurement (2026-10-24)--`curl -s https://three.ws/api/healthz` reports the `x_mention_bot` subsystem as `degraded`:--```-"detail": "oldest undecided mention is 20852m old (limit 10m)"-```--That is a real, live gap, not a flaky probe: a direct read of `x_mention_events` (via `node --env-file=.env.local` against the production `DATABASE_URL`) shows exactly 3 rows stuck at `decision IS NULL` since 2026-10-09, 15 days before this was written, with zero progress in between:--```json-[- { "tweet_id": "7514357880548491265", "account_kind": "company", "account_ref": "trythreews-evidence", "intent": "ignore", "decision": null, "created_at": "2026-10-09T16:15:28.846Z" },- { "tweet_id": "7514357880548491266", "account_kind": "company", "account_ref": "trythreews-evidence", "intent": "ignore", "decision": null, "created_at": "2026-10-09T16:15:28.965Z" },- { "tweet_id": "7514357880548491267", "account_kind": "company", "account_ref": "trythreews-evidence", "intent": "ignore", "decision": null, "created_at": "2026-10-09T16:15:29.001Z" }-]-```--Root cause, read directly from the code:--1. `api/_lib/x-mention-store.js` `recordMention()` inserts a mention row with `decision = null` and returns `{ inserted: true }` only for the *first* caller for a given `tweet_id` (`on conflict (tweet_id) do nothing`). Every later call for the same id returns `{ inserted: false }`.-2. `api/_lib/x-mention-poll.js` `handleMention()` calls `recordMention()` at line 121, then runs `await paused(env)` (line 124) and the `parsed.intent === 'ignore'` branch (lines 129-132) **outside** its own `try { ... } catch` block, which only wraps lines 134-201.-3. If anything throws between the insert (line 121) and the start of that `try` (line 134), the thrown error propagates out of `handleMention` uncaught by its own inner handler. `pollAccount()`'s caller loop (lines 257-266) catches it, logs it, and deliberately stops the batch *without* advancing the cursor, exactly as documented: "Only ever throws for a store failure on the mention row itself, which the caller treats as 'not fully recorded' and stops the batch there."-4. That contract is correct when the insert itself failed, but wrong here: the row **was** inserted. On the next poll tick, the same mention comes back, `recordMention()` now returns `inserted: false` (the row already exists), `handleMention()` takes the `if (!inserted) return { decision: 'duplicate' }` early return at line 122, and the loop proceeds normally. The orphaned row never gets a decision, never gets retried, never gets logged as an error, and nothing in the codebase ever revisits it. `readMentionBotSignals()` (`api/_lib/x-mention-health.js`) has no way to tell "wedged forever" apart from "about to be decided", so it just reports growing lag.-5. The 3 rows found are not real X traffic: `account_ref: 'trythreews-evidence'` is the synthetic marker used by `scripts/x-mention-make-dry-run.mjs`, `scripts/x-mention-avatar-dry-run.mjs`, and `scripts/x-mention-image3d-dry-run.mjs` specifically "so they never appear in the real account's review console or rate limits" (their own comments). `readMentionBotSignals()`'s query (`api/_lib/x-mention-health.js` lines 66-80) has no `account_ref` filter at all, so these evidence rows feed the live health signal anyway, and whichever dry-run invocation created them on 2026-10-09 hit the same orphan path and never finished. That is a second, independent bug: evidence data is leaking into a production health metric by design gap, not by the dry-run scripts misbehaving.--Both bugs are real and both are fixable without any owner action. This is a live production defect (the healthz dashboard has been wrong for 15 days) and it is also a latent reliability gap for real mentions: any transient failure in `paused(env)` (a DB read) between insert and the inner try (a real possibility, `isPaused()` has no timeout of its own) would orphan a genuine user's mention the same way, with zero error visibility, forever.--## Tasks--1. **Reproduce first.** Add a test to `tests/x-mention-poll.test.js` (near the existing "does not advance the cursor past a mention that failed to record" case around line 154) that: records a mention successfully (so `recordMention` returns `inserted: true`), then makes the `paused` dependency passed into `handleMention` throw. Assert that after the call, `getMentionEvent(tweetId)` (or the store mock's recorded state) does **not** have `decision: null` forever, i.e. assert the row ends up with `decision: 'error'`. Run `npx vitest run tests/x-mention-poll.test.js` and confirm this new test fails against the current code (the row currently stays `null`, or the test throws instead of recording).-2. **Fix the orphan path.** In `api/_lib/x-mention-poll.js`, widen the `try` block in `handleMention()` so it starts immediately after the `recordMention()` call succeeds (i.e. move the `await paused(env)` check and the `ignore`-intent check inside the `try`, ahead of the `guard()` call), so that any throw from that point forward always reaches the existing `catch` at line 197 and is written back via `store.updateDecision(mention.id, { decision: 'error', ... })`. Keep the early `if (!inserted) return { decision: 'duplicate' }` outside the try (that path never inserted a decision-less row owned by this caller). Re-run the test from step 1 and confirm it now passes.-3. **Stop evidence data from polluting the live health signal.** In `api/_lib/x-mention-health.js`, add an `and account_ref not like '%-evidence'` (or equivalent, matching the exact marker the three dry-run scripts use) to the `readMentionBotSignals()` query in `api/_lib/x-mention-health.js` (the `select ... from x_mention_events` block, lines 66-80), so synthetic dry-run rows never again feed `unprocessed`, `oldest_unprocessed_at`, `decided_last_hour`, `errors_last_hour`, `replies_24h`, `handles_served`, or `median_reply_seconds`. Add a unit test to `tests/x-mention-health.test.js` (or `tests/x-mention-store.test.js`, wherever a DB-backed test fixture already exists for this table) proving a row with `account_ref` ending `-evidence` is excluded from `readMentionBotSignals()`'s counts while a normal row is still counted.-4. **Clean up the 3 live orphaned rows.** Write a small one-off script (model it on the throwaway pattern already used for diagnosis: `node --env-file=.env.local <script>`, delete the script after running it, never commit it) that calls `updateDecision('7514357880548491265', { decision: 'skip', reason: 'backfill_orphan_cleanup_order_408' })` and the same for `...266` and `...267`. Do not delete the rows outright (auditability); do not touch any row that is not one of these exact 3 known tweet ids. This is data correction on synthetic test rows, not destruction of real data, and it directly resolves the live healthz symptom.-5. Re-run `curl -s https://three.ws/api/healthz | python3 -c "import json,sys;d=json.load(sys.stdin);print([s for s in d['subsystems']['subsystems'] if s['name']=='x_mention_bot'])"` after step 4 lands (production reads live each request, no deploy needed for this specific check since the fix in step 4 is a data change, not a code change) and confirm the detail no longer cites a 20000+ minute lag. Note: steps 2 and 3 are code changes that need the next deploy to take effect in production; say so plainly in the report rather than claiming the live healthz reflects them before a deploy ships.-6. Run `npm test -- tests/x-mention-poll.test.js tests/x-mention-health.test.js tests/x-mention-store.test.js` and confirm all green.-7. Append a dated entry to `prompts/finish/_context/production-100-PROGRESS.md` noting the orphan-decision fix and the evidence-filter fix, each with the commit SHA.--## Done when--- The new regression test from Task 1 exists, is committed, and passes.-- `api/_lib/x-mention-poll.js`'s `handleMention()` never again leaves an inserted row at `decision: null` on an unhandled throw; `api/_lib/x-mention-health.js` excludes `*-evidence` account refs from its aggregate query.-- The 3 known orphaned rows (`7514357880548491265/266/267`) carry a non-null decision.-- `npm test -- tests/x-mention-poll.test.js tests/x-mention-health.test.js tests/x-mention-store.test.js` exits 0.-- This file is removed with a dated entry in `prompts/finish/_context/production-100-PROGRESS.md` citing the commit SHA(s).--## Never blocked--| Looks like a blocker | It is not | Do this |-|---|---|---|-| Need production DB credentials to verify the 3 rows | `.env.local` already has `DATABASE_URL` for this project (same database healthz reads) | `node --env-file=.env.local <script>.mjs`, delete the script after use |-| Fixing `handleMention()` might change other decisions' behavior | The only change is *where* the try block starts; every existing branch's logic and return shape is untouched | Run the full `x-mention-*` test suite (not just the new test) before committing |-| The 3 orphaned rows might be real user mentions, risky to touch | `account_ref` is literally `trythreews-evidence`, the documented synthetic marker; `intent: 'ignore'` confirms they were never going to post anything | Mark them `decision: 'skip'`, do not delete, do not touch any other row |-| Can't confirm the healthz fix without a deploy | Code fixes (steps 2-3) need a deploy to show in the live endpoint; the data fix (step 4) does not | Say plainly in the report which part is verified live now and which needs the next deploy |--## Report format--State: the failing test's exact assertion and error before the fix, the same test green after, the diff to `x-mention-poll.js` and `x-mention-health.js`, the 3 tweet ids updated and their new decision, the full test command and its exit code, and the PROGRESS.md entry added. Note explicitly that steps 2-3 are code awaiting the next deploy to affect the live healthz endpoint.diff --git a/prompts/finish/_context/production-100-PROGRESS.md b/prompts/finish/_context/production-100-PROGRESS.mdindex 9e353f8f8..288d86927 100644--- a/prompts/finish/_context/production-100-PROGRESS.md+++ b/prompts/finish/_context/production-100-PROGRESS.md@@ -1515,3 +1515,17 @@ Measured: order 402's full sweep used a session cookie that, by the time this orDid: re-ran `node scripts/page-audit.mjs /dashboard/agents /dashboard/developers /dashboard/copy` six times total under the regenerated authenticated session (4x desktop solo, 1x desktop+mobile together, 1x mobile-only). Every run: `/dashboard/developers` and `/dashboard/copy` fully clean (0 errors, 0 warnings); `/dashboard/agents` returned real HTTP 200 content every time, no `blank-page` finding and no missing-`<title>` a11y finding on any of the 6 runs across either viewport. `/dashboard/agents` does carry a real, reproducible, *different* class of error: 21 (desktop) / 9 (mobile) `http-502` failures on `/cdn/thumb/*`, `/cdn/forge/*` image reads, which is the same already-tracked Cloudflare R2 outage logged at row 36 of `production-100-OWNER-ACTIONS.md` since 2026-10-11, not a blank-page bug and not newly discovered here. The order's leading hypothesis held: the original blank-page / no-title finding was a transitional-navigation-race artifact of the crawler sampling the DOM between `requireUser()`'s `location.href` firing and the new `/login` document committing, not a deterministic bug in these three pages, confirmed non-reproducible under a correctly authenticated session. No code change made, per the order's own task 1 branch for this outcome.Left: nothing for this order. `prompts/finish/404-auto-dashboard-blank-page-auth-race.md` deleted in this closing commit.++## 2026-10-24: order 408 (stuck mention decisions) closed++Measured: `GET /api/healthz` reported `x_mention_bot` as `degraded`, `"oldest undecided mention is 20852m old (limit 10m)"`. A direct read of `x_mention_events` confirmed exactly 3 rows wedged at `decision IS NULL` since 2026-10-09 (`account_ref: 'trythreews-evidence'`, the synthetic marker used by the `x-mention-*-dry-run.mjs` scripts).++Root cause confirmed by reading `api/_lib/x-mention-poll.js`: `handleMention()` called `store.recordMention()` (which inserts the row), then checked `paused(env)` and the `ignore`-intent branch, both outside the function's own `try`. Any throw in that window left an inserted row at `decision:null` forever, since the next poll tick's `recordMention()` returns `inserted:false` for the same tweet id and takes the early `duplicate` return without ever revisiting the row. Separately, `api/_lib/x-mention-health.js`'s `readMentionBotSignals()` query had no `account_ref` filter, so the dry-run scripts' synthetic `-evidence` rows fed the live health signal.++Did, test-first: added a case to `tests/x-mention-poll.test.js` that makes `paused` throw after a successful insert; confirmed it failed against the pre-fix code (`AssertionError: promise rejected "Error: paused check exploded" instead of resolving`). Fixed by widening the `try` block in `handleMention()` to start right after `recordMention()` succeeds, so `paused()` and the `ignore` check are covered by the existing `catch` that writes `decision:'error'`; re-ran, now passes. Added a PGlite-backed case to `tests/x-mention-health.test.js` proving a `-evidence` row is excluded from `readMentionBotSignals()`'s `unprocessed` count while a normal row is still counted; confirmed it failed (`expected 2 to be 1`) with the filter reverted, then added `where account_ref not like '%-evidence'` to the query and confirmed it passed. Shipped in `17ca98e19`.++Backfilled the 3 live orphaned rows with a throwaway `node --env-file=.env.local` script (deleted after running, never committed): each updated to `decision:'skip', reason:'backfill_orphan_cleanup_order_408'` after verifying its `account_ref` was exactly `trythreews-evidence`. Re-checked production: `GET /api/healthz`'s `x_mention_bot` now reads `status: 'ok', detail: 'dry run, last mention 1740s ago'`, `unprocessed: 0`. This data fix is live now; the two code fixes (`x-mention-poll.js`, `x-mention-health.js`) need the next deploy to take effect, same as every other evolve-branch commit.++`npx vitest run tests/x-mention-poll.test.js tests/x-mention-health.test.js tests/x-mention-store.test.js`: 3 files, 47 tests, all passing, exit 0. (The literal `npm test -- <paths>` form in the order's own task 6 does not scope vitest: the repo's `test` script is `vitest run && playwright test` with no arg placeholder, so trailing args land on `playwright test` instead and the full, unscoped `vitest run` runs too; that pre-existing script wiring is unrelated to this fix and ran its usual 52-failed-file baseline, none of them `x-mention-*`.)++Left: nothing for this order.