Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(backlog): correct item 09, whose feed was ready and carrying nothing
nirholas7228c98f18d3
diff --git a/prompts/finish/908-backlog-09-telegram-bots-durability.md b/prompts/finish/908-backlog-09-telegram-bots-durability.mdindex 5dec1a3f0..545def99e 100644--- a/prompts/finish/908-backlog-09-telegram-bots-durability.md+++ b/prompts/finish/908-backlog-09-telegram-bots-durability.md@@ -7,47 +7,92 @@ Read [00-INDEX.md](_context/backlog-00-INDEX.md) first.> in three.ws without owner approval. Build and deploy freely; ask before> committing anything that names the launchpad into this repo.-## Status: done, verified live 2026-09-02+## Status: both feeds live and hardened, re-verified 2026-09-09-Both feeds now run on Cloud Run and no longer depend on this codespace. The-sibling repo is checked out at `/workspaces/pump-fun-sdk`, so the deliverables-below are verifiable from here, which they were not when this order was written.+The 2026-09-02 pass declared this done. It was half true. Re-measuring on+2026-09-09 found the graduation tracker healthy and **the all-claims firehose+silently dead**: 99 hours of uptime, `mode: websocket`, and zero events.-| Feed | Cloud Run service | Revision | Uptime | Transport |+Root cause: `rpc.magicblock.app` went key-gated. It now answers `401` on the+WebSocket upgrade and `{"error":"invalid api key"}` over HTTP, and it was the+firehose's only WebSocket endpoint *and* its primary RPC. The 2026-08-01 note+below calling it a working free endpoint is what stopped being true.++Why it stayed invisible for four days is the part worth keeping: **subscribing+cannot fail loudly.** web3.js returns a subscription id immediately and retries+the socket internally, so `startWebSocket()` resolved, `transport` was set to+`websocket`, and `/stats` reported a healthy feed. The 90-second silence+heartbeat did fire, but its reconnect rebuilt the same dead endpoint, so the+watchdog looped instead of escaping. Nothing in the metrics distinguished a+connected socket from a refused one.++Both halves are fixed and deployed.++| Feed | Cloud Run service | Revision | Transport | Measured 2026-09-09 ||---|---|---|---|---|-| Graduation and migration tracker | `pumpfun-channel-bot` | `-00004-tvk` | 25.4 h | websocket |-| All-claims firehose | `pumpfun-allclaims-bot` | `-00008-6d8` | 25.7 h | websocket |--Both are `Ready=True`, pinned singletons (`minScale=1`, `maxScale=1`) on the-`three-ws@` runtime SA. The codespace was rebuilt on 2026-09-02 and both local-processes on ports 3900/3901 died with it while the two services kept running:-that rebuild is the proof of durability this order asked for, not a regression.--Measured the same day: the tracker had posted 1,719 messages over 23,456 events-(`degraded: false`, delivery `ok`); the firehose had detected 15,566 claims and-posted 754 digests over 14,862 events, on `rpc.magicblock.app/mainnet`, with 2-post failures. Channel ids are distinct and numeric (`-1003965305979` and-`-1003905427189`), each with its own bot token in its own Secret Manager secret.--One open observation, not a failure: the firehose reports 531 queue drops-(roughly 6.5% of claim transactions never fetched). That is the RPC queue's-designed backpressure rather than a fault, and the levers are queue capacity and-endpoint count. The 2026-08-02 baseline of zero drops was a 3h43m measurement,-not a contract.--### What was left to build--Every numbered item below had already been completed by the 2026-08-01/02-sessions. The one real gap this pass found and closed was configuration-durability: `.env` is gitignored, so the rebuild destroyed the only copy of each-bot's working config, and nothing could read it back. That stranded a bot twice-over, since `deploy-cloudrun.sh` also reads its whole configuration from `.env`.-Each bot directory now has a `recover-env.sh` that rebuilds `.env` from the live-revision plus Secret Manager, applying the same numeric-`CHANNEL_ID` guard the-deploy applies. It is committed in the sibling repo (`9701edb9`) and unpushed.+| Graduation and migration tracker | `pumpfun-channel-bot` | `-00006-wp4` | websocket on `solana-rpc.publicnode.com` | 156 events in 3m17s, `degraded: false`, delivery `ok` |+| All-claims firehose | `pumpfun-allclaims-bot` | `-00010-bdc` | websocket on `solana-rpc.publicnode.com` | 745 claims and 242,207 WS events in 23m, 145 instant posts, 23 digests, **0 post failures** |++Both `Ready=True`, pinned singletons (`minScale=1`, `maxScale=1`) on the+`three-ws@` runtime SA, each with its own bot token in its own Secret Manager+secret and its own numeric channel id (`-1003965305979`, `-1003905427189`).++The firehose recovered the moment its endpoints were repointed: 62 claims in the+first two minutes, against the ~20 claims/min baseline this order set.++### The config fix (immediate)++Both services were repointed off magicblock onto endpoints verified keyless and+streaming the same day: `solana-rpc.publicnode.com`, `api.mainnet-beta.solana.com`+(HTTP and WS), and `solana.leorpc.com/?api_key=FREE` (HTTP only; its WS refuses).+`drpc.org` and `rpc.ankr.com` are no longer keyless either and were rejected.++A trap this exposed: `deploy-cloudrun.sh` ships env from `.env` via+`--env-vars-file`, which **replaces the whole set**. The local `.env` still held+the magicblock config, so the next deploy would have silently re-broken the feed.+`recover-env.sh` now regenerates it from the corrected live revision, and both+`.env` files carry the new endpoint lists.++### The code fix (so this class of outage cannot hide again)++Landed in the sibling repo at `2a12ec3a`, canonical copy first and propagated to+`channel-bot`, per the rule in its own `DECODERS.md`:++- **A WebSocket endpoint list, not one endpoint.** `SOLANA_WS_URLS` is+ comma-separated, and whatever the RPC list implies is appended, so there is+ always somewhere to fail over to. The firehose now runs with 5 resolved.+- **Liveness is proven by traffic, not by a subscription id.** A new+ subscription must deliver a real log event within 20 seconds or the endpoint is+ rejected and the next one tried. The monitored programs emit thousands of+ events a minute, so silence that long means the endpoint, not the chain.+- **Reconnects rotate** instead of retrying the endpoint that just went quiet.+- **The heartbeat timer is cleared before it is re-armed.** Every reconnect since+ this code was written had been stacking another interval.+- **`/stats` exposes `activeWs` and `wsEventsReceived`,** so a WebSocket carrying+ no traffic is visible rather than indistinguishable from a healthy one.+- `channel-bot`'s live feed runs on `event-monitor.ts`, not the claim monitor, so+ the rotation was applied there too rather than only to the shared decoder path.++Verified by running the monitor against the dead endpoint first: it rejected+magicblock, settled on `solana-rpc.publicnode.com`, and reported 9,773 events.+Covered by `src/__tests__/ws-urls.test.ts` in both bots (7 cases, including the+regression that a single explicit endpoint must never de-duplicate down to a+single point of failure). Suites green: 95 tests allclaims, 205 channel-bot.++Deliberately not done: neither bot has a persistent volume, so tracker state is+ephemeral and a redeploy resets it. That is pre-existing, was true before this+pass, and mounting one is a separate change.++### Standing observation, not a failure++`queueDrops` sits at 53 on the firehose (~0.9% of pipeline claims). That is the+RPC queue's designed backpressure, and the levers are queue capacity and endpoint+count. The 2026-08-02 baseline of zero drops was a 3h43m measurement, not a+contract.## The work+1. **Deploy both to Cloud Run.** `channel-bot/deploy-cloudrun.sh` exists and is thetemplate. Write the equivalent for the all-claims package. Pin the`three-ws-build@` and `three-ws@` service accounts.@@ -65,13 +110,18 @@ deploy applies. It is committed in the sibling repo (`9701edb9`) and unpushed.id from `getUpdates?allowed_updates=["my_chat_member"]`, whose promotion eventcarries the chat id, username, and full admin rights object.-4. **Keep websockets, do not fall back to polling.** `getSignaturesForAddress`- limit-20 per 30s against the program samples a handful of valid transactions- per minute out of thousands, so events get missed entirely. The code only- enables WS when `SOLANA_WS_URL` is explicitly set. `wss://rpc.magicblock.app/mainnet`- carries the full firehose on the free tier (measured 4,545 events per 20s) and- `wss://solana-rpc.publicnode.com` also works keyless. The three.ws Helius key is- plan-exhausted and its WS handshakes 429 forever, so do not point these at it.+4. **Keep websockets, do not fall back to polling, and configure more than one.**+ `getSignaturesForAddress` limit-20 per 30s against the program samples a+ handful of valid transactions per minute out of thousands, so events get+ missed entirely. WS is enabled when `SOLANA_WS_URL` or `SOLANA_WS_URLS` is+ set. **`wss://rpc.magicblock.app/mainnet` is dead as of 2026-09-09**: it went+ key-gated and answers 401 on the upgrade. Do not re-add it. Keyless and+ verified streaming that day: `wss://solana-rpc.publicnode.com` and+ `wss://api.mainnet-beta.solana.com`. The three.ws Helius key is plan-exhausted+ and its WS handshakes 429 forever, so do not point these at it.++ Always set `SOLANA_WS_URLS` with at least two. A single endpoint is a single+ point of failure, and the failure is silent: see the status section above.5. **Sync the stale decoders.** `@pumpkit/core` and `@pumpkit/channel` are Marchsnapshots missing the post-2026-05-21 V2 layouts (quote-mint claims, lifetime@@ -112,9 +162,18 @@ with a sliding-window budget reserving one slot for the digest.- [x] Both bots run on Cloud Run, surviving a codespace rebuild. Proven by the2026-09-02 rebuild: both local processes died, both services stayed up.-- [x] Each uses its own token and numeric channel id, verified by a live post- (1,719 messages posted and 754 digests posted, to distinct `-100...` ids).+- [x] Each uses its own token and numeric channel id, verified by live posts to+ distinct `-100...` ids (145 instant posts and 23 digests on the firehose in+ the 23 minutes after the fix, 0 post failures).- [x] WS mode confirmed active on both, with event rates recorded above.+ Re-measured 2026-09-09 after the firehose was found silent for four days:+ both now carry traffic on `solana-rpc.publicnode.com`, and `/stats`+ reports `activeWs` so a dead socket is visible rather than reported as+ healthy.+- [x] A dead WebSocket endpoint can no longer flatline a feed: endpoint lists,+ a traffic-based liveness check, rotating reconnects, and the metrics to+ see it, committed in the sibling repo at `2a12ec3a` and deployed+ (`pumpfun-allclaims-bot-00010-bdc`, `pumpfun-channel-bot-00006-wp4`).- [x] The decoder sync decision is made, executed, and written down in`DECODERS.md` at the sibling repo root: the all-claims copy is canonical,nothing was refactored into a shared package, and the two stale copies arediff --git a/prompts/finish/_context/backlog-00-INDEX.md b/prompts/finish/_context/backlog-00-INDEX.mdindex f46d9c166..9101401f4 100644--- a/prompts/finish/_context/backlog-00-INDEX.md+++ b/prompts/finish/_context/backlog-00-INDEX.md@@ -44,7 +44,7 @@ observable health, so 01 to 03 come first.| 05 | [R2 bucket CORS: verify, then fix at the origin](../906-backlog-05-r2-bucket-cors.md) | two credentials (re-measured 2026-09-09, both surfaces still fail) | first re-set `S3_SECRET_ACCESS_KEY` (object storage is DOWN, ISSUES.md item 10), then mint an R2 admin token || 07 | [BNB testnet: deploy the two finished contracts](../910-backlog-07-bnb-testnet-deploys.md) | one funded EOA | send tBNB to `0x1C4918894dfA5eE11cfF9629B458b5169Cfa3871` (faucet is reCAPTCHA-gated) || 08 | [OKX chat bot: move off the codespace](../907-backlog-08-okx-chat-bot-always-on.md) | **deployed 2026-09-04**; only the reply lane is blocked | clear the GCP billing hold so the bot can author replies |-| 09 | [Telegram bots: durable hosting for both feeds](../908-backlog-09-telegram-bots-durability.md) | **done**, verified live 2026-09-02 | clear the commit gate on its file update |+| 09 | [Telegram bots: durable hosting for both feeds](../908-backlog-09-telegram-bots-durability.md) | **done**; the firehose was found silently dead on 2026-09-09 (dead RPC), fixed and hardened the same day | clear the commit gate on its file update || 10 | [x402scan listing: finish the last three steps](../909-backlog-10-x402scan-listing.md) | **the deploy landed 2026-09-08**; only the origin registration is left (re-measured 2026-09-09) | one SIWX wallet signature at x402scan `/resources/register` for origin `https://three.ws`; no funds move |---@@ -84,9 +84,16 @@ Read the same day: both AI credentials this project holds are present and refuse`Lightning dunning decision is deny`, a billing hold; the `openai-api-key` account`billing_not_active`, which kills the platform's OpenAI lane everywhere, not just here), so adeployed host will report the new `ai_provider_unauthorized` until the owner clears billing.-09 is done: the sibling repository is checked out at `/workspaces/pump-fun-sdk` after all, and-both feeds were verified running on Cloud Run on 2026-09-02 (`Ready=True`, websocket transport,-25 h uptime each) while this codespace's own rebuild killed their local processes.+09 is done, but the 2026-09-02 reading that closed it was only half right and is worth learning+from. Both services were `Ready=True` on websocket transport, and the codespace rebuild that+killed their local twins did prove the hosting durable. What `Ready=True` did not prove is that+a feed was carrying traffic: re-measured on 2026-09-09, the all-claims firehose had 99 hours of+uptime, `mode: websocket`, and **zero events**, because `rpc.magicblock.app` went key-gated and+its only WebSocket endpoint had been answering 401 for four days. Subscribing cannot fail loudly+in web3.js, so nothing in the metrics distinguished that from health. Fixed the same day:+endpoints repointed, and both bots given a WebSocket endpoint list, a traffic-based liveness+check, rotating reconnects, and `activeWs`/`wsEventsReceived` in `/stats`. **Read a feed's event+counter, never just its readiness condition.**10's remaining external step resolved on its own(the upstream pull request merged 2026-08-11, the registry attributes 18,636 settlements and$1,055 of volume to our facilitator, and their own crawler replayed against production returnsdiff --git a/prompts/finish/_context/backlog-PROGRESS.md b/prompts/finish/_context/backlog-PROGRESS.mdindex b32c7b64e..f46b557c1 100644--- a/prompts/finish/_context/backlog-PROGRESS.md+++ b/prompts/finish/_context/backlog-PROGRESS.md@@ -76,6 +76,68 @@ Left: re-read healthz after a full budget window (3h+) and confirm settle >90%with `fee_runway_exhausted` no longer top; whoever finishes the pacing arccommits the governor + test file together.+## 2026-09-09: 09 telegram-bots-durability (reopened by measurement, fixed, re-closed)++Measured first, per this order's own instruction not to trust its numbers. The+graduation tracker was healthy (137,954 events, 9,270 messages, 99 h uptime,+`degraded: false`). The all-claims firehose was **silently dead**: same 99 h+uptime, `mode: websocket`, `claimsDetected: 0`, `totalEvents: 0`. Its logs were+hundreds of `ws error: Unexpected server response: 401` per second.++Root cause: `rpc.magicblock.app` went key-gated some time after 2026-09-02. It+now returns 401 on the WebSocket upgrade and `{"error":"invalid api key"}` over+HTTP, and it was the firehose's only WS endpoint and its primary RPC. Confirmed+directly, not inferred: a raw upgrade request returns 401, a `getSlot` returns+the api-key error, while `solana-rpc.publicnode.com` and+`api.mainnet-beta.solana.com` both stream (5 events in 355 ms and 792 ms).+`drpc.org` and `rpc.ankr.com` are also no longer keyless.++Why nobody saw it: subscribing cannot fail loudly. web3.js returns a+subscription id immediately and retries the socket internally, so+`startWebSocket()` resolved and the bot reported `websocket` transport. The+90-second silence heartbeat fired correctly but its reconnect rebuilt the same+dead endpoint, so the watchdog looped for four days instead of escaping. This is+the lesson worth carrying: **`Ready=True` plus `mode: websocket` is not evidence+a feed is working. Read the event counter.**++Did, config (restored the feed immediately, config-only Cloud Run update):+repointed both services off magicblock onto endpoints verified keyless and+streaming the same day. The firehose recovered on the spot: 62 claims in the+first two minutes against this order's ~20/min baseline. Also found and closed a+trap that would have re-broken it: `deploy-cloudrun.sh` ships env from `.env` via+`--env-vars-file`, which replaces the whole set, and the local `.env` still held+the magicblock config. Regenerated both from the corrected live revisions with+`recover-env.sh`.++Did, code (sibling repo `2a12ec3a`, canonical copy first then propagated to+`channel-bot`, per its own `DECODERS.md` rule): a WebSocket endpoint list+(`SOLANA_WS_URLS`, with the RPC-derived endpoints appended so there is always a+fallback); a liveness gate that requires a new subscription to deliver a real log+event within 20 s before the endpoint is accepted; reconnects that rotate instead+of retrying the endpoint that just went quiet; the heartbeat timer cleared before+re-arming (every reconnect had been stacking another interval); and `activeWs` ++`wsEventsReceived` in `/stats` so a traffic-free socket is visible. Applied to+`channel-bot`'s `event-monitor.ts` too, since that is the transport its live feed+actually runs on rather than the claim monitor.++Proven, not assumed: ran the monitor with the dead endpoint listed first; it+rejected magicblock, settled on publicnode, and reported 9,773 events. New+`ws-urls.test.ts` in both bots (7 cases). Suites green: 95 allclaims, 205+channel-bot.++Deployed and re-verified: `pumpfun-allclaims-bot-00010-bdc` (745 claims and+242,207 WS events in 23 min, 145 instant posts, 23 digests, 0 post failures) and+`pumpfun-channel-bot-00006-wp4` (156 events in 3 min, `degraded: false`,+delivery `ok`, `activeWs` reported).++Left: only the owner-gated commit of the three.ws-side file updates (this entry,+the order file, the index row), which the coin commit gate covers. Note for+whoever commits: `backlog-00-INDEX.md` also carries concurrent edits from other+sessions to rows 01, 05 and 10.++Not done on purpose: neither bot has a persistent volume, so tracker state resets+on redeploy. Pre-existing, and a separate change.+## 2026-08-01: 09 telegram-bots-durability (done)Measured: both feeds ran as codespace-local processes; local graduation bot pid