Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(backlog): correct item 09, whose feed was ready and carrying nothing

nirholas7228c98f18d3
diff --git a/prompts/finish/908-backlog-09-telegram-bots-durability.md b/prompts/finish/908-backlog-09-telegram-bots-durability.md
index 5dec1a3f0..545def99e 100644
--- a/prompts/finish/908-backlog-09-telegram-bots-durability.md
+++ b/prompts/finish/908-backlog-09-telegram-bots-durability.md
@@ -7,47 +7,92 @@ Read [00-INDEX.md](_context/backlog-00-INDEX.md) first.
> in three.ws without owner approval. Build and deploy freely; ask before
> committing anything that names the launchpad into this repo.
-## Status: done, verified live 2026-09-02
+## Status: both feeds live and hardened, re-verified 2026-09-09
-Both feeds now run on Cloud Run and no longer depend on this codespace. The
-sibling repo is checked out at `/workspaces/pump-fun-sdk`, so the deliverables
-below are verifiable from here, which they were not when this order was written.
+The 2026-09-02 pass declared this done. It was half true. Re-measuring on
+2026-09-09 found the graduation tracker healthy and **the all-claims firehose
+silently dead**: 99 hours of uptime, `mode: websocket`, and zero events.
-| Feed | Cloud Run service | Revision | Uptime | Transport |
+Root cause: `rpc.magicblock.app` went key-gated. It now answers `401` on the
+WebSocket upgrade and `{"error":"invalid api key"}` over HTTP, and it was the
+firehose's only WebSocket endpoint *and* its primary RPC. The 2026-08-01 note
+below calling it a working free endpoint is what stopped being true.
+
+Why it stayed invisible for four days is the part worth keeping: **subscribing
+cannot fail loudly.** web3.js returns a subscription id immediately and retries
+the socket internally, so `startWebSocket()` resolved, `transport` was set to
+`websocket`, and `/stats` reported a healthy feed. The 90-second silence
+heartbeat did fire, but its reconnect rebuilt the same dead endpoint, so the
+watchdog looped instead of escaping. Nothing in the metrics distinguished a
+connected socket from a refused one.
+
+Both halves are fixed and deployed.
+
+| Feed | Cloud Run service | Revision | Transport | Measured 2026-09-09 |
|---|---|---|---|---|
-| Graduation and migration tracker | `pumpfun-channel-bot` | `-00004-tvk` | 25.4 h | websocket |
-| All-claims firehose | `pumpfun-allclaims-bot` | `-00008-6d8` | 25.7 h | websocket |
-
-Both are `Ready=True`, pinned singletons (`minScale=1`, `maxScale=1`) on the
-`three-ws@` runtime SA. The codespace was rebuilt on 2026-09-02 and both local
-processes on ports 3900/3901 died with it while the two services kept running:
-that rebuild is the proof of durability this order asked for, not a regression.
-
-Measured the same day: the tracker had posted 1,719 messages over 23,456 events
-(`degraded: false`, delivery `ok`); the firehose had detected 15,566 claims and
-posted 754 digests over 14,862 events, on `rpc.magicblock.app/mainnet`, with 2
-post failures. Channel ids are distinct and numeric (`-1003965305979` and
-`-1003905427189`), each with its own bot token in its own Secret Manager secret.
-
-One open observation, not a failure: the firehose reports 531 queue drops
-(roughly 6.5% of claim transactions never fetched). That is the RPC queue's
-designed backpressure rather than a fault, and the levers are queue capacity and
-endpoint count. The 2026-08-02 baseline of zero drops was a 3h43m measurement,
-not a contract.
-
-### What was left to build
-
-Every numbered item below had already been completed by the 2026-08-01/02
-sessions. The one real gap this pass found and closed was configuration
-durability: `.env` is gitignored, so the rebuild destroyed the only copy of each
-bot's working config, and nothing could read it back. That stranded a bot twice
-over, since `deploy-cloudrun.sh` also reads its whole configuration from `.env`.
-Each bot directory now has a `recover-env.sh` that rebuilds `.env` from the live
-revision plus Secret Manager, applying the same numeric-`CHANNEL_ID` guard the
-deploy applies. It is committed in the sibling repo (`9701edb9`) and unpushed.
+| Graduation and migration tracker | `pumpfun-channel-bot` | `-00006-wp4` | websocket on `solana-rpc.publicnode.com` | 156 events in 3m17s, `degraded: false`, delivery `ok` |
+| All-claims firehose | `pumpfun-allclaims-bot` | `-00010-bdc` | websocket on `solana-rpc.publicnode.com` | 745 claims and 242,207 WS events in 23m, 145 instant posts, 23 digests, **0 post failures** |
+
+Both `Ready=True`, pinned singletons (`minScale=1`, `maxScale=1`) on the
+`three-ws@` runtime SA, each with its own bot token in its own Secret Manager
+secret and its own numeric channel id (`-1003965305979`, `-1003905427189`).
+
+The firehose recovered the moment its endpoints were repointed: 62 claims in the
+first two minutes, against the ~20 claims/min baseline this order set.
+
+### The config fix (immediate)
+
+Both services were repointed off magicblock onto endpoints verified keyless and
+streaming the same day: `solana-rpc.publicnode.com`, `api.mainnet-beta.solana.com`
+(HTTP and WS), and `solana.leorpc.com/?api_key=FREE` (HTTP only; its WS refuses).
+`drpc.org` and `rpc.ankr.com` are no longer keyless either and were rejected.
+
+A trap this exposed: `deploy-cloudrun.sh` ships env from `.env` via
+`--env-vars-file`, which **replaces the whole set**. The local `.env` still held
+the magicblock config, so the next deploy would have silently re-broken the feed.
+`recover-env.sh` now regenerates it from the corrected live revision, and both
+`.env` files carry the new endpoint lists.
+
+### The code fix (so this class of outage cannot hide again)
+
+Landed in the sibling repo at `2a12ec3a`, canonical copy first and propagated to
+`channel-bot`, per the rule in its own `DECODERS.md`:
+
+- **A WebSocket endpoint list, not one endpoint.** `SOLANA_WS_URLS` is
+ comma-separated, and whatever the RPC list implies is appended, so there is
+ always somewhere to fail over to. The firehose now runs with 5 resolved.
+- **Liveness is proven by traffic, not by a subscription id.** A new
+ subscription must deliver a real log event within 20 seconds or the endpoint is
+ rejected and the next one tried. The monitored programs emit thousands of
+ events a minute, so silence that long means the endpoint, not the chain.
+- **Reconnects rotate** instead of retrying the endpoint that just went quiet.
+- **The heartbeat timer is cleared before it is re-armed.** Every reconnect since
+ this code was written had been stacking another interval.
+- **`/stats` exposes `activeWs` and `wsEventsReceived`,** so a WebSocket carrying
+ no traffic is visible rather than indistinguishable from a healthy one.
+- `channel-bot`'s live feed runs on `event-monitor.ts`, not the claim monitor, so
+ the rotation was applied there too rather than only to the shared decoder path.
+
+Verified by running the monitor against the dead endpoint first: it rejected
+magicblock, settled on `solana-rpc.publicnode.com`, and reported 9,773 events.
+Covered by `src/__tests__/ws-urls.test.ts` in both bots (7 cases, including the
+regression that a single explicit endpoint must never de-duplicate down to a
+single point of failure). Suites green: 95 tests allclaims, 205 channel-bot.
+
+Deliberately not done: neither bot has a persistent volume, so tracker state is
+ephemeral and a redeploy resets it. That is pre-existing, was true before this
+pass, and mounting one is a separate change.
+
+### Standing observation, not a failure
+
+`queueDrops` sits at 53 on the firehose (~0.9% of pipeline claims). That is the
+RPC queue's designed backpressure, and the levers are queue capacity and endpoint
+count. The 2026-08-02 baseline of zero drops was a 3h43m measurement, not a
+contract.
## The work
+
1. **Deploy both to Cloud Run.** `channel-bot/deploy-cloudrun.sh` exists and is the
template. Write the equivalent for the all-claims package. Pin the
`three-ws-build@` and `three-ws@` service accounts.
@@ -65,13 +110,18 @@ deploy applies. It is committed in the sibling repo (`9701edb9`) and unpushed.
id from `getUpdates?allowed_updates=["my_chat_member"]`, whose promotion event
carries the chat id, username, and full admin rights object.
-4. **Keep websockets, do not fall back to polling.** `getSignaturesForAddress`
- limit-20 per 30s against the program samples a handful of valid transactions
- per minute out of thousands, so events get missed entirely. The code only
- enables WS when `SOLANA_WS_URL` is explicitly set. `wss://rpc.magicblock.app/mainnet`
- carries the full firehose on the free tier (measured 4,545 events per 20s) and
- `wss://solana-rpc.publicnode.com` also works keyless. The three.ws Helius key is
- plan-exhausted and its WS handshakes 429 forever, so do not point these at it.
+4. **Keep websockets, do not fall back to polling, and configure more than one.**
+ `getSignaturesForAddress` limit-20 per 30s against the program samples a
+ handful of valid transactions per minute out of thousands, so events get
+ missed entirely. WS is enabled when `SOLANA_WS_URL` or `SOLANA_WS_URLS` is
+ set. **`wss://rpc.magicblock.app/mainnet` is dead as of 2026-09-09**: it went
+ key-gated and answers 401 on the upgrade. Do not re-add it. Keyless and
+ verified streaming that day: `wss://solana-rpc.publicnode.com` and
+ `wss://api.mainnet-beta.solana.com`. The three.ws Helius key is plan-exhausted
+ and its WS handshakes 429 forever, so do not point these at it.
+
+ Always set `SOLANA_WS_URLS` with at least two. A single endpoint is a single
+ point of failure, and the failure is silent: see the status section above.
5. **Sync the stale decoders.** `@pumpkit/core` and `@pumpkit/channel` are March
snapshots missing the post-2026-05-21 V2 layouts (quote-mint claims, lifetime
@@ -112,9 +162,18 @@ with a sliding-window budget reserving one slot for the digest.
- [x] Both bots run on Cloud Run, surviving a codespace rebuild. Proven by the
2026-09-02 rebuild: both local processes died, both services stayed up.
-- [x] Each uses its own token and numeric channel id, verified by a live post
- (1,719 messages posted and 754 digests posted, to distinct `-100...` ids).
+- [x] Each uses its own token and numeric channel id, verified by live posts to
+ distinct `-100...` ids (145 instant posts and 23 digests on the firehose in
+ the 23 minutes after the fix, 0 post failures).
- [x] WS mode confirmed active on both, with event rates recorded above.
+ Re-measured 2026-09-09 after the firehose was found silent for four days:
+ both now carry traffic on `solana-rpc.publicnode.com`, and `/stats`
+ reports `activeWs` so a dead socket is visible rather than reported as
+ healthy.
+- [x] A dead WebSocket endpoint can no longer flatline a feed: endpoint lists,
+ a traffic-based liveness check, rotating reconnects, and the metrics to
+ see it, committed in the sibling repo at `2a12ec3a` and deployed
+ (`pumpfun-allclaims-bot-00010-bdc`, `pumpfun-channel-bot-00006-wp4`).
- [x] The decoder sync decision is made, executed, and written down in
`DECODERS.md` at the sibling repo root: the all-claims copy is canonical,
nothing was refactored into a shared package, and the two stale copies are
diff --git a/prompts/finish/_context/backlog-00-INDEX.md b/prompts/finish/_context/backlog-00-INDEX.md
index f46d9c166..9101401f4 100644
--- a/prompts/finish/_context/backlog-00-INDEX.md
+++ b/prompts/finish/_context/backlog-00-INDEX.md
@@ -44,7 +44,7 @@ observable health, so 01 to 03 come first.
| 05 | [R2 bucket CORS: verify, then fix at the origin](../906-backlog-05-r2-bucket-cors.md) | two credentials (re-measured 2026-09-09, both surfaces still fail) | first re-set `S3_SECRET_ACCESS_KEY` (object storage is DOWN, ISSUES.md item 10), then mint an R2 admin token |
| 07 | [BNB testnet: deploy the two finished contracts](../910-backlog-07-bnb-testnet-deploys.md) | one funded EOA | send tBNB to `0x1C4918894dfA5eE11cfF9629B458b5169Cfa3871` (faucet is reCAPTCHA-gated) |
| 08 | [OKX chat bot: move off the codespace](../907-backlog-08-okx-chat-bot-always-on.md) | **deployed 2026-09-04**; only the reply lane is blocked | clear the GCP billing hold so the bot can author replies |
-| 09 | [Telegram bots: durable hosting for both feeds](../908-backlog-09-telegram-bots-durability.md) | **done**, verified live 2026-09-02 | clear the commit gate on its file update |
+| 09 | [Telegram bots: durable hosting for both feeds](../908-backlog-09-telegram-bots-durability.md) | **done**; the firehose was found silently dead on 2026-09-09 (dead RPC), fixed and hardened the same day | clear the commit gate on its file update |
| 10 | [x402scan listing: finish the last three steps](../909-backlog-10-x402scan-listing.md) | **the deploy landed 2026-09-08**; only the origin registration is left (re-measured 2026-09-09) | one SIWX wallet signature at x402scan `/resources/register` for origin `https://three.ws`; no funds move |
---
@@ -84,9 +84,16 @@ Read the same day: both AI credentials this project holds are present and refuse
`Lightning dunning decision is deny`, a billing hold; the `openai-api-key` account
`billing_not_active`, which kills the platform's OpenAI lane everywhere, not just here), so a
deployed host will report the new `ai_provider_unauthorized` until the owner clears billing.
-09 is done: the sibling repository is checked out at `/workspaces/pump-fun-sdk` after all, and
-both feeds were verified running on Cloud Run on 2026-09-02 (`Ready=True`, websocket transport,
-25 h uptime each) while this codespace's own rebuild killed their local processes.
+09 is done, but the 2026-09-02 reading that closed it was only half right and is worth learning
+from. Both services were `Ready=True` on websocket transport, and the codespace rebuild that
+killed their local twins did prove the hosting durable. What `Ready=True` did not prove is that
+a feed was carrying traffic: re-measured on 2026-09-09, the all-claims firehose had 99 hours of
+uptime, `mode: websocket`, and **zero events**, because `rpc.magicblock.app` went key-gated and
+its only WebSocket endpoint had been answering 401 for four days. Subscribing cannot fail loudly
+in web3.js, so nothing in the metrics distinguished that from health. Fixed the same day:
+endpoints repointed, and both bots given a WebSocket endpoint list, a traffic-based liveness
+check, rotating reconnects, and `activeWs`/`wsEventsReceived` in `/stats`. **Read a feed's event
+counter, never just its readiness condition.**
10's remaining external step resolved on its own
(the upstream pull request merged 2026-08-11, the registry attributes 18,636 settlements and
$1,055 of volume to our facilitator, and their own crawler replayed against production returns
diff --git a/prompts/finish/_context/backlog-PROGRESS.md b/prompts/finish/_context/backlog-PROGRESS.md
index b32c7b64e..f46b557c1 100644
--- a/prompts/finish/_context/backlog-PROGRESS.md
+++ b/prompts/finish/_context/backlog-PROGRESS.md
@@ -76,6 +76,68 @@ Left: re-read healthz after a full budget window (3h+) and confirm settle >90%
with `fee_runway_exhausted` no longer top; whoever finishes the pacing arc
commits the governor + test file together.
+## 2026-09-09: 09 telegram-bots-durability (reopened by measurement, fixed, re-closed)
+
+Measured first, per this order's own instruction not to trust its numbers. The
+graduation tracker was healthy (137,954 events, 9,270 messages, 99 h uptime,
+`degraded: false`). The all-claims firehose was **silently dead**: same 99 h
+uptime, `mode: websocket`, `claimsDetected: 0`, `totalEvents: 0`. Its logs were
+hundreds of `ws error: Unexpected server response: 401` per second.
+
+Root cause: `rpc.magicblock.app` went key-gated some time after 2026-09-02. It
+now returns 401 on the WebSocket upgrade and `{"error":"invalid api key"}` over
+HTTP, and it was the firehose's only WS endpoint and its primary RPC. Confirmed
+directly, not inferred: a raw upgrade request returns 401, a `getSlot` returns
+the api-key error, while `solana-rpc.publicnode.com` and
+`api.mainnet-beta.solana.com` both stream (5 events in 355 ms and 792 ms).
+`drpc.org` and `rpc.ankr.com` are also no longer keyless.
+
+Why nobody saw it: subscribing cannot fail loudly. web3.js returns a
+subscription id immediately and retries the socket internally, so
+`startWebSocket()` resolved and the bot reported `websocket` transport. The
+90-second silence heartbeat fired correctly but its reconnect rebuilt the same
+dead endpoint, so the watchdog looped for four days instead of escaping. This is
+the lesson worth carrying: **`Ready=True` plus `mode: websocket` is not evidence
+a feed is working. Read the event counter.**
+
+Did, config (restored the feed immediately, config-only Cloud Run update):
+repointed both services off magicblock onto endpoints verified keyless and
+streaming the same day. The firehose recovered on the spot: 62 claims in the
+first two minutes against this order's ~20/min baseline. Also found and closed a
+trap that would have re-broken it: `deploy-cloudrun.sh` ships env from `.env` via
+`--env-vars-file`, which replaces the whole set, and the local `.env` still held
+the magicblock config. Regenerated both from the corrected live revisions with
+`recover-env.sh`.
+
+Did, code (sibling repo `2a12ec3a`, canonical copy first then propagated to
+`channel-bot`, per its own `DECODERS.md` rule): a WebSocket endpoint list
+(`SOLANA_WS_URLS`, with the RPC-derived endpoints appended so there is always a
+fallback); a liveness gate that requires a new subscription to deliver a real log
+event within 20 s before the endpoint is accepted; reconnects that rotate instead
+of retrying the endpoint that just went quiet; the heartbeat timer cleared before
+re-arming (every reconnect had been stacking another interval); and `activeWs` +
+`wsEventsReceived` in `/stats` so a traffic-free socket is visible. Applied to
+`channel-bot`'s `event-monitor.ts` too, since that is the transport its live feed
+actually runs on rather than the claim monitor.
+
+Proven, not assumed: ran the monitor with the dead endpoint listed first; it
+rejected magicblock, settled on publicnode, and reported 9,773 events. New
+`ws-urls.test.ts` in both bots (7 cases). Suites green: 95 allclaims, 205
+channel-bot.
+
+Deployed and re-verified: `pumpfun-allclaims-bot-00010-bdc` (745 claims and
+242,207 WS events in 23 min, 145 instant posts, 23 digests, 0 post failures) and
+`pumpfun-channel-bot-00006-wp4` (156 events in 3 min, `degraded: false`,
+delivery `ok`, `activeWs` reported).
+
+Left: only the owner-gated commit of the three.ws-side file updates (this entry,
+the order file, the index row), which the coin commit gate covers. Note for
+whoever commits: `backlog-00-INDEX.md` also carries concurrent edits from other
+sessions to rows 01, 05 and 10.
+
+Not done on purpose: neither bot has a persistent volume, so tracker state resets
+on redeploy. Pre-existing, and a separate change.
+
## 2026-08-01: 09 telegram-bots-durability (done)
Measured: both feeds ran as codespace-local processes; local graduation bot pid