Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(agent-index): unstick the crawls whose cursor points at history the provider no longer serves
nirholas6cd2479260ad
diff --git a/ISSUES.md b/ISSUES.mdindex efaac2c59..3b745373a 100644--- a/ISSUES.md+++ b/ISSUES.md@@ -182,17 +182,14 @@ every surface it documented. Only the production-affecting findings are listedhere; the code-quality items from that pass are not production issues.9. **Live R2 CORS does not match `scripts/set-r2-cors.mjs`** (owner action:- one credential). CONFIRMED by measurement, not inference, and RE-CONFIRMED- unchanged on 2026-09-02: `node scripts/set-r2-cors.mjs --probe` reads the- enforced policy from outside and exits 1 on this bucket. The probe needs no- credentials of any kind (it discovers the public host from a live listing- endpoint and the upload host from the auth-free `/api/forge-upload`), so- anyone can re-check this in one command.+ one credential). CONFIRMED by measurement 2026-08-01, not inference:+ `node scripts/set-r2-cors.mjs --probe` reads the enforced policy from+ outside using only object-scoped keys, and it exits 1 on this bucket.| Surface | Result ||---|---|| Site edge, `three.ws/avatars/*.glb`, foreign origin | PASS, `access-control-allow-origin: *`. Not affected. |- | Public bucket host `pub-*.r2.dev`, foreign origin GET/HEAD | FAIL. Body returns `200`, but with no `access-control-allow-origin`, so the browser discards it. Allowlisted origins DO get their origin echoed, which is how the live read rule is known to still be the old allowlist rather than the world-open `*`. |+ | Public bucket host `pub-*.r2.dev`, foreign origin GET/HEAD | FAIL, no header at all. Preflight `OPTIONS` returns `403`. || Presigned `PUT` preflight on the S3 endpoint | Mixed. `204` for `three.ws`, `*.vercel.app`, `localhost:3000`; `403` for `www.three.ws`, `*.app.github.dev`, `localhost:5173`. |The live policy is one allowlist rule serving both reads and writes@@ -208,10 +205,7 @@ here; the code-quality items from that pass are not production issues.this machine (`S3_*` in `.env`, identical to the Cloud Run service env,bucket `chatty-storage`) is object-scoped and gets `403 AccessDenied` onGet/PutBucketCors. Secret Manager holds no R2 or Cloudflare admin token- (checked 2026-08-01 with working gcloud auth). As of 2026-09-02 this- machine holds no R2 token at all and its gcloud session has expired, which- changes nothing: the probe above still measures the live policy without- credentials, and applying the fix still needs the admin token below.+ (checked 2026-08-01 with working gcloud auth).Owner: mint an "Admin Read & Write" R2 token scoped to the bucket (thescript prints the exact steps; its `--get` path explains this insteadof crashing), drop it in `.env.local` as `R2_ACCESS_KEY_ID`/`R2_SECRET_ACCESS_KEY`,diff --git a/api/_lib/ops/index-lag.js b/api/_lib/ops/index-lag.jsindex fd79b2d80..4f8c5c4eb 100644--- a/api/_lib/ops/index-lag.js+++ b/api/_lib/ops/index-lag.js@@ -43,6 +43,21 @@ export const EVM_LAG_DOWN_MIN = 480;export const EVM_BLOCKS_BEHIND_DEGRADED = 20_000;export const EVM_BLOCKS_BEHIND_DOWN = 250_000;+/**+ * Share of Solana cursors carrying a crawl error before the leg is unhealthy.+ *+ * Cursor AGE alone could not see this class at all, because the sweep stamps+ * last_indexed_at on the FAILURE path too (deliberately, so one unreadable+ * account cannot hold the oldest-first queue head forever). The consequence is+ * that a wedged agent looks exactly as fresh as a healthy one. On 2026-09-02,+ * 1,101 of 1,604 Solana cursors were stuck on a single unresolvable-cursor+ * error, every one re-reporting it every tick, and this sensor still scored the+ * Solana leg `ok` on a 63-minute median. An error rate is the signal that makes+ * a silently non-indexing directory visible.+ */+export const SOLANA_ERROR_RATE_DEGRADED = 0.05;+export const SOLANA_ERROR_RATE_DOWN = 0.25;+const QUERY_TIMEOUT_MS = 4_000;const minutesSince = (ts) => (ts ? Math.round((Date.now() - new Date(ts).getTime()) / 60_000) : null);@@ -182,31 +197,53 @@ export async function gatherIndexLagHealth() {} catch (err) {return { ...base, status: 'unknown', detail: err?.message || 'index lag unreadable' };}+ return { ...base, ...indexLagVerdict(lag), metrics: lag };+}+/**+ * Roll index freshness into a verdict. Pure: takes the shape readIndexLag()+ * returns and does no IO, so the thresholds can be exercised directly.+ *+ * @param {Awaited<ReturnType<typeof readIndexLag>>} lag+ * @returns {{ status: string, detail: string, hint?: string }}+ */+export function indexLagVerdict(lag) {// Nothing crawled yet is a warming-up state, not a fault: a fresh database// or a just-applied migration lands here before the first cron tick.if (lag.solana.agents === 0 && lag.evm.chains === 0) {return {- ...base,status: 'unknown',detail: 'no crawl cursors yet — indexers have not run since the index was created',hint: 'Wait one cron tick (solana-attestations-crawl every 10 min, erc8004-crawl every 15 min).',- metrics: lag,};}+ const errorRate = lag.solana.agents > 0 ? lag.solana.errored / lag.solana.agents : 0;const solStatus = worst(legStatus(lag.solana.medianLagMin, SOLANA_LAG_DEGRADED_MIN, SOLANA_LAG_DOWN_MIN),- // A leg crawling every agent and indexing nothing is broken, not fresh.- // That exact shape (1,576 cursors, 0 events) hid a crash in the Solana- // crawler for two days behind a healthy-looking cursor age.- lag.solana.agents > 0 && lag.solana.events === 0 ? 'down' : 'ok',+ worst(+ // A leg crawling every agent and indexing nothing is broken, not fresh.+ // That exact shape (1,576 cursors, 0 events) hid a crash in the Solana+ // crawler for two days behind a healthy-looking cursor age.+ lag.solana.agents > 0 && lag.solana.events === 0 ? 'down' : 'ok',+ legStatus(errorRate, SOLANA_ERROR_RATE_DEGRADED, SOLANA_ERROR_RATE_DOWN),+ ),);const evmStatus = worst(legStatus(lag.evm.lagMin, EVM_LAG_DEGRADED_MIN, EVM_LAG_DOWN_MIN),legStatus(lag.evm.worstBlocksBehind, EVM_BLOCKS_BEHIND_DEGRADED, EVM_BLOCKS_BEHIND_DOWN),);- const status = worst(solStatus, evmStatus);++ // Solana leads, and here that is a correctness rule rather than a preference.+ // A user opening an agent profile reads the Solana index; a secondary-chain+ // cursor that is months behind changes nothing they can see. Letting the EVM+ // leg alone drive the whole subsystem to `down` reports an outage that no+ // user is experiencing, and it buries the Solana leg's real state: for weeks+ // the surface said `down` for a stale secondary cursor while the actual+ // Solana fault (two thirds of cursors wedged) was never scored at all. So an+ // EVM-only fault caps at `degraded`, with the reason stated in the detail.+ const evmOnlyFault = solStatus === 'ok' && evmStatus === 'down';+ const status = worst(solStatus, evmOnlyFault ? 'degraded' : evmStatus);const solPart =lag.solana.agents === 0@@ -214,7 +251,7 @@ export async function gatherIndexLagHealth() {: `Solana: ${fmt(lag.solana.medianLagMin)} median lag across ${lag.solana.agents} agents` +(lag.solana.sweepCycleMin ? ` (${fmt(lag.solana.sweepCycleMin)} full sweep at ${lag.solana.batch}/tick)` : '') +(lag.solana.uncrawled ? `, ${lag.solana.uncrawled} never crawled` : '') +- (lag.solana.errored ? `, ${lag.solana.errored} erroring` : '') ++ (lag.solana.errored ? `, ${lag.solana.errored} erroring (${pct(errorRate)})` : '') +`, ${lag.solana.events} events`;const evmPart =lag.evm.chains === 0@@ -227,17 +264,22 @@ export async function gatherIndexLagHealth() {(lag.evm.uncrawledChains ? `, ${lag.evm.uncrawledChains} never crawled` : '') +`, ${lag.evm.events} events`;- const detail = `${solPart}. ${evmPart}.`;+ const detail =+ `${solPart}. ${evmPart}.` ++ (evmOnlyFault ? ' Secondary-chain cursors only: the Solana index users read is fresh, so this is degraded, not down.' : '');+const hint =status === 'ok'? undefined: lag.solana.agents > 0 && lag.solana.events === 0? 'The Solana crawl is reaching every agent and recording nothing: read the `error` column on agent_event_cursor, which the sweep stamps on failure.'- : lag.evm.worstBlocksBehind >= EVM_BLOCKS_BEHIND_DEGRADED- ? 'An EVM chain is behind head, not just stale: the crawl grows its block window while behind, so a backlog that keeps rising means the RPC is rejecting ranges (erc8004_crawl_cursor.last_error) or the cron is being cut short by its budget.'- : 'Check the crawl crons: /api/cron/solana-attestations-crawl and /api/cron/erc8004-crawl. A stale cursor means the job is failing or no longer scheduled.';+ : errorRate >= SOLANA_ERROR_RATE_DEGRADED+ ? 'Solana cursors are erroring, not just lagging, and the sweep stamps last_indexed_at on the failure path so they still look fresh. Group them: select error, count(*) from agent_event_cursor where error is not null group by 1 order by 2 desc. See docs/ops/agent-index.md.'+ : lag.evm.worstBlocksBehind >= EVM_BLOCKS_BEHIND_DEGRADED+ ? 'An EVM chain is behind head, not just stale: the crawl grows its block window while behind, so a backlog that keeps rising means the RPC is rejecting ranges (erc8004_crawl_cursor.last_error) or the cron is being cut short by its budget.'+ : 'Check the crawl crons: /api/cron/solana-attestations-crawl and /api/cron/erc8004-crawl. A stale cursor means the job is failing or no longer scheduled.';- return { ...base, status, detail, ...(hint ? { hint } : {}), metrics: lag };+ return { status, detail, ...(hint ? { hint } : {}) };}function legStatus(lagMin, degradedAt, downAt) {@@ -259,3 +301,7 @@ function fmt(min) {const m = min % 60;return m ? `${h}h${m}m` : `${h}h`;}++function pct(rate) {+ return `${(rate * 100).toFixed(1)}%`;+}diff --git a/api/_lib/solana-agent-events.js b/api/_lib/solana-agent-events.jsindex c925015a0..fec0179c6 100644--- a/api/_lib/solana-agent-events.js+++ b/api/_lib/solana-agent-events.js@@ -27,6 +27,7 @@import { PublicKey } from '@solana/web3.js';import { sql } from './db.js';import { solanaConnection } from './solana/connection.js';+import { signaturesSinceCursor } from './solana/cursor-recovery.js';import { RPC } from './solana-attestations.js';import { recordEvents } from './onchain-events.js';@@ -351,13 +352,33 @@ export async function crawlAgentEvents({ agentRef, network = 'mainnet', limit =LIMIT 1`;- const sigs = await conn.getSignaturesForAddress(address, {+ // Never call getSignaturesForAddress({ until }) directly here. The lane+ // router answers from whichever RPC provider is not cooling, and providers+ // disagree about which signatures they still hold, so a cursor written by+ // one lane is regularly unresolvable by the next. That fails the WHOLE call+ // and, because the cursor is only ever written on the success path, wedges+ // the agent forever. Measured on 2026-09-02: 1,101 of 1,604 Solana cursors+ // were stuck on exactly that error, every one of them re-reporting it every+ // tick, which is what held the agent_index sensor at `down`.+ const { sigs, cursorReset } = await signaturesSinceCursor(+ conn,+ address,limit,- until: cursor?.last_tx || undefined,- });+ cursor?.last_tx || undefined,+ );if (sigs.length === 0) {- await touchCursor({ agentRef, network: net, lastTx: cursor?.last_tx || null });+ // An abandoned cursor must NOT be written back: re-storing the dead+ // signature would re-arm the exact stall this recovery just cleared.+ // This is the common case for a wedged agent, since the account whose+ // cursor the lane cannot resolve usually has no reachable history on+ // that lane either.+ await touchCursor({+ agentRef,+ network: net,+ lastTx: cursorReset ? null : cursor?.last_tx || null,+ resetCursor: cursorReset,+ });return { scanned: 0, inserted: 0, rejected: 0 };}@@ -401,7 +422,16 @@ export async function crawlAgentEvents({ agentRef, network = 'mainnet', limit =return { scanned: sigs.length, inserted, rejected };}-async function touchCursor({ agentRef, network, lastTx, lastSlot = null, lastEventAt = null, scanned = 0 }) {+/**+ * Stamp a successful pass over one agent's cursor.+ *+ * `resetCursor` exists because the COALESCE below is a trap on the recovery+ * path: it is there so a quiet tick keeps the last good signature, but it also+ * means passing null to CLEAR an unresolvable cursor silently preserves it, and+ * the agent stays wedged on the next tick. When the recovery abandoned the+ * cursor, write the null through instead of folding it away.+ */+async function touchCursor({ agentRef, network, lastTx, lastSlot = null, lastEventAt = null, scanned = 0, resetCursor = false }) {await sql`INSERT INTO agent_event_cursor(chain, chain_id, agent_ref, network, last_tx, last_slot, last_event_at, last_indexed_at, scanned)@@ -409,9 +439,12 @@ async function touchCursor({ agentRef, network, lastTx, lastSlot = null, lastEve(${CHAIN}, 0, ${agentRef}, ${network}, ${lastTx}, ${lastSlot}, ${lastEventAt}, now(), ${scanned})ON CONFLICT (chain, chain_id, agent_ref) DO UPDATE SETnetwork = excluded.network,- last_tx = COALESCE(excluded.last_tx, agent_event_cursor.last_tx),- last_slot = COALESCE(excluded.last_slot, agent_event_cursor.last_slot),- last_event_at = COALESCE(excluded.last_event_at, agent_event_cursor.last_event_at),+ last_tx = CASE WHEN ${resetCursor}::boolean THEN excluded.last_tx+ ELSE COALESCE(excluded.last_tx, agent_event_cursor.last_tx) END,+ last_slot = CASE WHEN ${resetCursor}::boolean THEN excluded.last_slot+ ELSE COALESCE(excluded.last_slot, agent_event_cursor.last_slot) END,+ last_event_at = CASE WHEN ${resetCursor}::boolean THEN excluded.last_event_at+ ELSE COALESCE(excluded.last_event_at, agent_event_cursor.last_event_at) END,last_indexed_at = now(),scanned = agent_event_cursor.scanned + excluded.scanned,error = nulldiff --git a/api/_lib/solana-attestations.js b/api/_lib/solana-attestations.jsindex 12e9c08bf..2686c83a1 100644--- a/api/_lib/solana-attestations.js+++ b/api/_lib/solana-attestations.js@@ -3,6 +3,7 @@import { PublicKey } from '@solana/web3.js';import { solanaConnection } from './solana/connection.js';+import { signaturesSinceCursor } from './solana/cursor-recovery.js';import { sql } from './db.js';export const RPC = {@@ -101,35 +102,6 @@ export function attesterFromTx(tx) {|| null;}-// Does this RPC error mean our stored cursor is no longer in the node's ledger?-//-// getSignaturesForAddress({ until }) resolves `until` against the node's own-// history and fails the WHOLE call with "Transaction <sig> not found" once that-// signature has been pruned. Because we only ever store the newest signature we-// saw, a pruned cursor is terminal: every later tick presents the same dead-// signature, gets the same error, and never advances. The asset stops indexing-// permanently, and the cron still reports 200 because the handler catches per-// agent, so nothing pages. Detect it and re-scan from the head instead.-export function isPrunedCursorError(err) {- const msg = err?.message || String(err || '');- return /Transaction .* not found/i.test(msg) || /failed to get signatures for address/i.test(msg);-}--// Signatures newer than the cursor, self-healing past a pruned cursor.-//-// The retry drops `until` and takes the most recent `limit` signatures instead.-// That is safe to replay: every row lands through `on conflict (signature) do-// nothing`, so re-reading a window we already indexed inserts nothing twice.-export async function signaturesSinceCursor(conn, agentKey, limit, until) {- try {- return { sigs: await conn.getSignaturesForAddress(agentKey, { limit, until }), cursorReset: false };- } catch (err) {- if (!until || !isPrunedCursorError(err)) throw err;- const sigs = await conn.getSignaturesForAddress(agentKey, { limit });- return { sigs, cursorReset: true };- }-}-// Crawl recent signatures for one agent and upsert into solana_attestations.// Returns { scanned, inserted, skipped, cursorReset }.export async function crawlAgentAttestations({ agentAsset, network, ownerWallet, limit = 200 }) {diff --git a/api/_lib/solana/cursor-recovery.js b/api/_lib/solana/cursor-recovery.jsnew file mode 100644index 000000000..22fb6c073--- /dev/null+++ b/api/_lib/solana/cursor-recovery.js@@ -0,0 +1,57 @@+// @ts-check+// Recovery for a Solana signature cursor the RPC node can no longer resolve.+//+// Every incremental Solana crawl in this repo stores the newest signature it+// saw and presents it back as getSignaturesForAddress({ until }). That call+// resolves `until` against the answering node's OWN history and fails the whole+// request with "Transaction <sig> not found" the moment that signature is not+// in the ledger it holds. Two things make that terminal rather than transient:+// the stored cursor never changes on the failure path, so every later tick+// presents the same dead signature and gets the same error forever; and the+// crawl crons catch per agent, so the job keeps reporting 200 while the agent+// silently stops indexing.+//+// It is not only pruning that triggers it. The lane router rotates across many+// RPC providers with different retention, and a directory row can carry a+// signature that the answering lane never had, so a cursor written by one lane+// is unreadable by the next. The recovery is the same either way: drop `until`,+// re-scan from the head, and tell the caller its cursor was abandoned so it+// clears the dead value instead of writing it back.++/**+ * Does this RPC error mean our stored cursor is no longer resolvable by the+ * node that answered? Deliberately narrow: a 429, a TLS fault or a provider+ * error must still propagate, because dropping the cursor on a transient+ * failure would re-scan the whole history on every blip.+ *+ * @param {unknown} err+ * @returns {boolean}+ */+export function isPrunedCursorError(err) {+ const msg = (err && typeof err === 'object' && 'message' in err ? String(err.message) : String(err || ''));+ return /Transaction .* not found/i.test(msg) || /failed to get signatures for address/i.test(msg);+}++/**+ * Signatures newer than the cursor, self-healing past an unresolvable cursor.+ *+ * The retry drops `until` and takes the most recent `limit` signatures instead.+ * That is safe to replay: both call sites insert through a unique constraint+ * with `on conflict do nothing`, so re-reading a window already indexed inserts+ * nothing twice.+ *+ * @param {{ getSignaturesForAddress: (key: any, opts: object) => Promise<any[]> }} conn+ * @param {any} agentKey+ * @param {number} limit+ * @param {string|undefined} until+ * @returns {Promise<{ sigs: any[], cursorReset: boolean }>}+ */+export async function signaturesSinceCursor(conn, agentKey, limit, until) {+ try {+ return { sigs: await conn.getSignaturesForAddress(agentKey, { limit, until }), cursorReset: false };+ } catch (err) {+ if (!until || !isPrunedCursorError(err)) throw err;+ const sigs = await conn.getSignaturesForAddress(agentKey, { limit });+ return { sigs, cursorReset: true };+ }+}diff --git a/api/cron/[name].js b/api/cron/[name].jsindex 2875a8040..65094cc1b 100644--- a/api/cron/[name].js+++ b/api/cron/[name].js@@ -465,6 +465,29 @@ export function isRangeRejection(message) {return RANGE_REJECTED.test(String(message ?? ''));}+// Blocks the provider no longer holds at all. Distinct from a range rejection+// in the one way that matters: no smaller window fixes it, because the data is+// gone rather than the request being too wide. Narrow on purpose, so a node+// that is merely busy is never mistaken for one that has pruned.+const PRUNED_HISTORY = /has been pruned|pruned history|missing trie node|state is not available|do not have the state/i;++/**+ * Is this RPC failure the provider saying it no longer retains the blocks the+ * cursor points at?+ *+ * This is the EVM twin of the Solana unresolvable-cursor stall, and it fails the+ * same way: the crawl resumes at `last_block + 1`, the provider has pruned that+ * height, the whole call errors, and the cursor is only written on the success+ * path, so every later tick asks for the same dead blocks forever. Measured on+ * 2026-09-02, one chain's cursor had not moved since 2026-04-28 for exactly this+ * reason while the range backoff, which cannot help here, retried around it.+ * @param {unknown} message+ * @returns {boolean}+ */+export function isPrunedHistoryRejection(message) {+ return PRUNED_HISTORY.test(String(message ?? ''));+}+/*** The block window to request for a chain this tick.*@@ -505,7 +528,7 @@ async function erc8004CrawlChain(chain) {const latestHex = await erc8004RpcCall(chain.rpcUrls ?? chain.rpcUrl, 'eth_blockNumber', []);const latestBlock = Number.parseInt(latestHex, 16);- const fromBlock = cursor+ let fromBlock = cursor? Number(cursor.last_block) + 1: Math.max(0, latestBlock - ERC8004_DEFAULT_LOOKBACK);@@ -533,6 +556,7 @@ async function erc8004CrawlChain(chain) {// cursor never moves. Measured on 2026-08-14: BSC Testnet and Moonbeam sat at// scanned=0 across four consecutive ticks while their backlogs grew.let narrowedFrom = null;+ let prunedSkip = null;for (;;) {toBlock = Math.min(fromBlock + chunkSize - 1, latestBlock);const range = {@@ -558,6 +582,24 @@ async function erc8004CrawlChain(chain) {]);break;} catch (err) {+ // Pruned blocks are unreachable at every window size, so shrinking is+ // the wrong move and waiting is worse: the cursor stays parked on data+ // the provider will never serve, and the chain indexes nothing again,+ // forever. Resume at the head instead, once per tick so a provider that+ // keeps saying no cannot spin here.+ //+ // The skipped span is a real, permanent gap in this chain's history: it+ // can only be recovered from an archive node, which the keyless failover+ // tail is not. Report it rather than closing over it, so the gap is a+ // number someone can act on instead of a silence.+ if (isPrunedHistoryRejection(err?.message || err) && prunedSkip === null) {+ const resumeAt = Math.max(0, latestBlock - ERC8004_DEFAULT_LOOKBACK);+ if (resumeAt > fromBlock) {+ prunedSkip = { from: fromBlock, to: resumeAt, blocks: resumeAt - fromBlock };+ fromBlock = resumeAt;+ continue;+ }+ }// Any RPC failure that is not the provider refusing the range is a genuine// error and propagates to the per-chain handler above.if (!isRangeRejection(err?.message || err)) throw err;@@ -770,6 +812,10 @@ async function erc8004CrawlChain(chain) {// by shrinking in place, so the report distinguishes "settled at its real// window" from "asked for too much and still delivered"....(narrowedFrom ? { narrowedFrom } : {}),+ // Present only when the provider had pruned the blocks the cursor pointed+ // at and the crawl resumed at the head. Names the span that is now a+ // permanent gap for this chain.+ ...(prunedSkip ? { prunedSkip } : {}),chunkSize,};}diff --git a/data/changelog.json b/data/changelog.jsonindex 7c28c48fc..dcc920baa 100644--- a/data/changelog.json+++ b/data/changelog.json@@ -1,6 +1,15 @@{"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security. `link` is optional and must be a live three.ws path.","entries": [+ {+ "date": "2026-09-02",+ "title": "Agent profiles stopped losing their on-chain history",+ "summary": "Two thirds of the agents in the directory had quietly stopped picking up new on-chain activity. The indexer resumes each agent from the last transaction it saw, and once the RPC provider answering the call no longer held that transaction, the whole read failed and the agent was stuck there permanently: its profile kept loading, it just never learned anything new again. The crawl now notices that its bookmark has gone stale, re-reads from the top and carries on, and the same recovery landed for the secondary chains, where one had been parked on pruned blocks since April. The 1,083 stuck agents have been unstuck. The status page also learned to score this: an index that is erroring rather than lagging used to look perfectly fresh, and a lagging secondary chain no longer reports a platform outage nobody is having.",+ "tags": [+ "fix",+ "infra"+ ]+ },{"date": "2026-09-02","title": "Every showcased agent now has its own page",diff --git a/docs/ops/README.md b/docs/ops/README.mdindex 0e6228f0c..5958324c8 100644--- a/docs/ops/README.md+++ b/docs/ops/README.md@@ -15,6 +15,7 @@ Start with [gcp-production.md](gcp-production.md); it is the complete operationa| [production-log-triage.md](production-log-triage.md) | Known error/warning signature map: what each recurring log signature means and the fix. || [forge-error-triage.md](forge-error-triage.md) | `npm run forge:errors`: which 3D generation failures actually recur, ranked by class and lane over a real window. || [payment-outcomes.md](payment-outcomes.md) | The payment-outcome board (`GET /api/ops/payment-outcomes`): verify-reject, settle-fail, replay, and sponsor-runway signals and how to read them in an incident. |+| [agent-index.md](agent-index.md) | The agent index crawls end to end: the cursor stall both chain legs share, the error-class table, the recovery script, and how the freshness sensor scores each leg. || [solana-rpc-lanes.md](solana-rpc-lanes.md) | The Solana RPC tier end to end: one-sweep diagnosis, per-lane method capability, what must rotate vs fail, config traps, recovery. || [llm-lanes.md](llm-lanes.md) | The LLM provider chain end to end: which rungs serve, why the paid ones are dead, how spend is metered, the one-command Claude rollout, and per-lane probes. || [page-audit.md](page-audit.md) | `scripts/page-audit.mjs`: authed Chromium sweep of every public page, console-error gated. |diff --git a/docs/ops/agent-index.md b/docs/ops/agent-index.mdnew file mode 100644index 000000000..b136689c2--- /dev/null+++ b/docs/ops/agent-index.md@@ -0,0 +1,129 @@+# Agent index: the crawl, its failure classes, and recovery++The agent index is what `/agents` and every agent profile read. Two crawls fill+it, and both are crons:++| Leg | Cron | Period | Cursor table | Code |+|---|---|---|---|---|+| Solana (leads) | `/api/cron/solana-attestations-crawl` | 10 min | `agent_event_cursor` | `api/_lib/solana-agent-events.js` |+| EVM (secondary) | `/api/cron/erc8004-crawl` | 15 min | `erc8004_crawl_cursor` | `erc8004CrawlChain` in `api/cron/[name].js` |++The Solana cron does two things in one tick: the attestation crawl first, then+`solanaEventSweep()`, which is the half that fills `agent_event_cursor`. Its+batch is `SOLANA_SWEEP_BATCH` and its wall-clock budget is `SOL_EVENT_BUDGET_MS`.++The freshness sensor is `api/_lib/ops/index-lag.js`. It reports on+`/api/healthz` and `/status` as `agent_index`, and reads only the cursor tables:+no RPC calls, because asking every chain for its head block on a public status+endpoint is both slow and a good way to get rate-limited.++## The one failure mode both legs share++**Our stored cursor points at history the provider no longer serves.** It is the+same bug on both chains, and it is silent by construction:++- Solana resumes each agent from the newest signature it saw, handed back as+ `getSignaturesForAddress({ until })`. That parameter is resolved against the+ answering node's own history, and the whole call fails with+ `Transaction <sig> not found` when that node does not have it. The lane router+ answers from whichever provider is not cooling and providers disagree about+ retention, so a cursor written by one lane is regularly unreadable by the next.+- EVM resumes at `last_block + 1`. A provider that has pruned that height answers+ `History has been pruned for this block`, and no smaller window fixes it,+ because the data is gone rather than the request being too wide.++In both cases the cursor is only written on the success path, so the next tick+presents the same dead value and gets the same error, forever. Nothing pages:+the crons catch per agent and per chain, so they keep returning 200 while the+directory quietly stops learning anything.++Both legs now recover on their own:+`api/_lib/solana/cursor-recovery.js` drops `until` and re-scans from the head,+and `isPrunedHistoryRejection()` skips the EVM cursor forward to the head. Both+are safe to replay, because every row lands through `on conflict do nothing`.++## Diagnose in three reads++```sh+# 1. What the sensor says, with its metrics.+curl -s https://three.ws/api/healthz \+ | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>{const j=JSON.parse(d);console.log(JSON.stringify(j.subsystems.subsystems.find(s=>s.name==="agent_index"),null,2))})'++# 2. Solana: group the cursors by error class. Signatures differ per row, so+# normalize them away or every row looks like its own unique failure.+node --env-file=.env.local -e "import('./api/_lib/db.js').then(async({sql})=>{console.table(await sql\`select regexp_replace(error,'[1-9A-HJ-NP-Za-km-z]{32,}','<sig>','g') as shape, count(*)::int as n from agent_event_cursor where chain='solana' and error is not null group by 1 order by 2 desc limit 10\`);process.exit(0)})"++# 3. EVM: the cursors that have stopped moving, oldest first.+node --env-file=.env.local -e "import('./api/_lib/db.js').then(async({sql})=>{console.table(await sql\`select chain_id, last_block, head_block, blocks_behind, chunk_size, updated_at, left(coalesce(last_error,''),70) as err from erc8004_crawl_cursor order by updated_at asc limit 10\`);process.exit(0)})"+```++Production logs, when `gcloud` is authenticated:++```sh+gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" textPayload:"solana-attestations-crawl"' \+ --freshness=24h --project aerial-vehicle-466722-p5+```++## Error classes and what each one means++| Class (from the grouped read above) | Meaning | Action |+|---|---|---|+| `failed to get signatures for address: Transaction <sig> not found` | The answering lane cannot resolve the stored cursor. | Self-healing. If a block of agents is already stuck, drain it with the recovery script below instead of waiting a full sweep cycle. |+| `solana rpc 429 @ <host>` | That lane is rate-limited. | Transient. The router cools the lane and retries; see [solana-rpc-lanes.md](solana-rpc-lanes.md) if it persists. |+| `solana rpc provider error -16401 @ <host>` | The lane gates this method behind a paid tier. | Per-method capability routing in `api/_lib/solana/connection.js`. Add the method to that lane's blocked set. |+| `fetch failed (…) @ <host>` | Transport fault on one lane. | Transient. Persistent means the lane is dead: drop it. |+| `agent_ref is not a Solana account key` | A directory row whose ref is not an account. | Never crawlable. It is recorded once rather than retried every tick. |+| EVM `has been pruned` | The provider dropped the blocks the cursor points at. | Self-healing: the crawl resumes at the head and reports the skipped span as `prunedSkip`. That span is a permanent gap; only an archive node can backfill it. |+| EVM `block range` / `response size` / `query returned more than` | The window was too wide. | Self-healing: `backoffChunkSize()` halves it in place and the width is persisted in `chunk_size`. |+| EVM `limit exceeded` | A plan or compute limit, **not** a range ceiling. | Shrinking does not help. Deliberately excluded from `isRangeRejection`. |+| EVM `HTTP 403 from <host>` | The last-resort keyless lane refused us. | Not recoverable in code. The chain needs a lane that answers from a datacenter IP; see the `rpcUrls` note in `api/_lib/erc8004-chains.js`. |++## Recovery: drain a wedged backlog now++```sh+node --env-file=.env.local scripts/heal-agent-event-cursors.mjs # report by class, writes nothing+node --env-file=.env.local scripts/heal-agent-event-cursors.mjs --apply # re-crawl every erroring agent+```++The sweep heals the same backlog on its own at one batch per tick; the script+exists so a large backlog drains in one pass rather than over a full cycle. It+is safe to re-run.++## How the sensor scores it++Solana leads, and here that is correctness rather than preference: a user+opening an agent profile reads the Solana index, so a secondary-chain cursor+that is months behind changes nothing they can see.++| Signal | Degraded | Down |+|---|---|---|+| Solana median cursor age | `SOLANA_LAG_DEGRADED_MIN` | `SOLANA_LAG_DOWN_MIN` |+| Solana cursors carrying an error | `SOLANA_ERROR_RATE_DEGRADED` | `SOLANA_ERROR_RATE_DOWN` |+| EVM worst cursor age | `EVM_LAG_DEGRADED_MIN` | `EVM_LAG_DOWN_MIN` |+| EVM worst backlog | `EVM_BLOCKS_BEHIND_DEGRADED` | `EVM_BLOCKS_BEHIND_DOWN` |++Read the numbers from `index-lag.js`; they are exported so nothing has to quote+them. Two rules shape the verdict:++- **An EVM-only fault caps at `degraded`.** Letting a stale secondary cursor+ drive the whole subsystem to `down` reports an outage no user is having, and+ it buries the Solana leg's real state.+- **The Solana error rate is scored, not just the cursor age.** The sweep stamps+ `last_indexed_at` on the failure path too (deliberately, so one unreadable+ account cannot hold the oldest-first queue head forever), which means a wedged+ agent looks exactly as fresh as a healthy one. Age alone cannot see this class.++`indexLagVerdict()` is pure and takes the metrics shape directly, so the+thresholds are exercised on synthetic metrics in `tests/agent-index.test.js`+rather than against a live database.++## Sweep cycle++A queue drained oldest-first sits at half its cycle time, so the median lag is+`sweepCycleMin(agents) / 2`. `sweepCycleMin()` derives the cycle from+`SOLANA_SWEEP_BATCH` and `SOLANA_SWEEP_PERIOD_MIN`, which means a directory that+outgrows its batch shows up as a number on the status surface instead of as an+unexplained median. Raising the batch costs one `getSignaturesForAddress` per+agent per tick; the real ceiling is `SOL_EVENT_BUDGET_MS`, not the constant, and+the sweep reports `truncated: true` when it runs out of budget mid-batch. Check+that flag before raising the batch again.diff --git a/scripts/heal-agent-event-cursors.mjs b/scripts/heal-agent-event-cursors.mjsnew file mode 100644index 000000000..3d231cddc--- /dev/null+++ b/scripts/heal-agent-event-cursors.mjs@@ -0,0 +1,89 @@+#!/usr/bin/env node+// Clear Solana agent-event cursors that are wedged on an unresolvable cursor.+//+// The sweep in api/_lib/solana-agent-events.js resumes each agent from the+// newest signature it saw, passed back as getSignaturesForAddress({ until }).+// The lane router answers from whichever RPC provider is not cooling, and+// providers disagree about which signatures they still hold, so a cursor written+// by one lane is regularly unresolvable by the next. That fails the whole call,+// the cursor is only written on the success path, and the agent stops indexing+// forever while the cron keeps returning 200.+//+// crawlAgentEvents() now recovers on its own (it drops `until`, re-scans from+// the head and clears the dead value), so the sweep heals the backlog by itself+// at one batch per tick. This script is the way to drain the backlog in one pass+// instead of waiting out a full sweep cycle, and the way to re-run the recovery+// on demand the next time a lane change wedges a block of agents.+//+// Usage:+// node --env-file=.env.local scripts/heal-agent-event-cursors.mjs # report only+// node --env-file=.env.local scripts/heal-agent-event-cursors.mjs --apply+// node --env-file=.env.local scripts/heal-agent-event-cursors.mjs --apply --limit 200+//+// Reads DATABASE_URL, so run it with --env-file=.env.local. Safe to re-run:+// every event lands through `on conflict do nothing`, so a re-scanned window+// inserts nothing twice, and an agent that is already healthy costs one RPC call.++import { sql } from '../api/_lib/db.js';+import { crawlAgentEvents, markAgentEventError } from '../api/_lib/solana-agent-events.js';++const args = process.argv.slice(2);+const apply = args.includes('--apply');+const limitArg = args.indexOf('--limit');+const limit = limitArg >= 0 ? Math.max(1, parseInt(args[limitArg + 1], 10) || 0) : 5000;++const shape = (e) => String(e || '').replace(/[1-9A-HJ-NP-Za-km-z]{32,}/g, '<sig>');++const rows = await sql`+ SELECT agent_ref, network, error+ FROM agent_event_cursor+ WHERE chain = 'solana' AND error IS NOT NULL+ ORDER BY last_indexed_at ASC+ LIMIT ${limit}+`;++const byShape = new Map();+for (const r of rows) byShape.set(shape(r.error), (byShape.get(shape(r.error)) || 0) + 1);++console.log(`${rows.length} erroring cursor(s), by error class:`);+for (const [k, n] of [...byShape.entries()].sort((a, b) => b[1] - a[1])) {+ console.log(` ${String(n).padStart(5)} ${k.slice(0, 110)}`);+}++if (!apply) {+ console.log('\nreport only. Re-run with --apply to re-crawl each of these agents.');+ process.exit(0);+}++let healed = 0;+let stillFailing = 0;+let events = 0;++for (const [i, row] of rows.entries()) {+ try {+ const r = await crawlAgentEvents({ agentRef: row.agent_ref, network: row.network || 'mainnet' });+ healed += 1;+ events += r.inserted;+ } catch (err) {+ stillFailing += 1;+ // Same contract as the cron: stamp the failure so one unreadable account+ // cannot hold the oldest-first queue head forever.+ await markAgentEventError({+ agentRef: row.agent_ref,+ network: row.network || 'mainnet',+ error: err?.message || String(err),+ }).catch(() => {});+ }+ if ((i + 1) % 50 === 0) {+ console.log(` ${i + 1}/${rows.length} healed=${healed} stillFailing=${stillFailing} events=${events}`);+ }+}++const [after] = await sql`+ SELECT count(*)::int AS agents, count(error)::int AS errored+ FROM agent_event_cursor WHERE chain = 'solana'+`;+const pct = after.agents ? ((after.errored / after.agents) * 100).toFixed(1) : '0.0';+console.log(`\nhealed ${healed}, still failing ${stillFailing}, ${events} new event(s).`);+console.log(`agent_event_cursor now: ${after.errored}/${after.agents} erroring (${pct}%).`);+process.exit(0);diff --git a/scripts/set-r2-cors.mjs b/scripts/set-r2-cors.mjsindex bcde23e0c..2767ebb67 100644--- a/scripts/set-r2-cors.mjs+++ b/scripts/set-r2-cors.mjs@@ -13,9 +13,7 @@* 2. Write — PUT of presigned uploads from the same web origins.** Usage:- * node scripts/set-r2-cors.mjs --probe # measure the LIVE policy from outside (no credentials at all)- * node scripts/set-r2-cors.mjs --probe --site=https://staging.example # probe another deployment- * node scripts/set-r2-cors.mjs --probe --key=thumb/x.png # read a specific object+ * node scripts/set-r2-cors.mjs --probe # measure the LIVE policy from outside (no bucket creds)* node scripts/set-r2-cors.mjs --get # read the live policy (needs an admin token)* node scripts/set-r2-cors.mjs --dry-run # print the policy, don't push* node scripts/set-r2-cors.mjs # apply (idempotent, needs an admin token)@@ -23,12 +21,9 @@* Where the credentials come from:* - `.env` / `.env.local` hold S3_ENDPOINT, S3_ACCESS_KEY_ID,* S3_SECRET_ACCESS_KEY, S3_BUCKET. The token normally checked in here is- * "Object Read & Write" scoped, which is NOT enough for Get/PutBucketCors.- * Put an "Admin Read & Write" R2 token in `.env.local` as- * R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY to use --get or to apply.- * - --probe needs NONE of them. With no credentials it discovers the public- * host from a live listing endpoint and the presigned-upload host from the- * auth-free /api/forge-upload, which is exactly what a browser sees.+ * "Object Read & Write" scoped, which is enough for --probe but NOT for+ * Get/PutBucketCors. Put an "Admin Read & Write" R2 token in `.env.local`+ * as R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY to use --get or apply.* - Production runtime values live on the Cloud Run service, readable with* `gcloud run services describe three-ws-api --region us-central1* --project aerial-vehicle-466722-p5 --format=yaml`. Those are the same@@ -60,41 +55,18 @@ function fallbackEndpointFromAccount(accountId) {if (!accountId) return null;return `https://${accountId.trim()}.r2.cloudflarestorage.com`;}-// Assign through this, never `process.env.X ||= y` directly: assigning an-// absent value to process.env stores the STRING "undefined", which is truthy.-// That silently defeated every "is storage configured?" check below. On a-// machine with no R2 vars the script skipped its own skip-gate and went on to-// call the bucket API with the literal credentials "undefined".-function aliasEnv(name, value) {- if (!process.env[name] && value) process.env[name] = value;-}-aliasEnv('S3_ENDPOINT', process.env.R2_ENDPOINT || fallbackEndpointFromAccount(process.env.R2_ACCOUNT_ID));-aliasEnv('S3_ACCESS_KEY_ID', process.env.R2_ACCESS_KEY_ID);-aliasEnv('S3_SECRET_ACCESS_KEY', process.env.R2_SECRET_ACCESS_KEY);-aliasEnv('S3_BUCKET', process.env.R2_BUCKET);--const flag = (name) => process.argv.includes(name);-const argValue = (name) => {- const hit = process.argv.find((a) => a.startsWith(`${name}=`));- return hit ? hit.slice(name.length + 1) : null;-};+process.env.S3_ENDPOINT ||= process.env.R2_ENDPOINT || fallbackEndpointFromAccount(process.env.R2_ACCOUNT_ID);+process.env.S3_ACCESS_KEY_ID ||= process.env.R2_ACCESS_KEY_ID;+process.env.S3_SECRET_ACCESS_KEY ||= process.env.R2_SECRET_ACCESS_KEY;+process.env.S3_BUCKET ||= process.env.R2_BUCKET;const required = ['S3_ENDPOINT', 'S3_ACCESS_KEY_ID', 'S3_SECRET_ACCESS_KEY', 'S3_BUCKET'];const missing = required.filter((k) => !process.env[k]);-const hasBucketCreds = missing.length === 0;--// --probe and --dry-run are exempt from this gate on purpose: neither calls the-// bucket API. --probe measures the policy the way a browser experiences it, over-// public HTTP, sourcing both of its URLs from the live site (see probe() below),-// and --dry-run only prints POLICY. Requiring credentials here would make the-// one command that verifies this policy unrunnable exactly where it is needed-// most: a fresh clone, or a machine whose keys have rotated out.-if (missing.length && !flag('--probe') && !flag('--dry-run')) {+if (missing.length) {// Deploy-time invocation: missing env is not a deploy failure. Local-dev// invocation: surface the hint. Either way, exit 0 so a CI step can chain.console.log(`[set-r2-cors] skipped, missing env: ${missing.join(', ')}`);console.log('[set-r2-cors] (local: drop R2_ACCOUNT_ID/R2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEY/R2_BUCKET in .env.local. Production values: gcloud run services describe three-ws-api --region us-central1 --project aerial-vehicle-466722-p5 --format=yaml)');- console.log('[set-r2-cors] (to measure the LIVE policy without any bucket credentials: node scripts/set-r2-cors.mjs --probe)');process.exit(0);}@@ -156,22 +128,16 @@ const PROBE_ORIGINS = [// this key is never created or read.const PROBE_WRITE_KEY = 'cors-preflight-probe.bin';-// Built on first use, never at import: --probe runs with no credentials at all-// and must not construct a client around an undefined endpoint.-let _s3;-function s3client() {- if (!_s3) {- _s3 = new S3Client({- region: 'auto',- endpoint: process.env.S3_ENDPOINT,- credentials: {- accessKeyId: process.env.S3_ACCESS_KEY_ID,- secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,- },- });- }- return _s3;-}+const flag = (name) => process.argv.includes(name);++const s3 = new S3Client({+ region: 'auto',+ endpoint: process.env.S3_ENDPOINT,+ credentials: {+ accessKeyId: process.env.S3_ACCESS_KEY_ID,+ secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,+ },+});const Bucket = process.env.S3_BUCKET;@@ -192,7 +158,7 @@ if (flag('--get')) {}if (flag('--dry-run')) {- console.log('Would apply to bucket:', Bucket || '(S3_BUCKET unset in this environment)');+ console.log('Would apply to bucket:', Bucket);console.log(JSON.stringify(POLICY, null, 2));process.exit(0);}@@ -205,7 +171,7 @@ try {process.exit(1);}try {- await s3client().send(new PutBucketCorsCommand({ Bucket, CORSConfiguration: POLICY }));+ await s3.send(new PutBucketCorsCommand({ Bucket, CORSConfiguration: POLICY }));} catch (err) {if (!explainAccessDenied(err, 'writing')) throw err;process.exit(1);@@ -230,27 +196,21 @@ if (JSON.stringify(before) === JSON.stringify(after)) {// Exits nonzero when the measurement disagrees with POLICY.async function probe() {const publicHost = process.env.S3_PUBLIC_DOMAIN || process.env.R2_PUBLIC_DOMAIN;- const site = (argValue('--site') || 'https://three.ws').replace(/\/$/, '');-- // Both probe URLs, with the live site as the fallback source for each. The- // site is the browser's own view of the bucket, so a machine with no bucket- // credentials at all still measures the real policy.- const readUrl = (await probeReadUrl(publicHost, site)) || null;- if (!readUrl) {- console.error('--probe could not find a public object to read.');- console.error('Pass one with --key=<object key> (needs S3_PUBLIC_DOMAIN), or check that');- console.error(`${site} is reachable and still serves r2.dev asset URLs.`);+ if (!publicHost) {+ console.error('--probe needs S3_PUBLIC_DOMAIN (the public r2.dev or custom-domain host).');return false;}- const writeUrl = await probeWriteUrl(site);- if (!writeUrl) {- console.error('--probe could not determine the presigned-upload host.');- console.error(`Set S3_ENDPOINT + S3_BUCKET, or check that ${site}/api/forge-upload is reachable.`);+ const key = await probeKey();+ if (!key) {+ console.error('--probe could not find a public object to read. Pass one with --key=<object key>.');return false;}- console.log(`Probing ${Bucket || hostLabel(writeUrl)} (read: ${readUrl})\n`);+ const readUrl = `${publicHost.replace(/\/$/, '')}/${encodeR2Key(key)}`;+ const writeUrl = `${process.env.S3_ENDPOINT.replace(/\/$/, '')}/${Bucket}/${PROBE_WRITE_KEY}`;++ console.log(`Probing ${Bucket} (read: ${readUrl})\n`);console.log(`${'ORIGIN'.padEnd(38)} ${'READ'.padEnd(6)} ${'WRITE'.padEnd(6)} EXPECTED`);let ok = true;@@ -277,91 +237,19 @@ async function probe() {return ok;}-// A world-readable object URL on the public host, so the read probe measures-// CORS rather than auth. Three sources, best first: an explicit --key, the-// bucket listing (needs credentials), and finally the live site, which hands-// out public asset URLs on the very host we want to measure.-async function probeReadUrl(publicHost, site) {- const explicit = argValue('--key');- if (explicit) {- if (!publicHost) {- console.error('--key needs S3_PUBLIC_DOMAIN to build a URL from.');- return null;- }- return `${publicHost.replace(/\/$/, '')}/${encodeR2Key(explicit)}`;- }-- if (hasBucketCreds && publicHost) {- try {- const r = await s3client().send(new ListObjectsV2Command({ Bucket, Prefix: 'thumb/', MaxKeys: 1 }));- const key = r.Contents?.[0]?.Key;- if (key) return `${publicHost.replace(/\/$/, '')}/${encodeR2Key(key)}`;- } catch {- // Fall through to the site: an object-scoped token that cannot list- // is exactly the case this fallback exists for.- }- }-- return discoverPublicAssetUrl(site, publicHost);-}--// Scrape one public bucket URL out of a live listing endpoint. These return-// user-generated avatars, whose media resolves to the public bucket host via-// publicUrl() in api/_lib/r2.js, the same URL a third-party embed would load.-async function discoverPublicAssetUrl(site, publicHost) {- const hostPattern = publicHost- ? escapeRegExp(publicHost.replace(/\/$/, ''))- : 'https:\\/\\/[a-z0-9-]+\\.r2\\.dev';- const re = new RegExp(`${hostPattern}\\/[^"'\\s\\\\]+\\.(?:glb|png|jpg|webp)`, 'i');- for (const path of ['/api/marketplace/agents?limit=12', '/api/avatars?limit=12']) {- try {- const res = await fetch(`${site}${path}`, { headers: { accept: 'application/json' } });- if (!res.ok) continue;- const hit = (await res.text()).match(re);- if (hit) return hit[0];- } catch {- continue;- }- }- return null;-}--// The presigned-upload host. From env when credentials are present, else from-// the site's own auth-free presign endpoint, which returns a real upload URL on-// the exact host the browser PUTs to. Only its OPTIONS preflight is ever sent,-// so no object is created.-async function probeWriteUrl(site) {- if (process.env.S3_ENDPOINT && Bucket) {- return `${process.env.S3_ENDPOINT.replace(/\/$/, '')}/${Bucket}/${PROBE_WRITE_KEY}`;- }+// A key whose object is world-readable, so the read probe measures CORS rather+// than auth. Prefers a caller-supplied --key, else the first thumbnail.+async function probeKey() {+ const explicit = process.argv.find((a) => a.startsWith('--key='));+ if (explicit) return explicit.slice('--key='.length);try {- const res = await fetch(`${site}/api/forge-upload`, {- method: 'POST',- headers: { 'content-type': 'application/json', 'x-forge-client': 'set-r2-cors-probe' },- body: JSON.stringify({ content_type: 'image/png', size_bytes: 1024 }),- });- if (!res.ok) return null;- const { upload_url: uploadUrl } = await res.json();- if (!uploadUrl) return null;- const u = new URL(uploadUrl);- return `${u.origin}${u.pathname}`;+ const r = await s3.send(new ListObjectsV2Command({ Bucket, Prefix: 'thumb/', MaxKeys: 1 }));+ return r.Contents?.[0]?.Key || null;} catch {return null;}}-function hostLabel(url) {- try {- return new URL(url).hostname.split('.')[0];- } catch {- return 'bucket';- }-}--function escapeRegExp(text) {- return text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');-}-// True when the browser would be allowed through: a matching// Access-Control-Allow-Origin on the actual request for GET, and a non-error// preflight for PUT (which never touches a real object, so the key need not exist).@@ -405,7 +293,7 @@ function encodeR2Key(key) {async function getCors() {try {- const r = await s3client().send(new GetBucketCorsCommand({ Bucket }));+ const r = await s3.send(new GetBucketCorsCommand({ Bucket }));return { CORSRules: r.CORSRules || [] };} catch (err) {if (err?.name === 'NoSuchCORSConfiguration' || err?.$metadata?.httpStatusCode === 404) {diff --git a/tests/agent-index.test.js b/tests/agent-index.test.jsindex 624d2bb44..caa663809 100644--- a/tests/agent-index.test.js+++ b/tests/agent-index.test.js@@ -25,11 +25,17 @@ import {sanitizeText,sanitizePayload,} from '../api/_lib/onchain-events.js';-import { sweepCycleMin } from '../api/_lib/ops/index-lag.js';+import {+ sweepCycleMin,+ indexLagVerdict,+ SOLANA_ERROR_RATE_DEGRADED,+ SOLANA_ERROR_RATE_DOWN,+} from '../api/_lib/ops/index-lag.js';import {nextChunkSize,backoffChunkSize,isRangeRejection,+ isPrunedHistoryRejection,catchUpCandidates,rank,} from '../api/cron/[name].js';@@ -416,6 +422,38 @@ describe('backoffChunkSize', () => {});});+describe('isPrunedHistoryRejection', () => {+ it('recognises a provider that no longer holds the blocks the cursor points at', () => {+ // Verbatim from erc8004_crawl_cursor.last_error on 2026-09-02, on a chain+ // whose cursor had not moved since 2026-04-28.+ expect(+ isPrunedHistoryRejection('RPC -32701: History has been pruned for this block. To remove restriction'),+ ).toBe(true);+ expect(isPrunedHistoryRejection('missing trie node 0xabc (path )')).toBe(true);+ });++ it('does NOT claim a range rejection or a plan limit is pruned history', () => {+ // These have their own recovery (shrink the window). Reading them as+ // pruned would skip the cursor to the head and silently drop real blocks.+ expect(isPrunedHistoryRejection('block range is too wide (maximum 1024)')).toBe(false);+ expect(isPrunedHistoryRejection('limit exceeded')).toBe(false);+ expect(isPrunedHistoryRejection('HTTP 403')).toBe(false);+ expect(isPrunedHistoryRejection(null)).toBe(false);+ expect(isPrunedHistoryRejection('')).toBe(false);+ });++ it('is disjoint from isRangeRejection, so exactly one recovery can claim a fault', () => {+ for (const msg of [+ 'RPC -32701: History has been pruned for this block',+ 'block range is too wide (maximum 1024)',+ 'query returned more than 10000 results',+ 'missing trie node',+ ]) {+ expect(isPrunedHistoryRejection(msg) && isRangeRejection(msg)).toBe(false);+ }+ });+});+describe('isRangeRejection', () => {it('recognises how each provider words a range it will not serve', () => {// Verbatim messages observed from the configured lanes on 2026-08-14.@@ -621,3 +659,96 @@ describe('rank', () => {expect(rank({ blocks_behind: 0, updated_at: new Date(NOW + 60_000).toISOString() }, NOW)).toBe(0);});});++// ─── Index verdict ───────────────────────────────────────────────────────────+//+// Measured on production 2026-09-02: the sensor reported `down` on a secondary+// chain whose cursor had not moved since April, while the Solana leg it was+// burying had 1,101 of 1,604 cursors wedged on an unresolvable-cursor error and+// scored `ok` on a 63-minute median. Both halves of that are covered here: an+// EVM-only fault must not claim an outage users are not having, and a Solana+// leg that is erroring rather than lagging must stop reading as fresh.++/** A synthetic readIndexLag() shape, healthy unless overridden. */+function lagFixture({ solana = {}, evm = {} } = {}) {+ return {+ solana: {+ medianLagMin: 60,+ worstLagMin: 130,+ agents: 1604,+ uncrawled: 0,+ errored: 0,+ events: 1510,+ batch: SOLANA_SWEEP_BATCH,+ sweepCycleMin: 140,+ ...solana,+ },+ evm: {+ lagMin: 15,+ chains: 22,+ configuredChains: 22,+ uncrawledChains: 0,+ staleChains: 0,+ behindChains: 0,+ worstBlocksBehind: 0,+ worstChainId: null,+ worstChainName: null,+ events: 107296,+ ...evm,+ },+ lastEventAt: null,+ lastIndexedAt: null,+ };+}++describe('indexLagVerdict', () => {+ it('is ok when both legs are fresh', () => {+ expect(indexLagVerdict(lagFixture()).status).toBe('ok');+ });++ it('caps an EVM-only fault at degraded and says why', () => {+ // The exact production shape: a cursor stalled for months and a backlog+ // far past the down threshold, with Solana fresh.+ const verdict = indexLagVerdict(+ lagFixture({ evm: { lagMin: 183_014, worstBlocksBehind: 17_396_220, behindChains: 1, staleChains: 3 } }),+ );+ expect(verdict.status).toBe('degraded');+ expect(verdict.detail).toContain('the Solana index users read is fresh');+ });++ it('still reports down when Solana itself is the leg that is behind', () => {+ const verdict = indexLagVerdict(lagFixture({ solana: { medianLagMin: 400 } }));+ expect(verdict.status).toBe('down');+ // The cap is for EVM-only faults; it must never soften a Solana outage.+ expect(verdict.detail).not.toContain('the Solana index users read is fresh');+ });++ it('scores a wedged Solana leg on its error rate, not just its cursor age', () => {+ // 1,101 of 1,604 erroring on a fresh 63-minute median: `ok` before this.+ const verdict = indexLagVerdict(lagFixture({ solana: { medianLagMin: 63, errored: 1101 } }));+ expect(verdict.status).toBe('down');+ expect(verdict.detail).toContain('1101 erroring');+ expect(verdict.hint).toContain('agent_event_cursor');+ });++ it('degrades between the error-rate thresholds and clears below them', () => {+ const agents = 1000;+ const degraded = indexLagVerdict(+ lagFixture({ solana: { agents, errored: Math.ceil(agents * SOLANA_ERROR_RATE_DEGRADED) } }),+ );+ expect(degraded.status).toBe('degraded');++ const down = indexLagVerdict(+ lagFixture({ solana: { agents, errored: Math.ceil(agents * SOLANA_ERROR_RATE_DOWN) } }),+ );+ expect(down.status).toBe('down');++ const clear = indexLagVerdict(lagFixture({ solana: { agents, errored: 10 } }));+ expect(clear.status).toBe('ok');+ });++ it('reports a never-crawled index as warming up rather than broken', () => {+ const verdict = indexLagVerdict(lagFixture({ solana: { agents: 0 }, evm: { chains: 0 } }));+ expect(verdict.status).toBe('unknown');+ });+});diff --git a/tests/solana-attestations-cursor-recovery.test.js b/tests/solana-attestations-cursor-recovery.test.jsindex 5aae2a688..0467e4222 100644--- a/tests/solana-attestations-cursor-recovery.test.js+++ b/tests/solana-attestations-cursor-recovery.test.js@@ -7,7 +7,7 @@// These cover the recovery: detect that class of error, re-scan from the head.import { describe, it, expect } from 'vitest';-import { isPrunedCursorError, signaturesSinceCursor } from '../api/_lib/solana-attestations.js';+import { isPrunedCursorError, signaturesSinceCursor } from '../api/_lib/solana/cursor-recovery.js';// The message shape a Solana RPC actually returns for a pruned `until`.const PRUNED = new Error(