Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(x402): stop a discovery sweep that straddles a rebuild from losing resources

nirholas7afb5bb95c5b
diff --git a/api/_lib/x402/discovery-resources.js b/api/_lib/x402/discovery-resources.js
index 9b3ac4e5f..6fa7becb4 100644
--- a/api/_lib/x402/discovery-resources.js
+++ b/api/_lib/x402/discovery-resources.js
@@ -150,11 +150,36 @@ export function projectDiscoveryResources(doc, { type, limit, offset, lastUpdate
// pagination sweep costs one build. A failed rebuild serves the last good doc
// rather than 500ing the crawler mid-sweep.
const DOC_TTL_MS = 300_000;
+
+// A sweep that straddles a rebuild loses rows. The crawler pages by offset with
+// no cursor and no snapshot token, so every row is served exactly once only if
+// one sweep reads one build: a rebuild landing between two pages shifts every
+// row after the churn point, which re-serves some and skips as many. Measured
+// on the live catalog 2026-09-02, a 46-page sweep that began ~50 s before the
+// TTL expired returned 4,519 rows holding 4,499 distinct resources.
+// So pages after the first keep the build the sweep started on past the TTL,
+// until this grace window closes; offset 0 always takes the fresh build. The
+// window bounds the staleness a slow crawler can pin (a 4,500-row catalog is
+// ~46 requests, far inside it), and a concurrent sweep starting at offset 0
+// still forces the rebuild both then see, because offset is the only sweep
+// identity the wire format carries.
+const DOC_SWEEP_GRACE_MS = 600_000;
+
+// True when the cached build of age `ageMs` may answer this request.
+// `continuation` marks a page past the first, i.e. a sweep already in flight.
+export function canServeCachedDoc({ ageMs, continuation }) {
+ if (!Number.isFinite(ageMs) || ageMs < 0) return false;
+ if (ageMs < DOC_TTL_MS) return true;
+ return Boolean(continuation) && ageMs < DOC_TTL_MS + DOC_SWEEP_GRACE_MS;
+}
+
let docCache = { at: 0, doc: null, inflight: null };
-async function getDiscoveryDoc() {
+async function getDiscoveryDoc({ continuation = false } = {}) {
const now = Date.now();
- if (docCache.doc && now - docCache.at < DOC_TTL_MS) return docCache.doc;
+ if (docCache.doc && canServeCachedDoc({ ageMs: now - docCache.at, continuation })) {
+ return docCache.doc;
+ }
if (!docCache.inflight) {
docCache.inflight = buildX402DiscoveryDoc()
.then((doc) => {
@@ -177,7 +202,9 @@ async function getDiscoveryDoc() {
}
export async function listDiscoveryResources(query = {}) {
- const doc = await getDiscoveryDoc();
+ const doc = await getDiscoveryDoc({
+ continuation: (Number.parseInt(query.offset, 10) || 0) > 0,
+ });
return projectDiscoveryResources(doc, {
...query,
lastUpdated: docCache.at ? new Date(docCache.at).toISOString() : undefined,
diff --git a/api/ops/payment-outcomes.js b/api/ops/payment-outcomes.js
index 763b7f8f2..91dccd7cb 100644
--- a/api/ops/payment-outcomes.js
+++ b/api/ops/payment-outcomes.js
@@ -144,6 +144,26 @@ async function sponsorBoard() {
return {
configured: Boolean(sponsor?.address),
address: sponsor?.address || null,
+ // Every configured ring wallet against its OWN floor, not the sponsor alone.
+ // "the wallets are dry" and "the sponsor is under the settle floor" are
+ // different outages with opposite fixes and identical symptoms from outside
+ // (the premise of the x402-economy-triage runbook), and a board that shows
+ // only the sponsor cannot tell them apart: a starved payer reads as healthy.
+ wallets: (ring.wallets || [])
+ .filter((w) => w.configured)
+ .map((w) => ({
+ role: w.role,
+ address: w.address,
+ sol: w.sol ?? null,
+ sol_floor: w.sol_floor ?? null,
+ sol_low: Boolean(w.sol_low),
+ usdc: w.usdc ?? null,
+ usdc_floor: w.usdc_floor ?? null,
+ usdc_low: Boolean(w.usdc_low),
+ })),
+ // Pre-rendered breach lines from the monitor, so the board's wording and the
+ // alert's wording are the same string rather than two drifting phrasings.
+ breaches: ring.breaches || [],
sol: sponsor?.sol ?? null,
// The ring's 1.5x watch floor (what `below_floor` compares against) and the
// facilitator's hard floor (where settling actually stops) are different
diff --git a/prompts/finish/backlog-01-x402-settle-runway.md b/prompts/finish/backlog-01-x402-settle-runway.md
index d53623e61..27e86f124 100644
--- a/prompts/finish/backlog-01-x402-settle-runway.md
+++ b/prompts/finish/backlog-01-x402-settle-runway.md
@@ -1,9 +1,46 @@
# 01. x402 settle: clear `fee_runway_exhausted` and hold the rate above 90%
-Read [00-INDEX.md](backlog-00-INDEX.md) first. This is the highest-priority open item:
-the only subsystem reporting **down** on the live healthz.
-
-## What is wrong
+Read [00-INDEX.md](backlog-00-INDEX.md) first.
+
+> ## Status, re-measured 2026-09-02 (read this before the body below)
+>
+> Every agent-doable line of this order is now shipped. What remains is capital,
+> and it is owner-owned. The body below is kept for its diagnosis method, but
+> four of its factual claims have been overtaken; do not act on them:
+>
+> 1. **The two config levers are already applied in production.**
+> `GET /api/x402/runway-lab` reports `runway_days: 1` and the 2026-08-01
+> entry in [PROGRESS.md](backlog-PROGRESS.md) records
+> `ECONOMY_MASTER_OPERATING_SOL=0.3` landing with it. Do not re-apply them.
+> 2. **Lever 2 is inert at this balance and was struck on 2026-08-02.** The
+> governor's budget is `max(minBudgetLamports, spendable / runwayDays)`. With
+> the sponsor at 0.0037 SOL the 10,000,000 lamport heartbeat floor already
+> exceeds spendable, so runway days change nothing.
+> 3. **The upstream cause is NOT a mis-tuned self-heal trigger.** The deficit has
+> been positive and the reclaim leg has been running for weeks. It cannot
+> complete: the wallets it targets are encrypted under the
+> WALLET_ENCRYPTION_KEY retired in the 2026-07 host migration and fail
+> AES-GCM decryption. 0.49 SOL is stranded that way, 0.35 of it customer
+> money. See `docs/ops/wallet-key-migration.md`.
+> 4. **The numbers below are a 2026-08-01 snapshot.** As of 2026-09-02 settle is
+> 55.0% and `degraded`, not 25.9% and `down`; demand has fallen to 177
+> attempts/hour from 3,726; measured burn is 0.0133 SOL/day spent against
+> ~0.043 SOL/day to serve every attempt.
+>
+> **The one thing that lifts the settle rate is SOL in two wallets.** The ring
+> payer sits 1,592 lamports under its 2,000,000 floor and the sponsor has
+> 1,727,883 spendable lamports. Send SOL to the economy master
+> `WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`, never to per-agent wallets.
+> That is stop-and-ask gate 1 and it is the owner's call.
+>
+> Shipped against this order: the caller-side fee admission gate
+> (`assessFeeAdmission`), the governor config recurrence guard
+> (`tests/x402-wallet-fee-governor.test.js`), `WALLET_ENCRYPTION_KEY_PREVIOUS`
+> key rotation, and the dry-run reclaim key gate plus the corrected
+> `sponsor_floor` hint (`afd349790`, 2026-09-02). Full evidence per line is in
+> [PROGRESS.md](backlog-PROGRESS.md).
+
+## What is wrong (as measured 2026-08-01)
`x402_settle` is at 25.9% (504 of 1948 paid attempts over 3 hours). The rail
faults look external (`http_502` x1180) but they are **our own facilitator
diff --git a/tests/x402-discovery-resources.test.js b/tests/x402-discovery-resources.test.js
index 63dec03b6..d9cf162e0 100644
--- a/tests/x402-discovery-resources.test.js
+++ b/tests/x402-discovery-resources.test.js
@@ -3,6 +3,7 @@ import {
toV1Accept,
toV1Item,
projectDiscoveryResources,
+ canServeCachedDoc,
DISCOVERY_DEFAULT_LIMIT,
DISCOVERY_MAX_LIMIT,
} from '../api/_lib/x402/discovery-resources.js';
@@ -212,3 +213,40 @@ describe('projectDiscoveryResources', () => {
expect(page.items.map((i) => i.metadata.toolName)).toEqual(['inspect', 'render', 'validate']);
});
});
+
+// ── Sweep coherence across a catalog rebuild ─────────────────────────────────
+//
+// The crawler pages by offset with no cursor, so a rebuild landing between two
+// pages shifts every row after the churn point: some rows come back twice and
+// as many are never served. Pages after the first therefore hold the build the
+// sweep started on past the freshness TTL, up to a bounded grace window.
+
+describe('canServeCachedDoc', () => {
+ const TTL = 300_000;
+ const GRACE = 600_000;
+
+ it('serves a fresh build to any page', () => {
+ expect(canServeCachedDoc({ ageMs: 0, continuation: false })).toBe(true);
+ expect(canServeCachedDoc({ ageMs: TTL - 1, continuation: true })).toBe(true);
+ });
+
+ it('rebuilds for the first page of a sweep once the build is stale', () => {
+ expect(canServeCachedDoc({ ageMs: TTL, continuation: false })).toBe(false);
+ expect(canServeCachedDoc({ ageMs: TTL + 1, continuation: false })).toBe(false);
+ });
+
+ it('keeps a sweep on the build it started on past the TTL', () => {
+ expect(canServeCachedDoc({ ageMs: TTL, continuation: true })).toBe(true);
+ expect(canServeCachedDoc({ ageMs: TTL + GRACE - 1, continuation: true })).toBe(true);
+ });
+
+ it('bounds how long a slow crawler can pin a stale build', () => {
+ expect(canServeCachedDoc({ ageMs: TTL + GRACE, continuation: true })).toBe(false);
+ });
+
+ it('refuses an unusable age rather than serving an unknown build', () => {
+ expect(canServeCachedDoc({ ageMs: Number.NaN, continuation: true })).toBe(false);
+ expect(canServeCachedDoc({ ageMs: -1, continuation: true })).toBe(false);
+ expect(canServeCachedDoc({ ageMs: Number.POSITIVE_INFINITY, continuation: true })).toBe(false);
+ });
+});