Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

31.4 KB
Raw Blame
// @ts-check
// x402 settle-success-rate sensor — the health signal the July 2026 502 wave
// proved was missing.
//
// The existing checks answer "is the ring ARMED?" (checkRing → invariants,
// checkX402Config → pay-to + fee payer present) and "is discovery UP?"
// (uptime-check probes /.well-known/x402.json). All three read healthy while a
// third of real settlements silently fail: from 2026-07-09 to 2026-07-17 the
// self-facilitator rejected ~300 settles/6h (same-signature collisions surfaced
// as `http_502`) yet every sensor above stayed green because the ring was
// configured, discovery answered 200, and *some* payments still landed. Eight
// days, no page. This module closes that hole by reading the only authoritative
// signal — the outcome the ring records for every paid call in
// `x402_autonomous_log` — and reporting the settle SUCCESS RATE.
//
// What counts, and the nuance that makes the signal usable
// --------------------------------------------------------
// `success=true, amount_atomic>0`  → a settlement LANDED (the numerator).
// A failure only counts against the rate when it is a PAYMENT-RAIL fault:
// facilitator/settle 5xx, a 402 (payment rejected), or an RPC/broadcast/confirm
// fault. Everything else the ring records is deliberately excluded, because the
// log is noisy with things that are NOT settle failures:
//   - caller/endpoint mismatches — http_400/404/405/409 (the ring POSTing to a
//     GET-only endpoint, or passing args a handler rejects). ~800/6h in prod:
//     including them would peg the sensor red and bury a real settle regression.
//   - benign guards — cap_would_exceed, insufficient_payer_usdc, breaker_tripped,
//     wallet_unconfigured, no_solana_accept: the ring CHOOSING not to pay. Ring
//     funding has its own monitor (x402/wallet-balance-monitor.js).
//   - downstream pipeline notes — *_calls_failed, config_missing: the payment
//     settled (or was never attempted); the failure is in value extraction.
// The fault set is an allowlist of rail faults, kept broad on the rail dimension
// (any 5xx, any RPC/broadcast/settle/confirm/simulation/timeout signature) so a
// NEW settle failure mode is caught without a code change — exactly what the 502
// wave needed. When a genuinely new benign reason starts showing up as a fault,
// add it to EXCLUDED_HINT and the caller docs, the same learn-once loop as the
// triage monitor's KNOWN_SIGNATURES.
//
// The window is 3h: the ring settles in bursts (an idle hour can hold zero
// settlements while it waits on funding or a settle-tick), so a short window
// reads `unknown` constantly. 3h reliably carries hundreds of attempts, smooths
// the bursts, and still surfaces a regression within hours — against an 8-day
// blind spot, decisive. Below MIN_ATTEMPTS the verdict is `unknown` (neutral,
// never pages): too little settling to judge is not a fault.
//
// The blind spot `unknown` opens, and why the governor count closes it
// --------------------------------------------------------------------
// "Too few attempts to judge" is the right verdict for a quiet ring and the
// WRONG one for a throttled one, and the two are identical in this log until you
// count the throttle. Since the wallet fee governor gained a caller-side
// admission check, a fee wallet with no daily budget left makes the ring SKIP
// its paid calls (reason `fee_runway_exhausted`) instead of settling them. Those
// skips are correctly not rail faults (nothing broke, we chose not to spend),
// but they also empty the numerator AND the denominator, so a rail that is
// 100% paced shut reads exactly like a rail nobody used. Measured on production
// 2026-08-01, before the admission check existed and while the same refusal
// still came back as a 502: 85,265 governed refusals against 562 rail-shaped
// failures, a settle rate of 25.9%, and a hint that sent the reader to the
// facilitator for a funding problem. `governorSkips` is counted separately here
// so the sensor can say "paced, top the fee wallet up" in both regimes.
//
// Consumed via gatherSubsystemHealth() → /api/healthz, /api/status, and the
// uptime-check escalation digest (pages on first sight, re-pages hourly, clears
// on recovery) — no new cron, no new alert path.

const WINDOW_INTERVAL = '3 hours';
// Enough settle attempts in-window to trust a rate. The ring runs hundreds/3h
// when active; 20 keeps a genuinely quiet period from reading as an outage.
const MIN_ATTEMPTS = 20;

// Rate bands. Vocabulary matches subsystem-health.js: `degraded` = functional
// but faulting; `down` = not functional. A rail settling ~73% of the time is
// impaired, not dead, so the collapse threshold sits well below that.
const OK_RATE = 0.9;
const DOWN_RATE = 0.5;

// Pipelines that record rows in x402_autonomous_log but never pay: their
// failures are a third party being slow, not our settle rail. The external
// service uptime monitor (`reliability`, price_atomic 0 in autonomous-registry.js)
// sends unpaid HEAD probes with a short timeout, and on 2026-09-30 its
// `This operation was aborted` rows for slow third-party hosts were 5 of the 60
// "rail faults" behind a degraded 74.9%, while not one of them touched a payment.
export const UNPAID_PROBE_PIPELINES = Object.freeze(['reliability']);

// A recorded failure is a SETTLE/PAYMENT-RAIL fault when its reason (the first
// `:`-delimited token of error_msg) matches these. Two rules: any 5xx or a 402,
// and any RPC/broadcast/settle/confirm/simulation/timeout signature.
const RAIL_STATUS = /^http_(5\d\d|402)$/i;
const RAIL_SIGNATURE =
	/broadcast_failed|settle_failed|not_confirmed|simulation|preflight|rpc_|insufficient_source|blockhash|aborted|operation was aborted|timeout/i;

/**
 * Is this reason token a settle/payment-rail fault (vs a caller error, a benign
 * guard, or a downstream note)?
 * @param {string} reason first `:`-token of error_msg
 * @returns {boolean}
 */
export function isRailFault(reason) {
	const r = String(reason || '');
	if (!r || r === 'none') return false;
	return RAIL_STATUS.test(r) || RAIL_SIGNATURE.test(r);
}

// The ring reports `no_solana_accept` when a 402 challenge carried no Solana
// accept it could pay. On OUR OWN endpoints that is never the ring's choice: it
// means buildRequirements() dropped the Solana accept, and the reason it drops
// one in production is sponsorKnownBelowFloor() (api/_lib/x402-paid-endpoint.js).
const NO_SOLANA_ACCEPT = 'no_solana_accept';
// The wallet fee governor's refusal, as both sides of the handshake record it:
// the caller-side admission check (api/_lib/x402/pay.js) skips the call with
// this reason, and the settle-path meter refuses with the same token. Never a
// rail fault: it is the platform pacing its own SOL burn, but it is the one
// benign reason that must still be COUNTED, because at volume it is the whole
// explanation for a rate or an attempt count that fell off a cliff.
const FEE_GOVERNOR = /^fee_runway_exhausted/i;
// The sponsor-floor refusal, as the ring records it. x402-spec.js answers 503
// `settlement_unavailable` for a fee wallet under its floor, so the reason token
// arrives either as the 503 prose or as the raw floor signature.
// `insufficient_lamports_for_fee` is the same starvation reported from inside
// the settle path (see runway-lab's bucketer, which files it under floor too).
// `sponsor_fee_unfunded` is the chain's own verdict on the same wallet, raised by
// assertSettleable() when simulation returns InsufficientFundsForRent on account
// index 0 (the fee payer, by definition) and that fee payer is OUR sponsor. It
// belongs here and not in RAIL_SIGNATURE: before it had its own class it read as
// `simulation_failed`, which /simulation/ matches, so a starved sponsor was
// counted as a rail fault. The buyer-side twin `payer_fee_unfunded` deliberately
// stays out: that wallet is not ours and needs no operator action.
const SPONSOR_FLOOR = /^(settlement temporarily unavailable|fee_wallet_below_floor|sponsor_sol_floor|sol_floor|insufficient_lamports_for_fee|sponsor_fee_unfunded)/i;

/**
 * Why did the settle rate fall? Two failure shapes are indistinguishable in the
 * rate alone and have opposite fixes, and the generic hint sent operators at the
 * wrong one for a full shift on 2026-07-29:
 *
 *   FAULTS ROSE      the denominator grew; settles are being attempted and
 *                    rejected. Look at the facilitator.
 *   NUMERATOR FELL   rail faults stayed flat while settlements stopped
 *                    happening, because the Solana accept was withdrawn from
 *                    every challenge and the (Solana-only) ring had nothing it
 *                    could pay. Look at the sponsor's SOL balance. On 07-29 this
 *                    read as "settle 22%" while faults sat at their normal
 *                    ~100/h and `no_solana_accept` went 0 to 374/h.
 *
 *   FLOOR REFUSED    the accept is still advertised and buyers still sign real
 *                    payments, but OUR OWN facilitator refuses every one of them
 *                    at the fee-wallet floor gate. Same wallet, same remedy as
 *                    WITHDRAWN, opposite evidence trail: the proof is in the
 *                    facilitator's reject book, not in the challenge. Measured
 *                    2026-09-09: 0 no_solana_accept, 164
 *                    `fee_wallet_below_floor:1167627<2000000`, and 126 `http_402`
 *                    from paid replays the gate turned away. Telling that
 *                    operator "do NOT start at the facilitator" points them away
 *                    from the only table that names the wallet and both numbers.
 *
 *   BUDGET PACED     nothing failed and nothing was withdrawn: the wallet fee
 *                    governor spent today's fee budget and is refusing the rest
 *                    of the day's settles on purpose. Look at the fee wallet's
 *                    SOL, not at the rail. On 2026-08-01 this shape was 85,265
 *                    governed refusals against 562 rail-shaped failures, and the
 *                    generic rail hint below sent the reader to the facilitator.
 *
 * Solana is the home chain, so a withdrawn Solana accept is a platform outage,
 * not the benign "ring chose not to pay" the rate deliberately excludes.
 * @param {{ noSolanaAccept: number, floorSignals: number, governorSkips?: number,
 *   settled: number, faults: number }} s
 * @returns {{ cause: 'sponsor_floor'|'fee_governor'|'rail',
 *   mechanism: 'accept_withdrawn'|'settle_refused'|'paced'|'rail', hint: string }}
 */
export function diagnoseSettleDrop({ noSolanaAccept, floorSignals, governorSkips = 0, settled, faults }) {
	// A floor refusal is proof on its own. Absent that, treat the home chain going
	// unpayable more often than it settles as the same cause: the accept is
	// withdrawn for most of the window, which is what a flapping floor looks like
	// through a cache-backed, fail-open check.
	//
	// The hard floor outranks the governor deliberately: both want SOL in the same
	// wallet, but under the floor EVERY settle fails closed, while a spent budget
	// still settles at the paced rate. Report the harder stop when both are lit.
	if (floorSignals > 0 || (noSolanaAccept > 0 && noSolanaAccept > settled)) {
		// Same wallet, same remedy, two opposite evidence trails. Reporting the
		// withdrawal unconditionally was wrong exactly when `noSolanaAccept` is 0:
		// the accept was still on every challenge, buyers signed real payments, and
		// our own facilitator turned each one away at the floor gate. The remedy
		// paragraph below is shared because the fix is identical; only the sentence
		// naming the mechanism and the place to look differs.
		const withdrawn = noSolanaAccept > 0;
		const mechanism = /** @type {const} */ (withdrawn ? 'accept_withdrawn' : 'settle_refused');
		const opening = withdrawn
			? 'The Solana accept is being WITHDRAWN, not rejected: ' +
				`${noSolanaAccept} no_solana_accept + ${floorSignals} floor refusal(s) against ${faults} rail faults. ` +
				'sponsorKnownBelowFloor() drops Solana from every 402 challenge while the sponsor sits under ' +
				'X402_SPONSOR_SOL_FLOOR_LAMPORTS, so the ring has nothing payable and settlements stop. ' +
				'Do NOT start at the facilitator: with no accept on the challenge it never saw these calls. ' +
				'Check the sponsor balance, then let the free self-heal run: '
			: 'Settles are being REFUSED at the sponsor floor, not withdrawn: ' +
				`0 no_solana_accept + ${floorSignals} floor refusal(s) against ${faults} rail faults. ` +
				'The Solana accept is still on every challenge and buyers are signing real payments; our own ' +
				'facilitator rejects each one at the floor gate, because the sponsor fee wallet sits under ' +
				'X402_SPONSOR_SOL_FLOOR_LAMPORTS. That is why the ring records them as `http_402` on the paid ' +
				'replay. START at the facilitator here: its reject book is the only ' +
				'place that names the wallet and both numbers ' +
				'(`fee_wallet_below_floor:<held><<floor>` in x402_self_facilitator_log.reject_reason, also ' +
				'under x402.self_facilitator.settle.fail_reasons on /api/healthz). Read the held number off ' +
				'that token before anything else: it is the wallet the money has to reach. Then let the free ' +
				'self-heal run: ';
		return {
			cause: /** @type {const} */ ('sponsor_floor'),
			mechanism,
			hint:
				opening +
				'POST /api/cron/treasury-topup?dry=1 (Bearer CRON_SECRET) to see the plan, then without ?dry=1 ' +
				'to apply. READ agent_reclaim.failed in that plan before you trust its total: a wallet whose ' +
				'secret does not decrypt is reported at stage `recover` and its SOL is unreachable until ' +
				'WALLET_ENCRYPTION_KEY (or WALLET_ENCRYPTION_KEY_PREVIOUS) can open it, so no cron run and no ' +
				'RPC tier will ever move it. Owner SOL is needed when every reclaim source reports ' +
				'at_or_below_floor or secret_undecryptable AND that plan reports ' +
				'skipped_floor_held_sol 0. A floor skip carries its numbers ' +
				'(at_or_below_floor:<held><<keep>), so a non-zero held total means the SOL exists on a ' +
				'platform wallet and a floor is fencing it, not that the fleet is empty. A ' +
				'below_min_sweep:<excess><<min> skip is a wallet ABOVE its keep line whose excess is under ' +
				'ECONOMY_SWEEPBACK_MIN_SOL: that SOL is released by the sweep minimum, not by a floor. ' +
				'See docs/ops/production-log-triage.md.',
		};
	}
	// Nothing failed and nothing was withdrawn: we are refusing our own settles to
	// pace the fee wallet's SOL. Dominance over faults is the test, because a
	// handful of governed skips is the governor working as designed on a healthy
	// rail, while a landslide of them IS the reason the rate moved.
	if (governorSkips > faults && governorSkips > 0) {
		return {
			cause: /** @type {const} */ ('fee_governor'),
			mechanism: /** @type {const} */ ('paced'),
			hint:
				'This is a GOVERNED THROTTLE, not a rail fault: ' +
				`${governorSkips} settle(s) skipped by the wallet fee governor against ${faults} rail fault(s). ` +
				'The fee wallet spent its daily SOL fee budget (spendable SOL / X402_WALLET_FEE_RUNWAY_DAYS, ' +
				'floored at X402_WALLET_FEE_MIN_BUDGET_LAMPORTS), so the rail is pacing itself until the budget ' +
				'refills. Do NOT debug the facilitator and do NOT lower the sponsor floor. Read the live budget ' +
				'at GET /api/x402/runway-lab, then fund the fee wallet: POST /api/cron/treasury-topup?dry=1 ' +
				'(Bearer CRON_SECRET) shows what the free self-heal can reclaim, and owner SOL is needed only ' +
				'when every reclaim source reports at_or_below_floor or an undecryptable secret AND that ' +
				'plan reports skipped_floor_held_sol 0; a non-zero held total is SOL we already own sitting ' +
				'behind a configured floor. See docs/x402-ring-economy.md "The wallet fee governor".',
		};
	}
	return {
		cause: /** @type {const} */ ('rail'),
		mechanism: /** @type {const} */ ('rail'),
		hint:
			'Payments are being rejected at settle. Check the self-facilitator: ' +
			'`npm run logs -- -s three-ws-api --grep "settle_failed" --since 3h`. ' +
			'A 502 cluster with empty simulation logs is a duplicate-signature or ' +
			'RPC-preflight fault; a 402 cluster is verify/facilitator rejection. A 503 ' +
			'cluster is a deliberate refusal (fee-governor budget or sponsor floor); ' +
			'both are reconciled against the facilitator book above, so a 503 surplus ' +
			'surviving into this hint means the facilitator log and the ring log ' +
			'disagree; read x402_self_facilitator_log.reject_reason directly. ' +
			'See docs/ops/production-log-triage.md.',
	};
}

// The admission gate exists so a spent fee budget costs one skipped call, not a
// signed transfer plus a simulated verify refused at the end. A caller that goes
// around it leaves no trace in the ring log (it is not the ring), so the rate
// this sensor reports stays green while the facilitator refuses thousands of
// settles an hour. Measured on production 2026-09-29: x402_settle read `ok, 97%`
// while 43,848 of 46,375 facilitator settle attempts in 24h were governor
// refusals, about 95% of them the x402 seed cron firing 60 sponsor-mode tips
// every two minutes against a budget that funded one.
//
// Degraded, not down: the calls that asked the gate still settle. It fires only
// when the bypass is both material (MIN_ATTEMPTS) and larger than everything the
// ring settled, so the residue of a cross-instance admission race (a handful an
// hour, each still recorded) never trips it.
function gateBypassVerdict(verdict, { gateBypass, settled, minAttempts }) {
	if (!(gateBypass >= minAttempts && gateBypass > settled)) return null;
	return {
		...verdict,
		status: /** @type {const} */ ('degraded'),
		cause: /** @type {const} */ ('fee_governor'),
		mechanism: /** @type {const} */ ('gate_bypass'),
		detail:
			`${verdict.detail}; ${gateBypass} settle(s) refused by the fee governor at the facilitator ` +
			'from a caller that skipped the admission gate',
		hint:
			'A caller is building, signing and verifying payments the wallet fee governor then refuses at ' +
			'settle (`fee_runway_exhausted` in x402_self_facilitator_log with no matching ring row). Each one ' +
			'costs a simulated verify and a 503 for nothing. Find it: group the same window of ' +
			'x402_self_facilitator_log by payer and amount_atomic (verify rows always carry the payer), then ' +
			'match the request logs by user agent. Fix the caller, not the budget: size the batch with ' +
			'reserveFeeAdmissions() or gate each call with assessFeeAdmission() / admitSponsorSettle() ' +
			'(api/_lib/x402/wallet-fee-meter.js, api/_lib/x402/pay.js). Funding the fee wallet only moves ' +
			'the line such a caller runs into. See docs/x402-ring-economy.md "The wallet fee governor".',
	};
}

/**
 * Classify pre-aggregated settle buckets into a subsystem verdict. Pure — no DB,
 * no clock — so the thresholds and the rail-fault split are unit-testable
 * against the exact bucket shape the DB returns.
 *
 * `facilitatorRejects` is the same window's deliberate-refusal counts from the
 * facilitator's own book (`x402_self_facilitator_log.reject_reason`). It exists
 * because the ring log is REASON-BLIND for refusals that arrive over HTTP: a
 * settle the wallet fee governor refused reaches `x402_autonomous_log` as a bare
 * `http_503` (pre-2026-08-06: `http_502`), matches RAIL_STATUS, and reads as a
 * rail fault, which is how 2026-08-05 production showed `cause: "rail"` for
 * ~1,200 refusals that were the budget governor pacing on purpose. The
 * reconciliation below re-attributes status-only 5xx faults to the governor /
 * floor, clamped by min() so a window skew between the two logs can never
 * invent faults or attribute more than actually happened.
 * @param {Array<{ success: boolean, paid: boolean, reason: string, rent?: boolean, n: number }>} buckets
 * @param {{ minAttempts?: number,
 *   facilitatorRejects?: { governor?: number, floor?: number } }} [opts]
 * @returns {{ status: 'ok'|'degraded'|'down'|'unknown', settled: number,
 *   faults: number, attempts: number, rate: number|null,
 *   faultClasses: Array<{ reason: string, n: number }>, detail: string,
 *   mechanism?: 'accept_withdrawn'|'settle_refused'|'paced'|'gate_bypass'|'rail', hint?: string,
 *   gateBypass?: number }}
 */
export function classifySettleBuckets(buckets, { minAttempts = MIN_ATTEMPTS, facilitatorRejects } = {}) {
	let settled = 0;
	let faults = 0;
	// Counted but deliberately kept OUT of the rate: these two explain a collapse
	// the rate can only report. Folding them into `faults` would change what the
	// percentage means (and peg it red on a benign quiet ring); dropping them
	// entirely is what made the 07-29 numerator collapse unreadable.
	let noSolanaAccept = 0;
	let floorSignals = 0;
	let governorSkips = 0;
	/** @type {Record<string, number>} */
	const faultBy = {};
	for (const b of Array.isArray(buckets) ? buckets : []) {
		const n = Number(b?.n) || 0;
		if (n <= 0) continue;
		if (b.success && b.paid) {
			settled += n;
			continue;
		}
		if (b.success) continue;
		if (b.reason === NO_SOLANA_ACCEPT) noSolanaAccept += n;
		if (SPONSOR_FLOOR.test(String(b.reason || ''))) floorSignals += n;
		if (FEE_GOVERNOR.test(String(b.reason || ''))) governorSkips += n;
		// A rent-exemption failure on the fee payer wears a rail-shaped reason
		// token (`simulation_failed`, `sweep_broadcast_failed`) while being the
		// opposite of a rail fault: the transaction never reached the rail because
		// the sponsor could not pay for it. Counting it as rail is what produced
		// the wrong hint on 2026-08-28, when 3 hours of a dry sponsor were reported
		// as `cause: rail` and the operator was pointed at duplicate signatures and
		// RPC preflight instead of at a wallet holding 0.000899 SOL. The flag is
		// computed from the full error_msg because the `:`-token this groups by
		// throws the rent detail away.
		if (b.rent) {
			floorSignals += n;
			continue;
		}
		if (isRailFault(b.reason)) {
			faults += n;
			faultBy[b.reason] = (faultBy[b.reason] || 0) + n;
		}
	}

	// Re-attribute status-only 5xx faults to the deliberate refusals the
	// facilitator book proves happened in the window. 503 drains first (the
	// current status both refusals wear), then 502 (the pre-2026-08-06 mapping,
	// and any path still routed through the generic settle_failed throw).
	const drainStatusFaults = (count) => {
		let remaining = Math.max(0, Math.floor(Number(count) || 0));
		let drained = 0;
		for (const key of ['http_503', 'http_502']) {
			if (remaining <= 0) break;
			const take = Math.min(faultBy[key] || 0, remaining);
			if (take <= 0) continue;
			faultBy[key] -= take;
			if (faultBy[key] === 0) delete faultBy[key];
			faults -= take;
			remaining -= take;
			drained += take;
		}
		return drained;
	};
	const facilitatorGovernor = Math.max(0, Math.floor(Number(facilitatorRejects?.governor) || 0));
	const drainedGovernor = drainStatusFaults(facilitatorGovernor);
	governorSkips += drainedGovernor;
	floorSignals += drainStatusFaults(facilitatorRejects?.floor);
	// Governor refusals the facilitator booked that NO ring row accounts for: a
	// caller that signed, verified and submitted a settle the budget could not
	// fund, without asking the caller-side admission gate first and without
	// writing to x402_autonomous_log. See gateBypassVerdict().
	const gateBypass = facilitatorGovernor - drainedGovernor;

	const attempts = settled + faults;
	const faultClasses = Object.entries(faultBy)
		.map(([reason, n]) => ({ reason, n }))
		.sort((a, b) => b.n - a.n);

	if (attempts < minAttempts) {
		// A sponsor under its floor is a HARD stop, not an unjudgeable window. Every
		// settle it touches fails closed, so the attempts that would have been
		// judged are precisely the ones the empty wallet prevented, and moving them
		// out of `faults` (which is correct: they never reached the rail) must not
		// let the sensor answer `unknown`. On 2026-08-28 that would have reported a
		// three-hour, zero-settle outage as "too few attempts to judge". Down rather
		// than degraded, per the precedence in diagnoseSettleDrop: under the floor
		// nothing settles at all, where a spent governor budget still settles at the
		// paced rate.
		if (floorSignals >= minAttempts) {
			const { cause, mechanism, hint } = diagnoseSettleDrop({
				noSolanaAccept, floorSignals, governorSkips, settled, faults,
			});
			return {
				status: 'down',
				settled, faults, attempts, rate: null, faultClasses,
				cause, mechanism, noSolanaAccept, floorSignals, governorSkips,
				detail:
					`settle halted: ${floorSignals} attempt(s) refused with the sponsor under its SOL floor ` +
					`in ${WINDOW_INTERVAL}, ${settled} settled`,
				hint,
			};
		}
		// A throttled rail is not an unjudgeable one. When the reason there is
		// nothing to judge is that the governor skipped the calls, say so and report
		// `degraded`: the funding action is identical to the low-rate case, and
		// `unknown` would let a fully paced-shut economy sit silent behind a neutral
		// verdict. The threshold is the same MIN_ATTEMPTS, so a handful of skips on
		// a genuinely idle ring still reads `unknown`.
		if (governorSkips >= minAttempts) {
			const { cause, mechanism, hint } = diagnoseSettleDrop({
				noSolanaAccept, floorSignals, governorSkips, settled, faults,
			});
			return {
				status: 'degraded',
				settled, faults, attempts, rate: null, faultClasses,
				cause, mechanism, noSolanaAccept, floorSignals, governorSkips,
				detail:
					`settle throttled: ${governorSkips} call(s) skipped by the wallet fee governor, ` +
					`${attempts} attempt(s) in ${WINDOW_INTERVAL}`,
				hint,
			};
		}
		const quiet = {
			status: /** @type {const} */ ('unknown'),
			settled, faults, attempts, rate: null, faultClasses, governorSkips, gateBypass,
			detail: `only ${attempts} settle attempts in ${WINDOW_INTERVAL}, too few to judge`,
		};
		return gateBypassVerdict(quiet, { gateBypass, settled, minAttempts }) || quiet;
	}

	const rate = settled / attempts;
	const pct = (rate * 100).toFixed(1);
	const topFaults = faultClasses
		.slice(0, 3)
		.map((f) => `${f.reason}×${f.n}`)
		.join(', ');
	const base = `settle ${pct}% (${settled}/${attempts} paid attempts, ${WINDOW_INTERVAL})`;

	if (rate >= OK_RATE) {
		// A healthy rate with the governor pacing behind it is still healthy: the
		// settles that ran, ran. Carry the skip count so a wallet quietly sliding
		// toward its budget shows up before the rate does.
		const healthy = {
			status: /** @type {const} */ ('ok'), settled, faults, attempts, rate, faultClasses,
			noSolanaAccept, floorSignals, governorSkips, gateBypass,
			detail: governorSkips > 0 ? `${base}; ${governorSkips} paced by the fee governor` : base,
		};
		return gateBypassVerdict(healthy, { gateBypass, settled, minAttempts }) || healthy;
	}
	const status = rate < DOWN_RATE ? 'down' : 'degraded';
	const { cause, mechanism, hint } = diagnoseSettleDrop({
		noSolanaAccept, floorSignals, governorSkips, settled, faults,
	});
	// Name the mechanism in `detail` too: the hint is one field deep in the JSON,
	// but `detail` is what the dashboard row, the digest and /api/status all print.
	// It has to agree with the counter printed beside it. "Solana accept withdrawn
	// (0 no_solana_accept)" was one clause contradicting itself, on the row an
	// operator reads first.
	const causeNote =
		mechanism === 'accept_withdrawn'
			? `; Solana accept withdrawn (${noSolanaAccept} no_solana_accept, sponsor under SOL floor)`
			: mechanism === 'settle_refused'
				? `; ${floorSignals} settle(s) refused at the fee-wallet floor (accept still advertised)`
				: mechanism === 'paced'
					? `; ${governorSkips} call(s) paced by the fee governor (budget spent, not a rail fault)`
					: '';
	return {
		status,
		settled, faults, attempts, rate, faultClasses,
		cause,
		mechanism,
		noSolanaAccept,
		floorSignals,
		governorSkips,
		gateBypass,
		detail: `${base}; ${faults} rail faults${topFaults ? ` (${topFaults})` : ''}${causeNote}`,
		hint,
	};
}

/**
 * Read the last WINDOW_INTERVAL of ring settle outcomes and classify them.
 * Fail-soft: a missing DB, an absent table (pre-migration), or any query error
 * resolves to `unknown` so a health read never throws. Pre-aggregates in SQL
 * (bucketed by success/paid/reason) so the row count is bounded regardless of
 * settle volume — safe on the per-request /api/healthz path.
 * @returns {Promise<{ name: 'x402_settle', label: string, status: string,
 *   detail: string, hint?: string, metrics?: object }>}
 */
export async function gatherX402SettleHealth() {
	const base = { name: /** @type {const} */ ('x402_settle'), label: 'x402 settlement success' };
	try {
		const { sql } = await import('../db.js');
		const buckets = /** @type {Array<{ success: boolean, paid: boolean, reason: string, rent?: boolean, n: number }>} */ (
			await sql`
				SELECT success,
				       (amount_atomic > 0) AS paid,
				       COALESCE(NULLIF(split_part(error_msg, ':', 1), ''), 'none') AS reason,
				       ((error_msg ILIKE '%InsufficientFundsForRent%' AND error_msg ILIKE '%"account_index":0%')
				         OR error_msg ILIKE '%account (0) with insufficient funds for rent%') AS rent,
				       count(*)::int AS n
				FROM x402_autonomous_log
				WHERE ts >= now() - ${WINDOW_INTERVAL}::interval
				  AND COALESCE(pipeline, '') <> ALL(${[...UNPAID_PROBE_PIPELINES]}::text[])
				GROUP BY success, paid, reason, rent
			`
		);
		// The facilitator's own book for the same window: the ring log only carries
		// `http_5xx` for a refusal that came back over HTTP, so this is the sole
		// record of WHY those settles were refused. Fail-soft to zero: a missing
		// table (pre-migration) must not take the whole sensor down with it.
		const rejects = await sql`
			SELECT split_part(COALESCE(reject_reason, ''), ':', 1) AS reason,
			       count(*)::int AS n
			FROM x402_self_facilitator_log
			WHERE ts >= now() - ${WINDOW_INTERVAL}::interval
			  AND action = 'settle' AND ok = false
			GROUP BY 1
		`.catch(() => []);
		const facilitatorRejects = { governor: 0, floor: 0 };
		for (const r of rejects) {
			const n = Number(r?.n) || 0;
			if (FEE_GOVERNOR.test(String(r?.reason || ''))) facilitatorRejects.governor += n;
			else if (SPONSOR_FLOOR.test(String(r?.reason || ''))) facilitatorRejects.floor += n;
		}
		const v = classifySettleBuckets(buckets, { facilitatorRejects });
		return {
			...base,
			status: v.status,
			detail: v.detail,
			...(v.hint ? { hint: v.hint } : {}),
			metrics: {
				windowHours: 3,
				settled: v.settled,
				faults: v.faults,
				attempts: v.attempts,
				rate: v.rate == null ? null : Math.round(v.rate * 1000) / 1000,
				topFaults: v.faultClasses.slice(0, 5),
				// The collapse-vs-rejection split. `cause` is what an operator should
				// read first; noSolanaAccept is the volume the rate cannot show.
				// `mechanism` is what to read SECOND, and it is not cosmetic: one
				// `cause: sponsor_floor` covers a withdrawn accept (the facilitator
				// never saw these calls) and a floor refusal (it saw and rejected
				// every one), which send an operator to opposite evidence. The triage
				// runbook's mechanism table keys off exactly this field.
				...(v.cause ? { cause: v.cause } : {}),
				...(v.mechanism ? { mechanism: v.mechanism } : {}),
				noSolanaAccept: v.noSolanaAccept ?? 0,
				floorSignals: v.floorSignals ?? 0,
				// Deliberate spend-pacing, not failure. Read it next to `rate`: the two
				// together separate "the rail broke" from "the rail is out of budget".
				governorSkips: v.governorSkips ?? 0,
				// Governor refusals at the facilitator with no ring row behind them:
				// callers that skipped the admission gate (see gateBypassVerdict).
				gateBypass: v.gateBypass ?? 0,
			},
		};
	} catch (err) {
		return { ...base, status: 'unknown', detail: err?.message || 'settle log unreadable' };
	}
}