Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Payment-outcome board
GET /api/ops/payment-outcomes is the single read that answers "are payments
healthy right now, and if not, which stage is failing?". It exists because the
x402 griefing classes (verify-reject floods, settle failures, sponsor-fee
starvation) historically became visible only after they halted the economy;
the 2026-07-11 fable audit filed the gap and this board closes it.
Two ways to read it: the JSON endpoint below (curl + jq, or wired into ops
tooling), and /payment-outcomes, the
internal page that renders the same response. Neither is a summary of the
other: the page renders every panel this endpoint returns and adds no number of
its own.
Auth
Same gate as /api/ops/health: an admin session, or x-ops-secret matching
OPS_SECRET (never CRON_SECRET; see api/_lib/ops-auth.js). Unset secret
means open in dev and denied in production. Read-only; it moves no funds and
never fires alerts (balance reads go through the ring monitor with alerting
disabled; the scheduled monitor owns paging).
Per-IP ceiling: the authedReadIp bucket (300 requests / 5 min), shared with
/api/ops/health and /api/ops/money-health. It is deliberately not the strict
authIp credential bucket, so polling a board cannot 429 an operator's login
from the same address. Over the ceiling the response is 429 rate_limited with
a retry-after header.
The response envelope
{ "ok": true, "degraded": [], "generated_at": "…", "inbound": {…}, "ring_settle": {…}, "sponsor": {…}, "stranded_custody": {…} }
The panels are read independently, so one failing never blanks the others
(an RPC outage is exactly when the settle panels matter most). ok reports
whether the BOARD rendered, never whether payments are healthy: that verdict is
per panel. A panel that threw is replaced by { "error": "…" }, named in
degraded, and the status code is 207 Multi-Status instead of 200, the same
convention /api/ops/health uses. Alert on degraded being non-empty: a board
that cannot see is not a board that is fine.
The panels
inbound (from x402_audit_log)
Counts and rates over 1h / 3h / 24h windows for payments arriving at our paid routes:
settled/settle_failed: outcomes ofsettlePaymentafter a valid verify.settle_success_rate= settled / (settled + settle_failed). A Solana settle that was broadcast but not yet confirmed when the response shipped (thesettlement_pendingpath, see x402-settlement-pending.md) is apayment_settledrow withsettlement_status = 'pending'until the reconcile cron promotes it, so it counts as settled here.unsettled_flush: a handler that ended its own response before settlement ran, so the buyer received the good without paying for it. Any nonzero count is a route bug (it should bestreaming: true), not buyer behaviour.verify_rejected:verifyPaymentrefusals, written durably aspayment_verify_rejectedevents (metadata carriesreasonandrejected_proofvsupstream_fault). A spike here with flat settles is the junk-X-PAYMENT griefing signature: the per-IP verify penalty bucket should be absorbing it, and this is the panel that proves whether it is.replay_rejected, split by stage inreplay_stages_24h:pre_handlermeans the durable spent-payment guard refused a replayed proof before the handler ran;post_settlemeans a lost insert race was refused at claim time. Any sustained nonzero rate is someone re-sending captured headers.top_failure_reasons_24hnames the dominant failure codes with the stage (verify vs settle) so a 502 cluster is distinguishable from a rejected-proof wave without a log dive.settled_volume_usd_24hfrom settledamount_atomics(USDC, 6 decimals).replay_reject_ratehas its own denominator,inbound_attempts(everything that arrived, replays included). Measuring replays against the verify denominator would report a rate above 1 during exactly the flood the number exists to detect, since replays are refused above verify. It is computed in the endpoint rather than in a renderer so the JSON, the page, and any ops script quote one number computed one way.
Replays are excluded from both rate denominators: they are refused before verify and would otherwise dilute the signal.
ring_settle (from gatherX402SettleHealth)
The outbound settle-success sensor over x402_autonomous_log (3h window,
rail-fault allowlist, and excluding unpaid probe pipelines such as the
reliability uptime monitor, whose third-party timeouts never touch a payment), verbatim: status, human detail, triage hint, and the
top fault signatures. This is the same object /api/healthz consumes; it is
included here so the board is complete without a second request. Its triage
map lives in production-log-triage.md.
The ring log is reason-blind for refusals that arrive over HTTP (they all read
as http_5xx), so the sensor reconciles status-only 5xx faults against the
facilitator's own book (x402_self_facilitator_log.reject_reason) for the same
window and names the dominant cause in ring_settle.metrics.cause:
sponsor_floor (the sponsor fee wallet is under its SOL floor),
fee_governor (deliberate spend pacing, a budget problem, not a rail fault),
or rail (genuine settle faults). governorSkips is carried even on a healthy
rate so a wallet sliding toward its budget shows up before the rate does.
ring_settle.metrics.mechanism is what to read second, because one cause can
point at opposite evidence: accept_withdrawn (the Solana accept is gone from
the challenge, so the facilitator never saw the calls) versus settle_refused
(the accept is still advertised and the facilitator refuses each settle at the
floor gate, naming the wallet in fee_wallet_below_floor:<held><<floor>), plus
paced, rail, and gate_bypass. gate_bypass (counted in
metrics.gateBypass) means a caller outside the ring is signing payments the fee
governor then refuses at settle without asking the admission gate first; it marks
the sensor degraded even while the ring's own rate looks healthy. The mechanism
table and where each one's proof lives are in
production-log-triage.md.
A rent-exemption failure on the fee payer (InsufficientFundsForRent on
account index 0, in either spelling the RPCs use) wears a rail-shaped reason
token (sweep_broadcast_failed, and simulation_failed on any path that has
not yet been split) while being the opposite of a rail fault: the transaction
never reached the rail because the sponsor could not pay for it. The verify
path stopped relying on that reconstruction on 2026-09-09 and now raises the
verdict as its own class, sponsor_fee_unfunded when the dry fee payer is our
sponsor and payer_fee_unfunded when it is the buyer's own wallet, so the
token alone carries the meaning. The sensor flags the remaining rail-shaped
rows from the full error_msg and counts both kinds as floor signals, not
faults, so a dry sponsor reads as
cause: sponsor_floor rather than pointing the operator at duplicate
signatures and RPC preflight (which is what happened for three hours on
2026-08-28). Because a sponsor under its floor is a hard stop, those attempts
also override the "too few attempts to judge" unknown verdict: once the
floor signals alone reach the minimum sample the status is down with a
settle halted detail, since under the floor nothing settles at all.
sponsor (live RPC + x402_self_facilitator_log)
The fee wallet that pays every settle's SOL fee:
solis the live balance. Two different floors are reported and they are not interchangeable:settle_floor_solis the facilitator's hard floor (X402_SPONSOR_SOL_FLOOR_LAMPORTS, default 0.02) where settlement is actually refused, andsol_flooris the ring monitor's watch floor at 1.5x that, which exists to warn early.below_floorcompares against the watch floor.burn_sol_per_dayis MEASURED fromfee_lamportsover successful settles, never quoted from memory: the folklore burn rate has been wrong by roughly 10x before (ISSUES.md item 6). It always ships withburn_window_days(the window it was measured over) andsettles_in_window(the sample size). A burn rate without its window is a rumour, so nothing here prints one without the other.runway_days= balance / measured burn (to empty, what a funding ask is sized against).runway_days_to_floor= (balance minus the settle floor) / burn, which is the operational figure: settling stops at the floor, not at zero.runway_statusis the verdict the alert fires on (see below), and the dashboard card colours from it so the board and the page cannot disagree.
stranded_custody (custodial key health, snapshot-cached)
Custody the platform has LOST: wallets whose encrypted_solana_secret no longer
decrypts, so their SOL is visible on chain and can never be signed for. It sits
on this board because it is a payment-outcome fact: the treasury self-heal cannot
reclaim that SOL and the customers holding it cannot withdraw it. Before this
panel the only way to see the number was running
scripts/audit-custodial-key-health.mjs by hand, so weeks passed between
readings. The measurement itself is shared with that script
(api/_lib/custodial-key-health.js), so the CLI and the board cannot disagree.
statusis the verdict andreasonsays why:stranded/sealed: funded wallets are sealed.sol_strandedand its_platform/_customersplit are a real measurement.unknown/partial_read: some sealed wallet never got a balance read (an RPC chunk failed), so the totals are a FLOOR, not a measurement. Re-read after the lane recovers rather than quoting the number.unknown/key_mismatch: every wallet failed to decrypt. That is one wrong key for this deployment, essentially never a fleet of separately sealed wallets.unknown/no_decryption_key: this process holds no custodial key at all (only possible outside production, where secret-box fails closed).clear: nothing funded is sealed.
- Under either
unknownkey verdict the SOL fields arenullrather than a number with a caveat attached. An unattributable total is exactly the phantom figure this surface exists to stop publishing. briefpoints at stranded-wallets.md, the standing owner decision on the customer balances, whenever customer SOL is confirmed stranded.- The panel is snapshot-cached for six hours per instance and concurrent reads
share one scan (
cache_age_secondsreports the snapshot's age). Custody loss is a rotation event, not a live metric, so polling the board never turns into a fleet-wide RPC sweep.
The runway alert
The runway is no longer render-only. checkRingWallets() in
api/_lib/x402/wallet-balance-monitor.js runs on the autonomous loop's 10-minute
tick, measures the burn, and calls sendOpsAlert through the normal alerting path
(dashboard row always, Telegram push when TELEGRAM_ALERTS_CHAT_ID is wired).
The verdict logic and the alert copy are pure functions in
api/_lib/x402/sponsor-runway.js, unit-tested in
tests/x402-sponsor-runway.test.js (the arithmetic and the rendered string) and
tests/x402-sponsor-runway-monitor.test.js (that the monitor actually sends it).
runway_status |
Condition | Alerts |
|---|---|---|
critical |
balance at or under the settle floor | yes, critical |
warn |
runway_days_to_floor under the threshold |
yes, warn |
ok |
runway at or above the threshold | no |
unknown |
balance unreadable, or no settles in the window | no |
unknown deliberately never pages. An idle rail has no measurable burn and
dividing by zero would page every quiet night, and an unreadable balance is an
RPC problem, not a funding one.
Two env knobs, both live without a redeploy:
| Var | Default | Meaning |
|---|---|---|
X402_SPONSOR_RUNWAY_ALERT_DAYS |
3 |
days of runway to the floor below which the alert fires |
X402_SPONSOR_BURN_WINDOW_DAYS |
7 |
window the burn rate is measured over |
The alert message carries the wallet, the balance, the floor, the measured burn
with its window and sample size, both runway figures, the threshold, and a
top-up size computed from the measured burn. It points at the free self-heal
(POST /api/cron/treasury-topup?dry=1) before any funding ask, and it repeats
the rule that has broken the rail before: top up the SPONSOR or the economy
master, NEVER per-agent wallets (that strands SOL in wallets that pay no fees).
It also names the symptom in advance, because the symptom is misleading: under
the floor, sponsorKnownBelowFloor() makes buildRequirements() withdraw the
Solana accept from every 402 challenge, so the Solana-only ring never attempts a
payment and there is nothing to reject. Settlements collapse while rail faults
stay flat. The settle sensor reports that as cause: sponsor_floor (distinct
from fee_governor and rail) with mechanism accept_withdrawn in
ring_settle.metrics; the accepts are checkable directly:
curl -s https://three.ws/api/x402/three-intel | jq '.accepts[].network'
# only eip155:8453 means the Solana accept has been withdrawn
The page
/payment-outcomes (pages/payment-outcomes.html)
renders the whole response. It takes the same gate as the endpoint, is
noindex, nofollow, and is deliberately absent from data/pages.json: it is
internal tooling, like /quality-bench and /likeness-bench, not a public
surface.
- The window selector (1h / 3h / 24h) is a re-render, not a request. All
three windows arrive in one response, so sweeping between them during an
incident costs no extra database or RPC read.
Refreshis the explicit refetch. - A rate with no attempts renders
n/a, never0%. A quiet rail and a totally failing rail must not look the same, so an empty window says so in words underneath. - A panel that failed renders its own error in place and names the other
panels as unaffected, matching the endpoint's 207 semantics. The banner at the
top lists whatever is in
degraded. - An unmeasured number is never a green zero. The stranded-custody panel
returns nulls when the process could not open the wallets at all, and the page
renders
not measuredin muted type rather than a reassuring0 SOL. - Panels: inbound rates for the selected window, top reject reasons and replay stages (both 24h), the outbound ring settle sensor with its triage hint, every configured ring wallet against its own floor plus the sponsor runway, and stranded custody.
Reading it in an incident
inbound.settle_success_ratelow butverify_reject_rateflat: the rail or facilitator is failing after verify. Checkring_settle.detailfor the fault mix;http_502clusters with empty simulation logs are RPC/preflight faults (see solana-rpc-lanes.md).verify_reject_ratespiking: someone is flooding junk proofs. Confirm the verify penalty limiter is holding (the reasons list will be dominated byverify_rejectedwithrejected_proof).Both fine but settles refused with 503s: look at
sponsor. Floor breach or sub-day runway is the answer. Readrunway_statusfirst, then run the free self-heal (POST /api/cron/treasury-topup?dry=1, then without?dry=1) before asking for funds: reclaimable SOL sitting in platform agent wallets is often the answer, and every reclaim run now leavesagent_reclaimrows ineconomy_master_ledgersaying whether it wasblocked(an RPC or key problem, free to fix) or foundnothing_reclaimable(the only case that genuinely needs the owner to send SOL).Read
agent_reclaim.failedin the dry plan before you trust its total. The plan-only run opens each wallet's key exactly as the real run does, so a wallet whose secret does not decrypt is listed there at stagerecoverwith reasonsecret_undecryptableinstead of being counted as reclaimable. That SOL is real, visible on chain, and unreachable: it is encrypted under a key this deploy does not hold, and no cron run, RPC tier, or deposit moves it (see wallet-key-migration.md for the recovery path, and put the retired key inWALLET_ENCRYPTION_KEY_PREVIOUSif you have it). Until 2026-09-02 the dry run skipped that check and reported those wallets as available on every tick, which sent two separate investigations away with "the cron will self-heal from here" when it could not. A dry total of 0 with a populatedfailedlist means the same thing asnothing_reclaimable: the owner has to send SOL.replay_stages_24hnonzero: captured X-PAYMENT headers are being re-sent; the guard is working if the settles panel is unaffected.stranded_custody.statusisstranded: capital is gone, not late. No cron run, RPC tier, or deposit reaches it. Ifsol_stranded_customeris above zero, that half is a support obligation and outranks every ops number on this board: read stranded-wallets.md. Akey_mismatchverdict is a deployment config problem instead, and the fix is the key, not the wallets.
Related
api/ops/payment-outcomes.js(the endpoint),pages/payment-outcomes.html(the page at/payment-outcomes),api/_lib/ops/x402-settle-health.js(the sensor),api/_lib/custodial-key-health.js(the custody-loss measurement shared withscripts/audit-custodial-key-health.mjs),api/_lib/x402/wallet-balance-monitor.js(balances, floors, and the runway alert),api/_lib/x402/sponsor-runway.js(the burn measurement, the verdict, and the alert copy),api/_lib/x402/audit-log.js(the durable ledger the inbound panel reads),api/_lib/economy-ledger.js(recordAgentReclaim, the durable trail for every self-heal attempt)./api/ops/money-healthis the complementary board: open reconciliation verdicts per money subsystem rather than live payment outcomes. It takes the same gate as this endpoint (admin session orx-ops-secret). Until 2026-08-14 it ran its own gate that accepted aCRON_SECRETbearer, so an ops script pointed at it with the cron credential now gets a 401: sendx-ops-secret: $OPS_SECRET(Secret Manager:ops-dashboard-secret) instead.