Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

feat(bots): tell the operator why an armed trading bot is not trading

nirholasf3b53a62af5b
diff --git a/docs/ops/trading-bot-report.md b/docs/ops/trading-bot-report.md
index 4ff371d00..e4f2870ff 100644
--- a/docs/ops/trading-bot-report.md
+++ b/docs/ops/trading-bot-report.md
@@ -4,6 +4,11 @@
the platform in one pass: what is armed, what it has traded, what it made or
lost, and what its wallet actually holds on-chain right now.
+It also answers the question the raw counts hide: **why** a bot is not trading.
+An armed bot with a dry wallet, a halted bot, and a healthy idle bot all look
+identical in a trade count, so every bot carries a posture, a solvency verdict,
+and a stall flag.
+
It is read-only. It never signs, funds, closes, or sweeps anything.
```bash
@@ -23,15 +28,18 @@ Two independent runtimes drive autonomous trading, and the report shows both:
2. **Strategy Object runtime** (`agent_strategy_equips` +
`agent_strategy_positions`), swept every 2 minutes by
`/api/cron/strategy-fanout`. A strategy is bound ("equipped") to an agent and
- fires against the same pump.fun launch feed.
+ fires against the same pump.fun launch feed. Equips answer to the per-owner
+ `strategy_kill_switch` table, shown as `KILLED` in that section.
## What each field means
| Field | Source | Meaning |
|---|---|---|
-| `ARMED` / `off` / `KILLED` | `enabled`, `kill_switch` | Posture. `KILLED` means the owner's global kill switch is engaged, so exits still mark to market but no entry can fire. |
+| `ARMED` / `off` / `KILLED` | `enabled`, `kill_switch` | Posture. While killed, exits still mark to market but no entry can fire. **The two runtimes have separate kill switches and they are not interchangeable**: a sniper arm answers only to its own `agent_sniper_strategies.kill_switch` column, while the per-owner `strategy_kill_switch` table is read exclusively by `engagedKillOwners()` in `api/_lib/agent-strategy-runtime.js` and governs Strategy Object equips. Joining the owner table onto sniper arms reports an arm as halted while the executor keeps trading it. |
+| `STALLED` | `max(opened_at)` | Armed, not halted, and has not *attempted* an entry in 7 days. Every trigger here is wired to feeds that produce candidates daily, so silence this long is a fault, not quiet markets. This is the flag that distinguishes "working and waiting" from "armed and dead". |
+| `STARVED` / `SHRUNK` | `walletTradeState()` | Whether the wallet can fund its configured entry size. Decided by `api/_lib/sniper-solvency.js`, which calls `resolveEntrySize()`, **the same sizing rule the executor uses to place or skip a real entry**. Never re-derive these thresholds here: a local copy drifts into calling a wallet tradeable that the executor sits out, which is the exact bug that module was written to catch. `SHRUNK` means it still trades, just below the configured size. |
| `trigger` | `trigger` | What wakes the bot: `new_mint`, `oracle_crossing`, `graduation_ride`, `intel_confirmed`. |
-| `closed` / `open` / `failed` | `agent_sniper_positions.status` | A `failed` row is an entry that never landed (most commonly `SIM_FAILED`, the pre-flight simulation rejecting the buy), not a losing trade. Failures cost gas, never principal. |
+| `closed` / `open` / `failed` | `agent_sniper_positions.status` | A `failed` row is an entry that never landed, not a losing trade. The dominant reason is `SIM_FAILED`: `estimateComputeUnits()` in `api/_lib/execution-engine.js` runs a real `simulateTransaction` before broadcast and throws when it reverts. That path never broadcasts, so a `SIM_FAILED` costs **one RPC call and zero lamports**, not gas. Each candidate mint is attempted once, so a high count is filter friction, not a retry loop. |
| `realized PnL` | `realized_pnl_lamports` | Sum over every position the bot has ever held, in SOL. |
| `deployed` | `entry_quote_lamports` | Total SOL that has ever gone through an entry, so PnL reads against the capital that actually worked. |
| `open positions … marked` | `last_value_lamports` | The last quote the sweep took for still-open positions. Marked, not realized. |
@@ -43,6 +51,20 @@ Two independent runtimes drive autonomous trading, and the report shows both:
A bot's wallet is resolved from its most recent position row, so a bot that has
never executed a trade reports "none yet" instead of a wrong address.
+## Fleet solvency
+
+The report closes with a fleet verdict from `summarizeFleetSolvency()`: how many
+armed wallets can still place an entry, the SOL needed to lift every starved
+wallet back to its own refill target, and the funding master's balance.
+
+`masterCanCover` is the line that matters operationally. It separates "the
+auto-funder will heal this on its next tick" from "a human has to move SOL". The
+master balance is read through `masterBalanceSol()`, which needs the launcher
+signer (`LAUNCHER_MASTER_SECRET_KEY_B64`); where that is absent the report says
+`unread` and `masterCanCover` stays null. An unread balance is never treated as
+zero, for the same reason `summarizeFleetSolvency` skips unread wallet balances:
+an RPC blip must not report a healthy fleet as starved.
+
## Environment
- `DATABASE_URL` (from `.env.local`) is required.
diff --git a/scripts/trading-bot-report.mjs b/scripts/trading-bot-report.mjs
index 6745a1069..89aae0683 100755
--- a/scripts/trading-bot-report.mjs
+++ b/scripts/trading-bot-report.mjs
@@ -9,6 +9,19 @@
// its wallet, and the LIVE on-chain balance of that wallet (SOL + surviving
// SPL token positions) read straight from a Solana RPC.
//
+// It also answers the question the raw numbers hide: WHY a bot is not trading.
+// An armed bot with a dry wallet looks identical to a healthy one in every
+// count, so each bot carries two verdicts:
+//
+// solvency: 'funded' / 'shrunk' / 'starved', decided by walletTradeState(),
+// the same sizing rule the executor itself uses to place or skip
+// an entry (api/_lib/sniper-solvency.js). Asked, never re-derived:
+// a threshold copied here would drift into calling a wallet
+// tradeable that the executor sits out.
+// activity: STALLED when a bot is armed but has not attempted an entry in
+// STALE_DAYS, which is how ten armed bots sat silent for weeks
+// while every liveness check reported them fine.
+//
// Read-only: it never signs, funds, or closes anything.
//
// Usage:
@@ -23,6 +36,7 @@
import { readFileSync } from 'node:fs';
import { neon } from '@neondatabase/serverless';
import { solPriceUsd } from '../api/_lib/sol-price.js';
+import { summarizeFleetSolvency, describeSolvency, walletTradeState } from '../api/_lib/sniper-solvency.js';
for (const file of ['.env.local', '.env']) {
try {
@@ -48,6 +62,9 @@ const RPC = process.env.SOLANA_RPC_URL || 'https://api.mainnet-beta.solana.com';
const SOL = 1e9;
const TOKEN_PROGRAM = 'TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA';
const TOKEN_2022 = 'TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb';
+// An armed bot that has not even ATTEMPTED an entry for this long is stalled,
+// not quiet: every arm here is wired to feeds that produce candidates daily.
+const STALE_DAYS = 7;
const num = (v) => (v == null ? 0 : Number(v));
const sol = (n, d = 4) => (n >= 0 ? ' ' : '') + n.toFixed(d);
@@ -163,17 +180,34 @@ async function loadSnipers() {
async function loadEquips() {
return sql`
select e.id, e.agent_id, e.network, e.active, e.fires_count, e.last_eval_at, e.last_fired_at,
- i.name agent_name, st.name strategy_name,
+ i.name agent_name,
+ coalesce(k.engaged, false) owner_killed,
+ st.name strategy_name,
(select count(*)::int from agent_strategy_positions p where p.equip_id = e.id) positions,
(select coalesce(sum(p.realized_pnl_lamports), 0)::float / 1e9 from agent_strategy_positions p
where p.equip_id = e.id) realized_pnl_sol
from agent_strategy_equips e
left join agent_identities i on i.id = e.agent_id
left join agent_strategies st on st.id = e.strategy_id
+ left join strategy_kill_switch k on k.owner_id = e.owner_id
order by e.active desc, e.fires_count desc`;
}
-const [snipers, equips, price] = await Promise.all([loadSnipers(), loadEquips(), livePrice()]);
+// The funding master is the wallet the auto-funder refills every bot from, so
+// "starved fleet" and "starved fleet the auto-funder can heal on its own" are
+// different operational answers. Reading it needs the launcher signer, which is
+// absent from a plain checkout; null is a first-class value here (it turns
+// masterCanCover into null, not false) rather than a reason to fail the report.
+async function masterSol() {
+ try {
+ const { masterBalanceSol } = await import('../api/_lib/launcher-funding.js');
+ return await masterBalanceSol('mainnet');
+ } catch {
+ return null;
+ }
+}
+
+const [snipers, equips, price, master] = await Promise.all([loadSnipers(), loadEquips(), livePrice(), masterSol()]);
const active = snipers.filter((b) => b.enabled || b.closed_count > 0 || b.open_count > 0);
const reported = includeIdle ? snipers : active;
@@ -190,6 +224,28 @@ if (!skipChain) {
}
}
+const staleBefore = Date.now() - STALE_DAYS * 86_400_000;
+for (const bot of reported) {
+ // The sniper arms answer to their OWN per-strategy kill_switch column only.
+ // The per-owner strategy_kill_switch table governs the Strategy Object runtime
+ // (api/_lib/agent-strategy-runtime.js engagedKillOwners()) and nothing reads it
+ // on this path, so joining it here would report an arm as halted while the
+ // executor keeps trading it.
+ bot.halted = Boolean(bot.kill_switch);
+ bot.stalled = Boolean(bot.enabled) && !bot.halted
+ && (!bot.last_trade_at || new Date(bot.last_trade_at).getTime() < staleBefore);
+ bot.solvency = bot.solBalance == null ? 'unknown' : walletTradeState(bot.solBalance, num(bot.per_trade_sol));
+}
+
+// Fleet verdict over the ARMED wallets only: an off bot not trading is a
+// decision, not a fault, and folding it in would understate the deficit.
+const fleet = summarizeFleetSolvency({
+ wallets: reported
+ .filter((b) => b.enabled && !b.halted)
+ .map((b) => ({ agentId: b.agent_id, address: b.wallet, balanceSol: b.solBalance ?? null, perTradeSol: num(b.per_trade_sol) })),
+ masterSol: master,
+});
+
const totals = reported.reduce(
(acc, b) => {
acc.realized += num(b.realized_pnl_sol);
@@ -208,7 +264,7 @@ const totals = reported.reduce(
);
if (asJson) {
- console.log(JSON.stringify({ generated_at: new Date().toISOString(), sol_price_usd: price, totals, snipers: reported, equips }, null, 2));
+ console.log(JSON.stringify({ generated_at: new Date().toISOString(), sol_price_usd: price, fleet, totals, snipers: reported, equips }, null, 2));
process.exit(0);
}
@@ -221,7 +277,10 @@ console.log(`SOL reference price: ${price ? `$${price.toFixed(2)}` : 'unknown'}
console.log('\nPUMP.FUN SNIPERS');
for (const b of reported) {
const posture = b.kill_switch ? 'KILLED' : b.enabled ? 'ARMED' : 'off';
- console.log(`\n ${b.agent_name || '(unnamed)'} [${b.label || 'no label'}] ${posture}`);
+ const flags = [b.stalled ? `STALLED (no entry attempt in ${STALE_DAYS}d)` : null, b.solvency === 'starved' ? 'STARVED (cannot fund an entry)' : b.solvency === 'shrunk' ? 'SHRUNK (trades below configured size)' : null]
+ .filter(Boolean)
+ .join(' ');
+ console.log(`\n ${b.agent_name || '(unnamed)'} [${b.label || 'no label'}] ${posture}${flags ? ` ${flags}` : ''}`);
console.log(` trigger ${b.trigger} · ${b.decision_mode || 'rules'} · ${b.network} · ${b.per_trade_sol} SOL/trade, ${b.daily_budget_sol} SOL/day, max ${b.max_concurrent_positions} open`);
console.log(` trades: ${b.closed_count} closed (${b.wins}W/${b.losses}L), ${b.open_count} open, ${b.failed_count} failed entries · last ${when(b.last_trade_at)}`);
console.log(` realized PnL: ${sol(num(b.realized_pnl_sol))} SOL${usd(num(b.realized_pnl_sol))} on ${num(b.deployed_sol).toFixed(3)} SOL deployed`);
@@ -244,11 +303,23 @@ for (const b of reported) {
console.log('\nSTRATEGY OBJECT RUNTIME (task-05 equips)');
for (const e of equips) {
- console.log(` ${e.active ? 'active' : 'off '} ${(e.agent_name || e.agent_id).slice(0, 24).padEnd(25)} ${(e.strategy_name || '-').slice(0, 22).padEnd(23)} fires ${String(e.fires_count).padStart(3)} · positions ${e.positions} · PnL ${sol(num(e.realized_pnl_sol))} SOL · last eval ${when(e.last_eval_at)}`);
+ console.log(` ${e.owner_killed ? 'KILLED' : e.active ? 'active' : 'off '} ${(e.agent_name || e.agent_id).slice(0, 24).padEnd(25)} ${(e.strategy_name || '-').slice(0, 22).padEnd(23)} fires ${String(e.fires_count).padStart(3)} · positions ${e.positions} · PnL ${sol(num(e.realized_pnl_sol))} SOL · last eval ${when(e.last_eval_at)}`);
+}
+
+console.log('\nFLEET SOLVENCY (armed bots)');
+console.log(` ${describeSolvency(fleet)}`);
+console.log(` ${fleet.funded} funded · ${fleet.shrunk} shrunk · ${fleet.starved} starved · ${fleet.tradeable}/${fleet.agents} able to place an entry`);
+if (fleet.deficitSol > 0) console.log(` SOL needed to lift every starved wallet to its refill target: ${fleet.deficitSol.toFixed(4)} SOL${usd(fleet.deficitSol)}`);
+console.log(` Funding master: ${fleet.masterSol == null ? 'unread (launcher signer not in this environment)' : `${fleet.masterSol.toFixed(4)} SOL`}`);
+if (fleet.masterCanCover === false) console.log(' The master CANNOT cover the deficit: the auto-funder will not heal this, a human has to move SOL.');
+if (fleet.masterCanCover === true) console.log(' The master can cover the deficit; the auto-funder should heal this on its next tick.');
+const stalled = reported.filter((b) => b.stalled);
+if (stalled.length) {
+ console.log(` Stalled and armed (${stalled.length}): ${stalled.map((b) => b.agent_name || b.agent_id).join(', ')}`);
}
console.log('\nTOTALS');
-console.log(` Bots reported: ${reported.length} (${reported.filter((b) => b.enabled && !b.kill_switch).length} armed)`);
+console.log(` Bots reported: ${reported.length} (${reported.filter((b) => b.enabled && !b.halted).length} armed)`);
console.log(` Closed trades: ${totals.wins + totals.losses} (${totals.wins}W / ${totals.losses}L${totals.wins + totals.losses ? `, ${((totals.wins / (totals.wins + totals.losses)) * 100).toFixed(1)}% win rate` : ''})`);
console.log(` Failed entries: ${totals.failed}`);
console.log(` Capital deployed into entries: ${totals.deployed.toFixed(3)} SOL${usd(totals.deployed)}`);