Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(okx-bot): probe the AI credential and stop paging a daemon that is booting

nirholas2942b332804a
diff --git a/STRUCTURE.md b/STRUCTURE.md
index 00ee6fd82..7ff45efd9 100644
--- a/STRUCTURE.md
+++ b/STRUCTURE.md
@@ -69,7 +69,7 @@ bottom.
| Skill bundles (seller builder + evidence pricing) | [pages/bundles.html](pages/bundles.html) `→ /bundles`, API [api/agents/[id]/bundles.js](api/agents/%5Bid%5D/bundles.js), arithmetic [api/_lib/bundle-pricing.js](api/_lib/bundle-pricing.js), checkout [api/marketplace/purchase-bundle.js](api/marketplace/purchase-bundle.js) | Live (seller side); no buyer-facing browser view yet | Sell several of one agent's skills as one purchase. The price is derived from the agent's own ledger rather than a rule of thumb: the public `?action=pricing` endpoint finds every buyer who already bought 2+ of the selected skills, suggests the median of what they really paid (`basis: median_basket`), and backtests any candidate price against those same baskets; with no such buyer yet it falls back to 20% off the sum of the parts and labels itself `discount_off_list` so a default is never read as a finding. Every amount is an atomic integer carried as a BigInt end to end, because a 9-decimal mint puts real sums past `Number.MAX_SAFE_INTEGER`. Checkout settles on-chain and writes one `skill_purchases` row per skill (`kind = 'bundle'`, amount 0) so one bundle sale is never counted as N skill sales. Docs: [docs/skill-bundles.md](docs/skill-bundles.md). Tests: [tests/bundle-pricing.test.js](tests/bundle-pricing.test.js), [tests/vercel-agents-subpath-routes.test.js](tests/vercel-agents-subpath-routes.test.js) |
| Marketplace analytics & trial funnel | [pages/marketplace-analytics.html](pages/marketplace-analytics.html) `→ /marketplace/analytics`, renderer [src/marketplace-analytics.js](src/marketplace-analytics.js), backend [api/marketplace/analytics.js](api/marketplace/analytics.js) | Live | Public transparency page for the skill marketplace: paid sales, volume, top skills/agents, and the trial conversion funnel. **Every money aggregate gates on `status = 'confirmed'` alone.** Counting `status IN ('confirmed','trial')` as a sale published 10,454 "sales" and 6.2M $THREE of "volume" on 2026-07-30 against a table holding zero confirmed purchases, because a `trial` row is a free grant whose `amount` is only the list price it would have cost. Free trials now render as their own stat and as a four-stage funnel (granted → used → exhausted → converted), which also names the two shapes that mean a wiring fault rather than weak demand: trials never spent, or trials exhausted with nothing converting. Docs: [docs/marketplace.md](docs/marketplace.md), tutorial [docs/tutorials/sell-a-skill-with-a-trial.md](docs/tutorials/sell-a-skill-with-a-trial.md). Tests: [tests/marketplace-analytics-paid-only.test.js](tests/marketplace-analytics-paid-only.test.js) |
| OKX.AI marketplace services | [api/okx/3d/[service].js](api/okx/3d/%5Bservice%5D.js) `→ /api/okx/3d/*` | Live | 3D services sold to other agents on OKX.AI (agent #2632). The **listed** line-up (rebuilt 2026-08-22) is **three.ws Forge**: four paid A2MCP rows (`forge-draft` $0.01 / `forge-standard` $0.05 / `forge-hd` $0.25 / `forge-image` $0.25) plus a free `forge-status` poll, tools in [api/_okx3d/forge.js](api/_okx3d/forge.js). Each row is a real MCP Streamable HTTP server whose paid `forge_3d` tool 402s with `eip155:196` leading; the buyer receives exactly what the ChatGPT custom GPT receives (GLB + concept image + viewer + AR link), because both fronts shape responses through [api/_mcp-studio/studio-shape.js](api/_mcp-studio/studio-shape.js). Agent Identity Studio and the single-capability REST endpoints are on the **back burner**: `listed: false` keeps them out of the listing and the submission payload while they stay deployed and payable. Catalog source of truth: [api/_lib/okx-catalog.js](api/_lib/okx-catalog.js); pipeline: [api/_okx3d/](api/_okx3d); showcase: [pages/agent-identities.html](pages/agent-identities.html) `→ /agent-identities` + [src/agent-identities.js](src/agent-identities.js), fed at runtime by [api/agent-identities.js](api/agent-identities.js) `→ /api/agent-identities` (demo runs from `data/agent-identities.json` merged with the live catalog row, so the price the page quotes is the price the paid endpoint charges); docs: [docs/okx-marketplace.md](docs/okx-marketplace.md), [docs/agent-identities.md](docs/agent-identities.md). Tests: [tests/api/okx-forge.test.js](tests/api/okx-forge.test.js) |
-| OKX.AI marketplace chat bot host | [workers/okx-chat-bot/](workers/okx-chat-bot) (Cloud Run service `okx-chat-bot`), local revive [scripts/okx-bot-revive.mjs](scripts/okx-bot-revive.mjs) `→ npm run okx:bot` | Built · deploy owner-gated | Always-on host for the XMTP chat behind agent #2632. Supervises the `okx-a2a` daemon plus the `onchainos` wallet session, persists the wallet/XMTP identity to GCS across revisions, and rebuilds the AI subsession's briefing from [api/_lib/okx-chat-briefing.js](api/_lib/okx-chat-briefing.js) on every boot so quoted prices can never drift from the catalog. It exists because the stack used to live on a codespace that sleeps, and its outage mode is silent: an expired wallet session leaves the process healthy while no chat is delivered, which flagged the listing offline once. Health reaches the platform as the `okx_chat_bot` subsystem on `/api/healthz` via a `bot_heartbeat` row; a logged-out session pages with the exact login commands. Deploy: [workers/okx-chat-bot/cloudbuild.yaml](workers/okx-chat-bot/cloudbuild.yaml) (`--max-instances=1` is load-bearing, one GCS state writer). Tests: [tests/okx-chat-bot.test.js](tests/okx-chat-bot.test.js) |
+| OKX.AI marketplace chat bot host | [workers/okx-chat-bot/](workers/okx-chat-bot) (Cloud Run service `okx-chat-bot`), local revive [scripts/okx-bot-revive.mjs](scripts/okx-bot-revive.mjs) `→ npm run okx:bot`, session seeding [scripts/okx-bot-seed-state.mjs](scripts/okx-bot-seed-state.mjs) `→ npm run okx:bot:seed-state` | Built · deploy owner-gated | Always-on host for the XMTP chat behind agent #2632. Supervises the `okx-a2a` daemon plus the `onchainos` wallet session, persists the wallet/XMTP identity to GCS across revisions, and rebuilds the AI subsession's briefing from [api/_lib/okx-chat-briefing.js](api/_lib/okx-chat-briefing.js) on every boot so quoted prices can never drift from the catalog. It exists because the stack used to live on a codespace that sleeps, and its outage mode is silent: an expired wallet session leaves the process healthy while no chat is delivered, which flagged the listing offline once. Health reaches the platform as the `okx_chat_bot` subsystem on `/api/healthz` via a `bot_heartbeat` row; a logged-out session pages with the exact login commands. The AI credential is probed against the provider rather than assumed present ([workers/okx-chat-bot/provider.js](workers/okx-chat-bot/provider.js)), because a key the provider refuses is indistinguishable from a working one until a buyer's message goes unanswered. Vertex AI is the deploy's transport, so the runtime service account is the credential and there is no secret to mint. Deploy: [workers/okx-chat-bot/cloudbuild.yaml](workers/okx-chat-bot/cloudbuild.yaml) (`--max-instances=1` is load-bearing, one GCS state writer). Tests: [tests/okx-chat-bot.test.js](tests/okx-chat-bot.test.js) |
| Crypto Data API docs / landing | [pages/crypto.html](pages/crypto.html) `→ /crypto` | In-repo · docs surface | Screenshot-worthy public docs + funnel-top for the free, keyless Crypto Data API (`/api/crypto/*`: token snapshot, security, holders, launches, bonding, whales, symbol, wallet, trending, portfolio, airdrops). Renders hero, endpoint table, tabbed quickstart (curl/JS/Python), per-endpoint request/response reference and a live "Try it" console. Probes production at runtime and marks each endpoint Live vs Coming soon (degrades gracefully while sibling endpoints roll out). CTAs funnel to paid uniques: [/launcher](pages/launcher.html), [/vanity-wallet](public/vanity-wallet.html), [/forge](pages/forge.html). Spec: x402-overhaul campaign (retired 2026-07-28; readable in git history) |
| Free Crypto Data API + discovery | [api/crypto/index.js](api/crypto/index.js) `→ /api/crypto`, [api/crypto/openapi.js](api/crypto/openapi.js) `→ /api/crypto/openapi.json`, docs [docs/crypto-api.md](docs/crypto-api.md) `→ /docs/crypto-api` | Live · docs surface | Front door + OpenAPI 3.1 for the free, keyless Crypto Data API. Catalog assembler [api/_lib/crypto-catalog/index.js](api/_lib/crypto-catalog/index.js) merges `api/_lib/crypto-catalog/*.js` self-describing descriptors (currently `airdrops`, `bonding`, `holders`, `launches`, `portfolio`, `security`, `symbol`, `token`, `trending`, `wallet`, `whales`) via a static import barrel (the production source of truth, since Vercel's bundler only ships what static imports reach) plus an additive dev-time directory glob; skips malformed/throwing entries, never throws, valid-empty when zero. New descriptor = drop the file AND add its import line to the barrel. OpenAPI built from the same catalog via [api/_lib/crypto-catalog/openapi.js](api/_lib/crypto-catalog/openapi.js). Index does HTML/JSON content negotiation. Endpoint handlers live in `api/crypto/*.js`. Spec: x402-overhaul campaign (retired 2026-07-28; readable in git history) |
| Unified Crypto API aggregator — storefront + OpenAPI | Registry [api/v1/_providers.js](api/v1/_providers.js) (`providerCatalog()`), front door [api/v1/x/[...slug].js](api/v1/x/%5B...slug%5D.js) `→ /api/v1/x/<provider>/<endpoint>`, discovery `→ GET /api/v1/x`, storefront [pages/crypto-api.html](pages/crypto-api.html) `→ /crypto-api`, OpenAPI 3.1 [api/openapi-json.js](api/openapi-json.js) `→ /openapi.json`, docs § in [docs/api-reference.md](docs/api-reference.md#unified-api--apiv1x-aggregator), dir guide [api/v1/README.md](api/v1/README.md) | Live | Third-party crypto/DeFi/on-chain APIs (CoinGecko, DefiLlama × 2, Jupiter, DexScreener, Solana RPC, OpenAI-compatible) re-offered as one bill: free per-IP tier → BYOK pass-through → three.ws plan → x402 pay-per-call, in that order. Everything (OpenAPI paths, the `/crypto-api` live table, the docs quota table) renders from `providerCatalog()` at request/page-load time — no hand-enumeration, so a new provider descriptor shows up everywhere with zero page edits. Test: [tests/openapi-aggregator.test.js](tests/openapi-aggregator.test.js). Not to be confused with the separate `/api/crypto/*` bundle below (different campaign, different registry). Spec: x402-catalog campaign (retired 2026-07-28; readable in git history) |
diff --git a/package.json b/package.json
index 6601638cf..43fba7155 100644
--- a/package.json
+++ b/package.json
@@ -407,6 +407,7 @@
"claude": "./scripts/claude.sh",
"typecheck": "tsc -p jsconfig.json",
"okx:bot": "node scripts/okx-bot-revive.mjs",
+ "okx:bot:seed-state": "node scripts/okx-bot-seed-state.mjs",
"okx:three-copy": "node scripts/okx-three-copy-check.mjs",
"okx:gauntlet": "node scripts/okx-e2e-gauntlet.mjs",
"check:conflicts": "node scripts/check-merge-conflicts.mjs",
diff --git a/prompts/finish/backlog-00-INDEX.md b/prompts/finish/backlog-00-INDEX.md
index 41d97a218..4ff737395 100644
--- a/prompts/finish/backlog-00-INDEX.md
+++ b/prompts/finish/backlog-00-INDEX.md
@@ -43,7 +43,7 @@ observable health, so 01 to 03 come first.
| 01 | [x402 settle: clear `fee_runway_exhausted`](backlog-01-x402-settle-runway.md) | capital and the deploy (three more code defects found and fixed 2026-09-04, incl. a thin top-up that funded the wrong wallet) | SOL to the economy master, and the deploy that carries the ordering fix; the config levers are already applied |
| 05 | [R2 bucket CORS: verify, then fix at the origin](backlog-05-r2-bucket-cors.md) | one credential | mint an R2 admin token |
| 07 | [BNB testnet: deploy the two finished contracts](backlog-07-bnb-testnet-deploys.md) | one funded EOA | send tBNB to `0x1C4918894dfA5eE11cfF9629B458b5169Cfa3871` (faucet is reCAPTCHA-gated) |
-| 08 | [OKX chat bot: move off the codespace](backlog-08-okx-chat-bot-always-on.md) | nothing | one email OTP login |
+| 08 | [OKX chat bot: move off the codespace](backlog-08-okx-chat-bot-always-on.md) | the deploy (everything it depended on now exists) | approve the deploy; clear the GCP billing hold so replies can be authored |
| 09 | [Telegram bots: durable hosting for both feeds](backlog-09-telegram-bots-durability.md) | **done**, verified live 2026-09-02 | clear the commit gate on its file update |
| 10 | [x402scan listing: finish the last three steps](backlog-10-x402scan-listing.md) | the deploy (the facilitator listing itself is live) | approve the deploy, then one wallet signature to re-register the origin |
@@ -73,7 +73,15 @@ node with the real `scripts/bnb-testnet-deploy-prove.mjs --broadcast`. Fund that
the retired 2026-08-02 one.
08's worker is built and committed; on 2026-09-02 it beat for the first time, from this
codespace, so `/api/healthz` now carries the `okx_chat_bot` subsystem and reports it as the
-stopgap it is (`hostDurable=false`). The Cloud Run host is still undeployed.
+stopgap it is (`hostDurable=false`). On 2026-09-04 the deploy stopped waiting on a credential
+nobody had: it now authenticates the AI subsession through Vertex with the runtime service
+account, so `--set-secrets` carries only the heartbeat database URL, and the session snapshot
+is seeded and verified so the first boot comes up authenticated rather than paging for an OTP.
+The Cloud Run host is still undeployed, and that is now the only agent-doable step left.
+Read the same day: both AI credentials this project holds are present and refused (Vertex
+`Lightning dunning decision is deny`, a billing hold; the `openai-api-key` account
+`billing_not_active`, which kills the platform's OpenAI lane everywhere, not just here), so a
+deployed host will report the new `ai_provider_unauthorized` until the owner clears billing.
09 is done: the sibling repository is checked out at `/workspaces/pump-fun-sdk` after all, and
both feeds were verified running on Cloud Run on 2026-09-02 (`Ready=True`, websocket transport,
25 h uptime each) while this codespace's own rebuild killed their local processes.
diff --git a/prompts/finish/backlog-PROGRESS.md b/prompts/finish/backlog-PROGRESS.md
index 27c9a72b3..e3d93d989 100644
--- a/prompts/finish/backlog-PROGRESS.md
+++ b/prompts/finish/backlog-PROGRESS.md
@@ -1576,3 +1576,38 @@ Left, and all of it is the owner's:
ordering.
- DoD lines 1 to 3 stay open behind those two. Lines 4 to 6 (recurrence guard + test,
changelog, this log) are done. The prompt file stays on disk.
+
+## 2026-09-04: 08 OKX chat bot always-on
+Measured: `onchainos wallet status` → `loggedIn: true, [email protected]` (no OTP needed).
+No `okx-chat-bot` Cloud Run service in `aerial-vehicle-466722-p5` (`gcloud run services
+list`), and no `anthropic-api-key` secret (`gcloud secrets list`), which is what the deploy
+demanded. Both credentials the project does hold refuse to serve: Vertex answers
+`403 PERMISSION_DENIED: Lightning dunning decision is deny for project: projects/93741856042`,
+and the `openai-api-key` secret's account answers `429 billing_not_active` on
+`/v1/chat/completions` while `GET /v1/models` returns 200 with 124 models. The daemon's boot
+log replayed a buyer message with `replayLagMs=71441816`: one message sat ~20 hours
+undelivered while the host was down.
+
+Did: revived the stopgap under the worker's own supervisor (`/readyz` 200, `online`,
+1 XMTP client, heartbeat writing so `/api/healthz` carries `okx_chat_bot` again as degraded
+per `hostDurable=false`). Removed the deploy's missing-secret blocker by making Vertex the
+AI transport (`CLAUDE_CODE_USE_VERTEX=1` on the service; `three-ws@` already holds
+`roles/aiplatform.user`, so there is no secret to mint) and stopped trusting presence as
+proof: `workers/okx-chat-bot/provider.js` asks the provider itself at boot and every 15 min
+and reports `ai_provider_unauthorized`, which fails `/readyz`, pages ops, and carries three
+ways out in the `remedy`. Also fixed codex auth (>=0.153 ignores `OPENAI_API_KEY` and needs
+`codex login --with-api-key`, now run at boot), set `OKX_A2A_AI_PERMISSION_PRESET=bypass` in
+the deploy so a headless subsession cannot stall on an approval prompt, and stopped the
+`daemon_down` false page a booting daemon fired on every restart (90s `daemon_starting`
+grace, observed live before the fix). Seeded the session snapshot with the new
+`npm run okx:bot:seed-state -- --apply` (329,396 bytes, generation 1788505342113858),
+verified by restoring it into a throwaway HOME and reading `loggedIn: true` back, so the
+first Cloud Run boot comes up authenticated. Workspace context verified by asking the
+subsession a platform question: it answered with the real catalog, `$THREE`, Solana and
+agent 2632.
+
+Left: the deploy itself (owner-gated, one command in `workers/okx-chat-bot/README.md` under
+"Ship it"; re-seed and stop the codespace stopgap first, the GCS object has exactly one
+writer), and the GCP billing hold (owner). Until the hold clears a deployed host reports
+`ai_provider_unauthorized`: chat is received and durable, replies are not authored. That is
+loud rather than silent, and clearing the hold needs no redeploy.
diff --git a/prompts/finish/okx-ai-RUNBOOK.md b/prompts/finish/okx-ai-RUNBOOK.md
index 03d92aea4..33137d1fe 100644
--- a/prompts/finish/okx-ai-RUNBOOK.md
+++ b/prompts/finish/okx-ai-RUNBOOK.md
@@ -89,6 +89,9 @@ PORT=8080 OKX_BOT_REPO_ROOT=/workspaces/three.ws \
curl -s localhost:8080/readyz # 200 with health.ready true when chat is deliverable
```
+If that port is taken the worker exits on `EADDRINUSE` before it ever spawns the daemon, which
+reads as "the bot will not start" rather than "the port is busy". Pick another `PORT`.
+
Run that way it supervises the daemon itself, beats to `bot_heartbeat` every 30s (so
`/api/healthz` finally shows the `okx_chat_bot` subsystem instead of `no heartbeat reported
yet`), and labels every beat `host=codespace:<name>`, `hostDurable=false`, which reads as
@@ -111,11 +114,27 @@ gcloud builds submit --config workers/okx-chat-bot/cloudbuild.yaml \
Until that lands, `npm run okx:bot` before an OKX retest window is still the stopgap: a
codespace cannot stay up on its own, so the local bot dies whenever this workspace sleeps.
-Two owner actions remain either way. The wallet OTP is unavoidable (OKX requires a human,
-and it never surfaces as a CLI prompt). The AI-provider credential is what lets the headless
-box author replies at all: set `ANTHROPIC_API_KEY` on the service (preferred), or
-`OPENAI_API_KEY` to run the Codex CLI instead. Without one the host still receives chat and
-reports `ai_provider_uncredentialed` rather than pretending to be healthy.
+**The deploy no longer waits on a credential (2026-09-04).** It used to demand an
+`anthropic-api-key` secret that exists nowhere, so it never ran. The service now authenticates
+the AI subsession through Vertex AI with the runtime service account (`three-ws@` already holds
+`roles/aiplatform.user`), so `--set-secrets` carries only the heartbeat database URL, and the
+session snapshot is seeded in `gs://three-ws-okx-bot-state`, so the first boot restores an
+authenticated wallet rather than paging for an OTP. Before shipping, re-seed from the host that
+holds the live session and stop it: `npm run okx:bot:seed-state -- --apply`, then stop the
+stopgap. That GCS object has exactly one writer.
+
+One owner action beyond the deploy: **the GCP billing hold**. Measured 2026-09-04, both AI
+credentials this project holds are present and refused, Vertex with
+`403 PERMISSION_DENIED: Lightning dunning decision is deny for project: projects/93741856042`
+and the `openai-api-key` secret's account with `429 billing_not_active` (that second one kills
+the platform's OpenAI lane everywhere, not just here). The host no longer takes a configured
+credential on trust: it asks the provider at boot and every 15 minutes and reports
+`ai_provider_unauthorized`, which fails `/readyz`, pages ops, and carries the three ways out in
+the `remedy`. Until billing clears, a deployed host receives chat durably and authors no
+replies. Clearing it needs no redeploy; the next probe flips the verdict on its own.
+
+The wallet OTP remains unavoidable whenever the session does expire (OKX requires a human, and
+it never surfaces as a CLI prompt).
Once deployed, the fix for a logged-out session is readable off the service itself, so
nobody has to reconstruct it from this runbook:
diff --git a/public/features.json b/public/features.json
index 10c973c4b..9535803c8 100644
--- a/public/features.json
+++ b/public/features.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-09-04T06:25:16.026Z",
+ "generated_at": "2026-09-04T07:17:23.363Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json",
"site": {
"name": "three.ws",
diff --git a/scripts/okx-bot-seed-state.mjs b/scripts/okx-bot-seed-state.mjs
new file mode 100644
index 000000000..cf899496f
--- /dev/null
+++ b/scripts/okx-bot-seed-state.mjs
@@ -0,0 +1,114 @@
+#!/usr/bin/env node
+// Seed the OKX chat bot's GCS state object from a machine that already holds a
+// live wallet session, so the first Cloud Run boot comes up authenticated
+// instead of paging a human for an email OTP it did not need to ask for.
+//
+// workers/okx-chat-bot/state.js writes the same archive on the deployed host,
+// but it authenticates through ADC and a developer codespace has none. This
+// script builds a byte-compatible archive from the SAME exported contract
+// (STATE_ROOTS / STATE_EXCLUDES, so the two can never drift) and uploads it with
+// gcloud, which is authenticated here.
+//
+// node scripts/okx-bot-seed-state.mjs # plan only, writes nothing remote
+// node scripts/okx-bot-seed-state.mjs --apply # build and upload
+// node scripts/okx-bot-seed-state.mjs --apply --force # upload with the daemon running
+//
+// The archive carries the wallet keyring and the XMTP identity. It belongs in
+// the private state bucket and nowhere else: never copy it elsewhere, never
+// attach it to a report, never commit it.
+
+import { execFile } from 'node:child_process';
+import { rm, stat } from 'node:fs/promises';
+import { homedir, tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { promisify } from 'node:util';
+import { STATE_ROOTS, STATE_EXCLUDES } from '../workers/okx-chat-bot/state.js';
+
+const run = promisify(execFile);
+
+const args = process.argv.slice(2);
+const APPLY = args.includes('--apply');
+const FORCE = args.includes('--force');
+const HOME = process.env.OKX_BOT_HOME || homedir();
+const BUCKET = process.env.OKX_BOT_STATE_BUCKET || 'three-ws-okx-bot-state';
+const OBJECT = process.env.OKX_BOT_STATE_OBJECT || 'okx-chat-bot/state.tar.gz';
+const PROJECT = process.env.GOOGLE_CLOUD_PROJECT || 'aerial-vehicle-466722-p5';
+const ARCHIVE = join(tmpdir(), 'okx-bot-seed.tar.gz');
+
+const say = (...m) => console.log(...m);
+const die = (msg, code = 1) => {
+ console.error(`\n FAILED: ${msg}\n`);
+ process.exit(code);
+};
+
+const exists = (p) => stat(p).then(() => true).catch(() => false);
+
+/**
+ * A snapshot taken while the daemon is writing can tear: the XMTP sqlite files
+ * are copied mid-write and the restored identity is corrupt, which costs a human
+ * OTP to recover. The deployed host avoids this by stopping the daemon before
+ * its shutdown snapshot; here the operator has to.
+ */
+async function daemonIsRunning() {
+ const r = await run('okx-a2a', ['daemon', 'status'], { timeout: 20_000 }).catch((err) => ({
+ stdout: String(err?.stdout || ''),
+ }));
+ return (r.stdout || '').trim().startsWith('running');
+}
+
+function gcloud(argv) {
+ return run('gcloud', [...argv, '--project', PROJECT], { timeout: 300_000, maxBuffer: 8 * 1024 * 1024 });
+}
+
+async function main() {
+ const roots = [];
+ for (const root of STATE_ROOTS) {
+ if (await exists(join(HOME, root))) roots.push(root);
+ else say(` note: ${root} is missing under ${HOME}, skipping it`);
+ }
+ if (!roots.length) die(`no state to seed: neither ${STATE_ROOTS.join(' nor ')} exists under ${HOME}`);
+
+ if (await daemonIsRunning()) {
+ if (!FORCE) {
+ die(
+ 'the okx-a2a daemon is running, so its sqlite files are being written and the archive could tear.\n' +
+ ' Stop it first (the worker does this on SIGTERM), or pass --force if you accept a live copy.',
+ );
+ }
+ say(' WARNING: --force with a running daemon: this is a live copy and may tear');
+ }
+
+ await rm(ARCHIVE, { force: true });
+ await run('tar', ['-czf', ARCHIVE, '-C', HOME, ...STATE_EXCLUDES.flatMap((p) => ['--exclude', p]), ...roots]);
+ const bytes = (await stat(ARCHIVE)).size;
+
+ say(`\n home ${HOME}`);
+ say(` roots ${roots.join(', ')}`);
+ say(` archive ${ARCHIVE} (${(bytes / 1024 / 1024).toFixed(1)} MB)`);
+ say(` target gs://${BUCKET}/${OBJECT}`);
+
+ if (!APPLY) {
+ await rm(ARCHIVE, { force: true });
+ say('\n plan only: nothing was uploaded. Re-run with --apply to seed the bucket.\n');
+ return;
+ }
+
+ try {
+ await gcloud(['storage', 'cp', ARCHIVE, `gs://${BUCKET}/${OBJECT}`]);
+ const { stdout } = await gcloud([
+ 'storage',
+ 'objects',
+ 'describe',
+ `gs://${BUCKET}/${OBJECT}`,
+ '--format=value(size,generation,updated)',
+ ]);
+ say(`\n seeded: ${stdout.trim()}`);
+ say('\n The next okx-chat-bot boot restores this session instead of asking for an OTP.');
+ say(' Exactly one host may write this object, so stop the seeding host before the');
+ say(' Cloud Run service starts, or the two interleave snapshots.\n');
+ } finally {
+ await rm(ARCHIVE, { force: true });
+ }
+}
+
+main().catch((err) => die(err?.message || String(err)));
diff --git a/tests/okx-chat-bot.test.js b/tests/okx-chat-bot.test.js
index 51d54eeba..1fb2bcbff 100644
--- a/tests/okx-chat-bot.test.js
+++ b/tests/okx-chat-bot.test.js
@@ -53,6 +53,26 @@ describe('okx-chat-bot session classifier', () => {
expect(classify({ ...ONLINE, daemon: 'stale pid=4242 lockPid=4242' }).reason).toBe('daemon_down');
});
+ // `daemon status` reads a lock the daemon writes seconds after it is spawned,
+ // so every healthy boot passes through a window that looks identical to a dead
+ // daemon. Paging critical there fired on every restart and recovered a minute
+ // later, which teaches people to skip the alert that matters.
+ it('does not page for a daemon that is still claiming its lock', () => {
+ const v = classify({ ...ONLINE, daemon: 'stopped', daemonChild: { pid: 4242, uptimeMs: 8_000 } });
+ expect(v).toMatchObject({ status: 'unknown', ready: false, reason: 'daemon_starting' });
+ });
+
+ // The grace window delays the claim, it does not withdraw it: a child that has
+ // been up for minutes without a lock is genuinely failing to start.
+ it('still calls a daemon down once the boot window has passed', () => {
+ const v = classify({ ...ONLINE, daemon: 'stopped', daemonChild: { pid: 4242, uptimeMs: 300_000 } });
+ expect(v).toMatchObject({ status: 'down', reason: 'daemon_down' });
+ });
+
+ it('calls a daemon with no child at all down immediately', () => {
+ expect(classify({ ...ONLINE, daemon: 'stopped', daemonChild: { pid: null, uptimeMs: 0 } }).reason).toBe('daemon_down');
+ });
+
it('flags a logged-out session as needing a human, not a restart', () => {
const v = classify({ ...ONLINE, wallet: { loggedIn: false }, agents: { agentCount: 0, activeClients: 0 } });
expect(v.status).toBe('down');
diff --git a/workers/README.md b/workers/README.md
index 606a2c7fb..2adf92eb1 100644
--- a/workers/README.md
+++ b/workers/README.md
@@ -80,8 +80,11 @@ delivered through, restores that identity from GCS on boot, and rebuilds the AI
subsession's briefing and skills from the image every time. Readiness is strict:
a bot that cannot receive a buyer's message reports 503 on `/readyz` even though
the process is alive, because the outage this worker exists to kill is silent.
-Must run `--min-instances=1 --max-instances=1` (single snapshot writer). Run
-locally with `npm run worker:okx-bot`. See [okx-chat-bot/README.md](okx-chat-bot/README.md).
+It also asks the AI provider whether it will actually serve the host, rather than
+checking that a key is present: a credential the provider refuses reads exactly
+like a working one until a buyer's message goes unanswered. Must run
+`--min-instances=1 --max-instances=1` (single snapshot writer). Run locally with
+`npm run worker:okx-bot`. See [okx-chat-bot/README.md](okx-chat-bot/README.md).
### `robinhood-feed/`
Robinhood Chain (4663) firehose: watches the NOXA and The Odyssey launchpads
diff --git a/workers/okx-chat-bot/README.md b/workers/okx-chat-bot/README.md
index b7bf4db48..d5ed19776 100644
--- a/workers/okx-chat-bot/README.md
+++ b/workers/okx-chat-bot/README.md
@@ -29,11 +29,13 @@ in every state below except `online`.
| `reason` | Status | Ready | What it means |
|---|---|---|---|
+| `daemon_starting` | unknown | no | The daemon child is alive but has not claimed its lock yet. Deliberately not a page: `daemon status` reads a lock file written seconds after the spawn, so every healthy boot passes through this window. |
| `daemon_down` | down | no | `okx-a2a` is not running. No chat is delivered. |
| `wallet_unreadable` | unknown | no | `onchainos wallet status` did not answer. Session state unknown. |
| `session_logged_out` | down | no | Session expired. Every XMTP client is offline. **Needs a human OTP.** |
| `no_active_client` | degraded | no | Logged in, but 0 XMTP clients serving. The daemon retries every minute. |
-| `ai_provider_uncredentialed` | degraded | no | Chat arrives but the subsession has no key, so it cannot author a reply. |
+| `ai_provider_uncredentialed` | degraded | no | Chat arrives but no credential is configured, so the subsession cannot author a reply. |
+| `ai_provider_unauthorized` | degraded | no | A credential IS configured and the provider **refuses** it (expired, revoked, or an account that cannot bill). Chat arrives and every reply dies. **Needs a human.** |
| `online` | ok | yes | At least one XMTP client is serving at least one agent identity. |
`classify()` in [session.js](session.js) is pure, so this state machine is
@@ -46,6 +48,7 @@ testable without a daemon, a wallet, or a network.
| `index.js` | Entrypoint. Boot order, session probe, heartbeat, ops alerts, graceful shutdown. |
| `config.js` | Env-driven config (`loadConfig`, `paths`) and AI-provider selection (`resolveProvider`). |
| `cli.js` | Timeout-bounded, non-throwing wrappers around the `okx-a2a` and `onchainos` binaries. |
+| `provider.js` | Asks the AI provider whether it will actually serve this host, and hands codex its key. |
| `session.js` | Pure health `classify()` plus `loginInstructions()`, the exact commands a human runs. |
| `health-server.js` | The HTTP surface: strict `/readyz`, always-200 liveness, and the `remedy` payload. |
| `state.js` | Tar the wallet/XMTP identity to GCS and restore it on boot (`snapshotState`, `restoreState`). |
@@ -100,6 +103,11 @@ locally with no code change. Defaults are the production posture.
| `OKX_BOT_STATE_OBJECT` | `okx-chat-bot/state.tar.gz` | Object name within that bucket. |
| `OKX_BOT_REPO_ROOT` | `/app` | Where the briefing and skills are read from. |
| `OKX_BOT_AI_PROVIDER` | auto | Pin the provider (`claude`, `codex`, `hermes`, `openclaw`). |
+| `CLAUDE_CODE_USE_VERTEX` | unset | `1` routes the Claude subsession to Vertex AI, authenticated by the runtime service account. The production posture: no secret exists to leak, rotate, or forget. |
+| `ANTHROPIC_VERTEX_PROJECT_ID` | unset | The GCP project Vertex bills. Required alongside the flag above. |
+| `CLOUD_ML_REGION` | `global` | Vertex region for both the subsession and the credential probe. |
+| `OKX_A2A_AI_PERMISSION_PRESET` | unset | `bypass` on a headless host. Without it the subsession stalls on a tool-approval prompt nobody is there to answer, which the buyer experiences as an unresponsive bot. |
+| `OKX_BOT_PROVIDER_PROBE_MS` | `900000` | How often the provider's own API is asked whether the credential still works. |
| `OKX_BOT_HOST_LABEL` | auto | Name this host on every beat. Cloud Run names itself from `K_SERVICE`. |
| `OKX_BOT_HOST_DURABLE` | unset | Set to `1` to claim a non-Cloud-Run host stays up on its own. |
| `OKX_BOT_DAEMON_BIN` | `okx-a2a` | The XMTP daemon binary the supervisor owns. |
@@ -113,12 +121,52 @@ locally with no code change. Defaults are the production posture.
**Provider selection is by credential, not by preference.** A provider CLI with
no key spawns, fails to authenticate, and produces exactly the symptom this
worker exists to kill: silence on the buyer's side. So an explicit
-`OKX_BOT_AI_PROVIDER` wins; otherwise `ANTHROPIC_API_KEY` or
-`CLAUDE_CODE_OAUTH_TOKEN` selects `claude`, and `OPENAI_API_KEY` selects
-`codex`. A developer host carries no key at all: its claude CLI was logged in by
-a human, so an existing `$OKX_BOT_HOME/.claude/.credentials.json` also counts as
-a credential. With no credential at all the worker boots, logs an error, and
-reports `ai_provider_uncredentialed` rather than pretending to be healthy.
+`OKX_BOT_AI_PROVIDER` wins; otherwise Vertex (`CLAUDE_CODE_USE_VERTEX=1` plus a
+project) selects `claude`, then `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`
+selects `claude`, then `OPENAI_API_KEY` selects `codex`. A developer host carries
+no key at all: its claude CLI was logged in by a human, so an existing
+`$OKX_BOT_HOME/.claude/.credentials.json` also counts as a credential. With no
+credential at all the worker boots, logs an error, and reports
+`ai_provider_uncredentialed` rather than pretending to be healthy.
+
+**Vertex wins over every key on purpose.** The credential is the runtime service
+account, reached through ADC, so there is nothing to mint, rotate, paste into a
+secret, or forget to renew, and the spend lands on the GCP credit pool the
+platform already prefers over paid third-party APIs. `three-ws@` holds
+`roles/aiplatform.user`, so the deploy needs no AI secret at all.
+
+**A configured credential is not a working one, and the probe says which.**
+This is the trap that would otherwise rebuild the worker's own defining failure
+one level up. Measured 2026-09-04, *both* credentials this project holds are
+present, well-formed, and refuse to serve: Vertex answers
+`PERMISSION_DENIED: Lightning dunning decision is deny` (a billing hold that
+reads like an IAM problem) and the `openai-api-key` secret's account answers
+`billing_not_active`. A presence check calls either one green, the host reports
+ready, a buyer's message lands, and no reply is ever authored.
+
+So [provider.js](provider.js) asks the provider itself: one tiny request at boot
+and every `OKX_BOT_PROVIDER_PROBE_MS`, classified into `ok`, `unauthorized`,
+`unreachable`, or `unprobed`. Only `unauthorized` fails readiness. `unreachable`
+deliberately does not: a provider outage is transient and self-heals, and paging
+a human for something no human can fix is how alerts stop being read. An
+interactive OAuth grant is `unprobed`, because the CLI refreshes that grant
+itself and no endpoint here can prove it without reimplementing the refresh.
+
+**One thing about Vertex that could not be tested here.** The billing hold denies
+every Vertex call on this project, so the credential probe was verified against
+the real endpoint (it returns the 403 above, and `classifyProbeStatus` calls that
+`unauthorized`) but the spawned subsession's own Vertex round trip was not. When
+the hold clears, watch the first reply: if the CLI reports an unknown model, pin
+one Vertex publishes with `ANTHROPIC_MODEL` on the service (a config-only
+`gcloud run services update --update-env-vars`, which is pre-approved). Nothing
+else about the transport is in question: the URL shape, the region and the
+service account's `roles/aiplatform.user` were all exercised.
+
+**Codex needs a login, not an env var.** Codex >= 0.153 authenticates from
+`~/.codex/auth.json` and ignores `OPENAI_API_KEY` in the environment: with the
+key set and no login, every request fails
+`401 Missing bearer or basic authentication in header`. `loginCodex()` runs
+`codex login --with-api-key` at boot, before the daemon can spawn a subsession.
`DATABASE_URL` is also required, for the heartbeat row.
@@ -163,96 +211,94 @@ someone has to go find.
## Deploying
-**Status: running as a stopgap on a developer codespace, not yet deployed.**
-There is no `okx-chat-bot` Cloud Run service in `aerial-vehicle-466722-p5`.
-Two of the three one-time prerequisites now exist (created 2026-09-02):
-`gs://three-ws-okx-bot-state` (versioned, `three-ws@` holds `objectAdmin`) and
-the `okx-chat-bot-database-url` secret (copied from the project's existing
-`DATABASE_URL` secret so the two cannot drift, `three-ws@` holds
-`secretAccessor`). **The one thing still missing is the AI-provider secret**,
-`anthropic-api-key`, whose value exists nowhere on this machine. The deploy
-references it in `--set-secrets` and will fail loudly without it, which is the
-intended trade: a bot that receives buyer chat and cannot answer is worse than a
-refused deploy.
-
-Since 2026-09-02 this worker has beat for the first time, from a codespace, so
-`/api/healthz` reports the `okx_chat_bot` subsystem instead of `unknown`. That
-is a stopgap and says so on the wire: every beat carries `host` and
-`hostDurable`, and a beat whose host cannot survive on its own reads as
-**degraded**, never `ok`, with the deploy command as its hint. Calling a
-codespace green would rebuild, one level up, the false-green this worker exists
-to kill.
-
-### The first boot pages for an OTP, unless the state object is seeded
-
-`gs://three-ws-okx-bot-state` holds no snapshot yet, so the first revision boots
-logged out and alerts for a human email OTP as `[email protected]`. That round trip
-is avoidable: a machine already holding a live session (the codespace stopgap
-does) can write the same archive this worker writes, and the new host then
-restores an authenticated session and comes up online.
+**Status: built, prerequisites complete, deploy owner-gated.** There is still no
+`okx-chat-bot` Cloud Run service in `aerial-vehicle-466722-p5`; a codespace runs
+the stopgap. Everything the deploy depends on now exists:
-```bash
-# On the machine holding the live session, with its daemon STOPPED so the sqlite
-# files are quiesced. The worker's SIGTERM path stops it for you.
-OKX_BOT_STATE_BUCKET=three-ws-okx-bot-state node -e "
- import('./workers/okx-chat-bot/state.js').then(async ({ snapshotState }) => {
- const { loadConfig } = await import('./workers/okx-chat-bot/config.js');
- console.log(await snapshotState(loadConfig(), { reason: 'seed' }));
- })"
-```
-
-Two caveats. The archive carries the wallet keyring and session, so it belongs in
-this private bucket and nowhere else, and `snapshotState` needs
-`GCP_SERVICE_ACCOUNT_JSON` on a machine that is not already on GCP. And the
-bucket has exactly one writer: stop the seeding host before the Cloud Run service
-starts, or the two interleave snapshots. Skipping the seed is not a failure, it
-just leaves the OTP that the first boot would have asked for anyway.
+| Prerequisite | State |
+|---|---|
+| `gs://three-ws-okx-bot-state` | created 2026-09-02, versioned, `three-ws@` holds `objectAdmin` |
+| `okx-chat-bot-database-url` secret | created 2026-09-02 from the project's own `DATABASE_URL`, `three-ws@` holds `secretAccessor` |
+| AI credential | **no secret needed.** `three-ws@` already holds `roles/aiplatform.user`, so `CLAUDE_CODE_USE_VERTEX=1` in the deploy authenticates through ADC |
+| Seeded session | seeded 2026-09-04, so the first boot restores an authenticated wallet instead of paging for an OTP |
+
+The AI-provider secret used to be the one blocker, and the deploy was written to
+fail loudly without it on the reasoning that a bot receiving chat it can never
+answer is worse than a refused deploy. That was right about the failure and wrong
+about the remedy: the key was never minted, the deploy never happened, and chat
+kept being served by a workspace that dies every night. Vertex removes the trade
+(the credential cannot go missing) and the case the gate really guarded against,
+a credential that is present but refused, is now caught at runtime by the
+credential probe and reported as `ai_provider_unauthorized` with the fix in the
+`remedy`.
+
+Since 2026-09-02 this worker has beat from a codespace, so `/api/healthz` reports
+the `okx_chat_bot` subsystem instead of `unknown`. That is a stopgap and says so
+on the wire: every beat carries `host` and `hostDurable`, and a beat whose host
+cannot survive on its own reads as **degraded**, never `ok`, with the deploy
+command as its hint. Calling a codespace green would rebuild, one level up, the
+false-green this worker exists to kill.
+
+### Ship it (one command, after the two steps below)
-The service must run `--min-instances=1 --max-instances=1`. This is not a
-capacity choice: the GCS snapshot has exactly one writer, and concurrent
-revisions would interleave snapshots and corrupt the identity.
+```bash
+# 1. Refresh the seeded session from the host that holds it, daemon stopped:
+npm run okx:bot:seed-state -- --apply
-Shipping is one command once the setup below exists:
+# 2. Stop that host. The GCS object has exactly one writer; a codespace stopgap
+# and the Cloud Run service running at once interleave snapshots.
-```bash
+# 3. Deploy.
gcloud builds submit --config workers/okx-chat-bot/cloudbuild.yaml \
--region us-central1 --project aerial-vehicle-466722-p5 \
--substitutions=SHORT_SHA=manual$(date +%s) .
```
The build pins `three-ws-build@` and the service runs as `three-ws@`; the
-project's default compute service account was deleted, so both pins are
-required. The bucket and secret commands are at the top of
-[cloudbuild.yaml](cloudbuild.yaml).
+project's default compute service account was deleted, so both pins are required.
+The service must run `--min-instances=1 --max-instances=1`. That is not a capacity
+choice: the GCS snapshot has exactly one writer, and concurrent revisions would
+interleave snapshots and corrupt the identity.
Cloud Run's startup probe is wired to `/healthz` and deliberately **not** to
`/readyz`, for the same reason liveness is always-200.
-One-time setup (none of it has run yet; the exact commands are at the top of
-[cloudbuild.yaml](cloudbuild.yaml)):
+### Seeding the session (why the first boot does not page)
+
+`gs://three-ws-okx-bot-state` holds a snapshot taken 2026-09-04 from the codespace
+stopgap, verified by restoring it into a throwaway HOME and reading
+`onchainos wallet status` back as `loggedIn: true, [email protected]`. So the first
+Cloud Run revision restores an authenticated session and comes up online rather
+than alerting for an email OTP.
+
+[scripts/okx-bot-seed-state.mjs](../../scripts/okx-bot-seed-state.mjs) (`npm run
+okx:bot:seed-state`) writes that archive from any machine holding a live session.
+It builds the tar from the same exported `STATE_ROOTS` / `STATE_EXCLUDES` the
+worker uses, so the two cannot drift, and uploads it with `gcloud` (the deployed
+host uses ADC; a developer codespace has none). A bare run is plan-only; `--apply`
+uploads. It **refuses to run while the daemon is up**, because a live copy of the
+XMTP sqlite files can tear, and a torn identity costs a human OTP to recover.
+
+The archive carries the wallet keyring and the XMTP identity. It belongs in that
+private bucket and nowhere else.
-1. Create the state bucket and grant the runtime service account
- `roles/storage.objectAdmin` on it.
-2. Create the `okx-chat-bot-database-url` and `anthropic-api-key` secrets and
- grant the runtime service account `roles/secretmanager.secretAccessor` on
- both. The deploy step wires them by name and **fails if either is missing**,
- which is deliberate: a bot with no AI credential receives buyer chat and
- never answers it. `OKX_BOT_STATE_BUCKET` needs no manual step, the deploy
- sets it.
-3. Confirm the image has both CLIs installed and the repo at `OKX_BOT_REPO_ROOT`.
+Skipping the seed is not a failure; it just leaves the OTP the first boot would
+otherwise have asked for.
-Never patch the AI key onto the service by hand. `--set-secrets` in the deploy
-step replaces the whole secret set, so a hand-added key survives exactly until
-the next deploy and then vanishes without a single error line.
+### Switching off Vertex
-The first boot logs `no state snapshot yet: first boot for this bucket` and
-comes up logged out, which pages a human to complete the initial OTP as
-`[email protected]`. Every boot after that restores the session.
+Never patch an AI key onto the service by hand. `--set-secrets` in the deploy step
+replaces the whole secret set, so a hand-added key survives exactly until the next
+deploy and then vanishes without a single error line. To use a key instead of
+Vertex, edit [cloudbuild.yaml](cloudbuild.yaml): drop `CLAUDE_CODE_USE_VERTEX`
+from `--set-env-vars` (it wins over every key) and add
+`ANTHROPIC_API_KEY=anthropic-api-key:latest` to `--set-secrets`, after creating
+that secret and granting `three-ws@` `secretAccessor` on it.
-Shutdown order matters and is handled on SIGTERM: the daemon is stopped
-**before** the final snapshot, so the sqlite files are quiesced rather than
-copied mid-write. The periodic timer snapshot is a live copy and is
-best-effort by design, so an ungraceful kill loses minutes, not the identity.
+Shutdown order matters and is handled on SIGTERM: the daemon is stopped **before**
+the final snapshot, so the sqlite files are quiesced rather than copied mid-write.
+The periodic timer snapshot is a live copy and is best-effort by design, so an
+ungraceful kill loses minutes, not the identity.
## Monitoring
@@ -282,11 +328,12 @@ URL=$(gcloud run services describe okx-chat-bot --region us-central1 \
curl -s -H "Authorization: Bearer $(gcloud auth print-identity-token)" "$URL/readyz" | jq .remedy
```
-Two signatures are classified in
+Three signatures are classified in
[scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), so `npm run triage:gcp`
explains them instead of filing them as unknown: `okx-bot-session-logged-out`
-(owner action, needs the OTP) and `okx-bot-daemon-restart` (self-healing unless
-the restart count climbs continuously).
+(owner action, needs the OTP), `okx-bot-provider-unauthorized` (owner action, the
+AI credential is present and refused) and `okx-bot-daemon-restart` (self-healing
+unless the restart count climbs continuously).
A transition into a bad state fires an ops alert through `sendOpsAlert`, at most
once per `OKX_BOT_ALERT_REPEAT_MS` while it persists, so one overnight expiry
@@ -298,5 +345,6 @@ Daemon stdout and stderr are forwarded into the worker's own log stream under a
## Related
- [scripts/okx-bot-revive.mjs](../../scripts/okx-bot-revive.mjs) stages the same workspace locally.
+- [scripts/okx-bot-seed-state.mjs](../../scripts/okx-bot-seed-state.mjs) seeds the GCS session snapshot (`npm run okx:bot:seed-state`).
- [api/_lib/okx-chat-briefing.js](../../api/_lib/okx-chat-briefing.js) generates the subsession briefing.
- [workers/README.md](../README.md) is the worker index.
diff --git a/workers/okx-chat-bot/index.js b/workers/okx-chat-bot/index.js
index 4e9ed2614..148cf1bda 100644
--- a/workers/okx-chat-bot/index.js
+++ b/workers/okx-chat-bot/index.js
@@ -45,7 +45,7 @@ const live = {
login: null,
workspace: null,
stateRestore: null,
- // What the provider's own API last answered. Undefined until the first probe,
+ // What the provider's own API last answered. Null until the first probe lands,
// which is why classify() treats a missing verdict as "no opinion" rather than
// as a failure: a host must not read red for a probe that has not run yet.
providerProbe: null,
@@ -159,6 +159,7 @@ async function runProbe(cfg, supervisor) {
agents,
providerCredentialed: cfg.providerCredentialed,
providerProbe: live.providerProbe,
+ daemonChild: supervisor.stats(),
});
// Mint a fresh login URL only when one is actually needed, and only when we
@@ -252,8 +253,10 @@ async function main() {
// Run the first credential probe immediately, but off the boot path: a slow
// provider must delay the verdict, never the daemon that receives the chat.
probeProviderCredential(cfg).catch((err) => log.warn('provider probe failed', { err: err?.message }));
- // Give the daemon a moment to bind XMTP before the first verdict, so a normal
- // boot does not page as "down".
+ // A first verdict early enough to be useful. It can legitimately land before
+ // the daemon has claimed its lock, which classify() reports as
+ // `daemon_starting` rather than `daemon_down`, so an ordinary boot no longer
+ // pages critical and recovers a minute later.
setTimeout(probe, 10_000);
let draining = false;
diff --git a/workers/okx-chat-bot/session.js b/workers/okx-chat-bot/session.js
index a5f07ff7b..7da220a1f 100644
--- a/workers/okx-chat-bot/session.js
+++ b/workers/okx-chat-bot/session.js
@@ -9,6 +9,13 @@
// So readiness here is deliberately strict: a bot that cannot receive a message
// is NOT ready, even though the process is perfectly alive.
+// How long a freshly spawned daemon may go without claiming its lock before the
+// silence counts as a failure. Measured boots take ~11s on this host; a cold
+// container restoring a state tree first is slower, so the window is generous.
+// It only delays the verdict, never the recovery: the supervisor is already
+// restarting a child that truly died.
+const DAEMON_BOOT_GRACE_MS = 90_000;
+
/** The exact commands a human runs to bring an expired session back. */
export function loginInstructions(loginUrl, authSessionId) {
const url = loginUrl || 'run `onchainos wallet login --phase init` to mint one';
@@ -58,13 +65,30 @@ export function providerInstructions(detail = '') {
* @param {{ agentCount: number, activeClients: number }} probe.agents
* @param {boolean} probe.providerCredentialed
* @param {{ code: string, detail: string }} [probe.providerProbe] what the provider's own API answered
+ * @param {{ pid: number|null, uptimeMs: number }} [probe.daemonChild] supervisor.stats(), so a daemon
+ * still writing its lock is told apart from one that never came up
* @returns {{ status: 'ok'|'degraded'|'down'|'unknown', ready: boolean, reason: string, detail: string,
* needsHumanLogin: boolean }}
*/
-export function classify({ daemon, wallet, agents, providerCredentialed, providerProbe }) {
+export function classify({ daemon, wallet, agents, providerCredentialed, providerProbe, daemonChild }) {
const daemonRunning = typeof daemon === 'string' && daemon.startsWith('running');
if (!daemonRunning) {
+ // `okx-a2a daemon status` reads a lock file the daemon writes seconds after
+ // it is spawned, so a healthy boot spends a window looking exactly like a
+ // dead one. Calling that window `down` paged critical on every single
+ // restart and then recovered a minute later, and an alert that cries wolf
+ // on every deploy is an alert people learn to skip. A live child inside the
+ // grace window is `unknown`: still not ready, but not yet a claim.
+ if (daemonChild?.pid && daemonChild.uptimeMs < DAEMON_BOOT_GRACE_MS) {
+ return {
+ status: 'unknown',
+ ready: false,
+ reason: 'daemon_starting',
+ detail: `okx-a2a was spawned ${Math.round(daemonChild.uptimeMs / 1000)}s ago and has not claimed its lock yet`,
+ needsHumanLogin: false,
+ };
+ }
return {
status: 'down',
ready: false,