Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

33.5 KB
Raw Blame

okx-chat-bot

Always-on host for the OKX.AI marketplace chat bot (agent #2632).

Buyers message our marketplace listing over XMTP. That chat is delivered to a local okx-a2a daemon, which reads agent identities through an onchainos wallet session and spawns an AI subsession to author the reply and drive the task lifecycle (accept, negotiate, deliver). Both CLIs used to run on a developer codespace, which cannot stay up: a rebuild wipes the CLIs and an idle nap kills the daemon (observed alive at 21:09, dead by 03:13 the same night). OKX's own chat test then reports "no delivery in 30 minutes" and flags the listing offline.

This worker is the durable host for that pair. It restores the wallet identity, supervises the daemon, rebuilds the AI subsession's world knowledge on every boot, and makes the one failure a human must clear (an expired OKX session) page loudly with the exact commands to fix it.

The failure it exists to kill

The dangerous outage here is silent. The process stays up, the container reports healthy, and chat is simply never delivered because the wallet session expired or no XMTP client came online. From the outside that is indistinguishable from "nobody messaged us".

So readiness is deliberately strict: a bot that cannot receive a message is not ready, even though the process is perfectly alive. /readyz returns 503 in every state below except online.

reason Status Ready What it means
daemon_starting unknown no The daemon child is alive but has not claimed its lock yet. Deliberately not a page: daemon status reads a lock file written seconds after the spawn, so every healthy boot passes through this window.
daemon_down down no okx-a2a is not running. No chat is delivered.
wallet_unreadable unknown no onchainos wallet status did not answer. Session state unknown.
session_logged_out down no Session expired. Every XMTP client is offline. Needs a human OTP.
no_active_client degraded no Logged in, but 0 XMTP clients serving. The daemon retries every minute.
ai_provider_uncredentialed degraded no Chat arrives but no credential is configured, so the subsession cannot author a reply.
ai_provider_unauthorized degraded no A credential IS configured and the provider refuses it (expired, revoked, or an account that cannot bill). Chat arrives and every reply dies. Needs a human.
online ok yes At least one XMTP client is serving at least one agent identity.

classify() in session.js is pure, so this state machine is testable without a daemon, a wallet, or a network.

Architecture

File Role
index.js Entrypoint. Boot order, session probe, heartbeat, ops alerts, graceful shutdown.
config.js Env-driven config (loadConfig, paths) and the AI lane chain (providerLanes, resolveProviderChain, applyLane).
cli.js Timeout-bounded, non-throwing wrappers around the okx-a2a and onchainos binaries.
provider.js Probes each AI lane (probeLane), elects the first that serves (electProvider), and hands codex its key.
session.js Pure health classify() plus loginInstructions(), the exact commands a human runs.
health-server.js The HTTP surface: strict /readyz, always-200 liveness, and the remedy payload.
state.js Tar the wallet/XMTP identity to GCS and restore it on boot (snapshotState, restoreState).
supervisor.js Owns the okx-a2a run child with capped exponential backoff restarts.
lease.js The single-writer lease: no two hosts restore, run or snapshot the bot identity at once, rollouts included (createLease, sqlLeaseStore).
workspace.js Rebuilds the AI subsession's briefing and skills from the image on every boot.
log.js Structured JSON lines for Cloud Logging.

Four things that are load-bearing

The daemon runs in the foreground, never via daemon start. okx-a2a daemon start delegates to an OS autostart unit (systemd/launchd). There is no systemd in a container, so that call installs a unit and silently leaves the daemon down: the exact trap that made the local bot look staged but offline. The supported foreground entrypoint is okx-a2a run, and supervisor.js owns that child directly. It also clears the lock file before every spawn, because a crashed daemon leaves one behind that blocks the next start.

Identity lives on disk, not in a database. ~/.onchainos/keyring.enc plus session.json and machine-identity are what make the wallet session survive, and ~/.okx-agent-task/ holds the XMTP client database. Cloud Run's filesystem is in-memory and dies with the revision, so state.js tars both trees to one GCS object and restores it on boot. Without that, every deploy would log the bot out and need a fresh human OTP.

A daemon that cannot be spawned must not take the host with it. A missing or unrunnable okx-a2a binary raises error on the child, not just exit. Node throws on an unhandled error event, so without a listener the whole worker dies and "the daemon binary is missing" surfaces as "the host is gone": no health verdict, no heartbeat, no alert naming the real problem. supervisor.js handles both events through one restart path, and a spawn that raises both never schedules two restarts.

The AI workspace is rebuilt from the image, not from the snapshot. The adapter spawns the AI CLI with cwd set to ~/.okx-agent-task/workspace, and whatever is in that directory is the subsession's world knowledge. A naive containerisation ships an agent that knows nothing about three.ws and improvises answers to paying buyers. workspace.js writes a briefing (as both CLAUDE.md and AGENTS.md, since which one is read depends on the spawned CLI) and copies 12 skills in on every boot, so a redeploy always ships the current catalog.

Configuration

Every knob is env-driven, so the same image runs on Cloud Run, on a plain VM, and locally with no code change. Defaults are the production posture.

Env Default Purpose
OKX_BOT_HOME $HOME HOME for both CLIs. Decides where all durable state lands.
PORT unset Health server port. Unset means no health server (fine locally, never on Cloud Run).
OKX_BOT_AGENT_ID 2632 The marketplace agent this bot answers for.
OKX_BOT_STATE_BUCKET unset GCS bucket for the state snapshot. Unset means ephemeral mode.
OKX_BOT_STATE_OBJECT okx-chat-bot/state.tar.gz Object name within that bucket.
OKX_BOT_REPO_ROOT /app Where the briefing and skills are read from.
OKX_BOT_AI_PROVIDER auto Pin the provider (claude, codex, hermes, openclaw). Narrows the chain to that CLI's lanes rather than emptying it.
CLAUDE_CODE_USE_VERTEX unset 1 routes the Claude subsession to Vertex AI, authenticated by the runtime service account. The production posture: no secret exists to leak, rotate, or forget.
ANTHROPIC_VERTEX_PROJECT_ID unset The GCP project Vertex bills. Required alongside the flag above.
CLOUD_ML_REGION global Vertex region for both the subsession and the credential probe.
OKX_A2A_AI_PERMISSION_PRESET unset bypass on a headless host. Without it the subsession stalls on a tool-approval prompt nobody is there to answer, which the buyer experiences as an unresponsive bot.
OKX_BOT_ANTHROPIC_BASE_URL unset Base URL of an Anthropic-wire-format gateway. The CLI appends /v1/messages. The deploy sets three.ws's own proxy, https://three.ws/api/llm/anthropic/agents/<metering agent>; OpenRouter's would be https://openrouter.ai/api, not .../api/v1.
OKX_BOT_ANTHROPIC_AUTH_TOKEN unset That gateway's credential. Both it and the base URL are required, or no gateway lane exists. On three.ws's proxy it is an API key of the service account that owns the metering agent, from the okx-chat-bot-llm-gateway-token secret.
OKX_BOT_ANTHROPIC_MODEL unset Model id in the gateway's catalog. The deploy sets nvidia/nemotron-3-super-120b-a12b, a free model on the proxy's chain.
OKX_BOT_LEASE_TTL_MS 120000 How long an unrenewed single-writer lease stays taken. Bounds how long a host that died holding it blocks its successor.
OKX_BOT_LEASE_RENEW_MS 20000 Lease renewal cadence. Must be under half the TTL.
OKX_BOT_LEASE_POLL_MS 5000 How often a waiting host asks for the lease again.
OKX_BOT_LEGACY_BEAT_STALE_MS 3 x heartbeat How old the heartbeat of a host built before the lease must be before a new host stops waiting for it.
OKX_BOT_PROVIDER_PROBE_MS 900000 How often every lane's own API is asked whether it will still serve. Also how quickly a recovered lane is picked up.
OKX_BOT_HOST_LABEL auto Name this host on every beat. Cloud Run names itself from K_SERVICE.
OKX_BOT_HOST_DURABLE unset Set to 1 to claim a non-Cloud-Run host stays up on its own.
OKX_BOT_DAEMON_BIN okx-a2a The XMTP daemon binary the supervisor owns.
OKX_BOT_HEARTBEAT_MS 30000 How often the bot_heartbeat row is written. Its own timer, not the probe's.
OKX_BOT_SESSION_PROBE_MS 60000 How often health is re-probed.
OKX_BOT_SNAPSHOT_MS 300000 Periodic state snapshot cadence.
OKX_BOT_ALERT_REPEAT_MS 21600000 Re-alert ceiling while a bad state persists (6 h).
OKX_BOT_RESTART_BASE_MS 2000 Daemon restart backoff floor.
OKX_BOT_RESTART_MAX_MS 60000 Daemon restart backoff ceiling.

The AI chain: whichever lane is funded is the lane that serves

The host used to hold exactly one AI lane, which is a chain with a single rung. On 2026-09-04 the rung snapped: this GCP project's Vertex access started answering PERMISSION_DENIED: Lightning dunning decision is deny, and the bot went on receiving every buyer message and authoring no reply at all, for days, while other credentials sat unused on the same service.

So providerLanes() builds an ordered chain and electProvider() probes it, best lane first, and elects the first one that actually answers. The election re-runs every OKX_BOT_PROVIDER_PROBE_MS, so a lane that recovers (or one the owner funds) is picked up with no deploy: the config is re-asserted, codex is re-logged-in if it is the new lane, and the daemon is restarted so the subsession it spawns carries the new credentials.

# Lane Selected by Why here
1 vertex CLAUDE_CODE_USE_VERTEX=1 + a project GCP credits, authenticated by the runtime service account. Nothing to mint, rotate or forget, and the spend lands on the pool the platform prefers over any paid third-party API.
2 anthropic-key ANTHROPIC_API_KEY A first-party key.
3 anthropic-gateway OKX_BOT_ANTHROPIC_BASE_URL + ..._AUTH_TOKEN Any service that speaks the Anthropic wire format. The deploy points it at three.ws's own we-pay proxy on the free model chain, so it needs no payment anywhere: see The payment-free lane below. Behind Vertex, so Claude on GCP credits leads again the moment the billing hold clears.
4 anthropic-login $OKX_BOT_HOME/.claude/.credentials.json A developer host, whose claude CLI a human logged in.
5 openai-key OPENAI_API_KEY The codex CLI.

Every lane spawns a genuinely agentic CLI, and that is not negotiable. The adapter does not read a reply out of the CLI's stdout: the spawned subsession sends the reply itself through okx-a2a and drives the task lifecycle (accept / negotiate / deliver). A one-shot completion call would not just answer worse, it would break the lifecycle. The gateway lane exists precisely because it keeps the same claude CLI and only moves its endpoint.

A provider CLI with no key spawns, fails to authenticate, and produces exactly the symptom this worker exists to kill: silence on the buyer's side. So with an empty chain the worker boots, logs an error, and reports ai_provider_uncredentialed rather than pretending to be healthy.

Vertex wins over every key on purpose. The credential is the runtime service account, reached through ADC, so there is nothing to mint, rotate, paste into a secret, or forget to renew, and the spend lands on the GCP credit pool the platform already prefers over paid third-party APIs. three-ws@ holds roles/aiplatform.user, so the deploy needs no AI secret at all.

A configured credential is not a working one, and the probe says which. This is the trap that would otherwise rebuild the worker's own defining failure one level up. Measured 2026-09-04, both credentials this project holds are present, well-formed, and refuse to serve: Vertex answers PERMISSION_DENIED: Lightning dunning decision is deny (a billing hold that reads like an IAM problem) and the openai-api-key secret's account answers billing_not_active. A presence check calls either one green, the host reports ready, a buyer's message lands, and no reply is ever authored.

So provider.js asks each lane's own API: one tiny request at boot and every OKX_BOT_PROVIDER_PROBE_MS, classified into ok, unauthorized, unreachable, or unprobed. Only unauthorized fails readiness. unreachable deliberately does not: a provider outage is transient and self-heals, and paging a human for something no human can fix is how alerts stop being read. An interactive OAuth grant is unprobed, because the CLI refreshes that grant itself and no endpoint here can prove it without reimplementing the refresh, and unprobed is the one non-ok verdict a host may still be elected on.

Every lane's verdict, not just the elected one's, is on /readyz under provider.chain and in the remedy. A human deciding which credential to fund needs the whole picture, not the news that one of them was refused.

A gateway lane is elected on a 2xx and nothing else. classifyProbeStatus() reads a 400 or a 404 as proof the credential works, because api.anthropic.com had to authenticate the caller before it could object to the request. That reasoning does not survive a gateway: a base URL one path segment too deep, or a model id from a different catalog, answers exactly those statuses and then fails every real reply. Measured 2026-09-09, https://openrouter.ai/api/v1 makes the CLI request /api/v1/v1/messages and answers 404 forever, which is why the base URL in cloudbuild.yaml ends at /api.

The payment-free lane: three.ws's own proxy

The gateway lane does not need a third party. three.ws serves the Anthropic Messages format itself at /api/llm/anthropic (the proxy every avatar embed talks to, on the free model chain), and an SDK client reaches it at the agent-scoped base URL /api/llm/anthropic/agents/<agent>/v1/messages, the path the claude CLI builds from ANTHROPIC_BASE_URL. So the deploy sets:

OKX_BOT_ANTHROPIC_BASE_URL=https://three.ws/api/llm/anthropic/agents/<metering agent>
OKX_BOT_ANTHROPIC_MODEL=nvidia/nemotron-3-super-120b-a12b
OKX_BOT_ANTHROPIC_AUTH_TOKEN=okx-chat-bot-llm-gateway-token:latest   (secret)

Measured 2026-09-18 against production: the CLI's real request (the full agentic system prompt, 22 tools, streaming, max_tokens 32000, and an environment block sent as a system turn inside messages) ran through the proxy on nvidia/nemotron-3-super-120b-a12b, made a real Bash tool call, read the output back, and answered in 14 seconds. Two proxy changes made that possible and ship with the API: the agent-scoped path, and accepting the request shape the CLI sends by default (block-array system, in-thread system turns, a clamped rather than rejected max_tokens).

The proxy meters per agent, so the lane meters against a dedicated, unpublished agent owned by a platform service account ([email protected]), never anyone else's. scripts/okx-bot-llm-gateway.mjs (npm run okx:bot:gateway) provisions all of it idempotently and proves it with this worker's own probeLane:

  • Why a service account, not the platform's main account. The reply subsession runs with tool access on buyer-supplied text and can read its own environment, token included. A key for the main account would expose every agent and wallet the platform owns; a key for an account that owns one unpublished agent is worth that agent's AI budget and nothing else.
  • Why its own embed policy. A Claude Code turn is a 25k to 35k token prompt and several calls per reply. The embed default (10 calls/min, 1M tokens a month) would starve it within a few conversations, so the agent gets 30 calls/min, 20,000 calls and 30M tokens a month. On the free chain that ceiling is a runaway guard rather than a spend.
  • The key is minted straight into Secret Manager and never printed or written to disk; --rotate mints a new one before revoking the old.

One thing about Vertex that could not be tested here. The billing hold denies every Vertex call on this project, so the credential probe was verified against the real endpoint (it returns the 403 above, and classifyProbeStatus calls that unauthorized) but the spawned subsession's own Vertex round trip was not. When the hold clears, watch the first reply: if the CLI reports an unknown model, pin one Vertex publishes with ANTHROPIC_MODEL on the service (a config-only gcloud run services update --update-env-vars, which is pre-approved). Nothing else about the transport is in question: the URL shape, the region and the service account's roles/aiplatform.user were all exercised.

Codex needs a login, not an env var. Codex >= 0.153 authenticates from ~/.codex/auth.json and ignores OPENAI_API_KEY in the environment: with the key set and no login, every request fails 401 Missing bearer or basic authentication in header. loginCodex() runs codex login --with-api-key at boot, before the daemon can spawn a subsession.

DATABASE_URL is also required, for the heartbeat row.

Running it

Both CLIs must be on PATH. cliEnv() prepends $HOME/.local/bin (where the onchainos installer drops its binary) so a local run works whether or not the binary has been relocated to /usr/local/bin.

# Local, ephemeral: no state bucket, no health server.
npm run worker:okx-bot

# Local, with a health server and an explicit provider.
PORT=8080 OKX_BOT_AI_PROVIDER=claude npm run worker:okx-bot

Then read the health verdict:

curl -s localhost:8080/readyz | jq '{ready: .health.ready, reason: .health.reason, agents}'

To stage the same workspace on a developer machine without running the worker, use scripts/okx-bot-revive.mjs, which keeps the identical skill list. It refuses to start a daemon while any other host is serving agent 2632: see One writer, enforced, across rollouts too below.

One writer, enforced, across rollouts too

--max-instances=1 is per revision. A Cloud Run rollout starts the new revision's instance, waits for its startup probe, moves traffic, and only then stops the old one, so for that window two instances run. Before the lease below, both would have restored the identity and run a daemon against it, and the old instance's SIGTERM snapshot would then have overwritten the new one's state.

lease.js closes it. A host must hold a single-writer lease (a row in the bot_heartbeat table, worker = 'okx-chat-bot:lease', driven by one conditional upsert per transition, all on the database's clock) before it restores state, starts the daemon or writes a snapshot:

Moment What happens
New instance boots Health server first (so the startup probe passes and the rollout can move traffic), then waits, reporting lease_wait on /readyz with lease.waitingOn naming what it waits for. It writes no heartbeat while waiting, so /api/healthz keeps describing the host that is actually serving.
Old instance gets SIGTERM Stops its daemon, writes the final snapshot while it still holds the lease, then releases it.
Lease released The new instance takes it, restores exactly that snapshot, starts its daemon.
A holder dies without releasing Waited out for OKX_BOT_LEASE_TTL_MS (2 minutes).
A holder cannot renew (database unreachable) Fences itself before the TTL can lapse: stops the daemon and exits without a snapshot, since another host may by then be writing. Cloud Run restarts it into the wait.
Old revision predates the lease Its heartbeat carries no leaseHolder, so the new instance waits until that heartbeat is OKX_BOT_LEGACY_BEAT_STALE_MS (90 s) old. This is what makes the first rollout onto this code safe; the live revision on 2026-09-18, okx-chat-bot-00001-926, is such a host.

The SQL was verified against the production database on a throwaway key (acquire, refuse a second holder, renew, release, hand over, expire, lose), and the state machine is covered in tests/okx-chat-bot.test.js.

A second machine

The wallet keyring and the XMTP client database are one state object with exactly one writer, which is what --max-instances=1 protects on Cloud Run. Nothing protected it from a second machine, and npm run okx:bot is the one command that starts one. On 2026-09-09 it did: run from a codespace while the Cloud Run host was serving, it installed the CLIs, started a daemon, and came up agentCount=1 activeClients=1 against the same inbox. It was stopped within three minutes and the deployed host never missed a beat, but nothing about the command said no.

Now it asks first, through scripts/lib/okx-bot-host-guard.mjs:

What the health endpoint says Verdict
unreachable allowed, with a warning. An emergency revive must not need the internet to work
no heartbeat reported yet allowed. Nothing has ever hosted this bot
down, heartbeat stale allowed. The host is gone; this is the emergency the script exists for
any beat, host is this machine allowed. Re-staging the workspace where the daemon already runs adds no writer
any beat, another host refused, exit 3, --force to override
any beat, host not named refused. An API build older than the host field answers exactly this, and reading it as an all-clear is what started the rival daemon

The read is GET /api/healthz and nothing else: no DATABASE_URL, no gcloud login, no secret. The machine most likely to run this by mistake is a fresh clone with none of those, so the guard has to work there or it does not work at all. That is also why subsystem-health.js now puts host and hostDurable on the okx_chat_bot subsystem as fields rather than only inside a sentence.

HTTP surface

Path Behaviour
/readyz Strict. 200 only when health.ready is true, 503 otherwise.
any other path Liveness. Always 200, with the same status body under ok: true.

Liveness is deliberately always-200: Cloud Run must not restart the container for a logged-out session. A restart cannot fix it (only a human OTP can) and a restart loop would destroy the state snapshot cadence.

When a human is needed, the response carries a remedy array with the real commands, so the fix travels with the status instead of living in a runbook someone has to go find.

Deploying

Status: redeployed 2026-09-29. Service okx-chat-bot in aerial-vehicle-466722-p5, revision okx-chat-bot-00002-s4s, at https://okx-chat-bot-lp642k3kpa-uc.a.run.app (authenticated invocations only). It took over from the first revision, okx-chat-bot-00001-926 (deployed 2026-09-04), through the single-writer lease: it waited until the old host's heartbeat was 90 s quiet, restored the snapshot that host wrote on SIGTERM, and came up loggedIn: true with no OTP, one XMTP client serving agent 2632. The codespace stopgap is stopped and must stay stopped: the GCS state object has exactly one writer and Cloud Run owns it now.

It reports online and /api/healthz reads ok. Vertex still answers the billing hold (Lightning dunning decision is deny), so the election skips it and replies run on the anthropic-gateway lane through three.ws's own proxy. Clearing the hold needs no redeploy: the next 15-minute election moves the bot back to Claude on Vertex by itself.

Everything the deploy depends on:

Prerequisite State
gs://three-ws-okx-bot-state created 2026-09-02, versioned, three-ws@ holds objectAdmin
okx-chat-bot-database-url secret created 2026-09-02 from the project's own DATABASE_URL, three-ws@ holds secretAccessor
AI credential no secret needed. three-ws@ already holds roles/aiplatform.user, so CLAUDE_CODE_USE_VERTEX=1 in the deploy authenticates through ADC
Seeded session seeded 2026-09-04 and proven: the first revision restored it and needed no OTP
Payment-free AI lane npm run okx:bot:deploy provisions it (metering agent, service-account key, okx-chat-bot-llm-gateway-token secret with three-ws@ as secretAccessor) through scripts/okx-bot-llm-gateway.mjs --apply. Provisioned 2026-09-29: service account [email protected], metering agent 55479fa8-13c6-44b3-a245-3156f2c836e5, one active key in the secret
API carrying the agent-scoped proxy path commits 53687d994 and d3074c1c8, live since API c8f10f437 (2026-09-24)

The AI-provider secret used to be the one blocker, and the deploy was written to fail loudly without it on the reasoning that a bot receiving chat it can never answer is worse than a refused deploy. That was right about the failure and wrong about the remedy: the key was never minted, the deploy never happened, and chat kept being served by a workspace that dies every night. Vertex removes the trade (the credential cannot go missing) and the case the gate really guarded against, a credential that is present but refused, is now caught at runtime by the credential probe and reported as ai_provider_unauthorized with the fix in the remedy.

Since 2026-09-02 this worker has beat from a codespace, so /api/healthz reports the okx_chat_bot subsystem instead of unknown. That is a stopgap and says so on the wire: every beat carries host and hostDurable, and a beat whose host cannot survive on its own reads as degraded, never ok, with the deploy command as its hint. Calling a codespace green would rebuild, one level up, the false-green this worker exists to kill.

Re-shipping it (one command)

npm run okx:bot:deploy -- --dry-run   # the whole plan, writes and submits nothing
npm run okx:bot:deploy                # provision the gateway lane, build, deploy, verify

scripts/okx-bot-deploy.mjs checks gcloud auth; builds from a clean worktree of HEAD (the image copies all of api/, so the shared working tree would ship other people's in-flight edits) and refuses while workers/okx-chat-bot/ has uncommitted changes; runs okx-bot-llm-gateway.mjs --apply and passes its metering agent to the build as _GATEWAY_AGENT; reports whether the live API already serves the agent-scoped proxy path; submits cloudbuild.yaml; and polls /api/healthz until the okx_chat_bot subsystem names the new revision as its host. Expect that last step to take a few minutes: the new instance waits for the single-writer lease while the old one drains.

The worker and the API can land in either order. The gateway lane is elected on a live probe every 15 minutes, so a worker deployed before the API simply stays on its current lane until the proxy answers, then moves by itself.

The seeded session only needs refreshing if the GCS snapshot is lost; the running service keeps it current:

npm run okx:bot:seed-state -- --apply   # from a host holding a live session, daemon stopped

The build pins three-ws-build@ and the service runs as three-ws@; the project's default compute service account was deleted, so both pins are required. The service must run --min-instances=1 --max-instances=1. That is not a capacity choice: the GCS snapshot has exactly one writer, and concurrent revisions would interleave snapshots and corrupt the identity.

Cloud Run's startup probe is wired to /healthz and deliberately not to /readyz, for the same reason liveness is always-200.

Seeding the session (why the first boot does not page)

gs://three-ws-okx-bot-state holds a snapshot taken 2026-09-04 from the codespace stopgap, verified by restoring it into a throwaway HOME and reading onchainos wallet status back as loggedIn: true, [email protected]. So the first Cloud Run revision restores an authenticated session and comes up online rather than alerting for an email OTP.

scripts/okx-bot-seed-state.mjs (npm run okx:bot:seed-state) writes that archive from any machine holding a live session. It builds the tar from the same exported STATE_ROOTS / STATE_EXCLUDES the worker uses, so the two cannot drift, and uploads it with gcloud (the deployed host uses ADC; a developer codespace has none). A bare run is plan-only; --apply uploads. It refuses to run while the daemon is up, because a live copy of the XMTP sqlite files can tear, and a torn identity costs a human OTP to recover.

The archive carries the wallet keyring and the XMTP identity. It belongs in that private bucket and nowhere else.

Skipping the seed is not a failure; it just leaves the OTP the first boot would otherwise have asked for.

Adding a lane

Never patch an AI key onto the service by hand. --set-secrets in the deploy step replaces the whole secret set, so a hand-added key survives exactly until the next deploy and then vanishes without a single error line. A lane belongs in cloudbuild.yaml, where the next deploy carries it too:

  • A first-party Anthropic key: create the secret, grant three-ws@ secretAccessor on it, and add ANTHROPIC_API_KEY=anthropic-api-key:latest to --set-secrets. It ranks below Vertex, so nothing else has to change; drop CLAUDE_CODE_USE_VERTEX only if you want the key to lead.
  • A gateway: set OKX_BOT_ANTHROPIC_BASE_URL and OKX_BOT_ANTHROPIC_MODEL in --set-env-vars and OKX_BOT_ANTHROPIC_AUTH_TOKEN=<secret>:latest in --set-secrets. The deploy already carries one: three.ws's own proxy, above.

Because the chain elects on a live probe rather than on presence, a lane with no funds behind it is inert (it probes unauthorized and is never elected) and costs nothing to leave configured. That is the point: funding it is then a billing action, not a deploy.

Shutdown order matters and is handled on SIGTERM: the daemon is stopped before the final snapshot, so the sqlite files are quiesced rather than copied mid-write. The periodic timer snapshot is a live copy and is best-effort by design, so an ungraceful kill loses minutes, not the identity.

Monitoring

The worker writes a bot_heartbeat row (keyed on worker = 'okx-chat-bot') carrying the current verdict, agent and client counts, provider, and restart count. api/_lib/ops/subsystem-health.js turns it into the okx_chat_bot subsystem, so the bot's reachability shows up next to every other platform dependency:

curl -s https://three.ws/api/healthz \
  | jq '.subsystems.subsystems[] | select(.name=="okx_chat_bot")'

A host that stops beating reads as down rather than silently vanishing, which is the whole point: a dead host cannot report that it is dead. The beat runs on its own timer (OKX_BOT_HEARTBEAT_MS) rather than at the end of a probe, because a probe is bounded at 15s + 30s + 90s of CLI calls and can outlast the two-minute freshness window /api/healthz judges the host by. One slow wallet call must not read as "the host is gone".

What a human has to do, read off the service itself:

URL=$(gcloud run services describe okx-chat-bot --region us-central1 \
  --project aerial-vehicle-466722-p5 --format='value(status.url)')
curl -s -H "Authorization: Bearer $(gcloud auth print-identity-token)" "$URL/readyz" | jq .remedy

Three signatures are classified in scripts/gcp-triage.mjs, so npm run triage:gcp explains them instead of filing them as unknown: okx-bot-session-logged-out (owner action, needs the OTP), okx-bot-provider-unauthorized (owner action, the AI credential is present and refused) and okx-bot-daemon-restart (self-healing unless the restart count climbs continuously).

A transition into a bad state fires an ops alert through sendOpsAlert, at most once per OKX_BOT_ALERT_REPEAT_MS while it persists, so one overnight expiry does not become a hundred notifications. Recovery fires a matching info alert.

Daemon stdout and stderr are forwarded into the worker's own log stream under a daemon prefix, since that output is the only window into XMTP delivery.

Related