Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

8.3 KB
Raw Blame

Ops runbooks

Operator-only documentation for running three.ws in production. This directory (like docs/security/ and docs/internal/) is deliberately excluded from the public site build: the copy-static-docs plugin in vite.config.js and scripts/combine-docs.mjs both skip it, so nothing here ships to dist/ or docs/ALL.md. Keep it that way: these files name the GCP project, service accounts, and env-var gates.

Start with gcp-production.md; it is the complete operational record. The rest are focused runbooks:

Runbook What it covers
gcp-production.md The production platform end to end: Cloud Run services, LB/DNS/TLS, env, deploy, rollback, recovery.
hosting-and-credentials.md Hosting source of truth, Vercel GitHub App removal, GCP authentication, Secret Manager boundaries, R2 rotation, and exposed-credential response.
gcp-credits-plan.md Standing map of the ~$100k GCP credit spend: fleet, quota, pre-approved scaling, what to do next without asking.
gcp-credits.md The Vertex AI and GCP footprint runbook backing the credits plan.
gcp-model-workers.md Self-hosted GPU generation lanes on Cloud Run: workers, weights, deploys.
livepeer-federation.md Phase 4 compute federation: the Livepeer text-to-image adapter, the measured gateway state, and the recommendation on whether to expand.
gcp-logs.md Production log reading and automated triage tools for the Cloud Run fleet.
production-log-triage.md Known error/warning signature map: what each recurring log signature means and the fix.
forge-error-triage.md npm run forge:errors: which 3D generation failures actually recur, ranked by class and lane over a real window.
forge-funnel.md The Forge funnel board (GET /api/ops/forge-funnel, npm run forge:funnel): useful-output rate, first and second asset rates, retention, what each model was for, attempts per kept model by engine, and x402 revenue per kept model.
mcp-clients.md The MCP clients board (GET /api/ops/mcp-clients, /mcp-clients): sessions, tool calls and top tools per AI client (Claude, ChatGPT, Cursor, Grok Bot, ...), from the clientInfo each sends on initialize.
forge-desktop-release.md Cutting a three.ws Forge desktop release: version and notes, the Cloud Build Linux and Windows build, the Mac build, signing secrets, publish order, verification and rollback of the /releases/forge/ feed.
cron-auth.md The two locks on /api/cron/*: the handler gate, the edge gate, and the header trap that makes attaching Cloud Scheduler OIDC take the whole fleet down if done naively.
materialize-fulfillment.md Running physical print fulfillment: the operator console, its authorization doors, the adapter contract for wiring a partner, webhook idempotency, and the stall sweep.
payment-outcomes.md The payment-outcome board (GET /api/ops/payment-outcomes): verify-reject, settle-fail, replay, and sponsor-runway signals and how to read them in an incident.
home-operations.md The Home Assistant lane: why a per-tenant failure never alerts and a correlated one always does, the SLOs (confirmation integrity has none to spend), the three alerts, the correlation query to run before calling anything an incident, and the reliability half: the measured scale envelope, the five-rung backpressure ladder, the seven chaos scenarios and the Cloud Run sizing each number argued for.
security-exceptions.md Accepted npm audit findings with no upstream patch (bigint-buffer, lodash.set, node-forge): reachability analysis and why the root @solana/pay stays on 0.2.6.
agent-index.md The agent index crawls end to end: the cursor stall both chain legs share, the error-class table, the recovery script, and how the freshness sensor scores each leg.
solana-rpc-lanes.md The Solana RPC tier end to end: one-sweep diagnosis, per-lane method capability, what must rotate vs fail, config traps, recovery.
llm-lanes.md The LLM provider chain end to end: which rungs serve, why the paid ones are dead, how spend is metered, the one-command Claude rollout, and per-lane probes.
evolve.md npm run evolve:start: the unattended loop that runs the work queue, refills it, and sweeps production on the signed-in Claude subscription, with every owner gate denied.
page-audit.md scripts/page-audit.mjs: authed Chromium sweep of every public page, console-error gated.
x-story-findings-2026-09-30.md What filming every feature for the X queue found: features that could not be announced because they do not work, and the defects seen on the ones that do.
guard-wiring.md Every check:* / audit:* script classified: what it checks, its measured exit code and runtime, and whether it runs in gate, on the deploy path, in the pre-push hook, or only by hand.
swarm-100-audit.md Reconciles the prompts/swarm-100/ self-deleting work-order ledger against git history and measures every still-open order.
avatar-asset-orphans.md Avatar rows whose model or thumbnail is gone from the bucket: why they happened, how to find and repair them.
avatar-reconstruction-capacity.md The photo-to-avatar lane sized to the 10k avatars/day launch target: measured numbers, the autoscaler-blindness trap, and the applied config.
db-retention.md Keeping the Neon Postgres branch under its storage cap: what grows, what gets pruned.
redis.md Upstash Redis quota, burn rate, and which limiters are distributed vs local.
runtime-flags.md DB-backed feature flags that flip platform behavior without a redeploy.
x402-discovery-listings.md Getting paid endpoints listed and ranked on x402scan, the Bazaar, and other directories.
seo-keyword-plan.md Verified keyword landscape and content calendar (snapshot dated 2026-07-17).
search-indexing.md How pages reach a search index: the three serving paths, the crawler-page duplicate cluster and its fix, robots.txt vs noindex, SPA soft 404s, and how to check a URL as a crawler.
nvidia-inception-response-2026-09-14.md Private correspondence record and owner-action runbook for NVIDIA's catalog, co-marketing, technical-resource, and invite-only GTC Pavilion reply.
wallet-key-migration.md Incident record: stranded pool-agent wallets after the WALLET_ENCRYPTION_KEY migration. Also the standing runbook for where production's credentials live in Secret Manager, how to publish a new version, and the owner-gated master-wallet rotation.
stranded-wallets.md The standing owner decision on custodial wallets sealed by that rotation: measured list, why recovery is impossible, what crediting or writing off the customer balances costs, and the commands for each.
wallet-rescue.md Runbook for a leaked wallet key: dry-run and execute the atomic Jito rescue (scripts/rescue-wallet.mjs), plus how Jito tip accounts are sourced. Linked from the wallets-leak-scan alarm.
forge-multiview-migration-handoff.md Historical hand-off for the forge multi-view migrations (June 2026); superseded by gcp-production.md.
examples-repo-export.md How the public three-ws/examples satellite repo is assembled and published from this monorepo.

Related, also private: the security review records in ../security/ (SECURITY_AUDIT.md, SECURITY_REMEDIATION.md, review-2026-06-24.md), all point-in-time records preserved as written.