Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(ops): record the prep:worktree and submit-gate steps in the deploy runbook
nirholas5b80cd1fc088
diff --git a/CLAUDE.md b/CLAUDE.mdindex 708844cad..4443cdf02 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -286,13 +286,13 @@ When reverting, do NOT use git's default `Revert "<original title>"` message: itRemember stop-and-ask gate 2: production deploys need owner approval unless the owner's current instruction is itself the approval. Full runbook incl. LB/DNS/TLS/env/rollback/recovery: `docs/ops/gcp-production.md`. GPU workers deploy from their own `workers/<name>/cloudbuild.yaml`.0. **Reclaim disk first: `npm run clean:worktrees` (add `--apply` to act).** Deploy worktrees are created by every deploy and removed by nobody. On 2026-08-04 ten had accumulated and filled the 126 GB disk to 100%, which fails in a way that reads like a different bug entirely: `git worktree add` dies mid-checkout with `No space left on device` and leaves a half-written tree behind. The script only reclaims a linked worktree that is detached, clean, and idle past `--min-age-hours` (default 2, so a concurrent agent's live build is never deleted); anything with uncommitted work is reported and kept.-1. **Prepare a CLEAN worktree.** `git worktree add --detach /workspaces/.deploy-wt HEAD`, then hardlink THREE artifacts with `cp -al` on the same filesystem (not just the first): `node_modules`, `chat/node_modules`, and `character-studio/build`, plus copy `.env`. Omitting either nested one kills the build minutes in: an OOM exit 144 on the avatar-studio rebuild, or `Cannot find package '@sveltejs/vite-plugin-svelte'` on the chat build. **Remove it when the deploy lands** (`git worktree remove --force <path>`); that is what step 0 exists to clean up after. See `docs/ops/gcp-production.md`.+1. **Prepare a CLEAN worktree: `npm run prep:worktree -- --apply`.** That is the whole step; it encodes everything below, and running it beats staging by hand. Plan first with a bare `npm run prep:worktree` (writes nothing), `--path <dir>` to stage your own tree instead of colliding with a concurrent agent on the default `/workspaces/.deploy-wt`, `--force` to replace an existing tree (refused if it holds uncommitted work). What it does: `git worktree add --detach <path> HEAD`, then hardlinks THREE artifacts with `cp -al` on the same filesystem (not just the first): `node_modules`, `chat/node_modules`, and `character-studio/build`, plus copies BOTH `.env` and `.env.local`. Omitting either nested one kills the build minutes in: an OOM exit 144 on the avatar-studio rebuild, or `Cannot find package '@sveltejs/vite-plugin-svelte'` on the chat build. A nested artifact that is missing from the source tree is BUILT first (`deps:chat`, `build:avatar-studio`) rather than skipped, which is the failure a hand-run `cp -al` produces on a machine that has never deployed. `.env.local` is the one that carries `DATABASE_URL`, and without it step 3's migration gate cannot read the schema state and refuses to submit. **Remove it when the deploy lands** (`git worktree remove --force <path>`); that is what step 0 exists to clean up after. See `docs/ops/gcp-production.md`.2. **Build: `npm run build:gcp`.** ORDER IS LOAD-BEARING and `build:gcp` already encodes it. Never hand-run the steps in a different order. The full chain, in order:`check:conflicts` → `check:browser-graph` → `ensure:avatar-studio` → `build:info:snapshot` → `build:chat` → `build` (the frontend `vite build`) → `build:lib:full` (the UMD lib) → `publish:lib` → `build:info` → `check:dist` → `check:pages`Why the order is load-bearing: `check:conflicts` runs first so an unresolved merge-conflict marker fails in seconds instead of after a full build. Then the frontend `vite build` runs with `emptyOutDir`, so it WIPES `dist/` and everything that writes into `dist/` must come after it. `publish:lib` mirrors the UMD lib into `dist/`, so it must follow `build:lib:full`, and `check:dist` fails without that bundle present. The first two steps are exactly why step 1 stages worktree artifacts: `ensure:avatar-studio` consumes `character-studio/build` and `build:chat` consumes `chat/node_modules`. Do NOT hand-run `build:vercel` as the frontend build: it builds the SDK/lib sub-artifacts, NOT the static HTML pages, so `dist/` ends up without `/`, `/create`, etc. and `check:dist` fails.-3. **Submit: `gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5`.** EVERY cloudbuild config must pin `serviceAccount: .../three-ws-build@...` (the default compute SA was deleted), and manual submits need `--substitutions=SHORT_SHA=manual$(date +%s)` when the config tags images with `$SHORT_SHA`.+3. **Submit: `npm run deploy:gcp:submit`.** That is `db:check` (exit 4 if any migration is pending) followed by the same `gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5`. Do NOT hand-run the bare `gcloud builds submit`: it is the only step that would have caught new code shipping over an old schema, and skipping it is how production ended up serving handlers that query columns their migration never applied (`column "blocks_behind" does not exist`, live on 2026-08-14). If the gate says migrations are pending, apply them first (`npm run db:status`, then `npm run db:migrate`); if it says `DATABASE_URL is not set`, step 1 did not copy `.env.local`. EVERY cloudbuild config must pin `serviceAccount: .../three-ws-build@...` (the default compute SA was deleted), and manual submits need `--substitutions=SHORT_SHA=manual$(date +%s)` when the config tags images with `$SHORT_SHA`.4. **Purge CDN: `npm run deploy:gcp:purge-cdn`.** The purge is synchronous on purpose. Do not re-add `--async`, or post-deploy checks read stale edge content and report phantom failures.5. **Verify the deploy landed.** `curl -s https://three.ws/api/version` returns the live commit SHA + Cloud Run revision, and `npm run smoke:prod` sweeps every page declared in `data/pages.json` against the live site (`deploy:gcp` runs it automatically after the purge).diff --git a/docs/ops/gcp-production.md b/docs/ops/gcp-production.mdindex bca005527..17bf24618 100644--- a/docs/ops/gcp-production.md+++ b/docs/ops/gcp-production.md@@ -156,13 +156,35 @@ npm run smoke:prod>> ```bash> npm run clean:worktrees -- --apply # reclaim old deploy trees before staging a new one+> npm run prep:worktree # plan only: shows what it would stage or build+> npm run prep:worktree -- --apply # stage /workspaces/.deploy-wt, artifacts and env included+> ```+>+> `prep:worktree` is the supported way to stage a deploy tree: it is the sequence+> below plus the checks a hand-run misses. Use `--path <dir>` to stage your own+> tree when a concurrent agent holds the default, and `--force` to replace an+> existing tree (refused when that tree holds uncommitted work). It BUILDS a+> nested artifact that is absent from the source tree instead of dying on the+> `cp -al`, which is what a machine that has never deployed hits first. It copies+> both env files rather than hardlinking them, and it rejects a `--path` on+> another filesystem up front, because `cp -al` cannot hardlink across a device+> boundary and leaves the tree half-staged when it tries. The equivalent by hand,+> for reference:+>+> ```bash> git worktree add --detach /workspaces/.deploy-wt HEAD> cp -al /workspaces/three.ws/node_modules /workspaces/.deploy-wt/node_modules> cp -al /workspaces/three.ws/chat/node_modules /workspaces/.deploy-wt/chat/node_modules> cp -al /workspaces/three.ws/character-studio/build /workspaces/.deploy-wt/character-studio/build> cp /workspaces/three.ws/.env /workspaces/.deploy-wt/.env+> cp /workspaces/three.ws/.env.local /workspaces/.deploy-wt/.env.local> ```>+> Both env files, not just `.env`. `DATABASE_URL` lives in `.env.local`, and it+> is what the deploy's migration gate reads; a worktree without it fails+> `db:check` with `DATABASE_URL is not set` and tempts whoever is deploying to+> skip the gate entirely.+>> **Remove the worktree once the deploy lands** (`git worktree remove --force> /workspaces/.deploy-wt`). Nobody did for three days in early August 2026 and> ten trees filled the 126 GB disk to 100%. The failure does not name its cause:@@ -223,7 +245,7 @@ automatically — the flow is:```bashnpm run db:status # dry run: list applied/pending against DATABASE_URLnpm run db:migrate # apply pending migrations (do this BEFORE deploy:gcp)-npm run db:check # what deploy:gcp runs: exits 4 if anything is pending+npm run db:check # what deploy:gcp and deploy:gcp:submit run: exits 4 if anything is pending```- `DATABASE_URL` comes from `.env.local` and must point at the production Neon@@ -232,6 +254,17 @@ npm run db:check # what deploy:gcp runs: exits 4 if anything is pendingthe sha256 drift and refuses (exit 3). Roll forward with a new file.- Apply migrations before deploying the code that needs them: migrations areadditive, so old code + new schema is safe; new code + old schema is not.+ When that order is inverted the symptom appears far from the cause: on+ 2026-08-14 `/api/healthz` reported the EVM agent index `unknown` with+ `column "blocks_behind" does not exist`, because the code that reads and+ writes those columns shipped while+ `20260813210000_erc8004_crawl_head_tracking.sql` was still pending. Submit+ through `npm run deploy:gcp:submit` (or `deploy:gcp`), never a bare+ `gcloud builds submit`, so `db:check` gets its say.+- `db:migrate` stops at the first migration that fails and reports every+ migration still queued behind it (exit 5). Those are unapplied: the schema is+ behind the code by exactly that list until the failing file is fixed with a+ NEW migration and the run drains the rest.- Lockfile unchanged → layer cache skips the workspace `npm ci`: **~3–5 min**.- Lockfile changed → full install: **~12 min**.diff --git a/package.json b/package.jsonindex 6a14cf0d6..42bf4b649 100644--- a/package.json+++ b/package.json@@ -268,6 +268,7 @@"sync:repos": "node scripts/sync-standalone-repos.mjs --execute","sync:repos:dry": "node scripts/sync-standalone-repos.mjs --dry-run","build:chat": "cd chat && node scripts/ensure-deps.mjs && npm run build && cd .. && mkdir -p dist/chat && cp -rf public/chat/. dist/chat/",+ "deps:chat": "cd chat && node scripts/ensure-deps.mjs","build:avatar-studio": "npm run build --prefix character-studio","build:fast": "npm run build","build:vercel": "node scripts/build-vercel.mjs",@@ -335,6 +336,7 @@"ensure:avatar-studio": "test -f character-studio/build/index.html || npm run build:avatar-studio","deploy:gcp": "npm run check:dist && npm run check:pages && npm run db:check && gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5 && npm run deploy:gcp:purge-cdn && npm run smoke:prod","deploy:gcp:full": "npm run build:gcp && npm run deploy:gcp",+ "deploy:gcp:submit": "npm run db:check && gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5","deploy:gcp:purge-cdn": "gcloud compute url-maps invalidate-cdn-cache three-ws-lb --path '/*' --project aerial-vehicle-466722-p5","postversion": "git push threews HEAD && git push threews --tags","validate:cards": "node scripts/validate-agent-cards.mjs",@@ -367,6 +369,7 @@"okx:gauntlet": "node scripts/okx-e2e-gauntlet.mjs","check:conflicts": "node scripts/check-merge-conflicts.mjs","clean:worktrees": "node scripts/clean-deploy-worktrees.mjs",+ "prep:worktree": "node scripts/prepare-deploy-worktree.mjs","check:browser-graph": "node scripts/check-browser-graph.mjs","prove:guards": "node scripts/prove-guards.mjs","prove:a2a-spend": "node scripts/a2a-spend-hardening-proof.mjs",