Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(deploy): create Scheduler jobs for newly declared crons on every deploy

nirholasfe2a8b24fb9f
diff --git a/CLAUDE.md b/CLAUDE.md
index f7b81a5d0..f59e0a4fb 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -299,7 +299,7 @@ Remember stop-and-ask gate 2: production deploys need owner approval unless the
Why the order is load-bearing: `check:conflicts` runs first so an unresolved merge-conflict marker fails in seconds instead of after a full build. `check:tdz-bootstrap` sits beside it for the same reason: it refuses a browser module whose top-level bootstrap call runs above the `let`s that call writes. JavaScriptCore checks an assignment target's temporal dead zone before evaluating the right-hand side and V8 does not, so that ordering ships green on Chrome and throws "Cannot access uninitialized variable." on every Safari, which is how `/avatars/:id` went dead on iOS and macOS while every desktop check passed. `build:lib:full` writes `dist-lib/agent-3d.js`, `build:avatar-sdk` copies that bundle into `avatar-sdk/dist/index.mjs`, and the frontend `vite build` resolves `avatar-sdk/src/agent.js` against it, so both must come BEFORE `build`; skipping them shipped whatever stale `avatar-sdk/dist` the build machine happened to hold, and in a fresh worktree failed the build outright with `Could not resolve "../dist/index.mjs"` (2026-08-17). The frontend `vite build` runs with `emptyOutDir`, so it WIPES `dist/` and everything that writes into `dist/` must come after it: that is why `publish:lib`, which mirrors the UMD lib into `dist/`, follows `build` rather than sitting next to `build:lib:full`, and `check:dist` fails without that bundle present. `publish:lib` writes only the moving channels (`latest`, `1`, `1.5`) from the fresh build; every released version directory is downloaded from the write-once archive `gs://three-ws-lib-releases` and checked against `data/agent-3d-releases.json`, so the build needs network access to storage.googleapis.com, and it fails if `package.json` names a version that was never cut. A versioned URL is immutable and SRI-pinned by embedders, so it must never change bytes: cut a new version with `npm run release:lib` after bumping `package.json`, never by rebuilding an old one (2026-10-01: rebuilding 1.5.2 on every deploy broke every published SRI pin). `check:dist` also fails when a documented SRI pin disagrees with the ledger; `npm run check:sri-pins -- --fix` rewrites stale pins. `ensure:avatar-studio` consumes `character-studio/build` and `build:chat` consumes `chat/node_modules`, which is exactly why step 1 stages worktree artifacts. `docs:freshness` measures documentation drift against git history and writes `public/docs-freshness.json` plus its summary, so it must run before `build` copies `public/` into `dist/`; running it here is what keeps `/docs/freshness` and every docs-page freshness badge describing the commit actually deployed instead of whenever someone last ran it by hand. Do NOT hand-run `build:vercel` as the frontend build: it builds the SDK/lib sub-artifacts, NOT the static HTML pages, so `dist/` ends up without `/`, `/create`, etc. and `check:dist` fails.
3. **Submit: `npm run deploy:gcp:submit`.** That is `db:check` (exit 4 if any migration is pending), then `check:gcloudignore`, then `audit:deploy` (no committed symlinks, no unsatisfied peer deps, no undeclared `api/` imports, decoder assets present, and an installed `node_modules` that matches `package-lock.json`), then `check:api-imports` (imports every module under `api/` and fails on any that throws while loading, since the server imports a handler on first hit and a load-time throw is a 500 on every request to that route rather than a build error), then the same `gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5`. `check:gcloudignore` simulates the upload and fails when a file the server imports at runtime would be excluded from the build context: an omitted `services/home-relay/src/token.js` shipped revision 00412 answering `/api/healthz` with `ERR_MODULE_NOT_FOUND` on every request (2026-09-04). The lockfile match exists because the image installs from the lockfile while every local test, and every deploy worktree that hardlinks this `node_modules`, runs whatever is on disk: a dependency bump nobody reinstalled shipped `@x402/extensions` 2.25.0 untested and every paid x402 route answered 500 on import (2026-09-17). If it fails, run `npm install`, then `npm test`, before deploying. Do NOT hand-run the bare `gcloud builds submit`: it is the only step that would have caught new code shipping over an old schema, and skipping it is how production ended up serving handlers that query columns their migration never applied (`column "blocks_behind" does not exist`, live on 2026-08-14). If the gate says migrations are pending, apply them first (`npm run db:status`, then `npm run db:migrate`); if it says `DATABASE_URL is not set`, step 1 did not copy `.env.local`. EVERY cloudbuild config must pin `serviceAccount: .../three-ws-build@...` (the default compute SA was deleted), and manual submits need `--substitutions=SHORT_SHA=manual$(date +%s)` when the config tags images with `$SHORT_SHA`.
-4. **Purge CDN: `npm run deploy:gcp:purge-cdn`.** The purge is synchronous on purpose. Do not re-add `--async`, or post-deploy checks read stale edge content and report phantom failures.
+4. **Sync new crons, then purge CDN: `npm run deploy:gcp:sync-crons`, then `npm run deploy:gcp:purge-cdn`.** The sync creates a Cloud Scheduler job for every cron in `vercel.json` that has none and touches no existing job; without it a new cron never fires (four sat dead for up to three weeks until 2026-10-08). The purge is synchronous on purpose. Do not re-add `--async`, or post-deploy checks read stale edge content and report phantom failures.
5. **Verify the deploy landed.** `curl -s https://three.ws/api/version` returns the live commit SHA + Cloud Run revision, and `npm run smoke:prod` sweeps every page declared in `data/pages.json` against the live site (`deploy:gcp` runs it automatically after the purge).
The whole build + submit + purge in one command: `npm run deploy:gcp:full`.
diff --git a/docs/ops/gcp-production.md b/docs/ops/gcp-production.md
index fe1fbef3c..1780e81e6 100644
--- a/docs/ops/gcp-production.md
+++ b/docs/ops/gcp-production.md
@@ -134,8 +134,11 @@ npm run build:gcp
# Build image on Cloud Build (32-vCPU + BuildKit layer cache) + push + deploy.
# Gated on check:dist, check:pages AND db:check so an incomplete dist/, an
-# unreachable page, or an out-of-date database can no longer ship. Ends with a
-# CDN purge and then smoke:prod against the live site.
+# unreachable page, or an out-of-date database can no longer ship. After the
+# submit it runs deploy:gcp:sync-crons, which creates a Cloud Scheduler job for
+# any cron declared in vercel.json that has none yet (existing jobs are never
+# touched), so a new cron starts firing with the code that serves it. Ends with
+# a CDN purge and then smoke:prod against the live site.
npm run deploy:gcp
# Or do build + submit + CDN purge in one (from a clean worktree):
diff --git a/package.json b/package.json
index 6e998e0ef..f8e4c366e 100644
--- a/package.json
+++ b/package.json
@@ -431,9 +431,10 @@
"build:info:snapshot": "node scripts/write-build-info.mjs --snapshot",
"build:gcp": "npm run check:conflicts && npm run check:browser-graph && npm run check:tdz-bootstrap && npm run ensure:avatar-studio && npm run build:info:snapshot && npm run docs:freshness && npm run build:lib:full && npm run build:avatar-sdk && npm run build:chat && npm run build && npm run publish:lib && npm run build:info && npm run check:dist && npm run check:pages",
"ensure:avatar-studio": "test -f character-studio/build/index.html || npm run build:avatar-studio",
- "deploy:gcp": "npm run check:dist && npm run check:pages && npm run deploy:gcp:submit && npm run deploy:gcp:purge-cdn && npm run smoke:prod",
+ "deploy:gcp": "npm run check:dist && npm run check:pages && npm run deploy:gcp:submit && npm run deploy:gcp:sync-crons && npm run deploy:gcp:purge-cdn && npm run smoke:prod",
"deploy:gcp:full": "npm run build:gcp && npm run deploy:gcp",
"deploy:gcp:submit": "npm run db:check && npm run check:gcloudignore && npm run audit:deploy && npm run check:api-imports && gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5",
+ "deploy:gcp:sync-crons": "node scripts/create-gcp-scheduler.mjs --missing-only",
"deploy:gcp:purge-cdn": "gcloud compute url-maps invalidate-cdn-cache three-ws-lb --path '/*' --project aerial-vehicle-466722-p5",
"postversion": "git push threews HEAD && git push threews --tags",
"validate:cards": "node scripts/validate-agent-cards.mjs",
diff --git a/scripts/create-gcp-scheduler.mjs b/scripts/create-gcp-scheduler.mjs
index 12986f528..01055f15b 100644
--- a/scripts/create-gcp-scheduler.mjs
+++ b/scripts/create-gcp-scheduler.mjs
@@ -38,6 +38,10 @@
//
// node scripts/create-gcp-scheduler.mjs --env-file <prod.env> --only garment-job-sweep
//
+// `--missing-only` creates just the declared crons that have no live job and
+// touches nothing else. `npm run deploy:gcp:sync-crons` runs it after every
+// deploy so a newly declared cron starts firing with the code that serves it.
+//
// Auth: each job sends `Authorization: Bearer $CRON_SECRET`, exactly what the
// api/cron/* handlers already validate. The secret is read from --env-file, then
// process.env, then the live three-ws-api Cloud Run service (production's
@@ -147,6 +151,20 @@ export function selectCrons(crons, argv) {
return picked;
}
+/**
+ * Narrow a sync to the declared crons that have no live job, for
+ * `--missing-only`. `deploy:gcp` runs that mode after every submit, because a
+ * cron added to vercel.json otherwise never fires until someone remembers to
+ * sync by hand: four handlers (x402-settlement-reconcile among them) shipped
+ * with no Scheduler job and sat dead for up to three weeks before 2026-10-08.
+ * Existing jobs are never touched in this mode, so it cannot rewrite a
+ * schedule, a credential, or a deliberate incident pause.
+ */
+export function missingCrons(crons, liveJobNames) {
+ const live = new Set([...liveJobNames].map((name) => String(name).split('/').pop()));
+ return crons.filter((c) => !live.has(jobId(c.path)));
+}
+
/** Read CRON_SECRET out of a `--env-file <path>` argument, if one was given. */
export function cronSecretFromArgs(argv, readFile = (p) => readFileSync(p, 'utf8')) {
const idx = argv.indexOf('--env-file');
@@ -317,6 +335,12 @@ async function main() {
if (selected.length !== crons.length) {
console.log(`--only: syncing ${selected.length} of ${crons.length} declared crons.`);
}
+ if (argv.includes('--missing-only')) {
+ const { stdout } = await gcloud(['list', '--format=value(name)']);
+ selected = missingCrons(selected, stdout.split('\n').filter(Boolean));
+ console.log(`--missing-only: ${selected.length} declared cron(s) have no live job.`);
+ if (!selected.length) return;
+ }
const queue = [...selected];
const results = [];
diff --git a/tests/cron-scheduler-sync.test.js b/tests/cron-scheduler-sync.test.js
index 178d08089..f10d9aa71 100644
--- a/tests/cron-scheduler-sync.test.js
+++ b/tests/cron-scheduler-sync.test.js
@@ -22,6 +22,7 @@ import {
cronSecretFromService,
selectCrons,
credentialFlags,
+ missingCrons,
} from '../scripts/create-gcp-scheduler.mjs';
describe('runStateAction', () => {
@@ -282,3 +283,23 @@ describe('credentialFlags', () => {
expect(deployed).toContain(email);
});
});
+
+describe('missingCrons', () => {
+ const crons = [
+ { path: '/api/cron/economy-tick', schedule: '* * * * *' },
+ { path: '/api/cron/duels-tick', schedule: '*/15 * * * *' },
+ ];
+
+ it('keeps only the declared crons with no live job', () => {
+ const live = ['projects/p/locations/us-central1/jobs/cron--api-cron-economy-tick'];
+ expect(missingCrons(crons, live)).toEqual([crons[1]]);
+ });
+
+ it('accepts bare job ids as well as full resource names', () => {
+ expect(missingCrons(crons, ['cron--api-cron-economy-tick', 'cron--api-cron-duels-tick'])).toEqual([]);
+ });
+
+ it('reports every cron missing when nothing is live', () => {
+ expect(missingCrons(crons, [])).toEqual(crons);
+ });
+});