Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(ops): escalate R2 row 36 from flap to 35+ hour sustained outage

nirholasee94b259dedb
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 7e2cff456..3632425a1 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -1253,7 +1253,7 @@ ERROR 502 POST /api/pump/launch-agent
`api/render/glb.js` and `api/render/avatar-clip.js` (502, same lib), and
GET `api/animations/[id].js` (`/api/animations/clips/<id>`, 502
`storage_error` from `getObjectBuffer`).
-- **What it means:** Cloudflare R2 flapping (healthz `object_storage`
+- **What it means:** Cloudflare R2 unreachable (healthz `object_storage`
`down`/`degraded`, tracked at
[production-100-OWNER-ACTIONS.md](../../prompts/finish/_context/production-100-OWNER-ACTIONS.md)
row 36) fans out into 5xx on this whole family at once with zero matching
@@ -1263,7 +1263,20 @@ ERROR 502 POST /api/pump/launch-agent
reproduced it live for `/api/render/animate` by curling the failing
avatar's own `model_url` (`https://pub-<account>.r2.dev/u/<owner>/<slug>.glb`)
directly and getting the same `403` Cloudflare answers for every other
- route in this family.
+ route in this family. 2026-10-13 (order 400): this was previously described
+ as a flap (many `RESOLVED` events minutes apart); re-measurement found a
+ 35-plus-hour unbroken `DOWN` streak since the last resolve at
+ 2026-10-11T06:55:05Z (60-plus hours and still ongoing by the time this note
+ was written), with the public `*.r2.dev` fallback domain also edge-blocked
+ (Cloudflare-branded 403) and `/cdn/*`'s own signed-read-then-public-fallback
+ therefore 502ing on both legs at once. Treat a live recurrence as a
+ sustained outage, not a flap, until row 36 says otherwise. Order 400 also
+ added one bounded retry (200-500ms backoff, inside a single
+ `gatherObjectStorageHealth()` call) so a single transient blip no longer
+ pages as `DOWN`, and routed most generated thumbnail/GLB URLs returned by
+ the API through `/cdn/` instead of a raw `*.r2.dev` URL, so a future
+ single-leg flap (only the signed path or only the public domain degraded)
+ gets `/cdn/`'s existing failover instead of no protection at all.
- **Resolve:** check healthz first:
`curl -s https://three.ws/api/healthz | jq '.subsystems.subsystems[] | select(.name=="object_storage")'`.
`down`/`degraded` confirms this family and the fix is the owner-side
diff --git a/prompts/finish/_context/production-100-OWNER-ACTIONS.md b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
index 34f7e52a4..962f88576 100644
--- a/prompts/finish/_context/production-100-OWNER-ACTIONS.md
+++ b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
@@ -46,7 +46,7 @@ on it; some resolve themselves as other work lands.
| 32 | **Decide retention for the six large tables the database retention cron never prunes** (added 2026-10-09 by the evolve loop triage sweep). The Neon footprint reached 11,606 MB, past the 8192 MB high-water mark since 2026-09-09, so five write-heavy crons (`launcher-tick`, `coin-intel-observe`, `smart-money-rollup`, `recompute-reputation`, `intel-learn`) skipped every tick for a month. The sweep raised `DB_RETENTION_HIGH_WATER_MB` to 16384 (config-only, pre-approved; real cap `neon.max_cluster_size` is 16TB), which buys about six weeks at the measured ~114 MB a day. The growth is in tables outside the retention set, and pruning them deletes history, so it is not an agent's call: `oracle_training_set` 1.8 GB (1.37M rows), `x402_self_facilitator_log` 1.4 GB (3.0M rows; the code already names it a product decision about lifetime revenue totals), `firewall_decisions` 1.2 GB, `usage_events` 0.7 GB, `club_tips` 0.6 GB (1.2M rows), `ops_alerts` 0.5 GB (693k one-off `503 settlement_unavailable` rows from July to October). Name a window per table (or keep forever and accept the storage bill); an agent then adds each to `TIME_SERIES_TABLES` in `api/cron/db-retention.js`. | decision | Keeping the write crons running past mid-November without raising the mark again | [docs/ops/db-retention.md](../../../docs/ops/db-retention.md) |
| 33 | **Create a production xAI API key so a Grok model's live MCP calls against three.ws can be proven** (added 2026-10-10 by the evolve loop, x-grok order 039, which already owns this exact gate as order 927: `prompts/finish/927-x-grok-46-owner-grok-api-key.md`). Step 0 re-derivation this session found no `GROK_API_KEY`/`XAI_API_KEY` anywhere: not in `process.env`, not in `.env`/`.env.local`, not on the `three-ws-api` Cloud Run service (`node scripts/read-service-env.mjs '^(GROK\|XAI)_API_KEY$' --names` returned nothing). `scripts/xai-mcp-e2e.mjs` (npm script `e2e:xai-mcp`) is built and verified end to end in dry-run mode: it matches xAI's current Responses API remote-MCP schema field for field and names this gate when no key is found. Create a key at console.x.ai, then either set it directly (`gcloud run services update three-ws-api --region us-central1 --update-env-vars GROK_API_KEY=<key>`) or hand it to order 927 to wire through Secret Manager; afterward `npm run e2e:xai-mcp` should exit 0 against production. | minutes, console only | Closes x-grok order 039's one remaining Definition of Done line, and order 927 | [_context/x-grok-PROGRESS.md](x-grok-PROGRESS.md), 2026-10-10 entry |
| 35 | **Apply the agent mention-policy migration, then ship it with the next deploy** (added 2026-10-11 by the evolve loop, x-grok order 061, which is otherwise complete and retired). `api/_lib/migrations/20261011220000_agent_mention_policy.sql` adds `mention_replies_enabled`, `mention_replies_dry_run`, `mention_replies_daily_cap` and `mention_replies_allowed_intents` to `agent_x_policies`, the four fields the new "Answer mentions" panel on `pages/agent-edit.html` (Social tab) reads and writes. `npm run db:status` on 2026-10-11 lists it as the only pending migration. The read path (`GET /api/x/agents`) already works without it, since the query is `select *` and `rowToPolicy()` defaults the missing columns; only the write path breaks: `saveAgentPolicy`'s `INSERT` names the four columns explicitly, so a save attempt against the unmigrated table answers 500 with Postgres `42703 undefined_column`, verified live in a local run of the full UI against the real dev database on 2026-10-11 (every other state, no-connection/missing-scope/connected/populated-decisions, was verified working by temporarily granting `X_OAUTH_CLIENT_ID`/`SECRET` in `.env.local` only, never in production). Apply it with `node scripts/apply-migrations.mjs --apply --file 20261011220000_agent_mention_policy.sql`, or `npm run db:status` then `npm run db:migrate` (nothing else is pending as of this writing). `npm run db:check` blocks the next deploy on it until then. Separately, and not blocking this row: production has never set `X_OAUTH_CLIENT_ID`/`X_OAUTH_CLIENT_SECRET` at all (`node scripts/read-service-env.mjs '^X_OAUTH_CLIENT'` finds nothing), so the whole per-agent "connect this agent's own X account" flow (distinct from the company bot account's `X_API_KEY`/`X_ACCESS_TOKEN` posting credentials, which are set) answers "X connections are not enabled on this deployment" for every agent today; that is a pre-existing gap this order did not create and its settings UI already degrades to that state correctly. | one command | The "Answer mentions" save button in the agent settings UI | [x-grok-PROGRESS.md](x-grok-PROGRESS.md), 2026-10-11 entry for order 061 |
-| 36 | **Check Cloudflare R2's own status/dashboard for the bucket behind `S3_BUCKET`: it flaps between healthy and unreachable without any corresponding change on our side** (added 2026-10-11 by the evolve scout lane). Queried `ops_alerts` directly (columns: `signature, title, detail, severity, count, environment, first_seen, last_seen`): since 2026-09-08 the `object_storage` sensor (`api/_lib/ops/object-storage-health.js`) has logged roughly 167 "DEGRADED: Object storage (Cloudflare R2)" events (`bucket reachable but slow, signed list ~1095ms` against a 1000ms threshold) and about 18 "DOWN" events, interleaved with many "RESOLVED" events minutes apart showing a healthy 150 to 300ms. Reproduced live with production's own `S3_ACCESS_KEY_ID`/`S3_SECRET_ACCESS_KEY`/`S3_BUCKET`/`S3_ENDPOINT` (pulled via `node scripts/read-service-env.mjs` one variable at a time, used in-process, then deleted from disk immediately): `gatherObjectStorageHealth()` returned `{"status":"down","detail":"bucket unreachable: Please enable R2 through the Cloudflare Dashboard.","metrics":{"latencyMs":135,"credentialFault":false}}`, a 135ms response, i.e. the bucket answered fast but Cloudflare's own API told us R2 was not enabled, which cannot be true most of the time this account is clearly serving reads. Ruled out as a code cause: the credential is not rejected (`credentialFault:false`, no `SignatureDoesNotMatch`), and the AWS SDK's Node HTTP handler already defaults to keep-alive on (`keepAlive:true, maxSockets:50` in `@smithy/node-http-handler`), so this is not the classic missing-keep-alive latency bug. The probe itself is a single signed `ListObjectsV2Command` with no retry (`api/_lib/ops/object-storage-health.js:76-92`), so a genuinely transient one-off Cloudflare hiccup is indistinguishable from a sustained outage in this sensor today, which may be inflating the DOWN/DEGRADED counts above a one-retry version would show. Two independent things to do, neither owned by this reconciliation order: (a) check Cloudflare's own R2 dashboard/status page for the account behind `S3_BUCKET` for incidents correlated with the alert timestamps above (owner/dashboard access this agent does not have); (b) if the pattern looks like a true transient flap rather than sustained Cloudflare-side degradation, a future order could add one immediate retry before `gatherObjectStorageHealth()` classifies `down`/`degraded`, with a test proving a single synthetic failure followed by a synthetic success no longer pages. Not attempted here because the current evidence does not confidently distinguish "add a retry" from "mute a real recurring problem", and CLAUDE.md is explicit that a check does not get weakened without a dated, measured reason; this row exists so neither path gets silently skipped. 2026-10-13 (evolve scout lane, re-measured before considering it as a work order): queried `ops_alerts` again, grouped by failure text. The `"bucket unreachable: Please enable R2 through the Cloudflare Dashboard"` DOWN signature (the one live healthz reports right now) spans `first_seen` 2026-09-08T19:35Z to `last_seen` 2026-10-13T19:10Z (this instant), 58 alert rows, i.e. it has been recurring continuously for over a month and is still firing as this note is written: a sustained, ongoing condition, not a one-off transient blip. The retry idea in option (b) above does not apply to this dominant failure mode: retrying a single probe cannot paper over something that has been reproducing for 30-plus days, and doing so would be exactly the "mute a real recurring problem" case CLAUDE.md warns against, so no work order was written for it this pass. The narrower "DEGRADED: bucket reachable but slow" variant (option (b)'s original target) is a separate, lower-severity code path from this DOWN/unreachable one and was not re-measured this pass; a future pass that wants to pursue the retry idea should scope it to that DEGRADED signature specifically and re-confirm its flap pattern first, never the DOWN/unreachable signature above. | Cloudflare dashboard check (owner): this is the one for the account behind `S3_BUCKET`. The retry idea is ruled out for the dominant failure mode; do not file it as a work order without re-scoping to the DEGRADED-only signature | Every avatar, thumbnail and generated GLB reads through this bucket; `/cdn/*` and the forge both degrade when it flaps | `ops_alerts` table, queried 2026-10-11 and re-queried 2026-10-13 |
+| 36 | **Check Cloudflare R2's own status/dashboard for the bucket behind `S3_BUCKET`: it flaps between healthy and unreachable without any corresponding change on our side** (added 2026-10-11 by the evolve scout lane). Queried `ops_alerts` directly (columns: `signature, title, detail, severity, count, environment, first_seen, last_seen`): since 2026-09-08 the `object_storage` sensor (`api/_lib/ops/object-storage-health.js`) has logged roughly 167 "DEGRADED: Object storage (Cloudflare R2)" events (`bucket reachable but slow, signed list ~1095ms` against a 1000ms threshold) and about 18 "DOWN" events, interleaved with many "RESOLVED" events minutes apart showing a healthy 150 to 300ms. Reproduced live with production's own `S3_ACCESS_KEY_ID`/`S3_SECRET_ACCESS_KEY`/`S3_BUCKET`/`S3_ENDPOINT` (pulled via `node scripts/read-service-env.mjs` one variable at a time, used in-process, then deleted from disk immediately): `gatherObjectStorageHealth()` returned `{"status":"down","detail":"bucket unreachable: Please enable R2 through the Cloudflare Dashboard.","metrics":{"latencyMs":135,"credentialFault":false}}`, a 135ms response, i.e. the bucket answered fast but Cloudflare's own API told us R2 was not enabled, which cannot be true most of the time this account is clearly serving reads. Ruled out as a code cause: the credential is not rejected (`credentialFault:false`, no `SignatureDoesNotMatch`), and the AWS SDK's Node HTTP handler already defaults to keep-alive on (`keepAlive:true, maxSockets:50` in `@smithy/node-http-handler`), so this is not the classic missing-keep-alive latency bug. The probe itself is a single signed `ListObjectsV2Command` with no retry (`api/_lib/ops/object-storage-health.js:76-92`), so a genuinely transient one-off Cloudflare hiccup is indistinguishable from a sustained outage in this sensor today, which may be inflating the DOWN/DEGRADED counts above a one-retry version would show. Two independent things to do, neither owned by this reconciliation order: (a) check Cloudflare's own R2 dashboard/status page for the account behind `S3_BUCKET` for incidents correlated with the alert timestamps above (owner/dashboard access this agent does not have); (b) if the pattern looks like a true transient flap rather than sustained Cloudflare-side degradation, a future order could add one immediate retry before `gatherObjectStorageHealth()` classifies `down`/`degraded`, with a test proving a single synthetic failure followed by a synthetic success no longer pages. Not attempted here because the current evidence does not confidently distinguish "add a retry" from "mute a real recurring problem", and CLAUDE.md is explicit that a check does not get weakened without a dated, measured reason; this row exists so neither path gets silently skipped. 2026-10-13 (evolve scout lane, re-measured before considering it as a work order): queried `ops_alerts` again, grouped by failure text. The `"bucket unreachable: Please enable R2 through the Cloudflare Dashboard"` DOWN signature (the one live healthz reports right now) spans `first_seen` 2026-09-08T19:35Z to `last_seen` 2026-10-13T19:10Z (this instant), 58 alert rows, i.e. it has been recurring continuously for over a month and is still firing as this note is written: a sustained, ongoing condition, not a one-off transient blip. The retry idea in option (b) above does not apply to this dominant failure mode: retrying a single probe cannot paper over something that has been reproducing for 30-plus days, and doing so would be exactly the "mute a real recurring problem" case CLAUDE.md warns against, so no work order was written for it this pass. The narrower "DEGRADED: bucket reachable but slow" variant (option (b)'s original target) is a separate, lower-severity code path from this DOWN/unreachable one and was not re-measured this pass; a future pass that wants to pursue the retry idea should scope it to that DEGRADED signature specifically and re-confirm its flap pattern first, never the DOWN/unreachable signature above. 2026-10-13 (order 400, evolve loop, severity escalation): re-measured again, 18:10-18:30 UTC. The "flaps between healthy and unreachable" framing at the top of this row is now stale and understates the severity: the last `RESOLVED: Object storage (Cloudflare R2)` row is 2026-10-11T06:55:05Z, and every hourly sensor tick since then has been `DOWN` with zero resolves, a 35-plus-hour unbroken streak at measurement time (now 60-plus hours as this note is written, still ongoing). Two things that were working as designed have also failed: the public `*.r2.dev` domain itself now answers a Cloudflare-branded 403 (an edge-level block on the whole R2 product for this account, not a bucket-level 404/403), and `/cdn/<key>` (`api/cdn-object.js`), whose job is to fail over from the signed read to that same public domain, now 502s because both of its legs are down simultaneously. The "reads fail over to the public bucket domain meanwhile" hint text in `api/_lib/ops/object-storage-health.js` is temporarily false while this persists. Enabling R2 in the Cloudflare dashboard for the account behind `S3_BUCKET` is now urgent, not routine; this row's classification stays owner/Cloudflare-dashboard-only, unchanged. Separately, order 400 implemented the retry this row had deferred, but scoped narrowly: one extra attempt after a 200-500ms backoff, inside a single `gatherObjectStorageHealth()` invocation (`api/_lib/ops/object-storage-health.js`, tests in `tests/object-storage-health.test.js`). This does not contradict the 2026-10-13 re-measurement above that ruled out retry for the dominant DOWN signature: one retry a few hundred milliseconds later mathematically cannot turn 60-plus hours of genuine downtime into `ok`, so it cannot mute this sustained outage. It only absorbs a single transient network blip inside one invocation, which is the DEGRADED-flap case this row originally proposed it for, scoped exactly as the warning above required. | Cloudflare dashboard check (owner): this is the one for the account behind `S3_BUCKET`. The retry idea is ruled out for the dominant failure mode; do not file it as a work order without re-scoping to the DEGRADED-only signature | Every avatar, thumbnail and generated GLB reads through this bucket; `/cdn/*` and the forge both degrade when it flaps | `ops_alerts` table, queried 2026-10-11 and re-queried 2026-10-13 |
| 37 | **Ship the cron time-budget fix with the next deploy** (added 2026-10-11 by the evolve loop, order 403, otherwise complete and retired). `wrapCron()` in `api/_lib/http.js` now races every cron handler against a real time budget (`maxDurationMs`, wired from each handler's own `export const maxDuration * 1000`, defaulting to 120000ms) and answers a clean `503 { reason: 'cron_timeout' }` instead of hanging until Cloud Run's blind 900s ingress kill, which is what `/api/cron/x402-ring-leak-scan` was hitting repeatedly before order 400 landed its own fix. Verified locally: a real `http.createServer` hosting `wrapCron` with an 800ms budget and a never-resolving handler answered `503 cron_timeout` at 0.809s (not 900s, not instantly); the new `tests/http-cron-timeout.test.js` (4 tests) and the added `wrapCron` case in `tests/http-db-unavailable-boundary.test.js` pass 10/10, and the timeout test was proven red (indefinite hang) against the pre-fix code via a path-scoped `git stash` before the fix landed. No migration, no env var, nothing else needed; this is a pure code fix sitting on `evolve` waiting for row 1's routine deploy cadence to carry it to `main` and production. Ship with `npm run deploy:gcp:full` (row 1 already tracks the general "approve the next deploy" gate; this row exists only so the new behavior is named for verification). Post-deploy verification: re-run the step 0 query from `prompts/finish/403-auto-cron-timeout-enforcement.md` (`gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" textPayload:"maximum request timeout"' --freshness=24h --project aerial-vehicle-466722-p5`) after the deploy window; a healthy rollout shows that query going quiet (or near-quiet) while `textPayload:"exceeded its"` picks up any genuine stuck cron with its name attached, cross-referenced against order 400's `x402-ring-leak-scan` fix so a recurrence is distinguished from a different handler entirely. | one deploy (already covered by row 1's cadence) | Clean, diagnosable cron timeouts platform-wide instead of a blank "maximum request timeout" | `docs/ops/production-log-triage.md`, "HTTP timeout with no cron context" section |
| 38 | **Deploy `model-trellis2` (the TRELLIS.2 GPU worker), then point `MODEL_TRELLIS2_URL` at it** (re-measured 2026-10-11 by the evolve loop, best3d order 067, built and tested on 2026-10-09, not yet deployed). `workers/model-trellis2/` is complete (Dockerfile, `cloudbuild.yaml` pinned to `three-ws-build@`, 64 of 65 local tests passing, 1 skipped), the Forge lane registry already makes `trellis2` the default image-to-3D backend with the old chain (`trellis_selfhost`, `hunyuan3d`, `trellis`, `modly`, `triposg`) as failover, and the territory router already keeps the EU, the UK and South Korea off every Tencent lane regardless. Confirmed live today: `gcloud run services describe model-trellis2 --region us-central1` answers "Cannot find service", and `three-ws-api`'s env has no `MODEL_TRELLIS2_URL`, so the registry's default lane is currently unreachable and every image-to-3D request falls through to the next lane in the chain, same as the 2026-10-09 measurement. This is a production deploy (new Cloud Run GPU service, gate 2), so no agent runs it. Commands, in order: `gcloud builds submit --config workers/model-trellis2/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5 .` from the repo root, then `gcloud run services update three-ws-api --region us-central1 --update-env-vars=MODEL_TRELLIS2_URL=<service-url>` (never `--set-env-vars`, which replaces the whole set). After that, run one real image-to-3D generation and confirm the `forge_creations` row for it shows `backend: 'trellis2'`, and validate the returned GLB with `packages/glb-tools`; both are the two remaining Definition of done lines in the order file. | one build + one env update + one test generation | Closes best3d order 067; makes TRELLIS.2 (MIT, EU/UK/KR-safe) the live default instead of the untested fallback chain | [067-best3d-01-trellis2-default-lane.md](../067-best3d-01-trellis2-default-lane.md), [best3d-PROGRESS.md](best3d-PROGRESS.md) |
| 39 | **Deploy `model-pixal3d` (the Pixal3D fidelity-lane GPU worker), then point `MODEL_PIXAL3D_URL` at it** (added 2026-10-11 by the evolve loop, best3d order 068, built and tested same day). `workers/model-pixal3d/` is complete (Dockerfile, `cloudbuild.yaml` pinned to `three-ws-build@`, 81 of 82 local tests passing, 1 skipped), `api/forge.js`/`api/_lib/forge-tiers.js`/`api/_providers/gcp.js` already register and wire the `pixal3d` backend (1-4 photos, `view_labels`, `fill_other_sides`, opt-in only: never a named default or a failover target), and `src/forge.js`/`pages/forge.html` already ship the "Match my photo" engine button and panel. Confirmed live today: `gcloud run services describe model-pixal3d --region us-central1` answers "Cannot find service", and `three-ws-api`'s env has no `MODEL_PIXAL3D_URL`, so the lane reports `backend_unconfigured` today and the engine button does not render in the catalog (verified against a synthetic catalog entry in a local browser session, since the real worker isn't deployed yet). This is a production deploy (new Cloud Run GPU service, gate 2), so no agent runs it. Commands, in order (from `workers/model-pixal3d/README.md`): `workers/model-pixal3d/stage_weights.sh` (one time, idempotent weight mirror), then `gcloud builds submit --config workers/model-pixal3d/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5 --substitutions=SHORT_SHA=manual$(date +%s) .` from the repo root, then `gcloud run services update three-ws-api --region us-central1 --update-env-vars MODEL_PIXAL3D_URL=$(gcloud run services describe model-pixal3d --region us-central1 --format='value(status.url)')` (never `--set-env-vars`, which replaces the whole set). After that, run one real 3-image generation with `backend: "pixal3d"` and confirm the `forge_creations` row records all three input URLs and the returned GLB is valid; that is the order's last unverified Definition of done line. | one weight stage + one build + one env update + one test generation | Closes best3d order 068; makes "Match my photo" (pixel-aligned fidelity to the caller's own photos, MIT + Apache-2.0) live in the Forge engine grid | [068-best3d-02-pixal3d-multiview.md](../068-best3d-02-pixal3d-multiview.md), [best3d-PROGRESS.md](best3d-PROGRESS.md) |