Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(db-retention): record the 16384 MB high-water mark and the month the write crons sat skipped, and ask the owner for table retention

nirholascff1f3f8afe8
diff --git a/docs/ops/db-retention.md b/docs/ops/db-retention.md
index 7ad518d06..3f5316359 100644
--- a/docs/ops/db-retention.md
+++ b/docs/ops/db-retention.md
@@ -139,9 +139,22 @@ while still leaving the valve as a genuine runaway backstop.
```sh
gcloud run services update three-ws-api --region us-central1 \
- --project aerial-vehicle-466722-p5 --update-env-vars DB_RETENTION_HIGH_WATER_MB=8192
+ --project aerial-vehicle-466722-p5 --update-env-vars DB_RETENTION_HIGH_WATER_MB=16384
```
+**Raised to `16384` on 2026-10-09.** The footprint crossed 8192 MB on 2026-09-09 and
+reached 11,606 MB a month later (about 114 MB a day), with `neon.max_cluster_size`
+still `16TB`. For that whole month `launcher-tick`, `coin-intel-observe`,
+`smart-money-rollup`, `recompute-reputation` and `intel-learn` preflight-skipped on
+every tick, which is the "too low" failure above: the `db retention pressure valve
+engaged` ops alert fired about 2,200 times a day, nobody acted on it, and nothing was
+ingested. The growth is not the pump.fun firehose the valve sheds: the largest tables
+are outside the retention set (`oracle_training_set` 1.8 GB, `x402_self_facilitator_log`
+1.4 GB, `firewall_decisions` 1.2 GB, `usage_events` 0.7 GB, `club_tips` 0.6 GB,
+`ops_alerts` 0.5 GB). At the current rate 16384 is about six weeks of headroom, so
+re-measure before it runs out and decide retention for those tables rather than
+raising the mark again.
+
## Upgrade trigger
The valve keeps the branch **healthy** on the free tier, but at the cost of a
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index a94c18148..a5909e4ea 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -783,7 +783,7 @@ Container failed to become healthy. Startup probes timed out after 4m (1 attempt
npm run logs -- --all --grep "project size limit" --since 7d # real 53100s, if any
```
- If `max_cluster_size` is far above the mark and there are no `project size limit` errors, the skips are **self-inflicted** and the fix is to raise the mark (config-only, pre-approved). On 2026-07-29 that read 16TB against a ~2.5 GB footprint while the mark sat at 3072, so the mark was raised to `8192`. See [db-retention.md](db-retention.md#sizing-the-high-water-mark-read-before-changing-it) for the sizing rule.
+ If `max_cluster_size` is far above the mark and there are no `project size limit` errors, the skips are **self-inflicted** and the fix is to raise the mark (config-only, pre-approved). On 2026-07-29 that read 16TB against a ~2.5 GB footprint while the mark sat at 3072, so the mark was raised to `8192`; on 2026-10-09 the footprint had reached 11.6 GB and it was raised again to `16384`. See [db-retention.md](db-retention.md#sizing-the-high-water-mark-read-before-changing-it) for the sizing rule.
- **This gate has starved the money path.** On 2026-07-28 at 18:42 the branch crossed a mark set equal to the assumed cap, and 56 crons skipped in one minute — including `economy-rebalance`, which refills the x402 fee wallet. The fee wallet then starved and every settle returned `fee_wallet_below_floor` for four hours. `economy-rebalance` is deliberately **not** gated any more; do not re-add `requireWriteCapacity` to a cron on the funding path.
- **Resolve (capacity):** the write crons stay skipped only while `pg_database_size > high-water`. In order: (a) raise `DB_RETENTION_HIGH_WATER_MB` if the real cap has headroom (the usual answer, see above); (b) raise the **Neon compute/storage plan** if it genuinely does not; (c) tighten `PUMP_INTEL_RETENTION_DAYS` / `PUMP_INTEL_MIN_RETENTION_DAYS` to shed the firehose faster. None is a code change — the gate and the valve are covered by [tests/cron-storage-backoff.test.js](../../tests/cron-storage-backoff.test.js).
diff --git a/prompts/finish/_context/production-100-OWNER-ACTIONS.md b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
index a6f0eb72a..dfc629808 100644
--- a/prompts/finish/_context/production-100-OWNER-ACTIONS.md
+++ b/prompts/finish/_context/production-100-OWNER-ACTIONS.md
@@ -9,7 +9,7 @@ it and, if that unblocks a work order, say which one in the owning pack's `produ
Rows carry the date they were added so staleness is visible. Re-verify a row before acting
on it; some resolve themselves as other work lands.
-## Open rows (2026-10-09: rows 1, 2 and 24 re-measured by the triage sweep and row 31 added for the GPU worker redeploy; row 30 added for the Grok Bot CLI setup rerun; 2026-10-08: row 26 added for the coin-gated MCP catalog regeneration; rows 1 and 2 re-measured by the triage sweep, row 24 added for the paid Solana RPC credits, row 25 added for the MCP connector OAuth deploy; 2026-09-30: row 23 added for the $THREE receiver wallets; 2026-09-29: row 19 re-measured after the agent-sniper deploy; 2026-09-09: row 19 added, and row 1 corrected because its outage premise no longer holds; rows 1, 2, 3, 9, 13, 15 and 16 re-measured 2026-09-04; the former row 18 was cleared that day and its number is not reused; the 2026-09-02 and 2026-08-09 passes are folded in below)
+## Open rows (2026-10-09: rows 1, 2 and 24 re-measured by the triage sweep, row 31 added for the GPU worker redeploy and row 32 for database table retention; row 30 added for the Grok Bot CLI setup rerun; 2026-10-08: row 26 added for the coin-gated MCP catalog regeneration; rows 1 and 2 re-measured by the triage sweep, row 24 added for the paid Solana RPC credits, row 25 added for the MCP connector OAuth deploy; 2026-09-30: row 23 added for the $THREE receiver wallets; 2026-09-29: row 19 re-measured after the agent-sniper deploy; 2026-09-09: row 19 added, and row 1 corrected because its outage premise no longer holds; rows 1, 2, 3, 9, 13, 15 and 16 re-measured 2026-09-04; the former row 18 was cleared that day and its number is not reused; the 2026-09-02 and 2026-08-09 passes are folded in below)
| # | Action | Size | Unblocks | Detail lives in |
|---|---|---|---|---|
@@ -43,6 +43,7 @@ on it; some resolve themselves as other work lands.
| 29 | **After the row 25 deploy, rerun the Grok Bot doc samples against production** (added 2026-10-09 by the evolve loop, x-grok order 035, which is otherwise complete and retired). `docs/grok-bot.md` (`ae20babcd`) documents `/api/mcp-grok`, install tokens, `get_job` and `idempotency_key`, none of which production (`fe2a8b24f`) serves yet, so its ten samples were run against a server built from this tree (all passed, `npm run probe:mcp-clients -- --only mcp-grok` included) and against production, where they answer 404. Once the deploy lands, run `set -a; . ./.env; set +a; node scripts/run-doc-samples.mjs docs/grok-bot.md` and expect `0 of 10 samples failed` with real JSON in every output (it makes one free `forge_free` generation). Nothing to approve beyond the deploy itself. | one command after the deploy | Proof that every Grok Bot doc sample runs on production as written | [docs/grok-bot.md](../../../docs/grok-bot.md#keeping-this-page-true) |
| 30 | **After the row 25 deploy, rerun `npx three-ws setup --client grok-bot` against production** (added 2026-10-09 by the evolve loop, x-grok order 037, which is otherwise complete and retired). The CLI (`cd9b7f5be`) reads the live directory, so on production today (`fe2a8b24f`) it connects the free studio `three-ws-studio` and its live check passes; `--auth install` answers that the deployment cannot mint install tokens yet, and `--auth key` without `--connector-key` answers that it cannot mint connector keys from the CLI yet, because the bearer `preset: "connector"` on `POST /api/api-keys` (`3350e513c`) is not deployed. All four modes were verified against a server built from this tree, including a real connector-key mint (scope capped to the connector set, then revoked). Once the deploy lands, run `node packages/three-ws-cli/src/cli.js setup --client grok-bot --yes`, then the same with `--auth install` and `--auth oauth`; each must name `three-ws-grok` and exit 0. Nothing to approve beyond the deploy itself. | three commands after the deploy | Proof the Grok Bot CLI setup reaches `/api/mcp-grok`, install tokens and connector keys on production | [docs/cli.md](../../../docs/cli.md#grok-bot) |
| 31 | **Redeploy the GPU workers that fetch caller images, so a Wikimedia reference photo stops failing** (added 2026-10-09 by the evolve loop triage sweep). `upload.wikimedia.org` answers httpx's default `python-httpx` agent with 403, so `model-triposg` failed every job whose reference photo was a Wikimedia link before inference (4 logged 403 failures in the 24 hours to 2026-10-09 03:00 UTC). `d886d09e1` sends a descriptive User-Agent from the shared `worker_security.py`, mirrored to all 17 vendored copies, but a worker only picks it up when its image is rebuilt. Start with the one that failed: `gcloud builds submit --config workers/model-triposg/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5 --substitutions=SHORT_SHA=manual$(date +%s) .` (the command in `workers/model-triposg/README.md`); the other image-fetching workers (`model-trellis`, `model-hunyuan3d`, `rig`, `texture`, `stylize`, `segment`, `remesh`) inherit it on their next routine deploy. Separately worth a look: `model-trellis` was killed by its own liveness probe (`/health` timing out during inference, then signal 9) three times in the 24 hours to 2026-10-09, which left 147 task polls answering 503; not root-caused this session. | one build | Image-to-3D jobs whose reference photo is a Wikimedia link | [workers/model-triposg/README.md](../../../workers/model-triposg/README.md) |
+| 32 | **Decide retention for the six large tables the database retention cron never prunes** (added 2026-10-09 by the evolve loop triage sweep). The Neon footprint reached 11,606 MB, past the 8192 MB high-water mark since 2026-09-09, so five write-heavy crons (`launcher-tick`, `coin-intel-observe`, `smart-money-rollup`, `recompute-reputation`, `intel-learn`) skipped every tick for a month. The sweep raised `DB_RETENTION_HIGH_WATER_MB` to 16384 (config-only, pre-approved; real cap `neon.max_cluster_size` is 16TB), which buys about six weeks at the measured ~114 MB a day. The growth is in tables outside the retention set, and pruning them deletes history, so it is not an agent's call: `oracle_training_set` 1.8 GB (1.37M rows), `x402_self_facilitator_log` 1.4 GB (3.0M rows; the code already names it a product decision about lifetime revenue totals), `firewall_decisions` 1.2 GB, `usage_events` 0.7 GB, `club_tips` 0.6 GB (1.2M rows), `ops_alerts` 0.5 GB (693k one-off `503 settlement_unavailable` rows from July to October). Name a window per table (or keep forever and accept the storage bill); an agent then adds each to `TIME_SERIES_TABLES` in `api/cron/db-retention.js`. | decision | Keeping the write crons running past mid-November without raising the mark again | [docs/ops/db-retention.md](../../../docs/ops/db-retention.md) |
Deleted 2026-09-01: the former row 5 (`WALLET_ENCRYPTION_KEY_PREVIOUS` policy), which
declared itself "delete on reading" and had outlived that by three weeks.