Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: refusals take doctor's rates from the last forecast instead of walking the week under the counter locks; the capacity notice no longer claims the call saved
garrytanb6bfb6520bd9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 064a4721a8..cd66e6c966 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,15 +12,15 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A write that saved now says so when it carries a low-capacity warning, and the warning no longer asks the user to raise a write cap by tens of gigabytes.** When a cumulative write cap (receipt bytes or request ids) was at 80% or forecast to run out within 30 days, every write receipt's `write_capacity_low` notice ended with `actor: host_admin` and `next: tell_user_to_run: gbrain config set persistence.limits.principal_terminal_bytes <value>`. An agent that read only the end of a successful `remember` took that for a refusal and told its user the memory was not saved. The value was also oversized: it projected the last 24 hours for a whole year, so one bulk sync day could recommend several times the current cap. The notice now opens with "This write was saved and writes still work", its fix is the read-only `gbrain doctor --only persistence_capacity --json`, and the one-more-year value uses the 7-day mean rate. Days-to-exhaustion warnings still react to a bulk day. Doctor's finding keeps the `gbrain config set` command for the brain host's operator.+**The low-capacity warning now says it is not a refusal, and it no longer asks the user to raise a write cap by tens of gigabytes.** When a cumulative write cap (receipt bytes or request ids) was at 80% or forecast to run out within 30 days, every write receipt's `write_capacity_low` notice ended with `actor: host_admin` and `next: tell_user_to_run: gbrain config set persistence.limits.principal_terminal_bytes <value>`. An agent that read only the end of a successful `remember` took that for a refusal and told its user the memory was not saved. The value was also oversized: it projected the last 24 hours for a whole year, so one bulk sync day could recommend several times the current cap. The notice now opens with "This notice is not a refusal: writes still work, and this call's own result says what it did", its fix is the read-only `gbrain doctor --only persistence_capacity --json` (the `get_brain_health` call over MCP), and the one-more-year value uses the 7-day mean rate. Days-to-exhaustion warnings still react to a bulk day. Doctor's finding keeps the `gbrain config set` command for the brain host's operator.### Itemized changes-- `src/core/persistence/capacity-snapshot.ts`: `write_capacity_low` says the write was saved and names doctor's read-only check (actor `agent`) as its fix, with no cap raise in the notice; over MCP that fix is the read-scope `get_brain_health` call, so a remote agent runs it instead of relaying a command to its user.+- `src/core/persistence/capacity-snapshot.ts`: `write_capacity_low` says the write was saved and names doctor's read-only check (actor `agent`) as its fix, with no cap raise in the notice. It never claims the call saved anything, because sync status, cycle output, dry runs and failed batches carry it too. Over MCP that fix is the read-scope `get_brain_health` call, so a remote agent runs it instead of relaying a command to its user.- `src/core/persistence/capacity-forecast.ts`: `recommended_limit` (doctor `persistence_capacity`, the notice snapshot, `gbrain sync status` and `/metrics`) covers one more year at the 7-day mean gross and net rates instead of the larger of the mean and the last 24 hours.-- `src/core/persistence/limits.ts`: `oneYearCapacity` (every `write_quota_exhausted`, `write_quota_insufficient` and background-pause refusal, and doctor's fallback) uses the scope's 7-day window over the forecast's window length while the scope admitted anything that week, and keeps the latest-1,000 sample for a scope idle all week. Like the forecast, the brain and the local CLI writer count the week's source lifecycle changes. A scope's rates are read at most once a minute per brain, because these refusals run under the counter locks and a paused background producer is refused on every retry. The compacted-size SQL moves to `receipt-bytes.ts` so both reads count the same net bytes.+- `src/core/persistence/limits.ts`: `oneYearCapacity` (every `write_quota_exhausted`, `write_quota_insufficient` and background-pause refusal, and doctor's fallback) uses the 7-day mean rates the last forecast recorded for the scope (`readCapacityForecast` records them per engine and brain for a minute; doctor, `/metrics` and the receipts' capacity snapshot read it), so the refusal names doctor's value, lifecycle changes included. These refusals run under the counter locks, so without a forecast from the last minute it samples the scope's latest 1,000 admissions as before and never walks the week. The sample's compacted size now uses the forecast's SQL (`receipt-bytes.ts`).- Docs: `docs/guides/write-refusals.md` (`write_capacity_low` row), `docs/guides/concurrent-writes.md` (the notice, the mean-rate value, and returning a raised `persistence.receipt_retention_days` to the default before raising a cap), `docs/guides/repair.md` (unset the 90-day retention once a failed-writes recovery is done), `docs/architecture/key-files/core-persistence-capacity.md`.-- Tests: `test/persistence-capacity-notices.test.ts` (the notice's fix and wording; a burst of 28 admissions in the last day over a 10-a-day week recommends the mean-rate value, ids exactly and receipt bytes under the mean-rate ceiling; after 1,000 admissions in one hour on a 100-a-week principal, a real `write_quota_exhausted` refusal names exactly the value doctor recommends, ids and receipt bytes, on PGLite and Postgres; the notice rendered over HTTP and stdio runs `get_brain_health`; a week of lifecycle changes moves the brain's value to doctor's; repeated refusals within a minute walk the window once per scope).+- Tests: `test/persistence-capacity-notices.test.ts` (the notice's fix and wording; a burst of 28 admissions in the last day over a 10-a-day week recommends the mean-rate value, ids exactly and receipt bytes under the mean-rate ceiling; after 1,000 admissions in one hour on a 100-a-week principal, a real `write_quota_exhausted` refusal names exactly the value doctor recommends, ids and receipt bytes, on PGLite and Postgres; the notice rendered over HTTP and stdio runs `get_brain_health`; a week of lifecycle changes moves the brain's value to doctor's; a refusal reads at most the latest 1,000 admissions, and none after a forecast).## [0.60.621.0] - 2026-11-09**A page a member-bound credential creates in the `default` source is now readable only by that member.** A token bound with `gbrain auth rescope --member <id>` and no `--sources` writes to `default`, the source every credential without a source grant reads, and so does a token granted `--sources default` and an OAuth client left at its registration default. Its `put_page` (and every page write through the same path) used to create an ordinary page there, which another member and an unbound token read through `get_page`, `search`, `query`, `recall` and `list_pages`; its `remember` facts were already kept on the member's own page. A page such a credential creates is now written with `readable_by: [member:<id>]`, in the database row on a repo-less brain and in the file on a repo brain; rewriting it without `readable_by` keeps the stored readers, and the member widens it by naming readers. An edit to an existing page that names no readers leaves it shared, and `POST /ingest`, which cannot keep a page to a member, refuses such a credential with `permission_denied`. Member-bound credentials whose write source is another source (the documented `--sources private-<member>,default` shape) and unbound credentials write exactly as before, apart from slugs a member now creates. Pages derived from such a page (extracted atoms, chronicle events, `derived_from` pages, and a synthesized concept that uses it) follow the rule for pages with readers and are readable by no remote caller, the writer included. A page that teammates should read, such as a shared `people/` or `companies/` entity page or a page in a folder a rule already scopes to a group, names its readers when it is created; otherwise the first member to create a slug keeps it, and every other remote credential writing that slug (another member, an unbound token, a shared agent) gets `page_not_found`. The operator widens such a page from the brain host by adding `readable_by` to it (the local CLI is not member-scoped). A member-bound `put_page` over a page deleted within the last 72 hours is a create, so it is kept to the member too. Pages written before this release keep whatever readers they have: `gbrain access who-can-read <slug>` shows who reads one.diff --git a/docs/architecture/key-files/core-persistence-capacity.md b/docs/architecture/key-files/core-persistence-capacity.mdindex b1e7effefe..c70fc6b057 100644--- a/docs/architecture/key-files/core-persistence-capacity.md+++ b/docs/architecture/key-files/core-persistence-capacity.md@@ -8,7 +8,7 @@ Cumulative write caps (permanent request ids, receipt bytes) and the paths that- `src/core/persistence/owner-verification.ts` — #6741 C: the claim's owner verification (`verifyOwnerRoot`: paths recorded, physical root, marker and reservation intact; a device-only change is `restampable`), shared by writer onboarding and background admissions; `assertBackgroundOwner` refuses a background admission `owner_unavailable` once per source per run (cache: admission-origin.ts `backgroundOwnerCheck`). Tests: `test/background-owner-verification.test.ts`.- `src/core/persistence/capacity-forecast.ts` — `readCapacityForecast`: per principal, one statement walks `persistence_requests_principal` newest first until `created_at` leaves the 7-day window and buckets by day; gross and net (compacted) receipt rates, `days_to_exhaustion` (at the larger of the 7-day mean and the last 24 hours), `basis` (`:legacy_under_reserved` while receipts admitted before the intent-sized reservation remain), `recommended_limit` (one more year at the 7-day mean, GST-2107, so one bulk day is not projected for a year). `light` skips the reclaim read and table size. Thresholds `persistence.capacity.warn_days` (30) and `fail_days` (7).- `src/core/engine-key.ts` + `src/core/engine-root.ts` — the key every per-engine cache uses. `engineKey(engine)` returns the engine itself through the `ENGINE_ROOT` symbol (each engine stores itself there; a page-transaction clone inherits it, forwarding proxies and a recovered degraded engine return it), and otherwise walks a page-transaction clone, a read-through view or a statement-deadline wrapper (`engineCacheKey`) to the engine behind it. A process with mounted brains keeps one entry per brain in the search telemetry writer (`getTelemetryWriter(engine)`), `search.track_retrieval`, minion admission policy, `mcp.strict_params` last-known-good, publication ceiling, `write_capacity_low` throttle, `search.hnsw_iterative_scan` and `search.vector_legacy_guard`, and a transaction or view shares its engine's entry. A one-brain process behaves as before. Test: `test/engine-keyed-caches.test.ts`.-- `src/core/persistence/capacity-snapshot.ts` — `cachedCapacitySnapshot`: one light forecast per engine per 60 s, shared by every caller; `capacityNotices` builds `write_capacity_low` (kind `info`, muteable; it says the write was saved and its fix is the read-only doctor check, GST-2107; hourly per engine and scope on write and sync receipts, `scopes` limits a remote caller to the brain and its own principal) and `write_capacity_backpressure` (once per pause start). Read by `mcp/dispatch.ts`, `cli/main.ts`, `commands/sync/{run,operator}.ts`, `cycle.ts` and the admin `/metrics` gauges in `serve-http-metrics.ts`. #6803: the cache keys on the real engine (`engineCacheKey`), the shared fetch is bounded at `CAPACITY_FETCH_BOUND_MS` (10 s) and a timed-out fetch is never cached; `capacitySnapshotWithin` (sync status, write and sync receipts: `CAPACITY_READ_WAIT_MS`, 1.5 s) returns `unknown` for that call alone, and `capacityNotices` on an unknown snapshot returns no notice and leaves throttle and pause state as it was. Test: `test/capacity-bounded-read-6803.test.ts`.+- `src/core/persistence/capacity-snapshot.ts` — `cachedCapacitySnapshot`: one light forecast per engine per 60 s, shared by every caller; `capacityNotices` builds `write_capacity_low` (kind `info`, muteable; it says it is not a refusal and its fix is the read-only doctor check, `get_brain_health` over MCP, GST-2107; hourly per engine and scope on write and sync receipts, `scopes` limits a remote caller to the brain and its own principal) and `write_capacity_backpressure` (once per pause start). Read by `mcp/dispatch.ts`, `cli/main.ts`, `commands/sync/{run,operator}.ts`, `cycle.ts` and the admin `/metrics` gauges in `serve-http-metrics.ts`. #6803: the cache keys on the real engine (`engineCacheKey`), the shared fetch is bounded at `CAPACITY_FETCH_BOUND_MS` (10 s) and a timed-out fetch is never cached; `capacitySnapshotWithin` (sync status, write and sync receipts: `CAPACITY_READ_WAIT_MS`, 1.5 s) returns `unknown` for that call alone, and `capacityNotices` on an unknown snapshot returns no notice and leaves throttle and pause state as it was. Test: `test/capacity-bounded-read-6803.test.ts`.- `src/core/persistence/compaction-pass.ts` — `runCompactionPass`, the consumer's idle and volume maintenance: `compactWriteReceiptsDetailed`, one sweep step, and the `persistence.maintenance.last_compaction` record (`at`, `host_id`, `compacted`, `freed_bytes`, `skipped_by_reason`). Each step resumes from the timer queue (`journal.ts` `yieldToEventLoop`), as do the consumer's maintenance chain, the per-table queue `VACUUM` and the effects drain passes (`NOOP_SETTLE_BATCH` 50), so on PGLite a request waits behind one step at most. `journal.ts` `compactReceipt` compacts one receipt in its own transaction (counters first, `lock_timeout` 2 s, `statement_timeout` 15 s) and skips open managed-sync runs (`OPEN_SYNC_CURSOR_EXCLUSION`) and the newest sync receipt per worktree (`newestSyncReceipt`).- `src/core/repair/receipt-compaction.ts` — `gbrain repair receipt-compaction`: explicit-only, preview-bound, destructive consent, `publication: 'projection'`; `--older-than` (the `pages purge-deleted` grammar) floored at the configured retention unless `--force`, committed only unless `--include-failures`; the approved set binds an absolute cutoff and the newest sequence. Test: `test/repair-receipt-compaction.test.ts`.- `src/core/persistence/refusal-breaker.ts` — `assertRefusalBreakerClosed`: `submitMaintenance` refuses before admission after `persistence.maintenance.refusal_breaker_threshold` (3; 0 off) same-code terminal refusals in 24 h for the same request `digest` (source, slug, intent with its revision and planner); `listSuppressedWrites` feeds doctor `maintenance_write_drift`. The ghost case (`canonical_file_missing`, tagged by `page-prepare.ts`, kept in `error_detail.origin`, returned by `liveFileDrift`) refuses before admission in `prepared-maintenance.ts`. Test: `test/maintenance-ghost-breaker-6696.test.ts`.diff --git a/docs/guides/concurrent-writes.md b/docs/guides/concurrent-writes.mdindex e370035b4a..b2ddc77620 100644--- a/docs/guides/concurrent-writes.md+++ b/docs/guides/concurrent-writes.md@@ -1014,9 +1014,9 @@ the recovery is done.**Warnings before the wall.** A cap at or past 80%, or one that runs outwithin `persistence.capacity.warn_days` (default 30), adds the-`[gbrain notice write_capacity_low]` notice (kind `info`). It rides a write-that succeeded, so it opens with "This write was saved" and its fix is the-read-only `gbrain doctor --only persistence_capacity --json`; the+`[gbrain notice write_capacity_low]` notice (kind `info`). It rides+receipts and status output whatever the call did, so it opens with "This+notice is not a refusal" and its fix is the read-only `gbrain doctor --only persistence_capacity --json`; the`gbrain config set` command stays in doctor's finding for the brain host. Write and sync receipts carry it at most once anhour per engine per scope (a remote client sees only the brain and its ownprincipal); `gbrain sync status`, doctor and autopilot cycle output carry itdiff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 305c2fbf2f..76aea045ed 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -125,7 +125,7 @@ of making a duplicate.| <a id="upload_too_many_parts"></a>`upload_too_many_parts` | `upload_too_many_parts` | An `upload_page_part` call would be part number 64 plus one per 64 KiB of the declared `total_bytes` (one per eighth of `part_max_frame_bytes` when that is under 512 KiB) or later without completing the content. Each part is a stored row, so a session of tiny parts is bounded. The part was not stored. | Send all remaining bytes in that part if they fit within `part_max_frame_bytes`; otherwise abort and begin again, cutting parts near `part_max_frame_bytes` (`get_page_upload` returns it). || <a id="part_splits_code_point"></a>`part_splits_code_point` | `part_splits_code_point` | An `upload_page_part` text starts or ends with half of a surrogate pair, which has no UTF-8 encoding on its own. The part was not stored. | Move the cut before the high surrogate so the pair travels in one part, then send the part again. || <a id="write_quota_insufficient"></a>`write_quota_insufficient` (exit 12) | `write_quota_insufficient` | A bulk run (a managed sync catch-up, a reclone, a repair) estimated from its manifest more new receipts (ids and 16 KiB reservations) than the capacity left before its producer's pause line, so it did not start. Other writes still work. | Run the `gbrain config set <key> <value>` the message prints on the brain host, then rerun the same command. Verify with `gbrain doctor --only persistence_capacity --json`. |-| <a id="write_capacity_low"></a>`write_capacity_low` (notice) | `write_capacity_low` | Not a refusal: a `[gbrain notice]` on write receipts, `sync` receipts, `gbrain sync status` and autopilot cycle output when a cumulative cap is at or past 80% or would run out within `persistence.capacity.warn_days` (30) days at its 7-day rate. Receipts carry it at most once an hour per engine and scope; `sync status` and doctor carry it every time. The write that carries it was saved; the notice says so first and its fix is the read-only `gbrain doctor --only persistence_capacity --json`, never a cap raise. The structured `capacity` object in `gbrain sync status` names the scope, resource, days left and the config command. | Nothing to redo. Read the forecast with `gbrain doctor --only persistence_capacity --json` and follow the [runbook](concurrent-writes.md#quota-near-exhaustion) on the brain host; a raised `persistence.receipt_retention_days` comes back to the default before any cap raise. |+| <a id="write_capacity_low"></a>`write_capacity_low` (notice) | `write_capacity_low` | Not a refusal: a `[gbrain notice]` on write receipts, `sync` receipts, `gbrain sync status` and autopilot cycle output when a cumulative cap is at or past 80% or would run out within `persistence.capacity.warn_days` (30) days at its 7-day rate. Receipts carry it at most once an hour per engine and scope; `sync status` and doctor carry it every time. The notice says first that it is not a refusal (it never claims the call saved anything) and its fix is the read-only `gbrain doctor --only persistence_capacity --json`, never a cap raise. The structured `capacity` object in `gbrain sync status` names the scope, resource, days left and the config command. | Nothing to redo. Read the forecast with `gbrain doctor --only persistence_capacity --json` and follow the [runbook](concurrent-writes.md#quota-near-exhaustion) on the brain host; a raised `persistence.receipt_retention_days` comes back to the default before any cap raise. || <a id="write_capacity_backpressure"></a>`write_capacity_backpressure` (exit 12) | `write_capacity_backpressure` | A background producer (managed maintenance, conversation facts, an autopilot cycle phase, an autopilot catch-up sync) stopped admitting because a cumulative cap of its writer or the brain is past `persistence.capacity.background_pause_pct` (default 0.8). Interactive writes (`remember`, `put_page`, `edit_page`, an operator's `gbrain sync`, an approved `gbrain repair`) keep the remainder. No receipt was written. `reason` names the resource. Doctor `persistence_capacity` names the paused producers. | Depends on `reason`. `lifetime_ids`: ids are never freed, so only the brain host's `gbrain config set <key> <value>` from the `fix` resumes background work; waiting does not. `terminal_bytes`: the same reversible raise resumes it at once, or it resumes by itself as retained receipts compact. Verify with `gbrain doctor --only persistence_capacity --json`. |### Other error codesdiff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.tsindex 44f6c51375..4760ba516b 100644--- a/src/core/behavior-change-notice.ts+++ b/src/core/behavior-change-notice.ts@@ -1490,7 +1490,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;{ since: '0.60.619.0', id: 'managed_commit_push_unpublished_gst1923', text: 'On a managed checkout, `gbrain git commit-push -m <msg> -- <paths>` now reports how many commits the upstream lacks as last fetched (`unpublished` in `--json`, `null` when there is no upstream) and says nothing was pushed, instead of answering `nothing to commit; <remote>/<branch> at -` and logging `[push] ok-already-on-remote` for a commit that already existed. It writes no push-log line on a managed checkout. It still never pushes there; `gbrain git commit-push --push-only --keep-checkout --path <checkout>` publishes.' },{ since: '0.60.620.0', id: 'refresh_publish_diverged_default_gbra386', text: 'A diverged managed checkout now publishes its local commits on `gbrain sources refresh` by default when the worktree\'s Git durability is on: `sources.refresh_publish_diverged` unset now means on there (it used to mean off), merged with the upstream in memory, never rebased or forced. The refresh publish first scans what the local commits add (messages, added lines, merge resolutions) and refuses the new `publish_secret_found` naming the file, line and rule, never the value, when one looks like a secret; `gbrain git commit-push --push-only --scan-secrets` runs the same gate. Unset with durability not on still refuses, naming `gbrain sources writer git-durability <source> --enable`; `false` turns it off.' },{ since: '0.60.621.0', id: 'ungranted_member_writes_gst2051', text: 'A token or OAuth client bound to a member (`gbrain auth rescope --member <id>`) whose write source is `default` (no `--sources`, `--sources default`, or an OAuth client at its registration default) writes where every credential without a source grant reads. A page it creates there is now written with `readable_by: [member:<id>]`, so other members and unbound tokens no longer read it; rewriting it without `readable_by` keeps those readers, naming readers widens it, an edit to an existing page with no readers leaves it shared, and `POST /ingest` refuses such a credential. Pages derived from such a page (atoms, chronicle events, `derived_from` pages, a synthesized concept using it) are readable by no remote caller, the writer included; a page teammates should read names its readers when created, because every other remote credential, unbound tokens included, gets page_not_found writing a slug a member created until the operator adds readable_by on the brain host. Credentials writing to another source and unbound credentials are unchanged, and pages written before the upgrade keep their readers (`gbrain access who-can-read <slug>`).' },- { since: 'TBD', id: 'write_capacity_low_saved_gst2107', text: 'The `write_capacity_low` notice a write receipt carries now opens with "This write was saved" and its fix is the read-only `gbrain doctor --only persistence_capacity --json` (actor agent) instead of a `gbrain config set` cap raise for the user, so an agent reading the end of a successful write no longer reports it as refused. The one-more-year value doctor, the notice snapshot and `/metrics` recommend now uses the 7-day mean rate, so one bulk day is no longer projected for a year; days-to-exhaustion still uses the larger of the mean and the last 24 hours. A `write_quota_exhausted`, `write_quota_insufficient` or background-pause refusal names the same value doctor does instead of sizing the year from the latest 1,000 admissions.' },+ { since: 'TBD', id: 'write_capacity_low_saved_gst2107', text: 'The `write_capacity_low` notice a write receipt carries now opens with "This notice is not a refusal" (never that the call saved anything: sync status, dry runs and failed batches carry it too) and its fix is the read-only `gbrain doctor --only persistence_capacity --json` (actor agent; `get_brain_health` over MCP) instead of a `gbrain config set` cap raise for the user, so an agent reading the end of a successful write no longer reports it as refused. The one-more-year value doctor, the notice snapshot and `/metrics` recommend now uses the 7-day mean rate, so one bulk day is no longer projected for a year; days-to-exhaustion still uses the larger of the mean and the last 24 hours. A `write_quota_exhausted`, `write_quota_insufficient` or background-pause refusal names the same value doctor does when a forecast ran in the last minute, and otherwise still samples the latest 1,000 admissions.' },];/** The newest disclosed change's release: the notice id moves only when a release adds rows. */diff --git a/src/core/persistence/capacity-forecast.ts b/src/core/persistence/capacity-forecast.tsindex 00ea17955f..036a715ca1 100644--- a/src/core/persistence/capacity-forecast.ts+++ b/src/core/persistence/capacity-forecast.ts@@ -22,7 +22,7 @@*/import type { SqlEngine } from './model.ts';import { DEFAULT_JOURNAL_LIMITS, type JournalLimits } from './model.ts';-import { DEFAULT_RECEIPT_RETENTION_DAYS, journalLimitKey, parseJournalConfigValue, readJournalLimits, RECEIPT_RETENTION_KEY, recommendedLimit } from './limits.ts';+import { DEFAULT_RECEIPT_RETENTION_DAYS, journalLimitKey, parseJournalConfigValue, readJournalLimits, RECEIPT_RETENTION_KEY, recommendedLimit, recordForecastRates } from './limits.ts';import { BACKGROUND_PAUSE_KEY, backgroundPauseShare } from './admission-origin.ts';import { COMPACTED_RECEIPT_BYTES_SQL } from './receipt-bytes.ts';@@ -154,8 +154,9 @@ export async function readCapacityForecast(engine: SqlEngine & { kind?: string }COALESCE(SUM(terminal_bytes) FILTER (WHERE created_at>now()-interval '1 day'),0)::text AS recent_gross,COALESCE(SUM(terminal_bytes) FILTER (WHERE created_at>now()-interval '1 day'),0)::text AS recent_netFROM persistence_topology_changes WHERE created_at>now()-($1::double precision*interval '1 day') GROUP BY 1,2`, [WINDOW_DAYS]);- const history = engine.executeRaw<{ age_days: number | null; compaction_age_days: number | null }>(`SELECT+ const history = engine.executeRaw<{ age_days: number | null; compaction_age_days: number | null; first_id: string | null }>(`SELECT(SELECT EXTRACT(EPOCH FROM now()-created_at)/86400 FROM persistence_requests ORDER BY sequence LIMIT 1)::float8 AS age_days,+ (SELECT id::text FROM persistence_requests ORDER BY sequence LIMIT 1) AS first_id,(SELECT EXTRACT(EPOCH FROM now()-min(completed_at))/86400 FROM persistence_requestsWHERE recovery IS NULL AND NOT compacted AND state IN ('committed','conflict','failed','cancelled'))::float8 AS compaction_age_days`);// The uncompacted terminal receipts (the compactable partial index): what compaction would free, and the legacy under-reserved ones.@@ -177,6 +178,7 @@ export async function readCapacityForecast(engine: SqlEngine & { kind?: string }const untilCompaction = Math.max(0, retention - Number(historyRow?.compaction_age_days ?? 0));const now = Date.now();const rows: CapacityForecastRow[] = [];+ const means: Parameters<typeof recordForecastRates>[3] = [];for (const counter of counters) {const scopeDays = counter.key === 'brain' ? days : days.filter(day => day.scope === counter.key);const byDay = new Map<string, Day>();@@ -200,6 +202,7 @@ export async function readCapacityForecast(engine: SqlEngine & { kind?: string }const maxDay = Math.max(0, ...[...byDay.values()].map(day => ids ? day.ids : day.gross));const mean = total(ids ? 'ids' : 'gross') / windowDays;const netMean = ids ? mean : total('net') / windowDays;+ means.push({ scope: counter.key, resource, perDay: mean, netPerDay: netMean });const spike = [...byDay.values()].find(day => mean > 0 && (ids ? day.ids : day.gross) > SPIKE_FACTOR * mean)?.day ?? null;const remaining = Math.max(0, limit - used);let daysLeft: number | null;@@ -222,6 +225,7 @@ export async function readCapacityForecast(engine: SqlEngine & { kind?: string }basis, recommended_limit: recommendedLimit(resource, used, limit, { perDay: mean, netPerDay: netMean }, retention), status });}}+ recordForecastRates(engine, historyRow?.first_id ?? null, retention, means);return { rows, thresholds: { warn_days: warnDays, fail_days: failDays, warn_pct: CAPACITY_WARN_PCT, fail_pct: CAPACITY_FAIL_PCT },retention_days: retention, table_bytes: Number.isFinite(tableBytes) ? tableBytes : null, window_days: Number(windowDays.toFixed(2)) };}diff --git a/src/core/persistence/capacity-snapshot.ts b/src/core/persistence/capacity-snapshot.tsindex 31776c377c..260eb09096 100644--- a/src/core/persistence/capacity-snapshot.ts+++ b/src/core/persistence/capacity-snapshot.ts@@ -104,15 +104,16 @@ const describe = (row: CapacityAttention) => `${row.scope} ${row.resource} ${rowconst VERIFY = CAPACITY_VERIFY;/**- * GST-2107: this notice rides a write that succeeded, so it says so first and its fix is the read-only forecast; the cap raise- * stays in doctor's finding for the brain host's operator, never in the tail of a write an agent may read as a refusal.+ * GST-2107: this notice rides receipts, sync status and cycle output whatever the call did, so it says first that it is not a+ * refusal and its fix is the read-only forecast; the cap raise stays in doctor's finding for the brain host's operator, never in+ * the tail of a write an agent may read as a refusal.*/function lowNotice(rows: CapacityAttention[]): Notice {return { code: 'write_capacity_low', kind: 'info',- why: `This write was saved and writes still work. Write capacity is running low: ${rows.map(describe).join('; ')}; admission refuses only at the cap. `+ why: `This notice is not a refusal: writes still work, and this call's own result says what it did. Write capacity is running low: ${rows.map(describe).join('; ')}; admission refuses only at the cap. `+ 'gbrain doctor --only persistence_capacity --json reads the forecast and the remedy for the brain host.',fix: { argv: [...VERIFY.argv], mcp: VERIFY.mcp, consent: [], actor: 'agent', requires_exclusive: false,- why: 'Reads which cap is running low, its forecast and the remedy, read-only; nothing needs to change for this write.', verify: VERIFY,+ why: 'Reads which cap is running low, its forecast and the remedy, read-only; nothing needs to change for this call.', verify: VERIFY,docs: 'docs/guides/concurrent-writes.md#quota-near-exhaustion' } };}diff --git a/src/core/persistence/limits.ts b/src/core/persistence/limits.tsindex a0c4ba906d..613e524a1d 100644--- a/src/core/persistence/limits.ts+++ b/src/core/persistence/limits.ts@@ -2,7 +2,7 @@ import { opError, type OperationError } from '../ops/contract.ts';import { PAGE_BODY_CONFIG_KEYS, pageRecoveryBytes, segmentedPageCeiling } from '../page-body/limits.ts';import { DEFAULT_JOURNAL_LIMITS, principalKey, type JournalLimits, type Principal, type SqlEngine } from './model.ts';import { engineKey } from '../engine-key.ts';-import { COMPACTED_HEADROOM_BYTES, COMPACTED_RECEIPT_BYTES_SQL, EFFECT_BYTES_SQL, RECEIPT_RESERVATION_FLOOR } from './receipt-bytes.ts';+import { COMPACTED_RECEIPT_BYTES_SQL, EFFECT_BYTES_SQL, RECEIPT_RESERVATION_FLOOR } from './receipt-bytes.ts';/** Fallback compacted receipt size when no receipt is sampled; a compacted put_page measures about 3.9 KB. */export const RETAINED_RECEIPT_BYTES = 4096;@@ -69,66 +69,45 @@ export function recommendedLimit(resource: CapacityResource, used: number, limitreturn resource === 'terminal_bytes' ? Math.ceil(value / 1024 ** 2) * 1024 ** 2 : value;}-/** Newest admissions a one-year read walks for the 7-day window (the walk reads the sequence index newest first). */-const WINDOW_WALK_ROWS = 200_000;+/** How long the 7-day mean rates a forecast read stay usable by an admission refusal. */+export const FORECAST_RATES_TTL_MS = 60_000;+const forecastRates = new WeakMap<object, { at: number; brain: string | null; retention: number; rates: Map<string, { perDay: number; netPerDay: number }> }>();/**- * A cumulative cap value that covers one more year, never less than double the current cap. GST-2107: while the scope- * admitted anything in the last 7 days it uses the window's mean rates, source lifecycle changes included, over the window- * `readCapacityForecast` uses (7 days, or the brain's age, at least a day), so an admission refusal names the same value as doctor and one bulk hour- * is not projected for a year. A scope idle for a week is sized from its latest 1,000 admissions, however old. Receipt- * bytes use measured reservations: what receipts reserve while the retention window holds them, and their compacted size- * after (`recommendedLimit`).+ * GST-2107: `readCapacityForecast` records each scope's 7-day mean rates here, per engine, for `oneYearCapacity`; `brain`+ * (the brain's first request) keeps a reused engine on a different brain from reading them.+ */+export function recordForecastRates(engine:SqlEngine,brain:string|null,retention:number,rates:Array<{scope:string;resource:CapacityResource;perDay:number;netPerDay:number}>):void {+ forecastRates.set(engineKey(engine),{at:Date.now(),brain,retention,rates:new Map(rates.map(r=>[`${r.scope}|${r.resource}`,{perDay:r.perDay,netPerDay:r.netPerDay}]))});+}++/**+ * A cumulative cap value that covers one more year, never less than double the current cap. Admission refusals call this+ * under the counter locks, so it never walks the week itself. GST-2107: when a forecast read in the last minute (doctor,+ * `/metrics`, or the capacity snapshot write and sync receipts read) recorded the scope's 7-day mean rates, it uses them,+ * so the refusal names doctor's value and one bulk hour is not projected for a year. Otherwise it samples the scope's+ * latest 1,000 admissions. Receipt bytes use measured reservations: what receipts reserve while the retention window+ * holds them, and their compacted size after (`recommendedLimit`).*/export async function oneYearCapacity(engine:SqlEngine,scope:string,setting:'LifetimeIds'|'TerminalBytes',used:number,limit:number):Promise<number> {+ const resource:CapacityResource=setting==='LifetimeIds' ? 'lifetime_ids' : 'terminal_bytes';+ const known=forecastRates.get(engineKey(engine));+ const [first]=known && Date.now()-known.at<FORECAST_RATES_TTL_MS+ ? await engine.executeRaw<{id:string}>('SELECT id::text AS id FROM persistence_requests ORDER BY sequence LIMIT 1') : [];+ const rate=known && first && first.id===known.brain ? known.rates.get(`${scope}|${resource}`) : undefined;+ if(rate && known) return recommendedLimit(resource,used,limit,rate,known.retention);const principal=/^principal:([^:]+):(.+)$/.exec(scope);- const where=principal ? 'WHERE principal_kind=$1 AND principal_id=$2' : '';- const params=principal ? [principal[1],principal[2]] : [];- // Refusals run under the counter locks, and a paused background producer is refused on every retry, so a scope's- // rates are read once a minute per brain (its oldest request names the brain) rather than walking the window each time.- const [oldest]=await engine.executeRaw<{id:string}>('SELECT id::text AS id FROM persistence_requests ORDER BY sequence LIMIT 1');- const cache=rateSamples.get(engineKey(engine)) ?? new Map<string,{at:number;brain:string|undefined;sample:RateSample}>();- rateSamples.set(engineKey(engine),cache);- const hit=cache.get(scope);- const sample=hit && hit.brain===oldest?.id && Date.now()-hit.at<RATE_SAMPLE_TTL_MS ? hit.sample : await readRateSample(engine,scope,where,params);- if(sample!==hit?.sample) cache.set(scope,{at:Date.now(),brain:oldest?.id,sample});+ const [sample]=await engine.executeRaw<{admissions:number;age_seconds:number|null;gross:number|null;net:number|null}>(`WITH recent AS (+ SELECT id,created_at,terminal_reservation,authority,outcome,error_detail,client_request_id FROM persistence_requests+ ${principal ? 'WHERE principal_kind=$1 AND principal_id=$2' : ''} ORDER BY sequence DESC LIMIT 1000),+ e AS (SELECT request_id,SUM(${EFFECT_BYTES_SQL}) AS bytes FROM persistence_effects WHERE request_id IN (SELECT id FROM recent) GROUP BY request_id)+ SELECT count(*)::int AS admissions,EXTRACT(EPOCH FROM (now()-min(created_at)))::float8 AS age_seconds,avg(terminal_reservation)::float8 AS gross,+ avg(${COMPACTED_RECEIPT_BYTES_SQL('recent', 'e')})::float8 AS net FROM recent LEFT JOIN e ON e.request_id=recent.id`,+ principal ? [principal[1],principal[2]] : []);+ const perDay=sample!.admissions/Math.max(1/24,(sample!.age_seconds ?? 0)/86_400);const retention=await readReceiptRetentionDays(engine).catch(()=>DEFAULT_RECEIPT_RETENTION_DAYS);- const windowIds=Number(sample.window_ids ?? 0)+Number(sample.topology_ids ?? 0);- if(windowIds>0) {- // A week busier than the walk reads the walk's own span, so the rate is never diluted over days it did not read.- const windowDays=Number(sample.window_ids ?? 0)>=WINDOW_WALK_ROWS ? Math.max(1/24,Number(sample.window_span_days ?? 0)) : Math.min(7,Math.max(1,Number(sample.brain_age_days ?? 7)));- const topologyBytes=Number(sample.topology_bytes ?? 0);- return setting==='LifetimeIds' ? recommendedLimit('lifetime_ids',used,limit,{perDay:windowIds/windowDays},retention)- : recommendedLimit('terminal_bytes',used,limit,{perDay:(Number(sample.window_gross)+topologyBytes)/windowDays,netPerDay:(Number(sample.window_net)+topologyBytes)/windowDays},retention);- }- const perDay=sample.admissions/Math.max(1/24,(sample.age_seconds ?? 0)/86_400);const measured=(value:number|null|undefined,fallback:number)=>Number.isFinite(Number(value)) && Number(value)>0 ? Number(value) : fallback;- return setting==='LifetimeIds' ? recommendedLimit('lifetime_ids',used,limit,{perDay},retention)- : recommendedLimit('terminal_bytes',used,limit,{perDay:perDay*measured(sample.gross,RECEIPT_RESERVATION_FLOOR),netPerDay:perDay*measured(sample.net,RETAINED_RECEIPT_BYTES)},retention);-}--type RateSample={admissions:number;age_seconds:number|null;gross:number|null;net:number|null;window_ids?:number|null;window_gross?:number|null;- window_net?:number|null;window_span_days?:number|null;brain_age_days?:number|null;topology_ids?:number|null;topology_bytes?:number|null};-const RATE_SAMPLE_TTL_MS=60_000;-const rateSamples=new WeakMap<object,Map<string,{at:number;brain:string|undefined;sample:RateSample}>>();--/** The scope's admission sample and 7-day window; lifecycle changes count for the brain and for the local CLI writer, as in readCapacityForecast. */-async function readRateSample(engine:SqlEngine,scope:string,where:string,params:string[]):Promise<RateSample> {- const topologyWhere=scope==='brain' ? '' : params[0]==='local_cli' ? 'AND principal_id::text=$2' : 'AND false';- const [sample]=await engine.executeRaw<RateSample>(`WITH recent AS (- SELECT id,created_at,terminal_reservation,authority,outcome,error_detail,client_request_id FROM persistence_requests ${where} ORDER BY sequence DESC LIMIT 1000),- w AS (SELECT * FROM (SELECT id,created_at,terminal_reservation,authority,outcome,error_detail,client_request_id FROM persistence_requests ${where}- ORDER BY sequence DESC LIMIT ${WINDOW_WALK_ROWS}) walk WHERE created_at>now()-interval '7 days'),- e AS (SELECT request_id,SUM(${EFFECT_BYTES_SQL}) AS bytes FROM persistence_effects WHERE request_id IN (SELECT id FROM w) GROUP BY request_id)- SELECT (SELECT count(*)::int FROM recent) AS admissions,(SELECT EXTRACT(EPOCH FROM (now()-min(created_at)))::float8 FROM recent) AS age_seconds,- (SELECT avg(terminal_reservation)::float8 FROM recent) AS gross,- (SELECT avg(LEAST(terminal_reservation,octet_length(authority::text)+COALESCE(octet_length(outcome::text),0)+COALESCE(octet_length(error_detail::text),0)+${COMPACTED_HEADROOM_BYTES}))::float8 FROM recent) AS net,- (SELECT count(*)::int FROM w) AS window_ids,(SELECT EXTRACT(EPOCH FROM now()-min(created_at))/86400 FROM w)::float8 AS window_span_days,(SELECT COALESCE(SUM(terminal_reservation),0)::float8 FROM w) AS window_gross,- (SELECT COALESCE(SUM(${COMPACTED_RECEIPT_BYTES_SQL('w', 'e')}),0)::float8 FROM w LEFT JOIN e ON e.request_id=w.id) AS window_net,- (SELECT EXTRACT(EPOCH FROM now()-created_at)/86400 FROM persistence_requests ORDER BY sequence LIMIT 1)::float8 AS brain_age_days,- (SELECT count(*)::int FROM persistence_topology_changes WHERE created_at>now()-interval '7 days' ${topologyWhere}) AS topology_ids,- (SELECT COALESCE(SUM(terminal_bytes),0)::float8 FROM persistence_topology_changes WHERE created_at>now()-interval '7 days' ${topologyWhere}) AS topology_bytes`,- params);- return sample!;+ return resource==='lifetime_ids' ? recommendedLimit(resource,used,limit,{perDay},retention)+ : recommendedLimit(resource,used,limit,{perDay:perDay*measured(sample!.gross,RECEIPT_RESERVATION_FLOOR),netPerDay:perDay*measured(sample!.net,RETAINED_RECEIPT_BYTES)},retention);}/**diff --git a/test/persistence-capacity-notices.test.ts b/test/persistence-capacity-notices.test.tsindex c59bb42618..d0697951a1 100644--- a/test/persistence-capacity-notices.test.ts+++ b/test/persistence-capacity-notices.test.ts@@ -68,7 +68,7 @@ for (const kind of testBackends()) {// GST-2107: the notice rides a write that succeeded, so its last lines name a read-only check, never a cap raise for the user.expect(first[0]).toMatchObject({ kind: 'info', fix: { actor: 'agent', argv: ['gbrain', 'doctor', '--only', 'persistence_capacity', '--json'] } });expect(first[0]!.why).toContain('81%');- expect(first[0]!.why).toContain('This write was saved');+ expect(first[0]!.why).toContain('This notice is not a refusal');expect(first[0]!.why).not.toContain('config set');// A remote agent reads the rendered notice: over MCP its fix is the read-scope get_brain_health call, never a step to relay.for (const transport of ['http', 'stdio'] as const) {@@ -145,9 +145,10 @@ for (const kind of testBackends()) {await tx.executeRaw("UPDATE persistence_requests SET created_at=now()-interval '2 days' WHERE id=ANY($1::uuid[])", [ids.slice(1, 101)]);});await engine.setConfig('persistence.limits.principal_lifetime_ids', '1101');+ // The forecast a receipt, doctor or /metrics read in the last minute records the rates the refusal uses under its locks.+ const forecast = await readCapacityForecast(engine);const refusal = await fixture.admit('one-more').then(() => null, (error: { code?: string; fix?: { argv?: string[] } }) => error);expect(refusal).toMatchObject({ code: 'write_quota_exhausted', reason: 'lifetime_ids' });- const forecast = await readCapacityForecast(engine);const doctor = forecast.rows.find(r => r.scope.startsWith('principal:') && r.resource === 'lifetime_ids')!;expect(doctor.recommended_limit).toBe(Math.ceil(1101 + (1100 / 7) * 365));expect(Number(refusal!.fix!.argv!.at(-1))).toBe(doctor.recommended_limit);@@ -173,24 +174,22 @@ for (const kind of testBackends()) {}});- check('repeated refusals within a minute reuse one rate read per scope, as a paused producer retries under the counter locks', async engine => {+ check('a refusal under the counter locks reads at most the latest 1,000 admissions, and none after a forecast recorded the rates', async engine => {const fixture = await requestFixture(engine);for (let i = 0; i < 3; i++) await fixture.admit(`rate-${i}`);- let walks = 0;+ await engine.setConfig('persistence.limits.brain_lifetime_ids', '10');+ const reads: string[] = [];const counted = { ...engine, executeRaw: (sql: string, params?: unknown[]) => {- if (sql.includes('LIMIT 200000')) walks++;+ if (/FROM persistence_requests/.test(sql)) reads.push(sql);return engine.executeRaw(sql, params);} } as unknown as BrainEngine;- const first = await oneYearCapacity(counted, 'brain', 'LifetimeIds', 3, 10);- expect(await oneYearCapacity(counted, 'brain', 'LifetimeIds', 3, 10)).toBe(first);- expect(await oneYearCapacity(counted, 'brain', 'TerminalBytes', 3, 10)).toBeGreaterThan(0);- expect(walks).toBe(1);- await fixture.admit('rate-new');- expect(await oneYearCapacity(counted, 'brain', 'LifetimeIds', 4, 10)).toBeGreaterThanOrEqual(first);- expect(walks).toBe(1);- const principal = (await readCapacityForecast(engine)).rows.find(r => r.scope.startsWith('principal:'))!.scope;- await oneYearCapacity(counted, principal, 'LifetimeIds', 4, 10);- expect(walks).toBe(2);+ await oneYearCapacity(counted, 'brain', 'LifetimeIds', 3, 10);+ expect(reads.some(sql => sql.includes('LIMIT 1000'))).toBe(true);+ expect(reads.some(sql => /LIMIT (?!1\b|1000\b)\d+/.test(sql))).toBe(false);+ const doctor = (await readCapacityForecast(counted)).rows.find(r => r.scope === 'brain' && r.resource === 'lifetime_ids')!;+ reads.length = 0;+ expect(await oneYearCapacity(counted, 'brain', 'LifetimeIds', doctor.used, doctor.limit)).toBe(doctor.recommended_limit);+ expect(reads.some(sql => sql.includes('LIMIT 1000'))).toBe(false);});check('writes within a minute share one snapshot; the next minute reads again', async engine => {