Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Book the rest of the daily embed cap before upkeep spends it

garrytanf65c53a515e8
diff --git a/changelog.d/capy-gbra438-upkeep-core.md b/changelog.d/capy-gbra438-upkeep-core.md
index 3c3aaa077c..f2dd83559a 100644
--- a/changelog.d/capy-gbra438-upkeep-core.md
+++ b/changelog.d/capy-gbra438-upkeep-core.md
@@ -1,6 +1,6 @@
### A keyless brain keeps itself tidy, and doctor says when upkeep last ran
-**`gbrain maintain --unattended` runs the free upkeep a brain needs, and is safe to run from a scheduler.** It runs the same steps as `gbrain maintain --safe` (link and timeline extraction, stale-page checks, the free repair lanes) plus a new bookkeeping cleanup step. It never deletes a page and never starts a paid model call, with one exception you can switch off. On a brain with an embedding key, it embeds pages and facts that are missing embeddings, capped at $1 a day per brain (`maintenance.embed_daily_usd`, 0 turns it off). Two runs that overlap embed once, because the second skips that step. It works the same on a brain with no git repository and no API keys. A run that has nothing to do, or that finds the brain busy, degraded, behind on migrations or already looked after by `gbrain autopilot`, exits 0 and says why. A step that fails three runs in a row is retried once a week instead of every run, and doctor names it until the step succeeds or stops being part of upkeep.
+**`gbrain maintain --unattended` runs the free upkeep a brain needs, and is safe to run from a scheduler.** It runs the same steps as `gbrain maintain --safe` (link and timeline extraction, stale-page checks, the free repair lanes) plus a new bookkeeping cleanup step. It never deletes a page and never starts a paid model call, with one exception you can switch off. On a brain with an embedding key, it embeds pages and facts that are missing embeddings, capped at $1 a day per brain (`maintenance.embed_daily_usd`, 0 turns it off). Two runs that overlap embed once, because the second skips that step. A run that is killed while embedding cannot spend that day's cap a second time, because the step books the rest of the cap before it spends and settles it afterwards. It works the same on a brain with no git repository and no API keys. A run that has nothing to do, or that finds the brain busy, degraded, behind on migrations or already looked after by `gbrain autopilot`, exits 0 and says why. A step that fails three runs in a row is retried once a week instead of every run, and doctor names it until the step succeeds or stops being part of upkeep.
- `gbrain maintain --status` shows when upkeep last ran, what each step did, what failed, and when the next run is due.
- A new doctor check, `unattended_maintenance`, reports the same thing. It warns when upkeep hasn't run for 48 hours or a step keeps failing, and its fix is `gbrain maintain --unattended`. A brain where `maintenance.auto` is `false`, or where autopilot runs upkeep, reads as ok.
diff --git a/src/core/maintenance/run.ts b/src/core/maintenance/run.ts
index 0fe2161954..90f98f0292 100644
--- a/src/core/maintenance/run.ts
+++ b/src/core/maintenance/run.ts
@@ -288,6 +288,7 @@ async function runEmbedStep(engine: BrainEngine, now: Date, signal: AbortSignal
}
}
+/** Reserves the rest of the day's cap in the ledger before any spend and settles it to the actual spend afterwards, so a run killed mid-step cannot spend the cap again. */
async function embedWithinDailyCap(engine: BrainEngine, now: Date, signal: AbortSignal | undefined): Promise<{ step: UpkeepStep; embed: { spent_today_usd: number; cap_usd: number } }> {
const cap = await readEmbedDailyCapUsd(engine);
const spentBefore = await readEmbedSpentToday(engine, now);
@@ -301,6 +302,7 @@ async function embedWithinDailyCap(engine: BrainEngine, now: Date, signal: Abort
const [{ embedCostCap }, { loadPricingOverrides }, { runEmbedCore }, { embedStaleFacts }] = await Promise.all([
import('../embed-cost-cap.ts'), import('../budget/budget-tracker.ts'), import('../../commands/embed.ts'), import('../embed-facts.ts')]);
const costCap = embedCostCap(remaining, 'user', await loadPricingOverrides(engine).catch(() => undefined))!;
+ await engine.setConfig(MAINTENANCE_EMBED_SPEND_KEY, JSON.stringify({ day: utcDay(now), usd: cap }));
const combined = anyAbortSignal(signal ? [costCap.signal, signal] : [costCap.signal]);
let spent = 0;
let status: UpkeepStepStatus = 'ok';
diff --git a/test/maintenance-unattended.test.ts b/test/maintenance-unattended.test.ts
index 2ef970527c..6e119a994b 100644
--- a/test/maintenance-unattended.test.ts
+++ b/test/maintenance-unattended.test.ts
@@ -23,7 +23,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { ALL_PHASES, PHASE_SCOPE, runCycle, type CyclePhase } from '../src/core/cycle.ts';
import { MANAGED_PHASE_TABLE } from '../src/core/cycle/phase-table.ts';
import {
- MAINTAIN_SAFE_PHASES, MAINTENANCE_EMBED_SPEND_KEY, UPKEEP_INTERVAL_MS, readUpkeepRecord, runUnattendedMaintenance, unattendedPlan,
+ MAINTAIN_SAFE_PHASES, MAINTENANCE_EMBED_SPEND_KEY, UPKEEP_INTERVAL_MS, readEmbedSpentToday, readUpkeepRecord, runUnattendedMaintenance, unattendedPlan,
type SafePhaseProviders,
} from '../src/core/maintenance/run.ts';
import { checkUnattendedMaintenance } from '../src/commands/doctor/checks/unattended-maintenance.ts';
@@ -363,6 +363,22 @@ for (const kind of testBackends()) {
expect(embedSteps).toContainEqual({ name: 'embed', scope: 'global', status: 'skipped', reason: 'embed_running' });
});
+ test('the ledger holds the rest of the cap while the step spends, so a run killed mid-step cannot spend it again', async () => {
+ await keyed();
+ const [col] = await engine.executeRaw<{ dims: number }>("SELECT atttypmod AS dims FROM pg_attribute WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'");
+ const midStep: number[] = [];
+ __setEmbedTransportForTests((async ({ values }: { values: string[] }) => {
+ midStep.push(await readEmbedSpentToday(engine, new Date()));
+ return { embeddings: values.map(() => new Array(Number(col!.dims)).fill(0.01)) };
+ }) as never);
+ await engine.putPage('notes/embed-me', { type: 'concept', title: 'Embed', compiled_truth: 'A page whose chunks need vectors.' });
+ const outcome = await upkeep(engine, { runner: 'test', providers: KEYED, cycle: quick });
+ expect(outcome.record!.steps).toContainEqual(expect.objectContaining({ name: 'embed', status: 'ok' }));
+ expect(midStep.length).toBeGreaterThan(0);
+ expect(midStep.every((usd) => usd === 1)).toBe(true);
+ expect(await readEmbedSpentToday(engine, new Date())).toBeLessThan(1);
+ });
+
test('the embed step does not start once the run budget is spent', async () => {
await keyed();
await engine.putPage('notes/embed-me', { type: 'concept', title: 'Embed', compiled_truth: 'A page whose chunks need vectors.' });