Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Show the parked pages a manual purge would free in its preview, and write nothing before consent

garrytandaa856c4bf4a
diff --git a/docs/architecture/key-files/memory-purge.md b/docs/architecture/key-files/memory-purge.md
index e8e3a17657..f21a428ecb 100644
--- a/docs/architecture/key-files/memory-purge.md
+++ b/docs/architecture/key-files/memory-purge.md
@@ -34,6 +34,6 @@ User-facing contract: [expire versus purge](../../guides/memory-boundaries.md#pu
- `src/commands/doctor/checks/atom-fence-rows.ts` — doctor `atom_fence_rows` (BRAIN, a host-only `WAVE_CHECKS` repair entry cleared by the explicit-only `atom-fence-rows`, so remediation classifies it `explicit_kind_required`): runs `findAtomFenceRows` per source, each statement bounded by `PLAN_STATEMENT_MS`; `warn` names the count per source and the read-only preview `gbrain repair atom-fence-rows --source <id> --json` as an agent `fix`, never a slug or row text; an overrun reports the count as unknown.
- `src/core/persistence/purge-parks.ts` — #6741 C: per-tombstone purge parks in op_checkpoints (op `purge-deleted-park`, key source+slug+deleted_at, so a re-deleted page starts fresh); park at `PURGE_PARK_LIMIT` (5) non-deferral failures, a success clears it, and parks of a source seen inactive rearm the first run its writer is active again (`notePurgeSourceWriter`). `purgeStaleCheckpoints` exempts the op; `prunePurgeParks` drops parks whose tombstone is gone. A park records the gbrain `version` of its last failure; `isParked` (the managed and unmanaged selection, doctor `purge_parked_pages`) counts only the running version's, so a park another version wrote (or one with no version) gets one automatic attempt and a failure re-parks it under the running version. Tests: `test/purge-deleted-parks.test.ts`, `test/purge-parks-version.test.ts` (Postgres: `test/e2e/purge-parks-version-postgres.test.ts`).
-- `src/core/persistence/derived-origin-hold.ts` + `src/core/repair/origin-carry.ts` — GST-2689: a page that reads as world by its own label but is hidden from remote callers only through a private origin (`derived_from`, an atom's `source_slug`, followed through `slug_aliases` and `slug_rename_history`, or a concept's provenance atom) keeps that privacy once the origin row is gone. `origin-carry.ts` finds the dependents with `private-labels.ts` `dependentsSql('remove')` (it reads only the pages whose frontmatter names a removed slug, or a slug a rename moved to it, through `idx_pages_frontmatter`, one jsonpath scan per 100 slugs of a source, so its cost does not grow with the source; the postimage drops the origin rows; a page that names readers is returned when its listed readers would gain it, and gets the label with its `readable_by` kept; a life-chronicle event is returned when the removal opens it, never when its `event.depth` page is removed, since it then fails closed) and gives each `visibility: private` through the label door (`publishMaintenancePages` with `keepTrust` and the label edit on a managed brain, batch `derived-origin-carry:<request_id>`; `writeLabelFile` or `importAtRevision` otherwise). `deletePageCarryingPrivate` is the `delete_page` handler body: a soft delete (local or remote) labels after it committed and returns the unchanged receipt; a purge labels first and, when that fails, soft-deletes instead (an existing tombstone stays), parks the origin and returns `purge_held`. `carryPrivateBeforePurge` runs before every expiry over all the origins (`type = 'atom'` or hidden by `privacyPartSql`) expired or expiring within `CARRY_LEAD_HOURS` (24) at once, so a page private through two of them is found and none crosses the cutoff mid-run uncarried; it publishes the labels 200 per maintenance batch, parks the blocked ones and returns the ones whose park no longer applies, which `purgeDeletedPagesCoordinated` clears only after its purge, so an origin parked when the purge was previewed stays that run; a consented manual purge (`gbrain pages purge-deleted`, `purge_deleted_pages`) first runs `retryParkedOrigins` so its preview lists the origins it frees. A soft-deleted dependent is returned too but never labelled: it blocks its origin (`dependent_soft_deleted`) until it is purged or restored, since a restore would bring it back with the origin row gone; a removed origin tied to another removed origin counts as such a dependent (judged on its own row), since another hold may keep it. Inside each delete transaction `parkUncarriedInPurge` rechecks the origins about to go (the page purge for an explicit or managed retention purge, the unmanaged bulk delete, and each recorded tombstone): an origin a page written after the carry still depends on is parked in that transaction and its row is kept (a live page is soft-deleted instead, and the receipt carries `purge_held` naming no page), so the next purge labels the page and the one after removes the origin. The origin is looked up by the lowercased slug, as `submitPageMutation` deletes it. `derived-origin-hold.ts` is the park (op_checkpoints op `derived-origin-park`, fingerprint the page id, with `blocking_slug`, `code` and `reason`), `derivedOriginPurgeHold` (no engine-sql imports), prune of restored or purged pages, and `readDerivedOriginParks` for doctor `derived_origin_parked`; `purgeStaleCheckpoints` exempts the op. Tests: `test/derived-origin-carry-private.test.ts` (repo-less, minted tokens), `test/derived-origin-carry-managed.test.ts` (managed Git brain), `test/derived-origin-recheck-plan.test.ts` (pages rows the recheck visits).
+- `src/core/persistence/derived-origin-hold.ts` + `src/core/repair/origin-carry.ts` — GST-2689: a page that reads as world by its own label but is hidden from remote callers only through a private origin (`derived_from`, an atom's `source_slug`, followed through `slug_aliases` and `slug_rename_history`, or a concept's provenance atom) keeps that privacy once the origin row is gone. `origin-carry.ts` finds the dependents with `private-labels.ts` `dependentsSql('remove')` (it reads only the pages whose frontmatter names a removed slug, or a slug a rename moved to it, through `idx_pages_frontmatter`, one jsonpath scan per 100 slugs of a source, so its cost does not grow with the source; the postimage drops the origin rows; a page that names readers is returned when its listed readers would gain it, and gets the label with its `readable_by` kept; a life-chronicle event is returned when the removal opens it, never when its `event.depth` page is removed, since it then fails closed) and gives each `visibility: private` through the label door (`publishMaintenancePages` with `keepTrust` and the label edit on a managed brain, batch `derived-origin-carry:<request_id>`; `writeLabelFile` or `importAtRevision` otherwise). `deletePageCarryingPrivate` is the `delete_page` handler body: a soft delete (local or remote) labels after it committed and returns the unchanged receipt; a purge labels first and, when that fails, soft-deletes instead (an existing tombstone stays), parks the origin and returns `purge_held`. `carryPrivateBeforePurge` runs before every expiry over all the origins (`type = 'atom'` or hidden by `privacyPartSql`) expired or expiring within `CARRY_LEAD_HOURS` (24) at once, so a page private through two of them is found and none crosses the cutoff mid-run uncarried; it publishes the labels 200 per maintenance batch, parks the blocked ones and returns the ones whose park no longer applies, which `purgeDeletedPagesCoordinated` clears only after its purge, so an origin parked when the purge was previewed stays that run; a consented manual purge (`gbrain pages purge-deleted`, `purge_deleted_pages`) previews with `forecastCarry` (read-only: the parked origins the carry would free, `released_origins`, and the origins it would park, both judged as `carryPrivate` judges them) passed to `selectPurgeCandidates`, so `--dry-run` and the consent list exactly what `--yes` purges and nothing is labelled before the consent; the run passes the freed ids as `release`, and `purgeDeletedPagesCoordinated` clears those parks once its carry landed, before it purges. A soft-deleted dependent is returned too but never labelled: it blocks its origin (`dependent_soft_deleted`) until it is purged or restored, since a restore would bring it back with the origin row gone; a removed origin tied to another removed origin counts as such a dependent (judged on its own row), since another hold may keep it. Inside each delete transaction `parkUncarriedInPurge` rechecks the origins about to go (the page purge for an explicit or managed retention purge, the unmanaged bulk delete, and each recorded tombstone): an origin a page written after the carry still depends on is parked in that transaction and its row is kept (a live page is soft-deleted instead, and the receipt carries `purge_held` naming no page), so the next purge labels the page and the one after removes the origin. The origin is looked up by the lowercased slug, as `submitPageMutation` deletes it. `derived-origin-hold.ts` is the park (op_checkpoints op `derived-origin-park`, fingerprint the page id, with `blocking_slug`, `code` and `reason`), `derivedOriginPurgeHold` (no engine-sql imports), prune of restored or purged pages, and `readDerivedOriginParks` for doctor `derived_origin_parked`; `purgeStaleCheckpoints` exempts the op. Tests: `test/derived-origin-carry-private.test.ts` (repo-less, minted tokens), `test/derived-origin-carry-managed.test.ts` (managed Git brain), `test/derived-origin-recheck-plan.test.ts` (pages rows the recheck visits).
- `src/core/repair/segmented-mirror.ts` — `gbrain repair segmented-mirror` (LPH a), explicit-only, preview-bound, destructive consent, `checks: []` (no doctor scan: detection reads every segmented file). `scanSegmentedMirrors` (read-only, owner host; a non-owner source is held `not_owner_host`) walks live `body_storage='segmented'` pages of sources with a worktree binding whose snapshot ledger applies (`ledgerApplies`), runs `prepareFileTarget(..., null, { allowMissing })` (the mirror's own drift check; `file_database_drift` or any other `OperationError` is held, never rewritten), then lists the page only when overlaying the ledger on the file's facts fences changes its region digests (`fileBodyDigests` vs `overlaidFileBodyDigests` over `FileBytes`, bounded memory) and the streamed render (`unreadPageFile`) hashes differently. The hash (`segmented-mirror-v1`) binds source selection, slug, page id, revision, path and both SHA-256s; the approved set lives in `op_checkpoints`. Apply re-checks revision and file hash of every page not yet admitted (`preview_changed`), then per page admits and commits, in one transaction like a withdrawal, a `repair_segmented_mirror` request (request id derived from the hash; its receipt) that inserts a version-2 targeted `withdrawal-mirror` + `git` effect (no embedding) whose `data.approved_file` carries the preview's file and render SHA-256 (`mirrorPage` skips the target `preview_changed` when the file's before-hash or the streamed render differs, before the gate or any recovery record; `publishGitPage` skips it when the file is not the approved render; the outcome is `refused` `preview_changed` with a read-only fix re-running the preview); it wakes this process's consumer and waits up to the write wait for them (`mirrored` / `mirror_queued` / `held` / `failed`; a replay is `replayed`). Test: `test/pathological/lp-a-segmented-mirror.test.ts`.
diff --git a/src/commands/pages.ts b/src/commands/pages.ts
index dba0e03d4a..50f069d2de 100644
--- a/src/commands/pages.ts
+++ b/src/commands/pages.ts
@@ -47,10 +47,13 @@ async function runPurgeDeleted(engine: BrainEngine, args: string[]): Promise<voi
const { olderThanHours, dryRun, json, retryParked } = parsePurgeArgs(args);
// #6741: one selector for the preview, the consent and the run (selectPurgeCandidates): the same cutoff, rename
// holds, writer state and parks; --retry-parked puts the parked pages the cutoff selects back among the candidates.
- if (!dryRun) await (await import('../core/repair/origin-carry.ts')).retryParkedOrigins(engine, olderThanHours);
- const selection = await selectPurgeCandidates(engine, olderThanHours, { retryParked });
+ // GST-2689: the preview reads ahead what the run's carry does (forecastCarry, read-only), so it lists the parked origins
+ // the run releases and leaves out the ones it parks; nothing is labelled before the consent.
+ const forecast = await (await import('../core/repair/origin-carry.ts')).forecastCarry(engine, olderThanHours);
+ const selection = await selectPurgeCandidates(engine, olderThanHours, { retryParked, forecast });
const retried = selection.parked.filter(park => park.retried).length;
- const notes = { ...(selection.parked.length ? { parked: selection.parked, retry_parked: retryParked } : {}), ...(selection.blocked.length ? { blocked: selection.blocked } : {}) };
+ const notes = { ...(selection.parked.length ? { parked: selection.parked, retry_parked: retryParked } : {}), ...(selection.blocked.length ? { blocked: selection.blocked } : {}),
+ ...(selection.released_origins.length ? { released_origins: selection.released_origins } : {}) };
if (dryRun) {
// Read-only: the same predicate and DB now() clock as the real purge; only the verb differs.
@@ -65,6 +68,7 @@ async function runPurgeDeleted(engine: BrainEngine, args: string[]): Promise<voi
for (const c of selection.candidates) console.log(` ${c.source_id}/${c.slug}${c.park ? ' (parked; retried once more)' : ''}${c.sync_unreviewed ? ' (deleted by a managed sync; no live twin)' : ''}`);
for (const p of selection.parked.filter(park => !park.retried)) console.log(` parked: ${p.source_id}/${p.slug} after ${p.failures} failures (${p.code ?? 'error'}); skipped unless --retry-parked`);
for (const b of selection.blocked.filter(entry => entry.code === 'writer_not_active')) console.log(` waiting: ${b.source_id}/${b.slug}: ${b.reason}`);
+ for (const o of selection.released_origins) console.log(` released: ${o.source_id}/${o.slug} is parked (gbrain doctor: derived_origin_parked); the run labels the pages derived from it private first and keeps it parked if that fails`);
return;
}
@@ -82,7 +86,7 @@ async function runPurgeDeleted(engine: BrainEngine, args: string[]): Promise<voi
const authorized = await consentGate(purgeConsentRequest(selection.count, olderThanHours, json, args, retryParked, retried, selection.sync_unreviewed), { json });
if (!authorized) return;
- const result = await purgeDeletedPagesCoordinated(engine, olderThanHours, { retryParked });
+ const result = await purgeDeletedPagesCoordinated(engine, olderThanHours, { retryParked, release: forecast.release.map(origin => origin.id) });
if (json) {
console.log(JSON.stringify({ older_than_hours: olderThanHours, count: result.count, slugs: result.slugs,
...(result.capacity_deferred ? { capacity_deferred: result.capacity_deferred } : {}), ...(result.blocked.length ? { blocked: result.blocked } : {}) }, null, 2));
diff --git a/src/core/persistence/purge-deleted.ts b/src/core/persistence/purge-deleted.ts
index ee609ad8e2..fa3da414c3 100644
--- a/src/core/persistence/purge-deleted.ts
+++ b/src/core/persistence/purge-deleted.ts
@@ -16,6 +16,7 @@ import { atomRetirementPurgeHold } from './atom-retirement-hold.ts';
import { derivedRetirementPurgeHold } from './derived-retirement-hold.ts';
import { faultPoint } from './fault-points.ts';
import { clearDerivedOriginParks, DERIVED_ORIGIN_PARKED_CODE, derivedOriginPurgeHold, readDerivedOriginParks, type DerivedOriginPark } from './derived-origin-hold.ts';
+import type { CarryForecast } from '../repair/origin-carry.ts';
import { writerOnboardingPreflight } from './onboarding.ts';
import { clearPurgePark, isParked, notePurgeSourceWriter, PURGE_PARK_LIMIT, purgeParkKey, prunePurgeParks, readPurgeParks, recordPurgeFailure, type PurgePark } from './purge-parks.ts';
import { ItemRun, type ItemFailure } from '../item-failure.ts';
@@ -91,23 +92,23 @@ export function purgeConsentRequest(count: number, olderThanHours: number, json:
/** The `purge_deleted_pages` op (`gbrain call`): the D6 consent first (`yes` is `--yes`); an empty purge set needs none. */
export async function consentedPurgeDeletedPages(engine: BrainEngine, olderThanHours: number, yes: boolean) {
- await (await import('../repair/origin-carry.ts')).retryParkedOrigins(engine, olderThanHours);
- const preview = await selectPurgeCandidates(engine, olderThanHours);
+ const forecast = await (await import('../repair/origin-carry.ts')).forecastCarry(engine, olderThanHours);
+ const preview = await selectPurgeCandidates(engine, olderThanHours, { forecast });
if (preview.count === 0) return { status: 'purged', count: 0, slugs: [] };
await requireConsent(purgeConsentRequest(preview.count, olderThanHours, false, yes ? ['--yes'] : [], false, 0, preview.sync_unreviewed));
- const result = await purgeDeletedPagesCoordinated(engine, olderThanHours);
+ const result = await purgeDeletedPagesCoordinated(engine, olderThanHours, { release: forecast.release.map(origin => origin.id) });
return { status: result.failed ? 'partial' : 'purged', count: result.count, slugs: result.slugs, ...(result.blocked.length ? { blocked: result.blocked } : {}) };
}
/** An expired tombstone of a source the belief ledger records (unmanaged brain), in scan order, with its live failure record. */
interface RecordedTombstone { id: number; source_id: string; slug: string; deleted_us: string; key: string; park?: PurgePark }
-async function recordedTombstones(engine: BrainEngine, hours: string): Promise<RecordedTombstone[]> {
+async function recordedTombstones(engine: BrainEngine, hours: string, released: readonly number[] = []): Promise<RecordedTombstone[]> {
const rows = await engine.executeRaw<{ id: number; source_id: string; slug: string; deleted_us: string }>(`SELECT p.id, p.source_id, p.slug,
floor(extract(epoch FROM p.deleted_at)*1000000)::bigint::text AS deleted_us FROM pages p
- WHERE p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($1 || ' hours')::interval AND ${identityDowngradePurgeHold('p')} AND ${atomRetirementPurgeHold('p')} AND ${derivedRetirementPurgeHold('p')} AND ${derivedOriginPurgeHold('p')}
+ WHERE p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($1 || ' hours')::interval AND ${identityDowngradePurgeHold('p')} AND ${atomRetirementPurgeHold('p')} AND ${derivedRetirementPurgeHold('p')} AND (${derivedOriginPurgeHold('p')} OR p.id = ANY($2::int[]))
AND EXISTS (SELECT 1 FROM belief_events e WHERE e.source_id = p.source_id AND e.subject_kind = 'coverage')
- ORDER BY p.deleted_at ASC, p.source_id ASC, p.slug ASC`, [hours]);
+ ORDER BY p.deleted_at ASC, p.source_id ASC, p.slug ASC`, [hours, [...released]]);
const parks = rows.length ? await readPurgeParks(engine) : new Map<string, PurgePark>();
return rows.map(row => {
const key = purgeParkKey(row.source_id, row.slug, row.deleted_us);
@@ -219,23 +220,44 @@ export interface PurgeSelection {
sync_unreviewed: number;
/** Per source in the scan: whether its writer passed the owner verification (the park rearm reads it). */
writers: Record<string, boolean>;
+ /** GST-2689: the candidates that are parked origins the run's carry releases first (`forecast.release`). */
+ released_origins: Array<{ source_id: string; slug: string }>;
}
-/** What keeps an expired tombstone of page `page` (source `source`) out of every managed purge: an archived source, or the rename-recovery, restorable-atom, derived-retirement or derived-origin hold. */
-export function purgeHoldsSql(page: string, source: string): string {
- return `NOT ${source}.archived AND ${identityDowngradePurgeHold(page)} AND ${atomRetirementPurgeHold(page)} AND ${derivedRetirementPurgeHold(page)} AND ${derivedOriginPurgeHold(page)}`;
+/**
+ * What keeps an expired tombstone of page `page` (source `source`) out of every managed purge: an archived source, or the
+ * rename-recovery, restorable-atom, derived-retirement or derived-origin hold; `released` (an SQL int[]) lifts the
+ * derived-origin hold for the parked origins a manual purge's preview counts as released.
+ */
+export function purgeHoldsSql(page: string, source: string, released?: string): string {
+ const origin = released ? `(${derivedOriginPurgeHold(page)} OR ${page}.id = ANY(${released}))` : derivedOriginPurgeHold(page);
+ return `NOT ${source}.archived AND ${identityDowngradePurgeHold(page)} AND ${atomRetirementPurgeHold(page)} AND ${derivedRetirementPurgeHold(page)} AND ${origin}`;
}
-export async function selectPurgeCandidates(engine: BrainEngine, olderThanHours: number, opts: { retryParked?: boolean; automatic?: boolean } = {}): Promise<PurgeSelection> {
+/**
+ * GST-2689 `forecast` (repair/origin-carry.ts forecastCarry; the consented manual purge's preview and consent): the
+ * selection the run makes once its carry has run, so it counts the parked origins the carry releases and leaves out the
+ * origins it parks; the run itself reads the parks as they are.
+ */
+export async function selectPurgeCandidates(engine: BrainEngine, olderThanHours: number, opts: { retryParked?: boolean; automatic?: boolean; forecast?: CarryForecast } = {}): Promise<PurgeSelection> {
+ const release = (opts.forecast?.release ?? []).map(origin => origin.id), hold = (opts.forecast?.hold ?? []).map(origin => origin.id);
if (!await managedPersistenceEnabled(engine)) {
- const parked = (await recordedTombstones(engine, String(Math.max(0, Math.floor(olderThanHours)))))
+ const parked = (await recordedTombstones(engine, String(Math.max(0, Math.floor(olderThanHours))), release))
.filter((row): row is RecordedTombstone & { park: PurgePark } => isParked(row.park));
const skipped = new Set(opts.retryParked ? [] : parked.map(row => row.id));
const unreviewed = (await readUnreviewedSyncTombstones(engine, olderThanHours)).filter(row => !skipped.has(row.id));
const kept = opts.automatic ? unreviewed : [];
- const preview = await engine.purgeDeletedPages(olderThanHours, { dryRun: true, excludeIds: [...skipped, ...kept.map(row => row.id)] });
+ const excludeIds = [...skipped, ...kept.map(row => row.id), ...hold];
+ const preview = await engine.purgeDeletedPages(olderThanHours, { dryRun: true, excludeIds });
+ // The engine's preview keeps every parked origin; the ones the run frees are read here under the purge's other holds.
+ const released = release.length ? (await engine.executeRaw<{ slug: string; source_id: string; deleted_at: Date | string }>(`SELECT p.slug, p.source_id, p.deleted_at FROM pages p
+ WHERE p.id = ANY($1::int[]) AND p.id <> ALL($2::int[]) AND p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($3 || ' hours')::interval
+ AND ${identityDowngradePurgeHold('p')} AND ${atomRetirementPurgeHold('p')} AND ${derivedRetirementPurgeHold('p')}`,
+ [release, excludeIds, String(Math.max(0, Math.floor(olderThanHours)))])).map(row => ({ slug: row.slug, source_id: row.source_id, deleted_at: new Date(row.deleted_at) }))
+ .filter(row => !(preview.pages ?? []).some(page => page.source_id === row.source_id && page.slug === row.slug)) : [];
+ const pages = [...(preview.pages ?? []), ...released].sort((a, b) => a.deleted_at.getTime() - b.deleted_at.getTime() || (a.slug < b.slug ? -1 : a.slug > b.slug ? 1 : 0));
const held = kept.filter(row => row.expired);
- return { managed: false, count: preview.count, slugs: preview.slugs, pages: preview.pages, candidates: [],
+ return { managed: false, count: pages.length, slugs: pages.map(page => page.slug), pages, candidates: [], released_origins: released.map(({ source_id, slug }) => ({ source_id, slug })),
blocked: [...(opts.retryParked ? [] : parked.map(row => ({ source_id: row.source_id, slug: row.slug, code: 'purge_parked', reason: parkedReason(row.park) }))),
...held.map(row => syncDeleteHeld(row, olderThanHours))],
parked: parked.map(row => ({ source_id: row.source_id, slug: row.slug, failures: row.park.failures, code: row.park.code, retried: opts.retryParked === true })),
@@ -245,13 +267,13 @@ export async function selectPurgeCandidates(engine: BrainEngine, olderThanHours:
const rows = await engine.executeRaw<{ id: number | string; source_id: string; slug: string; cutoff: string; deleted_us: string }>(
`SELECT p.id, p.source_id, p.slug, (now() - ($1 || ' hours')::interval)::text AS cutoff,
floor(extract(epoch FROM p.deleted_at)*1000000)::bigint::text AS deleted_us FROM pages p JOIN sources s ON s.id = p.source_id
- WHERE p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($1 || ' hours')::interval AND ${purgeHoldsSql('p', 's')}
- ORDER BY p.deleted_at ASC, p.source_id ASC, p.slug ASC`, [String(hours)]);
+ WHERE p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($1 || ' hours')::interval AND ${purgeHoldsSql('p', 's', '$2::int[]')} AND p.id <> ALL($3::int[])
+ ORDER BY p.deleted_at ASC, p.source_id ASC, p.slug ASC`, [String(hours), release, hold]);
const states = new Map<string, { state: string; physical_error?: string }>();
if (rows.length) for (const source of (await writerOnboardingPreflight(engine)).sources) states.set(source.source_id, source);
const parks = rows.length ? await readPurgeParks(engine) : new Map<string, PurgePark>();
const unreviewed = new Set(rows.length ? (await readUnreviewedSyncTombstones(engine, hours)).map(row => row.id) : []);
- const selection: PurgeSelection = { managed: true, count: 0, slugs: [], candidates: [], blocked: [], parked: [], inactive: 0, held: 0, sync_unreviewed: 0, writers: {} };
+ const selection: PurgeSelection = { managed: true, count: 0, slugs: [], candidates: [], blocked: [], parked: [], inactive: 0, held: 0, sync_unreviewed: 0, writers: {}, released_origins: [] };
for (const { id, ...row } of rows) {
const writer = states.get(row.source_id);
const active = writer?.state !== 'recovery_required';
@@ -279,6 +301,7 @@ export async function selectPurgeCandidates(engine: BrainEngine, olderThanHours:
}
}
if (syncUnreviewed) selection.sync_unreviewed++;
+ if (release.includes(Number(id))) selection.released_origins.push({ source_id: row.source_id, slug: row.slug });
selection.candidates.push({ ...row, key, ...(park ? { park } : {}), ...(syncUnreviewed ? { sync_unreviewed: true as const } : {}) });
}
selection.count = selection.candidates.length;
@@ -334,11 +357,14 @@ export function purgePhaseNote(result: DeletedPagePurge): string {
/** Limits a purge to pages the caller already showed the person (`gbrain maintain --purge --expect`). */
export type PurgeFilter = (page: { source_id: string; slug: string }) => boolean;
-export async function purgeDeletedPagesCoordinated(engine: BrainEngine, olderThanHours: number, opts: { retryParked?: boolean; automatic?: boolean; only?: PurgeFilter } = {}): Promise<DeletedPagePurge> {
+export async function purgeDeletedPagesCoordinated(engine: BrainEngine, olderThanHours: number, opts: { retryParked?: boolean; automatic?: boolean; only?: PurgeFilter; release?: readonly number[] } = {}): Promise<DeletedPagePurge> {
+ const { release = [], ...purge } = opts;
const carried = await (await import('../repair/origin-carry.ts')).carryPrivateBeforePurge(engine, olderThanHours);
const origins = originParked(carried.parks);
+ // GST-2689: a consented manual purge releases now the parked origins its preview counted, once their carry landed.
+ await clearDerivedOriginParks(engine, carried.unparked.filter(origin => release.includes(origin.id)).map(origin => origin.id));
await faultPoint('purge:after_carry');
- const result = await purgeExpired(engine, olderThanHours, opts);
+ const result = await purgeExpired(engine, olderThanHours, purge);
const reparked = new Set(result.blocked.filter(b => b.code === DERIVED_ORIGIN_PARKED_CODE).map(b => `${b.source_id}/${b.slug}`));
await clearDerivedOriginParks(engine, carried.unparked.filter(origin => !reparked.has(`${origin.source_id}/${origin.slug}`)).map(origin => origin.id));
const parked = origins.length + (result.origin_parked ?? 0);
diff --git a/src/core/repair/origin-carry.ts b/src/core/repair/origin-carry.ts
index a6baac09f0..b6df9baa26 100644
--- a/src/core/repair/origin-carry.ts
+++ b/src/core/repair/origin-carry.ts
@@ -33,7 +33,7 @@ const EDIT: LabelEdit = { set: { visibility: 'private' }, remove: [] };
/** A page whose removal can open another: hidden by the privacy rules itself, or an atom a concept's provenance names. */
const originSql = (alias: string) => `(${alias}.type = 'atom' OR NOT ${privacyPartSql(alias)})`;
-interface Origin { id: number; source_id: string; slug: string }
+export interface Origin { id: number; source_id: string; slug: string }
interface Dependent { id: number; origin_id: number; source_id: string; slug: string; deleted: boolean }
interface Failure { code: string | null; reason: string }
export interface CarryBlock extends Failure { slug: string | null }
@@ -175,6 +175,16 @@ export async function deletePageCarryingPrivate(ctx: OperationContext, p: Record
/** Hours before its cutoff a tombstone is carried, so none crosses the cutoff while the carry runs and is purged uncarried. */
export const CARRY_LEAD_HOURS = 24;
+/** The soft-deleted origins expired or expiring within CARRY_LEAD_HOURS of this cutoff, the set every carry judges at once. */
+async function expiringOrigins(engine: BrainEngine, olderThanHours: number): Promise<Origin[]> {
+ return withoutCallerAccess(async () => {
+ await readHonorAliases(engine);
+ return (await engine.executeRaw<{ id: number; source_id: string; slug: string }>(`SELECT p.id, p.source_id, p.slug FROM pages p
+ WHERE p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($1 || ' hours')::interval AND ${originSql('p')} ORDER BY p.id`,
+ [String(Math.max(0, Math.floor(olderThanHours) - CARRY_LEAD_HOURS))], bounded)).map(row => ({ id: Number(row.id), source_id: row.source_id, slug: row.slug }));
+ });
+}
+
/**
* Before a purge: the soft-deleted origins expired or expiring within CARRY_LEAD_HOURS get their dependents labelled
* private, judged against all of them at once (a page private through two of them opens only when both go); each one
@@ -183,15 +193,9 @@ export const CARRY_LEAD_HOURS = 24;
*/
export async function carryPrivateBeforePurge(engine: BrainEngine, olderThanHours: number): Promise<{ parks: DerivedOriginPark[]; unparked: Origin[] }> {
await pruneDerivedOriginParks(engine);
- const origins = await withoutCallerAccess(async () => {
- await readHonorAliases(engine);
- return engine.executeRaw<{ id: number; source_id: string; slug: string }>(`SELECT p.id, p.source_id, p.slug FROM pages p
- WHERE p.deleted_at IS NOT NULL AND p.deleted_at < now() - ($1 || ' hours')::interval AND ${originSql('p')} ORDER BY p.id`,
- [String(Math.max(0, Math.floor(olderThanHours) - CARRY_LEAD_HOURS))], bounded);
- });
const parks: DerivedOriginPark[] = [];
const batch = `derived-origin-carry:purge-${randomUUID()}`;
- const expired = origins.map(row => ({ id: Number(row.id), source_id: row.source_id, slug: row.slug }));
+ const expired = await expiringOrigins(engine, olderThanHours);
const blocked = await carryPrivate(engine, expired, batch);
for (const origin of expired) {
const block = blocked.get(origin.id);
@@ -202,13 +206,34 @@ export async function carryPrivateBeforePurge(engine: BrainEngine, olderThanHour
return { parks, unparked: expired.filter(origin => !blocked.has(origin.id)) };
}
+/** What the carry of a consented manual purge would do, read ahead: the parked origins it frees and the others it parks. */
+export interface CarryForecast { release: Origin[]; hold: Origin[] }
+
/**
- * A consented manual purge first retries the label for every parked origin and unparks the ones it frees, so its preview
- * and consent list them (otherwise a brain whose only expired pages are parked origins never gets past "nothing to purge").
+ * Read-only, for a consented manual purge's preview and consent: what carryPrivateBeforePurge would do over the same
+ * expiring origins, judged as carryPrivate judges them. An origin is blocked when a page it alone keeps private is
+ * soft-deleted, stored in segments or unreadable; the label write is assumed to land on every other page. A parked origin
+ * not blocked is released (the run clears its park once its carry lands, purgeDeletedPagesCoordinated `release`), and an
+ * origin not parked but blocked is held. A failed read holds every origin, as a failed carry parks every origin.
*/
-export async function retryParkedOrigins(engine: BrainEngine, olderThanHours: number): Promise<void> {
- if (!(await readDerivedOriginParks(engine)).length) return;
- await clearDerivedOriginParks(engine, (await carryPrivateBeforePurge(engine, olderThanHours)).unparked.map(origin => origin.id));
+export async function forecastCarry(engine: BrainEngine, olderThanHours: number): Promise<CarryForecast> {
+ const parks = new Set((await readDerivedOriginParks(engine)).map(park => park.page_id));
+ return withoutCallerAccess(async () => {
+ let origins: Origin[] = [];
+ try {
+ origins = await expiringOrigins(engine, olderThanHours);
+ const blocked = new Set<number>();
+ if (origins.length) for (const page of await dependentsOf(engine, origins)) {
+ if (blocked.has(page.origin_id)) continue;
+ const snapshot = page.deleted ? null : await engine.readPageSnapshot(page.slug, { sourceId: page.source_id });
+ if (!snapshot || Number(snapshot.page.id) !== page.id || isSegmentedPage(snapshot.page)) blocked.add(page.origin_id);
+ }
+ return { release: origins.filter(origin => parks.has(origin.id) && !blocked.has(origin.id)),
+ hold: origins.filter(origin => !parks.has(origin.id) && blocked.has(origin.id)) };
+ } catch {
+ return { release: [], hold: origins.filter(origin => !parks.has(origin.id)) };
+ }
+ });
}
/** The receipt note of an explicit purge held because its origin lookup failed; it names no page and counts none. */
diff --git a/test/derived-origin-carry-managed.test.ts b/test/derived-origin-carry-managed.test.ts
index 806d68ef9e..db4828c3de 100644
--- a/test/derived-origin-carry-managed.test.ts
+++ b/test/derived-origin-carry-managed.test.ts
@@ -27,6 +27,9 @@ import { runPages } from '../src/commands/pages.ts';
import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
import { installFaultHook } from '../src/core/persistence/fault-points.ts';
import { purgeDeletedPagesCoordinated } from '../src/core/persistence/purge-deleted.ts';
+import * as preparedMaintenance from '../src/core/persistence/prepared-maintenance.ts';
+import { readDerivedOriginParks } from '../src/core/persistence/derived-origin-hold.ts';
+import { _resetCliExitVerdictForTests, currentExitCode } from '../src/core/cli-force-exit.ts';
import { importFromContent } from '../src/core/import-file.ts';
import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';
import { sqlQueryForEngine } from '../src/core/sql-query.ts';
@@ -62,7 +65,7 @@ function fileText(k: string, shape: Shape): string {
};
return ['---', `title: ${k} ${shape}`, ...fm[shape], '---', '', `Harbor ledger planning notes ${canaryOf(k, shape)} for the roadmap.`, ''].join('\n');
}
-const KEYS = ['gitrm', 'purge', 'remote', 'midpurge'];
+const KEYS = ['gitrm', 'purge', 'remote', 'midpurge', 'preview'];
/** A private origin, a public meeting, an event derived from the origin whose depth is the meeting, and an event whose depth is the origin. */
const EVENT_SLUGS = { origin: 'notes/event-origin', depth: 'notes/event-depth', event: 'notes/event-event', closed: 'notes/event-closed' };
const EVENT_FILES: Record<keyof typeof EVENT_SLUGS, string[]> = {
@@ -100,6 +103,21 @@ async function cli(engine: BrainEngine, args: string[]): Promise<void> {
try { await withEnv({ GBRAIN_CONSENT: undefined }, () => runPages(engine, args)); }
finally { log.mockRestore(); err.mockRestore(); }
}
+/** `gbrain pages <args>` as the CLI runs it: its JSON output and exit verdict. */
+async function cliJson(engine: BrainEngine, args: string[]): Promise<{ out: Record<string, any> | null; exit: number }> {
+ _resetCliExitVerdictForTests();
+ const prior = process.exitCode;
+ const lines: string[] = [];
+ const log = spyOn(console, 'log').mockImplementation((...parts: unknown[]) => { lines.push(parts.join(' ')); });
+ const err = spyOn(console, 'error').mockImplementation(() => {});
+ try { await withEnv({ GBRAIN_CONSENT: undefined }, () => runPages(engine, args)); }
+ finally { log.mockRestore(); err.mockRestore(); }
+ const exit = currentExitCode();
+ _resetCliExitVerdictForTests();
+ process.exitCode = prior;
+ const text = lines.join('\n').trim();
+ return { out: text.startsWith('{') ? JSON.parse(text) : null, exit };
+}
const page = async (engine: BrainEngine, slug: string) => (await engine.executeRaw<{ deleted_at: string | null; frontmatter: unknown }>(
"SELECT deleted_at, frontmatter FROM pages WHERE source_id='default' AND slug=$1", [slug]))[0];
const fmOf = async (engine: BrainEngine, slug: string) => { const fm = (await page(engine, slug))!.frontmatter; return (typeof fm === 'string' ? JSON.parse(fm) : fm) as Record<string, unknown>; };
@@ -162,6 +180,36 @@ for (const backend of testBackends()) {
await expectCarried(engine, root, k, files);
}, { setup, databaseUrl }), 300_000);
+ test('purge-deleted --dry-run lists the parked origin the consented run releases and labels nothing; a refused consent labels nothing; --yes purges exactly that list', () => managedBrain(async ({ engine, root }) => {
+ await performManagedSync(engine, OPTS);
+ const k = 'preview', origin = slugOf(k, 'origin'), digest = slugOf(k, 'digest');
+ const files = await before(engine, root, k);
+ git(root, 'rm', '-q', `${origin}.md`); commit(root, 'remove the origin');
+ await performManagedSync(engine, OPTS);
+ await age(engine, 100);
+ const failing = spyOn(preparedMaintenance, 'publishMaintenancePages').mockImplementation(async () => { throw new Error('forced label failure'); });
+ try { expect((await purgeDeletedPagesCoordinated(engine, 72, { automatic: true })).slugs).not.toContain(origin); }
+ finally { failing.mockRestore(); }
+ const untouched = async () => {
+ await settleManagedGitEffects(engine);
+ expect(readFileSync(join(root, `${digest}.md`), 'utf8')).toBe(files.digest!);
+ expect((await fmOf(engine, digest)).visibility).toBeUndefined();
+ expect((await readDerivedOriginParks(engine)).map(park => park.slug)).toContain(origin);
+ expect((await page(engine, origin))?.deleted_at).not.toBeNull();
+ };
+ await untouched();
+ const preview = await cliJson(engine, ['purge-deleted', '--dry-run', '--json']);
+ expect(preview.exit).toBe(0);
+ expect(preview.out!.slugs).toContain(origin);
+ await untouched();
+ expect((await cliJson(engine, ['purge-deleted', '--json'])).exit).toBe(3);
+ await untouched();
+ const run = await cliJson(engine, ['purge-deleted', '--json', '--yes']);
+ expect(run.exit).toBe(0);
+ expect([...run.out!.slugs].sort()).toEqual([...preview.out!.slugs].sort());
+ await expectCarried(engine, root, k, files);
+ }, { setup, databaseUrl }), 300_000);
+
test('the managed retention purge rechecks inside its transaction: a derived page written after the carry parks the origin and skips its row', () => managedBrain(async ({ engine, root }) => {
await performManagedSync(engine, OPTS);
const k = 'midpurge', origin = slugOf(k, 'origin'), late = `notes/${k}-late`;
diff --git a/test/derived-origin-carry-private.test.ts b/test/derived-origin-carry-private.test.ts
index 137bce55e4..f705edf522 100644
--- a/test/derived-origin-carry-private.test.ts
+++ b/test/derived-origin-carry-private.test.ts
@@ -35,6 +35,9 @@ import { setOwner } from '../src/core/access/owner.ts';
import { bindCredential } from '../src/core/access/members.ts';
import { consentedPurgeDeletedPages, purgeDeletedPagesCoordinated, selectPurgeCandidates } from '../src/core/persistence/purge-deleted.ts';
import { checkDerivedOriginParked } from '../src/commands/doctor/checks/purge-parks.ts';
+import { runPages } from '../src/commands/pages.ts';
+import { _resetCliExitVerdictForTests, currentExitCode } from '../src/core/cli-force-exit.ts';
+import { readDerivedOriginParks } from '../src/core/persistence/derived-origin-hold.ts';
import { installFaultHook } from '../src/core/persistence/fault-points.ts';
import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
import * as labelWrite from '../src/core/repair/label-write.ts';
@@ -394,6 +397,54 @@ for (const kind of testBackends()) {
}
}
+ /** `gbrain pages purge-deleted <args>` as the CLI runs it: its JSON output and exit verdict. */
+ async function pagesCli(args: string[]): Promise<{ out: Record<string, any> | null; exit: number }> {
+ _resetCliExitVerdictForTests();
+ const prior = process.exitCode;
+ const lines: string[] = [];
+ const log = spyOn(console, 'log').mockImplementation((...parts: unknown[]) => { lines.push(parts.join(' ')); });
+ const err = spyOn(console, 'error').mockImplementation(() => {});
+ try { await withEnv({ GBRAIN_CONSENT: undefined, GBRAIN_HOME: home }, () => runPages(engine, ['purge-deleted', ...args])); }
+ finally { log.mockRestore(); err.mockRestore(); }
+ const exit = currentExitCode();
+ _resetCliExitVerdictForTests();
+ process.exitCode = prior;
+ const text = lines.join('\n').trim();
+ return { out: text.startsWith('{') ? JSON.parse(text) : null, exit };
+ }
+
+ test('purge-deleted --dry-run lists the parked origin a consented run releases and writes nothing; refused consent writes nothing; --yes purges exactly that list', async () => {
+ const k = 'preview';
+ await seed(k);
+ const origin = slugOf(k, 'origin'), blocking = slugOf(k, 'digest'), plain = slugOf(k, 'plain');
+ await parkByFailedLabel(origin);
+ await softDelete(plain);
+ await backdate(origin);
+ await backdate(plain);
+ const untouched = async () => {
+ expect((await fm(blocking)).visibility).toBeUndefined();
+ expect((await readDerivedOriginParks(engine)).map(park => park.slug)).toContain(origin);
+ expect((await row(origin))?.deleted_at).not.toBeNull();
+ expect((await row(plain))?.deleted_at).not.toBeNull();
+ };
+ const preview = await pagesCli(['--dry-run', '--json']);
+ expect(preview.exit).toBe(0);
+ expect(preview.out!.slugs).toEqual(expect.arrayContaining([origin, plain]));
+ await untouched();
+ const refused = await pagesCli(['--json']);
+ expect(refused.exit).toBe(3);
+ await untouched();
+ await expect(consentedPurgeDeletedPages(engine, 72, false)).rejects.toThrow();
+ await untouched();
+ const run = await pagesCli(['--json', '--yes']);
+ expect(run.exit).toBe(0);
+ expect([...run.out!.slugs].sort()).toEqual([...preview.out!.slugs].sort());
+ expect((await fm(blocking)).visibility).toBe('private');
+ await nothingKept(origin);
+ expect(await row(plain)).toBeUndefined();
+ await expectCarriedHidden(k);
+ }, 240_000);
+
test('an explicit purge rechecks inside its transaction: a derived page written after the carry keeps the origin soft-deleted and parked', async () => {
const k = 'midsingle';
await seed(k);