Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: visibility guard also reads the parsed value (block scalar, next-line value, list); cover the extracted-session and sole-source cells
garrytan9346e1a3443a
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex e2233699cb..05e410a36d 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -132,7 +132,7 @@ a receipt can report has a row here.| Reason | Error code | What it means | Recovery || --- | --- | --- | --- || `writer_pool_capacity` | `writer_pool_capacity` | The canonical owner has no free publication slot right now; the write stays queued. | Wait and poll the receipt (`gbrain write-request <id>`). If it persists, check `gbrain sources writer status <source>`. |-| <a id="visibility_not_applied"></a>`noncanonical_value`, `not_kept` | `visibility_not_applied` | A page write's frontmatter asked for `visibility: private`, but the page would have stayed world-readable, so nothing was written and no no-op was reported (#7841). `noncanonical_value`: the key or value is in another case (`Visibility: private`, `visibility: Private`), which page privacy does not read as private. `not_kept`: the request did not survive into the page that would be stored. The check only refuses; it never widens a page or changes what a remote caller may read or write. | Read the page and its revision (`gbrain get --source <id> -- <slug>`), write `visibility: private` on its own frontmatter line, lowercase, and submit `put_page` again with a new `request_id`. That is the supported way to make an existing page private, on a git-canonical source too (the page publishes database-only, see [`git_private_commit_refused`](#git_private_commit_refused)). |+| <a id="visibility_not_applied"></a>`noncanonical_value`, `not_kept` | `visibility_not_applied` | A page write's frontmatter asked for `visibility: private`, but the page would have stayed world-readable, so nothing was written and no no-op was reported (#7841). `noncanonical_value`: the key or value is in another case (`Visibility: private`, `visibility: Private`) or another shape (a value on the next line, a block scalar, a list), which page privacy does not read as private. `not_kept`: the request did not survive into the page that would be stored. The check only refuses; it never widens a page or changes what a remote caller may read or write. | Read the page and its revision (`gbrain get --source <id> -- <slug>`), write `visibility: private` on its own frontmatter line, lowercase, and submit `put_page` again with a new `request_id`. That is the supported way to make an existing page private, on a git-canonical source too (the page publishes database-only, see [`git_private_commit_refused`](#git_private_commit_refused)). || `revision_required` | `revision_required` | The operation must be bound to a revision you reviewed (for example `--if-version` or `expected_revision`), and none was given: a `put_page` of a page that already exists without `expected_revision` reports this since #5385 (earlier releases said `revision_conflict`). | Preview first, then repeat with the revision the preview printed. || `revision_conflict` | `revision_conflict` | The page changed after the revision this write was bound to. Nothing was overwritten. | Re-read the page, merge your change, and save with the new revision. || `revision_changed` | `revision_changed` | An `append_page` with `expected_revision`, or a `commit_page_upload` whose session was bound to a revision, found the page at a newer revision. Nothing was appended or written, and the upload session is aborted with its staged bytes released. | Read the page again (`get_page` with `body: 'none'`), check whether the text is still missing, then append against the new revision with a new `request_id` (for an upload, begin a new session). |diff --git a/scripts/function-size-baseline.tsv b/scripts/function-size-baseline.tsvindex 0cd3f5054d..cd70f3608d 100644--- a/scripts/function-size-baseline.tsv+++ b/scripts/function-size-baseline.tsv@@ -48,7 +48,7 @@ src/core/minions/queue.ts MinionQueue.add>result=this.engine.transaction(#0) 359src/core/minions/worker.ts MinionWorker.start 470 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)src/core/ops/facts.ts recall.handler 357 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +3 #4830 (W14 PR3): every hit names its source_id, falling back to the scope's single source (the federated case stays absent); +3 W5: recall threads the page-unit ranking cut into its hybrid page arm (logic in search/page-rank-cut.ts; delivery reports it); +3 #7043: copies of one claim collapse before the limit cut (the collapse lives in facts/capture-dedup.ts collapseFactReads) and the row names its entity_slugs/collapsed_countsrc/core/ops/search.ts query.handler 338 +5 GBRA-81 item 5 per-stage search timings: the request's SearchTimings and setup-stage binding, the image branch's limit read hoisted out of the timed vector call, and its response stage wrapper (collector logic in search/stage-timings.ts); seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +2 GBRA-49 explain_target: one TargetTrace prep line and its HybridSearchOpts thread (diagnosis/meta live in prepareExplainTarget/finishExplainTarget); +1 W5 (#6445): query passes the page-unit ranking cut to hybrid search (logic in search/page-rank-cut.ts), merged with master growth-src/core/persistence/page-prepare.ts prepareInlineOrSegmentedMutation 302 +2 #7841: a put_page that asks for visibility: private is kept or refused (visibility_not_applied), checked before the no-op test and after the trust stamp (logic in persistence/visibility-request.ts)+src/core/persistence/page-prepare.ts prepareInlineOrSegmentedMutation 303 +3 #7841: a put_page that asks for visibility: private is kept or refused (visibility_not_applied), checked before the no-op test and after the trust stamp (logic in persistence/visibility-request.ts)src/core/progressive-batch/orchestrator.ts runProgressiveBatch 348 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)src/core/remediation/run.ts runRemediation 428 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)src/core/skillopt/orchestrator.ts runOptimizationLoop 704 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)diff --git a/src/core/persistence/page-prepare.ts b/src/core/persistence/page-prepare.tsindex e57b1db5d6..41a5cbeb68 100644--- a/src/core/persistence/page-prepare.ts+++ b/src/core/persistence/page-prepare.ts@@ -67,7 +67,7 @@ import { DEFAULT_WRITE_GATE_CONFIG, writeGateApplies } from '../write-gate.ts';import { materializeTimeline, prepareCanonicalProjections, type TimelineRowsRemoved } from './canonical-projections.ts';import { assertTimelineNotOmitted, timelineWritePolicy } from './timeline-omission.ts';import { preserveProtectedTakes } from './protected-takes.ts';-import { assertPrivateRequestKept, declaredVisibilities } from './visibility-request.ts';+import { assertPrivateRequestKept, readPrivateRequest } from './visibility-request.ts';import { isAutoLinkEnabled } from '../link-extraction.ts';import { prepareAutomaticLinks } from './links-preparation.ts';import { loadActivePackForEngine } from '../schema-pack/engine-resolution.ts';@@ -833,7 +833,8 @@ async function prepareInlineOrSegmentedMutation(unbounded: BrainEngine, row: Wriconst { timeline, materialized } = await materializeTimeline(engine,parsed,row.slug,snapshot,writer,timelinePolicy);if (materialized) content = serializePageToMarkdown({...snapshot.page,...parsed,timeline,type:parsed.typeExplicit ? parsed.type : snapshot.page.type},parsed.tags);}- if (typeof content === 'string' && declaredVisibilities(content).includes('private')) assertPrivateRequestKept(row, content, parseMarkdown(content, row.slug, { activePack }).frontmatter);+ const privateRequest = typeof content === 'string' ? readPrivateRequest(content, () => parseMarkdown(content as string, row.slug, { activePack }).frontmatter) : null;+ if (privateRequest) assertPrivateRequestKept(row, privateRequest.frontmatter);const storedTier = snapshot ? await storedPageTier(engine, row.source_id, row.slug) : null;// Detect an exact canonical no-op before ingestion can invoke any provider.// Revision/identity checks above still apply to stale identical replacements.@@ -875,7 +876,7 @@ async function prepareInlineOrSegmentedMutation(unbounded: BrainEngine, row: Wriprovenance = putProvenance(row, snapshot, page);suppliedTrustMarker = page.frontmatter.trust_tier;stampPageTrustMarker(row, page.frontmatter, trust, storedTier);- if (typeof content === 'string') assertPrivateRequestKept(row, content, page.frontmatter);+ if (privateRequest) assertPrivateRequestKept(row, page.frontmatter);},prepare: async value => { prepared = value; return value.result; },}).catch(error => {diff --git a/src/core/persistence/visibility-request.ts b/src/core/persistence/visibility-request.tsindex 616c037c03..1a68d8d9c7 100644--- a/src/core/persistence/visibility-request.ts+++ b/src/core/persistence/visibility-request.ts@@ -4,12 +4,13 @@* comes back as a no-op, with the page still world-readable.** The request is read from the content's leading frontmatter block line by line,- * independently of the YAML parse, so a request the parse does not keep is still- * seen: a key or value in another case (`Visibility: private`,- * `visibility: Private`, which page privacy does not read as private), or a line- * the parse drops. Only a request for `private` is checked, so the guard can only- * refuse: it never widens a page and never changes what a remote caller may read- * or write. A request that is kept passes unchanged.+ * and from the parse of a visibility line, so a request the stored page would not+ * keep is still seen: a key or value in another case (`Visibility: private`,+ * `visibility: Private`, which page privacy does not read as private), a value+ * the parse reshapes (a block scalar, a list) or a line the parse drops. Only a+ * request for `private` is checked, so the guard can only refuse: it never widens+ * a page and never changes what a remote caller may read or write. A request that+ * is kept passes unchanged.*/import { opError, type OperationError } from '../ops/contract.ts';import { readFix } from '../ops/op-fix.ts';@@ -30,11 +31,31 @@ export function declaredVisibilities(content: string): string[] {return [];}-/** Refuses a write whose content asks for `visibility: private` when `published` (the frontmatter it would store) is not private. */-export function assertPrivateRequestKept(row: Pick<WriteRequest, 'slug' | 'source_id'>, content: string, published: Record<string, unknown> | null | undefined): void {- if (!declaredVisibilities(content).includes('private') || published?.visibility === 'private') return;- throw visibilityNotApplied(row, typeof published?.visibility === 'string' && published.visibility.trim().toLowerCase() === 'private'- || Object.keys(published ?? {}).some(key => key !== 'visibility' && key.toLowerCase() === 'visibility') ? 'noncanonical_value' : 'not_kept');+/** Every string the frontmatter's `visibility` key (any case) holds, trimmed and lowercased; a list contributes its string items. */+function visibilityValues(frontmatter: Record<string, unknown> | null | undefined): string[] {+ const flatten = (value: unknown): string[] => typeof value === 'string' ? [value.trim().toLowerCase()] : Array.isArray(value) ? value.flatMap(flatten) : [];+ return Object.entries(frontmatter ?? {}).flatMap(([key, value]) => key.toLowerCase() === 'visibility' ? flatten(value) : []);+}++/**+ * The private request `content` carries, with the frontmatter it parses to, or null.+ * A visibility line asks for private when its own text says so or when the parse+ * reads it as private (a value on the next line, a block scalar, a flow list), so a+ * spelling the line scan cannot read is still seen. Parsing waits for a visibility key.+ */+export function readPrivateRequest(content: string, parse: () => Record<string, unknown> | null | undefined): { frontmatter: Record<string, unknown> | null | undefined } | null {+ const declared = declaredVisibilities(content);+ if (!declared.length) return null;+ const inline = declared.includes('private');+ let frontmatter: Record<string, unknown> | null | undefined;+ try { frontmatter = parse(); } catch (error) { if (inline) throw error; return null; }+ return inline || visibilityValues(frontmatter).includes('private') ? { frontmatter } : null;+}++/** Refuses a write that asked for `visibility: private` when `published` (the frontmatter it would store) does not carry exactly that. */+export function assertPrivateRequestKept(row: Pick<WriteRequest, 'slug' | 'source_id'>, published: Record<string, unknown> | null | undefined): void {+ if (published?.visibility === 'private') return;+ throw visibilityNotApplied(row, visibilityValues(published).includes('private') ? 'noncanonical_value' : 'not_kept');}function visibilityNotApplied(row: Pick<WriteRequest, 'slug' | 'source_id'>, reason: 'noncanonical_value' | 'not_kept'): OperationError {diff --git a/test/put-visibility-private-7841.test.ts b/test/put-visibility-private-7841.test.tsindex 3fe383a282..59705d82c1 100644--- a/test/put-visibility-private-7841.test.ts+++ b/test/put-visibility-private-7841.test.ts@@ -96,6 +96,10 @@ test('declaredVisibilities reads the frontmatter block only, any key case, lowerexpect(declaredVisibilities(page('summary: >\n visibility: private\n'))).toEqual([]);expect(declaredVisibilities('A body.\nvisibility: private\n')).toEqual([]);expect(declaredVisibilities('---\nvisibility: private\n')).toEqual([]);+ expect(declaredVisibilities('\uFEFF\n\n---\r\nvisibility: private\r\n---\r\nA body.\r\n')).toEqual(['private']);+ expect(declaredVisibilities('---\nvisibility: private\n...\nA body.\n')).toEqual(['private']);+ expect(declaredVisibilities(page('"visibility": private # keep\n'))).toEqual(['private']);+ expect(declaredVisibilities(page('visibility: private\nvisibility: public\n'))).toEqual(['private', 'public']);});for (const managed of [false, true]) describe(`${managed ? 'managed brain with a Git checkout' : 'repo-less brain'}: put_page visibility: private (#7841)`, () => {@@ -113,7 +117,8 @@ for (const managed of [false, true]) describe(`${managed ? 'managed brain with a});}, 120_000);- for (const [label, extra] of [['visibility: Private', 'visibility: Private\n'], ['Visibility: private', 'Visibility: private\n'], ['VISIBILITY: "PRIVATE"', 'VISIBILITY: "PRIVATE"\n']] as const) {+ for (const [label, extra] of [['visibility: Private', 'visibility: Private\n'], ['Visibility: private', 'Visibility: private\n'], ['VISIBILITY: "PRIVATE"', 'VISIBILITY: "PRIVATE"\n'],+ ['visibility with Private on the next line', 'visibility:\n Private\n'], ['visibility as a block scalar', 'visibility: |\n private\n'], ['visibility as a flow list', 'visibility: [private]\n']] as const) {test(`${label} on an existing world page is refused noncanonical_value, and the page and its revision are unchanged`, async () => {await brain(managed, async (ctx, engine) => {await put(ctx, 'notes/example', page());@@ -128,6 +133,27 @@ for (const managed of [false, true]) describe(`${managed ? 'managed brain with a}, 120_000);}+ test('a quoted key with a trailing comment still publishes private, with CRLF line endings too', async () => {+ await brain(managed, async (ctx, engine) => {+ for (const [slug, content] of [['notes/quoted', page('"visibility": private # keep\n')], ['notes/crlf', page('visibility: private\n').replace(/\n/g, '\r\n')]] as const) {+ expect((await put(ctx, slug, content)).thrown).toBeUndefined();+ expect(isPrivatePage((await snapshot(engine, ctx, slug))!.page)).toBe(true);+ }+ });+ }, 120_000);++ test('a remote caller is refused the same way, and its plain private request still publishes private', async () => {+ await brain(managed, async (ctx, engine) => {+ const remote = { ...ctx, remote: true };+ await put(ctx, 'notes/example', page());+ const before = (await snapshot(engine, ctx, 'notes/example'))!;+ expect(await put(remote, 'notes/example', page('visibility: Private\n'), before.revision)).toMatchObject({ thrown: 'visibility_not_applied' });+ expect((await snapshot(engine, ctx, 'notes/example'))!.revision).toBe(before.revision);+ expect((await put(remote, 'notes/remote-example', page('visibility: private\n'))).thrown).toBeUndefined();+ expect(isPrivatePage((await snapshot(engine, ctx, 'notes/remote-example'))!.page)).toBe(true);+ });+ }, 120_000);+test('a new page created with visibility: Private is refused, never stored world-readable', async () => {await brain(managed, async (ctx, engine) => {expect(await put(ctx, 'notes/new-example', page('visibility: Private\n'))).toMatchObject({ thrown: 'visibility_not_applied' });diff --git a/test/synthesize-session-source-7805.test.ts b/test/synthesize-session-source-7805.test.tsindex 2896875a90..3bef42b5d7 100644--- a/test/synthesize-session-source-7805.test.ts+++ b/test/synthesize-session-source-7805.test.ts@@ -15,7 +15,7 @@ import { afterAll, beforeAll, describe, expect, test } from 'bun:test';import { randomUUID } from 'node:crypto';import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';-import { basename, join } from 'node:path';+import { basename, dirname, join } from 'node:path';import type { BrainEngine } from '../src/core/engine.ts';import { PGLiteEngine } from '../src/core/pglite-engine.ts';import { runPhaseSynthesize } from '../src/core/cycle/synthesize.ts';@@ -62,23 +62,28 @@ const usage = { input_tokens: 100, output_tokens: 100, cache_read_tokens: 0, cacinterface Fixture { engine: BrainEngine; cycle: { id: string; root: string }; other: { id: string; root: string }; corpus: string }-async function fixture(managed: boolean, run: (f: Fixture) => Promise<void>) {+async function fixture(managed: boolean, run: (f: Fixture) => Promise<void>, solo = false) {for (const engine of engines) {const dir = mkdtempSync(join(tmpdir(), 'gbrain-synth-source-'));const corpus = join(dir, 'corpus');const tag = randomUUID().slice(0, 8);- const cycle = { id: `source-a-${tag}`, root: join(dir, 'brain-a') };+ const cycle = { id: solo ? 'default' : `source-a-${tag}`, root: join(dir, 'brain-a') };const other = { id: `source-b-${tag}`, root: join(dir, 'brain-b') };mkdirSync(corpus);try {await withEnv({ GBRAIN_HOME: join(dir, 'home'), ANTHROPIC_API_KEY: 'sk-test-synth-source', GBRAIN_SOURCE: undefined }, async () => {await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');- for (const s of [cycle, other]) {+ const sources = solo ? [cycle] : [cycle, other];+ for (const s of sources) {mkdirSync(s.root);- await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [s.id, s.root]);+ if (solo) {+ await engine.executeRaw("DELETE FROM sources WHERE id <> 'default'");+ await engine.executeRaw('UPDATE sources SET local_path=$2 WHERE id=$1', [s.id, s.root]);+ }+ else await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [s.id, s.root]);}await engine.setConfig('sync.write_through', 'true');- for (const s of [cycle, other]) await claimWorktree(engine, s.id, s.root);+ if (!solo) for (const s of sources) await claimWorktree(engine, s.id, s.root);for (const [key, value] of Object.entries({'dream.synthesize.enabled': 'true', 'dream.synthesize.cooldown_hours': '0','dream.synthesize.min_chars': '100', 'dream.synthesize.link_manifest': 'false',@@ -182,4 +187,51 @@ for (const managed of [false, true]) describe(`${managed ? 'managed' : 'unmanageexpect(await synthesizedSources(engine, { cycle, other })).toEqual({ 'sess-extracted.txt': [cycle.id] });});}, 120_000);++ test('an --input file outside the corpus is kept for the cycle source, in no routing bucket', async () => {+ await fixture(managed, async ({ engine, cycle, other, corpus }) => {+ const outside = join(dirname(corpus), 'meeting-notes.txt');+ writeTranscript(outside, 'the quarterly offsite');+ const result = await runPhaseSynthesize(engine, { brainDir: cycle.root, sourceId: cycle.id, dryRun: false, inputFile: outside });+ expect(result.status).toBe('ok');+ expect(result.details.source_routing).toBeUndefined();+ expect(await synthesizedSources(engine, { cycle, other })).toEqual({ 'meeting-notes.txt': [cycle.id] });+ });+ }, 120_000);+});++test('a sole-source brain (unmanaged) still synthesizes a legacy top-level file (#7805)', async () => {+ await fixture(false, async ({ engine, cycle, other, corpus }) => {+ writeTranscript(join(corpus, 'sess-solo.txt'), 'the office move');+ const result = await runPhaseSynthesize(engine, { brainDir: cycle.root, sourceId: cycle.id, dryRun: false });+ expect(result.status).toBe('ok');+ expect(result.details.source_routing).toBeUndefined();+ expect(await synthesizedSources(engine, { cycle, other })).toEqual({ 'sess-solo.txt': ['default'] });+ }, true);+}, 120_000);++describe('assign-corpus listing includes sessions the sweep already extracted (#7805)', () => {+ const extracted = (dir: string, name: string) => { writeTranscript(join(dir, name), 'the office move'); writeFileSync(join(dir, name + '.ingested'), '{}\n'); };++ test('only a held extracted session is listed: not a stamped or frozen one, not a sole-source legacy file, not a file outside the corpus', () => {+ const corpus = mkdtempSync(join(tmpdir(), 'gbrain-assign-corpus-'));+ try {+ const spool = ensureCorpusSpoolDir(corpus);+ extracted(spool, sessionCorpusFileName('sess-held', CORPUS_UNRESOLVED_STAMP));+ extracted(spool, sessionCorpusFileName('sess-stamped', 'source-a'));+ extracted(spool, sessionCorpusFileName('sess-frozen', CORPUS_UNRESOLVED_STAMP));+ expect(freezeSessionSource(corpus, 'sess-frozen', 'source-b', 'operator')?.source_id).toBe('source-b');+ extracted(corpus, 'sess-legacy.txt');+ const outside = join(dirname(corpus), `${basename(corpus)}-elsewhere`);+ mkdirSync(outside);+ extracted(outside, 'sess-elsewhere.txt');++ expect(listCorpusSourceHolds(corpus, false).sessions).toEqual([]);+ expect(listCorpusSourceHolds(corpus, false, { includeIngested: true })).toMatchObject({ unresolved: 1, legacy_unstamped: 1, sessions: ['sess-held', 'sess-legacy'] });+ expect(listCorpusSourceHolds(corpus, true, { includeIngested: true }).sessions).toEqual(['sess-held']);+ expect(assignCorpusSessions(corpus, ['sess-held', 'sess-frozen'], 'source-a')).toEqual(['sess-held']);+ expect(freezeSessionSource(corpus, 'sess-frozen', 'source-a', 'operator')?.source_id).toBe('source-b');+ rmSync(outside, { recursive: true, force: true });+ } finally { rmSync(corpus, { recursive: true, force: true }); }+ });});