Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: visibility guard also reads the parsed value (block scalar, next-line value, list); cover the extracted-session and sole-source cells

garrytan9346e1a3443a
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index e2233699cb..05e410a36d 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -132,7 +132,7 @@ a receipt can report has a row here.
| Reason | Error code | What it means | Recovery |
| --- | --- | --- | --- |
| `writer_pool_capacity` | `writer_pool_capacity` | The canonical owner has no free publication slot right now; the write stays queued. | Wait and poll the receipt (`gbrain write-request <id>`). If it persists, check `gbrain sources writer status <source>`. |
-| <a id="visibility_not_applied"></a>`noncanonical_value`, `not_kept` | `visibility_not_applied` | A page write's frontmatter asked for `visibility: private`, but the page would have stayed world-readable, so nothing was written and no no-op was reported (#7841). `noncanonical_value`: the key or value is in another case (`Visibility: private`, `visibility: Private`), which page privacy does not read as private. `not_kept`: the request did not survive into the page that would be stored. The check only refuses; it never widens a page or changes what a remote caller may read or write. | Read the page and its revision (`gbrain get --source <id> -- <slug>`), write `visibility: private` on its own frontmatter line, lowercase, and submit `put_page` again with a new `request_id`. That is the supported way to make an existing page private, on a git-canonical source too (the page publishes database-only, see [`git_private_commit_refused`](#git_private_commit_refused)). |
+| <a id="visibility_not_applied"></a>`noncanonical_value`, `not_kept` | `visibility_not_applied` | A page write's frontmatter asked for `visibility: private`, but the page would have stayed world-readable, so nothing was written and no no-op was reported (#7841). `noncanonical_value`: the key or value is in another case (`Visibility: private`, `visibility: Private`) or another shape (a value on the next line, a block scalar, a list), which page privacy does not read as private. `not_kept`: the request did not survive into the page that would be stored. The check only refuses; it never widens a page or changes what a remote caller may read or write. | Read the page and its revision (`gbrain get --source <id> -- <slug>`), write `visibility: private` on its own frontmatter line, lowercase, and submit `put_page` again with a new `request_id`. That is the supported way to make an existing page private, on a git-canonical source too (the page publishes database-only, see [`git_private_commit_refused`](#git_private_commit_refused)). |
| `revision_required` | `revision_required` | The operation must be bound to a revision you reviewed (for example `--if-version` or `expected_revision`), and none was given: a `put_page` of a page that already exists without `expected_revision` reports this since #5385 (earlier releases said `revision_conflict`). | Preview first, then repeat with the revision the preview printed. |
| `revision_conflict` | `revision_conflict` | The page changed after the revision this write was bound to. Nothing was overwritten. | Re-read the page, merge your change, and save with the new revision. |
| `revision_changed` | `revision_changed` | An `append_page` with `expected_revision`, or a `commit_page_upload` whose session was bound to a revision, found the page at a newer revision. Nothing was appended or written, and the upload session is aborted with its staged bytes released. | Read the page again (`get_page` with `body: 'none'`), check whether the text is still missing, then append against the new revision with a new `request_id` (for an upload, begin a new session). |
diff --git a/scripts/function-size-baseline.tsv b/scripts/function-size-baseline.tsv
index 0cd3f5054d..cd70f3608d 100644
--- a/scripts/function-size-baseline.tsv
+++ b/scripts/function-size-baseline.tsv
@@ -48,7 +48,7 @@ src/core/minions/queue.ts MinionQueue.add>result=this.engine.transaction(#0) 359
src/core/minions/worker.ts MinionWorker.start 470 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)
src/core/ops/facts.ts recall.handler 357 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +3 #4830 (W14 PR3): every hit names its source_id, falling back to the scope's single source (the federated case stays absent); +3 W5: recall threads the page-unit ranking cut into its hybrid page arm (logic in search/page-rank-cut.ts; delivery reports it); +3 #7043: copies of one claim collapse before the limit cut (the collapse lives in facts/capture-dedup.ts collapseFactReads) and the row names its entity_slugs/collapsed_count
src/core/ops/search.ts query.handler 338 +5 GBRA-81 item 5 per-stage search timings: the request's SearchTimings and setup-stage binding, the image branch's limit read hoisted out of the timed vector call, and its response stage wrapper (collector logic in search/stage-timings.ts); seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +2 GBRA-49 explain_target: one TargetTrace prep line and its HybridSearchOpts thread (diagnosis/meta live in prepareExplainTarget/finishExplainTarget); +1 W5 (#6445): query passes the page-unit ranking cut to hybrid search (logic in search/page-rank-cut.ts), merged with master growth
-src/core/persistence/page-prepare.ts prepareInlineOrSegmentedMutation 302 +2 #7841: a put_page that asks for visibility: private is kept or refused (visibility_not_applied), checked before the no-op test and after the trust stamp (logic in persistence/visibility-request.ts)
+src/core/persistence/page-prepare.ts prepareInlineOrSegmentedMutation 303 +3 #7841: a put_page that asks for visibility: private is kept or refused (visibility_not_applied), checked before the no-op test and after the trust stamp (logic in persistence/visibility-request.ts)
src/core/progressive-batch/orchestrator.ts runProgressiveBatch 348 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)
src/core/remediation/run.ts runRemediation 428 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)
src/core/skillopt/orchestrator.ts runOptimizationLoop 704 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)
diff --git a/src/core/persistence/page-prepare.ts b/src/core/persistence/page-prepare.ts
index e57b1db5d6..41a5cbeb68 100644
--- a/src/core/persistence/page-prepare.ts
+++ b/src/core/persistence/page-prepare.ts
@@ -67,7 +67,7 @@ import { DEFAULT_WRITE_GATE_CONFIG, writeGateApplies } from '../write-gate.ts';
import { materializeTimeline, prepareCanonicalProjections, type TimelineRowsRemoved } from './canonical-projections.ts';
import { assertTimelineNotOmitted, timelineWritePolicy } from './timeline-omission.ts';
import { preserveProtectedTakes } from './protected-takes.ts';
-import { assertPrivateRequestKept, declaredVisibilities } from './visibility-request.ts';
+import { assertPrivateRequestKept, readPrivateRequest } from './visibility-request.ts';
import { isAutoLinkEnabled } from '../link-extraction.ts';
import { prepareAutomaticLinks } from './links-preparation.ts';
import { loadActivePackForEngine } from '../schema-pack/engine-resolution.ts';
@@ -833,7 +833,8 @@ async function prepareInlineOrSegmentedMutation(unbounded: BrainEngine, row: Wri
const { timeline, materialized } = await materializeTimeline(engine,parsed,row.slug,snapshot,writer,timelinePolicy);
if (materialized) content = serializePageToMarkdown({...snapshot.page,...parsed,timeline,type:parsed.typeExplicit ? parsed.type : snapshot.page.type},parsed.tags);
}
- if (typeof content === 'string' && declaredVisibilities(content).includes('private')) assertPrivateRequestKept(row, content, parseMarkdown(content, row.slug, { activePack }).frontmatter);
+ const privateRequest = typeof content === 'string' ? readPrivateRequest(content, () => parseMarkdown(content as string, row.slug, { activePack }).frontmatter) : null;
+ if (privateRequest) assertPrivateRequestKept(row, privateRequest.frontmatter);
const storedTier = snapshot ? await storedPageTier(engine, row.source_id, row.slug) : null;
// Detect an exact canonical no-op before ingestion can invoke any provider.
// Revision/identity checks above still apply to stale identical replacements.
@@ -875,7 +876,7 @@ async function prepareInlineOrSegmentedMutation(unbounded: BrainEngine, row: Wri
provenance = putProvenance(row, snapshot, page);
suppliedTrustMarker = page.frontmatter.trust_tier;
stampPageTrustMarker(row, page.frontmatter, trust, storedTier);
- if (typeof content === 'string') assertPrivateRequestKept(row, content, page.frontmatter);
+ if (privateRequest) assertPrivateRequestKept(row, page.frontmatter);
},
prepare: async value => { prepared = value; return value.result; },
}).catch(error => {
diff --git a/src/core/persistence/visibility-request.ts b/src/core/persistence/visibility-request.ts
index 616c037c03..1a68d8d9c7 100644
--- a/src/core/persistence/visibility-request.ts
+++ b/src/core/persistence/visibility-request.ts
@@ -4,12 +4,13 @@
* comes back as a no-op, with the page still world-readable.
*
* The request is read from the content's leading frontmatter block line by line,
- * independently of the YAML parse, so a request the parse does not keep is still
- * seen: a key or value in another case (`Visibility: private`,
- * `visibility: Private`, which page privacy does not read as private), or a line
- * the parse drops. Only a request for `private` is checked, so the guard can only
- * refuse: it never widens a page and never changes what a remote caller may read
- * or write. A request that is kept passes unchanged.
+ * and from the parse of a visibility line, so a request the stored page would not
+ * keep is still seen: a key or value in another case (`Visibility: private`,
+ * `visibility: Private`, which page privacy does not read as private), a value
+ * the parse reshapes (a block scalar, a list) or a line the parse drops. Only a
+ * request for `private` is checked, so the guard can only refuse: it never widens
+ * a page and never changes what a remote caller may read or write. A request that
+ * is kept passes unchanged.
*/
import { opError, type OperationError } from '../ops/contract.ts';
import { readFix } from '../ops/op-fix.ts';
@@ -30,11 +31,31 @@ export function declaredVisibilities(content: string): string[] {
return [];
}
-/** Refuses a write whose content asks for `visibility: private` when `published` (the frontmatter it would store) is not private. */
-export function assertPrivateRequestKept(row: Pick<WriteRequest, 'slug' | 'source_id'>, content: string, published: Record<string, unknown> | null | undefined): void {
- if (!declaredVisibilities(content).includes('private') || published?.visibility === 'private') return;
- throw visibilityNotApplied(row, typeof published?.visibility === 'string' && published.visibility.trim().toLowerCase() === 'private'
- || Object.keys(published ?? {}).some(key => key !== 'visibility' && key.toLowerCase() === 'visibility') ? 'noncanonical_value' : 'not_kept');
+/** Every string the frontmatter's `visibility` key (any case) holds, trimmed and lowercased; a list contributes its string items. */
+function visibilityValues(frontmatter: Record<string, unknown> | null | undefined): string[] {
+ const flatten = (value: unknown): string[] => typeof value === 'string' ? [value.trim().toLowerCase()] : Array.isArray(value) ? value.flatMap(flatten) : [];
+ return Object.entries(frontmatter ?? {}).flatMap(([key, value]) => key.toLowerCase() === 'visibility' ? flatten(value) : []);
+}
+
+/**
+ * The private request `content` carries, with the frontmatter it parses to, or null.
+ * A visibility line asks for private when its own text says so or when the parse
+ * reads it as private (a value on the next line, a block scalar, a flow list), so a
+ * spelling the line scan cannot read is still seen. Parsing waits for a visibility key.
+ */
+export function readPrivateRequest(content: string, parse: () => Record<string, unknown> | null | undefined): { frontmatter: Record<string, unknown> | null | undefined } | null {
+ const declared = declaredVisibilities(content);
+ if (!declared.length) return null;
+ const inline = declared.includes('private');
+ let frontmatter: Record<string, unknown> | null | undefined;
+ try { frontmatter = parse(); } catch (error) { if (inline) throw error; return null; }
+ return inline || visibilityValues(frontmatter).includes('private') ? { frontmatter } : null;
+}
+
+/** Refuses a write that asked for `visibility: private` when `published` (the frontmatter it would store) does not carry exactly that. */
+export function assertPrivateRequestKept(row: Pick<WriteRequest, 'slug' | 'source_id'>, published: Record<string, unknown> | null | undefined): void {
+ if (published?.visibility === 'private') return;
+ throw visibilityNotApplied(row, visibilityValues(published).includes('private') ? 'noncanonical_value' : 'not_kept');
}
function visibilityNotApplied(row: Pick<WriteRequest, 'slug' | 'source_id'>, reason: 'noncanonical_value' | 'not_kept'): OperationError {
diff --git a/test/put-visibility-private-7841.test.ts b/test/put-visibility-private-7841.test.ts
index 3fe383a282..59705d82c1 100644
--- a/test/put-visibility-private-7841.test.ts
+++ b/test/put-visibility-private-7841.test.ts
@@ -96,6 +96,10 @@ test('declaredVisibilities reads the frontmatter block only, any key case, lower
expect(declaredVisibilities(page('summary: >\n visibility: private\n'))).toEqual([]);
expect(declaredVisibilities('A body.\nvisibility: private\n')).toEqual([]);
expect(declaredVisibilities('---\nvisibility: private\n')).toEqual([]);
+ expect(declaredVisibilities('\uFEFF\n\n---\r\nvisibility: private\r\n---\r\nA body.\r\n')).toEqual(['private']);
+ expect(declaredVisibilities('---\nvisibility: private\n...\nA body.\n')).toEqual(['private']);
+ expect(declaredVisibilities(page('"visibility": private # keep\n'))).toEqual(['private']);
+ expect(declaredVisibilities(page('visibility: private\nvisibility: public\n'))).toEqual(['private', 'public']);
});
for (const managed of [false, true]) describe(`${managed ? 'managed brain with a Git checkout' : 'repo-less brain'}: put_page visibility: private (#7841)`, () => {
@@ -113,7 +117,8 @@ for (const managed of [false, true]) describe(`${managed ? 'managed brain with a
});
}, 120_000);
- for (const [label, extra] of [['visibility: Private', 'visibility: Private\n'], ['Visibility: private', 'Visibility: private\n'], ['VISIBILITY: "PRIVATE"', 'VISIBILITY: "PRIVATE"\n']] as const) {
+ for (const [label, extra] of [['visibility: Private', 'visibility: Private\n'], ['Visibility: private', 'Visibility: private\n'], ['VISIBILITY: "PRIVATE"', 'VISIBILITY: "PRIVATE"\n'],
+ ['visibility with Private on the next line', 'visibility:\n Private\n'], ['visibility as a block scalar', 'visibility: |\n private\n'], ['visibility as a flow list', 'visibility: [private]\n']] as const) {
test(`${label} on an existing world page is refused noncanonical_value, and the page and its revision are unchanged`, async () => {
await brain(managed, async (ctx, engine) => {
await put(ctx, 'notes/example', page());
@@ -128,6 +133,27 @@ for (const managed of [false, true]) describe(`${managed ? 'managed brain with a
}, 120_000);
}
+ test('a quoted key with a trailing comment still publishes private, with CRLF line endings too', async () => {
+ await brain(managed, async (ctx, engine) => {
+ for (const [slug, content] of [['notes/quoted', page('"visibility": private # keep\n')], ['notes/crlf', page('visibility: private\n').replace(/\n/g, '\r\n')]] as const) {
+ expect((await put(ctx, slug, content)).thrown).toBeUndefined();
+ expect(isPrivatePage((await snapshot(engine, ctx, slug))!.page)).toBe(true);
+ }
+ });
+ }, 120_000);
+
+ test('a remote caller is refused the same way, and its plain private request still publishes private', async () => {
+ await brain(managed, async (ctx, engine) => {
+ const remote = { ...ctx, remote: true };
+ await put(ctx, 'notes/example', page());
+ const before = (await snapshot(engine, ctx, 'notes/example'))!;
+ expect(await put(remote, 'notes/example', page('visibility: Private\n'), before.revision)).toMatchObject({ thrown: 'visibility_not_applied' });
+ expect((await snapshot(engine, ctx, 'notes/example'))!.revision).toBe(before.revision);
+ expect((await put(remote, 'notes/remote-example', page('visibility: private\n'))).thrown).toBeUndefined();
+ expect(isPrivatePage((await snapshot(engine, ctx, 'notes/remote-example'))!.page)).toBe(true);
+ });
+ }, 120_000);
+
test('a new page created with visibility: Private is refused, never stored world-readable', async () => {
await brain(managed, async (ctx, engine) => {
expect(await put(ctx, 'notes/new-example', page('visibility: Private\n'))).toMatchObject({ thrown: 'visibility_not_applied' });
diff --git a/test/synthesize-session-source-7805.test.ts b/test/synthesize-session-source-7805.test.ts
index 2896875a90..3bef42b5d7 100644
--- a/test/synthesize-session-source-7805.test.ts
+++ b/test/synthesize-session-source-7805.test.ts
@@ -15,7 +15,7 @@ import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { randomUUID } from 'node:crypto';
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
-import { basename, join } from 'node:path';
+import { basename, dirname, join } from 'node:path';
import type { BrainEngine } from '../src/core/engine.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runPhaseSynthesize } from '../src/core/cycle/synthesize.ts';
@@ -62,23 +62,28 @@ const usage = { input_tokens: 100, output_tokens: 100, cache_read_tokens: 0, cac
interface Fixture { engine: BrainEngine; cycle: { id: string; root: string }; other: { id: string; root: string }; corpus: string }
-async function fixture(managed: boolean, run: (f: Fixture) => Promise<void>) {
+async function fixture(managed: boolean, run: (f: Fixture) => Promise<void>, solo = false) {
for (const engine of engines) {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-synth-source-'));
const corpus = join(dir, 'corpus');
const tag = randomUUID().slice(0, 8);
- const cycle = { id: `source-a-${tag}`, root: join(dir, 'brain-a') };
+ const cycle = { id: solo ? 'default' : `source-a-${tag}`, root: join(dir, 'brain-a') };
const other = { id: `source-b-${tag}`, root: join(dir, 'brain-b') };
mkdirSync(corpus);
try {
await withEnv({ GBRAIN_HOME: join(dir, 'home'), ANTHROPIC_API_KEY: 'sk-test-synth-source', GBRAIN_SOURCE: undefined }, async () => {
await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');
- for (const s of [cycle, other]) {
+ const sources = solo ? [cycle] : [cycle, other];
+ for (const s of sources) {
mkdirSync(s.root);
- await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [s.id, s.root]);
+ if (solo) {
+ await engine.executeRaw("DELETE FROM sources WHERE id <> 'default'");
+ await engine.executeRaw('UPDATE sources SET local_path=$2 WHERE id=$1', [s.id, s.root]);
+ }
+ else await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [s.id, s.root]);
}
await engine.setConfig('sync.write_through', 'true');
- for (const s of [cycle, other]) await claimWorktree(engine, s.id, s.root);
+ if (!solo) for (const s of sources) await claimWorktree(engine, s.id, s.root);
for (const [key, value] of Object.entries({
'dream.synthesize.enabled': 'true', 'dream.synthesize.cooldown_hours': '0',
'dream.synthesize.min_chars': '100', 'dream.synthesize.link_manifest': 'false',
@@ -182,4 +187,51 @@ for (const managed of [false, true]) describe(`${managed ? 'managed' : 'unmanage
expect(await synthesizedSources(engine, { cycle, other })).toEqual({ 'sess-extracted.txt': [cycle.id] });
});
}, 120_000);
+
+ test('an --input file outside the corpus is kept for the cycle source, in no routing bucket', async () => {
+ await fixture(managed, async ({ engine, cycle, other, corpus }) => {
+ const outside = join(dirname(corpus), 'meeting-notes.txt');
+ writeTranscript(outside, 'the quarterly offsite');
+ const result = await runPhaseSynthesize(engine, { brainDir: cycle.root, sourceId: cycle.id, dryRun: false, inputFile: outside });
+ expect(result.status).toBe('ok');
+ expect(result.details.source_routing).toBeUndefined();
+ expect(await synthesizedSources(engine, { cycle, other })).toEqual({ 'meeting-notes.txt': [cycle.id] });
+ });
+ }, 120_000);
+});
+
+test('a sole-source brain (unmanaged) still synthesizes a legacy top-level file (#7805)', async () => {
+ await fixture(false, async ({ engine, cycle, other, corpus }) => {
+ writeTranscript(join(corpus, 'sess-solo.txt'), 'the office move');
+ const result = await runPhaseSynthesize(engine, { brainDir: cycle.root, sourceId: cycle.id, dryRun: false });
+ expect(result.status).toBe('ok');
+ expect(result.details.source_routing).toBeUndefined();
+ expect(await synthesizedSources(engine, { cycle, other })).toEqual({ 'sess-solo.txt': ['default'] });
+ }, true);
+}, 120_000);
+
+describe('assign-corpus listing includes sessions the sweep already extracted (#7805)', () => {
+ const extracted = (dir: string, name: string) => { writeTranscript(join(dir, name), 'the office move'); writeFileSync(join(dir, name + '.ingested'), '{}\n'); };
+
+ test('only a held extracted session is listed: not a stamped or frozen one, not a sole-source legacy file, not a file outside the corpus', () => {
+ const corpus = mkdtempSync(join(tmpdir(), 'gbrain-assign-corpus-'));
+ try {
+ const spool = ensureCorpusSpoolDir(corpus);
+ extracted(spool, sessionCorpusFileName('sess-held', CORPUS_UNRESOLVED_STAMP));
+ extracted(spool, sessionCorpusFileName('sess-stamped', 'source-a'));
+ extracted(spool, sessionCorpusFileName('sess-frozen', CORPUS_UNRESOLVED_STAMP));
+ expect(freezeSessionSource(corpus, 'sess-frozen', 'source-b', 'operator')?.source_id).toBe('source-b');
+ extracted(corpus, 'sess-legacy.txt');
+ const outside = join(dirname(corpus), `${basename(corpus)}-elsewhere`);
+ mkdirSync(outside);
+ extracted(outside, 'sess-elsewhere.txt');
+
+ expect(listCorpusSourceHolds(corpus, false).sessions).toEqual([]);
+ expect(listCorpusSourceHolds(corpus, false, { includeIngested: true })).toMatchObject({ unresolved: 1, legacy_unstamped: 1, sessions: ['sess-held', 'sess-legacy'] });
+ expect(listCorpusSourceHolds(corpus, true, { includeIngested: true }).sessions).toEqual(['sess-held']);
+ expect(assignCorpusSessions(corpus, ['sess-held', 'sess-frozen'], 'source-a')).toEqual(['sess-held']);
+ expect(freezeSessionSource(corpus, 'sess-frozen', 'source-a', 'operator')?.source_id).toBe('source-b');
+ rmSync(outside, { recursive: true, force: true });
+ } finally { rmSync(corpus, { recursive: true, force: true }); }
+ });
});