Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(enrich): a model-written timeline no longer holds republication as lossy (#7663)
garrytan0fcb666e337a
diff --git a/docs/architecture/key-files/core-cycle-patterns.md b/docs/architecture/key-files/core-cycle-patterns.mdindex a085e4d6a3..815b63d71e 100644--- a/docs/architecture/key-files/core-cycle-patterns.md+++ b/docs/architecture/key-files/core-cycle-patterns.md@@ -22,7 +22,7 @@ Patterns, drift reports and the cycle modules listed after them.- `src/core/cycle/triage-rescue.ts` — THE dream triage gate (`passesTriageGate`) + the verified-segment rescue (`applyTriageRescue`). Rescue fires only for scores in `[floor, threshold)` (floor inclusive; at/above threshold is the plain gate's job) with content_type in the buried-signal allowlist (default mixed/reflection/idea/strategy/people — never routine/technical) and ≥ minSegments of the judge's own segments verifying as normalized transcript substrings (`normForGrounding`, ≥40 chars, deduped by normalized quote — repetition is not more evidence). $0 — no LLM calls; fabricated segments cannot fire it; works on cached verdicts (gate-time only). Fail-closed on every malformed shape (null score, missing/short/non-string segments), never throws. `minSegments: 0` is the kill switch (gate degenerates to the plain threshold). ONE-GATE RULE: `runTriagePass` (report construction — `worth`/`rescued`/telemetry/dry-run), the synthesize fan-out, and `dream retriage` (reconcile-queue cancels + `--audit-rejects` sampling) all read this predicate — a second hand-rolled `score >= threshold` check is how an operator sweep cancels exactly the jobs the rescue admitted. Config: `dream.triage.rescue_floor` / `rescue_min_segments` / `rescue_content_types`. Pinned by `test/cycle-triage-rescue.test.ts` + the rescue suites in `test/cycle-synthesize-triage.test.ts` / `test/dream-retriage.test.ts`. System One S7: a decide verdict (`model` starting with `DECIDE_VERDICT_PREFIX`) is gated against the S7 slot threshold (the optional fifth argument) and never enters the rescue band.- `src/core/cycle/derived-write-through.ts` — opt-in markdown write-through for generated pages on an unmanaged brain (#5041). `cycle.extract_atoms.write_through` / `cycle.synthesize_concepts.write_through` (off by default; `true`/`1`/`yes`/`on`) make `derivedWriteThrough` return a hook (null when off, managed or dry run): `extract_atoms` writes each item's atoms after the completion flip and removes the files of atoms it retired; `synthesize_concepts` writes through `publishClassicConcept`, which also rewrites a concept that already has a file whatever the key says. Creating the hook writes up to 200 generated pages per run that have no file yet (`source_path` NULL; pending atoms excluded), so enabling the key backfills. `writeDerivedPageThrough` keeps a page database-only when its target is ignored by git or sits in a declared `storage.db_only` directory (`derivedFileTarget`), replaces an existing file only when `replaceableFile` (`facts/file-body-guard.ts`) finds it equal to the page or one of its recent page_versions rows, then calls `writePageThrough` bound to the bytes compared and returns its result (`written`, `skipped`, `error`) so `publishClassicConcept` can surface a failed write. #8073: every file write and retired-file unlink is uncommitted (`deferCommit`) and the batch (an item's atoms plus its retired atoms' files, the backfill, or one concept) commits once through the shared-checkout door `commitPathsLocked` (`git/commit-push.ts`) under the source filesystem lock, naming only the batch's paths that exist or are tracked; a directory that is not a Git checkout, or whose checkout lookup fails (`classifyGitCheckout` `unknown`), has nothing to commit. A refused commit leaves a file whose change HEAD already holds (another writer's locked commit swept it from the ledger) as committed, restores each other file still holding the bytes the batch wrote (`WriteThroughResult.content`; prior bytes, or removed), clears the `source_path` the write bound so the next backfill rewrites it, and returns a typed `DerivedCommitFailure` (`{code, message, paths, kept}`); the hook then writes nothing more that run and the phases report `details.write_through_failed` with `warn` (`reportDerivedWriteThrough`), while `writeDerivedPageThrough` returns it as `error` so the concept is held. Each file written or removed is also recorded in the door's pending ledger (`notePending`) right after the write, so a crash before the commit leaves it for the next `commitPathsLocked` in that checkout to sweep (ledger paths inside the source root that still differ from HEAD; only paths gbrain wrote are recorded). The dream reverse-write (`writeDreamPageFile`, `removeDreamPageFile`, `reverseWriteDreamRefs`) commits the same way: one batch per `reverseWriteDreamRefs` call (committed in a `finally`, so an abort still commits what it wrote) and one per summary write or retired-file removal; a refused commit rolls the files back and returns or pushes a `write_through_failed` hold, which the phases report through their existing `file_holds`. A checkout carrying a canonical owner marker commits through `commitAndPush`'s named-paths-only path (`commitManagedPaths`), and paths git ignores are left uncommitted. Tests: `test/cycle/derived-write-through-commit.test.ts` (with the `derivedWriteSeams.beforeCommit` crash fault point) and `test/cycle/dream-reverse-write-commit.test.ts` (PGLite and Postgres). A retired atom's file is removed under the same filter and comparison, the unlink rechecking the bytes under the source filesystem lock. A file holding an edit the database lacks (or an unindexed file at a backfilled page's path) is left as it is and listed in the hook's `held`, which both phases report as `details.file_holds` (`file_database_drift`) with a `warn` status. Tests: `test/cycle/derived-write-through.test.ts`, `test/pathological/d10-derived-atom-file-overwrite.test.ts`, `test/pathological/d11-derived-retired-atom-unlink.test.ts`. #6982: `writeDreamPageFile` is the dream reverse-write (synthesize, patterns, the cycle summary): nothing with `sync.write_through` off, for an owned url clone or a `storage.db_only` slug; the path is `resolvePageWriteTarget`'s (recorded source_path), falling back to the caller's brainDir layout when the source has no root or its root is missing; the file is replaced only when `replaceableFile` (facts/file-body-guard.ts) finds nothing in it the database lacks and, once the replacement is staged, still holds the compared bytes right before the rename, else the result carries the `file_database_drift` hold. It writes directly (not `writePageThrough`/`atomicWriteFileSync`) because a checkout a subagent tool write claimed through the coordinator refuses those with `writer_coordinator_required`. `reportFileHolds` lists every phase's holds in `details.file_holds` and turns ok into warn.-- `src/core/cycle/concept-publication.ts` — managed `synthesize_concepts` publication: `publishManagedConcept` publishes through `publishMaintenancePage` (a concept with a recorded `source_path` republishes to its file unless a `db_only` tier covers it; new concepts are database-only), and `composeConceptRepublication` keeps an existing concept's facts/takes fences, timeline, tags and other frontmatter, holding the concept (`concept_preservation_hold`) when a fence is duplicated, unbalanced, unparseable or below the timeline sentinel. `replaceConceptProvenanceLinks` banks the concept-provenance edges and retires the edges of atoms that left the group in one transaction (coordinated on a managed brain), only while the concept is at the revision its publication produced (GBRA-161 PR 5, C-4). On an unmanaged brain `publishClassicConcept` writes under the page lock the fence writers take: it re-reads the page, defers (`revision_conflict`) when the narrative changed since synthesis started, another writer owns it, or the page's canonical file holds anything the database lacks (`replaceableFile`: body, fences, timeline, frontmatter, tags, type or title equal to neither the page nor a recent page_versions row; formatting is ignored; this also covers an unindexed file at a new concept's path, and the message names the page with `file_database_drift`), and otherwise composes the new narrative with the database's latest fences, timeline, tags and frontmatter, so a take added during synthesis survives (`test/pathological/d12-concept-stale-fences.test.ts`, `d13-concept-occupied-path.test.ts`). The file write after `importPage` passes the bytes read under the lock as `expectedFileBytes`, so an edit landing during the import is kept and the concept deferred; a failed file write holds the concept (`concept_write_through_failed`) instead of reporting success (W9F item 3, `test/cycle/concept-publication-file-recheck.test.ts`). `stripFenceSections` is the shared narrative view. Tests: `test/cycle/synthesize-concepts-fences.test.ts`. The phase keeps the private-first, provenance, then promotion order; revision races are deferred and holds rep... (line truncated to 2000 chars)+- `src/core/cycle/concept-publication.ts` — managed `synthesize_concepts` publication: `publishManagedConcept` publishes through `publishMaintenancePage` (a concept with a recorded `source_path` republishes to its file unless a `db_only` tier covers it; new concepts are database-only), and `composeConceptRepublication` keeps an existing concept's facts/takes fences, timeline, tags and other frontmatter (a timeline the replacement writes itself is dropped, `test/cycle/concept-publication-timeline.test.ts`), holding the concept (`concept_preservation_hold`) when a fence is duplicated, unbalanced, unparseable or below the timeline sentinel. `replaceConceptProvenanceLinks` banks the concept-provenance edges and retires the edges of atoms that left the group in one transaction (coordinated on a managed brain), only while the concept is at the revision its publication produced (GBRA-161 PR 5, C-4). On an unmanaged brain `publishClassicConcept` writes under the page lock the fence writers take: it re-reads the page, defers (`revision_conflict`) when the narrative changed since synthesis started, another writer owns it, or the page's canonical file holds anything the database lacks (`replaceableFile`: body, fences, timeline, frontmatter, tags, type or title equal to neither the page nor a recent page_versions row; formatting is ignored; this also covers an unindexed file at a new concept's path, and the message names the page with `file_database_drift`), and otherwise composes the new narrative with the database's latest fences, timeline, tags and frontmatter, so a take added during synthesis survives (`test/pathological/d12-concept-stale-fences.test.ts`, `d13-concept-occupied-path.test.ts`). The file write after `importPage` passes the bytes read under the lock as `expectedFileBytes`, so an edit landing during the import is kept and the concept deferred; a failed file write holds the concept (`concept_write_through_failed`) instead of reporting success (W9F item 3, `test/cycle/concept-publication-file-recheck.test.ts`). `stripFenceSections` is the shared narrative view. Tests: `test/cycle/synthesize-concepts-fences.test.ts`. The phase keeps the private-first, provenance, then promotion order; revision races are deferred and holds rep... (line truncated to 2000 chars)- `src/core/cycle/connector-atoms.ts` — the connector atom setting (#5856). `CONNECTOR_ATOM_PAGES_KEY` (`cycle.extract_atoms.connector_pages`, on when unset; `connectorAtomPagesEnabled` treats only false/0/off/no as opted out) gates `CONNECTOR_ATOM_PAGE_TYPES` (`email`, `meeting`) of sources whose `config.kind` is a connector kind (`CONNECTOR_SOURCE_KINDS`). `connectorAtomExclusionSql` returns the `pages p` predicate atom discovery and the backlog count append (empty unless opted out), so the routine cycle, `dream --drain` and autopilot auto-drain extract connector mail and calendar pages under the auto-drain budget until the user opts out. Other page types of a connector source and every checkout-backed page are unaffected. Tests: `test/connector-atom-pages.test.ts`, `test/autopilot-auto-drain-dispatch.test.ts`, and the `atom_drain_default_on` / `atom_drain_opted_out` cases of `test/helpers/managed-connector-job-contract.ts`, run by `test/managed-connector-job-contract.test.ts` and its Postgres twin `test/e2e/managed-connector-job-contract.test.ts`.diff --git a/src/core/cycle/concept-publication.ts b/src/core/cycle/concept-publication.tsindex 5d9e2cc977..b5e9c3a4ea 100644--- a/src/core/cycle/concept-publication.ts+++ b/src/core/cycle/concept-publication.ts@@ -12,7 +12,7 @@ import type { BrainEngine, LinkBatchInput } from '../engine.ts';import { preservedConceptSections, stripPreservedConceptSections } from './concept-sections.ts';import type { Page } from '../types.ts';import { assertInlineBody, isSegmentedPage, pageBodySegmentedError } from '../page-body/page-shape.ts';-import { serializeMarkdown, parseMarkdown } from '../markdown.ts';+import { serializeMarkdown, parseMarkdown, splitBody } from '../markdown.ts';import { FACTS_FENCE_BEGIN, FACTS_FENCE_END, parseFactsFence, replaceOrInsertFactsFence, stripFactsFence } from '../facts-fence.ts';import { TAKES_FENCE_BEGIN, TAKES_FENCE_END, parseTakesFence, stripTakesFence } from '../takes-fence.ts';import { isDbOnly, loadStorageConfig } from '../storage-config.ts';@@ -158,6 +158,8 @@ export function composeConceptRepublication(page: Pick<Page, 'type' | 'title' |* body written by a model or a synthesis phase, which owns only the prose. Any* fence in the replacement is dropped and the original blocks are inserted* verbatim, so publication neither expires fence facts nor deletes takes.+ * A timeline the replacement carries (below a sentinel, or a bare dated+ * `## Timeline` section) is dropped the same way; the page keeps its own.* Curated `## Facets` / `## Merged` sections (#6161: the record of a concept* merge) are carried the same way. Throws a hold* (`concept_preservation_hold`) when the original fences are ambiguous or the@@ -169,7 +171,7 @@ export function preserveCanonicalFences(page: Pick<Page, 'compiled_truth' | 'timconst hold = conceptPreservationHold(page);if (hold) throw conceptHoldError(hold);const body = page.compiled_truth ?? '';- let compiled = stripPreservedConceptSections(stripTakesFence(stripFactsFence(replacement))).trim();+ let compiled = stripPreservedConceptSections(stripTakesFence(stripFactsFence(splitBody(replacement).compiled_truth))).trim();const sections = preservedConceptSections(body);if (sections.length) compiled = [compiled, ...sections].join('\n\n');const facts = fenceBlock(body, FACTS_FENCE_BEGIN, FACTS_FENCE_END);diff --git a/test/cycle/concept-publication-timeline.test.ts b/test/cycle/concept-publication-timeline.test.tsnew file mode 100644index 0000000000..b75cc58557--- /dev/null+++ b/test/cycle/concept-publication-timeline.test.ts@@ -0,0 +1,46 @@+// #7663: preserveCanonicalFences treats a timeline the model wrote itself+// like a fence: dropped from the replacement, with the page's own timeline+// carried over.+//+// Protects: enrich and synthesize_concepts republish a page whose model+// write-up ends with a `## Timeline` section instead of holding it with+// CONCEPT_REPUBLICATION_LOSSY (which left the page stale and re-bought the+// model call every night).+// Fails when: the replacement's timeline section reaches the composed body,+// where the final parse reads it as the page timeline.++import { describe, test, expect } from 'bun:test';+import { preserveCanonicalFences } from '../../src/core/cycle/concept-publication.ts';+import { parseMarkdown, serializeMarkdown } from '../../src/core/markdown.ts';++const PAGE_TIMELINE = '- **2026-09-01** | Joined acme-example';+const REPLACEMENTS = [+ 'Para one.\n\n## Role\n\nPara two.',+ 'Para one.\n\n---\n\nPara two.',+ 'Para one.\n\n## Facts\n\n- a fact',+ 'Para one.\n\n## Recent activity\n\n- **2026-10-01** | x',+ 'Para one.\n\n## Timeline\n\n- **2026-10-01** | x',+];++function page(timeline: string) {+ return { compiled_truth: 'Old narrative.', timeline, body_storage: 'inline' as const };+}++describe('preserveCanonicalFences and a model-written timeline (#7663)', () => {+ for (const timeline of ['', PAGE_TIMELINE]) {+ for (const replacement of REPLACEMENTS) {+ test(`${JSON.stringify(replacement)} with ${timeline ? 'a' : 'no'} page timeline republishes with the page timeline`, () => {+ const compiled = preserveCanonicalFences(page(timeline), replacement);+ const out = parseMarkdown(serializeMarkdown({}, compiled, timeline, { type: 'person', title: 'alice example', tags: [] }), 'page');+ expect(out.timeline).toBe(timeline);+ expect(out.compiled_truth).toContain('Para one.');+ expect(out.compiled_truth).not.toMatch(/^##\s+Timeline/m);+ });+ }+ }++ test('the model timeline section is dropped and the narrative around it is kept', () => {+ const compiled = preserveCanonicalFences(page(''), 'Para one.\n\n## Timeline\n\n- **2026-10-01** | x\n\n## Role\n\nPara two.');+ expect(compiled).toBe('Para one.\n\n## Role\n\nPara two.');+ });+});