Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(timeline): citation timeline entries keep the text in backticks
garrytan6508d35e8b2c
diff --git a/docs/architecture/key-files/core-utilities-2.md b/docs/architecture/key-files/core-utilities-2.mdindex 2e88a2a908..eca37ab876 100644--- a/docs/architecture/key-files/core-utilities-2.md+++ b/docs/architecture/key-files/core-utilities-2.md@@ -46,7 +46,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/surface-audit.ts` — `writeSurfaceChangeAudit(engine, audit)`: the surface-mutation audit trail. Every surface mutation (rescope CLI, `POST /admin/api/rescope-client`, `request_tools` persist) writes one typed `mcp_request_log` row — `operation='surface_change'`, params a RAW object `{actor, client_id, old, new, via}` via `executeRawJsonb` (never `JSON.stringify` into `::jsonb`). Zero new DDL; rides `idx_mcp_log_time_agent` + the retention TODO. Best-effort: a failed audit write warns to stderr, never fails the committed mutation (callers needing fail-closed semantics inspect the returned boolean). CLI-actor rows are written even though stdio ops don't otherwise log (documented exception — audit outranks the transport-logging convention). The usage reader excludes these rows from op-call stats.- `src/core/timeline-marker.ts` — the `<!-- gbrain:materialized v1 <12-hex> -->` marker that precedes a materialized timeline bullet: `timelineKey` (the one normalization for prior-set membership, dedup, stored-row matching and marker hashes), `timelineTupleHash`, `materializedMarker`, marker-line detection and `stripMaterializedMarkers`. Both timeline extractors skip marker lines (a marker never becomes bullet or header detail) and `sanitizeRemoteBody` drops them from chunk text unless `keepMaterializedMarkers` is set (remote `get_page`/`fetch` keep them so an edit round-trips). A marker whose hash no longer matches the bullet below it marks nothing.-- `src/core/timeline-citations.ts` — `parseInlineCitationTimelineEntries`: inline `[Source: <source>, YYYY-MM-DD]` citations become timeline candidates whose summary is the citing paragraph without the citation; a page with no `[Source:` returns none before any paragraph is built. #6184: a comment-only line is a paragraph boundary and `stripHtmlComments` removes `<!-- … -->` spans (and dangling `<!--`/`-->`) before the summary is built, so section and materialized-row markers are never filed as entries. #6226: a citation files one entry per dated `source, date` segment (a `;` splits sources only after a date), summaries unwrap paired emphasis, and `supersededInlineCitationEntries` returns what the older one-source reading filed and the current reading no longer does; `timeline-extract.ts:supersededCitationTimeline` runs it with both extractors' bullet skips and `retractRemovedTimelineEntries` retracts those rows when their detail is extractor-written (`hasExtractorDetail`: empty or `Source: <source>`). The dated-bullet regexes (`TIMELINE_LINE_RE`, `_CN`, `isDatedTimelineLine`) live here and `link-extraction.ts` imports them.+- `src/core/timeline-citations.ts` — `parseInlineCitationTimelineEntries`: inline `[Source: <source>, YYYY-MM-DD]` citations become timeline candidates whose summary is the citing paragraph without the citation; a page with no `[Source:` returns none before any paragraph is built. #6184: a comment-only line is a paragraph boundary and `stripHtmlComments` removes `<!-- … -->` spans (and dangling `<!--`/`-->`) before the summary is built, so section and materialized-row markers are never filed as entries. #6226: a citation files one entry per dated `source, date` segment (a `;` splits sources only after a date), summaries unwrap paired emphasis and keep inline code (#8893), and `supersededInlineCitationEntries` returns older readings' entries the current one drops; `timeline-extract.ts:supersededCitationTimeline` runs it with both extractors' bullet skips and `retractRemovedTimelineEntries` retracts those rows when their detail is extractor-written (`hasExtractorDetail`: empty or `Source: <source>`). The dated-bullet regexes (`TIMELINE_LINE_RE`, `_CN`, `isDatedTimelineLine`) live here and `link-extraction.ts` imports them.- `src/core/transcripts.ts` — `listRecentTranscripts(engine, opts)` library reused by both the `gbrain transcripts recent` CLI and the `get_recent_transcripts` MCP op. Reads `dream.synthesize.session_corpus_dir` + `dream.synthesize.meeting_transcripts_dir` config (same as `discoverTranscripts`); walks `.txt` files within `days`; applies the `isDreamOutput` guard from `transcript-discovery.ts` (skips dream-generated files); returns `{path, date, mtime, length, summary}[]` sorted newest-first. Summary mode (default true) = first non-empty line + ~250 trailing chars; full mode caps at 100KB/file. Missing/non-existent corpus dirs return `[]`, not error. Trust gate lives in the op handler, not here: the op throws `permission_denied` for `ctx.remote === true`; this is a trusted library function used by both the gated op and the local CLI.- `src/core/eligibility/fence-overlay.ts` + `src/core/trust/fence-append.ts` — the facts-fence eligibility overlay for chunks (#5575 ENG-1). `loadFenceChunkOverlay(engine, { sourceId, slug, compiled_truth, timeline })` reads, in one query and only when the body has world fence rows, which rows to omit (a `write_gate_holds` hold on the row's fingerprint, through the `registerFactHoldFingerprint` seam for the write gate's `holdFingerprint`; a `fact_purges` tombstone, the backstop to L3's import overlay) and which to demote (stored `facts.trust_tier` below the page tier, or a row a fence append is publishing, declared with `withPendingFenceRows`). `prepareMarkdownChunks(page, maxTokens, overlay)` (`markdown-chunks.ts`) drops both from the page's chunks and emits demoted rows as trailing chunks whose first line is `fenceTrustMarker(tier)` (a `compactTrustLabel` line with origin `facts-fence`); without overlay rows the output is byte-identical. The importer (`prepareFenceAwareMarkdownChunks`), projection rebuilds (`ProjectionSnapshot.fenceOverlay`), embed's chunkless rebuilds (`prepareProseChunks`) and evidence delivery's page document (`pageEvidenceText`) all cut through it; `healOversizedChunks` keeps the marker on every split piece. `fenceLowTierRows` / `renderFenceLowTier` give the demoted rows of one text for the segment reader's low-tier block. `eligibility/stamp.ts` labels a row carrying a marker min(page tier, marker tier) with origin `facts-fence` and applies floors at that tier; a marker can only lower. `withPageTierKept(tx, page, fn)` restores the page's prior `trust_tier`/`write_origin` after a fence append's page write (promotion capped at the prior tier when raising back), used by `fence-write.ts` (legacy mirror), `memory-prepare.ts` (remember) and `facts-prepare.ts` (managed entity facts). Tests: `test/fence-chunk-overlay.test.ts` (PGLite and Postgres via `test/e2e/fence-chunk-overlay-postgres.test.ts`).diff --git a/src/core/timeline-citations.ts b/src/core/timeline-citations.tsindex 85ca0dcd22..9f41cf2fb1 100644--- a/src/core/timeline-citations.ts+++ b/src/core/timeline-citations.ts@@ -1,3 +1,4 @@+import { inlineCodeSpans } from './fence-scan.ts';import { stripCodeBlocks } from './markdown-code.ts';const CITATION_TIMELINE_RE = /\[Source:\s*([^\]]+?,\s*\d{4}-\d{2}-\d{2})\s*\]/g;@@ -48,7 +49,37 @@ export interface InlineCitationTimelineCandidate {}interface CitationParagraph {+ /** Code masked: where citations are found. */text: string;+ /** The same lines with each inline code span as a placeholder for its text in `codes` (#8893). */+ withCode: string;+ codes: string[];+}++const CODE_PLACEHOLDER_RE = /\uE000(\d+)\uE001/g;++/**+ * #8893: `masked` (stripCodeBlocks of `raw`, same length) with each same-line+ * inline span it blanked replaced by a placeholder for the span's text, so+ * summary rewriting never touches code and fenced text never comes back. A+ * span holding citation or comment markup stays blank: that is markup quoted+ * as an example, never summary text. Placeholder characters already in the+ * text become spaces, so only ours are restored. Linear in the line.+ */+function placeInlineCode(raw: string, masked: string, codes: string[]): string {+ const plain = (text: string) => text.replace(/[\uE000\uE001]/g, ' ');+ let out = '';+ let at = 0;+ for (const [start, end] of inlineCodeSpans(raw)) {+ if (masked.slice(start, end).trim()) continue;+ const run = /^`+/.exec(raw.slice(start, end))![0].length;+ const inner = raw.slice(start + run, end - run);+ if (/\[Source:|<!--|-->/.test(inner)) continue;+ const padded = inner.length > 2 && inner.startsWith(' ') && inner.endsWith(' ') && inner.trim() !== '';+ out += `${plain(masked.slice(at, start))}\uE000${codes.push(plain(padded ? inner.slice(1, -1) : inner)) - 1}\uE001`;+ at = end;+ }+ return out + plain(masked.slice(at));}/** #6184: HTML comments (section markers, materialized-row markers) are markup, never summary text. */@@ -87,18 +118,24 @@ function startsMarkdownBlock(line: string): boolean {function citationParagraphs(content: string,opts: { skipLine?: (line: string) => boolean } = {},+ keepCode = false,): CitationParagraph[] {const paragraphs: CitationParagraph[] = [];let lines: string[] = [];+ let codeLines: string[] = [];+ let codes: string[] = [];let skippedBlock = false;const flush = () => {if (lines.length === 0) return;- paragraphs.push({ text: lines.map((line) => line.trim()).join(' ') });+ paragraphs.push({ text: lines.map((line) => line.trim()).join(' '), withCode: codeLines.map((line) => line.trim()).join(' '), codes });lines = [];+ codeLines = [];+ codes = [];};- for (const line of stripCodeBlocks(content).split(/\r?\n/)) {+ const rawLines = keepCode ? content.split(/\r?\n/) : [];+ for (const [index, line] of stripCodeBlocks(content).split(/\r?\n/).entries()) {if (line.trim().length === 0 || isCommentOnlyLine(line)) {flush();continue;@@ -113,6 +150,7 @@ function citationParagraphs(skippedBlock = false;if (lines.length > 0 && startsMarkdownBlock(line)) flush();lines.push(line);+ if (keepCode) codeLines.push(placeInlineCode(rawLines[index] ?? line, line, codes));}flush();@@ -121,19 +159,28 @@ function citationParagraphs(type CitationOpts = { skipLine?: (line: string) => boolean };-function readCitations(content: string, opts: CitationOpts, legacy: boolean): InlineCitationTimelineCandidate[] {+/**+ * `current` keeps inline code text (#8893); `code_masked` is the reading before+ * it (inline code blanked); `legacy` the one before #6226 (code blanked, the+ * whole citation body one source, emphasis kept).+ */+type CitationReading = 'current' | 'code_masked' | 'legacy';++function readCitations(content: string, opts: CitationOpts, reading: CitationReading): InlineCitationTimelineCandidate[] {+ const legacy = reading === 'legacy';const result: InlineCitationTimelineCandidate[] = [];if (!content.includes('[Source:')) return result;- for (const paragraph of citationParagraphs(content, opts)) {+ const summaryOf = (text: string, codes?: string[]) => {+ const stripped = stripHtmlComments(text).replace(/\[Source:[^\]]*\](?:\((?:[^()]|\([^()]*\))*\))?/g, '');+ const lead = (legacy ? stripped : unwrapEmphasis(stripped)).replace(/^[-*>#\s]+/, '');+ return (codes ? lead.replace(CODE_PLACEHOLDER_RE, (_, i: string) => codes[Number(i)]!) : lead).replace(/\s+/g, ' ').trim().slice(0, 300);+ };+ for (const paragraph of citationParagraphs(content, opts, reading === 'current')) {const matches = [...paragraph.text.matchAll(CITATION_TIMELINE_RE)];if (matches.length === 0) continue;- const text = stripHtmlComments(paragraph.text)- .replace(/\[Source:[^\]]*\](?:\((?:[^()]|\([^()]*\))*\))?/g, '');- const summary = (legacy ? text : unwrapEmphasis(text))- .replace(/^[-*>#\s]+/, '')- .replace(/\s+/g, ' ')- .trim()- .slice(0, 300);+ const withCode = reading === 'current' ? summaryOf(paragraph.withCode, paragraph.codes) : '';+ // Restored code that completes citation or comment markup running outside its span reads as the masked text did.+ const summary = withCode && !/\[Source:|<!--|-->/.test(withCode) ? withCode : summaryOf(paragraph.text);if (!summary) continue;const seen = new Set<string>();for (const m of matches) {@@ -154,22 +201,27 @@ function readCitations(content: string, opts: CitationOpts, legacy: boolean): In* One entry per dated source of each `[Source: …, YYYY-MM-DD]` citation* (#6226: `[Source: A, 2026-10-06; B, 2026-09-28]` files A on 2026-10-06 and* B on 2026-09-28), summarized by the paragraph it annotates with citations,- * HTML comments (#6184) and paired emphasis removed.+ * HTML comments (#6184), paired emphasis and inline-code backticks removed.*/export function parseInlineCitationTimelineEntries(content: string, opts: CitationOpts = {}): InlineCitationTimelineCandidate[] {- return readCitations(content, opts, false);+ return readCitations(content, opts, 'current');}/**- * The entries the reading before #6226 filed for `content` (a multi-source- * citation as one source with its final date, emphasis kept) that the- * current reading no longer produces. Stored rows equal to one of them are+ * The entries the older readings filed for `content` (before #8893 inline+ * code blanked; before #6226 also a multi-source citation as one source with+ * its final date, emphasis kept) that the current reading no longer produces. Stored rows equal to one of them are* that older reading's output, so projection and retraction can retire them* instead of writing them back into the page.*/export function supersededInlineCitationEntries(content: string, opts: CitationOpts = {}): InlineCitationTimelineCandidate[] {- const current = new Set(readCitations(content, opts, false).map(e => JSON.stringify([e.date, e.source, e.summary])));- return readCitations(content, opts, true).filter(e => !current.has(JSON.stringify([e.date, e.source, e.summary])));+ const seen = new Set(readCitations(content, opts, 'current').map(e => JSON.stringify([e.date, e.source, e.summary])));+ return [...readCitations(content, opts, 'code_masked'), ...readCitations(content, opts, 'legacy')].filter((e) => {+ const key = JSON.stringify([e.date, e.source, e.summary]);+ if (seen.has(key)) return false;+ seen.add(key);+ return true;+ });}export function isValidDate(s: string): boolean {diff --git a/test/timeline-citations-inline-code.test.ts b/test/timeline-citations-inline-code.test.tsnew file mode 100644index 0000000000..c85bdcd61e--- /dev/null+++ b/test/timeline-citations-inline-code.test.ts@@ -0,0 +1,80 @@+/**+ * #8893: a citation-derived timeline summary keeps the text of its inline code.+ *+ * Protects: the summary of `Shipped \`build-42.tar\` [Source: …]` names the+ * file (backticks dropped, code text never emphasis-unwrapped), fenced code+ * still never reaches a summary, a citation inside code still files nothing,+ * and a stored row written by the code-masking reading is retired on+ * reconcile instead of lingering next to the new row.+ * Fails when: the summary is built from the code-masked text (inline code+ * stored as blanks), or the masked reading is not listed as superseded.+ * Seams: none; pure parser plus real PGLite for the retraction.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { parseInlineCitationTimelineEntries, supersededInlineCitationEntries } from '../src/core/timeline-citations.ts';+import { retractRemovedTimelineEntries } from '../src/core/timeline-extract.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { cpuMs } from './helpers/cpu-time.ts';++const tick = '`';++describe('#8893 inline code in citation summaries', () => {+ test('inline code text stays in the summary without its backticks', () => {+ expect(parseInlineCitationTimelineEntries(`Shipped ${tick}build-42.tar${tick} to acme-example. [Source: deploy log, 2026-01-02]`))+ .toEqual([{ date: '2026-01-02', source: 'deploy log', summary: 'Shipped build-42.tar to acme-example.' }]);+ });++ test('code text is not emphasis-unwrapped and double-backtick spans keep their inner backtick', () => {+ expect(parseInlineCitationTimelineEntries(`Renamed ${tick}__init__.py${tick} and ${tick}${tick} a${tick}b ${tick}${tick}. [Source: review, 2026-01-03]`).map(e => e.summary))+ .toEqual([`Renamed __init__.py and a${tick}b.`]);+ });++ test('a fenced block next to a cited paragraph never enters the summary', () => {+ const body = `Deployed the fix. [Source: deploy log, 2026-01-04]\n${tick.repeat(3)}\nsecret-looking build output\n${tick.repeat(3)}`;+ expect(parseInlineCitationTimelineEntries(body).map(e => e.summary)).toEqual(['Deployed the fix.']);+ });++ test('a citation written inside inline code still files nothing and never enters a summary', () => {+ expect(parseInlineCitationTimelineEntries(`Use ${tick}[Source: x, 2026-01-02]${tick} literally.`)).toEqual([]);+ expect(parseInlineCitationTimelineEntries(`Quoted ${tick}[Source: x, 2026-01-02]${tick} and ${tick}<!-- note -->${tick}. [Source: style guide, 2026-01-05]`).map(e => e.summary))+ .toEqual(['Quoted and .']);+ });++ test('a long code span with a leading space parses in linear time (remote put_page bodies)', () => {+ const body = `Intro [Source: A, 2026-01-01]\n\nx ${tick} ${'a'.repeat(200_000)}${tick} y [Source: B, 2026-01-02]`;+ expect(cpuMs(() => { parseInlineCitationTimelineEntries(body); supersededInlineCitationEntries(body); })).toBeLessThan(1_000);+ }, 60_000);++ test('restored code never completes citation or comment markup that continues outside the span', () => {+ expect(parseInlineCitationTimelineEntries(`See ${tick}[Sour${tick}ce: X, 2026-03-03] now [Source: B, 2026-02-02]`).map(e => e.summary)).toEqual(['See ce: X, 2026-03-03] now']);+ expect(parseInlineCitationTimelineEntries(`Note ${tick}<!${tick}-- hidden tail [Source: B, 2026-02-02]`).map(e => e.summary)).toEqual(['Note -- hidden tail']);+ });++ test('placeholder characters already in the text are never replaced by code', () => {+ expect(parseInlineCitationTimelineEntries(`Literal \uE0000\uE001 then ${tick}code${tick} [Source: B, 2026-02-02]`).map(e => e.summary))+ .toEqual(['Literal 0 then code']);+ });++ test('the code-masking reading is reported as superseded only where the summary changed', () => {+ expect(supersededInlineCitationEntries(`Shipped ${tick}build-42.tar${tick} to acme-example. [Source: deploy log, 2026-01-02]`))+ .toEqual([{ date: '2026-01-02', source: 'deploy log', summary: 'Shipped to acme-example.' }]);+ expect(supersededInlineCitationEntries('Shipped the beta. [Source: deploy log, 2026-01-02]')).toEqual([]);+ });+});++describe('#8893 a row the code-masking reading stored is retired on reconcile', () => {+ const text = `Shipped ${tick}build-42.tar${tick} to acme-example. [Source: deploy log, 2026-01-02]`;+ let engine: PGLiteEngine;+ beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 60_000);+ afterAll(async () => { await engine.disconnect(); }, 60_000);++ test('the masked-summary row is retracted and the current row stays', async () => {+ await engine.putPage('companies/acme-example', { type: 'company', title: 'Acme Example', compiled_truth: text, timeline: '' });+ await engine.addTimelineEntry('companies/acme-example', { date: '2026-01-02', source: 'deploy log', summary: 'Shipped to acme-example.', detail: 'Source: deploy log' });+ await engine.addTimelineEntry('companies/acme-example', { date: '2026-01-02', source: 'deploy log', summary: 'Shipped build-42.tar to acme-example.', detail: 'Source: deploy log' });+ const retired = await retractRemovedTimelineEntries(engine, 'companies/acme-example', 'default', text);+ expect(retired.map(r => r.summary)).toEqual(['Shipped to acme-example.']);+ const left = await engine.executeRaw<{ summary: string }>('SELECT summary FROM timeline_entries');+ expect(left.map(r => r.summary)).toEqual(['Shipped build-42.tar to acme-example.']);+ });+});