Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(redaction): keep link and graph endpoint slugs exact in retrieval output

garrytana500330ff83a
diff --git a/docs/architecture/key-files/core-search-2.md b/docs/architecture/key-files/core-search-2.md
index 23df17085a..da156ac8f9 100644
--- a/docs/architecture/key-files/core-search-2.md
+++ b/docs/architecture/key-files/core-search-2.md
@@ -24,7 +24,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/search/segmented-evidence.ts` — Wave B PR 2 (spec 7.4): the windows an evidence block of a segmented page is cut from. `readSegmentedWindows` opens the body (`page-body/read-range.ts`), places each hit from its chunk's occurrence and segment-relative projected span through the projected layout (a span is trusted only while it still holds the chunk's text; a hit whose chunk has no span is placed by a unique exact match inside its own segment, else positionally), and reads with `readBodyRange` (projected space, the remote read projection with the per-segment withdrawal and fence overlays) only the hits' neighbor chunks (`window`), the hit's segment (`section`) or the whole text while within the inline read size (`page`), widened to whole lines and merged where they meet. Returns the runs in document order with chunk spans in UTF-16; `evidence-delivery.ts` `planSegmentedBlock` joins them. Never builds a page text or calls `locateChunks`. Tests: `test/large-pages/segmented-evidence.test.ts` (privacy, projection parity, separate windows), `test/large-pages/segmented-evidence-read-cost.test.ts` (40 MiB page: at most two segment texts loaded per hit).
- `src/core/search/explain-target.ts` — `explain_target` diagnosis. A `TargetTrace` (HybridSearchOpts.explainTarget) records the target page's presence and 1-based rank at each pipeline stage: every arm instance (`arm:<label>`, plus `arm:keyword_raw`/`arm:title_raw` before relaxed-row demotion), `fused`, `deduped`, `reranked`, `return_pool`, `limit_slice`, `token_budget`. `diagnoseTrace` returns one state (`retrieved` / `dropped` / `reinjected` / `not_retrieved` / `not_recorded`) naming the first stage that lost the page; boost stages are never reported as drops (they only re-order). `probeTarget` + `diagnoseProbe` run one read under the caller's read policy and separate not-indexed / stale projection / unsealed safe chunks / ambiguous slug; an unreadable page and an absent page return the same `target_not_found_or_not_visible`. Every diagnosis carries an Action-contract fix the `search`/`query` ops emit as a notice. Pinned by `test/search/explain.test.ts`.
-- `src/core/search/output-redaction.ts` — copies retrieval results and metadata before output sanitation, sharing the canonical `secret-scan.ts` plan and echo dictionary. Keyword, hybrid, image, exact and CRAG paths sanitize after internal grading/capture and before snippet truncation; CLI search/query JSON, explain and human rendering repeat the defense-in-depth pass. Every field is planned with `highEntropy: true`, so `KEY=value` assignments redact alongside the named shapes. Identity fields (`id`, `slug`, `source_id`, `page_id`, `chunk_id`, `message_id`, `thread_id`, `graph_session_prefix`, relational seed/path, `superseded_by`, `fact_id`, `entity_slug`) remain byte-identical: never put secrets in IDs. By default each field is capped at 64 Ki UTF-16 characters, the response at 1 Mi characters/8192 text fields, and nesting at 24 levels; oversized/unscannable text becomes `<REDACTED:output_limit>`. `RetrievalRedactionOpts.uncapped` scans every field whole (no `output_limit` replacement; one echo dictionary, every field planned before any write; the depth cap stays) and `verbatim` copies the named keys of each top-level result unscanned. `withOutputRedaction(op)` is the registration-time wrapper behind `Operation.outputRedaction` (`src/core/ops/contract.ts`): `operations.ts` wraps every `'retrieval'` op's handler once, so the CLI, stdio and HTTP MCP, subagent tools and `gbrain call` all pass the whole response (early returns included) through the uncapped redactor, linear in output size; `{ retrieval: { localVerbatim } }` leaves the named keys raw only when `ctx.remote === false`. The memory verbs: `recall`, `context_pack` and `delta` redact their presentation sets with one echo dictionary before packing, so budgets, `budget_used` and the delta cursor follow the delivered text, and their rendered `text` is never replaced by `output_limit`; facts (`fact`, `context`, `source`) are `localVerbatim`, so a remembered credential is readable raw only by the trusted local CLI on the brain host (stdio MCP and thin clients are remote). Handlers that clip or summarize stored text (transcript summaries, entity-card and compiled-context excerpts) redact the whole field before clipping. THREAT MODEL: this defends against accidental exposure of stored credentials into agent context and transcripts through retrieval ops. It does not cover the raw-read exemptions (`get_page`, `fetch`, `get_chunks`, `get_raw_data`, `get_versions`, the skill catalog, admin job ops), stored content (unchanged), or configured providers (embeddings at ingest, the hosted reranker, synthesis/think generation still receive unredacted text, except private keys, which the credential-safe projection keeps out of chunks). Unknown formats and short fragments can escape detection. This is output hygiene, not authorization or provider/eval no-egress. Tests: `test/search/output-redaction.serial.test.ts`, `test/search/output-redaction.test.ts`, `test/retrieval-op-redaction.test.ts` (every `'retrieval'` and `'no_stored_text'` op run through MCP dispatch with `remote = true` against a planted secret, plus the registration wrapper), `test/transcripts-redaction.test.ts`.
+- `src/core/search/output-redaction.ts` — copies retrieval results and metadata before output sanitation, sharing the canonical `secret-scan.ts` plan and echo dictionary. Keyword, hybrid, image, exact and CRAG paths sanitize after internal grading/capture and before snippet truncation; CLI search/query JSON, explain and human rendering repeat the defense-in-depth pass. Every field is planned with `highEntropy: true`, so `KEY=value` assignments redact alongside the named shapes. Identity fields (`id`, `slug`, `source_id`, `page_id`, `chunk_id`, `message_id`, `thread_id`, `graph_session_prefix`, relational seed/path, `superseded_by`, `fact_id`, `entity_slug`, and the link and graph endpoints `from_slug`, `to_slug`, `origin_slug` and their `*_source_id`) remain byte-identical: never put secrets in IDs. By default each field is capped at 64 Ki UTF-16 characters, the response at 1 Mi characters/8192 text fields, and nesting at 24 levels; oversized/unscannable text becomes `<REDACTED:output_limit>`. `RetrievalRedactionOpts.uncapped` scans every field whole (no `output_limit` replacement; one echo dictionary, every field planned before any write; the depth cap stays) and `verbatim` copies the named keys of each top-level result unscanned. `withOutputRedaction(op)` is the registration-time wrapper behind `Operation.outputRedaction` (`src/core/ops/contract.ts`): `operations.ts` wraps every `'retrieval'` op's handler once, so the CLI, stdio and HTTP MCP, subagent tools and `gbrain call` all pass the whole response (early returns included) through the uncapped redactor, linear in output size; `{ retrieval: { localVerbatim } }` leaves the named keys raw only when `ctx.remote === false`. The memory verbs: `recall`, `context_pack` and `delta` redact their presentation sets with one echo dictionary before packing, so budgets, `budget_used` and the delta cursor follow the delivered text, and their rendered `text` is never replaced by `output_limit`; facts (`fact`, `context`, `source`) are `localVerbatim`, so a remembered credential is readable raw only by the trusted local CLI on the brain host (stdio MCP and thin clients are remote). Handlers that clip or summarize stored text (transcript summaries, entity-card and compiled-context excerpts) redact the whole field before clipping. THREAT MODEL: this defends against accidental exposure of stored credentials into agent context and transcripts through retrieval ops. It does not cover the raw-read exemptions (`get_page`, `fetch`, `get_chunks`, `get_raw_data`, `get_versions`, the skill catalog, admin job ops), stored content (unchanged), or configured providers (embeddings at ingest, the hosted reranker, synthesis/think generation still receive unredacted text, except private keys, which the credential-safe projection keeps out of chunks). Unknown formats and short fragments can escape detection. This is output hygiene, not authorization or provider/eval no-egress. Tests: `test/search/output-redaction.serial.test.ts`, `test/search/output-redaction.test.ts`, `test/retrieval-op-redaction.test.ts` (every `'retrieval'` and `'no_stored_text'` op run through MCP dispatch with `remote = true` against a planted secret, plus the registration wrapper), `test/transcripts-redaction.test.ts`.
- `src/core/search/return-policy.ts` (default OFF) — intent-aware adaptive return-sizing. Pure, dependency-light module that trims the final ranked candidate set to an intent-driven cap instead of returning the full top-K. `entity` intent gets a tight cap; `temporal`/`event`/`general` get a recall-preserving cap (`concept` is coerced to `general` by `hybrid.ts` before the call — concept queries want breadth). A `minKeep` failsafe (≥1) guarantees a human never gets a silent blank when candidates exist. WHY a cap, not a score-cliff detector: PrecisionMemBench instrumentation (gbrain-evals) measured the rank1→rank2 RRF gap is ~identical whether rank-1 is correct (0.602) or wrong (0.569) — mechanical decay, not a separatrix; rank-1 is right in 94% of single-answer cases, so "return a tight set" is the whole win and cliff-cutting just adds noise. Exports `AdaptiveReturnConfig`, `DEFAULT_ADAPTIVE_RETURN` (frozen: `enabled=false`, `entityMax=2`, `otherMax=6`, `minKeep=1`), `AdaptiveReturnDecision` (`{applied, intent, cap, kept, total}`), `AdaptiveReturnInput` (`boolean | Partial<AdaptiveReturnConfig> | undefined`), `adaptiveReturnFromConfig(cfg)`, `resolveAdaptiveReturn(perCall, fromConfig)` (defaults → config → per-call merge), `adaptiveReturnEnabled(...)` (cache-skip gate check), `applyAdaptiveReturn(results, intent, cfg)` (the trim). Config knobs (DB or file plane): `search.adaptive_return` (master switch), `search.adaptive_return_entity_max`, `search.adaptive_return_other_max`, `search.adaptive_return_min_keep` (each clamped ≥1). Wired into `hybridSearch` AFTER `applyReranker`, BEFORE the `limit` slice, and ONLY on the first page (`offset===0`) — paginating a confidence-gated set is incoherent, so paginated calls fall through to the fixed limit. Stamps the decision onto `HybridSearchMeta.adaptive_return` for `gbrain search --explain`. `hybridSearchCached` SKIPS the cache when the gate is on (a trimmed set must not be served to a gate-off lookup and vice versa). `SearchOpts.adaptiveReturn` + `HybridSearchMeta.adaptive_return` declared in `src/core/types.ts`. Agent-facing: the `query` op (`src/core/operations.ts`) exposes an `adaptive_return` boolean param whose description instructs the agent WHEN to set it (single-answer → on; breadth/exploration → off; pass `limit:1` for a hard single-answer cap), threaded into `hybridSearchCached` — end users never touch the config knob; their agent decides per query (same pattern as `salience`/`recency`). Pinned by `test/search/return-policy.test.ts` (mechanism) + `test/search/query-op-adaptive-return.test.ts` (agent surface: param exists + description teaches both directions + the never-empty contract).
diff --git a/src/core/search/output-redaction.ts b/src/core/search/output-redaction.ts
index e8ce86d72b..01ae1d9069 100644
--- a/src/core/search/output-redaction.ts
+++ b/src/core/search/output-redaction.ts
@@ -9,7 +9,7 @@ export const OUTPUT_REDACTION_LIMIT = '<REDACTED:output_limit>';
const IDENTITY_FIELDS = new Set([
'id', 'slug', 'source_id', 'page_id', 'chunk_id', 'message_id', 'thread_id',
'graph_session_prefix', 'relational_seed', 'relational_path', 'superseded_by',
- 'fact_id', 'entity_slug',
+ 'fact_id', 'entity_slug', 'from_slug', 'to_slug', 'origin_slug', 'from_source_id', 'to_source_id', 'origin_source_id',
]);
const MAX_DEPTH = 24;
const MAX_TEXT_FIELDS = 8192;
diff --git a/test/search/output-redaction-link-slugs-8545.test.ts b/test/search/output-redaction-link-slugs-8545.test.ts
new file mode 100644
index 0000000000..2a0c1fa72d
--- /dev/null
+++ b/test/search/output-redaction-link-slugs-8545.test.ts
@@ -0,0 +1,51 @@
+/**
+ * #8545: link and graph endpoint slugs are identities, like `slug`: a slug
+ * that happens to look like a credential (`pa-` plus 20 characters reads as a
+ * Voyage key) must come back exact from get_links, get_backlinks and
+ * traverse_graph, or the caller cannot follow the edge. Context text is still
+ * scanned. PGLite, remote MCP dispatch, synthetic slugs.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
+import { redactRetrievalOutput } from '../../src/core/search/output-redaction.ts';
+import { dispatchToolCall } from '../../src/mcp/dispatch.ts';
+import { withEnv } from '../helpers/with-env.ts';
+
+const FROM = 'notes/alice-example';
+const TO = 'districts/pa-synthetic-example-district-1234';
+const REMOTE = { remote: true, transport: 'http' as const, sourceId: 'default' };
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ await engine.putPage(FROM, { type: 'note', title: 'Alice example', compiled_truth: 'Notes on a district.' } as never);
+ await engine.putPage(TO, { type: 'note', title: 'Synthetic district', compiled_truth: 'A district page.' } as never);
+ await engine.addLink(FROM, TO, 'see also', 'mentions', 'manual');
+}, 60_000);
+
+afterAll(async () => { await engine?.disconnect(); });
+
+const call = (tool: string, args: Record<string, unknown>) =>
+ withEnv({ GBRAIN_BACKUP_CHECK: '0' }, () => dispatchToolCall(engine, tool, args, REMOTE));
+const body = (res: { content: Array<{ text: string }> }) => res.content.map(c => c.text).join('\n');
+
+describe('#8545 link and graph slugs survive output redaction', () => {
+ test('redactRetrievalOutput keeps from_slug, to_slug and origin_slug exact and still scans context', () => {
+ const { results } = redactRetrievalOutput([{ from_slug: FROM, to_slug: TO, origin_slug: TO, from_source_id: 'default', to_source_id: 'default',
+ origin_source_id: 'default', context: 'x', links: [{ to_slug: TO, link_type: 'mentions' }] }], {});
+ expect(results[0]).toMatchObject({ from_slug: FROM, to_slug: TO, origin_slug: TO, links: [{ to_slug: TO }] });
+ const key = ['pa', 'Zq81LmX0vR4tYw7NcB2KfJ6'].join('-');
+ expect(redactRetrievalOutput([{ to_slug: TO, context: key }], {}).results[0]).toMatchObject({ to_slug: TO, context: '<REDACTED:voyage>' });
+ });
+
+ test('get_links, get_backlinks and traverse_graph over MCP return the endpoint slug', async () => {
+ for (const [tool, args] of [['get_links', { slug: FROM }], ['get_backlinks', { slug: TO }], ['traverse_graph', { slug: FROM, depth: 1 }]] as const) {
+ const text = body(await call(tool, args));
+ expect(text).toContain(TO);
+ expect(text).not.toContain('<REDACTED:');
+ }
+ });
+});