Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

feat(review): near-name flags and unattributed facts in extraction_pending (#2942, #2947 L1)

garrytandbd500308bd4
diff --git a/docs/TOOL_CATALOG.md b/docs/TOOL_CATALOG.md
index c5a85e00c9..89ccb351d5 100644
--- a/docs/TOOL_CATALOG.md
+++ b/docs/TOOL_CATALOG.md
@@ -61,7 +61,7 @@ Every non-localOnly operation on the MCP surface: 162 tools across 25 areas. **S
|---|---|---|---|---|
| `entity_identity_list` | List cross-source entity identity groups and their member pages. | read | | |
| `extract_entities` | Extract entity names (people, companies) from text and create/update their brain stub pages. | write | | |
-| `extraction_pending` | List unverified auto-extracted entity stubs awaiting owner review (the quarantine lane from extract_entities). | read | | |
+| `extraction_pending` | List what waits for review: unverified auto-extracted entity stubs (pending; promote or reject them with extraction_review), facts saved without an entity (unattributed; relink each with remember entity + replaces, never forget first), and, when include names it, recently created entity pages whose name is one edit or one acronym letter from another entity (possible_duplicates; flag only, never merged). | read | | |
## facts
diff --git a/docs/architecture/key-files/core-memory.md b/docs/architecture/key-files/core-memory.md
index c4582623af..935b56a796 100644
--- a/docs/architecture/key-files/core-memory.md
+++ b/docs/architecture/key-files/core-memory.md
@@ -64,3 +64,5 @@ notice). User guide: [core memory](../../guides/core-memory.md).
`test/remember-batch-entity-7432.test.ts`,
`test/trust-cache-invalidation.test.ts` (core lane after trust transitions),
`test/e2e/core-memory-locks-postgres.test.ts`.
+- `src/core/entities/near-name.ts` — Flag-only near-name matcher for #2942 (never merges, never writes `entity_identities`; `entity-identity.ts` stays manual-only). `classifyNearName` returns `same` (normalized-equal after corporate suffixes, or a strong `name-variants.ts` shape), `typo` (Damerau distance 1 on 3–64 normalized characters) or `acronym` (one word equals a multi-word name's initials, at most one letter different). Candidates come from symmetric one-deletion keys plus initials keys, so recall is complete for those kinds; past `NEAR_NAME_MAX_KEYS_PER_SHARD` the index splits into shards by key hash (a pair shares a key, so it lands whole in one shard). `loadEntityNames` reads one source's fact-entity pages plus the source pack's linkable types, leaving out private pages for callers that exclude them. Pinned by `test/review-lists.test.ts`.
+- `src/core/review-lists.ts` — The two lists `extraction_pending` adds beside `pending`: `unattributed` (active facts with no entity, newest first, world-only for remote callers, each with a `remember` + `entity` + `replaces` relink action, never a forget) and opt-in `possible_duplicates` (entity pages created in the last `since_days` whose name is near another entity's name or alias; pairs in `aliases` or `distinct_from` are skipped). The duplicates cursor walks recent pages within a shard, then the next shard, and reports `complete` only after the last shard; index builds are serialized per process. Pinned by `test/review-lists.test.ts` and the Postgres parity file.
diff --git a/docs/guides/entity-detection.md b/docs/guides/entity-detection.md
index 16813176f1..e61d92f654 100644
--- a/docs/guides/entity-detection.md
+++ b/docs/guides/entity-detection.md
@@ -183,6 +183,19 @@ to Alice" only works if Alice's page links back to every mention.
Variant spellings, nicknames, and company abbreviations cause duplicates.
"Alice Example" and "Alice" might be the same person.
+6. **Review near names, never auto-merge them.** A misheard name ("ACMX" for
+ "ACME") becomes a new page beside the real one. `gbrain extraction-pending
+ --include possible_duplicates` (MCP `extraction_pending` with `include:
+ ["possible_duplicates"]`) lists entity pages created in the last 30 days
+ whose name is one edit, or one acronym letter, away from another entity's
+ name or alias, whoever wrote them. It only flags: link the pair with
+ `entity-identity-link` (owner, manual), or add the name to the other
+ page's `aliases` frontmatter, which silences the pair. A page whose
+ `distinct_from` frontmatter lists the other slug is never flagged. The
+ same command lists facts saved without an entity (`unattributed`); relink
+ one with `remember` plus `entity` and `replaces`, never by forgetting it
+ first.
+
## How to Verify
First leave capture off and send a substantive message: verify no write occurs.
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index bfa01bceb3..9d238a5d26 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -47,4 +47,4 @@ src/commands/doctor/checks/extraction-sync.ts 1500 ratchet +7 GBRA-130 A-1: link
src/core/search/evidence-delivery.ts 1733 ratchet +5 W5: EvidencePlan.rankCut resolved with the plan (logic in search/page-rank-cut.ts); +20 W5 whole_pages: wholeCost prices a block as delivery would emit it whole (needs the private Block/enrichBlock internals) and the cut is wired before allocate (cutWholePages lives in search/page-rank-cut.ts); new row: crossed the 1500 unlisted cap at the #6367 merge (master 1479, its explicit-budget cap) by the C1 date header threaded through every block path, cap included; pagePlanHits (search/page-plan-hits.ts) and the header loader (evidence-date.ts blockDateHeaders) were peeled first; peel candidate: allocateCapped; +8 at the #6396 merge: page evidence text cut through the fence-chunk trust overlay and the caller minTrust floor (re-summed: our 1504 + master +8); +119 Cat 40 next wave F2: the search.long_page_unit policy (withLongPagePolicy, selectLongPages size preflight with page/byte/deadline limits) and the long-page fetch and fallback in deliverEvidence, kept in the shared seam the plan names (gbrain-evals docs/plans/2026-10-11-cat40-next-wave, F2); +17 F2 rework to the frozen-comparison policy: search.long_page_min_chars (rendered-body characters, default 20,000) read in withLongPagePolicy, a character count in the preflight, the rendered-body confirmation in deliverEvidence, and the long-page row exemption in capDeliveredSnippets (re-summed at the 0.60.225.0 merge: master 1537 + F2 136); +39 Wave B PR 2 lane 2b: planSegmentedBlock joins the windows search/segmented-evidence.ts reads around a segmented page's hits into one run-marked document (needs the private Doc/Piece/buildDoc/locateAnchor internals), planBlock's unit cut split into blockFromDoc for both, sections bounded by their run, selectionCost counts a gap between runs, and the fetch skips the whole-body fence overlay for a segmented page; +4 segmented pages read at once before planning (segmentedReads; the read is search/segmented-evidence.ts, the statements page-body/read-range.ts); +17 GBRA-156: tail trim plan flag, DeliverOptions.query, DeliveryMeta.tail_trim, the trim call and the frozen-hit score/query library fields (logic in search/evidence-tail-trim.ts; interface fields cannot move)
src/commands/config.ts 1524 ratchet new row: crossed the 1500 unlisted cap in wave 15 P7.4 #5461: set-time validation of think.pages_budget_chars / think.excerpt_ceiling_chars (range + parser live in core/think/pages-budget.ts), mirroring the cycle.consolidate.cluster_threshold hook; +1 wave 16 G5.1 (#6495): set-time refusal hook for search.keyword_statement_timeout_ms (check lives in commands/config/keyword-timeout-value.ts); +1 wave 18b lane 5 (#5012 item 6): search.contextual_retrieval prints the existing-pages hint after the write (text lives in commands/config/contextual-mode-hint.ts); +1 #6712 PR 2: retired/rollback sync key dispatch line (logic in core/sync-config-keys.ts handleSyncKeyConfig); +1 #6865: sync.slug_conflicts.auto_remove boolean validation at set time (refusal text in repair/slug-autoheal.ts); +1 #6948: sync.origin_twins.auto_resolve boolean validation at set time (refusal text in repair/origin-twins-heal.ts); +2 #6795: config set chat_model/expansion_model help line and the model-pin import (shadow report and file-pin unset live in commands/config/model-pin-shadow.ts); +1 #6712 PR 4b: sync.admission set-time hook (value logic in commands/config/sync-admission-value.ts); +4 #7130: set-time refusal hook for sync.max_deletes / sync.max_delete_ratio (and .source.<id>) / purge.retention_hours (check lives in core/sync-reconcile.ts deleteSafetyConfigRefusal); +1 GBRA-150: sync.metafile_pages set-time route (body in config/sync-metafile-pages-value.ts); re-summed at the v0.60.330.0 merge (1520 measured); +2 #7242 minions.terminal_hook file-plane key and its set branch (logic in minions/terminal-hook.ts); +1 GBRA-161: generations.* set-time route (refusal in commands/config/generations-values.ts); +1 GBRA-207 #7630: sync.repo_path set-time refusal hook (refusal in core/sync-config-keys.ts syncRepoPathRefusal)
src/commands/mcp-expose.ts 1512 ratchet hosted_option: Session carries the gbrain.io coaching notice on the first-receipt run and Tailscale/service friction exits (copy, refs and gating live in core/onboard/hosted-option.ts)
-src/core/behavior-change-notice.ts 1518 ratchet the BEHAVIOR_CHANGES table grows by one row per disclosed change: +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2); the behavior-change table split (#7607) retires this row
+src/core/behavior-change-notice.ts 1519 ratchet +1 GBRA-215 L1: extraction_pending review lists row; the BEHAVIOR_CHANGES table grows by one row per disclosed change: +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2); the behavior-change table split (#7607) retires this row
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 59b8c1aa07..8b0cdf9e21 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1320,6 +1320,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.471.0', id: 'door_planned_state', text: 'On a source with `canonical: git`, a page write now commits exactly the bytes it published, or nothing. A file edited on disk while the write was publishing refuses `git_target_changed`: nothing is committed, the edit stays on disk and the source\'s writes pause until `gbrain repair stuck-recovery` resolves it (before, Git committed the edit while the database recorded the write\'s content). A staged change of the page\'s file refuses `git_index_conflict`, with the index untouched; commit or unstage it, then retry (before, the write replaced the staged change in the index). When the index still holds the version from before an earlier gbrain commit whose index update failed, the next write restores it and says so with a `git_index_restored` notice.' },
// #7679 / #7685 orphaned worktree refresh. `since` is stamped at the merge slot.
{ since: '0.60.472.0', id: 'refresh_orphan_7679', text: 'A managed `gbrain sources refresh` whose process died (for example a container replaced mid-refresh) no longer wedges the checkout: the next plain `gbrain sources refresh` on the owner host finishes the orphaned refresh through the `--resume` path before its own refresh and reports it as `recovered`, and the owner completes an orphaned `syncing` refresh once every member has synced to its target. A refresh whose holder is still running, or is recorded on another host, is never taken over. `gbrain sources writer status` and `gbrain sync status` now name the holder and its `liveness` (`live`, `orphaned`, `unknown`) with a fix naming `--resume`. With `--json`, a refusal that carries a structured fix (`refresh_in_progress`, `refresh_diverged`, a git read failure) now prints `fix` as an object (argv, actor, next) instead of a string; the prose stays in `suggestion`, so read that when a script needs text. A plain refresh of a checkout whose interrupted refresh needs a reset decision now refuses `refresh_recovery_required` (it refused `refresh_in_progress`).' },
+ { since: 'TBD', id: 'extraction_pending_review_lists', text: '`extraction_pending` (CLI `gbrain extraction-pending`) now also returns `unattributed`: active facts saved without an entity, newest first (world facts only for remote callers), each with the relink action (remember the same claim with `entity` and `replaces`). `pending` and `count` are unchanged; the result adds `include` and `unattributed`. Pass `include` to choose lists, and add `possible_duplicates` to it to list entity pages created in the last 30 days whose name is one edit or one acronym letter away from another entity, for review only (nothing is merged).' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/entities/near-name.ts b/src/core/entities/near-name.ts
new file mode 100644
index 0000000000..fe5d2b2306
--- /dev/null
+++ b/src/core/entities/near-name.ts
@@ -0,0 +1,186 @@
+/**
+ * #2942: flag a never-seen entity name that sits next to an existing one (flag only, never merge).
+ *
+ * A transcription slip ("ACMX" for "ACME") mints a new entity page beside the real one. This module finds
+ * those pairs so a person can decide: link them (entity_identity_link, still manual-only), add an alias, or
+ * mark them distinct. It never writes, never merges and never touches entity_identities.
+ *
+ * Kinds:
+ * - same: the names normalize equal, or the entity card treats them as one name (honorific, reordered,
+ * diminutive, initial). Offered as an alias, never reported as a different entity.
+ * - typo: Damerau distance exactly 1 between normalized names of 3 to 64 characters.
+ * - acronym: a one-word name equals the initials of a multi-word name, with at most one letter different.
+ *
+ * Candidates come from symmetric one-deletion keys (a distance-1 pair always shares a key) plus initials
+ * keys, so recall is complete within those kinds. Work is bounded: when the key count passes the budget the
+ * index is split into shards by key hash (a pair shares a key, so it lands whole in one shard) and callers
+ * walk the shards with a cursor.
+ */
+
+import type { BrainEngine } from '../engine.ts';
+import { nameWords, matchNameVariant, editDistance, type NameMatchKind } from './name-variants.ts';
+import { isFactEntityPage } from './resolve.ts';
+import { privatePagesFilterFragment } from '../search/private-visibility.ts';
+
+export type NearNameKind = 'same' | 'typo' | 'acronym';
+
+const CORPORATE_SUFFIXES: ReadonlySet<string> = new Set([
+ 'inc', 'incorporated', 'llc', 'ltd', 'limited', 'corp', 'corporation', 'co', 'company', 'labs', 'lab', 'gmbh',
+ 'plc', 'ag', 'sa', 'bv', 'the',
+]);
+const SAME_VARIANTS: ReadonlySet<NameMatchKind> = new Set(['normalized', 'honorific', 'reordered', 'diminutive', 'initial']);
+export const NEAR_NAME_MIN_CHARS = 3;
+export const NEAR_NAME_MAX_CHARS = 64;
+
+/** Name words without corporate suffixes; a name that is only suffixes keeps them. */
+export function entityNameWords(name: string): string[] {
+ const words = nameWords(name);
+ const core = words.filter(w => !CORPORATE_SUFFIXES.has(w));
+ return core.length ? core : words;
+}
+
+export function normalizeEntityName(name: string): string {
+ return entityNameWords(name).join('');
+}
+
+function initialsOf(words: string[]): string | null {
+ return words.length >= 2 ? words.map(w => w[0]).join('') : null;
+}
+
+/** How two entity names relate, or null when they are unrelated. */
+export function classifyNearName(a: string, b: string): NearNameKind | null {
+ const wa = entityNameWords(a), wb = entityNameWords(b);
+ const na = wa.join(''), nb = wb.join('');
+ if (!na || !nb) return null;
+ if (na === nb) return 'same';
+ const variant = matchNameVariant(a, b) ?? matchNameVariant(b, a);
+ if (variant && SAME_VARIANTS.has(variant)) return 'same';
+ const checkable = (n: string) => n.length >= NEAR_NAME_MIN_CHARS && n.length <= NEAR_NAME_MAX_CHARS;
+ if (checkable(na) && checkable(nb) && editDistance(na, nb, 1) === 1) return 'typo';
+ const acronym = (single: string[], multi: string[]) => {
+ const initials = initialsOf(multi);
+ return single.length === 1 && initials !== null && single[0]!.length >= NEAR_NAME_MIN_CHARS
+ && initials.length === single[0]!.length && editDistance(single[0]!, initials, 1) <= 1;
+ };
+ if (acronym(wa, wb) || acronym(wb, wa)) return 'acronym';
+ return null;
+}
+
+function deletionKeys(s: string): string[] {
+ const keys = new Set<string>([s]);
+ for (let i = 0; i < s.length; i++) keys.add(s.slice(0, i) + s.slice(i + 1));
+ return [...keys];
+}
+
+/** Every index key a name contributes: one-deletion keys of the normalized name, and of the initials or acronym. */
+export function nameKeys(name: string): string[] {
+ const words = entityNameWords(name);
+ const normalized = words.join('');
+ if (normalized.length < NEAR_NAME_MIN_CHARS || normalized.length > NEAR_NAME_MAX_CHARS) return [];
+ const keys = deletionKeys(normalized).map(k => `n:${k}`);
+ const acronym = words.length === 1 ? words[0]! : initialsOf(words);
+ if (acronym && acronym.length >= NEAR_NAME_MIN_CHARS && acronym.length <= 12) keys.push(...deletionKeys(acronym).map(k => `a:${k}`));
+ return keys;
+}
+
+function keyShard(key: string, shards: number): number {
+ let h = 2166136261;
+ for (let i = 0; i < key.length; i++) h = Math.imul(h ^ key.charCodeAt(i), 16777619);
+ return (h >>> 0) % shards;
+}
+
+export interface EntityName { slug: string; name: string; created_at?: string }
+
+export interface NearNameIndex {
+ shard: number;
+ shards: number;
+ entries: EntityName[];
+ keys: Map<string, number[]>;
+}
+
+/** Build one shard of the index. A pair of names sharing a key is in exactly the shard that key hashes to. */
+export function buildNearNameIndex(names: EntityName[], shard = 0, shards = 1): NearNameIndex {
+ const keys = new Map<string, number[]>();
+ names.forEach((entry, i) => {
+ for (const key of nameKeys(entry.name)) {
+ if (shards > 1 && keyShard(key, shards) !== shard) continue;
+ const list = keys.get(key);
+ if (list) { if (list[list.length - 1] !== i) list.push(i); } else keys.set(key, [i]);
+ }
+ });
+ return { shard, shards, entries: names, keys };
+}
+
+/** Count the keys a name set would produce, to pick a shard count before building. */
+export function countNameKeys(names: EntityName[]): number {
+ let n = 0;
+ for (const entry of names) n += nameKeys(entry.name).length;
+ return n;
+}
+
+export interface NearNameMatch { slug: string; name: string; kind: Exclude<NearNameKind, 'same'> | 'same' }
+
+/** Names in this shard that are near `name` (any kind, `same` included), excluding `excludeSlug`. */
+export function queryNearNames(index: NearNameIndex, name: string, excludeSlug?: string): NearNameMatch[] {
+ const seen = new Set<number>();
+ const out: NearNameMatch[] = [];
+ for (const key of nameKeys(name)) {
+ if (index.shards > 1 && keyShard(key, index.shards) !== index.shard) continue;
+ for (const i of index.keys.get(key) ?? []) {
+ if (seen.has(i)) continue;
+ seen.add(i);
+ const entry = index.entries[i]!;
+ if (entry.slug === excludeSlug) continue;
+ const kind = classifyNearName(name, entry.name);
+ if (kind) out.push({ slug: entry.slug, name: entry.name, kind });
+ }
+ }
+ return out;
+}
+
+/** Index keys allowed in one shard; past this the index splits into more shards. */
+export const NEAR_NAME_MAX_KEYS_PER_SHARD = 500_000;
+/** Entity names loaded per source; past this possible_duplicates reports incomplete instead of reading more. */
+export const NEAR_NAME_MAX_ENTITIES = 200_000;
+
+export interface EntityNameLoad { names: EntityName[]; truncated: boolean; aliasesBySlug: Map<string, Set<string>>; distinctBySlug: Map<string, Set<string>> }
+
+/**
+ * The live entity names (title, aliases, slug basename) of one source a caller may read: fact-entity pages
+ * plus the source pack's linkable entity types. Private pages are left out for callers that exclude them,
+ * so a hidden page never shapes what a restricted caller is shown.
+ */
+export async function loadEntityNames(engine: BrainEngine, sourceId: string, opts: { excludePrivate?: boolean; max?: number } = {}): Promise<EntityNameLoad> {
+ const max = opts.max ?? NEAR_NAME_MAX_ENTITIES;
+ const privacy = opts.excludePrivate ? `AND ${privatePagesFilterFragment('p')}` : '';
+ const rows = await engine.executeRaw<{ slug: string; type: string | null; title: string | null; aliases: unknown; distinct_from: unknown; created_at: string }>(
+ `SELECT p.slug, p.type, p.title, p.frontmatter -> 'aliases' AS aliases, p.frontmatter -> 'distinct_from' AS distinct_from,
+ p.created_at::text AS created_at
+ FROM pages p WHERE p.source_id = $1 AND p.deleted_at IS NULL ${privacy}
+ ORDER BY p.id LIMIT $2`,
+ [sourceId, max + 1],
+ );
+ let linkable: Set<string> | null = null;
+ const names: EntityName[] = [];
+ const aliasesBySlug = new Map<string, Set<string>>();
+ const distinctBySlug = new Map<string, Set<string>>();
+ const list = (v: unknown): string[] => Array.isArray(v) ? v.filter((x): x is string => typeof x === 'string') : typeof v === 'string' ? [v] : [];
+ for (const row of rows.slice(0, max)) {
+ if (!isFactEntityPage(row.slug, row.type)) {
+ linkable ??= new Set((await (await import('../mentions/policy.ts')).loadLinkableTypes(engine, sourceId).catch(() => ({ types: [] as string[] }))).types);
+ if (row.type == null || !linkable.has(row.type)) continue;
+ }
+ const aliases = list(row.aliases);
+ const own = new Set<string>([row.title ?? '', row.slug.split('/').pop()!.replace(/-/g, ' '), ...aliases].filter(Boolean));
+ for (const name of own) names.push({ slug: row.slug, name, created_at: row.created_at });
+ aliasesBySlug.set(row.slug, new Set([...own].map(normalizeEntityName)));
+ const distinct = list(row.distinct_from);
+ if (distinct.length) distinctBySlug.set(row.slug, new Set(distinct));
+ }
+ return { names, truncated: rows.length > max, aliasesBySlug, distinctBySlug };
+}
+
+/** Shards needed so no shard passes the key budget. */
+export function shardCountFor(names: EntityName[], maxKeysPerShard = NEAR_NAME_MAX_KEYS_PER_SHARD): number {
+ return Math.max(1, Math.ceil(countNameKeys(names) / maxKeysPerShard));
+}
diff --git a/src/core/entity-identity.ts b/src/core/entity-identity.ts
index 93bd2ebc3c..56bcb9a710 100644
--- a/src/core/entity-identity.ts
+++ b/src/core/entity-identity.ts
@@ -14,6 +14,8 @@
* - MANUAL-ONLY. Rows come exclusively from the entity_identity_link op.
* No auto-matching, no name-similarity heuristics — a wrong identity
* merge silently corrupts retrieval, so v1 keeps a human in the loop.
+ * Near names are only flagged for review (entities/near-name.ts,
+ * extraction_pending possible_duplicates); nothing here is written from them.
* - A page belongs to at most one identity (UNIQUE (source_id, page_id));
* re-linking MOVES it.
* - Retrieval union is OFF by default, gated by the
diff --git a/src/core/operation-manifest.generated.ts b/src/core/operation-manifest.generated.ts
index abb2c1a536..47e9a6568b 100644
--- a/src/core/operation-manifest.generated.ts
+++ b/src/core/operation-manifest.generated.ts
@@ -5456,17 +5456,40 @@ export const OPERATION_MANIFEST: OperationMeta[] = [
"mutating": false,
"idempotent": true,
"outputRedaction": "retrieval",
- "description": "List unverified auto-extracted entity stubs awaiting owner review (the quarantine lane from extract_entities). Promote or reject them with extraction_review.",
+ "description": "List what waits for review: unverified auto-extracted entity stubs (pending; promote or reject them with extraction_review), facts saved without an entity (unattributed; relink each with remember entity + replaces, never forget first), and, when include names it, recently created entity pages whose name is one edit or one acronym letter from another entity (possible_duplicates; flag only, never merged).",
"params": {
"limit": {
"type": "number",
"required": false,
- "description": "Max rows (default 100, cap 500)."
+ "description": "Max rows per list (default 100 for pending, 50 for the others; cap 500)."
},
"offset": {
"type": "number",
"required": false,
- "description": "Pagination offset."
+ "description": "Pagination offset for pending."
+ },
+ "include": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ },
+ "required": false,
+ "description": "Lists to return: pending, unattributed, possible_duplicates. Default pending and unattributed."
+ },
+ "unattributed_offset": {
+ "type": "number",
+ "required": false,
+ "description": "Pagination offset for unattributed (its next_offset)."
+ },
+ "possible_duplicates_cursor": {
+ "type": "string",
+ "required": false,
+ "description": "Continuation for possible_duplicates (its next_cursor)."
+ },
+ "since_days": {
+ "type": "number",
+ "required": false,
+ "description": "possible_duplicates: entity pages created in the last N days (default 30)."
}
},
"scope": "read",
diff --git a/src/core/ops/extraction.ts b/src/core/ops/extraction.ts
index 52aa9e5467..506c61c1fe 100644
--- a/src/core/ops/extraction.ts
+++ b/src/core/ops/extraction.ts
@@ -110,13 +110,18 @@ const extraction_pending: Operation = {
mutating: false,
idempotent: true,
outputRedaction: 'retrieval',
- description: 'List unverified auto-extracted entity stubs awaiting owner review (the quarantine lane from extract_entities). Promote or reject them with extraction_review.',
+ description: 'List what waits for review: unverified auto-extracted entity stubs (pending; promote or reject them with extraction_review), facts saved without an entity (unattributed; relink each with remember entity + replaces, never forget first), and, when include names it, recently created entity pages whose name is one edit or one acronym letter from another entity (possible_duplicates; flag only, never merged).',
params: {
- limit: { type: 'number', required: false, description: 'Max rows (default 100, cap 500).' },
- offset: { type: 'number', required: false, description: 'Pagination offset.' },
+ limit: { type: 'number', required: false, description: 'Max rows per list (default 100 for pending, 50 for the others; cap 500).' },
+ offset: { type: 'number', required: false, description: 'Pagination offset for pending.' },
+ include: { type: 'array', items: { type: 'string' }, required: false, description: 'Lists to return: pending, unattributed, possible_duplicates. Default pending and unattributed.' },
+ unattributed_offset: { type: 'number', required: false, description: 'Pagination offset for unattributed (its next_offset).' },
+ possible_duplicates_cursor: { type: 'string', required: false, description: 'Continuation for possible_duplicates (its next_cursor).' },
+ since_days: { type: 'number', required: false, description: 'possible_duplicates: entity pages created in the last N days (default 30).' },
},
scope: 'read',
handler: async (ctx, p) => {
+ const include = parseInclude(p.include);
const limit = Math.min(Math.max(Number(p.limit ?? 100) || 100, 1), 500);
const offset = Math.max(Number(p.offset ?? 0) || 0, 0);
// Read-side source isolation: route through sourceScopeOpts (federated
@@ -148,11 +153,41 @@ const extraction_pending: Operation = {
LIMIT $${params.length - 1} OFFSET $${params.length}`,
params,
);
- return { count: rows.length, pending: rows };
+ const out: Record<string, unknown> = { count: rows.length, pending: include.has('pending') ? rows : [], include: [...include] };
+ const lists = await import('../review-lists.ts');
+ const listLimit = Math.min(Math.max(Number(p.limit ?? lists.UNATTRIBUTED_DEFAULT_LIMIT) || lists.UNATTRIBUTED_DEFAULT_LIMIT, 1), lists.REVIEW_LIST_MAX_LIMIT);
+ const sourceIds = scope.sourceIds?.length ? scope.sourceIds : [scope.sourceId ?? ctx.sourceId ?? 'default'];
+ if (include.has('unattributed')) {
+ out.unattributed = await lists.listUnattributedFacts(ctx.engine, ctx, sourceIds,
+ { limit: listLimit, offset: Math.max(Number(p.unattributed_offset ?? 0) || 0, 0) });
+ }
+ if (include.has('possible_duplicates')) {
+ const { resolveExcludePrivatePages } = await import('../search/private-visibility.ts');
+ out.possible_duplicates = await lists.listPossibleDuplicates(ctx.engine, ctx, sourceIds[0]!, {
+ limit: listLimit, cursor: p.possible_duplicates_cursor,
+ sinceDays: Math.max(Number(p.since_days ?? lists.POSSIBLE_DUPLICATES_DEFAULT_DAYS) || lists.POSSIBLE_DUPLICATES_DEFAULT_DAYS, 1),
+ excludePrivate: await resolveExcludePrivatePages(ctx.engine, ctx.remote),
+ });
+ }
+ return out;
},
cliHints: { name: 'extraction-pending' },
};
+const PENDING_LISTS = ['pending', 'unattributed', 'possible_duplicates'] as const;
+
+function parseInclude(raw: unknown): Set<string> {
+ const given = Array.isArray(raw) ? raw : typeof raw === 'string' ? raw.split(',') : null;
+ if (!given) return new Set(['pending', 'unattributed']);
+ const wanted = new Set(given.map(v => String(v).trim()).filter(Boolean));
+ const unknown = [...wanted].filter(v => !(PENDING_LISTS as readonly string[]).includes(v));
+ if (unknown.length) {
+ throw opError('invalid_params', `extraction_pending: unknown include ${unknown.map(v => JSON.stringify(v)).join(', ')}.`,
+ `Nothing was read. Pass include as any of ${PENDING_LISTS.join(', ')}.`);
+ }
+ return wanted;
+}
+
const REVIEW_SLUG = /^[a-z0-9][a-z0-9/_.-]{0,254}$/i;
function pendingFix(): Action {
diff --git a/src/core/review-lists.ts b/src/core/review-lists.ts
new file mode 100644
index 0000000000..82d83ff412
--- /dev/null
+++ b/src/core/review-lists.ts
@@ -0,0 +1,121 @@
+/**
+ * #2947 / #2942: the two review lists extraction_pending adds beside its quarantine stubs.
+ *
+ * - unattributed: active facts saved with no entity (memory/unattributed), each with the non-destructive
+ * relink action. Never "forget then remember": a forget without an entity withdraws the claim for every entity.
+ * - possible_duplicates (opt-in): entity pages created recently whose name is one edit or one acronym letter away
+ * from another entity's name or alias. Flag only: linking stays manual (entity_identity_link), and an alias
+ * edit or a later "distinct" decision silences a pair.
+ *
+ * Both lists are read-only and source-scoped, and both hide what a remote caller cannot read.
+ */
+
+import type { BrainEngine } from './engine.ts';
+import type { OperationContext } from './ops/contract.ts';
+import { buildNearNameIndex, loadEntityNames, normalizeEntityName, queryNearNames, shardCountFor, type NearNameKind } from './entities/near-name.ts';
+
+export interface ReviewAction { kind: string; run_by: 'caller' | 'owner'; argv: string[]; mcp?: { tool: string; arguments: Record<string, unknown> }; inputs?: Array<{ name: string; how: string }>; why: string }
+
+export interface UnattributedRow { fact_id: string; fact: string; kind: string; visibility: string; source_id: string; created_at: string; actions: ReviewAction[] }
+
+export const UNATTRIBUTED_DEFAULT_LIMIT = 50;
+export const REVIEW_LIST_MAX_LIMIT = 500;
+
+export async function listUnattributedFacts(engine: BrainEngine, ctx: OperationContext, sourceIds: string[], opts: { limit: number; offset: number }):
+ Promise<{ count: number; rows: UnattributedRow[]; next_offset: number | null; complete: boolean }> {
+ const worldOnly = ctx.remote !== false;
+ const rows = await engine.executeRaw<{ id: number | string; fact: string; kind: string | null; visibility: string; source_id: string; created_at: string }>(
+ `SELECT id, fact, kind, visibility, source_id, created_at::text AS created_at FROM facts
+ WHERE source_id = ANY($1::text[]) AND entity_slug IS NULL AND expired_at IS NULL
+ AND (valid_until IS NULL OR valid_until > now())
+ AND ($2::boolean = false OR visibility = 'world')
+ ORDER BY id DESC LIMIT $3 OFFSET $4`,
+ [sourceIds, worldOnly, opts.limit + 1, opts.offset],
+ );
+ const page = rows.slice(0, opts.limit).map((r): UnattributedRow => {
+ const id = String(r.id);
+ return {
+ fact_id: id, fact: r.fact, kind: r.kind ?? 'fact', visibility: r.visibility, source_id: r.source_id, created_at: r.created_at,
+ actions: [{
+ kind: 'relink', run_by: 'caller',
+ argv: ['gbrain', 'remember', r.fact, '--entity', '<entity-slug>', '--replaces', id, '--source', r.source_id],
+ mcp: { tool: 'remember', arguments: { fact: r.fact, entity: '<entity-slug>', replaces: id, visibility: r.visibility } },
+ inputs: [{ name: 'entity-slug', how: 'The slug of the person, company or project page the claim is about (search or whois finds it).' }],
+ why: 'Saves the same claim on the entity it is about and replaces this unattributed copy. Do not forget it first: a forget without an entity blocks the claim for every entity.',
+ }],
+ };
+ });
+ return { count: page.length, rows: page, next_offset: rows.length > opts.limit ? opts.offset + opts.limit : null, complete: rows.length <= opts.limit };
+}
+
+export interface DuplicateRow { slug: string; name: string; source_id: string; created_at: string; near: Array<{ slug: string; name: string; kind: Exclude<NearNameKind, 'same'> }>; actions: ReviewAction[] }
+
+export const POSSIBLE_DUPLICATES_DEFAULT_LIMIT = 50;
+export const POSSIBLE_DUPLICATES_DEFAULT_DAYS = 30;
+
+let buildQueue: Promise<unknown> = Promise.resolve();
+/** One index build at a time per process, so concurrent calls never hold several shards in memory at once. */
+function serialized<T>(fn: () => Promise<T>): Promise<T> {
+ const run = buildQueue.then(fn, fn);
+ buildQueue = run.catch(() => undefined);
+ return run;
+}
+
+function parseCursor(raw: unknown): { shard: number; offset: number } {
+ const m = typeof raw === 'string' ? /^s(\d+):(\d+)$/.exec(raw.trim()) : null;
+ return m ? { shard: Number(m[1]), offset: Number(m[2]) } : { shard: 0, offset: 0 };
+}
+
+/**
+ * One page of possible duplicates for one source. The cursor walks recent entity pages within a shard, then the
+ * next shard; complete is true only after the last page of the last shard, so a sharded index never reports a
+ * pair-free result it did not compare.
+ */
+export async function listPossibleDuplicates(engine: BrainEngine, ctx: OperationContext, sourceId: string,
+ opts: { limit: number; cursor?: unknown; sinceDays: number; excludePrivate: boolean }):
+ Promise<{ count: number; rows: DuplicateRow[]; next_cursor: string | null; complete: boolean; shards: number; reason?: string }> {
+ return serialized(async () => {
+ const loaded = await loadEntityNames(engine, sourceId, { excludePrivate: opts.excludePrivate });
+ const shards = shardCountFor(loaded.names);
+ const cursor = parseCursor(opts.cursor);
+ const shard = Math.min(cursor.shard, shards - 1);
+ const since = Date.now() - opts.sinceDays * 86_400_000;
+ const created = new Map<string, string>();
+ for (const n of loaded.names) if (n.created_at) created.set(n.slug, n.created_at);
+ const recent = [...created.entries()].filter(([, at]) => Date.parse(at) >= since)
+ .sort((a, b) => Date.parse(b[1]) - Date.parse(a[1]) || a[0].localeCompare(b[0]));
+ const index = buildNearNameIndex(loaded.names, shard, shards);
+ const window = recent.slice(cursor.offset, cursor.offset + opts.limit);
+ const local = ctx.remote === false;
+ const rows: DuplicateRow[] = [];
+ for (const [slug, createdAt] of window) {
+ const own = loaded.names.filter(n => n.slug === slug);
+ const near = new Map<string, { slug: string; name: string; kind: Exclude<NearNameKind, 'same'> }>();
+ for (const entry of own) {
+ for (const m of queryNearNames(index, entry.name, slug)) {
+ if (m.kind === 'same' || near.has(m.slug)) continue;
+ if (loaded.distinctBySlug.get(slug)?.has(m.slug) || loaded.distinctBySlug.get(m.slug)?.has(slug)) continue;
+ if (loaded.aliasesBySlug.get(m.slug)?.has(normalizeEntityName(entry.name)) || loaded.aliasesBySlug.get(slug)?.has(normalizeEntityName(m.name))) continue;
+ near.set(m.slug, { slug: m.slug, name: m.name, kind: m.kind });
+ }
+ }
+ if (!near.size) continue;
+ const first = [...near.values()][0]!;
+ rows.push({
+ slug, name: own[0]!.name, source_id: sourceId, created_at: createdAt, near: [...near.values()],
+ actions: [
+ { kind: 'link_identity', run_by: local ? 'caller' : 'owner', argv: ['gbrain', 'entity-identity-link', '<entity-id>', slug, '--source-id', sourceId],
+ inputs: [{ name: 'entity-id', how: 'Any stable identity handle; run it once for each of the two pages with the same handle.' }],
+ why: `Records that ${slug} and ${first.slug} are the same entity (link both pages under one entity id). Linking is manual by design.` },
+ { kind: 'add_alias', run_by: 'caller', argv: ['gbrain', 'get', first.slug],
+ mcp: { tool: 'edit_page', arguments: { slug: first.slug, expected_revision: '<from get_page>', edits: [] } },
+ why: `If "${own[0]!.name}" is another name for ${first.slug}, add it to that page's aliases frontmatter; the pair is then no longer flagged.` },
+ ],
+ });
+ }
+ const nextOffset = cursor.offset + opts.limit;
+ const next = nextOffset < recent.length ? `s${shard}:${nextOffset}` : shard + 1 < shards ? `s${shard + 1}:0` : null;
+ return { count: rows.length, rows, next_cursor: next, complete: next === null && !loaded.truncated, shards,
+ ...(loaded.truncated ? { reason: 'entity_count_over_cap' } : {}) };
+ });
+}
diff --git a/test/e2e/extraction-review-postgres.test.ts b/test/e2e/extraction-review-postgres.test.ts
index 439de35723..e8a6b4e4ea 100644
--- a/test/e2e/extraction-review-postgres.test.ts
+++ b/test/e2e/extraction-review-postgres.test.ts
@@ -107,4 +107,16 @@ d('extraction quarantine lane (live Postgres)', () => {
expect(rej.results[0].status).toBe('rejected');
expect(await engine.getPage('people/pg-reject')).toBeNull();
});
+ test('review lists (#2942/#2947): unattributed world-only for remote, possible_duplicates flags a near name', async () => {
+ await engine.executeRaw(`INSERT INTO facts (fact, source, source_id, visibility) VALUES ('pg world claim', 'fixture', 'default', 'world'), ('pg private claim', 'fixture', 'default', 'private')`);
+ await engine.putPage('companies/pg-acme', { title: 'PGACME', type: 'company', compiled_truth: '# PGACME', timeline: '', frontmatter: {} });
+ await engine.putPage('companies/pg-acmx', { title: 'PGACMX', type: 'company', compiled_truth: '# PGACMX', timeline: '', frontmatter: {} });
+ const out = await operationsByName['extraction_pending']!.handler(ctx(), { include: ['unattributed', 'possible_duplicates'] }) as Record<string, any>;
+ const facts = out.unattributed.rows.map((r: any) => r.fact);
+ expect(facts).toContain('pg world claim');
+ expect(facts).not.toContain('pg private claim');
+ const flagged = out.possible_duplicates.rows.flatMap((r: any) => [r.slug, ...r.near.map((n: any) => n.slug)]);
+ expect(flagged).toContain('companies/pg-acme');
+ expect(flagged).toContain('companies/pg-acmx');
+ });
});
diff --git a/test/fixtures/goldens/cli/tools-json.json b/test/fixtures/goldens/cli/tools-json.json
index 0f55a875dc..293362e44e 100644
--- a/test/fixtures/goldens/cli/tools-json.json
+++ b/test/fixtures/goldens/cli/tools-json.json
@@ -2,8 +2,8 @@
"golden": {
"exitCode": 0,
"stderr": "",
- "stdoutBytes": 264473,
- "stdoutSha256": "f5647f0a0d45a36ed8685afa000792cef233379d012eeceec6fa2f9366d1b0ee",
+ "stdoutBytes": 265643,
+ "stdoutSha256": "a3193f5bf16f3f06caa9c6a96be44a89434f470729fbb1ca70230d83803cc63c",
"toolNames": [
"remember",
"entity",
@@ -6406,24 +6406,47 @@
"scope": "write"
},
{
- "description": "List unverified auto-extracted entity stubs awaiting owner review (the quarantine lane from extract_entities). Promote or reject them with extraction_review.",
+ "description": "List what waits for review: unverified auto-extracted entity stubs (pending; promote or reject them with extraction_review), facts saved without an entity (unattributed; relink each with remember entity + replaces, never forget first), and, when include names it, recently created entity pages whose name is one edit or one acronym letter from another entity (possible_duplicates; flag only, never merged).",
"idempotent": true,
"local_only": false,
"mutating": false,
"name": "extraction_pending",
"parameters": {
+ "include": "array?",
"limit": "number?",
- "offset": "number?"
+ "offset": "number?",
+ "possible_duplicates_cursor": "string?",
+ "since_days": "number?",
+ "unattributed_offset": "number?"
},
"required_scopes": [],
"schema": {
"properties": {
+ "include": {
+ "description": "Lists to return: pending, unattributed, possible_duplicates. Default pending and unattributed.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
"limit": {
- "description": "Max rows (default 100, cap 500).",
+ "description": "Max rows per list (default 100 for pending, 50 for the others; cap 500).",
"type": "number"
},
"offset": {
- "description": "Pagination offset.",
+ "description": "Pagination offset for pending.",
+ "type": "number"
+ },
+ "possible_duplicates_cursor": {
+ "description": "Continuation for possible_duplicates (its next_cursor).",
+ "type": "string"
+ },
+ "since_days": {
+ "description": "possible_duplicates: entity pages created in the last N days (default 30).",
+ "type": "number"
+ },
+ "unattributed_offset": {
+ "description": "Pagination offset for unattributed (its next_offset).",
"type": "number"
}
},
diff --git a/test/review-lists.test.ts b/test/review-lists.test.ts
new file mode 100644
index 0000000000..b3d2c67543
--- /dev/null
+++ b/test/review-lists.test.ts
@@ -0,0 +1,204 @@
+/**
+ * #2942 / #2947 (GBRA-215 L1): the near-name matcher and the two review lists extraction_pending adds.
+ *
+ * Flag only: nothing here merges or writes entity_identities. The lists are read-only, so a repo-less brain
+ * (this PGLite brain has no repository) and a repository-backed one read the same rows; the Postgres parity
+ * run lives in test/e2e/extraction-review-postgres.test.ts.
+ */
+
+import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { classifyNearName, nameKeys, buildNearNameIndex, queryNearNames, normalizeEntityName, type EntityName } from '../src/core/entities/near-name.ts';
+import { operationsByName, OperationError, type OperationContext } from '../src/core/operations.ts';
+
+let engine: PGLiteEngine;
+const pending = operationsByName['extraction_pending']!;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+}, 60_000);
+
+afterAll(async () => { await engine.disconnect(); });
+
+beforeEach(async () => {
+ await engine.executeRaw('DELETE FROM facts');
+ await engine.executeRaw('DELETE FROM content_chunks');
+ await engine.executeRaw('DELETE FROM links');
+ await engine.executeRaw('DELETE FROM pages');
+ await engine.executeRaw('DELETE FROM entity_identities').catch(() => undefined);
+});
+
+function ctx(over: Partial<OperationContext> = {}): OperationContext {
+ return {
+ engine, config: {} as OperationContext['config'],
+ logger: { info() {}, warn() {}, error() {}, debug() {} } as unknown as OperationContext['logger'],
+ dryRun: false, remote: true, sourceId: 'default', ...over,
+ } as OperationContext;
+}
+
+async function page(slug: string, title: string, frontmatter: Record<string, unknown> = {}, type = slug.startsWith('people/') ? 'person' : 'company') {
+ await engine.putPage(slug, { title, type, compiled_truth: `# ${title}`, timeline: '', frontmatter });
+}
+
+async function fact(text: string, visibility: 'world' | 'private' = 'world', entity: string | null = null) {
+ await engine.executeRaw(
+ `INSERT INTO facts (fact, source, source_id, visibility, entity_slug) VALUES ($1, 'fixture', 'default', $2, $3)`,
+ [text, visibility, entity],
+ );
+}
+
+describe('classifyNearName', () => {
+ test('one-letter transcription slips are typos', () => {
+ expect(classifyNearName('ACMX', 'ACME')).toBe('typo');
+ expect(classifyNearName('Brex Example', 'Drex Example')).toBe('typo');
+ expect(classifyNearName('Acme Exmaple', 'Acme Example')).toBe('typo');
+ });
+ test('corporate suffixes and case normalize to the same name, never a different entity', () => {
+ expect(classifyNearName('Acme Example Inc.', 'acme example')).toBe('same');
+ expect(classifyNearName('Acme Labs', 'ACME')).toBe('same');
+ expect(classifyNearName('Dr. Alice Example', 'Alice Example')).toBe('same');
+ expect(classifyNearName('Allie Example', 'Alice Example')).toBe('same');
+ });
+ test('an acronym of a multi-word name, with at most one letter different', () => {
+ expect(classifyNearName('NXE', 'Northstar Xeno Example')).toBe('acronym');
+ expect(classifyNearName('NEL', 'Northstar Example Labs')).toBe(null);
+ expect(classifyNearName('NXF', 'Northstar Example')).toBe(null);
+ expect(classifyNearName('NEF', 'Northstar Example Fund')).toBe('acronym');
+ expect(classifyNearName('NEX', 'Northstar Example Fund')).toBe('acronym');
+ });
+ test('short names and unrelated names are never flagged', () => {
+ expect(classifyNearName('AB', 'AC')).toBe(null);
+ expect(classifyNearName('Acme Example', 'Globex Example')).toBe(null);
+ expect(nameKeys('ab')).toEqual([]);
+ expect(nameKeys('x'.repeat(65))).toEqual([]);
+ });
+});
+
+describe('near-name index', () => {
+ const names: EntityName[] = [
+ { slug: 'companies/acme', name: 'ACME' },
+ { slug: 'companies/brex-example', name: 'Brex Example' },
+ { slug: 'companies/xacme', name: 'Xacme' },
+ ...Array.from({ length: 200 }, (_, i) => ({ slug: `companies/filler-${i}`, name: `Filler Company Number ${i}` })),
+ ];
+
+ test('finds every distance-1 pair, including ones that differ in the first letter', () => {
+ const index = buildNearNameIndex(names);
+ expect(queryNearNames(index, 'ACMX').map(m => m.slug).sort()).toEqual(['companies/acme']);
+ expect(queryNearNames(index, 'Drex Example').map(m => m.slug)).toEqual(['companies/brex-example']);
+ expect(queryNearNames(index, 'Acme').find(m => m.slug === 'companies/xacme')?.kind).toBe('typo');
+ });
+
+ test('a sharded index finds the same pairs once every shard is walked', () => {
+ const queries = ['ACMX', 'Drex Example', 'Acme', 'Filler Company Numbr 7'];
+ const whole = queries.map(q => queryNearNames(buildNearNameIndex(names), q).map(m => m.slug).sort());
+ for (const shards of [2, 5, 13]) {
+ const sharded = queries.map(q => [...new Set(Array.from({ length: shards }, (_, s) =>
+ queryNearNames(buildNearNameIndex(names, s, shards), q).map(m => m.slug)).flat())].sort());
+ expect(sharded).toEqual(whole);
+ }
+ });
+
+ test('normalizeEntityName strips corporate suffixes', () => {
+ expect(normalizeEntityName('The Acme Example Company, LLC')).toBe('acmeexample');
+ });
+});
+
+describe('extraction_pending review lists', () => {
+ test('default output keeps pending and count, adds unattributed, and runs no matcher', async () => {
+ await fact('the quarterly offsite moved to march');
+ const out = await pending.handler(ctx(), {}) as Record<string, any>;
+ expect(out.count).toBe(0);
+ expect(out.pending).toEqual([]);
+ expect(out.include).toEqual(['pending', 'unattributed']);
+ expect(out.unattributed.rows.map((r: any) => r.fact)).toEqual(['the quarterly offsite moved to march']);
+ expect(out.possible_duplicates).toBeUndefined();
+ });
+
+ test('unattributed rows carry the relink action and never a forget', async () => {
+ await fact('a world claim');
+ const out = await pending.handler(ctx(), {}) as Record<string, any>;
+ const [action] = out.unattributed.rows[0].actions;
+ expect(action.kind).toBe('relink');
+ expect(action.mcp.tool).toBe('remember');
+ expect(action.mcp.arguments.replaces).toBe(out.unattributed.rows[0].fact_id);
+ expect(JSON.stringify(out.unattributed)).not.toContain('forget"');
+ });
+
+ test('remote callers see only world facts; the local owner sees private ones too', async () => {
+ await fact('world claim', 'world');
+ await fact('private claim', 'private');
+ await fact('linked claim', 'world', 'people/alice-example');
+ const remote = await pending.handler(ctx(), {}) as Record<string, any>;
+ expect(remote.unattributed.rows.map((r: any) => r.fact)).toEqual(['world claim']);
+ const local = await pending.handler(ctx({ remote: false }), {}) as Record<string, any>;
+ expect(local.unattributed.rows.map((r: any) => r.fact).sort()).toEqual(['private claim', 'world claim']);
+ });
+
+ test('unattributed pages with next_offset until complete', async () => {
+ for (let i = 0; i < 5; i++) await fact(`claim ${i}`);
+ const first = await pending.handler(ctx(), { limit: 2 }) as Record<string, any>;
+ expect(first.unattributed.rows.length).toBe(2);
+ expect(first.unattributed.complete).toBe(false);
+ const seen = [...first.unattributed.rows.map((r: any) => r.fact)];
+ let next = first.unattributed.next_offset;
+ while (next !== null) {
+ const more = await pending.handler(ctx(), { limit: 2, unattributed_offset: next }) as Record<string, any>;
+ seen.push(...more.unattributed.rows.map((r: any) => r.fact));
+ next = more.unattributed.next_offset;
+ }
+ expect(seen.sort()).toEqual(['claim 0', 'claim 1', 'claim 2', 'claim 3', 'claim 4']);
+ });
+
+ test('possible_duplicates flags a near name on request, without merging anything', async () => {
+ await page('companies/acme', 'ACME');
+ await page('companies/acmx', 'ACMX');
+ await page('companies/globex-example', 'Globex Example');
+ const out = await pending.handler(ctx(), { include: ['possible_duplicates'] }) as Record<string, any>;
+ const flagged = out.possible_duplicates.rows.map((r: any) => [r.slug, r.near.map((n: any) => `${n.slug}:${n.kind}`)]);
+ expect(flagged).toContainEqual(['companies/acmx', ['companies/acme:typo']]);
+ expect(flagged.find((f: any) => f[0] === 'companies/globex-example')).toBeUndefined();
+ expect(out.possible_duplicates.complete).toBe(true);
+ const [{ n }] = await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM entity_identities');
+ expect(n).toBe(0);
+ const link = out.possible_duplicates.rows[0].actions.find((a: any) => a.kind === 'link_identity');
+ expect(link.run_by).toBe('owner');
+ });
+
+ test('an alias or a distinct_from decision silences a pair', async () => {
+ await page('companies/acme', 'ACME', { aliases: ['ACMX'] });
+ await page('companies/acmx', 'ACMX');
+ await page('people/alice-example', 'Alice Example');
+ await page('people/alicia-example', 'Alicx Example', { distinct_from: ['people/alice-example'] });
+ const out = await pending.handler(ctx(), { include: ['possible_duplicates'] }) as Record<string, any>;
+ expect(out.possible_duplicates.rows).toEqual([]);
+ });
+
+ test('a private page never shapes what a remote caller is shown', async () => {
+ await engine.setConfig('search.exclude_private_pages', 'true').catch(() => undefined);
+ await page('companies/acme', 'ACME', { visibility: 'private' });
+ await page('companies/acmx', 'ACMX');
+ const remote = await pending.handler(ctx(), { include: ['possible_duplicates'] }) as Record<string, any>;
+ expect(JSON.stringify(remote.possible_duplicates)).not.toContain('companies/acme"');
+ const local = await pending.handler(ctx({ remote: false }), { include: ['possible_duplicates'] }) as Record<string, any>;
+ expect(local.possible_duplicates.rows.map((r: any) => r.slug)).toContain('companies/acmx');
+ });
+
+ test('possible_duplicates only looks at recently created pages', async () => {
+ await page('companies/acme', 'ACME');
+ await page('companies/acmx', 'ACMX');
+ await engine.executeRaw(`UPDATE pages SET created_at = now() - interval '90 days'`);
+ const out = await pending.handler(ctx(), { include: ['possible_duplicates'] }) as Record<string, any>;
+ expect(out.possible_duplicates.rows).toEqual([]);
+ const wider = await pending.handler(ctx(), { include: ['possible_duplicates'], since_days: 120 }) as Record<string, any>;
+ expect(wider.possible_duplicates.rows.length).toBeGreaterThan(0);
+ });
+
+ test('an unknown include is refused with the valid names', async () => {
+ const err = await pending.handler(ctx(), { include: ['everything'] }).catch(e => e);
+ expect(err).toBeInstanceOf(OperationError);
+ expect(String(err.message)).toContain('everything');
+ });
+});