Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
26.2 KB
Raw Blame

Key files — entity recall (mention index)

The mention index links every page that names an entity (by title, title subject, frontmatter alias or declared code) to that entity page, keeps the links current from gbrain extract --stale, and serves them on the entity card and get_backlinks. Guide: entity recall.

  • src/core/by-mention.ts — the pure mention matcher plus its gazetteer. buildGazetteer(engine, opts) loads linkable entity titles and aliases per source (drops below-minimum, generic, ambiguous-first-word and colliding names (a name claimed at one origin by an identity-sibling group stays, marked shared), reported as DroppedName), keyed by lowercase first token and sorted longest-first. findMentionedEntities(text, gazetteer, opts) masks code and HTML comments (the <!--- gbrain:... --> fence markers included) with stripCodeBlocks, offsets preserved, then picks the longest matching entry at each token (maximal munch), first mention per target only (a shared match links every sibling with those tokens, unless the scanning page is one of them), skipping self-links and other sources unless allowCrossSource (an own-source twin wins, except that a connector contact stub, connector_stub from google_contact_id frontmatter, yields under allowCrossSource to the one canonical same-name page in another source or the one sharing its slug; #6158). Word-run scripts match whole tokens; Han and Kana match character substrings. A Hangul name must start a word, keep its written spacing, and end at a non-Hangul character or an attached title/particle/copula suffix (HANGUL_NAME_SUFFIX_RE), so 지원에게 matches and 지원하는 does not (measurement). A name written only in Katakana (ー included) must be a whole contiguous Katakana run (hasKatakanaMatchBoundary): Katakana, small and supplementary kana, ー and voicing marks continue it, Hiragana, Han, Latin, punctuation and ・ end it, so デン matches 喫茶デン and デン・カフェ but not インシデント. A matcher change bumps MENTION_EXTRACTOR_VERSION (mentions/pass.ts, every page rescans once) and the semantics tag inside hashGazetteer (the extract links --by-mention resume checkpoint). scanStaleMentions is the read-only doctor scan for mention rows the current gazetteer no longer produces; each example carries the row's stored link_type (a verb on typed_ner rows), which doctor stale_mentions puts in that example's own gbrain unlink … --link-type <type> --link-source mentions command, built with shellQuote (argv and command in details.unlink) (#8814).
  • src/core/mentions/policy.ts — the one linkable-type resolver. linkableTypesFor(pack, policy): person, company, organization, entity (ALWAYS_LINKABLE_TYPES, never removable) plus every pack type with primitive: entity and its type aliases (product excluded), adjusted by mentions.entity_types (+type/type adds, -type removes). readMentionPolicy reads auto_link, mentions.auto_link, mentions.entity_types, mentions.ignore, mentions.exclude_slugs (#5829: listed pages get no gazetteer entry by title or alias, dropped as excluded_slug; JSON array or comma list, parseNameList), mentions.alias_deny, mentions.multiword_aliases and mentions.sibling_merge (both default on): from the request's config snapshot when one is open (config-snapshot.ts), else in one keyed config read. canonicalTypeOf / typeAliasPairs map a stored type through the pack's type aliases (untyped for none); pairs are bound as two text[]. loadSourcePack(engine, sourceId, {strict}) returns null on a resolution failure unless strict. Tests: test/mentions-policy-aliases.test.ts.
  • src/core/mentions/siblings.ts — identity siblings: pages in one source naming the same normalized title subject under pairwise different prefixes, all linkable entities of one canonical type other than person, none excluded or identity: separate, at most SIBLING_GROUP_CAP (3; a larger group is capped). siblingVerdict is the one rule (card, search fan-out, gazetteer); readIdentitySiblings reads a page's group with the private-page predicate for untrusted callers. siblingCanonical (the group's lowest slug, for fact attribution by name) and factEntitySlugs (the page, its siblings and its bare subject slug, read by the entity card's facts and recall {entity}). Nothing is written to entity_identities, and aliases and references are never unioned. Tests: test/mentions-identity-siblings.test.ts.
  • src/core/mentions/aliases.ts — derived names of an entity page, implementing docs/designs/ALIAS_CONVENTIONS.md. Stored aliases and card identity lines read declarations with anchored: true (a cue must attach to the page itself; search's per-result aliasDeclarations scan reads them all). titleSubject (text after the last : ), declarationsIn (each declaration with its line: prose cues, label and value lines and table rows whose label names an alias field (also a label opening a later sentence of a line, alias noun within its first two words, at most 7 words), the quoted-only "calls it" cue, a quoted defined term right after the page's own name; a capture is a quoted name or a run of up to 4 capitalized or code tokens ending at punctuation, a lowercase word or the line end, rejected on a possessive, a run of only common words or a cue after a relational noun; linear in the line length), declaredNames (its post-filtered names, shared with search's aliasDeclarations), deriveEntityAliases(page, {multiword, deny}) (subject origin='subject', declarations origin='declared' with their lines, read from publicBody: private takes and facts fences stripped; a single-token declaration is case_sensitive; mentions.alias_deny and frontmatter alias_deny reject as denied). aliasRejection: under 4 characters (unless a short code the page declares for itself: a declaration marked selfCode), a generic word, or the first word of the page's own multi-word name. Rejections are returned with their line for explain. Tests: test/mentions-policy-aliases.test.ts.
  • src/core/mentions/short-codes.ts — isShortCode (2-3 letters or digits with a capital, not in SHORT_CODE_STOPLIST of common capitals and business acronyms, compared upper-cased). declarationsIn marks a short code selfCode when an alias label or an also called / a.k.a. / known as / short for cue declares it in a sentence naming nothing before the cue but the page (empty, a pronoun or an own name); aliasRejection lets it through and by-mention.ts keeps a declared, case_sensitive short code in the gazetteer, matched only as written. Leaf module (aliases.ts imports by-mention.ts). Tests: test/mentions-policy-aliases.test.ts, test/mentions-pass.test.ts.
  • src/core/mentions/pass.ts — runMentionPass(engine, {sourceId?, deadline?, signal?, beforePublish?}), run by extractStaleFromDB (unmanaged) and extractManagedStaleLinks (managed, unless mentions: false). #7031: on a belief-covered source a batch's link adds and drops run in an unmanaged mention_pass belief point (page keys locked first), so they never read as ledger drift (test/belief-stale-links-7031.test.ts). Per source: policy off → disableSource deletes plain mention links (link_kind null or plain; typed_ner stays), derived alias rows, saved entries and page state once and records disabled; a changed policy fingerprint (types, cross-source flag, ignore list, mentions.alias_deny, mentions.multiword_aliases, mentions.sibling_merge) marks every page due and recounts pending; the status row's policy_fingerprint carries the extractor versions its count reflects after the hash (<hash>;v<mention>.<alias>, INDEX_VERSION_STAMP), and a stamp that differs from the binary's recounts pending once (version-behind pages are already due, so nothing is invalidated); refreshAliases rewrites derived rows of entity pages whose page_mention_state.alias_revision/alias_version differ (revision-checked, coordinated write when managed) and removes them from pages that stopped being entities; buildGazetteer({strict: true}) failing writes nothing and records failed; saveEntries diffs mention_gazetteer_entries under pg_advisory_xact_lock, bumps the source generation and marks due only pages linking to a removed target plus pages a mini-gazetteer of added names matches (ASCII names prefiltered with LIKE on lower-cased text, others scanned in JS; a first save marks the source due); reconcileSource scans due pages (no state row, or mention_revision/mention_version differ) in batches of 200 and publishes only when the source generations (FOR SHARE) and each page's knowledge_revision (FOR SHARE) are unchanged, writing link diffs (addLinksBatch, audit site extract.mentions) and page state in one transaction; then the status row gets state, pending (countIndexDuePages: mention-due pages plus linkable entity pages whose derived aliases are behind their content or ALIAS_DERIVATION_VERSION, counted when the source finishes or the deadline stops it) and last_pass_at. writeDerivedAliases / writePageAliases are import's and reindex --aliases' alias projection. MENTION_EXTRACTOR_VERSION and ALIAS_DERIVATION_VERSION are independent of LINK_EXTRACTOR_VERSION_TS. The scanned text is title, compiled truth and timeline; frontmatter mention_ignore names are skipped per page. Tests: test/mentions-pass.test.ts, test/mentions-coverage-version.test.ts, test/mentions-managed.test.ts, test/e2e/mentions-parity.test.ts.
  • src/core/mentions/stale.ts — the pass as a sweep phase: previewMentionPass (index-due pages per source, as countIndexDuePages counts them, and newest last_pass_at, for --dry-run, doctor links_extraction_lag and post-upgrade), linkPhaseDeadline (a budgeted sweep gives link work half its budget while mention-due pages exist), formatMentionSummary, mentionJsonFields.
  • src/core/mentions/coverage.ts — readMentionCoverage(engine, sourceIds): {state, pending_pages, last_pass_at, degraded?} from mention_index_status plus due pages written at or after the row's counted_at (states complete, pending, disabled, failed; the card adds type_not_linkable). A row whose version stamp differs from the binary's (counted by another release) is recounted once with countIndexDuePages and the count and stamp saved on the row (compare-and-set on the stored fingerprint; a failed write still returns the count), so after a version bump every page behind MENTION_EXTRACTOR_VERSION or ALIAS_DERIVATION_VERSION reads as pending whatever its updated_at, and later reads stay one status read plus the recent-pages query. Policy settings changes are detected by the pass, not here. mentionCoverageNotice builds the mention_index notice (kind degraded, a per-call code): the free catch-up sweep for pending/failed, the user's setting for disabled/type-not-linkable, each with a read-only verify.
  • src/core/mentions/referrers.ts — the shared referrer query behind referenced_by and get_backlinks {group:"page"}: one row per referring page (every link source), semi-join over links to the target (plus identity co-members under entity_identity.union), canonical type via bound alias arrays, date COALESCE(effective_date, updated_at) with date_source, previews (160 characters after an optional # heading, fences stripped, redactFindings). readReferrerGroups: per-type totals and a bounded top-10 per type, water-filled to 50 rows per card. readReferrerPage: keyset paging on (date DESC, source_id, slug) with an opaque cursor (microsecond UTC text). Untrusted scopes apply privatePagesFilterFragment to referrer and target and privateLinkOriginFilterFragment to the link, in rows and counts. #5575: quarantined referrers are left out (rows and counts) unless includeQuarantined, the scope's eligibility (pageEligibleSql) applies before LIMIT, and each row carries trust_tier, origin and unconfirmed from trustFields. Tests: test/mentions-card-backlinks.test.ts, test/context-pack-newer-mentions.test.ts, test/entity-card-perf.slow.test.ts (10,000-link hub gate). F1 records: referrersCteBatched is the same CTE with a target index (referrerFilters shared), readTargetReferences reads referenced_by_count plus enrolled-type records for many targets in two statements (totals per type; top RECORD_ROWS_PER_TARGET per target newest first; RECORD_ROWS_PER_CALL allotted in target order before hydration; rows, totals and cursors equal readReferrerPage {type}), readRecordMetadata adds page_status, amends (+ since from link_relationships.first_start in the caller's scope) and supersedes, also on readReferrerPage {recordMetadata} continuations. F3: pageStatusValue (string, trimmed, at most PAGE_STATUS_MAX_CHARS 40) is the one page_status rule; readPageStatusByRef reads it by (source_id, slug) in one statement (private pages hidden from untrusted callers, quarantined pages unless allowed) for readReferrerPage {pageStatus}, get_backlinks link rows and context_pack cards and newer-mention rows. Tests: test/entity-records.test.ts, test/e2e/entity-records-postgres.test.ts.
  • src/core/mentions/upgrade-notice.ts — gbrain post-upgrade prints an [AGENT] block naming gbrain extract --stale --catch-up with a time estimate (MENTION_PASS_PAGES_PER_SECOND) while index-due pages exist (previewMentionPass), including every page behind the binary's mention or alias version on an unchanged brain. Post-upgrade does not run the sweep; autopilot's cycle extract phase continues it on its stale-drain budget (CYCLE_STALE_DRAIN_BUDGET_MS).
  • src/core/ops/backlinks-paged.ts — get_backlinks' type / group / limit / cursor. With none, ops/links.ts returns the legacy Link[]. group: "page" → {rows, total, truncated, cursor, coverage} (target = the first source in scope holding the slug; referrers from the read scope). Array mode (#6876) is the keyset-paged Link[] from ops/links.ts / ops/link-paging.ts; filterBacklinkRows only filters each page by referrer type. With group: "page", limit outside 1..500, a foreign cursor, or an unknown type (not a pack type, alias, observed referrer type, untyped or a type of any page the caller can read in its scope, ops/stats-by.ts visiblePageTypes: the countPagesBy count summed per type, most pages first, also behind the empty list_pages notice) is invalid_params naming the valid values (only types of pages the caller can read); a type other visible pages have but no referrer has answers empty with an empty_retrieval info notice naming the referrer types (GBRA-449, test/backlinks-type-exists-gbra449.test.ts). F3 (entity.row_page_status, default on): withLinkPageStatus adds the referring page's page_status to link rows, and group: "page" rows get it through readReferrerPage {pageStatus}; a record type's continuation keeps it through recordMetadata when the key is off.
  • src/core/ops/graph-rank.ts — #7168: rank_entities (CLI rank-entities): validates every parameter with invalidParam (no coercion; type and link-type names are pattern-checked, non-ranking link types refused, limit 1-100), scopes with assertSourceInCallerScope / resolveRequestedScope then linkReadScopeOpts (remote scalar scopes promoted to sourceIds), defaults types to resolveEntityTypes, runs engine-sql/graph-rank.ts under RANK_ENTITIES_STATEMENT_MS (a 57014 becomes rank_entities_timeout), attaches possible_duplicates (readPossibleDuplicates) to at most RANK_ENTITIES_DUPLICATE_ROWS top rows within a 3 s budget, and coverage from readExtractionLag per source in scope (state only for remote callers). Guide: graph ranking. Tests: test/rank-entities.test.ts.
  • src/core/ops/whois.ts — GBRA-156: whois (CLI whois <query>), the exact identity lookup. A query is an email address (identityEmailKey: NFKC, lowercase, surrounding quotes and angle brackets dropped, whole address compared) or a name (identityNameKey: normalizeAlias, surrounding straight and typographic quotes dropped, NFKD with combining marks stripped, foldNonDecomposingLatin). Scope: parseSourceIdParam + federatedSearchScope + assertExplicitSourceLive, an unscoped trusted call reading every unarchived source; types defaults to resolveEntityTypes for that scope. Candidates come from engine-sql/exact-identity.ts; identityMatchKind keeps a candidate only when a frontmatter email/emails value (email queries), the title, or a normalizeAliasList alias folds to exactly the query, labeled email > title > alias (ordered so, then source id and slug). Returns matches, ambiguous (more than one match), types_status, and truncated + note past WHOIS_CANDIDATE_CAP candidates; no match is an empty list, never a search fallback. Registered labeled, output redaction retrieval. Tests: test/whois.test.ts, test/e2e/whois-postgres.test.ts.
  • src/commands/extract-mentions-explain.ts — gbrain extract mentions --explain <name|slug> [--page <slug>] [--source-id <id>] [--json] (dispatched from extract-explain.ts): the matched entries with origin and case flag, or one reason code (ambiguous_first_word, below_min_length, generic_token, alias_collision, case_mismatch, type_not_linkable, linking_disabled, pending, ignored_by_page, ignored_by_config, denied_by_config, not_a_known_name); only pending names a sweep. A slug query also lists each derived name with origin and declaring line, the rejected candidates with reasons, and the sibling decision. Tests: test/mentions-explain.test.ts.
  • src/core/verbs/entity-names.ts — entity with names (up to ENTITY_NAMES_MAX 50): lookupEntityNames reads entityLookupScope once per batch, then resolves each name through the card's candidate selection (selectEntityCandidates, no card assembled), 8 at a time, deduped in first-seen order, into compact rows {name, found, slug, title, type, aka, lead, siblings?}; the lead comes from the candidate row's bounded, withdrawal-overlaid compiled_truth prefix (page-window/bounded-lead.ts), siblings from readSiblingIdentity with the batch's policy and pack. Output is byte-identical to the old full-card path (test/entity-names-parity.test.ts, Postgres twin test/e2e/entity-names-parity-postgres.test.ts, fixtures and the reference resolver in test/helpers/entity-names-parity.ts). pageLead is the body's opening prose from boundedProjectedPrefix (the 64 KiB prefix, sanitized and credential-projected) plus secret redaction, headings, markers and table rows dropped, ENTITY_LEAD_CHARS 200. attachReferences adds referenced_by_count, the card next and records to every found page and sibling from one readRowReferences call. Tests: test/entity-names.test.ts, test/entity-names-parity.test.ts, test/entity-records.test.ts.
  • src/core/verbs/entity-references.ts — the F1 reference fields on compact rows: readRowReferences (config gate per surface names / pack / full_card, REFERENCES_DEADLINE_MS 2 s cooperative deadline, references_unavailable marker + degraded notice, per-type next to get_backlinks returning exactly the rest, record_types_undeclared notice), readPackReferences (context_pack pull verb only: card page + identity siblings), renderRecordLines / RECORDS_HEADER (packed text, "page date"). Config: src/core/entity-references-config.ts (entity.names_references default on, entity.record_types default empty). Tests: test/entity-records.test.ts, test/e2e/entity-records-postgres.test.ts.
  • src/core/verbs/entity-card-identity.ts — the entity card's aka_sources, identity_siblings, identity_excerpt (verbatim lines of the page and its siblings from sanitizeRemoteBody, tiered: declarations, label lines with a name-like value, other label lines, lines with another capitalized run or code; 600 characters per page, 1,200 per card, identity_excerpt_omitted), alias_guidance (surface-aware) and the identity_siblings_capped notice. A segmented page is excerpted from its trusted, withdrawal-overlaid lead (truthLead, EXCERPT_LEAD_BYTES = 64 KiB, cut back to a whole line), never its body getters; the card equals the inline page's when the qualifying lines sit in that lead (test/large-pages/segmented-read-refusals.test.ts). readSiblingIdentity is its siblings-and-aliases half (one alias read for the page and its siblings; optional preloaded mention policy and pack), used alone by entity names[]. Tests: test/mentions-identity-siblings.test.ts.
  • src/core/search/alias-fanout.ts — search and query's alias fan-out: resolveQueryEntity (query n-grams up to 4 tokens through resolveAliases and linkable exact titles, longest first, capitalized or code-shaped unless a multi-word n-gram resolves to one page, case-sensitive aliases as written, a tie only when the pages are identity siblings), the names of the entity and its siblings by origin then length, at most search.alias_fanout_max (4) keyword queries of 5 rows (searchAliasRequired: the name as a required websearch phrase, ranked by the query's other terms through SearchOpts.rankQuery on both engines), at most 8 spliced rows with matched_alias, and the fanout meta (alias_fanout notice when capped). Offset 0 only. resolving takes the handler's resolveQueryEntity started before the search, so its reads overlap it. Tests: test/search-alias-fanout.test.ts, test/search-other-names.test.ts, test/e2e/mentions-parity.test.ts. wholeQueryEntityNames: when the whole query is one entity's name or alias, match: "keyword" lists pages matching any of its names (one websearch OR of phrases; match_names in the meta and the count line; rows found by another name carry matched_alias). resolveQueryEntities returns every entity a text names; its alias, linkable-title and case-sensitive alias reads run concurrently (test/search-entity-resolve-concurrency.test.ts).
  • src/core/search/entity-facts.ts — entitySavedFacts: when a search names an entity, its newest active saved facts (factEntitySlugs, 5) lead the model-visible saved-facts block, then up to 3 facts each of up to 2 entities those facts name (linked_from), before the word-overlap matches. Same active, audit-row, visibility and read-eligibility rules as recall (projectionEligibleSql: quarantined-page and needs_rederive facts hidden, the caller's trust floor applied). Tests: test/memory-sibling-facts.test.ts, test/search-entity-facts-eligibility.test.ts (also the trajectory, entity-card count, standing-question evidence, local CLI recall and consolidate reads, both engines).
  • src/core/search/saved-facts-match.ts — matchingSavedFacts, the word-matched saved facts search and query report in retrieval.saved_facts after the named entity's facts (ops/search.ts buildRetrievalResponseMeta). Words match whole after stemming with the facts keyword index's expression (factsFtsDocument), CJK words as substrings; a fact needs three quarters of the query's words, at most four, question-framing words (QUESTION_FRAME_WORDS: count, number, total, amount) not counted; most matched words first, then newest (time, then id), at most five after the #7043 collapse. One statement behind the facts-only EXISTS probe (test/search/saved-facts-probe.test.ts). search.saved_facts_supersede (default on; false turns it off): a newer fact with a correction cue that shares two matched words with a returned fact is returned too, even below the threshold, and the facts it corrects move last with corrected_by (the newest correction); every correction of a returned fact comes back; a cue fact that adds no figure to the older fact (figures: numbers, with ISO and day-month dates and times left out, thousands separators and a zero fraction dropped; CORRECTION_CUE leaves out "corrected"/"correcting", which report a correction) corrects nothing when it restates one of its figures or the older fact is itself a correction, nor when it states no figure and withdraws nothing (RETRACTION, which the facts block's retraction cue in fact-relevance.ts extends); nor does one saved under another entity, of lower trust, or sharing none of the older fact's matched names (namedWords: a name head matches a query word by prefix or, read only when a cue fact is a candidate, by stem). Correctors rank newest first under the cap and a corrected fact whose corrector is cut is dropped; a failed read warns with its SQLSTATE and returns no facts. The result carries every pre-cap decision (correctedBy): buildRetrievalResponseMeta applies it after merging the entity facts, dropping a corrected fact with no returned corrector (repeated through chains), labelling the rest, and ordering each corrected fact after its corrector. Also owns FACT_MATCH_STOPWORDS. Tests: test/search-saved-facts-match.test.ts, test/search-saved-facts.test.ts.
  • src/core/schema-migrations/v206-entity-mention-index.ts — additive: page_aliases.origin (frontmatter default, declared, subject), case_sensitive, alias_text, unique key (source_id, alias_norm, slug, origin); tables page_mention_state, mention_gazetteer_entries, mention_index_status (RLS on when the role bypasses RLS). GUARDED_TABLE_COLUMNS lists the new page_aliases columns.

Alias readers apply precedence frontmatter > declared > subject per source, and a derived alias never answers for another live page's exact title that the caller can read (the title competitor gets the caller's pageReadFilter, so a private title a private-excluding caller can't see never hides a readable alias) (readAliases in src/core/search/read-enrichment.ts, shared by both engines' resolveAliases), so resolveEntitySlug, the retrieval reflex, the alias hop, intent weights and search-diagnose rank an exact title first. buildGazetteer (src/core/by-mention.ts) applies the same precedence, drops an alias claimed by two pages at its best origin (unless the claimants are identity siblings, which all keep it), yields to a same-source title, drops a single-word alias equal to the first word of another multi-word entity name, applies the generic-word list to every single-word name, and carries caseTokens for case-sensitive entries; opts.dropped records every dropped name with its reason.