Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
7.1 KB
Raw Blame

Multi-language full-text search

GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery). The tokenizer language is configurable via the GBRAIN_FTS_LANGUAGE environment variable. Default: english.

How it works

Postgres text-search configurations control stemming and stop-word removal. GBRAIN_FTS_LANGUAGE is read by src/core/fts-language.ts and applied on both sides of the search:

  • Query side — websearch_to_tsquery('<lang>', $query) in both engines (Postgres and PGLite).
  • Write side — the update_page_search_vector and update_chunk_search_vector trigger functions that populate pages.search_vector and content_chunks.search_vector.

The value is validated against /^[a-z][a-z0-9_]*$/ before it is ever interpolated into SQL (tsvector functions don't accept parameterized config names). Invalid values fall back to english with a warning.

Built-in languages

Set the env var to any configuration your Postgres instance ships:

export GBRAIN_FTS_LANGUAGE=portuguese
export GBRAIN_FTS_LANGUAGE=spanish
export GBRAIN_FTS_LANGUAGE=german

List what's available:

SELECT cfgname FROM pg_ts_config;

PGLite (the embedded default engine) ships the same built-in snowball configurations as stock Postgres.

First install vs. changing language later

On first install (or upgrade), the configurable_fts_language schema migration reads GBRAIN_FTS_LANGUAGE and stamps the trigger functions with that language. After the migration has run, changing the env var alone does NOT retokenize existing rows — the migration shows as applied and is skipped. Use the explicit command:

export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run    # preview: language + row counts
gbrain reindex-search-vector --yes        # recreate triggers + backfill + rebuild the facts keyword index

The stamp survives later schema work: initSchema() — including the replay behind gbrain init --migrate-only on every upgrade — applies the schema template under the configured language, so it re-creates the trigger functions as they already are instead of reverting them to english.

The command recreates both trigger functions under the new language and backfills every existing pages and content_chunks row in batches, streaming progress to stderr. It is idempotent: re-running with the same language produces identical vectors. --json prints a machine-readable result envelope but still requires --yes (or an interactive confirm).

No cache purge is needed. The resolved language is part of the query-cache key, so rows written under the previous language are unreachable after the switch — searches read the retokenized index immediately instead of being served pre-switch results for up to search.cache.ttl_seconds. Switching back reaches the original rows rather than rebuilding them.

Recipe: accent-insensitive Portuguese (pt_br)

Brazilian Portuguese content often mixes accented and unaccented spellings ("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via the unaccent extension, then stems with the portuguese snowball dictionary:

CREATE EXTENSION IF NOT EXISTS unaccent;

CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);

ALTER TEXT SEARCH CONFIGURATION pt_br
  ALTER MAPPING FOR hword, hword_part, word
  WITH unaccent, portuguese_stem;

Then point GBrain at it:

export GBRAIN_FTS_LANGUAGE=pt_br
gbrain reindex-search-vector --yes

Note: custom configurations require a real Postgres instance (e.g. the Supabase engine). The config must exist BEFORE the migration or the reindex command runs, or Postgres will reject the trigger recreation with text search configuration "pt_br" does not exist.

Caveats

  • One language per brain: the setting is global to the database, not per-source. Mixed-language brains should pick the dominant language (the vector-search arm is language-agnostic and covers the rest).
  • Keep GBRAIN_FTS_LANGUAGE set consistently in every environment that writes to the brain (CLI shells, MCP server, cron jobs) — a writer without the env var tokenizes new rows in english until the next reindex.
  • Remote title matching (MCP and other safe-chunks callers) reads the title lexemes stored in pages.search_vector, so it follows the language those vectors were built with. After a language change, run gbrain reindex-search-vector so remote title search tokenizes like the query.
  • Interrupted reindex: the trigger flip commits before the backfill, so a reindex-search-vector run killed mid-way (crash, full disk, SIGKILL) leaves new writes in the new language and un-backfilled rows in the old one — keyword search then matches only part of the corpus. The command records an in-progress marker plus a per-batch checkpoint, gbrain doctor fails with fts_reindex_incomplete until the run completes, and re-running with the same GBRAIN_FTS_LANGUAGE resumes from the checkpoint rather than starting over. Budget minutes, not seconds, on brains with 100K+ chunks.
  • CJK (Chinese / Japanese / Korean): none of the built-in snowball configurations can tokenize CJK text, so the FTS arm would return nothing for CJK queries. Both engines detect CJK queries and route them to a term-by-term ILIKE fallback with term-frequency ranking instead (shared SQL in src/core/search/cjk-keyword-sql.ts). The fallback is correct but not index-accelerated: it scans content_chunks.chunk_text, so latency grows with corpus size. Note the routing is query-driven: any query containing CJK characters takes the fallback today, even on a Postgres instance with a CJK-aware extension (pgroonga / zhparser) installed. Wiring a CJK-capable GBRAIN_FTS_LANGUAGE config past the fallback is a filed follow-up. In hybrid search the fallback keeps every term required first and, when that finds fewer rows than requested, matches any term, ranking chunks by how many terms they contain and then by term frequency (so 席位 升级 finds pages that mention either word). Results are unchanged whenever the all-terms match already fills the request. Direct keyword reads stay all-terms. The hybrid keyword arm also has one time budget, search.cjk_keyword_deadline_ms (integer milliseconds, default 3000, range 500 to 30000; gbrain config set search.cjk_keyword_deadline_ms 5000). On Postgres the normal scoring gets two thirds of it; if that times out, a capped retry scores at most 2,000 matching chunks (all-terms matches before partial ones) in the remaining third. On PGLite, which cannot interrupt a running query, a brain past 50,000 chunks always uses the capped stage. When the cap or the budget cut the candidates, the search reports the degraded stage keyword_candidates_incomplete: an incomplete keyword search is not evidence that the brain has nothing, and the next step is to narrow with --source-id (MCP source_id) or raise the deadline. The vector arm still runs either way. Ranking quality among the capped candidates has not been measured against relevance judgments.