Garry's Opinionated OpenClaw/Hermes Agent Brain
Multi-language full-text search
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
The tokenizer language is configurable via the GBRAIN_FTS_LANGUAGE
environment variable. Default: english.
How it works
Postgres text-search configurations control stemming and stop-word removal.
GBRAIN_FTS_LANGUAGE is read by src/core/fts-language.ts and applied on
both sides of the search:
- Query side —
websearch_to_tsquery('<lang>', $query)in both engines (Postgres and PGLite). - Write side — the
update_page_search_vectorandupdate_chunk_search_vectortrigger functions that populatepages.search_vectorandcontent_chunks.search_vector.
The value is validated against /^[a-z][a-z0-9_]*$/ before it is ever
interpolated into SQL (tsvector functions don't accept parameterized config
names). Invalid values fall back to english with a warning.
Built-in languages
Set the env var to any configuration your Postgres instance ships:
export GBRAIN_FTS_LANGUAGE=portuguese
export GBRAIN_FTS_LANGUAGE=spanish
export GBRAIN_FTS_LANGUAGE=german
List what's available:
SELECT cfgname FROM pg_ts_config;
PGLite (the embedded default engine) ships the same built-in snowball configurations as stock Postgres.
First install vs. changing language later
On first install (or upgrade), the configurable_fts_language schema
migration reads GBRAIN_FTS_LANGUAGE and stamps the trigger functions with
that language. After the migration has run, changing the env var alone does
NOT retokenize existing rows — the migration shows as applied and is skipped.
Use the explicit command:
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview: language + row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill + rebuild the facts keyword index
The stamp survives later schema work: initSchema() — including the replay
behind gbrain init --migrate-only on every upgrade — applies the schema
template under the configured language, so it re-creates the trigger
functions as they already are instead of reverting them to english.
The command recreates both trigger functions under the new language and
backfills every existing pages and content_chunks row in batches,
streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. --json prints a machine-readable
result envelope but still requires --yes (or an interactive confirm).
No cache purge is needed. The resolved language is part of the query-cache
key, so rows written under the previous language are unreachable after the
switch — searches read the retokenized index immediately instead of being
served pre-switch results for up to search.cache.ttl_seconds. Switching
back reaches the original rows rather than rebuilding them.
Recipe: accent-insensitive Portuguese (pt_br)
Brazilian Portuguese content often mixes accented and unaccented spellings
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
the unaccent extension, then stems with the portuguese snowball dictionary:
CREATE EXTENSION IF NOT EXISTS unaccent;
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
ALTER TEXT SEARCH CONFIGURATION pt_br
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, portuguese_stem;
Then point GBrain at it:
export GBRAIN_FTS_LANGUAGE=pt_br
gbrain reindex-search-vector --yes
Note: custom configurations require a real Postgres instance (e.g. the
Supabase engine). The config must exist BEFORE the migration or the reindex
command runs, or Postgres will reject the trigger recreation with
text search configuration "pt_br" does not exist.
Caveats
- One language per brain: the setting is global to the database, not per-source. Mixed-language brains should pick the dominant language (the vector-search arm is language-agnostic and covers the rest).
- Keep
GBRAIN_FTS_LANGUAGEset consistently in every environment that writes to the brain (CLI shells, MCP server, cron jobs) — a writer without the env var tokenizes new rows inenglishuntil the next reindex. - Remote title matching (MCP and other safe-chunks callers) reads the title
lexemes stored in
pages.search_vector, so it follows the language those vectors were built with. After a language change, rungbrain reindex-search-vectorso remote title search tokenizes like the query. - Interrupted reindex: the trigger flip commits before the backfill, so a
reindex-search-vectorrun killed mid-way (crash, full disk, SIGKILL) leaves new writes in the new language and un-backfilled rows in the old one — keyword search then matches only part of the corpus. The command records an in-progress marker plus a per-batch checkpoint,gbrain doctorfails withfts_reindex_incompleteuntil the run completes, and re-running with the sameGBRAIN_FTS_LANGUAGEresumes from the checkpoint rather than starting over. Budget minutes, not seconds, on brains with 100K+ chunks. - CJK (Chinese / Japanese / Korean): none of the built-in snowball
configurations can tokenize CJK text, so the FTS arm would return nothing
for CJK queries. Both engines detect CJK queries and route them to a
term-by-term
ILIKEfallback with term-frequency ranking instead (shared SQL insrc/core/search/cjk-keyword-sql.ts). The fallback is correct but not index-accelerated: it scanscontent_chunks.chunk_text, so latency grows with corpus size. Note the routing is query-driven: any query containing CJK characters takes the fallback today, even on a Postgres instance with a CJK-aware extension (pgroonga/zhparser) installed. Wiring a CJK-capableGBRAIN_FTS_LANGUAGEconfig past the fallback is a filed follow-up. In hybrid search the fallback keeps every term required first and, when that finds fewer rows than requested, matches any term, ranking chunks by how many terms they contain and then by term frequency (so席位 升级finds pages that mention either word). Results are unchanged whenever the all-terms match already fills the request. Direct keyword reads stay all-terms. The hybrid keyword arm also has one time budget,search.cjk_keyword_deadline_ms(integer milliseconds, default 3000, range 500 to 30000;gbrain config set search.cjk_keyword_deadline_ms 5000). On Postgres the normal scoring gets two thirds of it; if that times out, a capped retry scores at most 2,000 matching chunks (all-terms matches before partial ones) in the remaining third. On PGLite, which cannot interrupt a running query, a brain past 50,000 chunks always uses the capped stage. When the cap or the budget cut the candidates, the search reports the degraded stagekeyword_candidates_incomplete: an incomplete keyword search is not evidence that the brain has nothing, and the next step is to narrow with--source-id(MCPsource_id) or raise the deadline. The vector arm still runs either way. Ranking quality among the capped candidates has not been measured against relevance judgments.