Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(search): address review on MessageSearch defaults and the SQLite contains build
Luisfc71d6f0f828
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ab5f17000..5de9ab21c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,8 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added-- Add `match=contains` to `GET /api/search`, which needs `SEARCH_CONTAINS_ENABLED=true` (off by default; the route answers `501` otherwise): case- and accent-insensitive substring search on a trigram index (FTS5 on SQLite, `pg_trgm` and `unaccent` on PostgreSQL) that ranks the 1,000 most recently stored matches, so `total` is at most 1,000 in this mode; `match=word` stays the default and the SDKs accept `match`. Thanks @luisffm.-- Add the `MessageSearch` MCP tool, which runs the global message search for agents. It defaults to `match=contains` only with `SEARCH_CONTAINS_ENABLED=true`, and searches by word otherwise or while the contains index is not ready. Thanks @luisffm.+- Add `match=contains` to `GET /api/search`, which needs `SEARCH_CONTAINS_ENABLED=true`: case- and accent-insensitive substring search on a trigram index (FTS5 on SQLite, `pg_trgm` and `unaccent` on PostgreSQL) that ranks the 1,000 most recently stored matches, so `total` is at most 1,000 in this mode; `match=word` stays the default and the SDKs accept `match`. Thanks @luisffm.+- Add the `MessageSearch` MCP tool, which runs the global message search for agents, with `match=contains` by default only when `SEARCH_CONTAINS_ENABLED=true`. Thanks @luisffm.## [0.26.0] - 2026-10-28diff --git a/docs/26-global-search.md b/docs/26-global-search.mdindex eaca1ead0..e23096e8f 100644--- a/docs/26-global-search.md+++ b/docs/26-global-search.md@@ -80,7 +80,8 @@ building, `match=contains` answers `501`:`contains` finds everything `word` finds, plus partial words. Every whitespace-separated term mustappear, as with `word`; a repeated term (also one that differs only in case or accents) runs once. Atrigram index cannot look up a term it takes no trigram from, so such a term only narrows what another-term found, and a query made of such terms alone answers `400`. What counts differs by dialect: SQLite+term found, and a query made of such terms alone, two-character CJK words such as `東京` included,+answers `400`. What counts differs by dialect: SQLitetakes trigrams from any three characters, so a term needs three (`id` and `pm` only narrow, `:-)`searches); `pg_trgm` takes them from letters and digits only, so on Postgres a term needs three of themin a row (`:-)`, `...`, `😂😂😂` and `a.b` only narrow there, and alone answer `400`). The query grammar of@@ -115,7 +116,9 @@ Postgres's `unaccent` also transliterates letters that have no decomposition: `and `ё` become `L`, `ss`, `O`, `i`, `oe` and `е`. So on Postgres `lodz` finds "Łódź", `strasse` finds"Straße" and `orsted` finds "Ørsted", while SQLite's `remove_diacritics` keeps those letters and findsnone of them. The application compares and highlights the way SQLite folds, so those extra Postgres hits-come back with `score` 0 and a snippet without `<mark>`.+come back with `score` 0 and a snippet without `<mark>`. A term under three characters narrows with `LIKE`+on SQLite, which folds ASCII case only and keeps accents, and with `ILIKE` on `openwa_unaccent(body)` on+Postgres, which folds both.**Requirements.** `SEARCH_CONTAINS_ENABLED=true`. On SQLite, `contains` needs FTS5 and SQLite 3.45+(the first with `remove_diacritics` for the trigram tokenizer); the bundled `better-sqlite3` ships both.@@ -298,8 +301,8 @@ answering its health probe (a boot that outlasts the Helm startup probe is restathe existing rows 1,000 rowids at a time, yielding to the event loop between batches; rows storedmeanwhile are indexed by the triggers. A trigram index holds every three-character window of the body:on a synthetic corpus of 1,000,000 messages it took 289 MB, about 3.1 times the message text, and a- 9.4 s backfill. Later boots only check it and re-add what is missing, so an interrupted backfill- resumes where it stopped.+ 9.4 s backfill. Later boots only check it; an index whose document count differs from `messages` (an+ interrupted backfill, or rowids renumbered by a table rebuild) is emptied and backfilled again.- **PostgreSQL** creates the `pg_trgm` and `unaccent` extensions when they are missing and the role may,the IMMUTABLE `openwa_unaccent(text)` wrapper, and the `idx_messages_body_trgm` GIN index on`openwa_unaccent(body)` with `CREATE INDEX CONCURRENTLY`, so writes go on during the build (it cannotdiff --git a/src/core/agent-tools/tools/search.tools.spec.ts b/src/core/agent-tools/tools/search.tools.spec.tsindex 696b426c8..ce59b400f 100644--- a/src/core/agent-tools/tools/search.tools.spec.ts+++ b/src/core/agent-tools/tools/search.tools.spec.ts@@ -51,6 +51,17 @@ describe('MessageSearch tool', () => {expect(search).toHaveBeenCalledWith({ q: 'invoice', match: 'contains' }, undefined, undefined);});+ it('bounds dateFrom and dateTo to the safe integer range, as REST does', async () => {+ const key = { id: 'k1', allowedSessions: null, allowedChats: null };++ await expect(call(key, { q: 'invoice', dateFrom: 1e22 })).rejects.toBeInstanceOf(BadRequestException);+ await expect(call(key, { q: 'invoice', dateTo: -1e22 })).rejects.toBeInstanceOf(BadRequestException);+ expect(search).not.toHaveBeenCalled();++ await call(key, { q: 'invoice', dateFrom: -Number.MAX_SAFE_INTEGER, dateTo: Number.MAX_SAFE_INTEGER });+ expect(search).toHaveBeenCalledTimes(1);+ });+it('rejects an unknown match mode', async () => {await expect(call({ id: 'k1', allowedSessions: null }, { q: 'invoice', match: 'fuzzy' })).rejects.toBeInstanceOf(BadRequestException,diff --git a/src/core/agent-tools/tools/search.tools.ts b/src/core/agent-tools/tools/search.tools.tsindex 6f42a0781..b4f70c824 100644--- a/src/core/agent-tools/tools/search.tools.ts+++ b/src/core/agent-tools/tools/search.tools.ts@@ -46,13 +46,23 @@ export function searchTools(search: SearchService | undefined, containsEnabled =q: z.string().trim().min(1).max(SEARCH_QUERY_MAX_LENGTH).describe('Search term'),match: z.enum(SEARCH_MATCH_MODES)- .default(defaultMatch)+ .optional().describe(`contains: any part of a word; word: whole words only. Default: ${defaultMatch}`),chatId: chatIdSchema.optional().describe('Restrict to a single chat JID'),from: z.string().optional().describe('Filter by sender phone or JID'),direction: z.enum(MessageDirection).optional(),- dateFrom: z.number().optional().describe('Epoch-ms lower bound (inclusive)'),- dateTo: z.number().optional().describe('Epoch-ms upper bound (inclusive)'),+ dateFrom: z+ .number()+ .min(-Number.MAX_SAFE_INTEGER)+ .max(Number.MAX_SAFE_INTEGER)+ .optional()+ .describe('Epoch-ms lower bound (inclusive)'),+ dateTo: z+ .number()+ .min(-Number.MAX_SAFE_INTEGER)+ .max(Number.MAX_SAFE_INTEGER)+ .optional()+ .describe('Epoch-ms upper bound (inclusive)'),limit: z.number().int().min(1).max(SEARCH_LIMIT_MAX).optional(),offset: z.number().int().min(0).max(SEARCH_OFFSET_MAX).optional(),}),@@ -60,7 +70,14 @@ export function searchTools(search: SearchService | undefined, containsEnabled =if (!search) throw new NotImplementedException('Search is disabled (SEARCH_ENABLED=false).');// Both scopes come from the authenticated key, never the input, exactly as SearchController// passes them: SearchService overwrites any caller-supplied scope at the provider boundary.- return search.search(input, apiKey.allowedSessions ?? undefined, await chatScope?.idsForFilter(apiKey));+ const sessionIds = apiKey.allowedSessions ?? undefined;+ const chatIds = await chatScope?.idsForFilter(apiKey);+ try {+ return await search.search({ ...input, match: input.match ?? defaultMatch }, sessionIds, chatIds);+ } catch (error) {+ if (input.match || defaultMatch === 'word' || !(error instanceof NotImplementedException)) throw error;+ return search.search({ ...input, match: 'word' }, sessionIds, chatIds);+ }},}),];diff --git a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts b/src/modules/search/providers/builtin-fts.provider.contains.spec.tsindex 2eca15abb..742e04336 100644--- a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts+++ b/src/modules/search/providers/builtin-fts.provider.contains.spec.ts@@ -190,8 +190,7 @@ describe('BuiltInFtsProvider trigram index build (sqlite)', () => {const trigramTables = async () =>(await ds.query<unknown[]>(`SELECT name FROM sqlite_master WHERE name = 'messages_trgm'`)).length;- // The `rank = 1` form checks the index against the content table; valid here because no fixture body- // is NULL (see builtin-fts.provider.backfill.spec.ts).+ // The `rank = 1` form checks the index against the content table, NULL bodies included.const integrityCheck = () => ds.query(`INSERT INTO messages_trgm(messages_trgm, rank) VALUES ('integrity-check', 1)`);const insertBulk = async (count: number, body: (i: number) => string) => {@@ -220,6 +219,7 @@ describe('BuiltInFtsProvider trigram index build (sqlite)', () => {it('builds over existing rows twice into a sound index the triggers then keep current', async () => {await insertBulk(2500, i => `Bulk receipt ${i}`);+ await ds.query(`UPDATE messages SET body = NULL WHERE body = 'Bulk receipt 7'`);const provider = new BuiltInFtsProvider(ds, containsOn);await provider.buildContainsIndex();diff --git a/src/modules/search/providers/builtin-fts.provider.ts b/src/modules/search/providers/builtin-fts.provider.tsindex 31f8713af..07344f79e 100644--- a/src/modules/search/providers/builtin-fts.provider.ts+++ b/src/modules/search/providers/builtin-fts.provider.ts@@ -10,6 +10,7 @@ import { ConfigService } from '@nestjs/config';import { HttpAdapterHost } from '@nestjs/core';import { setImmediate } from 'node:timers/promises';import { createLogger } from '../../../common/services/logger.service';+import { awaitNoOpenTransaction } from '../../../common/utils/shared-connection';import { InjectDataSource } from '@nestjs/typeorm';import { DataSource, QueryRunner } from 'typeorm';import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';@@ -388,7 +389,14 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppliawait this.dataSource.query(`INSERT INTO "${table}"("${table}") VALUES ('delete-all')`);}// A batched index is filled after its triggers exist instead (backfillSqliteIndexInBatches).- if (!batched) {+ if (batched) {+ const [counts] = await this.dataSource.query<Array<{ messages: number; indexed: number }>>(+ `SELECT (SELECT count(*) FROM "messages") AS messages, (SELECT count(*) FROM "${docsize}") AS indexed`,+ );+ if (Number(counts.messages) !== Number(counts.indexed)) {+ await this.dataSource.query(`INSERT INTO "${table}"("${table}") VALUES ('delete-all')`);+ }+ } else {const gapRow: unknown[] = await this.dataSource.query(`SELECT EXISTS(SELECT 1 FROM "messages" m WHERE m."body" IS NOT NULLAND NOT EXISTS (SELECT 1 FROM "${docsize}" d WHERE d."rowid" = m."rowid")) AS missing`,@@ -440,10 +448,11 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli`SELECT max("rowid") AS last FROM "messages"`,);for (let from = 0; from < (last ?? 0); from += TRIGRAM_BACKFILL_BATCH) {+ if (!(await awaitNoOpenTransaction(this.dataSource))) throw new Error('Another database transaction stayed open');await this.dataSource.query(`INSERT INTO "${table}"("rowid", "body")SELECT m."rowid", m."body" FROM "messages" m- WHERE m."rowid" > ? AND m."rowid" <= ? AND m."body" IS NOT NULL+ WHERE m."rowid" > ? AND m."rowid" <= ?AND NOT EXISTS (SELECT 1 FROM "${table}_docsize" d WHERE d."rowid" = m."rowid")`,[from, from + TRIGRAM_BACKFILL_BATCH],);@@ -478,6 +487,7 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppliif (!(await this.sqliteHasFts5())) return CONTAINS_UNAVAILABLE.sqliteTrigram;const rows: Array<{ version: string }> = await this.dataSource.query(`SELECT sqlite_version() AS version`);if (!sqliteSupportsTrigramDiacritics(rows[0]?.version ?? '')) return CONTAINS_UNAVAILABLE.sqliteTrigram;+ if (!(await awaitNoOpenTransaction(this.dataSource))) throw new Error('Another database transaction stayed open');await this.ensureSqliteIndex(TRIGRAM_INDEX);return null;}diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.tsindex f21997b88..2eed09cdf 100644--- a/src/modules/search/search-terms.ts+++ b/src/modules/search/search-terms.ts@@ -50,12 +50,6 @@ function wordsOf(word: string): number {return words;}-/**- * How many words a query holds: each run of letters or digits is one, so punctuation separates words as- * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and a- * symbol or emoji that stands apart counts as a word of its own. A whitespace-separated word said again- * is run once and counts once, and a word with no letter, digit or symbol counts as one.- *//*** Folds text the way the match=contains indexes compare it: accents dropped (NFD, then every combining* mark removed, as remove_diacritics and unaccent do for Latin text) and case lowered. Used to compare@@ -79,6 +73,12 @@ export function containsTerms(q: string): string[] {return [...terms.values()];}+/**+ * How many words a query holds: each run of letters or digits is one, so punctuation separates words as+ * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and a+ * symbol or emoji that stands apart counts as a word of its own. A whitespace-separated word said again+ * is run once and counts once, and a word with no letter, digit or symbol counts as one.+ */export function searchTermCount(q: string): number {let count = 0;for (const word of new Set(q.trim().split(/\s+/))) {