Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(search): address review on MessageSearch defaults and the SQLite contains build

Luisfc71d6f0f828
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ab5f17000..5de9ab21c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,8 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
-- Add `match=contains` to `GET /api/search`, which needs `SEARCH_CONTAINS_ENABLED=true` (off by default; the route answers `501` otherwise): case- and accent-insensitive substring search on a trigram index (FTS5 on SQLite, `pg_trgm` and `unaccent` on PostgreSQL) that ranks the 1,000 most recently stored matches, so `total` is at most 1,000 in this mode; `match=word` stays the default and the SDKs accept `match`. Thanks @luisffm.
-- Add the `MessageSearch` MCP tool, which runs the global message search for agents. It defaults to `match=contains` only with `SEARCH_CONTAINS_ENABLED=true`, and searches by word otherwise or while the contains index is not ready. Thanks @luisffm.
+- Add `match=contains` to `GET /api/search`, which needs `SEARCH_CONTAINS_ENABLED=true`: case- and accent-insensitive substring search on a trigram index (FTS5 on SQLite, `pg_trgm` and `unaccent` on PostgreSQL) that ranks the 1,000 most recently stored matches, so `total` is at most 1,000 in this mode; `match=word` stays the default and the SDKs accept `match`. Thanks @luisffm.
+- Add the `MessageSearch` MCP tool, which runs the global message search for agents, with `match=contains` by default only when `SEARCH_CONTAINS_ENABLED=true`. Thanks @luisffm.
## [0.26.0] - 2026-10-28
diff --git a/docs/26-global-search.md b/docs/26-global-search.md
index eaca1ead0..e23096e8f 100644
--- a/docs/26-global-search.md
+++ b/docs/26-global-search.md
@@ -80,7 +80,8 @@ building, `match=contains` answers `501`:
`contains` finds everything `word` finds, plus partial words. Every whitespace-separated term must
appear, as with `word`; a repeated term (also one that differs only in case or accents) runs once. A
trigram index cannot look up a term it takes no trigram from, so such a term only narrows what another
-term found, and a query made of such terms alone answers `400`. What counts differs by dialect: SQLite
+term found, and a query made of such terms alone, two-character CJK words such as `東京` included,
+answers `400`. What counts differs by dialect: SQLite
takes trigrams from any three characters, so a term needs three (`id` and `pm` only narrow, `:-)`
searches); `pg_trgm` takes them from letters and digits only, so on Postgres a term needs three of them
in a row (`:-)`, `...`, `😂😂😂` and `a.b` only narrow there, and alone answer `400`). The query grammar of
@@ -115,7 +116,9 @@ Postgres's `unaccent` also transliterates letters that have no decomposition: `
and `ё` become `L`, `ss`, `O`, `i`, `oe` and `е`. So on Postgres `lodz` finds "Łódź", `strasse` finds
"Straße" and `orsted` finds "Ørsted", while SQLite's `remove_diacritics` keeps those letters and finds
none of them. The application compares and highlights the way SQLite folds, so those extra Postgres hits
-come back with `score` 0 and a snippet without `<mark>`.
+come back with `score` 0 and a snippet without `<mark>`. A term under three characters narrows with `LIKE`
+on SQLite, which folds ASCII case only and keeps accents, and with `ILIKE` on `openwa_unaccent(body)` on
+Postgres, which folds both.
**Requirements.** `SEARCH_CONTAINS_ENABLED=true`. On SQLite, `contains` needs FTS5 and SQLite 3.45+
(the first with `remove_diacritics` for the trigram tokenizer); the bundled `better-sqlite3` ships both.
@@ -298,8 +301,8 @@ answering its health probe (a boot that outlasts the Helm startup probe is resta
the existing rows 1,000 rowids at a time, yielding to the event loop between batches; rows stored
meanwhile are indexed by the triggers. A trigram index holds every three-character window of the body:
on a synthetic corpus of 1,000,000 messages it took 289 MB, about 3.1 times the message text, and a
- 9.4 s backfill. Later boots only check it and re-add what is missing, so an interrupted backfill
- resumes where it stopped.
+ 9.4 s backfill. Later boots only check it; an index whose document count differs from `messages` (an
+ interrupted backfill, or rowids renumbered by a table rebuild) is emptied and backfilled again.
- **PostgreSQL** creates the `pg_trgm` and `unaccent` extensions when they are missing and the role may,
the IMMUTABLE `openwa_unaccent(text)` wrapper, and the `idx_messages_body_trgm` GIN index on
`openwa_unaccent(body)` with `CREATE INDEX CONCURRENTLY`, so writes go on during the build (it cannot
diff --git a/src/core/agent-tools/tools/search.tools.spec.ts b/src/core/agent-tools/tools/search.tools.spec.ts
index 696b426c8..ce59b400f 100644
--- a/src/core/agent-tools/tools/search.tools.spec.ts
+++ b/src/core/agent-tools/tools/search.tools.spec.ts
@@ -51,6 +51,17 @@ describe('MessageSearch tool', () => {
expect(search).toHaveBeenCalledWith({ q: 'invoice', match: 'contains' }, undefined, undefined);
});
+ it('bounds dateFrom and dateTo to the safe integer range, as REST does', async () => {
+ const key = { id: 'k1', allowedSessions: null, allowedChats: null };
+
+ await expect(call(key, { q: 'invoice', dateFrom: 1e22 })).rejects.toBeInstanceOf(BadRequestException);
+ await expect(call(key, { q: 'invoice', dateTo: -1e22 })).rejects.toBeInstanceOf(BadRequestException);
+ expect(search).not.toHaveBeenCalled();
+
+ await call(key, { q: 'invoice', dateFrom: -Number.MAX_SAFE_INTEGER, dateTo: Number.MAX_SAFE_INTEGER });
+ expect(search).toHaveBeenCalledTimes(1);
+ });
+
it('rejects an unknown match mode', async () => {
await expect(call({ id: 'k1', allowedSessions: null }, { q: 'invoice', match: 'fuzzy' })).rejects.toBeInstanceOf(
BadRequestException,
diff --git a/src/core/agent-tools/tools/search.tools.ts b/src/core/agent-tools/tools/search.tools.ts
index 6f42a0781..b4f70c824 100644
--- a/src/core/agent-tools/tools/search.tools.ts
+++ b/src/core/agent-tools/tools/search.tools.ts
@@ -46,13 +46,23 @@ export function searchTools(search: SearchService | undefined, containsEnabled =
q: z.string().trim().min(1).max(SEARCH_QUERY_MAX_LENGTH).describe('Search term'),
match: z
.enum(SEARCH_MATCH_MODES)
- .default(defaultMatch)
+ .optional()
.describe(`contains: any part of a word; word: whole words only. Default: ${defaultMatch}`),
chatId: chatIdSchema.optional().describe('Restrict to a single chat JID'),
from: z.string().optional().describe('Filter by sender phone or JID'),
direction: z.enum(MessageDirection).optional(),
- dateFrom: z.number().optional().describe('Epoch-ms lower bound (inclusive)'),
- dateTo: z.number().optional().describe('Epoch-ms upper bound (inclusive)'),
+ dateFrom: z
+ .number()
+ .min(-Number.MAX_SAFE_INTEGER)
+ .max(Number.MAX_SAFE_INTEGER)
+ .optional()
+ .describe('Epoch-ms lower bound (inclusive)'),
+ dateTo: z
+ .number()
+ .min(-Number.MAX_SAFE_INTEGER)
+ .max(Number.MAX_SAFE_INTEGER)
+ .optional()
+ .describe('Epoch-ms upper bound (inclusive)'),
limit: z.number().int().min(1).max(SEARCH_LIMIT_MAX).optional(),
offset: z.number().int().min(0).max(SEARCH_OFFSET_MAX).optional(),
}),
@@ -60,7 +70,14 @@ export function searchTools(search: SearchService | undefined, containsEnabled =
if (!search) throw new NotImplementedException('Search is disabled (SEARCH_ENABLED=false).');
// Both scopes come from the authenticated key, never the input, exactly as SearchController
// passes them: SearchService overwrites any caller-supplied scope at the provider boundary.
- return search.search(input, apiKey.allowedSessions ?? undefined, await chatScope?.idsForFilter(apiKey));
+ const sessionIds = apiKey.allowedSessions ?? undefined;
+ const chatIds = await chatScope?.idsForFilter(apiKey);
+ try {
+ return await search.search({ ...input, match: input.match ?? defaultMatch }, sessionIds, chatIds);
+ } catch (error) {
+ if (input.match || defaultMatch === 'word' || !(error instanceof NotImplementedException)) throw error;
+ return search.search({ ...input, match: 'word' }, sessionIds, chatIds);
+ }
},
}),
];
diff --git a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts b/src/modules/search/providers/builtin-fts.provider.contains.spec.ts
index 2eca15abb..742e04336 100644
--- a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts
+++ b/src/modules/search/providers/builtin-fts.provider.contains.spec.ts
@@ -190,8 +190,7 @@ describe('BuiltInFtsProvider trigram index build (sqlite)', () => {
const trigramTables = async () =>
(await ds.query<unknown[]>(`SELECT name FROM sqlite_master WHERE name = 'messages_trgm'`)).length;
- // The `rank = 1` form checks the index against the content table; valid here because no fixture body
- // is NULL (see builtin-fts.provider.backfill.spec.ts).
+ // The `rank = 1` form checks the index against the content table, NULL bodies included.
const integrityCheck = () => ds.query(`INSERT INTO messages_trgm(messages_trgm, rank) VALUES ('integrity-check', 1)`);
const insertBulk = async (count: number, body: (i: number) => string) => {
@@ -220,6 +219,7 @@ describe('BuiltInFtsProvider trigram index build (sqlite)', () => {
it('builds over existing rows twice into a sound index the triggers then keep current', async () => {
await insertBulk(2500, i => `Bulk receipt ${i}`);
+ await ds.query(`UPDATE messages SET body = NULL WHERE body = 'Bulk receipt 7'`);
const provider = new BuiltInFtsProvider(ds, containsOn);
await provider.buildContainsIndex();
diff --git a/src/modules/search/providers/builtin-fts.provider.ts b/src/modules/search/providers/builtin-fts.provider.ts
index 31f8713af..07344f79e 100644
--- a/src/modules/search/providers/builtin-fts.provider.ts
+++ b/src/modules/search/providers/builtin-fts.provider.ts
@@ -10,6 +10,7 @@ import { ConfigService } from '@nestjs/config';
import { HttpAdapterHost } from '@nestjs/core';
import { setImmediate } from 'node:timers/promises';
import { createLogger } from '../../../common/services/logger.service';
+import { awaitNoOpenTransaction } from '../../../common/utils/shared-connection';
import { InjectDataSource } from '@nestjs/typeorm';
import { DataSource, QueryRunner } from 'typeorm';
import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';
@@ -388,7 +389,14 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli
await this.dataSource.query(`INSERT INTO "${table}"("${table}") VALUES ('delete-all')`);
}
// A batched index is filled after its triggers exist instead (backfillSqliteIndexInBatches).
- if (!batched) {
+ if (batched) {
+ const [counts] = await this.dataSource.query<Array<{ messages: number; indexed: number }>>(
+ `SELECT (SELECT count(*) FROM "messages") AS messages, (SELECT count(*) FROM "${docsize}") AS indexed`,
+ );
+ if (Number(counts.messages) !== Number(counts.indexed)) {
+ await this.dataSource.query(`INSERT INTO "${table}"("${table}") VALUES ('delete-all')`);
+ }
+ } else {
const gapRow: unknown[] = await this.dataSource.query(
`SELECT EXISTS(SELECT 1 FROM "messages" m WHERE m."body" IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM "${docsize}" d WHERE d."rowid" = m."rowid")) AS missing`,
@@ -440,10 +448,11 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli
`SELECT max("rowid") AS last FROM "messages"`,
);
for (let from = 0; from < (last ?? 0); from += TRIGRAM_BACKFILL_BATCH) {
+ if (!(await awaitNoOpenTransaction(this.dataSource))) throw new Error('Another database transaction stayed open');
await this.dataSource.query(
`INSERT INTO "${table}"("rowid", "body")
SELECT m."rowid", m."body" FROM "messages" m
- WHERE m."rowid" > ? AND m."rowid" <= ? AND m."body" IS NOT NULL
+ WHERE m."rowid" > ? AND m."rowid" <= ?
AND NOT EXISTS (SELECT 1 FROM "${table}_docsize" d WHERE d."rowid" = m."rowid")`,
[from, from + TRIGRAM_BACKFILL_BATCH],
);
@@ -478,6 +487,7 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli
if (!(await this.sqliteHasFts5())) return CONTAINS_UNAVAILABLE.sqliteTrigram;
const rows: Array<{ version: string }> = await this.dataSource.query(`SELECT sqlite_version() AS version`);
if (!sqliteSupportsTrigramDiacritics(rows[0]?.version ?? '')) return CONTAINS_UNAVAILABLE.sqliteTrigram;
+ if (!(await awaitNoOpenTransaction(this.dataSource))) throw new Error('Another database transaction stayed open');
await this.ensureSqliteIndex(TRIGRAM_INDEX);
return null;
}
diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.ts
index f21997b88..2eed09cdf 100644
--- a/src/modules/search/search-terms.ts
+++ b/src/modules/search/search-terms.ts
@@ -50,12 +50,6 @@ function wordsOf(word: string): number {
return words;
}
-/**
- * How many words a query holds: each run of letters or digits is one, so punctuation separates words as
- * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and a
- * symbol or emoji that stands apart counts as a word of its own. A whitespace-separated word said again
- * is run once and counts once, and a word with no letter, digit or symbol counts as one.
- */
/**
* Folds text the way the match=contains indexes compare it: accents dropped (NFD, then every combining
* mark removed, as remove_diacritics and unaccent do for Latin text) and case lowered. Used to compare
@@ -79,6 +73,12 @@ export function containsTerms(q: string): string[] {
return [...terms.values()];
}
+/**
+ * How many words a query holds: each run of letters or digits is one, so punctuation separates words as
+ * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and a
+ * symbol or emoji that stands apart counts as a word of its own. A whitespace-separated word said again
+ * is run once and counts once, and a word with no letter, digit or symbol counts as one.
+ */
export function searchTermCount(q: string): number {
let count = 0;
for (const word of new Set(q.trim().split(/\s+/))) {