Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(search): bound contains queries and fold text as the index does
Yudhi Armyndharis431e19a6580b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b340752ac..2b9639d43 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -30,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Index a Baileys clear, delete, archive or unread by the same chat jid as its message key, so the phone applies it (#1891). Thanks @rjsebening.- Key Baileys chat writes and read receipts to the gateway's newest stored message when the session holds none in memory (#1891). Thanks @rjsebening.- Group dashboard phone numbers in threes and fours, so `+49 157 333 4 4455` now reads `+49 157 3334 4455`; 10-digit numbers keep their shape (#1940).+- Answer `400` to a NUL character in `q` on `GET /api/search`, which failed with `500` on PostgreSQL.## [0.26.0] - 2026-10-28diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 42756a3a6..d2a20dfae 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6603,19 +6603,19 @@ Search messages across sessions (active search provider).**Query parameters**-| Name | Type | Required | Default | Description |-| ----------- | ------------------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words, a symbol or emoji that stands apart counts as a word, a whitespace-separated word said again counts once). Passed to the active provider's native full-text matcher. |-| `match` | enum (`word` \| `contains`) | No | `word` | `word` matches whole words. `contains` matches any part of a word, ignoring case and accents (`voic` finds "Invoice", `cafe` finds "Café"); every term must appear, at least one term needs 3+ characters, on PostgreSQL 3+ letters or digits in a row (`400` otherwise), and `%`/`_` match literally. It ranks the 1,000 most recently stored matches by match quality (whole word, then word start, then inside a word; newest first within a score), so `total` is at most 1,000. Opt-in on the built-in provider: `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true` and has finished building the index (on PostgreSQL also the `pg_trgm` and `unaccent` extensions). PostgreSQL also folds letters such as `Ł`, `ß` and `Ø` that SQLite keeps. See [26 - Global Search](./26-global-search.md#match-modes). |-| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |-| `chatId` | string | No | — | Restrict to a single chat id. With the built-in provider, a phone also matches the chat's other stored forms (the other user dialect, any lid that resolves to it). For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are also narrowed to the allowlist. |-| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |-| `type` | string | No | — | Filter by stored message `type` (e.g. `text`, `image`, `video`). Compared against `messages.type`; not an enum validation, any string is accepted and unmatched values simply return no hits. |-| `from` | string | No | — | Filter by sender; with the built-in provider, matches `from` or a group message's `author`, and a phone also matches any lid that resolves to it. |-| `dateFrom` | integer (epoch ms) | No | — | Inclusive lower bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |-| `dateTo` | integer (epoch ms) | No | — | Inclusive upper bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |-| `limit` | integer (≥ 1) | No | `50` | Max hits to return. Clamped to `SEARCH_LIMIT_MAX` (default `100`). A non-integer value is rejected with `400`. |-| `offset` | integer (≥ 0) | No | `0` | Pagination offset, at most `100000`. A non-integer or larger value is rejected with `400`. |+| Name | Type | Required | Default | Description |+| ----------- | ------------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, over 32 words (punctuation separates words, a symbol or emoji that stands apart counts as a word, a whitespace-separated word said again counts once), or holding a NUL character. Passed to the active provider's native full-text matcher. |+| `match` | enum (`word` \| `contains`) | No | `word` | `word` matches whole words. `contains` matches any part of a word, ignoring case and accents (`voic` finds "Invoice", `cafe` finds "Café"); every term must appear, at least one term needs 3+ characters, on PostgreSQL 3+ letters or digits in a row, the terms together may hold at most 64 characters, spaces not counted (`400` otherwise, so a query of two-character words alone, as most Korean and many Chinese and Japanese words are, needs `word`), and `%`/`_` match literally. It ranks the 1,000 most recently stored matches by match quality (whole word, then word start, then inside a word; newest first within a score), so `total` is at most 1,000. Opt-in on the built-in provider: `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true` and has finished building the index (on PostgreSQL also the `pg_trgm` and `unaccent` extensions). PostgreSQL also folds letters such as `Ł`, `ß` and `Ø` that SQLite keeps. See [26 - Global Search](./26-global-search.md#match-modes). |+| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |+| `chatId` | string | No | — | Restrict to a single chat id. With the built-in provider, a phone also matches the chat's other stored forms (the other user dialect, any lid that resolves to it). For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are also narrowed to the allowlist. |+| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |+| `type` | string | No | — | Filter by stored message `type` (e.g. `text`, `image`, `video`). Compared against `messages.type`; not an enum validation, any string is accepted and unmatched values simply return no hits. |+| `from` | string | No | — | Filter by sender; with the built-in provider, matches `from` or a group message's `author`, and a phone also matches any lid that resolves to it. |+| `dateFrom` | integer (epoch ms) | No | — | Inclusive lower bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |+| `dateTo` | integer (epoch ms) | No | — | Inclusive upper bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |+| `limit` | integer (≥ 1) | No | `50` | Max hits to return. Clamped to `SEARCH_LIMIT_MAX` (default `100`). A non-integer value is rejected with `400`. |+| `offset` | integer (≥ 0) | No | `0` | Pagination offset, at most `100000`. A non-integer or larger value is rejected with `400`. |**Response** `200` — `SearchResults`@@ -6659,8 +6659,8 @@ Search messages across sessions (active search provider).the match quality on both dialects (3 per term found as a whole word, 2 at a word start, 1 inside aword) and every key gets it, since it reads only its own message.-**Errors:** `400` empty/whitespace `q`, a non-numeric `dateFrom`/`dateTo`/`limit`/`offset`, a `dateFrom`/`dateTo` beyond 9007199254740991 in either direction, an `offset` above `100000`, an unknown `match`, a-`match=contains` query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row), or a malformed+**Errors:** `400` empty/whitespace `q` or one holding a NUL character, a non-numeric `dateFrom`/`dateTo`/`limit`/`offset`, a `dateFrom`/`dateTo` beyond 9007199254740991 in either direction, an `offset` above `100000`, an unknown `match`, a+`match=contains` query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row) or with more than 64 characters in its terms, or a malformedSQLite FTS5 query (unbalanced quote/paren, bare operator) — Postgres's `websearch_to_tsquery` istolerant and has no equivalent · `401` missing/invalid `X-API-Key` · `403` key role below `OPERATOR`, or a chat-restricted keythat names a `chatId` outside its allowlist or uses a plugin provider ·diff --git a/docs/07-api-collection.md b/docs/07-api-collection.mdindex 954255c57..3ce4a5f83 100644--- a/docs/07-api-collection.md+++ b/docs/07-api-collection.md@@ -1752,7 +1752,7 @@ Search messages across sessions. `q` is required (non-empty after trim); optiona`sessionId`, `chatId`, `direction` (`incoming` | `outgoing`), `type`, `from`, `dateFrom`/`dateTo`(epoch ms), plus `limit` (default 50, clamped to `SEARCH_LIMIT_MAX` = 100) and `offset` (at most100000). `match` is `word` (the default, whole words) or `contains` (any part of a word, ignoring case-and accents), which answers `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true`. `q` is at most 1000 characters and 32 words, and a longer one answers `400`. A key restricted to+and accents), which answers `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true` and `400` without a term of 3 or more characters or with more than 64 characters in its terms. `q` is at most 1000 characters and 32 words with no NUL character, and any other answers `400`. A key restricted toselected chats gets `403` for a `chatId` outside its list and for a plugin search provider. Hit`snippet` fields carry `<mark>` highlight markers — render them as text, never as HTML.diff --git a/docs/26-global-search.md b/docs/26-global-search.mdindex e23096e8f..8e8454cf7 100644--- a/docs/26-global-search.md+++ b/docs/26-global-search.md@@ -68,7 +68,8 @@ never assign the raw snippet to `innerHTML`.### Match modesThe `match` query parameter picks how `q` matches a message body. Each mode has its own DB-maintained-index, so both are index lookups, never table scans. The `contains` index is opt-in+index, and every search looks its terms up there; `contains` refuses a query its index cannot look up+rather than read the whole table (how much of the index it reads is below). The `contains` index is opt-in(`SEARCH_CONTAINS_ENABLED=true`, see [26.9](#269-enabling-matchcontains)); while it is off or stillbuilding, `match=contains` answers `501`:@@ -77,26 +78,35 @@ building, `match=contains` answers `501`:| `word` (default) | whole words, as described above | `body_ts` (Postgres) / `messages_fts` (SQLite) | `ts_rank` / FTS5 `rank`, over every match || `contains` | any part of a word of 3+ characters, ignoring case and accents (`voic` finds "Invoice", `cafe` finds "Café") | a `pg_trgm` GIN index on `openwa_unaccent(body)` (Postgres) / an FTS5 `trigram` table with `remove_diacritics`, `messages_trgm` (SQLite) | match quality, over the 1,000 most recently stored matches |-`contains` finds everything `word` finds, plus partial words. Every whitespace-separated term must-appear, as with `word`; a repeated term (also one that differs only in case or accents) runs once. A-trigram index cannot look up a term it takes no trigram from, so such a term only narrows what another-term found, and a query made of such terms alone, two-character CJK words such as `東京` included,-answers `400`. What counts differs by dialect: SQLite-takes trigrams from any three characters, so a term needs three (`id` and `pm` only narrow, `:-)`-searches); `pg_trgm` takes them from letters and digits only, so on Postgres a term needs three of them-in a row (`:-)`, `...`, `😂😂😂` and `a.b` only narrow there, and alone answer `400`). The query grammar of-`word` mode (quoted phrases, `OR`, `-` on Postgres) does not apply: every term, `%` and `_` included,-matches literally.+`contains` finds parts of words as well as whole ones, but it does not run every query `word` runs.+Every whitespace-separated term must appear, as with `word`; a repeated term (also one that differs+only in case or accents) runs once. A trigram index cannot look up a term it takes no trigram from, so+such a term only narrows what another term found, and a query made of such terms alone answers `400`.+That includes every query of two-character words, as most Korean words (`서울`) and many Chinese and+Japanese ones (`東京`) are: search those with `word`. What counts differs by dialect: SQLite takes+trigrams from any three characters, so a term needs three (`id` and `pm` only narrow, `:-)` searches,+and a Hangul syllable is one character however it is encoded); `pg_trgm` takes them from letters and+digits only, so on Postgres a term needs three of them in a row (`:-)`, `...`, `😂😂😂`, `a.b` and+superscript digits such as `¹²³` only narrow there, and alone answer `400`). The terms may also hold+at most **64 characters** together, spaces not counted, or the query answers `400`: every character of+a term is checked on every message holding the term's trigrams, on SQLite on the event loop every+session shares, and on 200k messages a query of 1,000 characters took 2.8 s where one of 64 took about+100 ms. The query grammar of `word` mode (quoted phrases, `OR`, `-` on Postgres) does not apply: every+term, `%` and `_` included, matches literally.**Ranking runs in two phases.** Ranking every match in SQL costs time in proportion to the number ofmatches: measured on 1M messages with a term found in 200k of them, about 100 ms on SQLite (where thattime blocks the event loop every session shares) and 1.6 s on Postgres. So `contains` bounds the work:1. The database returns at most **1,000 candidates**: the most recently stored messages that contain- every term, newest first. Both dialects answer that order from an index and stop at 1,000, however- many messages match. SQLite reads its trigram index in insertion order; Postgres walks the existing+ every term, newest first. Both dialects answer that order from an index and stop once they hold+ 1,000 matches. SQLite reads its trigram index in insertion order; Postgres walks the existing`IDX_messages_createdAt` b-tree backward for a common term, or reads the few matches of a rare one- from the GIN index.+ from the GIN index. The window bounds the matches, not the messages read to find them: a message+ the index yields that another term then rules out (a short term, or one `pg_trgm` cannot look up)+ costs a check and counts for nothing. So a query whose looked-up term is in most messages while a+ narrowing term is in none reads every message holding the looked-up term, on Postgres each+ rechecked through `openwa_unaccent` (which wraps the STABLE `unaccent` and so cannot be inlined).2. The application ranks the candidates and pages them: each term scores **3** as a whole word, **2**at the start of a word and **1** inside a word (compared without case or accents), the hit's `score`is the sum over the terms, and ties go to the newest message, then to `messageId`. The same code@@ -202,7 +212,7 @@ GET /api/search?q=<term>&match=<word|contains>&sessionId=<id>&chatId=<id>&directis a `400`.- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one- longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and a symbol or emoji that stands apart counting as one; a whitespace-separated word said again counts once). Numeric params+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and a symbol or emoji that stands apart counting as one; a whitespace-separated word said again counts once; and one holding a NUL character, which PostgreSQL refuses as a parameter). Numeric paramsare coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,never as a `NaN` SQL parameter, and so does a `dateFrom`/`dateTo` beyond 9007199254740991 in either direction(no timestamp reaches it, and the cap keeps the value exactly representable, on both databases).@@ -223,9 +233,10 @@ GET /api/search?q=<term>&match=<word|contains>&sessionId=<id>&chatId=<id>&direct`direction`, `from`, and optional `score`. `total` is an exact count (bounded; computed lazily onlywhen the page could be full). `tookMs` is the provider-side query time. `provider` names whichbackend answered (e.g. `builtin-fts`).-- **Errors:** `400` empty/whitespace `q`, a non-numeric numeric param, an unknown `match`, or a- `match=contains` query with no term the trigram index can look up (three characters; on Postgres,- three letters or digits in a row) (an FTS5 query-grammar error is also mapped to `400`, but since the+- **Errors:** `400` empty/whitespace `q` or one holding a NUL character, a non-numeric numeric param,+ an unknown `match`, or a `match=contains` query with no term the trigram index can look up (three+ characters; on Postgres, three letters or digits in a row) or with more than 64 characters in its+ terms (an FTS5 query-grammar error is also mapped to `400`, but since theSQLite path quotes every token ordinary input no longer reaches that fallback) · `401`/`403` auth ·`501` no provider configured / FTS schema absent (e.g. a non-FTS5 SQLite build) / for`match=contains`, `SEARCH_CONTAINS_ENABLED` off, the index still building, or no way to build it (e.g.diff --git a/openapi.json b/openapi.jsonindex 920daee31..f31bc5b58 100644--- a/openapi.json+++ b/openapi.json@@ -18037,7 +18037,7 @@"name": "q","required": true,"in": "query",- "description": "Search term (required, non-empty, at most 1000 characters and 32 words)",+ "description": "Search term (required, non-empty, at most 1000 characters and 32 words, no NUL character)","schema": {"maxLength": 1000,"type": "string"@@ -18047,7 +18047,7 @@"name": "match","required": false,"in": "query",- "description": "word (default): whole words. contains: any part of a word, ignoring case and accents; ranks the 1000 most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true",+ "description": "word (default): whole words. contains: any part of a word, ignoring case and accents; ranks the 1000 most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true, a term of 3 or more characters and at most 64 characters in all its terms","schema": {"enum": ["word",@@ -18161,7 +18161,7 @@}},"400": {- "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 words; an unknown \"match\"; or a \"match=contains\" query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row)",+ "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 words or holding a NUL character; an unknown \"match\"; or a \"match=contains\" query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row) or with more than 64 characters in its terms","content": {"application/json": {"schema": {diff --git a/src/core/agent-tools/tools/search.tools.spec.ts b/src/core/agent-tools/tools/search.tools.spec.tsindex 9ff91cdc8..eb8a2c844 100644--- a/src/core/agent-tools/tools/search.tools.spec.ts+++ b/src/core/agent-tools/tools/search.tools.spec.ts@@ -4,6 +4,7 @@ import type { AuthService } from '../../../modules/auth/auth.service';import { ApiKeyRole } from '../../../modules/auth/entities/api-key.entity';import type { SearchService } from '../../../modules/search/search.service';import { invokeTool } from '../tool-invoker';+import { ContainsTermsRefusedError } from '../../../modules/search/search-terms';import { searchTools } from './search.tools';const CHAT = '[email protected]';@@ -64,13 +65,38 @@ describe('MessageSearch tool', () => {await expect(call(key, { q: 'invoice', match: 'contains' })).rejects.toBeInstanceOf(NotImplementedException);expect(search).toHaveBeenCalledTimes(1);- // Only the 501 of an unready index falls back; any other failure reaches the agent as it was.+ // Only the 501 of an unready index and a refusal of the terms (below) fall back; any other failure+ // reaches the agent as it was.search.mockClear();search.mockRejectedValueOnce(new BadRequestException('Every term is under three characters'));await expect(call(key, { q: 'in' })).rejects.toBeInstanceOf(BadRequestException);expect(search).toHaveBeenCalledTimes(1);});+ // REST's default word mode answers `ok`, `東京` or a pasted sentence; contains refuses them for their terms.+ it.each([+ ['too short', 'ok', 'A contains search needs at least one term of 3 or more characters.'],+ ['too long', 'a'.repeat(65), 'The terms of a contains search may hold at most 64 characters together.'],+ ])('searches by word when contains refuses %s terms, unless the agent asked for contains', async (_n, q, refusal) => {+ const key = { id: 'k1', allowedSessions: null, allowedChats: null };+ search.mockRejectedValueOnce(new ContainsTermsRefusedError(refusal));++ await expect(call(key, { q })).resolves.toBe(results);+ expect(search).toHaveBeenNthCalledWith(1, { q, match: 'contains' }, undefined, undefined);+ expect(search).toHaveBeenNthCalledWith(2, { q, match: 'word' }, undefined, undefined);++ search.mockClear();+ search.mockRejectedValueOnce(new ContainsTermsRefusedError(refusal));+ await expect(call(key, { q, match: 'contains' })).rejects.toBeInstanceOf(BadRequestException);+ expect(search).toHaveBeenCalledTimes(1);+ });++ it('refuses a NUL character in the query before searching, which PostgreSQL would fail on', async () => {+ const key = { id: 'k1', allowedSessions: null, allowedChats: null };+ await expect(call(key, { q: `invoice${String.fromCharCode(0)}` })).rejects.toBeInstanceOf(BadRequestException);+ expect(search).not.toHaveBeenCalled();+ });+it('bounds dateFrom and dateTo to the safe integer range, as REST does', async () => {const key = { id: 'k1', allowedSessions: null, allowedChats: null };diff --git a/src/core/agent-tools/tools/search.tools.ts b/src/core/agent-tools/tools/search.tools.tsindex b4f70c824..0f08fd1f5 100644--- a/src/core/agent-tools/tools/search.tools.ts+++ b/src/core/agent-tools/tools/search.tools.ts@@ -4,12 +4,15 @@ import { ApiKeyRole } from '../../../modules/auth/entities/api-key.entity';import { MessageDirection } from '../../../modules/message/entities/message.entity';import type { SearchService } from '../../../modules/search/search.service';import {+ SEARCH_CONTAINS_MIN_TERM_LENGTH,+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,SEARCH_CONTAINS_WINDOW,SEARCH_LIMIT_MAX,SEARCH_MATCH_MODES,SEARCH_OFFSET_MAX,SEARCH_QUERY_MAX_LENGTH,} from '../../../modules/search/search.constants';+import { ContainsTermsRefusedError } from '../../../modules/search/search-terms';import { chatIdSchema } from '../chat-id-schema';import { defineTool, type AnyToolDescriptor } from '../tool-descriptor';@@ -21,14 +24,16 @@ import { defineTool, type AnyToolDescriptor } from '../tool-descriptor';* configuration; calling it answers 501, as the REST route answers 404 then.** `containsEnabled` is SEARCH_CONTAINS_ENABLED. With it on, `match` defaults to contains, unlike REST- * (word): an agent often knows only part of a word or types it without accents, and contains finds- * everything word finds. With it off the built-in provider answers 501 to contains, so the tool- * defaults to word like REST.+ * (word): an agent often knows only part of a word or types it without accents. Contains does not run+ * every query word does (two-character words alone, as most Korean and many CJK words are, give the+ * trigram index nothing to look up, and the terms together are bounded), so a query it refuses for its+ * terms runs as a word search, as does one while its index is not ready. With it off the built-in+ * provider answers 501 to contains, so the tool defaults to word like REST.*/export function searchTools(search: SearchService | undefined, containsEnabled = false): AnyToolDescriptor[] {const defaultMatch = containsEnabled ? 'contains' : 'word';const matchHelp = containsEnabled- ? `match=contains (the default) finds any part of a word, ignoring case and accents, and ranks the ${SEARCH_CONTAINS_WINDOW} most recently stored matches (whole words first, then newest), so total is at most ${SEARCH_CONTAINS_WINDOW}: narrow with more terms, chatId or dateTo to reach older ones. match=word finds whole words only.`+ ? `match=contains (the default) finds any part of a word, ignoring case and accents, and ranks the ${SEARCH_CONTAINS_WINDOW} most recently stored matches (whole words first, then newest), so total is at most ${SEARCH_CONTAINS_WINDOW}: narrow with more terms, chatId or dateTo to reach older ones. It needs a term of ${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more characters and at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in all its terms; without match, a query outside those bounds runs as match=word, which finds whole words only.`: 'match=word (the default) finds whole words; match=contains needs SEARCH_CONTAINS_ENABLED=true on the server.';return [defineTool({@@ -75,7 +80,10 @@ export function searchTools(search: SearchService | undefined, containsEnabled =try {return await search.search({ ...input, match: input.match ?? defaultMatch }, sessionIds, chatIds);} catch (error) {- if (input.match || defaultMatch === 'word' || !(error instanceof NotImplementedException)) throw error;+ // Unless the agent asked for contains, a query contains cannot run yet (its index is not ready)+ // or cannot run at all for its terms (too short or too long) is answered as a word search.+ const fallback = error instanceof NotImplementedException || error instanceof ContainsTermsRefusedError;+ if (input.match || defaultMatch === 'word' || !fallback) throw error;return search.search({ ...input, match: 'word' }, sessionIds, chatIds);}},diff --git a/src/modules/search/dto/search-query.dto.spec.ts b/src/modules/search/dto/search-query.dto.spec.tsindex ad07fe711..52a5450e1 100644--- a/src/modules/search/dto/search-query.dto.spec.ts+++ b/src/modules/search/dto/search-query.dto.spec.ts@@ -43,6 +43,12 @@ describe('SearchQueryDto', () => {);});+ // PostgreSQL refuses U+0000 in a bound parameter in either match mode, and SQLite's FTS5 in a contains query.+ it('rejects a q holding a NUL character', () => {+ const errors = validateSync(fromQuery({ q: `invoice${String.fromCharCode(0)}` }));+ expect(errors.map(e => [e.property, Object.keys(e.constraints ?? {})])).toEqual([['q', ['notContains']]]);+ });+it('rejects limit < 1 (@Min(1))', () => {const dto = fromQuery({ q: 'hello', limit: '0' });expect(validateSync(dto).some(e => e.property === 'limit')).toBe(true);diff --git a/src/modules/search/dto/search-query.dto.ts b/src/modules/search/dto/search-query.dto.tsindex 536157e31..f1bb72da0 100644--- a/src/modules/search/dto/search-query.dto.ts+++ b/src/modules/search/dto/search-query.dto.ts@@ -4,8 +4,10 @@ import { Type } from 'class-transformer';import { MessageDirection } from '../../message/entities/message.entity';import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';import { ToStrictNumber, ToStrictString } from '../../../common/utils/strict-boolean';+import { NoNulCharacter } from '../../../common/validation/no-nul-character';import {SEARCH_CONTAINS_MIN_TERM_LENGTH,+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,SEARCH_CONTAINS_WINDOW,SEARCH_MATCH_MODES,SEARCH_OFFSET_MAX,@@ -28,13 +30,15 @@ import type { SearchMatch } from '../search.types';*/export class SearchQueryDto {@ApiProperty({- description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} words)`,+ description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} words, no NUL character)`,maxLength: SEARCH_QUERY_MAX_LENGTH,})@ToStrictString()@IsString()@IsNotEmpty()@MaxLength(SEARCH_QUERY_MAX_LENGTH)+ // PostgreSQL refuses U+0000 in a bound parameter, and SQLite's FTS5 in a contains query: both a 500.+ @NoNulCharacter()q!: string;@ApiPropertyOptional({@@ -44,7 +48,9 @@ export class SearchQueryDto {'`word` (default) matches whole words. `contains` matches any part of a word, ignoring case and ' +`accents (\`voic\` finds "Invoice", \`cafe\` finds "Café"); it is opt-in on the built-in provider ` +'(SEARCH_CONTAINS_ENABLED=true, else 501) and needs at least one term of ' +- `${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more characters (on PostgreSQL, ${SEARCH_CONTAINS_MIN_TERM_LENGTH} letters or digits in a row). ` ++ `${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more characters (on PostgreSQL, ${SEARCH_CONTAINS_MIN_TERM_LENGTH} letters or digits in a row) ` ++ `and at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in all its terms, spaces not counted (400 otherwise), so a ` ++ 'query of two-character words alone, as most Korean and many Chinese and Japanese words are, needs `word`. ' +`It ranks the ${SEARCH_CONTAINS_WINDOW} most recently stored matches by match quality, so \`total\` is at most ` +`${SEARCH_CONTAINS_WINDOW}.`,})diff --git a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts b/src/modules/search/providers/builtin-fts.provider.contains.spec.tsindex 742e04336..bc75ca6a9 100644--- a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts+++ b/src/modules/search/providers/builtin-fts.provider.contains.spec.ts@@ -5,7 +5,8 @@ import { DataSource } from 'typeorm';import { Message, MessageDirection } from '../../../modules/message/entities/message.entity';import { Session } from '../../../modules/session/entities/session.entity';import { AddMessagesFts1782400000000 } from '../../../database/migrations/1782400000000-AddMessagesFts';-import { SEARCH_CONTAINS_WINDOW } from '../search.constants';+import { SEARCH_CONTAINS_TERMS_MAX_LENGTH, SEARCH_CONTAINS_WINDOW } from '../search.constants';+import { ContainsTermsRefusedError } from '../search-terms';import {BuiltInFtsProvider,postgresUnaccentFunctionDdl,@@ -97,6 +98,41 @@ describe('BuiltInFtsProvider match=contains (sqlite)', () => {await expect(provider.search({ q: 'id to', match: 'contains' })).rejects.toBeInstanceOf(BadRequestException);});+ it('counts a Hangul syllable as one character, so a two-syllable word only narrows', async () => {+ await ds+ .getRepository(Message)+ .insert([+ row('s1', 'c5', '서울역에서 만나요', MessageDirection.INCOMING, 50),+ row('s1', 'c6', 'invoice 한국 지사로 보냄', MessageDirection.INCOMING, 60),+ ]);++ await expect(provider.search({ q: '서울', match: 'contains' })).rejects.toBeInstanceOf(ContainsTermsRefusedError);+ const station = await provider.search({ q: '서울역', match: 'contains' });+ expect(station.hits.map(h => [h.chatId, h.snippet])).toEqual([['c5', '<mark>서울역</mark>에서 만나요']]);+ expect((await provider.search({ q: 'invoice 한국', match: 'contains' })).hits.map(h => h.chatId)).toEqual(['c6']);+ });++ it('marks a Greek word whose final sigma the body writes in capitals', async () => {+ await ds.getRepository(Message).insert(row('s1', 'c5', 'ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', MessageDirection.INCOMING, 50));+ const query = jest.spyOn(ds, 'query');++ const res = await provider.search({ q: 'δρομος', match: 'contains' });++ expect(res.hits.map(h => [h.snippet, h.score])).toEqual([['ΚΑΛΟΣ <mark>ΔΡΟΜΟΣ</mark> σήμερα', 3]]);+ // Only the application reads final sigma as sigma: the database is sent the term as written.+ expect(query.mock.calls.at(-1)?.[1]).toContain('"δρομος"');+ });++ it('bounds the characters of the terms together, spaces not counted', async () => {+ const max = 'a'.repeat(SEARCH_CONTAINS_TERMS_MAX_LENGTH);+ expect((await provider.search({ q: max, match: 'contains' })).total).toBe(0);+ for (const q of [`${max}b`, `${max.slice(0, 40)} ${max.slice(0, 25)}x`]) {+ await expect(provider.search({ q, match: 'contains' })).rejects.toBeInstanceOf(ContainsTermsRefusedError);+ }+ // A repeated term counts once, as it runs once.+ expect((await provider.search({ q: `${max} ${max.toUpperCase()}`, match: 'contains' })).total).toBe(0);+ });+it('matches a LIKE wildcard in a term literally', async () => {expect((await provider.search({ q: '10%', match: 'contains' })).total).toBe(1);expect((await provider.search({ q: 'invoice _', match: 'contains' })).total).toBe(0);diff --git a/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts b/src/modules/search/providers/builtin-fts.provider.postgres.spec.tsindex 2e8bd6a51..1253f3379 100644--- a/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts+++ b/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts@@ -1,7 +1,9 @@import 'reflect-metadata';import { NotImplementedException } from '@nestjs/common';+import { ConfigService } from '@nestjs/config';import { LoggerService } from '../../../common/services/logger.service';import { DataSource } from 'typeorm';+import { ContainsTermsRefusedError } from '../search-terms';import { BuiltInFtsProvider } from './builtin-fts.provider';/**@@ -186,3 +188,32 @@ describe('BuiltInFtsProvider (postgres probe)', () => {await expect(provider.search({ q: 'hello' })).resolves.toMatchObject({ provider: 'builtin-fts' });});});++describe('BuiltInFtsProvider match=contains terms (postgres)', () => {+ /** A provider whose catalog read finds the trigram index ready, over a database that matches nothing. */+ const containsReady = () => {+ const query = jest.fn().mockResolvedValue([]);+ const manager = {+ query: jest.fn().mockResolvedValue([{ trgm: 'public', unaccent: 'public', fn: true, idx: true }]),+ };+ const ds = { options: { type: 'postgres' }, query, manager } as unknown as DataSource;+ return { provider: new BuiltInFtsProvider(ds, new ConfigService({ search: { containsEnabled: true } })), query };+ };++ // As show_trgm(openwa_unaccent(term)) reads them on PostgreSQL 16, with glibc and with musl: a Hangul+ // word of two syllables, a modifier letter unaccent turns into an apostrophe and superscript digits+ // give pg_trgm no trigram to look up, so the search would read every row.+ it.each(['안녕', '\u02bc'.repeat(3), '¹²³'])('refuses %j alone, which pg_trgm cannot look up', async q => {+ const { provider, query } = containsReady();+ await expect(provider.search({ q, match: 'contains' })).rejects.toBeInstanceOf(ContainsTermsRefusedError);+ expect(query).not.toHaveBeenCalled();+ });++ // Hangul syllables, a katakana word with its prolonged sound mark, and letter numbers (which unaccent+ // spells in Latin) give it trigrams.+ it.each(['서울역', 'ラーメン', 'ⅫⅫⅫ'])('looks %j up', async q => {+ const { provider, query } = containsReady();+ await expect(provider.search({ q, match: 'contains' })).resolves.toMatchObject({ total: 0 });+ expect(query).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/modules/search/providers/builtin-fts.provider.ts b/src/modules/search/providers/builtin-fts.provider.tsindex 07344f79e..c2ee46177 100644--- a/src/modules/search/providers/builtin-fts.provider.ts+++ b/src/modules/search/providers/builtin-fts.provider.ts@@ -16,8 +16,13 @@ import { DataSource, QueryRunner } from 'typeorm';import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';import { MessageDirection } from '../../message/entities/message.entity';import type { SearchProvider, SearchQuery, SearchResults, SearchHit } from '../search.types';-import { SEARCH_CONTAINS_MIN_TERM_LENGTH, SEARCH_CONTAINS_WINDOW, SEARCH_LIMIT_MAX } from '../search.constants';-import { containsTerms, foldSearchText } from '../search-terms';+import {+ SEARCH_CONTAINS_MIN_TERM_LENGTH,+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,+ SEARCH_CONTAINS_WINDOW,+ SEARCH_LIMIT_MAX,+} from '../search.constants';+import { ContainsTermsRefusedError, containsTerms, foldSearchText } from '../search-terms';import { containsSnippet } from '../search-snippet';import { containsScore } from '../search-score';@@ -70,8 +75,16 @@ const CONTAINS_UNAVAILABLE = {'Contains search is unavailable: this SQLite build has no FTS5 trigram tokenizer (SQLite 3.45+ is required).',} as const;-/** Three letters or digits in a row: pg_trgm takes trigrams from nothing else. */-const POSTGRES_TRIGRAM_SOURCE = new RegExp(`[\\p{L}\\p{N}]{${SEARCH_CONTAINS_MIN_TERM_LENGTH}}`, 'u');+/**+ * Three letters or digits in a row, the only characters pg_trgm takes trigrams from: letters of every+ * kind, decimal digits and letter numbers (`Ⅻ`), never other numbers (`²`, `①`). The nine modifier+ * letters left out are the ones openwa_unaccent turns into punctuation before pg_trgm reads them (`ʼ`+ * into an apostrophe, `ˆ` into `^`).+ */+const POSTGRES_TRIGRAM_SOURCE = new RegExp(+ `(?:(?![\\u02b9-\\u02bd\\u02c6\\u02c8\\u02cb\\u02d0])[\\p{L}\\p{Nl}\\p{Nd}]){${SEARCH_CONTAINS_MIN_TERM_LENGTH}}`,+ 'u',+);/*** The contains candidate SELECT list. It carries no snippet or score column: searchContains computes@@ -624,8 +637,10 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli** 1. The database returns at most SEARCH_CONTAINS_WINDOW candidates: the most recently stored* messages that contain every term, found through the trigram index, newest first. Both- * dialects answer that order from an index and stop at the window, so the work is bounded- * however many messages match.+ * dialects answer that order from an index and stop once the window is full, so a common match+ * costs no more than a rare one. A message the index yields that a narrowing term then rules out+ * does not fill it, so the messages holding the looked-up terms bound the work, multiplied by+ * the length of the terms (SEARCH_CONTAINS_TERMS_MAX_LENGTH).* 2. The application ranks the candidates with containsScore (whole word, then word start, then* inside a word), newest first within a score and `id` last for a total order, and pages them in* memory.@@ -730,17 +745,24 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli* it takes no trigram from (FTS5 returns nothing for it, pg_trgm scans every row), so such terms only* narrow the rows another term already found, and a query made of them alone is a 400 rather than a* full-table scan. SQLite takes trigrams from any three characters, pg_trgm only from three letters or- * digits in a row (see postgresCanIndex).+ * digits in a row (see postgresCanIndex). The terms together are bounded by+ * SEARCH_CONTAINS_TERMS_MAX_LENGTH, since each of their characters costs a check on every candidate.*/private static containsTermsOf(q: string, isPostgres: boolean): string[] {const terms = containsTerms(q);const canIndex = isPostgres ? postgresCanIndex : (term: string) => !isShortTerm(term);if (!terms.some(canIndex)) {const unit = isPostgres ? 'letters or digits in a row' : 'characters';- throw new BadRequestException(+ throw new ContainsTermsRefusedError(`A contains search needs at least one term of ${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more ${unit}.`,);}+ const length = terms.reduce((sum, term) => sum + [...foldSearchText(term)].length, 0);+ if (length > SEARCH_CONTAINS_TERMS_MAX_LENGTH) {+ throw new ContainsTermsRefusedError(+ `The terms of a contains search may hold at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters together, spaces not counted.`,+ );+ }return terms;}diff --git a/src/modules/search/search-score.spec.ts b/src/modules/search/search-score.spec.tsindex e813a588e..7122fb258 100644--- a/src/modules/search/search-score.spec.ts+++ b/src/modules/search/search-score.spec.ts@@ -21,6 +21,11 @@ describe('containsScore', () => {expect(containsScore('Thanks, I will pay the invoice today', ['invoice', 'to'])).toBe(5);});+ it('reads final sigma as sigma, whichever side writes it', () => {+ expect(containsScore('ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', ['δρομος'])).toBe(3);+ expect(containsScore('ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', ['δρομοσ'])).toBe(3);+ });+it('scores 0 for a term the body does not contain', () => {expect(containsScore('Order shipped', ['invoice'])).toBe(0);});diff --git a/src/modules/search/search-score.ts b/src/modules/search/search-score.tsindex 32b4793ae..9ddde0fcd 100644--- a/src/modules/search/search-score.ts+++ b/src/modules/search/search-score.ts@@ -1,4 +1,4 @@-import { foldSearchText } from './search-terms';+import { foldForMatching } from './search-terms';/*** Relevance of a match=contains candidate, computed in the application so both dialects rank alike@@ -14,9 +14,9 @@ import { foldSearchText } from './search-terms';* caller's scope.*/export function containsScore(body: string, terms: string[]): number {- const folded = foldSearchText(body);+ const folded = foldForMatching(body);let score = 0;- for (const term of terms.map(foldSearchText)) {+ for (const term of terms.map(foldForMatching)) {let best = 0;for (let at = folded.indexOf(term); at !== -1 && best < 3; at = folded.indexOf(term, at + 1)) {const startsWord = !isWordChar(folded[at - 1]);diff --git a/src/modules/search/search-snippet.spec.ts b/src/modules/search/search-snippet.spec.tsindex 61abf910d..fb377de3f 100644--- a/src/modules/search/search-snippet.spec.ts+++ b/src/modules/search/search-snippet.spec.ts@@ -6,6 +6,11 @@ describe('foldSearchText', () => {expect(foldSearchText('Meeting at the CAFÉ')).toBe('meeting at the cafe');expect(foldSearchText('Ação')).toBe('acao');});++ it('keeps a Hangul syllable one character, however the text spells it', () => {+ expect([...foldSearchText('서울')]).toEqual(['서', '울']);+ expect(foldSearchText('서울'.normalize('NFD'))).toBe('서울');+ });});describe('containsTerms', () => {@@ -35,6 +40,16 @@ describe('containsSnippet', () => {expect(containsSnippet(`meet at the ${decomposed}`, ['cafe'], 24)).toBe(`meet at the <mark>${decomposed}</mark>`);});+ it('marks a Hangul word in a body that spells its syllables as jamo', () => {+ const [word, rest] = ['서울역'.normalize('NFD'), '에서 만나요'.normalize('NFD')];+ expect(containsSnippet(`${word}${rest}`, ['서울역'], 24)).toBe(`<mark>${word}</mark>${rest}`);+ });++ it('marks a word ending in final sigma in a body written in capitals, and the other way round', () => {+ expect(containsSnippet('ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', ['δρομος'], 24)).toBe('ΚΑΛΟΣ <mark>ΔΡΟΜΟΣ</mark> σήμερα');+ expect(containsSnippet('ο δρομος', ['ΔΡΟΜΟΣ'], 24)).toBe('ο <mark>δρομος</mark>');+ });+it('opens the window a few words before a late match and marks where it cut', () => {const words = Array.from({ length: 40 }, (_, i) => `w${i}`);words[30] = 'invoice';diff --git a/src/modules/search/search-snippet.ts b/src/modules/search/search-snippet.tsindex 883f5064a..17cd5d94f 100644--- a/src/modules/search/search-snippet.ts+++ b/src/modules/search/search-snippet.ts@@ -1,4 +1,4 @@-import { foldSearchText } from './search-terms';+import { foldForMatching } from './search-terms';/*** Snippets for match=contains hits, built in the application rather than the database so both@@ -20,27 +20,31 @@ interface FoldedText {ends: number[];}+/**+ * What folds as one unit: a character with the combining marks after it, or a Hangul syllable spelled+ * as jamo (leading consonants, vowels, trailing consonants), which NFC composes into one syllable.+ */+const FOLD_UNIT = /[\u1100-\u115f]+[\u1160-\u11a7]*[\u11a8-\u11ff]*\p{M}*|[\s\S]\p{M}*/gu;+/*** Folding can change a text's length: a body that spells `é` as `e` followed by a combining accent* loses that accent unit. Matches are therefore found in the folded copy and mapped back through- * per-unit offsets, rather than assuming positions line up.+ * per-unit offsets, rather than assuming positions line up. The text folds one FOLD_UNIT at a time, as+ * the whole body folds for the score: a letter's accents fold away with it, so a match ending on it+ * covers them instead of leaving a dangling accent outside </mark>, and jamo recompose into the+ * syllable the term folds to.*/function foldWithOffsets(text: string): FoldedText {let folded = '';const starts: number[] = [];const ends: number[] = [];- let offset = 0;- for (const char of text) {- const mapped = foldSearchText(char);- // A combining mark folds to nothing; it belongs to the letter before it, so a match ending on that- // letter also covers the mark instead of leaving a dangling accent outside </mark>.- if (!mapped && ends.length) ends[ends.length - 1] = offset + char.length;+ for (const { 0: unit, index } of text.matchAll(FOLD_UNIT)) {+ const mapped = foldForMatching(unit);for (let i = 0; i < mapped.length; i++) {- starts.push(offset);- ends.push(offset + char.length);+ starts.push(index);+ ends.push(index + unit.length);}folded += mapped;- offset += char.length;}return { folded, starts, ends };}@@ -49,7 +53,7 @@ function foldWithOffsets(text: string): FoldedText {function matchRanges(body: string, terms: string[]): Array<[number, number]> {const { folded, starts, ends } = foldWithOffsets(body);const ranges: Array<[number, number]> = [];- for (const term of terms.map(foldSearchText).filter(Boolean)) {+ for (const term of terms.map(foldForMatching).filter(Boolean)) {for (let at = folded.indexOf(term); at !== -1; at = folded.indexOf(term, at + term.length)) {ranges.push([starts[at], ends[at + term.length - 1]]);}diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.tsindex 2eed09cdf..8a375df09 100644--- a/src/modules/search/search-terms.ts+++ b/src/modules/search/search-terms.ts@@ -1,3 +1,5 @@+import { BadRequestException } from '@nestjs/common';+/*** What the built-in index (SQLite FTS5, unicode61) makes of a query, as an upper bound on its words. A* run of letters or digits is one word. ASCII punctuation separates. Any other character outside ASCII@@ -52,13 +54,31 @@ function wordsOf(word: string): number {/*** Folds text the way the match=contains indexes compare it: accents dropped (NFD, then every combining- * mark removed, as remove_diacritics and unaccent do for Latin text) and case lowered. Used to compare- * terms and to place highlights, never to build SQL, which leaves folding to the database.+ * mark removed, as remove_diacritics and unaccent do for Latin text), recomposed (NFC: NFD splits a+ * Hangul syllable into conjoining jamo, which are letters, so `서울` would count five characters where+ * the database sees two) and case lowered. Used to compare terms and to place highlights, never to+ * build SQL, which leaves folding to the database.*/export function foldSearchText(text: string): string {- return text.normalize('NFD').replace(/\p{M}/gu, '').toLowerCase();+ return text.normalize('NFD').replace(/\p{M}/gu, '').normalize('NFC').toLowerCase();+}++/**+ * foldSearchText for the application's own comparisons, the score and the highlight, with final sigma+ * read as sigma: lowercasing a whole body turns a word-final `Σ` into `ς`, and one character alone into+ * `σ`, so a match the score counted got no highlight. The terms sent to the database are not affected.+ */+export function foldForMatching(text: string): string {+ return foldSearchText(text).replace(/ς/g, 'σ');}+/**+ * A match=contains query refused for its terms (none long enough for a trigram index, or more characters+ * than SEARCH_CONTAINS_TERMS_MAX_LENGTH), which a word search can still answer: the MessageSearch tool+ * runs one instead when the agent did not ask for contains.+ */+export class ContainsTermsRefusedError extends BadRequestException {}+/*** The whitespace-separated terms of a match=contains query, each kept as written, with a term that* folds to one already seen dropped (`Café cafe` is one term): it would add nothing to an AND butdiff --git a/src/modules/search/search.constants.ts b/src/modules/search/search.constants.tsindex 4d293527d..b864e7d4c 100644--- a/src/modules/search/search.constants.ts+++ b/src/modules/search/search.constants.ts@@ -42,10 +42,19 @@ export const SEARCH_MATCH_MODES = ['word', 'contains'] as const satisfies readon*/export const SEARCH_CONTAINS_MIN_TERM_LENGTH = 3;+/**+ * Most characters the terms of a match=contains query may hold together, counted after folding and+ * without the spaces between them. The trigram index checks every trigram of a term on every message+ * that holds all of them, and on SQLite synchronously: a query of `the` repeated, on 200k messages of+ * which 40% hold `the`, `het` and `eth`, took about 100 ms at 64 characters whether written as one term+ * or as several, and 2.8 s at 1000. A longer query answers 400; match=word runs it.+ */+export const SEARCH_CONTAINS_TERMS_MAX_LENGTH = 64;+/*** How many candidates a match=contains search ranks: the most recently stored matches, fetched from- * the index in recency order so the database stops at this many however many messages match, then- * ranked in the application. It bounds the work of one search (ranking every match in SQL took about+ * the index in recency order so the database stops once it holds this many, then ranked in the+ * application. It bounds the ranking work of one search (ranking every match in SQL took about* 100 ms on SQLite and 1.6 s on Postgres for 200k matches) and is therefore also the most `total` can* report in this mode. Fixed rather than env-driven, like SEARCH_OFFSET_MAX.*/diff --git a/src/modules/search/search.controller.ts b/src/modules/search/search.controller.tsindex 3aaac81d2..1b1e8dc5f 100644--- a/src/modules/search/search.controller.ts+++ b/src/modules/search/search.controller.ts@@ -6,7 +6,12 @@ import { ChatScopeService } from '../auth/chat-scope.service';import { ApiKey, ApiKeyRole } from '../auth/entities/api-key.entity';import { SearchService } from './search.service';import { SearchQueryDto } from './dto/search-query.dto';-import { SEARCH_CONTAINS_WINDOW, SEARCH_MATCH_MODES, SEARCH_OFFSET_MAX } from './search.constants';+import {+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,+ SEARCH_CONTAINS_WINDOW,+ SEARCH_MATCH_MODES,+ SEARCH_OFFSET_MAX,+} from './search.constants';import type { SearchResults } from './search.types';@ApiTags('search')@@ -25,8 +30,9 @@ export class SearchController {@ApiResponse({status: 400,description:- 'Empty or whitespace-only "q", or one over 1000 characters or 32 words; an unknown "match"; or a ' +- '"match=contains" query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row)',+ 'Empty or whitespace-only "q", or one over 1000 characters or 32 words or holding a NUL character; an ' ++ 'unknown "match"; or a "match=contains" query with no term of 3 or more characters (on PostgreSQL, 3 ' ++ `letters or digits in a row) or with more than ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in its terms`,})@ApiResponse({ status: 403, description: 'Chat scope denied or chat-restricted search uses a plugin provider' })@ApiResponse({@@ -51,7 +57,7 @@ export class SearchController {@ApiQuery({name: 'q',required: true,- description: 'Search term (required, non-empty, at most 1000 characters and 32 words)',+ description: 'Search term (required, non-empty, at most 1000 characters and 32 words, no NUL character)',})@ApiQuery({name: 'match',@@ -59,7 +65,8 @@ export class SearchController {enum: SEARCH_MATCH_MODES,description:'word (default): whole words. contains: any part of a word, ignoring case and accents; ranks the ' +- `${SEARCH_CONTAINS_WINDOW} most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true`,+ `${SEARCH_CONTAINS_WINDOW} most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true, ` ++ `a term of 3 or more characters and at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in all its terms`,})@ApiQuery({ name: 'sessionId', required: false, description: 'Restrict to a single session' })@ApiQuery({