Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(search): bound contains queries and fold text as the index does

Yudhi Armyndharis431e19a6580b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b340752ac..2b9639d43 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -30,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Index a Baileys clear, delete, archive or unread by the same chat jid as its message key, so the phone applies it (#1891). Thanks @rjsebening.
- Key Baileys chat writes and read receipts to the gateway's newest stored message when the session holds none in memory (#1891). Thanks @rjsebening.
- Group dashboard phone numbers in threes and fours, so `+49 157 333 4 4455` now reads `+49 157 3334 4455`; 10-digit numbers keep their shape (#1940).
+- Answer `400` to a NUL character in `q` on `GET /api/search`, which failed with `500` on PostgreSQL.
## [0.26.0] - 2026-10-28
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 42756a3a6..d2a20dfae 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -6603,19 +6603,19 @@ Search messages across sessions (active search provider).
**Query parameters**
-| Name | Type | Required | Default | Description |
-| ----------- | ------------------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words, a symbol or emoji that stands apart counts as a word, a whitespace-separated word said again counts once). Passed to the active provider's native full-text matcher. |
-| `match` | enum (`word` \| `contains`) | No | `word` | `word` matches whole words. `contains` matches any part of a word, ignoring case and accents (`voic` finds "Invoice", `cafe` finds "Café"); every term must appear, at least one term needs 3+ characters, on PostgreSQL 3+ letters or digits in a row (`400` otherwise), and `%`/`_` match literally. It ranks the 1,000 most recently stored matches by match quality (whole word, then word start, then inside a word; newest first within a score), so `total` is at most 1,000. Opt-in on the built-in provider: `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true` and has finished building the index (on PostgreSQL also the `pg_trgm` and `unaccent` extensions). PostgreSQL also folds letters such as `Ł`, `ß` and `Ø` that SQLite keeps. See [26 - Global Search](./26-global-search.md#match-modes). |
-| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |
-| `chatId` | string | No | — | Restrict to a single chat id. With the built-in provider, a phone also matches the chat's other stored forms (the other user dialect, any lid that resolves to it). For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are also narrowed to the allowlist. |
-| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |
-| `type` | string | No | — | Filter by stored message `type` (e.g. `text`, `image`, `video`). Compared against `messages.type`; not an enum validation, any string is accepted and unmatched values simply return no hits. |
-| `from` | string | No | — | Filter by sender; with the built-in provider, matches `from` or a group message's `author`, and a phone also matches any lid that resolves to it. |
-| `dateFrom` | integer (epoch ms) | No | — | Inclusive lower bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |
-| `dateTo` | integer (epoch ms) | No | — | Inclusive upper bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |
-| `limit` | integer (≥ 1) | No | `50` | Max hits to return. Clamped to `SEARCH_LIMIT_MAX` (default `100`). A non-integer value is rejected with `400`. |
-| `offset` | integer (≥ 0) | No | `0` | Pagination offset, at most `100000`. A non-integer or larger value is rejected with `400`. |
+| Name | Type | Required | Default | Description |
+| ----------- | ------------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, over 32 words (punctuation separates words, a symbol or emoji that stands apart counts as a word, a whitespace-separated word said again counts once), or holding a NUL character. Passed to the active provider's native full-text matcher. |
+| `match` | enum (`word` \| `contains`) | No | `word` | `word` matches whole words. `contains` matches any part of a word, ignoring case and accents (`voic` finds "Invoice", `cafe` finds "Café"); every term must appear, at least one term needs 3+ characters, on PostgreSQL 3+ letters or digits in a row, the terms together may hold at most 64 characters, spaces not counted (`400` otherwise, so a query of two-character words alone, as most Korean and many Chinese and Japanese words are, needs `word`), and `%`/`_` match literally. It ranks the 1,000 most recently stored matches by match quality (whole word, then word start, then inside a word; newest first within a score), so `total` is at most 1,000. Opt-in on the built-in provider: `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true` and has finished building the index (on PostgreSQL also the `pg_trgm` and `unaccent` extensions). PostgreSQL also folds letters such as `Ł`, `ß` and `Ø` that SQLite keeps. See [26 - Global Search](./26-global-search.md#match-modes). |
+| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |
+| `chatId` | string | No | — | Restrict to a single chat id. With the built-in provider, a phone also matches the chat's other stored forms (the other user dialect, any lid that resolves to it). For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are also narrowed to the allowlist. |
+| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |
+| `type` | string | No | — | Filter by stored message `type` (e.g. `text`, `image`, `video`). Compared against `messages.type`; not an enum validation, any string is accepted and unmatched values simply return no hits. |
+| `from` | string | No | — | Filter by sender; with the built-in provider, matches `from` or a group message's `author`, and a phone also matches any lid that resolves to it. |
+| `dateFrom` | integer (epoch ms) | No | — | Inclusive lower bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |
+| `dateTo` | integer (epoch ms) | No | — | Inclusive upper bound on `timestamp`. Must be between -9007199254740991 and 9007199254740991; a non-numeric or out-of-range value is rejected with `400`. |
+| `limit` | integer (≥ 1) | No | `50` | Max hits to return. Clamped to `SEARCH_LIMIT_MAX` (default `100`). A non-integer value is rejected with `400`. |
+| `offset` | integer (≥ 0) | No | `0` | Pagination offset, at most `100000`. A non-integer or larger value is rejected with `400`. |
**Response** `200` — `SearchResults`
@@ -6659,8 +6659,8 @@ Search messages across sessions (active search provider).
the match quality on both dialects (3 per term found as a whole word, 2 at a word start, 1 inside a
word) and every key gets it, since it reads only its own message.
-**Errors:** `400` empty/whitespace `q`, a non-numeric `dateFrom`/`dateTo`/`limit`/`offset`, a `dateFrom`/`dateTo` beyond 9007199254740991 in either direction, an `offset` above `100000`, an unknown `match`, a
-`match=contains` query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row), or a malformed
+**Errors:** `400` empty/whitespace `q` or one holding a NUL character, a non-numeric `dateFrom`/`dateTo`/`limit`/`offset`, a `dateFrom`/`dateTo` beyond 9007199254740991 in either direction, an `offset` above `100000`, an unknown `match`, a
+`match=contains` query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row) or with more than 64 characters in its terms, or a malformed
SQLite FTS5 query (unbalanced quote/paren, bare operator) — Postgres's `websearch_to_tsquery` is
tolerant and has no equivalent · `401` missing/invalid `X-API-Key` · `403` key role below `OPERATOR`, or a chat-restricted key
that names a `chatId` outside its allowlist or uses a plugin provider ·
diff --git a/docs/07-api-collection.md b/docs/07-api-collection.md
index 954255c57..3ce4a5f83 100644
--- a/docs/07-api-collection.md
+++ b/docs/07-api-collection.md
@@ -1752,7 +1752,7 @@ Search messages across sessions. `q` is required (non-empty after trim); optiona
`sessionId`, `chatId`, `direction` (`incoming` | `outgoing`), `type`, `from`, `dateFrom`/`dateTo`
(epoch ms), plus `limit` (default 50, clamped to `SEARCH_LIMIT_MAX` = 100) and `offset` (at most
100000). `match` is `word` (the default, whole words) or `contains` (any part of a word, ignoring case
-and accents), which answers `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true`. `q` is at most 1000 characters and 32 words, and a longer one answers `400`. A key restricted to
+and accents), which answers `501` unless the server sets `SEARCH_CONTAINS_ENABLED=true` and `400` without a term of 3 or more characters or with more than 64 characters in its terms. `q` is at most 1000 characters and 32 words with no NUL character, and any other answers `400`. A key restricted to
selected chats gets `403` for a `chatId` outside its list and for a plugin search provider. Hit
`snippet` fields carry `<mark>` highlight markers — render them as text, never as HTML.
diff --git a/docs/26-global-search.md b/docs/26-global-search.md
index e23096e8f..8e8454cf7 100644
--- a/docs/26-global-search.md
+++ b/docs/26-global-search.md
@@ -68,7 +68,8 @@ never assign the raw snippet to `innerHTML`.
### Match modes
The `match` query parameter picks how `q` matches a message body. Each mode has its own DB-maintained
-index, so both are index lookups, never table scans. The `contains` index is opt-in
+index, and every search looks its terms up there; `contains` refuses a query its index cannot look up
+rather than read the whole table (how much of the index it reads is below). The `contains` index is opt-in
(`SEARCH_CONTAINS_ENABLED=true`, see [26.9](#269-enabling-matchcontains)); while it is off or still
building, `match=contains` answers `501`:
@@ -77,26 +78,35 @@ building, `match=contains` answers `501`:
| `word` (default) | whole words, as described above | `body_ts` (Postgres) / `messages_fts` (SQLite) | `ts_rank` / FTS5 `rank`, over every match |
| `contains` | any part of a word of 3+ characters, ignoring case and accents (`voic` finds "Invoice", `cafe` finds "Café") | a `pg_trgm` GIN index on `openwa_unaccent(body)` (Postgres) / an FTS5 `trigram` table with `remove_diacritics`, `messages_trgm` (SQLite) | match quality, over the 1,000 most recently stored matches |
-`contains` finds everything `word` finds, plus partial words. Every whitespace-separated term must
-appear, as with `word`; a repeated term (also one that differs only in case or accents) runs once. A
-trigram index cannot look up a term it takes no trigram from, so such a term only narrows what another
-term found, and a query made of such terms alone, two-character CJK words such as `東京` included,
-answers `400`. What counts differs by dialect: SQLite
-takes trigrams from any three characters, so a term needs three (`id` and `pm` only narrow, `:-)`
-searches); `pg_trgm` takes them from letters and digits only, so on Postgres a term needs three of them
-in a row (`:-)`, `...`, `😂😂😂` and `a.b` only narrow there, and alone answer `400`). The query grammar of
-`word` mode (quoted phrases, `OR`, `-` on Postgres) does not apply: every term, `%` and `_` included,
-matches literally.
+`contains` finds parts of words as well as whole ones, but it does not run every query `word` runs.
+Every whitespace-separated term must appear, as with `word`; a repeated term (also one that differs
+only in case or accents) runs once. A trigram index cannot look up a term it takes no trigram from, so
+such a term only narrows what another term found, and a query made of such terms alone answers `400`.
+That includes every query of two-character words, as most Korean words (`서울`) and many Chinese and
+Japanese ones (`東京`) are: search those with `word`. What counts differs by dialect: SQLite takes
+trigrams from any three characters, so a term needs three (`id` and `pm` only narrow, `:-)` searches,
+and a Hangul syllable is one character however it is encoded); `pg_trgm` takes them from letters and
+digits only, so on Postgres a term needs three of them in a row (`:-)`, `...`, `😂😂😂`, `a.b` and
+superscript digits such as `¹²³` only narrow there, and alone answer `400`). The terms may also hold
+at most **64 characters** together, spaces not counted, or the query answers `400`: every character of
+a term is checked on every message holding the term's trigrams, on SQLite on the event loop every
+session shares, and on 200k messages a query of 1,000 characters took 2.8 s where one of 64 took about
+100 ms. The query grammar of `word` mode (quoted phrases, `OR`, `-` on Postgres) does not apply: every
+term, `%` and `_` included, matches literally.
**Ranking runs in two phases.** Ranking every match in SQL costs time in proportion to the number of
matches: measured on 1M messages with a term found in 200k of them, about 100 ms on SQLite (where that
time blocks the event loop every session shares) and 1.6 s on Postgres. So `contains` bounds the work:
1. The database returns at most **1,000 candidates**: the most recently stored messages that contain
- every term, newest first. Both dialects answer that order from an index and stop at 1,000, however
- many messages match. SQLite reads its trigram index in insertion order; Postgres walks the existing
+ every term, newest first. Both dialects answer that order from an index and stop once they hold
+ 1,000 matches. SQLite reads its trigram index in insertion order; Postgres walks the existing
`IDX_messages_createdAt` b-tree backward for a common term, or reads the few matches of a rare one
- from the GIN index.
+ from the GIN index. The window bounds the matches, not the messages read to find them: a message
+ the index yields that another term then rules out (a short term, or one `pg_trgm` cannot look up)
+ costs a check and counts for nothing. So a query whose looked-up term is in most messages while a
+ narrowing term is in none reads every message holding the looked-up term, on Postgres each
+ rechecked through `openwa_unaccent` (which wraps the STABLE `unaccent` and so cannot be inlined).
2. The application ranks the candidates and pages them: each term scores **3** as a whole word, **2**
at the start of a word and **1** inside a word (compared without case or accents), the hit's `score`
is the sum over the terms, and ties go to the newest message, then to `messageId`. The same code
@@ -202,7 +212,7 @@ GET /api/search?q=<term>&match=<word|contains>&sessionId=<id>&chatId=<id>&direct
is a `400`.
- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one
- longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and a symbol or emoji that stands apart counting as one; a whitespace-separated word said again counts once). Numeric params
+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and a symbol or emoji that stands apart counting as one; a whitespace-separated word said again counts once; and one holding a NUL character, which PostgreSQL refuses as a parameter). Numeric params
are coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,
never as a `NaN` SQL parameter, and so does a `dateFrom`/`dateTo` beyond 9007199254740991 in either direction
(no timestamp reaches it, and the cap keeps the value exactly representable, on both databases).
@@ -223,9 +233,10 @@ GET /api/search?q=<term>&match=<word|contains>&sessionId=<id>&chatId=<id>&direct
`direction`, `from`, and optional `score`. `total` is an exact count (bounded; computed lazily only
when the page could be full). `tookMs` is the provider-side query time. `provider` names which
backend answered (e.g. `builtin-fts`).
-- **Errors:** `400` empty/whitespace `q`, a non-numeric numeric param, an unknown `match`, or a
- `match=contains` query with no term the trigram index can look up (three characters; on Postgres,
- three letters or digits in a row) (an FTS5 query-grammar error is also mapped to `400`, but since the
+- **Errors:** `400` empty/whitespace `q` or one holding a NUL character, a non-numeric numeric param,
+ an unknown `match`, or a `match=contains` query with no term the trigram index can look up (three
+ characters; on Postgres, three letters or digits in a row) or with more than 64 characters in its
+ terms (an FTS5 query-grammar error is also mapped to `400`, but since the
SQLite path quotes every token ordinary input no longer reaches that fallback) · `401`/`403` auth ·
`501` no provider configured / FTS schema absent (e.g. a non-FTS5 SQLite build) / for
`match=contains`, `SEARCH_CONTAINS_ENABLED` off, the index still building, or no way to build it (e.g.
diff --git a/openapi.json b/openapi.json
index 920daee31..f31bc5b58 100644
--- a/openapi.json
+++ b/openapi.json
@@ -18037,7 +18037,7 @@
"name": "q",
"required": true,
"in": "query",
- "description": "Search term (required, non-empty, at most 1000 characters and 32 words)",
+ "description": "Search term (required, non-empty, at most 1000 characters and 32 words, no NUL character)",
"schema": {
"maxLength": 1000,
"type": "string"
@@ -18047,7 +18047,7 @@
"name": "match",
"required": false,
"in": "query",
- "description": "word (default): whole words. contains: any part of a word, ignoring case and accents; ranks the 1000 most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true",
+ "description": "word (default): whole words. contains: any part of a word, ignoring case and accents; ranks the 1000 most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true, a term of 3 or more characters and at most 64 characters in all its terms",
"schema": {
"enum": [
"word",
@@ -18161,7 +18161,7 @@
}
},
"400": {
- "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 words; an unknown \"match\"; or a \"match=contains\" query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row)",
+ "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 words or holding a NUL character; an unknown \"match\"; or a \"match=contains\" query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row) or with more than 64 characters in its terms",
"content": {
"application/json": {
"schema": {
diff --git a/src/core/agent-tools/tools/search.tools.spec.ts b/src/core/agent-tools/tools/search.tools.spec.ts
index 9ff91cdc8..eb8a2c844 100644
--- a/src/core/agent-tools/tools/search.tools.spec.ts
+++ b/src/core/agent-tools/tools/search.tools.spec.ts
@@ -4,6 +4,7 @@ import type { AuthService } from '../../../modules/auth/auth.service';
import { ApiKeyRole } from '../../../modules/auth/entities/api-key.entity';
import type { SearchService } from '../../../modules/search/search.service';
import { invokeTool } from '../tool-invoker';
+import { ContainsTermsRefusedError } from '../../../modules/search/search-terms';
import { searchTools } from './search.tools';
const CHAT = '[email protected]';
@@ -64,13 +65,38 @@ describe('MessageSearch tool', () => {
await expect(call(key, { q: 'invoice', match: 'contains' })).rejects.toBeInstanceOf(NotImplementedException);
expect(search).toHaveBeenCalledTimes(1);
- // Only the 501 of an unready index falls back; any other failure reaches the agent as it was.
+ // Only the 501 of an unready index and a refusal of the terms (below) fall back; any other failure
+ // reaches the agent as it was.
search.mockClear();
search.mockRejectedValueOnce(new BadRequestException('Every term is under three characters'));
await expect(call(key, { q: 'in' })).rejects.toBeInstanceOf(BadRequestException);
expect(search).toHaveBeenCalledTimes(1);
});
+ // REST's default word mode answers `ok`, `東京` or a pasted sentence; contains refuses them for their terms.
+ it.each([
+ ['too short', 'ok', 'A contains search needs at least one term of 3 or more characters.'],
+ ['too long', 'a'.repeat(65), 'The terms of a contains search may hold at most 64 characters together.'],
+ ])('searches by word when contains refuses %s terms, unless the agent asked for contains', async (_n, q, refusal) => {
+ const key = { id: 'k1', allowedSessions: null, allowedChats: null };
+ search.mockRejectedValueOnce(new ContainsTermsRefusedError(refusal));
+
+ await expect(call(key, { q })).resolves.toBe(results);
+ expect(search).toHaveBeenNthCalledWith(1, { q, match: 'contains' }, undefined, undefined);
+ expect(search).toHaveBeenNthCalledWith(2, { q, match: 'word' }, undefined, undefined);
+
+ search.mockClear();
+ search.mockRejectedValueOnce(new ContainsTermsRefusedError(refusal));
+ await expect(call(key, { q, match: 'contains' })).rejects.toBeInstanceOf(BadRequestException);
+ expect(search).toHaveBeenCalledTimes(1);
+ });
+
+ it('refuses a NUL character in the query before searching, which PostgreSQL would fail on', async () => {
+ const key = { id: 'k1', allowedSessions: null, allowedChats: null };
+ await expect(call(key, { q: `invoice${String.fromCharCode(0)}` })).rejects.toBeInstanceOf(BadRequestException);
+ expect(search).not.toHaveBeenCalled();
+ });
+
it('bounds dateFrom and dateTo to the safe integer range, as REST does', async () => {
const key = { id: 'k1', allowedSessions: null, allowedChats: null };
diff --git a/src/core/agent-tools/tools/search.tools.ts b/src/core/agent-tools/tools/search.tools.ts
index b4f70c824..0f08fd1f5 100644
--- a/src/core/agent-tools/tools/search.tools.ts
+++ b/src/core/agent-tools/tools/search.tools.ts
@@ -4,12 +4,15 @@ import { ApiKeyRole } from '../../../modules/auth/entities/api-key.entity';
import { MessageDirection } from '../../../modules/message/entities/message.entity';
import type { SearchService } from '../../../modules/search/search.service';
import {
+ SEARCH_CONTAINS_MIN_TERM_LENGTH,
+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,
SEARCH_CONTAINS_WINDOW,
SEARCH_LIMIT_MAX,
SEARCH_MATCH_MODES,
SEARCH_OFFSET_MAX,
SEARCH_QUERY_MAX_LENGTH,
} from '../../../modules/search/search.constants';
+import { ContainsTermsRefusedError } from '../../../modules/search/search-terms';
import { chatIdSchema } from '../chat-id-schema';
import { defineTool, type AnyToolDescriptor } from '../tool-descriptor';
@@ -21,14 +24,16 @@ import { defineTool, type AnyToolDescriptor } from '../tool-descriptor';
* configuration; calling it answers 501, as the REST route answers 404 then.
*
* `containsEnabled` is SEARCH_CONTAINS_ENABLED. With it on, `match` defaults to contains, unlike REST
- * (word): an agent often knows only part of a word or types it without accents, and contains finds
- * everything word finds. With it off the built-in provider answers 501 to contains, so the tool
- * defaults to word like REST.
+ * (word): an agent often knows only part of a word or types it without accents. Contains does not run
+ * every query word does (two-character words alone, as most Korean and many CJK words are, give the
+ * trigram index nothing to look up, and the terms together are bounded), so a query it refuses for its
+ * terms runs as a word search, as does one while its index is not ready. With it off the built-in
+ * provider answers 501 to contains, so the tool defaults to word like REST.
*/
export function searchTools(search: SearchService | undefined, containsEnabled = false): AnyToolDescriptor[] {
const defaultMatch = containsEnabled ? 'contains' : 'word';
const matchHelp = containsEnabled
- ? `match=contains (the default) finds any part of a word, ignoring case and accents, and ranks the ${SEARCH_CONTAINS_WINDOW} most recently stored matches (whole words first, then newest), so total is at most ${SEARCH_CONTAINS_WINDOW}: narrow with more terms, chatId or dateTo to reach older ones. match=word finds whole words only.`
+ ? `match=contains (the default) finds any part of a word, ignoring case and accents, and ranks the ${SEARCH_CONTAINS_WINDOW} most recently stored matches (whole words first, then newest), so total is at most ${SEARCH_CONTAINS_WINDOW}: narrow with more terms, chatId or dateTo to reach older ones. It needs a term of ${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more characters and at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in all its terms; without match, a query outside those bounds runs as match=word, which finds whole words only.`
: 'match=word (the default) finds whole words; match=contains needs SEARCH_CONTAINS_ENABLED=true on the server.';
return [
defineTool({
@@ -75,7 +80,10 @@ export function searchTools(search: SearchService | undefined, containsEnabled =
try {
return await search.search({ ...input, match: input.match ?? defaultMatch }, sessionIds, chatIds);
} catch (error) {
- if (input.match || defaultMatch === 'word' || !(error instanceof NotImplementedException)) throw error;
+ // Unless the agent asked for contains, a query contains cannot run yet (its index is not ready)
+ // or cannot run at all for its terms (too short or too long) is answered as a word search.
+ const fallback = error instanceof NotImplementedException || error instanceof ContainsTermsRefusedError;
+ if (input.match || defaultMatch === 'word' || !fallback) throw error;
return search.search({ ...input, match: 'word' }, sessionIds, chatIds);
}
},
diff --git a/src/modules/search/dto/search-query.dto.spec.ts b/src/modules/search/dto/search-query.dto.spec.ts
index ad07fe711..52a5450e1 100644
--- a/src/modules/search/dto/search-query.dto.spec.ts
+++ b/src/modules/search/dto/search-query.dto.spec.ts
@@ -43,6 +43,12 @@ describe('SearchQueryDto', () => {
);
});
+ // PostgreSQL refuses U+0000 in a bound parameter in either match mode, and SQLite's FTS5 in a contains query.
+ it('rejects a q holding a NUL character', () => {
+ const errors = validateSync(fromQuery({ q: `invoice${String.fromCharCode(0)}` }));
+ expect(errors.map(e => [e.property, Object.keys(e.constraints ?? {})])).toEqual([['q', ['notContains']]]);
+ });
+
it('rejects limit < 1 (@Min(1))', () => {
const dto = fromQuery({ q: 'hello', limit: '0' });
expect(validateSync(dto).some(e => e.property === 'limit')).toBe(true);
diff --git a/src/modules/search/dto/search-query.dto.ts b/src/modules/search/dto/search-query.dto.ts
index 536157e31..f1bb72da0 100644
--- a/src/modules/search/dto/search-query.dto.ts
+++ b/src/modules/search/dto/search-query.dto.ts
@@ -4,8 +4,10 @@ import { Type } from 'class-transformer';
import { MessageDirection } from '../../message/entities/message.entity';
import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';
import { ToStrictNumber, ToStrictString } from '../../../common/utils/strict-boolean';
+import { NoNulCharacter } from '../../../common/validation/no-nul-character';
import {
SEARCH_CONTAINS_MIN_TERM_LENGTH,
+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,
SEARCH_CONTAINS_WINDOW,
SEARCH_MATCH_MODES,
SEARCH_OFFSET_MAX,
@@ -28,13 +30,15 @@ import type { SearchMatch } from '../search.types';
*/
export class SearchQueryDto {
@ApiProperty({
- description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} words)`,
+ description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} words, no NUL character)`,
maxLength: SEARCH_QUERY_MAX_LENGTH,
})
@ToStrictString()
@IsString()
@IsNotEmpty()
@MaxLength(SEARCH_QUERY_MAX_LENGTH)
+ // PostgreSQL refuses U+0000 in a bound parameter, and SQLite's FTS5 in a contains query: both a 500.
+ @NoNulCharacter()
q!: string;
@ApiPropertyOptional({
@@ -44,7 +48,9 @@ export class SearchQueryDto {
'`word` (default) matches whole words. `contains` matches any part of a word, ignoring case and ' +
`accents (\`voic\` finds "Invoice", \`cafe\` finds "Café"); it is opt-in on the built-in provider ` +
'(SEARCH_CONTAINS_ENABLED=true, else 501) and needs at least one term of ' +
- `${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more characters (on PostgreSQL, ${SEARCH_CONTAINS_MIN_TERM_LENGTH} letters or digits in a row). ` +
+ `${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more characters (on PostgreSQL, ${SEARCH_CONTAINS_MIN_TERM_LENGTH} letters or digits in a row) ` +
+ `and at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in all its terms, spaces not counted (400 otherwise), so a ` +
+ 'query of two-character words alone, as most Korean and many Chinese and Japanese words are, needs `word`. ' +
`It ranks the ${SEARCH_CONTAINS_WINDOW} most recently stored matches by match quality, so \`total\` is at most ` +
`${SEARCH_CONTAINS_WINDOW}.`,
})
diff --git a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts b/src/modules/search/providers/builtin-fts.provider.contains.spec.ts
index 742e04336..bc75ca6a9 100644
--- a/src/modules/search/providers/builtin-fts.provider.contains.spec.ts
+++ b/src/modules/search/providers/builtin-fts.provider.contains.spec.ts
@@ -5,7 +5,8 @@ import { DataSource } from 'typeorm';
import { Message, MessageDirection } from '../../../modules/message/entities/message.entity';
import { Session } from '../../../modules/session/entities/session.entity';
import { AddMessagesFts1782400000000 } from '../../../database/migrations/1782400000000-AddMessagesFts';
-import { SEARCH_CONTAINS_WINDOW } from '../search.constants';
+import { SEARCH_CONTAINS_TERMS_MAX_LENGTH, SEARCH_CONTAINS_WINDOW } from '../search.constants';
+import { ContainsTermsRefusedError } from '../search-terms';
import {
BuiltInFtsProvider,
postgresUnaccentFunctionDdl,
@@ -97,6 +98,41 @@ describe('BuiltInFtsProvider match=contains (sqlite)', () => {
await expect(provider.search({ q: 'id to', match: 'contains' })).rejects.toBeInstanceOf(BadRequestException);
});
+ it('counts a Hangul syllable as one character, so a two-syllable word only narrows', async () => {
+ await ds
+ .getRepository(Message)
+ .insert([
+ row('s1', 'c5', '서울역에서 만나요', MessageDirection.INCOMING, 50),
+ row('s1', 'c6', 'invoice 한국 지사로 보냄', MessageDirection.INCOMING, 60),
+ ]);
+
+ await expect(provider.search({ q: '서울', match: 'contains' })).rejects.toBeInstanceOf(ContainsTermsRefusedError);
+ const station = await provider.search({ q: '서울역', match: 'contains' });
+ expect(station.hits.map(h => [h.chatId, h.snippet])).toEqual([['c5', '<mark>서울역</mark>에서 만나요']]);
+ expect((await provider.search({ q: 'invoice 한국', match: 'contains' })).hits.map(h => h.chatId)).toEqual(['c6']);
+ });
+
+ it('marks a Greek word whose final sigma the body writes in capitals', async () => {
+ await ds.getRepository(Message).insert(row('s1', 'c5', 'ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', MessageDirection.INCOMING, 50));
+ const query = jest.spyOn(ds, 'query');
+
+ const res = await provider.search({ q: 'δρομος', match: 'contains' });
+
+ expect(res.hits.map(h => [h.snippet, h.score])).toEqual([['ΚΑΛΟΣ <mark>ΔΡΟΜΟΣ</mark> σήμερα', 3]]);
+ // Only the application reads final sigma as sigma: the database is sent the term as written.
+ expect(query.mock.calls.at(-1)?.[1]).toContain('"δρομος"');
+ });
+
+ it('bounds the characters of the terms together, spaces not counted', async () => {
+ const max = 'a'.repeat(SEARCH_CONTAINS_TERMS_MAX_LENGTH);
+ expect((await provider.search({ q: max, match: 'contains' })).total).toBe(0);
+ for (const q of [`${max}b`, `${max.slice(0, 40)} ${max.slice(0, 25)}x`]) {
+ await expect(provider.search({ q, match: 'contains' })).rejects.toBeInstanceOf(ContainsTermsRefusedError);
+ }
+ // A repeated term counts once, as it runs once.
+ expect((await provider.search({ q: `${max} ${max.toUpperCase()}`, match: 'contains' })).total).toBe(0);
+ });
+
it('matches a LIKE wildcard in a term literally', async () => {
expect((await provider.search({ q: '10%', match: 'contains' })).total).toBe(1);
expect((await provider.search({ q: 'invoice _', match: 'contains' })).total).toBe(0);
diff --git a/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts b/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts
index 2e8bd6a51..1253f3379 100644
--- a/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts
+++ b/src/modules/search/providers/builtin-fts.provider.postgres.spec.ts
@@ -1,7 +1,9 @@
import 'reflect-metadata';
import { NotImplementedException } from '@nestjs/common';
+import { ConfigService } from '@nestjs/config';
import { LoggerService } from '../../../common/services/logger.service';
import { DataSource } from 'typeorm';
+import { ContainsTermsRefusedError } from '../search-terms';
import { BuiltInFtsProvider } from './builtin-fts.provider';
/**
@@ -186,3 +188,32 @@ describe('BuiltInFtsProvider (postgres probe)', () => {
await expect(provider.search({ q: 'hello' })).resolves.toMatchObject({ provider: 'builtin-fts' });
});
});
+
+describe('BuiltInFtsProvider match=contains terms (postgres)', () => {
+ /** A provider whose catalog read finds the trigram index ready, over a database that matches nothing. */
+ const containsReady = () => {
+ const query = jest.fn().mockResolvedValue([]);
+ const manager = {
+ query: jest.fn().mockResolvedValue([{ trgm: 'public', unaccent: 'public', fn: true, idx: true }]),
+ };
+ const ds = { options: { type: 'postgres' }, query, manager } as unknown as DataSource;
+ return { provider: new BuiltInFtsProvider(ds, new ConfigService({ search: { containsEnabled: true } })), query };
+ };
+
+ // As show_trgm(openwa_unaccent(term)) reads them on PostgreSQL 16, with glibc and with musl: a Hangul
+ // word of two syllables, a modifier letter unaccent turns into an apostrophe and superscript digits
+ // give pg_trgm no trigram to look up, so the search would read every row.
+ it.each(['안녕', '\u02bc'.repeat(3), '¹²³'])('refuses %j alone, which pg_trgm cannot look up', async q => {
+ const { provider, query } = containsReady();
+ await expect(provider.search({ q, match: 'contains' })).rejects.toBeInstanceOf(ContainsTermsRefusedError);
+ expect(query).not.toHaveBeenCalled();
+ });
+
+ // Hangul syllables, a katakana word with its prolonged sound mark, and letter numbers (which unaccent
+ // spells in Latin) give it trigrams.
+ it.each(['서울역', 'ラーメン', 'ⅫⅫⅫ'])('looks %j up', async q => {
+ const { provider, query } = containsReady();
+ await expect(provider.search({ q, match: 'contains' })).resolves.toMatchObject({ total: 0 });
+ expect(query).toHaveBeenCalledTimes(1);
+ });
+});
diff --git a/src/modules/search/providers/builtin-fts.provider.ts b/src/modules/search/providers/builtin-fts.provider.ts
index 07344f79e..c2ee46177 100644
--- a/src/modules/search/providers/builtin-fts.provider.ts
+++ b/src/modules/search/providers/builtin-fts.provider.ts
@@ -16,8 +16,13 @@ import { DataSource, QueryRunner } from 'typeorm';
import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';
import { MessageDirection } from '../../message/entities/message.entity';
import type { SearchProvider, SearchQuery, SearchResults, SearchHit } from '../search.types';
-import { SEARCH_CONTAINS_MIN_TERM_LENGTH, SEARCH_CONTAINS_WINDOW, SEARCH_LIMIT_MAX } from '../search.constants';
-import { containsTerms, foldSearchText } from '../search-terms';
+import {
+ SEARCH_CONTAINS_MIN_TERM_LENGTH,
+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,
+ SEARCH_CONTAINS_WINDOW,
+ SEARCH_LIMIT_MAX,
+} from '../search.constants';
+import { ContainsTermsRefusedError, containsTerms, foldSearchText } from '../search-terms';
import { containsSnippet } from '../search-snippet';
import { containsScore } from '../search-score';
@@ -70,8 +75,16 @@ const CONTAINS_UNAVAILABLE = {
'Contains search is unavailable: this SQLite build has no FTS5 trigram tokenizer (SQLite 3.45+ is required).',
} as const;
-/** Three letters or digits in a row: pg_trgm takes trigrams from nothing else. */
-const POSTGRES_TRIGRAM_SOURCE = new RegExp(`[\\p{L}\\p{N}]{${SEARCH_CONTAINS_MIN_TERM_LENGTH}}`, 'u');
+/**
+ * Three letters or digits in a row, the only characters pg_trgm takes trigrams from: letters of every
+ * kind, decimal digits and letter numbers (`Ⅻ`), never other numbers (`²`, `①`). The nine modifier
+ * letters left out are the ones openwa_unaccent turns into punctuation before pg_trgm reads them (`ʼ`
+ * into an apostrophe, `ˆ` into `^`).
+ */
+const POSTGRES_TRIGRAM_SOURCE = new RegExp(
+ `(?:(?![\\u02b9-\\u02bd\\u02c6\\u02c8\\u02cb\\u02d0])[\\p{L}\\p{Nl}\\p{Nd}]){${SEARCH_CONTAINS_MIN_TERM_LENGTH}}`,
+ 'u',
+);
/**
* The contains candidate SELECT list. It carries no snippet or score column: searchContains computes
@@ -624,8 +637,10 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli
*
* 1. The database returns at most SEARCH_CONTAINS_WINDOW candidates: the most recently stored
* messages that contain every term, found through the trigram index, newest first. Both
- * dialects answer that order from an index and stop at the window, so the work is bounded
- * however many messages match.
+ * dialects answer that order from an index and stop once the window is full, so a common match
+ * costs no more than a rare one. A message the index yields that a narrowing term then rules out
+ * does not fill it, so the messages holding the looked-up terms bound the work, multiplied by
+ * the length of the terms (SEARCH_CONTAINS_TERMS_MAX_LENGTH).
* 2. The application ranks the candidates with containsScore (whole word, then word start, then
* inside a word), newest first within a score and `id` last for a total order, and pages them in
* memory.
@@ -730,17 +745,24 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit, OnAppli
* it takes no trigram from (FTS5 returns nothing for it, pg_trgm scans every row), so such terms only
* narrow the rows another term already found, and a query made of them alone is a 400 rather than a
* full-table scan. SQLite takes trigrams from any three characters, pg_trgm only from three letters or
- * digits in a row (see postgresCanIndex).
+ * digits in a row (see postgresCanIndex). The terms together are bounded by
+ * SEARCH_CONTAINS_TERMS_MAX_LENGTH, since each of their characters costs a check on every candidate.
*/
private static containsTermsOf(q: string, isPostgres: boolean): string[] {
const terms = containsTerms(q);
const canIndex = isPostgres ? postgresCanIndex : (term: string) => !isShortTerm(term);
if (!terms.some(canIndex)) {
const unit = isPostgres ? 'letters or digits in a row' : 'characters';
- throw new BadRequestException(
+ throw new ContainsTermsRefusedError(
`A contains search needs at least one term of ${SEARCH_CONTAINS_MIN_TERM_LENGTH} or more ${unit}.`,
);
}
+ const length = terms.reduce((sum, term) => sum + [...foldSearchText(term)].length, 0);
+ if (length > SEARCH_CONTAINS_TERMS_MAX_LENGTH) {
+ throw new ContainsTermsRefusedError(
+ `The terms of a contains search may hold at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters together, spaces not counted.`,
+ );
+ }
return terms;
}
diff --git a/src/modules/search/search-score.spec.ts b/src/modules/search/search-score.spec.ts
index e813a588e..7122fb258 100644
--- a/src/modules/search/search-score.spec.ts
+++ b/src/modules/search/search-score.spec.ts
@@ -21,6 +21,11 @@ describe('containsScore', () => {
expect(containsScore('Thanks, I will pay the invoice today', ['invoice', 'to'])).toBe(5);
});
+ it('reads final sigma as sigma, whichever side writes it', () => {
+ expect(containsScore('ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', ['δρομος'])).toBe(3);
+ expect(containsScore('ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', ['δρομοσ'])).toBe(3);
+ });
+
it('scores 0 for a term the body does not contain', () => {
expect(containsScore('Order shipped', ['invoice'])).toBe(0);
});
diff --git a/src/modules/search/search-score.ts b/src/modules/search/search-score.ts
index 32b4793ae..9ddde0fcd 100644
--- a/src/modules/search/search-score.ts
+++ b/src/modules/search/search-score.ts
@@ -1,4 +1,4 @@
-import { foldSearchText } from './search-terms';
+import { foldForMatching } from './search-terms';
/**
* Relevance of a match=contains candidate, computed in the application so both dialects rank alike
@@ -14,9 +14,9 @@ import { foldSearchText } from './search-terms';
* caller's scope.
*/
export function containsScore(body: string, terms: string[]): number {
- const folded = foldSearchText(body);
+ const folded = foldForMatching(body);
let score = 0;
- for (const term of terms.map(foldSearchText)) {
+ for (const term of terms.map(foldForMatching)) {
let best = 0;
for (let at = folded.indexOf(term); at !== -1 && best < 3; at = folded.indexOf(term, at + 1)) {
const startsWord = !isWordChar(folded[at - 1]);
diff --git a/src/modules/search/search-snippet.spec.ts b/src/modules/search/search-snippet.spec.ts
index 61abf910d..fb377de3f 100644
--- a/src/modules/search/search-snippet.spec.ts
+++ b/src/modules/search/search-snippet.spec.ts
@@ -6,6 +6,11 @@ describe('foldSearchText', () => {
expect(foldSearchText('Meeting at the CAFÉ')).toBe('meeting at the cafe');
expect(foldSearchText('Ação')).toBe('acao');
});
+
+ it('keeps a Hangul syllable one character, however the text spells it', () => {
+ expect([...foldSearchText('서울')]).toEqual(['서', '울']);
+ expect(foldSearchText('서울'.normalize('NFD'))).toBe('서울');
+ });
});
describe('containsTerms', () => {
@@ -35,6 +40,16 @@ describe('containsSnippet', () => {
expect(containsSnippet(`meet at the ${decomposed}`, ['cafe'], 24)).toBe(`meet at the <mark>${decomposed}</mark>`);
});
+ it('marks a Hangul word in a body that spells its syllables as jamo', () => {
+ const [word, rest] = ['서울역'.normalize('NFD'), '에서 만나요'.normalize('NFD')];
+ expect(containsSnippet(`${word}${rest}`, ['서울역'], 24)).toBe(`<mark>${word}</mark>${rest}`);
+ });
+
+ it('marks a word ending in final sigma in a body written in capitals, and the other way round', () => {
+ expect(containsSnippet('ΚΑΛΟΣ ΔΡΟΜΟΣ σήμερα', ['δρομος'], 24)).toBe('ΚΑΛΟΣ <mark>ΔΡΟΜΟΣ</mark> σήμερα');
+ expect(containsSnippet('ο δρομος', ['ΔΡΟΜΟΣ'], 24)).toBe('ο <mark>δρομος</mark>');
+ });
+
it('opens the window a few words before a late match and marks where it cut', () => {
const words = Array.from({ length: 40 }, (_, i) => `w${i}`);
words[30] = 'invoice';
diff --git a/src/modules/search/search-snippet.ts b/src/modules/search/search-snippet.ts
index 883f5064a..17cd5d94f 100644
--- a/src/modules/search/search-snippet.ts
+++ b/src/modules/search/search-snippet.ts
@@ -1,4 +1,4 @@
-import { foldSearchText } from './search-terms';
+import { foldForMatching } from './search-terms';
/**
* Snippets for match=contains hits, built in the application rather than the database so both
@@ -20,27 +20,31 @@ interface FoldedText {
ends: number[];
}
+/**
+ * What folds as one unit: a character with the combining marks after it, or a Hangul syllable spelled
+ * as jamo (leading consonants, vowels, trailing consonants), which NFC composes into one syllable.
+ */
+const FOLD_UNIT = /[\u1100-\u115f]+[\u1160-\u11a7]*[\u11a8-\u11ff]*\p{M}*|[\s\S]\p{M}*/gu;
+
/**
* Folding can change a text's length: a body that spells `é` as `e` followed by a combining accent
* loses that accent unit. Matches are therefore found in the folded copy and mapped back through
- * per-unit offsets, rather than assuming positions line up.
+ * per-unit offsets, rather than assuming positions line up. The text folds one FOLD_UNIT at a time, as
+ * the whole body folds for the score: a letter's accents fold away with it, so a match ending on it
+ * covers them instead of leaving a dangling accent outside </mark>, and jamo recompose into the
+ * syllable the term folds to.
*/
function foldWithOffsets(text: string): FoldedText {
let folded = '';
const starts: number[] = [];
const ends: number[] = [];
- let offset = 0;
- for (const char of text) {
- const mapped = foldSearchText(char);
- // A combining mark folds to nothing; it belongs to the letter before it, so a match ending on that
- // letter also covers the mark instead of leaving a dangling accent outside </mark>.
- if (!mapped && ends.length) ends[ends.length - 1] = offset + char.length;
+ for (const { 0: unit, index } of text.matchAll(FOLD_UNIT)) {
+ const mapped = foldForMatching(unit);
for (let i = 0; i < mapped.length; i++) {
- starts.push(offset);
- ends.push(offset + char.length);
+ starts.push(index);
+ ends.push(index + unit.length);
}
folded += mapped;
- offset += char.length;
}
return { folded, starts, ends };
}
@@ -49,7 +53,7 @@ function foldWithOffsets(text: string): FoldedText {
function matchRanges(body: string, terms: string[]): Array<[number, number]> {
const { folded, starts, ends } = foldWithOffsets(body);
const ranges: Array<[number, number]> = [];
- for (const term of terms.map(foldSearchText).filter(Boolean)) {
+ for (const term of terms.map(foldForMatching).filter(Boolean)) {
for (let at = folded.indexOf(term); at !== -1; at = folded.indexOf(term, at + term.length)) {
ranges.push([starts[at], ends[at + term.length - 1]]);
}
diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.ts
index 2eed09cdf..8a375df09 100644
--- a/src/modules/search/search-terms.ts
+++ b/src/modules/search/search-terms.ts
@@ -1,3 +1,5 @@
+import { BadRequestException } from '@nestjs/common';
+
/**
* What the built-in index (SQLite FTS5, unicode61) makes of a query, as an upper bound on its words. A
* run of letters or digits is one word. ASCII punctuation separates. Any other character outside ASCII
@@ -52,13 +54,31 @@ function wordsOf(word: string): number {
/**
* Folds text the way the match=contains indexes compare it: accents dropped (NFD, then every combining
- * mark removed, as remove_diacritics and unaccent do for Latin text) and case lowered. Used to compare
- * terms and to place highlights, never to build SQL, which leaves folding to the database.
+ * mark removed, as remove_diacritics and unaccent do for Latin text), recomposed (NFC: NFD splits a
+ * Hangul syllable into conjoining jamo, which are letters, so `서울` would count five characters where
+ * the database sees two) and case lowered. Used to compare terms and to place highlights, never to
+ * build SQL, which leaves folding to the database.
*/
export function foldSearchText(text: string): string {
- return text.normalize('NFD').replace(/\p{M}/gu, '').toLowerCase();
+ return text.normalize('NFD').replace(/\p{M}/gu, '').normalize('NFC').toLowerCase();
+}
+
+/**
+ * foldSearchText for the application's own comparisons, the score and the highlight, with final sigma
+ * read as sigma: lowercasing a whole body turns a word-final `Σ` into `ς`, and one character alone into
+ * `σ`, so a match the score counted got no highlight. The terms sent to the database are not affected.
+ */
+export function foldForMatching(text: string): string {
+ return foldSearchText(text).replace(/ς/g, 'σ');
}
+/**
+ * A match=contains query refused for its terms (none long enough for a trigram index, or more characters
+ * than SEARCH_CONTAINS_TERMS_MAX_LENGTH), which a word search can still answer: the MessageSearch tool
+ * runs one instead when the agent did not ask for contains.
+ */
+export class ContainsTermsRefusedError extends BadRequestException {}
+
/**
* The whitespace-separated terms of a match=contains query, each kept as written, with a term that
* folds to one already seen dropped (`Café cafe` is one term): it would add nothing to an AND but
diff --git a/src/modules/search/search.constants.ts b/src/modules/search/search.constants.ts
index 4d293527d..b864e7d4c 100644
--- a/src/modules/search/search.constants.ts
+++ b/src/modules/search/search.constants.ts
@@ -42,10 +42,19 @@ export const SEARCH_MATCH_MODES = ['word', 'contains'] as const satisfies readon
*/
export const SEARCH_CONTAINS_MIN_TERM_LENGTH = 3;
+/**
+ * Most characters the terms of a match=contains query may hold together, counted after folding and
+ * without the spaces between them. The trigram index checks every trigram of a term on every message
+ * that holds all of them, and on SQLite synchronously: a query of `the` repeated, on 200k messages of
+ * which 40% hold `the`, `het` and `eth`, took about 100 ms at 64 characters whether written as one term
+ * or as several, and 2.8 s at 1000. A longer query answers 400; match=word runs it.
+ */
+export const SEARCH_CONTAINS_TERMS_MAX_LENGTH = 64;
+
/**
* How many candidates a match=contains search ranks: the most recently stored matches, fetched from
- * the index in recency order so the database stops at this many however many messages match, then
- * ranked in the application. It bounds the work of one search (ranking every match in SQL took about
+ * the index in recency order so the database stops once it holds this many, then ranked in the
+ * application. It bounds the ranking work of one search (ranking every match in SQL took about
* 100 ms on SQLite and 1.6 s on Postgres for 200k matches) and is therefore also the most `total` can
* report in this mode. Fixed rather than env-driven, like SEARCH_OFFSET_MAX.
*/
diff --git a/src/modules/search/search.controller.ts b/src/modules/search/search.controller.ts
index 3aaac81d2..1b1e8dc5f 100644
--- a/src/modules/search/search.controller.ts
+++ b/src/modules/search/search.controller.ts
@@ -6,7 +6,12 @@ import { ChatScopeService } from '../auth/chat-scope.service';
import { ApiKey, ApiKeyRole } from '../auth/entities/api-key.entity';
import { SearchService } from './search.service';
import { SearchQueryDto } from './dto/search-query.dto';
-import { SEARCH_CONTAINS_WINDOW, SEARCH_MATCH_MODES, SEARCH_OFFSET_MAX } from './search.constants';
+import {
+ SEARCH_CONTAINS_TERMS_MAX_LENGTH,
+ SEARCH_CONTAINS_WINDOW,
+ SEARCH_MATCH_MODES,
+ SEARCH_OFFSET_MAX,
+} from './search.constants';
import type { SearchResults } from './search.types';
@ApiTags('search')
@@ -25,8 +30,9 @@ export class SearchController {
@ApiResponse({
status: 400,
description:
- 'Empty or whitespace-only "q", or one over 1000 characters or 32 words; an unknown "match"; or a ' +
- '"match=contains" query with no term of 3 or more characters (on PostgreSQL, 3 letters or digits in a row)',
+ 'Empty or whitespace-only "q", or one over 1000 characters or 32 words or holding a NUL character; an ' +
+ 'unknown "match"; or a "match=contains" query with no term of 3 or more characters (on PostgreSQL, 3 ' +
+ `letters or digits in a row) or with more than ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in its terms`,
})
@ApiResponse({ status: 403, description: 'Chat scope denied or chat-restricted search uses a plugin provider' })
@ApiResponse({
@@ -51,7 +57,7 @@ export class SearchController {
@ApiQuery({
name: 'q',
required: true,
- description: 'Search term (required, non-empty, at most 1000 characters and 32 words)',
+ description: 'Search term (required, non-empty, at most 1000 characters and 32 words, no NUL character)',
})
@ApiQuery({
name: 'match',
@@ -59,7 +65,8 @@ export class SearchController {
enum: SEARCH_MATCH_MODES,
description:
'word (default): whole words. contains: any part of a word, ignoring case and accents; ranks the ' +
- `${SEARCH_CONTAINS_WINDOW} most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true`,
+ `${SEARCH_CONTAINS_WINDOW} most recently stored matches by match quality; needs SEARCH_CONTAINS_ENABLED=true, ` +
+ `a term of 3 or more characters and at most ${SEARCH_CONTAINS_TERMS_MAX_LENGTH} characters in all its terms`,
})
@ApiQuery({ name: 'sessionId', required: false, description: 'Restrict to a single session' })
@ApiQuery({