Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(search): count punctuation-separated words toward the bound
Yudhi Armyndharisd39236ca6d65
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e04a9de7e..9fd2d82c6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -122,7 +122,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Drop a Baileys edit, revoke or reaction that the stored message proves foreign (another chat, or a one-to-one change by the other party), also when the engine's own message store no longer holds the original.- Name who reacted in a Baileys 1:1 reaction, whose sender id came out empty so the two parties overwrote each other's reaction, and flag a 1:1 message from a lid-only sender as `isLidSender`.- Read the duration of an outgoing Baileys audio file in a separate process and a voice note's waveform through ffmpeg, status voice notes included, so a crafted file can no longer abort or exhaust the gateway.-- Bound `GET /api/search` to 1000 characters and 32 distinct words and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.+- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator, and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.- Print the startup banner's API Docs URL only when Swagger is served, so a production boot no longer points at a `404`.- Decode numeric and common named character references in a Baileys link preview's title and description, so `’` and `é` no longer reach the recipient as written.- Log an error at whatsapp-web.js start when the message-id backport is only partly applied, naming the files it did not reach.@@ -162,7 +162,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- With `OPENWA_DATA_DIR` set, `backup.sh` and `restore.sh` read the default SQLite databases and `./data/...` paths in `./.env` under it; pass a path in the environment to keep it in the working directory.- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.- Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working on the REST routes, but over `/events` a subscribe that would hold more than 4096 rooms (the subscribed events times the key's chat ids, two per phone number) is refused with `TOO_MANY_SUBSCRIPTIONS`, so shorten it.-- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 distinct words; a repeated word counts once.+- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words); a repeated word counts once.- Without an ffmpeg binary (`FFMPEG_PATH`), a Baileys voice note goes out with flat waveform bars instead of its real ones; the official image ships ffmpeg.- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.- The first boot builds three indexes on `messages` (`IDX_messages_session_timestamp_id`, `IDX_messages_session_chat_timestamp_id` and `IDX_messages_session_direction_timestamp_id`) not concurrently, and the table with its indexes grows by about 240 MB per million messages. On PostgreSQL writes to that table wait until the boot migrations commit, after all three builds, and the Helm chart's startup probe (about 295 s) restarts a pod whose boot outlasts it, so on a large table run the migrations first (migration guide, section 14.5).diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 284f76cea..6d758ca08 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6502,7 +6502,7 @@ Search messages across sessions (active search provider).| Name | Type | Required | Default | Description || ----------- | ------------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; whitespace-only is rejected with `400`, and so is one longer than 1000 characters or with more than 32 distinct words (a repeated word counts once). Passed to the active provider's native full-text matcher. |+| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words as whitespace does; a repeated word is counted once). Passed to the active provider's native full-text matcher. || `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). || `chatId` | string | No | — | Restrict to a single chat id. For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are additionally narrowed to the allowlist. || `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |diff --git a/docs/26-global-search.md b/docs/26-global-search.mdindex c75c3b544..4b39ca0ae 100644--- a/docs/26-global-search.md+++ b/docs/26-global-search.md@@ -134,7 +134,7 @@ GET /api/search?q=<term>&sessionId=<id>&chatId=<id>&direction=<incoming|outgoing```- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one- longer than 1000 characters or with more than 32 distinct words; a repeated word counts once). Numeric params+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does; a repeated word counts once). Numeric paramsare coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,never as a `NaN` SQL parameter.- **Auth scoping is authoritative.** The caller's API-key `allowedSessions` is injected bydiff --git a/openapi.json b/openapi.jsonindex 64cee6dd6..4ef5956d9 100644--- a/openapi.json+++ b/openapi.json@@ -17723,7 +17723,7 @@"name": "q","required": true,"in": "query",- "description": "Search term (required, non-empty, at most 1000 characters and 32 distinct words)",+ "description": "Search term (required, non-empty, at most 1000 characters and 32 words)","schema": {"maxLength": 1000,"type": "string"@@ -17830,7 +17830,7 @@}},"400": {- "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 distinct words",+ "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 words","content": {"application/json": {"schema": {diff --git a/src/modules/search/dto/search-query.dto.ts b/src/modules/search/dto/search-query.dto.tsindex 0bfe02a94..76b14c764 100644--- a/src/modules/search/dto/search-query.dto.ts+++ b/src/modules/search/dto/search-query.dto.ts@@ -18,7 +18,7 @@ import { SEARCH_OFFSET_MAX, SEARCH_QUERY_MAX_LENGTH, SEARCH_TERMS_MAX } from '..*/export class SearchQueryDto {@ApiProperty({- description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} distinct words)`,+ description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} words)`,maxLength: SEARCH_QUERY_MAX_LENGTH,})@IsString()diff --git a/src/modules/search/search-terms.spec.ts b/src/modules/search/search-terms.spec.tsnew file mode 100644index 000000000..4f8f0ecf5--- /dev/null+++ b/src/modules/search/search-terms.spec.ts@@ -0,0 +1,63 @@+import 'reflect-metadata';+import { DataSource } from 'typeorm';+import { searchTermCount } from './search-terms';++describe('searchTermCount', () => {+ it('counts a run of letters or digits as one word and punctuation as a separator', () => {+ expect(searchTermCount('hello')).toBe(1);+ expect(searchTermCount('hello world')).toBe(2);+ expect(searchTermCount('a.b.c')).toBe(3);+ expect(searchTermCount('[email protected]')).toBe(3);+ expect(searchTermCount('2024-10-14')).toBe(3);+ });++ it('counts a whitespace-separated word said again once', () => {+ expect(searchTermCount('hi hi hi')).toBe(1);+ expect(searchTermCount('a.b a.b')).toBe(2);+ });++ it('counts a word with no letter or digit as one', () => {+ expect(searchTermCount('!!!')).toBe(1);+ });++ it('keeps the words of non-Latin text whole', () => {+ expect(searchTermCount('привет мир')).toBe(2);+ expect(searchTermCount('こんにちは')).toBe(1);+ });++ it('counts a character the index splits on although it is a letter as a separator', () => {+ expect(searchTermCount('a\u1CF2'.repeat(40))).toBe(40);+ });++ describe('against the bundled SQLite', () => {+ let ds: DataSource;+ beforeAll(async () => {+ ds = new DataSource({ type: 'better-sqlite3', database: ':memory:' });+ await ds.initialize();+ await ds.query('CREATE VIRTUAL TABLE probe USING fts5(body)');+ });+ afterAll(() => ds.destroy());++ it('never counts fewer words than the index splits a letter or digit into', async () => {+ const letters: number[] = [];+ for (let cp = 1; cp <= 0x10ffff; cp++) {+ if (cp >= 0xd800 && cp <= 0xdfff) continue;+ if (/[\p{L}\p{N}]/u.test(String.fromCodePoint(cp))) letters.push(cp);+ }+ for (let at = 0; at < letters.length; at += 5000) {+ const rows = letters+ .slice(at, at + 5000)+ .map(cp => `(${cp}, 'a${String.fromCodePoint(cp)}a')`)+ .join(',');+ await ds.query(`INSERT INTO probe (rowid, body) VALUES ${rows}`);+ }+ // `a<c>a` is two tokens exactly when the index treats <c> as a separator.+ const split = await ds.query<{ cp: number }[]>(`SELECT rowid AS cp FROM probe WHERE probe MATCH '"a a"'`);+ const undercounted = split+ .map(row => row.cp)+ .filter(cp => searchTermCount(`a${String.fromCodePoint(cp)}a`) < 2)+ .map(cp => cp.toString(16));+ expect(undercounted).toEqual([]);+ });+ });+});diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.tsnew file mode 100644index 000000000..e0f760dd4--- /dev/null+++ b/src/modules/search/search-terms.ts@@ -0,0 +1,20 @@+/**+ * What a word is made of, as the built-in index sees it: letters and digits, less the 21 characters+ * its tokenizer (SQLite FTS5, unicode61) still splits on although current Unicode tables class them+ * as letters. search-terms.spec.ts compares every letter and digit with the bundled SQLite, so an+ * upgrade that moves one fails there instead of silently loosening the bound.+ */+const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+/gu;++/**+ * How many words a query holds, counted as the index counts them: a run of letters or digits is one,+ * so punctuation separates words as whitespace does (`a.b.c` is three, and a quoted token of them is a+ * three-word phrase to match). A whitespace-separated word said again is run once and counts once.+ */+export function searchTermCount(q: string): number {+ let count = 0;+ for (const word of new Set(q.trim().split(/\s+/))) {+ count += word.match(WORD)?.length ?? 1;+ }+ return count;+}diff --git a/src/modules/search/search.constants.ts b/src/modules/search/search.constants.tsindex c6439091e..00b9accff 100644--- a/src/modules/search/search.constants.ts+++ b/src/modules/search/search.constants.ts@@ -23,8 +23,9 @@ export const SEARCH_DEFAULT_LIMIT = 50;export const SEARCH_QUERY_MAX_LENGTH = 1000;/**- * Most distinct whitespace-separated words in `q`. Each one costs a pass over every message that- * matches, and the built-in provider runs on the event loop, so a query of hundreds of words froze every- * session for seconds. A word repeated is still one word: the provider collapses repeats.+ * Most words in `q`, counted as the index counts them (see search-terms.ts). Each one costs a pass over+ * every message that matches, and the built-in provider runs on the event loop, so a query of hundreds+ * of words froze every session for seconds. A whitespace-separated word repeated is one word: the+ * provider collapses repeats.*/export const SEARCH_TERMS_MAX = 32;diff --git a/src/modules/search/search.controller.ts b/src/modules/search/search.controller.tsindex b062519b9..95cea2c76 100644--- a/src/modules/search/search.controller.ts+++ b/src/modules/search/search.controller.ts@@ -24,7 +24,7 @@ export class SearchController {@ApiResponse({ status: 200, description: 'Search results from the active provider', type: SearchResultsResponseDto })@ApiResponse({status: 400,- description: 'Empty or whitespace-only "q", or one over 1000 characters or 32 distinct words',+ description: 'Empty or whitespace-only "q", or one over 1000 characters or 32 words',})@ApiResponse({ status: 403, description: 'Chat scope denied or chat-restricted search uses a plugin provider' })@ApiResponse({ status: 501, description: 'No search provider configured' })@@ -43,7 +43,7 @@ export class SearchController {@ApiQuery({name: 'q',required: true,- description: 'Search term (required, non-empty, at most 1000 characters and 32 distinct words)',+ description: 'Search term (required, non-empty, at most 1000 characters and 32 words)',})@ApiQuery({ name: 'sessionId', required: false, description: 'Restrict to a single session' })@ApiQuery({ name: 'chatId', required: false, description: 'Restrict to a single chat id' })diff --git a/src/modules/search/search.service.spec.ts b/src/modules/search/search.service.spec.tsindex c004d2095..72ddbdb47 100644--- a/src/modules/search/search.service.spec.ts+++ b/src/modules/search/search.service.spec.ts@@ -51,7 +51,7 @@ describe('SearchService', () => {expect(search).toHaveBeenCalledWith(expect.objectContaining({ chatIds: ['[email protected]', '111@lid'] }));});- it('refuses a query of more than SEARCH_TERMS_MAX distinct words before any provider runs it', async () => {+ it('refuses a query of more than SEARCH_TERMS_MAX words before any provider runs it', async () => {const reg = new SearchProviderRegistry();const search = jest.fn().mockResolvedValue(emptyResults);reg.register(mkProvider('builtin-fts', search));@@ -67,6 +67,22 @@ describe('SearchService', () => {expect(search).toHaveBeenCalledTimes(2);});+ it('counts the words inside a token the way the index does, so punctuation cannot hide them', async () => {+ const reg = new SearchProviderRegistry();+ const search = jest.fn().mockResolvedValue(emptyResults);+ reg.register(mkProvider('builtin-fts', search));+ const svc = new SearchService(reg);+ const joined = (n: number, sep: string): string => Array.from({ length: n }, () => 'a').join(sep);++ for (const sep of ['.', '-', '_', '@', '\u1CF2']) {+ await expect(svc.search({ q: joined(SEARCH_TERMS_MAX + 1, sep) })).rejects.toBeInstanceOf(BadRequestException);+ }+ expect(search).not.toHaveBeenCalled();++ await svc.search({ q: joined(SEARCH_TERMS_MAX, '.') });+ expect(search).toHaveBeenCalledTimes(1);+ });+it('clamps an excessive limit to SEARCH_LIMIT_MAX before it reaches any provider', async () => {// Without the host-side clamp a plugin provider would receive the raw caller value (9999) and// could pressure host heap by returning an oversized page. The built-in already rejected this indiff --git a/src/modules/search/search.service.ts b/src/modules/search/search.service.tsindex a7be40bff..5b3974a45 100644--- a/src/modules/search/search.service.ts+++ b/src/modules/search/search.service.ts@@ -1,6 +1,7 @@import { BadRequestException, Injectable, NotImplementedException } from '@nestjs/common';import { SearchProviderRegistry } from './search-provider.registry';import { SEARCH_DEFAULT_LIMIT, SEARCH_LIMIT_MAX, SEARCH_OFFSET_MAX, SEARCH_TERMS_MAX } from './search.constants';+import { searchTermCount } from './search-terms';import type { SearchQuery, SearchResults } from './search.types';@Injectable()@@ -11,9 +12,9 @@ export class SearchService {const provider = this.registry.active();if (!provider) throw new NotImplementedException('Search is not configured (no active search provider).');// Bounded here, ahead of every provider: the cost of a query grows with its words, whatever backend- // runs it. A repeated word is one word.- if (new Set(query.q.trim().split(/\s+/)).size > SEARCH_TERMS_MAX) {- throw new BadRequestException(`q may hold at most ${SEARCH_TERMS_MAX} distinct words.`);+ // runs it. Counted as the index counts them, so a token with punctuation inside it is several.+ if (searchTermCount(query.q) > SEARCH_TERMS_MAX) {+ throw new BadRequestException(`q may hold at most ${SEARCH_TERMS_MAX} words, punctuation included.`);}// Auth scoping is authoritative — a caller cannot override it via the query. This covers both// fences: sessions (allowedSessions) and chats (allowedChats, expanded lid-aware by the caller),