Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(search): bound the words of a query and run a repeated one once
Yudhi Armyndharis65f868c1a8f7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2311903be..89cfdbf8d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -112,6 +112,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Keep one terminal webhook failure per receiver idempotency key.- Reject poll votes with no matching options while preserving explicit vote clearing.- Read the duration and waveform of an outgoing Baileys audio file in a separate process, so a crafted file can no longer abort or exhaust the gateway.+- Bound `GET /api/search` to 1000 characters and 32 distinct words and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.### Documentation@@ -146,6 +147,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- With `OPENWA_DATA_DIR` set, `backup.sh` and `restore.sh` read the default SQLite databases and `./data/...` paths in `./.env` under it; pass a path in the environment to keep it in the working directory.- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.- Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working until it is edited.+- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 distinct words; a repeated word counts once.- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.- The first boot builds three indexes on `messages` (`IDX_messages_session_timestamp_id`, `IDX_messages_session_chat_timestamp_id` and `IDX_messages_session_direction_timestamp_id`) not concurrently; on a large table boot takes longer, on PostgreSQL writes to that table wait until each build finishes, and the table with its indexes grows by about 240 MB per million messages.- Baileys answers `400` for an animated sticker with more than 500 frames.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 6d8c51614..805610466 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6500,18 +6500,18 @@ Search messages across sessions (active search provider).**Query parameters**-| Name | Type | Required | Default | Description |-| ----------- | ------------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; whitespace-only is rejected with `400`. Passed to the active provider's native full-text matcher. |-| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |-| `chatId` | string | No | — | Restrict to a single chat id. For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are additionally narrowed to the allowlist. |-| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |-| `type` | string | No | — | Filter by stored message `type` (e.g. `text`, `image`, `video`). Compared against `messages.type`; not an enum validation, any string is accepted and unmatched values simply return no hits. |-| `from` | string | No | — | Filter by sender. |-| `dateFrom` | integer (epoch ms) | No | — | Inclusive lower bound on `timestamp`. A non-numeric value is rejected with `400`. |-| `dateTo` | integer (epoch ms) | No | — | Inclusive upper bound on `timestamp`. A non-numeric value is rejected with `400`. |-| `limit` | integer (≥ 1) | No | `50` | Max hits to return. Clamped to `SEARCH_LIMIT_MAX` (default `100`). A non-integer value is rejected with `400`. |-| `offset` | integer (≥ 0) | No | `0` | Pagination offset, at most `100000`. A non-integer or larger value is rejected with `400`. |+| Name | Type | Required | Default | Description |+| ----------- | ------------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; whitespace-only is rejected with `400`, and so is one longer than 1000 characters or with more than 32 distinct words (a repeated word counts once). Passed to the active provider's native full-text matcher. |+| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |+| `chatId` | string | No | — | Restrict to a single chat id. For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are additionally narrowed to the allowlist. |+| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |+| `type` | string | No | — | Filter by stored message `type` (e.g. `text`, `image`, `video`). Compared against `messages.type`; not an enum validation, any string is accepted and unmatched values simply return no hits. |+| `from` | string | No | — | Filter by sender. |+| `dateFrom` | integer (epoch ms) | No | — | Inclusive lower bound on `timestamp`. A non-numeric value is rejected with `400`. |+| `dateTo` | integer (epoch ms) | No | — | Inclusive upper bound on `timestamp`. A non-numeric value is rejected with `400`. |+| `limit` | integer (≥ 1) | No | `50` | Max hits to return. Clamped to `SEARCH_LIMIT_MAX` (default `100`). A non-integer value is rejected with `400`. |+| `offset` | integer (≥ 0) | No | `0` | Pagination offset, at most `100000`. A non-integer or larger value is rejected with `400`. |**Response** `200` — `SearchResults`diff --git a/openapi.json b/openapi.jsonindex fe6bc8d4f..c8ef1f718 100644--- a/openapi.json+++ b/openapi.json@@ -17723,8 +17723,9 @@"name": "q","required": true,"in": "query",- "description": "Search term (required, non-empty)",+ "description": "Search term (required, non-empty, at most 1000 characters and 32 distinct words)","schema": {+ "maxLength": 1000,"type": "string"}},@@ -17829,7 +17830,7 @@}},"400": {- "description": "Empty or whitespace-only \"q\"",+ "description": "Empty or whitespace-only \"q\", or one over 1000 characters or 32 distinct words","content": {"application/json": {"schema": {diff --git a/src/modules/search/dto/search-query.dto.spec.ts b/src/modules/search/dto/search-query.dto.spec.tsindex 4c9f276f1..e5b548dd4 100644--- a/src/modules/search/dto/search-query.dto.spec.ts+++ b/src/modules/search/dto/search-query.dto.spec.ts@@ -2,7 +2,7 @@ import { DECORATORS } from '@nestjs/swagger';import { validateSync } from 'class-validator';import { plainToInstance } from 'class-transformer';import { SearchQueryDto } from './search-query.dto';-import { SEARCH_OFFSET_MAX } from '../search.constants';+import { SEARCH_OFFSET_MAX, SEARCH_QUERY_MAX_LENGTH } from '../search.constants';describe('SearchQueryDto', () => {// Mirrors how the global ValidationPipe (transform:true + enableImplicitConversion) instantiates@@ -36,6 +36,13 @@ describe('SearchQueryDto', () => {expect(validateSync(fromQuery({ q: 'hello', offset: '0.5' })).some(e => e.property === 'offset')).toBe(true);});+ it('rejects a q longer than SEARCH_QUERY_MAX_LENGTH and accepts one at the bound', () => {+ expect(validateSync(fromQuery({ q: 'a'.repeat(SEARCH_QUERY_MAX_LENGTH) }))).toHaveLength(0);+ expect(validateSync(fromQuery({ q: 'a'.repeat(SEARCH_QUERY_MAX_LENGTH + 1) })).some(e => e.property === 'q')).toBe(+ true,+ );+ });+it('rejects limit < 1 (@Min(1))', () => {const dto = fromQuery({ q: 'hello', limit: '0' });expect(validateSync(dto).some(e => e.property === 'limit')).toBe(true);diff --git a/src/modules/search/dto/search-query.dto.ts b/src/modules/search/dto/search-query.dto.tsindex ffa0f04dd..0bfe02a94 100644--- a/src/modules/search/dto/search-query.dto.ts+++ b/src/modules/search/dto/search-query.dto.ts@@ -1,10 +1,10 @@import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';-import { IsString, IsNotEmpty, IsOptional, IsEnum, IsInt, IsNumber, Max, Min } from 'class-validator';+import { IsString, IsNotEmpty, IsOptional, IsEnum, IsInt, IsNumber, Max, MaxLength, Min } from 'class-validator';import { Type } from 'class-transformer';import { MessageDirection } from '../../message/entities/message.entity';import type { MessageType } from '../../../engine/interfaces/whatsapp-engine.interface';import { ToStrictNumber } from '../../../common/utils/strict-boolean';-import { SEARCH_OFFSET_MAX } from '../search.constants';+import { SEARCH_OFFSET_MAX, SEARCH_QUERY_MAX_LENGTH, SEARCH_TERMS_MAX } from '../search.constants';/*** Request shape for GET /search. Numeric fields are coerced from query-string strings via@@ -17,9 +17,13 @@ import { SEARCH_OFFSET_MAX } from '../search.constants';* API-key allowedSessions/allowedChats (never user-supplied).*/export class SearchQueryDto {- @ApiProperty({ description: 'Search term (required, non-empty)' })+ @ApiProperty({+ description: `Search term (required, non-empty, at most ${SEARCH_QUERY_MAX_LENGTH} characters and ${SEARCH_TERMS_MAX} distinct words)`,+ maxLength: SEARCH_QUERY_MAX_LENGTH,+ })@IsString()@IsNotEmpty()+ @MaxLength(SEARCH_QUERY_MAX_LENGTH)q!: string;@ApiPropertyOptional({ description: 'Restrict to a single session' })diff --git a/src/modules/search/providers/builtin-fts.provider.spec.ts b/src/modules/search/providers/builtin-fts.provider.spec.tsindex 1e04a9fdc..51a27d80f 100644--- a/src/modules/search/providers/builtin-fts.provider.spec.ts+++ b/src/modules/search/providers/builtin-fts.provider.spec.ts@@ -187,6 +187,17 @@ describe('BuiltInFtsProvider (sqlite)', () => {}});+ it('runs a repeated word once, since a repeat adds nothing under AND but costs a pass per match', async () => {+ const query = jest.spyOn(ds, 'query');+ const repeated = await provider.search({ q: Array.from({ length: 200 }, () => 'hello').join(' ') });+ const single = await provider.search({ q: 'hello' });++ const matched = query.mock.calls.filter(([sql]) => String(sql).includes('MATCH')).map(([, params]) => params);+ expect((matched[0] as unknown[])[0]).toBe('"hello"');+ expect((matched[1] as unknown[])[0]).toBe('"hello"');+ expect(repeated.hits.map(h => h.messageId)).toEqual(single.hits.map(h => h.messageId));+ });+it('keeps multi-token AND semantics after quoting', async () => {// Both rows carry 'hello' AND 'world'... exactly one does ('goodbye world' lacks 'hello').const res = await provider.search({ q: 'hello world' });diff --git a/src/modules/search/providers/builtin-fts.provider.ts b/src/modules/search/providers/builtin-fts.provider.tsindex 079caf54a..fb4cd1f97 100644--- a/src/modules/search/providers/builtin-fts.provider.ts+++ b/src/modules/search/providers/builtin-fts.provider.ts@@ -314,7 +314,9 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit {* unchanged. Postgres already uses the tolerant websearch_to_tsquery and needs no equivalent.*/private static toFts5Query(raw: string): string {- const tokens = raw.trim().split(/\s+/).filter(Boolean);+ // A repeated token is dropped: `a a` is `a` under implicit AND, but FTS5 would pay for each copy+ // on every matching row.+ const tokens = [...new Set(raw.trim().split(/\s+/).filter(Boolean))];if (tokens.length === 0) return '""';return tokens.map(tok => `"${tok.replace(/"/g, '""')}"`).join(' ');}diff --git a/src/modules/search/search.constants.ts b/src/modules/search/search.constants.tsindex be8419d4a..c6439091e 100644--- a/src/modules/search/search.constants.ts+++ b/src/modules/search/search.constants.ts@@ -18,3 +18,13 @@ export const SEARCH_OFFSET_MAX = 100_000;/** Default page size when a caller omits `limit` (matches the built-in provider's historical default). */export const SEARCH_DEFAULT_LIMIT = 50;++/** Longest `q`, in characters. A generous bound: it exists so the request cannot be a megabyte of terms. */+export const SEARCH_QUERY_MAX_LENGTH = 1000;++/**+ * Most distinct whitespace-separated words in `q`. Each one costs a pass over every message that+ * matches, and the built-in provider runs on the event loop, so a query of hundreds of words froze every+ * session for seconds. A word repeated is still one word: the provider collapses repeats.+ */+export const SEARCH_TERMS_MAX = 32;diff --git a/src/modules/search/search.controller.ts b/src/modules/search/search.controller.tsindex 8e6256fbe..b062519b9 100644--- a/src/modules/search/search.controller.ts+++ b/src/modules/search/search.controller.ts@@ -22,7 +22,10 @@ export class SearchController {@RequireRole(ApiKeyRole.OPERATOR)@ApiOperation({ summary: 'Search messages across sessions (active search provider)' })@ApiResponse({ status: 200, description: 'Search results from the active provider', type: SearchResultsResponseDto })- @ApiResponse({ status: 400, description: 'Empty or whitespace-only "q"' })+ @ApiResponse({+ status: 400,+ description: 'Empty or whitespace-only "q", or one over 1000 characters or 32 distinct words',+ })@ApiResponse({ status: 403, description: 'Chat scope denied or chat-restricted search uses a plugin provider' })@ApiResponse({ status: 501, description: 'No search provider configured' })@ApiResponse({@@ -37,7 +40,11 @@ export class SearchController {'The active plugin search provider did not answer: its worker is not running, timed out, or reported ' +'a failure. The built-in provider never returns this. Retryable.',})- @ApiQuery({ name: 'q', required: true, description: 'Search term (required, non-empty)' })+ @ApiQuery({+ name: 'q',+ required: true,+ description: 'Search term (required, non-empty, at most 1000 characters and 32 distinct words)',+ })@ApiQuery({ name: 'sessionId', required: false, description: 'Restrict to a single session' })@ApiQuery({ name: 'chatId', required: false, description: 'Restrict to a single chat id' })@ApiQuery({ name: 'direction', required: false, description: 'incoming | outgoing' })diff --git a/src/modules/search/search.service.spec.ts b/src/modules/search/search.service.spec.tsindex 91285385d..c004d2095 100644--- a/src/modules/search/search.service.spec.ts+++ b/src/modules/search/search.service.spec.ts@@ -1,7 +1,7 @@-import { NotImplementedException } from '@nestjs/common';+import { BadRequestException, NotImplementedException } from '@nestjs/common';import { SearchService } from './search.service';import { SearchProviderRegistry } from './search-provider.registry';-import { SEARCH_DEFAULT_LIMIT, SEARCH_LIMIT_MAX, SEARCH_OFFSET_MAX } from './search.constants';+import { SEARCH_DEFAULT_LIMIT, SEARCH_LIMIT_MAX, SEARCH_OFFSET_MAX, SEARCH_TERMS_MAX } from './search.constants';import type { SearchProvider, SearchResults } from './search.types';function mkProvider(id: string, search: SearchProvider['search']): SearchProvider {@@ -51,6 +51,22 @@ describe('SearchService', () => {expect(search).toHaveBeenCalledWith(expect.objectContaining({ chatIds: ['[email protected]', '111@lid'] }));});+ it('refuses a query of more than SEARCH_TERMS_MAX distinct words before any provider runs it', async () => {+ const reg = new SearchProviderRegistry();+ const search = jest.fn().mockResolvedValue(emptyResults);+ reg.register(mkProvider('builtin-fts', search));+ const svc = new SearchService(reg);+ const words = (n: number): string => Array.from({ length: n }, (_, i) => `w${i}`).join(' ');++ await expect(svc.search({ q: words(SEARCH_TERMS_MAX + 1) })).rejects.toBeInstanceOf(BadRequestException);+ expect(search).not.toHaveBeenCalled();++ await svc.search({ q: words(SEARCH_TERMS_MAX) });+ // A word said many times is one word.+ await svc.search({ q: Array.from({ length: 300 }, () => 'hello').join(' ') });+ expect(search).toHaveBeenCalledTimes(2);+ });+it('clamps an excessive limit to SEARCH_LIMIT_MAX before it reaches any provider', async () => {// Without the host-side clamp a plugin provider would receive the raw caller value (9999) and// could pressure host heap by returning an oversized page. The built-in already rejected this indiff --git a/src/modules/search/search.service.ts b/src/modules/search/search.service.tsindex e6631bf19..a7be40bff 100644--- a/src/modules/search/search.service.ts+++ b/src/modules/search/search.service.ts@@ -1,6 +1,6 @@-import { Injectable, NotImplementedException } from '@nestjs/common';+import { BadRequestException, Injectable, NotImplementedException } from '@nestjs/common';import { SearchProviderRegistry } from './search-provider.registry';-import { SEARCH_DEFAULT_LIMIT, SEARCH_LIMIT_MAX, SEARCH_OFFSET_MAX } from './search.constants';+import { SEARCH_DEFAULT_LIMIT, SEARCH_LIMIT_MAX, SEARCH_OFFSET_MAX, SEARCH_TERMS_MAX } from './search.constants';import type { SearchQuery, SearchResults } from './search.types';@Injectable()@@ -10,6 +10,11 @@ export class SearchService {async search(query: SearchQuery, callerSessionIds?: string[], callerChatIds?: string[]): Promise<SearchResults> {const provider = this.registry.active();if (!provider) throw new NotImplementedException('Search is not configured (no active search provider).');+ // Bounded here, ahead of every provider: the cost of a query grows with its words, whatever backend+ // runs it. A repeated word is one word.+ if (new Set(query.q.trim().split(/\s+/)).size > SEARCH_TERMS_MAX) {+ throw new BadRequestException(`q may hold at most ${SEARCH_TERMS_MAX} distinct words.`);+ }// Auth scoping is authoritative — a caller cannot override it via the query. This covers both// fences: sessions (allowedSessions) and chats (allowedChats, expanded lid-aware by the caller),// each undefined for an unrestricted key. Bound pagination host-side so EVERY provider