Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(search): count every non-ASCII symbol toward the word bound
Yudhi Armyndharis71f3fb056c6f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 59e502131..26222bbf4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -124,7 +124,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Drop a Baileys edit, revoke or reaction that the stored message proves foreign (another chat, or a one-to-one change by the other party), also when the engine's own message store no longer holds the original.- Name who reacted in a Baileys 1:1 reaction, whose sender id came out empty so the two parties overwrote each other's reaction, and flag a 1:1 message from a lid-only sender as `isLidSender`.- Read the duration of an outgoing Baileys audio file in a separate process and a voice note's waveform through ffmpeg, status voice notes included, so a crafted file can no longer abort or exhaust the gateway.-- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator, and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.+- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator and each symbol or emoji as a word, and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.- Print the startup banner's API Docs URL only when Swagger is served, so a production boot no longer points at a `404`.- Decode numeric and common named character references in a Baileys link preview's title and description, so `’` and `é` no longer reach the recipient as written.- Log an error at whatsapp-web.js start when the message-id backport is only partly applied, naming the files it did not reach.@@ -164,7 +164,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- With `OPENWA_DATA_DIR` set, `backup.sh` and `restore.sh` read the default SQLite databases and `./data/...` paths in `./.env` under it; pass a path in the environment to keep it in the working directory.- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.- Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working on the REST routes, but over `/events` a subscribe that would hold more than 4096 rooms (the subscribed events times the key's chat ids, two per phone number) is refused with `TOO_MANY_SUBSCRIPTIONS`, so shorten it.-- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words); a repeated word counts once.+- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words and each symbol or emoji is a word); a repeated word counts once.- On SQLite, `GET /api/search` leaves `score` out of the hits of a key restricted to sessions or chats, because it is computed over the whole index; the hits stay in relevance order.- Without an ffmpeg binary (`FFMPEG_PATH`), a Baileys voice note goes out with flat waveform bars instead of its real ones; the official image ships ffmpeg.- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 817937d32..4a9b26c74 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6502,7 +6502,7 @@ Search messages across sessions (active search provider).| Name | Type | Required | Default | Description || ----------- | ------------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words as whitespace does; a repeated word is counted once). Passed to the active provider's native full-text matcher. |+| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words, each symbol is a word, a repeated word counts once). Passed to the active provider's native full-text matcher. || `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). || `chatId` | string | No | — | Restrict to a single chat id. For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are additionally narrowed to the allowlist. || `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |diff --git a/docs/26-global-search.md b/docs/26-global-search.mdindex 767853127..1b7bc1011 100644--- a/docs/26-global-search.md+++ b/docs/26-global-search.md@@ -134,7 +134,7 @@ GET /api/search?q=<term>&sessionId=<id>&chatId=<id>&direction=<incoming|outgoing```- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one- longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does; a repeated word counts once). Numeric params+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and each symbol or emoji counting as one; a repeated word counts once). Numeric paramsare coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,never as a `NaN` SQL parameter.- **Auth scoping is authoritative.** The caller's API-key `allowedSessions` is injected bydiff --git a/src/modules/search/search-terms.spec.ts b/src/modules/search/search-terms.spec.tsindex 4f8f0ecf5..b07264e7e 100644--- a/src/modules/search/search-terms.spec.ts+++ b/src/modules/search/search-terms.spec.ts@@ -2,8 +2,10 @@ import 'reflect-metadata';import { DataSource } from 'typeorm';import { searchTermCount } from './search-terms';+const chars = (...codePoints: number[]): string => String.fromCodePoint(...codePoints);+describe('searchTermCount', () => {- it('counts a run of letters or digits as one word and punctuation as a separator', () => {+ it('counts a run of letters or digits as one word and ASCII punctuation as a separator', () => {expect(searchTermCount('hello')).toBe(1);expect(searchTermCount('hello world')).toBe(2);expect(searchTermCount('a.b.c')).toBe(3);@@ -25,8 +27,13 @@ describe('searchTermCount', () => {expect(searchTermCount('こんにちは')).toBe(1);});- it('counts a character the index splits on although it is a letter as a separator', () => {- expect(searchTermCount('a\u1CF2'.repeat(40))).toBe(40);+ it('counts a character the index may split on, or glue, one by one', () => {+ // A letter its tokenizer still splits on, a private-use character and an emoji its Unicode table+ // predates: each is a word of its own between separators or next to a letter.+ for (const c of [chars(0x1cf2), chars(0xe000), chars(0x1f923), chars(0x1f3fb)]) {+ expect(searchTermCount(`${c}.`.repeat(40))).toBeGreaterThanOrEqual(40);+ expect(searchTermCount(`a${c}`.repeat(40))).toBeGreaterThanOrEqual(40);+ }});describe('against the bundled SQLite', () => {@@ -34,30 +41,40 @@ describe('searchTermCount', () => {beforeAll(async () => {ds = new DataSource({ type: 'better-sqlite3', database: ':memory:' });await ds.initialize();- await ds.query('CREATE VIRTUAL TABLE probe USING fts5(body)');- });+ await ds.query('CREATE VIRTUAL TABLE glue USING fts5(body)');+ await ds.query("CREATE VIRTUAL TABLE glue_tokens USING fts5vocab(glue, 'instance')");+ await ds.query('CREATE VIRTUAL TABLE split USING fts5(body)');+ await ds.query("CREATE VIRTUAL TABLE split_tokens USING fts5vocab(split, 'instance')");+ // Every code point but the surrogates, generated inside SQLite: c.c is two tokens when the index+ // treats c as part of a word, and aca is two when it treats c as a separator.+ const codePoints = 'WITH RECURSIVE n(c) AS (SELECT 1 UNION ALL SELECT c + 1 FROM n WHERE c < 1114111)';+ await ds.query(+ `${codePoints} INSERT INTO glue (rowid, body) SELECT c, char(c) || '.' || char(c) FROM n WHERE c NOT BETWEEN 55296 AND 57343`,+ );+ await ds.query(+ `${codePoints} INSERT INTO split (rowid, body) SELECT c, 'a' || char(c) || 'a' FROM n WHERE c NOT BETWEEN 55296 AND 57343`,+ );+ }, 60_000);afterAll(() => ds.destroy());- it('never counts fewer words than the index splits a letter or digit into', async () => {- const letters: number[] = [];- for (let cp = 1; cp <= 0x10ffff; cp++) {- if (cp >= 0xd800 && cp <= 0xdfff) continue;- if (/[\p{L}\p{N}]/u.test(String.fromCodePoint(cp))) letters.push(cp);- }- for (let at = 0; at < letters.length; at += 5000) {- const rows = letters- .slice(at, at + 5000)- .map(cp => `(${cp}, 'a${String.fromCodePoint(cp)}a')`)- .join(',');- await ds.query(`INSERT INTO probe (rowid, body) VALUES ${rows}`);- }- // `a<c>a` is two tokens exactly when the index treats <c> as a separator.- const split = await ds.query<{ cp: number }[]>(`SELECT rowid AS cp FROM probe WHERE probe MATCH '"a a"'`);- const undercounted = split- .map(row => row.cp)- .filter(cp => searchTermCount(`a${String.fromCodePoint(cp)}a`) < 2)- .map(cp => cp.toString(16));- expect(undercounted).toEqual([]);- });+ const undercounted = async (table: string, text: (c: string) => string): Promise<string[]> => {+ const tokens = await ds.query<{ cp: number; n: number }[]>(+ `SELECT doc AS cp, count(*) AS n FROM ${table} GROUP BY doc`,+ );+ // Whitespace never reaches the count inside a word: the query is split on it first.+ const notSpace = (row: { cp: number }): boolean => !/\s/u.test(String.fromCodePoint(row.cp));+ return tokens+ .filter(notSpace)+ .filter(row => searchTermCount(text(String.fromCodePoint(row.cp))) < row.n)+ .map(row => row.cp.toString(16));+ };++ it('never counts fewer words than the index finds around a character it treats as part of a word', async () => {+ expect(await undercounted('glue_tokens', c => `${c}.${c}`)).toEqual([]);+ }, 60_000);++ it('never counts fewer words than the index finds around a character it treats as a separator', async () => {+ expect(await undercounted('split_tokens', c => `a${c}a`)).toEqual([]);+ }, 60_000);});});diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.tsindex e0f760dd4..bf9738cde 100644--- a/src/modules/search/search-terms.ts+++ b/src/modules/search/search-terms.ts@@ -1,15 +1,20 @@/**- * What a word is made of, as the built-in index sees it: letters and digits, less the 21 characters- * its tokenizer (SQLite FTS5, unicode61) still splits on although current Unicode tables class them- * as letters. search-terms.spec.ts compares every letter and digit with the bundled SQLite, so an- * upgrade that moves one fails there instead of silently loosening the bound.+ * What counts as a word, as an upper bound on how the built-in index splits a query (SQLite FTS5,+ * unicode61): a run of letters or digits is one word, and so is every other character outside ASCII,+ * since the index may treat it as part of a word (private use, a code point its Unicode table does not+ * know, among them recent emoji) or as a separator. Only ASCII punctuation is known to separate.+ *+ * The 21 letters its tokenizer still splits on are left out of the run, so they count one by one.+ * search-terms.spec.ts compares every code point with the bundled SQLite, so an upgrade that moves one+ * fails there instead of silently loosening the bound.*/-const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+/gu;+const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+|[^\s\p{ASCII}]/gu;/**- * How many words a query holds, counted as the index counts them: a run of letters or digits is one,- * so punctuation separates words as whitespace does (`a.b.c` is three, and a quoted token of them is a- * three-word phrase to match). A whitespace-separated word said again is run once and counts once.+ * How many words a query holds: each run of letters or digits is one, so punctuation separates words as+ * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and+ * a symbol or emoji counts as a word of its own. A whitespace-separated word said again is run once and+ * counts once.*/export function searchTermCount(q: string): number {let count = 0;