Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(search): stop counting vowel signs and joiners as words
Yudhi Armyndharisa51d239644df
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 436ca3292..f212dbfca 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -134,7 +134,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Drop a Baileys edit, revoke or reaction that the stored message proves foreign (another chat, or a one-to-one change by the other party), also when the engine's own message store no longer holds the original.- Name who reacted in a Baileys 1:1 reaction, whose sender id came out empty so the two parties overwrote each other's reaction, and flag a 1:1 message from a lid-only sender as `isLidSender`.- Read the duration of an outgoing Baileys audio file in a separate process and a voice note's waveform through ffmpeg, status voice notes included, so a crafted file can no longer abort or exhaust the gateway.-- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator and each symbol or emoji as a word, and run a whitespace-separated word said again once, so one request can no longer hold the event loop on SQLite for seconds.+- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator and a symbol or emoji that stands apart as a word, and run a whitespace-separated word said again once, so one request can no longer hold the event loop on SQLite for seconds.- Print the startup banner's API Docs URL only when Swagger is served, so a production boot no longer points at a `404`.- Decode numeric and common named character references in a Baileys link preview's title and description, so `’` and `é` no longer reach the recipient as written.- Log an error at whatsapp-web.js start when the message-id backport is only partly applied, naming the files it did not reach, and have the patcher refuse such a tree, whichever file the interrupted install stopped at, instead of reporting version skew and leaving `.rej` files.@@ -195,7 +195,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- With `OPENWA_DATA_DIR` set, `backup.sh` and `restore.sh` read the default SQLite databases and `./data/...` paths in `./.env` under it; pass a path in the environment to keep it in the working directory.- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.- ⚠️ **Breaking (API).** Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working on the REST routes, but over `/events` a subscribe that would hold more than 4096 rooms (the subscribed events times the key's chat ids, two per phone number) is refused with `TOO_MANY_SUBSCRIPTIONS`, so shorten it.-- ⚠️ **Breaking (API).** `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words and each symbol or emoji is a word); a whitespace-separated word said again counts once.+- ⚠️ **Breaking (API).** `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words and a symbol or emoji that stands apart is a word); a whitespace-separated word said again counts once.- On SQLite, `GET /api/search` leaves `score` out of the hits of a key restricted to sessions or chats, because it is computed over the whole index; the hits stay in relevance order.- Without an ffmpeg binary (`FFMPEG_PATH`), a Baileys voice note goes out with flat waveform bars instead of its real ones; the official image ships ffmpeg.- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.diff --git a/docs/26-global-search.md b/docs/26-global-search.mdindex 9ab27d083..399d4a3e5 100644--- a/docs/26-global-search.md+++ b/docs/26-global-search.md@@ -134,7 +134,7 @@ GET /api/search?q=<term>&sessionId=<id>&chatId=<id>&direction=<incoming|outgoing```- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one- longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and each symbol or emoji counting as one; a whitespace-separated word said again counts once). Numeric params+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and a symbol or emoji that stands apart counting as one; a whitespace-separated word said again counts once). Numeric paramsare coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,never as a `NaN` SQL parameter.- **Auth scoping is authoritative.** The caller's API-key `allowedSessions` is injected bydiff --git a/src/modules/search/search-terms.spec.ts b/src/modules/search/search-terms.spec.tsindex b07264e7e..6c9d28f5a 100644--- a/src/modules/search/search-terms.spec.ts+++ b/src/modules/search/search-terms.spec.ts@@ -27,6 +27,17 @@ describe('searchTermCount', () => {expect(searchTermCount('こんにちは')).toBe(1);});+ it('keeps an ordinary sentence of a script with vowel signs and joiners under the 32-word limit', () => {+ // Combining marks and joiners between letters add no word, as the index splits on them too.+ expect(searchTermCount('मैं आज शाम को बाज़ार जाकर कुछ ताज़ी सब्ज़ियाँ और फल खरीदूँगा')).toBeLessThanOrEqual(32);+ expect(searchTermCount('نمی\u200Cخواهم')).toBe(2);+ expect(searchTermCount('క్రమం తెలుగు')).toBeLessThanOrEqual(8);+ // A mark or a symbol that stands alone, or a run of them, still counts.+ expect(searchTermCount(chars(0x0301))).toBe(1);+ expect(searchTermCount(chars(0x1f923).repeat(10))).toBe(5);+ expect(searchTermCount(`${chars(0x1f923)}${chars(0xb7)}`.repeat(10))).toBe(10);+ });+it('counts a character the index may split on, or glue, one by one', () => {// A letter its tokenizer still splits on, a private-use character and an emoji its Unicode table// predates: each is a word of its own between separators or next to a letter.@@ -76,5 +87,87 @@ describe('searchTermCount', () => {it('never counts fewer words than the index finds around a character it treats as a separator', async () => {expect(await undercounted('split_tokens', c => `a${c}a`)).toEqual([]);}, 60_000);++ // Strings of every kind of character at once, because a stretch of them is what the count has to+ // bound: a glued mark between letters joins them, a separator next to it does not, and the+ // two alternate. The same words are not run twice, as the count does not run them.+ it('never counts fewer words than the index finds in a string mixing every kind of character', async () => {+ await ds.query('CREATE VIRTUAL TABLE mixed USING fts5(body)');+ await ds.query("CREATE VIRTUAL TABLE mixed_tokens USING fts5vocab(mixed, 'instance')");+ const pieces = [+ 'a',+ 'b',+ 'k',+ '7',+ chars(0xe9),+ chars(0x915),+ '.',+ '-',+ "'",+ ' ',+ ' ',+ '/',+ chars(0x93e),+ chars(0x301),+ chars(0x902),+ chars(0x200c),+ chars(0x200d),+ chars(0xb7),+ chars(0x2014),+ chars(0x2026),+ chars(0x201c),+ chars(0x1e4ee),+ chars(0x11300),+ chars(0x1abf),+ chars(0x1f923),+ chars(0x1f3fb),+ chars(0xe000),+ chars(0x1cf2),+ chars(0x19b0),+ ];+ let seed = 20241014;+ const next = (limit: number): number => {+ seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0;+ return seed % limit;+ };+ const samples: string[] = [];+ for (let i = 0; i < 400; i++) {+ const raw = Array.from({ length: 1 + next(40) }, () => pieces[next(pieces.length)]).join('');+ samples.push([...new Set(raw.trim().split(/\s+/))].join(' '));+ }+ for (const [i, text] of samples.entries()) {+ await ds.query('INSERT INTO mixed (rowid, body) VALUES (?, ?)', [i + 1, text]);+ }+ const found = await ds.query<{ doc: number; n: number }[]>(+ 'SELECT doc, count(*) AS n FROM mixed_tokens GROUP BY doc',+ );+ const tokens = new Map(found.map(row => [row.doc, row.n]));+ const undercounts = samples+ .map((text, i) => ({ text, i, n: tokens.get(i + 1) ?? 0 }))+ .filter(({ text, n }) => text.trim() !== '' && searchTermCount(text) < n)+ .map(({ text, n }) => `${JSON.stringify(text)} counted ${searchTermCount(text)}, index finds ${n}`);+ expect(undercounts).toEqual([]);+ }, 60_000);++ it('counts a sentence of marked letters as the index splits it, not once more for every mark', async () => {+ const sentences = [+ 'मैं आज शाम को बाज़ार जाकर कुछ ताज़ी सब्ज़ियाँ और फल खरीदूँगा',+ 'నేను ఈ రోజు సాయంత్రం మార్కెట్కు వెళ్లి తాజా కూరగాయలు కొంటాను',+ ];+ await ds.query('CREATE VIRTUAL TABLE sentences USING fts5(body)');+ await ds.query("CREATE VIRTUAL TABLE sentences_tokens USING fts5vocab(sentences, 'instance')");+ for (const [i, text] of sentences.entries()) {+ await ds.query('INSERT INTO sentences (rowid, body) VALUES (?, ?)', [i + 1, text]);+ }+ const found = await ds.query<{ doc: number; n: number }[]>(+ 'SELECT doc, count(*) AS n FROM sentences_tokens GROUP BY doc',+ );+ for (const [i, text] of sentences.entries()) {+ const n = found.find(row => row.doc === i + 1)?.n ?? 0;+ expect(searchTermCount(text)).toBeGreaterThanOrEqual(n);+ // Close to what the index finds, and well under the limit the route enforces.+ expect(searchTermCount(text)).toBeLessThanOrEqual(Math.ceil(n * 1.3));+ }+ }, 60_000);});});diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.tsindex bf9738cde..9cd07cf62 100644--- a/src/modules/search/search-terms.ts+++ b/src/modules/search/search-terms.ts@@ -1,25 +1,65 @@/**- * What counts as a word, as an upper bound on how the built-in index splits a query (SQLite FTS5,- * unicode61): a run of letters or digits is one word, and so is every other character outside ASCII,- * since the index may treat it as part of a word (private use, a code point its Unicode table does not- * know, among them recent emoji) or as a separator. Only ASCII punctuation is known to separate.+ * What the built-in index (SQLite FTS5, unicode61) makes of a query, as an upper bound on its words. A+ * run of letters or digits is one word. ASCII punctuation separates. Any other character outside ASCII+ * may separate or be part of a word, depending on what the index's own Unicode table knows of it: the+ * ones it knows as marks, punctuation or symbols separate, and private use, a letter it does not split+ * on and every code point that table predates (among them recent emoji and marks) are parts of words.*- * The 21 letters its tokenizer still splits on are left out of the run, so they count one by one.- * search-terms.spec.ts compares every code point with the bundled SQLite, so an upgrade that moves one- * fails there instead of silently loosening the bound.+ * The 21 letters its tokenizer still splits on are left out of the run, so they count among the other+ * characters. search-terms.spec.ts compares every code point, and strings of them, with the bundled+ * SQLite, so an upgrade that moves one fails there instead of silently loosening the bound.*/-const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+|[^\s\p{ASCII}]/gu;+const RUN_CHAR = /^(?![ᦰ-ᧀᧈᧉᳲᳳ])[\p{L}\p{N}]$/u;++/**+ * The words of one whitespace-free string, at most: each run of letters or digits is one, and so are+ * the characters outside ASCII that are neither, which the index may take as words of their own.+ *+ * A stretch of m such characters adds at most ceil(m / 2) words, alternating one the index glues with+ * one it separates on. Next to a run of letters it adds fewer: a glued character there joins that+ * word, so at most floor(m / 2) words with one neighbour and floor((m - 1) / 2) with two. A vowel sign+ * or a joiner between letters therefore adds none, and Devanagari or Persian text counts the words the+ * index finds in it.+ */+function wordsOf(word: string): number {+ let words = 0;+ let inRun = false;+ let stretch = 0;+ let afterRun = false;+ const close = (beforeRun: boolean): void => {+ if (stretch === 0) return;+ const neighbours = (afterRun ? 1 : 0) + (beforeRun ? 1 : 0);+ words += neighbours === 0 ? Math.ceil(stretch / 2) : Math.floor((stretch - (neighbours === 2 ? 1 : 0)) / 2);+ stretch = 0;+ };+ for (const char of word) {+ if (RUN_CHAR.test(char)) {+ close(true);+ if (!inRun) words += 1;+ inRun = true;+ } else if ((char.codePointAt(0) ?? 0) > 0x7f) {+ if (stretch === 0) afterRun = inRun;+ stretch += 1;+ inRun = false;+ } else {+ close(false);+ inRun = false;+ }+ }+ close(false);+ return words;+}/*** How many words a query holds: each run of letters or digits is one, so punctuation separates words as- * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and- * a symbol or emoji counts as a word of its own. A whitespace-separated word said again is run once and- * counts once.+ * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and a+ * symbol or emoji that stands apart counts as a word of its own. A whitespace-separated word said again+ * is run once and counts once, and a word with no letter, digit or symbol counts as one.*/export function searchTermCount(q: string): number {let count = 0;for (const word of new Set(q.trim().split(/\s+/))) {- count += word.match(WORD)?.length ?? 1;+ count += wordsOf(word) || 1;}return count;}