Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(search): stop counting vowel signs and joiners as words

Yudhi Armyndharisa51d239644df
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 436ca3292..f212dbfca 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -134,7 +134,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Drop a Baileys edit, revoke or reaction that the stored message proves foreign (another chat, or a one-to-one change by the other party), also when the engine's own message store no longer holds the original.
- Name who reacted in a Baileys 1:1 reaction, whose sender id came out empty so the two parties overwrote each other's reaction, and flag a 1:1 message from a lid-only sender as `isLidSender`.
- Read the duration of an outgoing Baileys audio file in a separate process and a voice note's waveform through ffmpeg, status voice notes included, so a crafted file can no longer abort or exhaust the gateway.
-- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator and each symbol or emoji as a word, and run a whitespace-separated word said again once, so one request can no longer hold the event loop on SQLite for seconds.
+- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator and a symbol or emoji that stands apart as a word, and run a whitespace-separated word said again once, so one request can no longer hold the event loop on SQLite for seconds.
- Print the startup banner's API Docs URL only when Swagger is served, so a production boot no longer points at a `404`.
- Decode numeric and common named character references in a Baileys link preview's title and description, so `’` and `é` no longer reach the recipient as written.
- Log an error at whatsapp-web.js start when the message-id backport is only partly applied, naming the files it did not reach, and have the patcher refuse such a tree, whichever file the interrupted install stopped at, instead of reporting version skew and leaving `.rej` files.
@@ -195,7 +195,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- With `OPENWA_DATA_DIR` set, `backup.sh` and `restore.sh` read the default SQLite databases and `./data/...` paths in `./.env` under it; pass a path in the environment to keep it in the working directory.
- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.
- ⚠️ **Breaking (API).** Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working on the REST routes, but over `/events` a subscribe that would hold more than 4096 rooms (the subscribed events times the key's chat ids, two per phone number) is refused with `TOO_MANY_SUBSCRIPTIONS`, so shorten it.
-- ⚠️ **Breaking (API).** `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words and each symbol or emoji is a word); a whitespace-separated word said again counts once.
+- ⚠️ **Breaking (API).** `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words and a symbol or emoji that stands apart is a word); a whitespace-separated word said again counts once.
- On SQLite, `GET /api/search` leaves `score` out of the hits of a key restricted to sessions or chats, because it is computed over the whole index; the hits stay in relevance order.
- Without an ffmpeg binary (`FFMPEG_PATH`), a Baileys voice note goes out with flat waveform bars instead of its real ones; the official image ships ffmpeg.
- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.
diff --git a/docs/26-global-search.md b/docs/26-global-search.md
index 9ab27d083..399d4a3e5 100644
--- a/docs/26-global-search.md
+++ b/docs/26-global-search.md
@@ -134,7 +134,7 @@ GET /api/search?q=<term>&sessionId=<id>&chatId=<id>&direction=<incoming|outgoing
```
- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one
- longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and each symbol or emoji counting as one; a whitespace-separated word said again counts once). Numeric params
+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and a symbol or emoji that stands apart counting as one; a whitespace-separated word said again counts once). Numeric params
are coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,
never as a `NaN` SQL parameter.
- **Auth scoping is authoritative.** The caller's API-key `allowedSessions` is injected by
diff --git a/src/modules/search/search-terms.spec.ts b/src/modules/search/search-terms.spec.ts
index b07264e7e..6c9d28f5a 100644
--- a/src/modules/search/search-terms.spec.ts
+++ b/src/modules/search/search-terms.spec.ts
@@ -27,6 +27,17 @@ describe('searchTermCount', () => {
expect(searchTermCount('こんにちは')).toBe(1);
});
+ it('keeps an ordinary sentence of a script with vowel signs and joiners under the 32-word limit', () => {
+ // Combining marks and joiners between letters add no word, as the index splits on them too.
+ expect(searchTermCount('मैं आज शाम को बाज़ार जाकर कुछ ताज़ी सब्ज़ियाँ और फल खरीदूँगा')).toBeLessThanOrEqual(32);
+ expect(searchTermCount('نمی\u200Cخواهم')).toBe(2);
+ expect(searchTermCount('క్రమం తెలుగు')).toBeLessThanOrEqual(8);
+ // A mark or a symbol that stands alone, or a run of them, still counts.
+ expect(searchTermCount(chars(0x0301))).toBe(1);
+ expect(searchTermCount(chars(0x1f923).repeat(10))).toBe(5);
+ expect(searchTermCount(`${chars(0x1f923)}${chars(0xb7)}`.repeat(10))).toBe(10);
+ });
+
it('counts a character the index may split on, or glue, one by one', () => {
// A letter its tokenizer still splits on, a private-use character and an emoji its Unicode table
// predates: each is a word of its own between separators or next to a letter.
@@ -76,5 +87,87 @@ describe('searchTermCount', () => {
it('never counts fewer words than the index finds around a character it treats as a separator', async () => {
expect(await undercounted('split_tokens', c => `a${c}a`)).toEqual([]);
}, 60_000);
+
+ // Strings of every kind of character at once, because a stretch of them is what the count has to
+ // bound: a glued mark between letters joins them, a separator next to it does not, and the
+ // two alternate. The same words are not run twice, as the count does not run them.
+ it('never counts fewer words than the index finds in a string mixing every kind of character', async () => {
+ await ds.query('CREATE VIRTUAL TABLE mixed USING fts5(body)');
+ await ds.query("CREATE VIRTUAL TABLE mixed_tokens USING fts5vocab(mixed, 'instance')");
+ const pieces = [
+ 'a',
+ 'b',
+ 'k',
+ '7',
+ chars(0xe9),
+ chars(0x915),
+ '.',
+ '-',
+ "'",
+ ' ',
+ ' ',
+ '/',
+ chars(0x93e),
+ chars(0x301),
+ chars(0x902),
+ chars(0x200c),
+ chars(0x200d),
+ chars(0xb7),
+ chars(0x2014),
+ chars(0x2026),
+ chars(0x201c),
+ chars(0x1e4ee),
+ chars(0x11300),
+ chars(0x1abf),
+ chars(0x1f923),
+ chars(0x1f3fb),
+ chars(0xe000),
+ chars(0x1cf2),
+ chars(0x19b0),
+ ];
+ let seed = 20241014;
+ const next = (limit: number): number => {
+ seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0;
+ return seed % limit;
+ };
+ const samples: string[] = [];
+ for (let i = 0; i < 400; i++) {
+ const raw = Array.from({ length: 1 + next(40) }, () => pieces[next(pieces.length)]).join('');
+ samples.push([...new Set(raw.trim().split(/\s+/))].join(' '));
+ }
+ for (const [i, text] of samples.entries()) {
+ await ds.query('INSERT INTO mixed (rowid, body) VALUES (?, ?)', [i + 1, text]);
+ }
+ const found = await ds.query<{ doc: number; n: number }[]>(
+ 'SELECT doc, count(*) AS n FROM mixed_tokens GROUP BY doc',
+ );
+ const tokens = new Map(found.map(row => [row.doc, row.n]));
+ const undercounts = samples
+ .map((text, i) => ({ text, i, n: tokens.get(i + 1) ?? 0 }))
+ .filter(({ text, n }) => text.trim() !== '' && searchTermCount(text) < n)
+ .map(({ text, n }) => `${JSON.stringify(text)} counted ${searchTermCount(text)}, index finds ${n}`);
+ expect(undercounts).toEqual([]);
+ }, 60_000);
+
+ it('counts a sentence of marked letters as the index splits it, not once more for every mark', async () => {
+ const sentences = [
+ 'मैं आज शाम को बाज़ार जाकर कुछ ताज़ी सब्ज़ियाँ और फल खरीदूँगा',
+ 'నేను ఈ రోజు సాయంత్రం మార్కెట్‌కు వెళ్లి తాజా కూరగాయలు కొంటాను',
+ ];
+ await ds.query('CREATE VIRTUAL TABLE sentences USING fts5(body)');
+ await ds.query("CREATE VIRTUAL TABLE sentences_tokens USING fts5vocab(sentences, 'instance')");
+ for (const [i, text] of sentences.entries()) {
+ await ds.query('INSERT INTO sentences (rowid, body) VALUES (?, ?)', [i + 1, text]);
+ }
+ const found = await ds.query<{ doc: number; n: number }[]>(
+ 'SELECT doc, count(*) AS n FROM sentences_tokens GROUP BY doc',
+ );
+ for (const [i, text] of sentences.entries()) {
+ const n = found.find(row => row.doc === i + 1)?.n ?? 0;
+ expect(searchTermCount(text)).toBeGreaterThanOrEqual(n);
+ // Close to what the index finds, and well under the limit the route enforces.
+ expect(searchTermCount(text)).toBeLessThanOrEqual(Math.ceil(n * 1.3));
+ }
+ }, 60_000);
});
});
diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.ts
index bf9738cde..9cd07cf62 100644
--- a/src/modules/search/search-terms.ts
+++ b/src/modules/search/search-terms.ts
@@ -1,25 +1,65 @@
/**
- * What counts as a word, as an upper bound on how the built-in index splits a query (SQLite FTS5,
- * unicode61): a run of letters or digits is one word, and so is every other character outside ASCII,
- * since the index may treat it as part of a word (private use, a code point its Unicode table does not
- * know, among them recent emoji) or as a separator. Only ASCII punctuation is known to separate.
+ * What the built-in index (SQLite FTS5, unicode61) makes of a query, as an upper bound on its words. A
+ * run of letters or digits is one word. ASCII punctuation separates. Any other character outside ASCII
+ * may separate or be part of a word, depending on what the index's own Unicode table knows of it: the
+ * ones it knows as marks, punctuation or symbols separate, and private use, a letter it does not split
+ * on and every code point that table predates (among them recent emoji and marks) are parts of words.
*
- * The 21 letters its tokenizer still splits on are left out of the run, so they count one by one.
- * search-terms.spec.ts compares every code point with the bundled SQLite, so an upgrade that moves one
- * fails there instead of silently loosening the bound.
+ * The 21 letters its tokenizer still splits on are left out of the run, so they count among the other
+ * characters. search-terms.spec.ts compares every code point, and strings of them, with the bundled
+ * SQLite, so an upgrade that moves one fails there instead of silently loosening the bound.
*/
-const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+|[^\s\p{ASCII}]/gu;
+const RUN_CHAR = /^(?![ᦰ-ᧀᧈᧉᳲᳳ])[\p{L}\p{N}]$/u;
+
+/**
+ * The words of one whitespace-free string, at most: each run of letters or digits is one, and so are
+ * the characters outside ASCII that are neither, which the index may take as words of their own.
+ *
+ * A stretch of m such characters adds at most ceil(m / 2) words, alternating one the index glues with
+ * one it separates on. Next to a run of letters it adds fewer: a glued character there joins that
+ * word, so at most floor(m / 2) words with one neighbour and floor((m - 1) / 2) with two. A vowel sign
+ * or a joiner between letters therefore adds none, and Devanagari or Persian text counts the words the
+ * index finds in it.
+ */
+function wordsOf(word: string): number {
+ let words = 0;
+ let inRun = false;
+ let stretch = 0;
+ let afterRun = false;
+ const close = (beforeRun: boolean): void => {
+ if (stretch === 0) return;
+ const neighbours = (afterRun ? 1 : 0) + (beforeRun ? 1 : 0);
+ words += neighbours === 0 ? Math.ceil(stretch / 2) : Math.floor((stretch - (neighbours === 2 ? 1 : 0)) / 2);
+ stretch = 0;
+ };
+ for (const char of word) {
+ if (RUN_CHAR.test(char)) {
+ close(true);
+ if (!inRun) words += 1;
+ inRun = true;
+ } else if ((char.codePointAt(0) ?? 0) > 0x7f) {
+ if (stretch === 0) afterRun = inRun;
+ stretch += 1;
+ inRun = false;
+ } else {
+ close(false);
+ inRun = false;
+ }
+ }
+ close(false);
+ return words;
+}
/**
* How many words a query holds: each run of letters or digits is one, so punctuation separates words as
- * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and
- * a symbol or emoji counts as a word of its own. A whitespace-separated word said again is run once and
- * counts once.
+ * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and a
+ * symbol or emoji that stands apart counts as a word of its own. A whitespace-separated word said again
+ * is run once and counts once, and a word with no letter, digit or symbol counts as one.
*/
export function searchTermCount(q: string): number {
let count = 0;
for (const word of new Set(q.trim().split(/\s+/))) {
- count += word.match(WORD)?.length ?? 1;
+ count += wordsOf(word) || 1;
}
return count;
}