Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(search): count every non-ASCII symbol toward the word bound

Yudhi Armyndharis71f3fb056c6f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 59e502131..26222bbf4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -124,7 +124,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Drop a Baileys edit, revoke or reaction that the stored message proves foreign (another chat, or a one-to-one change by the other party), also when the engine's own message store no longer holds the original.
- Name who reacted in a Baileys 1:1 reaction, whose sender id came out empty so the two parties overwrote each other's reaction, and flag a 1:1 message from a lid-only sender as `isLidSender`.
- Read the duration of an outgoing Baileys audio file in a separate process and a voice note's waveform through ffmpeg, status voice notes included, so a crafted file can no longer abort or exhaust the gateway.
-- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator, and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.
+- Bound `GET /api/search` to 1000 characters and 32 words, punctuation counted as a separator and each symbol or emoji as a word, and run a repeated word once, so one request can no longer hold the event loop on SQLite for seconds.
- Print the startup banner's API Docs URL only when Swagger is served, so a production boot no longer points at a `404`.
- Decode numeric and common named character references in a Baileys link preview's title and description, so `’` and `é` no longer reach the recipient as written.
- Log an error at whatsapp-web.js start when the message-id backport is only partly applied, naming the files it did not reach.
@@ -164,7 +164,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- With `OPENWA_DATA_DIR` set, `backup.sh` and `restore.sh` read the default SQLite databases and `./data/...` paths in `./.env` under it; pass a path in the environment to keep it in the working directory.
- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.
- Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working on the REST routes, but over `/events` a subscribe that would hold more than 4096 rooms (the subscribed events times the key's chat ids, two per phone number) is refused with `TOO_MANY_SUBSCRIPTIONS`, so shorten it.
-- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words); a repeated word counts once.
+- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words and each symbol or emoji is a word); a repeated word counts once.
- On SQLite, `GET /api/search` leaves `score` out of the hits of a key restricted to sessions or chats, because it is computed over the whole index; the hits stay in relevance order.
- Without an ffmpeg binary (`FFMPEG_PATH`), a Baileys voice note goes out with flat waveform bars instead of its real ones; the official image ships ffmpeg.
- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 817937d32..4a9b26c74 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -6502,7 +6502,7 @@ Search messages across sessions (active search provider).
| Name | Type | Required | Default | Description |
| ----------- | ------------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words as whitespace does; a repeated word is counted once). Passed to the active provider's native full-text matcher. |
+| `q` | string | **Yes** | — | Search term. Must be non-empty after trim; `400` for whitespace-only, over 1000 characters, or over 32 words (punctuation separates words, each symbol is a word, a repeated word counts once). Passed to the active provider's native full-text matcher. |
| `sessionId` | string | No | — | Restrict to a single session id (intersected with the key's `allowedSessions` scope). |
| `chatId` | string | No | — | Restrict to a single chat id. For a key restricted to selected chats this must name an allowed chat (`403` otherwise), and its hits are additionally narrowed to the allowlist. |
| `direction` | enum (`incoming` \| `outgoing`) | No | — | Filter by message direction. |
diff --git a/docs/26-global-search.md b/docs/26-global-search.md
index 767853127..1b7bc1011 100644
--- a/docs/26-global-search.md
+++ b/docs/26-global-search.md
@@ -134,7 +134,7 @@ GET /api/search?q=<term>&sessionId=<id>&chatId=<id>&direction=<incoming|outgoing
```
- **`q`** is required and must be non-empty (whitespace-only is rejected with `400`, and so is one
- longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does; a repeated word counts once). Numeric params
+ longer than 1000 characters or with more than 32 words, punctuation separating words as whitespace does and each symbol or emoji counting as one; a repeated word counts once). Numeric params
are coerced and validated; a non-numeric `limit`/`offset`/`dateFrom`/`dateTo` surfaces as `400`,
never as a `NaN` SQL parameter.
- **Auth scoping is authoritative.** The caller's API-key `allowedSessions` is injected by
diff --git a/src/modules/search/search-terms.spec.ts b/src/modules/search/search-terms.spec.ts
index 4f8f0ecf5..b07264e7e 100644
--- a/src/modules/search/search-terms.spec.ts
+++ b/src/modules/search/search-terms.spec.ts
@@ -2,8 +2,10 @@ import 'reflect-metadata';
import { DataSource } from 'typeorm';
import { searchTermCount } from './search-terms';
+const chars = (...codePoints: number[]): string => String.fromCodePoint(...codePoints);
+
describe('searchTermCount', () => {
- it('counts a run of letters or digits as one word and punctuation as a separator', () => {
+ it('counts a run of letters or digits as one word and ASCII punctuation as a separator', () => {
expect(searchTermCount('hello')).toBe(1);
expect(searchTermCount('hello world')).toBe(2);
expect(searchTermCount('a.b.c')).toBe(3);
@@ -25,8 +27,13 @@ describe('searchTermCount', () => {
expect(searchTermCount('こんにちは')).toBe(1);
});
- it('counts a character the index splits on although it is a letter as a separator', () => {
- expect(searchTermCount('a\u1CF2'.repeat(40))).toBe(40);
+ it('counts a character the index may split on, or glue, one by one', () => {
+ // A letter its tokenizer still splits on, a private-use character and an emoji its Unicode table
+ // predates: each is a word of its own between separators or next to a letter.
+ for (const c of [chars(0x1cf2), chars(0xe000), chars(0x1f923), chars(0x1f3fb)]) {
+ expect(searchTermCount(`${c}.`.repeat(40))).toBeGreaterThanOrEqual(40);
+ expect(searchTermCount(`a${c}`.repeat(40))).toBeGreaterThanOrEqual(40);
+ }
});
describe('against the bundled SQLite', () => {
@@ -34,30 +41,40 @@ describe('searchTermCount', () => {
beforeAll(async () => {
ds = new DataSource({ type: 'better-sqlite3', database: ':memory:' });
await ds.initialize();
- await ds.query('CREATE VIRTUAL TABLE probe USING fts5(body)');
- });
+ await ds.query('CREATE VIRTUAL TABLE glue USING fts5(body)');
+ await ds.query("CREATE VIRTUAL TABLE glue_tokens USING fts5vocab(glue, 'instance')");
+ await ds.query('CREATE VIRTUAL TABLE split USING fts5(body)');
+ await ds.query("CREATE VIRTUAL TABLE split_tokens USING fts5vocab(split, 'instance')");
+ // Every code point but the surrogates, generated inside SQLite: c.c is two tokens when the index
+ // treats c as part of a word, and aca is two when it treats c as a separator.
+ const codePoints = 'WITH RECURSIVE n(c) AS (SELECT 1 UNION ALL SELECT c + 1 FROM n WHERE c < 1114111)';
+ await ds.query(
+ `${codePoints} INSERT INTO glue (rowid, body) SELECT c, char(c) || '.' || char(c) FROM n WHERE c NOT BETWEEN 55296 AND 57343`,
+ );
+ await ds.query(
+ `${codePoints} INSERT INTO split (rowid, body) SELECT c, 'a' || char(c) || 'a' FROM n WHERE c NOT BETWEEN 55296 AND 57343`,
+ );
+ }, 60_000);
afterAll(() => ds.destroy());
- it('never counts fewer words than the index splits a letter or digit into', async () => {
- const letters: number[] = [];
- for (let cp = 1; cp <= 0x10ffff; cp++) {
- if (cp >= 0xd800 && cp <= 0xdfff) continue;
- if (/[\p{L}\p{N}]/u.test(String.fromCodePoint(cp))) letters.push(cp);
- }
- for (let at = 0; at < letters.length; at += 5000) {
- const rows = letters
- .slice(at, at + 5000)
- .map(cp => `(${cp}, 'a${String.fromCodePoint(cp)}a')`)
- .join(',');
- await ds.query(`INSERT INTO probe (rowid, body) VALUES ${rows}`);
- }
- // `a<c>a` is two tokens exactly when the index treats <c> as a separator.
- const split = await ds.query<{ cp: number }[]>(`SELECT rowid AS cp FROM probe WHERE probe MATCH '"a a"'`);
- const undercounted = split
- .map(row => row.cp)
- .filter(cp => searchTermCount(`a${String.fromCodePoint(cp)}a`) < 2)
- .map(cp => cp.toString(16));
- expect(undercounted).toEqual([]);
- });
+ const undercounted = async (table: string, text: (c: string) => string): Promise<string[]> => {
+ const tokens = await ds.query<{ cp: number; n: number }[]>(
+ `SELECT doc AS cp, count(*) AS n FROM ${table} GROUP BY doc`,
+ );
+ // Whitespace never reaches the count inside a word: the query is split on it first.
+ const notSpace = (row: { cp: number }): boolean => !/\s/u.test(String.fromCodePoint(row.cp));
+ return tokens
+ .filter(notSpace)
+ .filter(row => searchTermCount(text(String.fromCodePoint(row.cp))) < row.n)
+ .map(row => row.cp.toString(16));
+ };
+
+ it('never counts fewer words than the index finds around a character it treats as part of a word', async () => {
+ expect(await undercounted('glue_tokens', c => `${c}.${c}`)).toEqual([]);
+ }, 60_000);
+
+ it('never counts fewer words than the index finds around a character it treats as a separator', async () => {
+ expect(await undercounted('split_tokens', c => `a${c}a`)).toEqual([]);
+ }, 60_000);
});
});
diff --git a/src/modules/search/search-terms.ts b/src/modules/search/search-terms.ts
index e0f760dd4..bf9738cde 100644
--- a/src/modules/search/search-terms.ts
+++ b/src/modules/search/search-terms.ts
@@ -1,15 +1,20 @@
/**
- * What a word is made of, as the built-in index sees it: letters and digits, less the 21 characters
- * its tokenizer (SQLite FTS5, unicode61) still splits on although current Unicode tables class them
- * as letters. search-terms.spec.ts compares every letter and digit with the bundled SQLite, so an
- * upgrade that moves one fails there instead of silently loosening the bound.
+ * What counts as a word, as an upper bound on how the built-in index splits a query (SQLite FTS5,
+ * unicode61): a run of letters or digits is one word, and so is every other character outside ASCII,
+ * since the index may treat it as part of a word (private use, a code point its Unicode table does not
+ * know, among them recent emoji) or as a separator. Only ASCII punctuation is known to separate.
+ *
+ * The 21 letters its tokenizer still splits on are left out of the run, so they count one by one.
+ * search-terms.spec.ts compares every code point with the bundled SQLite, so an upgrade that moves one
+ * fails there instead of silently loosening the bound.
*/
-const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+/gu;
+const WORD = /(?:(?![\u19B0-\u19C0\u19C8\u19C9\u1CF2\u1CF3])[\p{L}\p{N}])+|[^\s\p{ASCII}]/gu;
/**
- * How many words a query holds, counted as the index counts them: a run of letters or digits is one,
- * so punctuation separates words as whitespace does (`a.b.c` is three, and a quoted token of them is a
- * three-word phrase to match). A whitespace-separated word said again is run once and counts once.
+ * How many words a query holds: each run of letters or digits is one, so punctuation separates words as
+ * whitespace does (`a.b.c` is three, and a quoted token of them is a three-word phrase to match), and
+ * a symbol or emoji counts as a word of its own. A whitespace-separated word said again is run once and
+ * counts once.
*/
export function searchTermCount(q: string): number {
let count = 0;