Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(search): leave score out for session-scoped keys

Yudhi Armyndharis0f4bcd54628f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9fd2d82c6..498c40074 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -163,6 +163,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A storage import stops at `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB); raise it before importing a larger export.
- Creating or updating an API key with more than 2000 `allowedChats` entries answers `400`; a stored longer list keeps working on the REST routes, but over `/events` a subscribe that would hold more than 4096 rooms (the subscribed events times the key's chat ids, two per phone number) is refused with `TOO_MANY_SUBSCRIPTIONS`, so shorten it.
- `GET /api/search` answers `400` for a `q` over 1000 characters or with more than 32 words (punctuation separates words); a repeated word counts once.
+- On SQLite, `GET /api/search` leaves `score` out of the hits of a key restricted to sessions or chats, because it is computed over the whole index; the hits stay in relevance order.
- Without an ffmpeg binary (`FFMPEG_PATH`), a Baileys voice note goes out with flat waveform bars instead of its real ones; the official image ships ffmpeg.
- With `DATABASE_SSL=true`, the PostgreSQL certificate is now verified against `DATABASE_HOST` even when it is an IP address (`pg` 8.23.1). A certificate that names only `localhost` stops matching a connection by IP: use the certificate's DNS name as the host or add the IP to its subject alternative names. `DATABASE_SSL_REJECT_UNAUTHORIZED=false` remains the escape hatch.
- The first boot builds three indexes on `messages` (`IDX_messages_session_timestamp_id`, `IDX_messages_session_chat_timestamp_id` and `IDX_messages_session_direction_timestamp_id`) not concurrently, and the table with its indexes grows by about 240 MB per million messages. On PostgreSQL writes to that table wait until the boot migrations commit, after all three builds, and the Helm chart's startup probe (about 295 s) restarts a pod whose boot outlasts it, so on a large table run the migrations first (migration guide, section 14.5).
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 6d758ca08..817937d32 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -6546,7 +6546,7 @@ Search messages across sessions (active search provider).
id, so it changes when a plugin backend is selected via `SEARCH_PROVIDER`.
- `score` is optional and provider-specific (rank ordering is stable within a provider; cross-provider
scores are not comparable). The built-in provider on SQLite leaves it out for a key restricted to
- selected chats, since it is computed over the whole index.
+ selected sessions or chats, since it is computed over the whole index.
**Errors:** `400` empty/whitespace `q`, a non-numeric `dateFrom`/`dateTo`/`limit`/`offset`, an `offset` above `100000`, or a malformed
SQLite FTS5 query (unbalanced quote/paren, bare operator) — Postgres's `websearch_to_tsquery` is
diff --git a/docs/26-global-search.md b/docs/26-global-search.md
index 4b39ca0ae..767853127 100644
--- a/docs/26-global-search.md
+++ b/docs/26-global-search.md
@@ -145,9 +145,9 @@ GET /api/search?q=<term>&sessionId=<id>&chatId=<id>&direction=<incoming|outgoing
- **Chat scope applies the same way.** A key with a non-empty `allowedChats` searches only those
chats: the built-in provider narrows its hits before `limit`, `offset` and `total` apply, an explicit
`chatId` outside the allowlist answers `403`, and a plugin provider answers `403` for such a key
- (set `SEARCH_PROVIDER=builtin-fts` to allow scoped search). On SQLite such a key's hits carry no
- `score`, because the value is computed over the whole index and would reflect messages outside the
- allowed chats; the hits stay in relevance order.
+ (set `SEARCH_PROVIDER=builtin-fts` to allow scoped search). On SQLite the hits of a key scoped to
+ sessions or chats carry no `score`, because the value is computed over the whole index and would
+ reflect messages outside its scope; the hits stay in relevance order.
- **Response** is a `SearchResults` object: `{ hits: SearchHit[], total, tookMs, provider }`. Each hit
carries `messageId`, `waMessageId`, `sessionId`, `chatId`, `body`, `snippet`, `timestamp`, `type`,
`direction`, `from`, and optional `score`. `total` is an exact count (bounded; computed lazily only
diff --git a/src/modules/search/providers/builtin-fts.provider.spec.ts b/src/modules/search/providers/builtin-fts.provider.spec.ts
index 57372b9b2..9b95d6a87 100644
--- a/src/modules/search/providers/builtin-fts.provider.spec.ts
+++ b/src/modules/search/providers/builtin-fts.provider.spec.ts
@@ -162,18 +162,20 @@ describe('BuiltInFtsProvider (sqlite)', () => {
expect((await provider.search({ q: 'hello', chatIds: ['c2'], sessionIds: ['s1'] })).total).toBe(0);
});
- // bm25 weighs a term by how many messages the whole index holds, so the rank of a hit a chat-scoped
- // caller may read still reflects chats it may not. The hits stay ordered by it; the value is withheld.
- it('returns no score to a chat-scoped caller, and still returns one to the others', async () => {
+ // bm25 weighs a term by how many messages the whole index holds, so the rank of a hit a session- or
+ // chat-scoped caller may read still reflects sessions and chats it may not. The hits stay ordered by it;
+ // the value is withheld.
+ it('returns no score to a session- or chat-scoped caller, and still returns one to an unscoped key', async () => {
const unscoped = await provider.search({ q: 'hello' });
expect(unscoped.hits.map(hit => typeof hit.score)).toEqual(['number', 'number']);
- const scoped = await provider.search({ q: 'hello', chatIds: ['c2'] });
- expect(scoped.hits.map(hit => hit.chatId)).toEqual(['c2']);
- expect(scoped.hits[0].score).toBeUndefined();
+ const chats = await provider.search({ q: 'hello', chatIds: ['c2'] });
+ expect(chats.hits.map(hit => hit.chatId)).toEqual(['c2']);
+ expect(chats.hits[0].score).toBeUndefined();
const sessions = await provider.search({ q: 'hello', sessionIds: ['s1'] });
- expect(sessions.hits.map(hit => typeof hit.score)).toEqual(['number']);
+ expect(sessions.hits.map(hit => hit.sessionId)).toEqual(['s1']);
+ expect(sessions.hits[0].score).toBeUndefined();
});
it.each([0, 1])('returns no hits or count for an empty compiled chat scope at offset %s', async offset => {
diff --git a/src/modules/search/providers/builtin-fts.provider.ts b/src/modules/search/providers/builtin-fts.provider.ts
index 462e46013..b6cad93a1 100644
--- a/src/modules/search/providers/builtin-fts.provider.ts
+++ b/src/modules/search/providers/builtin-fts.provider.ts
@@ -336,9 +336,10 @@ export class BuiltInFtsProvider implements SearchProvider, OnModuleInit {
const where: string[] = [`messages_fts MATCH ${ph()}`];
params.push(BuiltInFtsProvider.toFts5Query(q.q));
this.applyFilters(where, params, q, 'm.', ph);
- // bm25 weighs a term by how many messages the whole index holds, so the value of a hit a chat-scoped
- // caller may read still reflects chats it may not. It keeps ordering the hits; only the value is withheld.
- const score = q.chatIds === undefined ? 'rank' : 'NULL';
+ // bm25 weighs a term by how many messages the whole index holds, so the value of a hit a caller
+ // scoped to some sessions or chats may read still reflects the ones it may not. It keeps ordering the
+ // hits; only the value is withheld.
+ const score = q.chatIds === undefined && q.sessionIds === undefined ? 'rank' : 'NULL';
const cols = `m."id", m."waMessageId" AS wa_message_id, m."sessionId" AS session_id, m."chatId" AS chat_id, m."from" AS "from", m."body", m."timestamp", m."type", m."direction", snippet(messages_fts, 0, '<mark>', '</mark>', '…', ${MAX_SNIPPET_WORDS}) AS snippet, ${score} AS score`;
const sql = `SELECT ${cols} FROM messages_fts JOIN messages m ON m."rowid" = messages_fts."rowid" WHERE ${where.join(' AND ')} ORDER BY rank, m."timestamp" DESC, m."id" DESC LIMIT ${ph()} OFFSET ${ph()}`;
params.push(limit, offset);