Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

docs(search): describe the backfill a search plugin can do

Yudhi Armyndharis6ecc0ab75f8f
diff --git a/docs/27-plugin-search-providers.md b/docs/27-plugin-search-providers.md
index 7dc1901d8..bff10a8d9 100644
--- a/docs/27-plugin-search-providers.md
+++ b/docs/27-plugin-search-providers.md
@@ -128,9 +128,28 @@ the same handler keeps your index in step.
Rows removed in bulk emit no `message:deleted`: deleting a session and message retention (`MESSAGE_RETENTION_DAYS`) leave the plugin's copies in its index until the plugin removes them itself.
**Backfill is the plugin's responsibility.** The hook fires only for live traffic. A plugin installed on
-a deployment with existing message history must perform its own one-time backfill (read the `messages`
-table via `ctx.engine.getChatHistory` or a direct query, and index) at `onEnable`. The built-in DB-FTS
-provider is unaffected (its index is DB-synced via triggers on every insert, including backfill).
+a deployment with existing message history must perform its own one-time backfill. The only sanctioned
+path is `ctx.engine.getChats(sessionId)` followed by `ctx.engine.getChatHistory(sessionId, chatId, limit)`,
+which needs the `engine:read` permission and a session in the plugin's scope. It reads live from WhatsApp,
+not from the core `messages` table, and returns at most the 100 most recent messages per chat.
+
+`getChatHistory` exists only on the whatsapp-web.js engine. On Baileys it rejects with
+`EngineNotSupportedError` (see [29 - Engine Capability Matrix](./29-engine-capability-matrix.md)), so a
+Baileys deployment has no backfill path and a provider there indexes live `message:persisted` traffic
+only. Catch that error once and skip the backfill rather than retrying it chat after chat.
+
+There is no capability that reads the `messages` table. Reading the database directly bypasses the
+capability model and is unsupported; see [30 - Plugin Sandboxing](./30-plugin-sandboxing.md) for what a
+loaded plugin can still reach.
+
+Backfilled items carry only the WhatsApp id (`id` there is the WhatsApp message id), not the core row PK.
+Keep a `waMessageId` lookup too, and when a later `message:persisted` arrives for a message you already
+backfilled, upsert onto that document instead of adding a second one.
+
+Start the backfill from `onEnable` without awaiting it: a sandboxed lifecycle call is cut off after 30 s,
+and walking every chat's history takes longer on a real deployment. Record a marker in `ctx.storage` when
+it finishes so a restart does not repeat it. The built-in DB-FTS provider is unaffected (its index is
+DB-synced via triggers on every insert, including backfill).
## 27.4 Host-side guarantees (the plugin author doesn't handle these)
@@ -174,7 +193,7 @@ plugins/my-search/
```js
module.exports = class MySearchPlugin {
async onEnable(ctx) {
- // 1. Index every persisted message (live traffic only — backfill separately at onEnable).
+ // 1. Index every persisted message (live traffic only — backfill separately, see 27.3).
ctx.registerHook('message:persisted', async hookCtx => {
const { message } = hookCtx.data;
await this._index(ctx, message);
diff --git a/package.json b/package.json
index 251e00b37..46676b7d7 100644
--- a/package.json
+++ b/package.json
@@ -42,7 +42,7 @@
"lint:fix": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"test": "jest",
"test:scripts": "node --test scripts/postinstall.spec.js scripts/patch-wwebjs-newsletter-preview.spec.js scripts/patch-wwebjs-status.spec.js scripts/patch-wwebjs-ready-sync.spec.js scripts/patch-wwebjs-participant-arity.spec.js scripts/patch-wwebjs-block.spec.js scripts/patch-wwebjs-group-description.spec.js scripts/patch-wwebjs-media-id.spec.js scripts/patch-wwebjs-send-error.spec.js scripts/patch-baileys-appstate.spec.js scripts/patch-baileys-newsletter-create.spec.js scripts/dockerfile-patchers.spec.js scripts/check-upstream-surface.spec.mjs scripts/check-sdk-docs.spec.mjs scripts/check-audit.spec.mjs scripts/check-contract-shapes.spec.mjs scripts/cli-entrypoint-guard.spec.mjs",
- "test:docs": "jest src/common/changelog-sections.spec.ts src/common/ci-privilege-gate.spec.ts src/common/docs-ci-jobs.spec.ts src/common/docs-coverage-policy.spec.ts src/common/docs-metrics-list.spec.ts src/common/docs-next-version.spec.ts src/common/openapi/docs-06-contract.spec.ts src/common/test-lane-partition.spec.ts src/config/docs-env-example.spec.ts src/config/openapi-contract.spec.ts src/config/openapi-export-pins.spec.ts src/config/wa-web-version-example-pin.spec.ts src/engine/docs-29-counts.spec.ts src/engine/engine-inventory-parity.spec.ts src/engine/engine-parity.spec.ts src/engine/inventory-mapping-direction.spec.ts src/engine/adapters/baileys-voice-waveform.spec.ts src/engine/adapters/wwebjs-newsletter-preview.spec.ts src/modules/audit/audit-coverage.spec.ts src/modules/auth/global-route-fence-coverage.spec.ts src/modules/auth/query-session-scope-coverage.spec.ts src/modules/docker/compose-network.spec.ts src/modules/queue/bull-board-mount-coverage.spec.ts src/common/release-gate-parity.spec.ts src/database/migrations/__tests__/migration-drift.spec.ts --testPathIgnorePatterns=/node_modules/",
+ "test:docs": "jest src/common/changelog-sections.spec.ts src/common/ci-privilege-gate.spec.ts src/common/docs-ci-jobs.spec.ts src/common/docs-coverage-policy.spec.ts src/common/docs-metrics-list.spec.ts src/common/docs-next-version.spec.ts src/common/openapi/docs-06-contract.spec.ts src/common/test-lane-partition.spec.ts src/config/docs-env-example.spec.ts src/config/openapi-contract.spec.ts src/config/openapi-export-pins.spec.ts src/config/wa-web-version-example-pin.spec.ts src/engine/docs-29-counts.spec.ts src/engine/docs-27-backfill-engine.spec.ts src/engine/engine-inventory-parity.spec.ts src/engine/engine-parity.spec.ts src/engine/inventory-mapping-direction.spec.ts src/engine/adapters/baileys-voice-waveform.spec.ts src/engine/adapters/wwebjs-newsletter-preview.spec.ts src/modules/audit/audit-coverage.spec.ts src/modules/auth/global-route-fence-coverage.spec.ts src/modules/auth/query-session-scope-coverage.spec.ts src/modules/docker/compose-network.spec.ts src/modules/queue/bull-board-mount-coverage.spec.ts src/common/release-gate-parity.spec.ts src/database/migrations/__tests__/migration-drift.spec.ts --testPathIgnorePatterns=/node_modules/",
"test:watch": "jest --watch",
"test:cov": "jest --coverage",
"test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
@@ -163,6 +163,7 @@
"src/config/openapi-export-pins\\.spec\\.ts$",
"src/config/wa-web-version-example-pin\\.spec\\.ts$",
"src/engine/docs-29-counts\\.spec\\.ts$",
+ "src/engine/docs-27-backfill-engine\\.spec\\.ts$",
"src/engine/engine-inventory-parity\\.spec\\.ts$",
"src/engine/engine-parity\\.spec\\.ts$",
"src/engine/inventory-mapping-direction\\.spec\\.ts$",
@@ -484,6 +485,7 @@
"src/config/openapi-export-pins\\.spec\\.ts$",
"src/config/wa-web-version-example-pin\\.spec\\.ts$",
"src/engine/docs-29-counts\\.spec\\.ts$",
+ "src/engine/docs-27-backfill-engine\\.spec\\.ts$",
"src/engine/engine-inventory-parity\\.spec\\.ts$",
"src/engine/engine-parity\\.spec\\.ts$",
"src/engine/inventory-mapping-direction\\.spec\\.ts$",
diff --git a/src/engine/docs-27-backfill-engine.spec.ts b/src/engine/docs-27-backfill-engine.spec.ts
new file mode 100644
index 000000000..1aadb8b9b
--- /dev/null
+++ b/src/engine/docs-27-backfill-engine.spec.ts
@@ -0,0 +1,39 @@
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { engineCapabilityMatrix } from './engine-capability-matrix';
+
+/**
+ * docs/27 tells search-provider authors to backfill through `ctx.engine.*` calls. Those calls go
+ * straight to the active engine adapter, so a method the capability matrix marks not-available on
+ * an engine rejects there with `EngineNotSupportedError`. The page must say so next to the method,
+ * or an author on that engine follows the documented path and every call fails.
+ */
+describe('docs/27 names the engines its ctx.engine backfill path does not work on', () => {
+ const repoRoot = join(__dirname, '..', '..');
+ const doc = readFileSync(join(repoRoot, 'docs', '27-plugin-search-providers.md'), 'utf8');
+ const paragraphs = doc.split(/\n\s*\n/);
+ const matrix = engineCapabilityMatrix();
+ const engineName = { wwjs: 'whatsapp-web.js', baileys: 'Baileys' } as const;
+
+ const methods = [...new Set([...doc.matchAll(/ctx\.engine\.(\w+)\(/g)].map(match => match[1]))];
+
+ it('finds the ctx.engine calls the page documents', () => {
+ expect(methods).toEqual(expect.arrayContaining(['getChats', 'getChatHistory']));
+ });
+
+ const gaps = methods.flatMap(method =>
+ (['wwjs', 'baileys'] as const)
+ .filter(engine => matrix[method]?.[engine].status === 'not-available')
+ .map(engine => [method, engineName[engine]] as const),
+ );
+
+ it.each(gaps)('says `%s` rejects on %s', (method, engine) => {
+ const qualified = paragraphs.some(
+ paragraph =>
+ paragraph.includes(`\`${method}\``) &&
+ paragraph.includes(engine) &&
+ paragraph.includes('EngineNotSupportedError'),
+ );
+ expect(qualified).toBe(true);
+ });
+});