Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(search): describe the backfill a search plugin can do
Yudhi Armyndharis6ecc0ab75f8f
diff --git a/docs/27-plugin-search-providers.md b/docs/27-plugin-search-providers.mdindex 7dc1901d8..bff10a8d9 100644--- a/docs/27-plugin-search-providers.md+++ b/docs/27-plugin-search-providers.md@@ -128,9 +128,28 @@ the same handler keeps your index in step.Rows removed in bulk emit no `message:deleted`: deleting a session and message retention (`MESSAGE_RETENTION_DAYS`) leave the plugin's copies in its index until the plugin removes them itself.**Backfill is the plugin's responsibility.** The hook fires only for live traffic. A plugin installed on-a deployment with existing message history must perform its own one-time backfill (read the `messages`-table via `ctx.engine.getChatHistory` or a direct query, and index) at `onEnable`. The built-in DB-FTS-provider is unaffected (its index is DB-synced via triggers on every insert, including backfill).+a deployment with existing message history must perform its own one-time backfill. The only sanctioned+path is `ctx.engine.getChats(sessionId)` followed by `ctx.engine.getChatHistory(sessionId, chatId, limit)`,+which needs the `engine:read` permission and a session in the plugin's scope. It reads live from WhatsApp,+not from the core `messages` table, and returns at most the 100 most recent messages per chat.++`getChatHistory` exists only on the whatsapp-web.js engine. On Baileys it rejects with+`EngineNotSupportedError` (see [29 - Engine Capability Matrix](./29-engine-capability-matrix.md)), so a+Baileys deployment has no backfill path and a provider there indexes live `message:persisted` traffic+only. Catch that error once and skip the backfill rather than retrying it chat after chat.++There is no capability that reads the `messages` table. Reading the database directly bypasses the+capability model and is unsupported; see [30 - Plugin Sandboxing](./30-plugin-sandboxing.md) for what a+loaded plugin can still reach.++Backfilled items carry only the WhatsApp id (`id` there is the WhatsApp message id), not the core row PK.+Keep a `waMessageId` lookup too, and when a later `message:persisted` arrives for a message you already+backfilled, upsert onto that document instead of adding a second one.++Start the backfill from `onEnable` without awaiting it: a sandboxed lifecycle call is cut off after 30 s,+and walking every chat's history takes longer on a real deployment. Record a marker in `ctx.storage` when+it finishes so a restart does not repeat it. The built-in DB-FTS provider is unaffected (its index is+DB-synced via triggers on every insert, including backfill).## 27.4 Host-side guarantees (the plugin author doesn't handle these)@@ -174,7 +193,7 @@ plugins/my-search/```jsmodule.exports = class MySearchPlugin {async onEnable(ctx) {- // 1. Index every persisted message (live traffic only — backfill separately at onEnable).+ // 1. Index every persisted message (live traffic only — backfill separately, see 27.3).ctx.registerHook('message:persisted', async hookCtx => {const { message } = hookCtx.data;await this._index(ctx, message);diff --git a/package.json b/package.jsonindex 251e00b37..46676b7d7 100644--- a/package.json+++ b/package.json@@ -42,7 +42,7 @@"lint:fix": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix","test": "jest","test:scripts": "node --test scripts/postinstall.spec.js scripts/patch-wwebjs-newsletter-preview.spec.js scripts/patch-wwebjs-status.spec.js scripts/patch-wwebjs-ready-sync.spec.js scripts/patch-wwebjs-participant-arity.spec.js scripts/patch-wwebjs-block.spec.js scripts/patch-wwebjs-group-description.spec.js scripts/patch-wwebjs-media-id.spec.js scripts/patch-wwebjs-send-error.spec.js scripts/patch-baileys-appstate.spec.js scripts/patch-baileys-newsletter-create.spec.js scripts/dockerfile-patchers.spec.js scripts/check-upstream-surface.spec.mjs scripts/check-sdk-docs.spec.mjs scripts/check-audit.spec.mjs scripts/check-contract-shapes.spec.mjs scripts/cli-entrypoint-guard.spec.mjs",- "test:docs": "jest src/common/changelog-sections.spec.ts src/common/ci-privilege-gate.spec.ts src/common/docs-ci-jobs.spec.ts src/common/docs-coverage-policy.spec.ts src/common/docs-metrics-list.spec.ts src/common/docs-next-version.spec.ts src/common/openapi/docs-06-contract.spec.ts src/common/test-lane-partition.spec.ts src/config/docs-env-example.spec.ts src/config/openapi-contract.spec.ts src/config/openapi-export-pins.spec.ts src/config/wa-web-version-example-pin.spec.ts src/engine/docs-29-counts.spec.ts src/engine/engine-inventory-parity.spec.ts src/engine/engine-parity.spec.ts src/engine/inventory-mapping-direction.spec.ts src/engine/adapters/baileys-voice-waveform.spec.ts src/engine/adapters/wwebjs-newsletter-preview.spec.ts src/modules/audit/audit-coverage.spec.ts src/modules/auth/global-route-fence-coverage.spec.ts src/modules/auth/query-session-scope-coverage.spec.ts src/modules/docker/compose-network.spec.ts src/modules/queue/bull-board-mount-coverage.spec.ts src/common/release-gate-parity.spec.ts src/database/migrations/__tests__/migration-drift.spec.ts --testPathIgnorePatterns=/node_modules/",+ "test:docs": "jest src/common/changelog-sections.spec.ts src/common/ci-privilege-gate.spec.ts src/common/docs-ci-jobs.spec.ts src/common/docs-coverage-policy.spec.ts src/common/docs-metrics-list.spec.ts src/common/docs-next-version.spec.ts src/common/openapi/docs-06-contract.spec.ts src/common/test-lane-partition.spec.ts src/config/docs-env-example.spec.ts src/config/openapi-contract.spec.ts src/config/openapi-export-pins.spec.ts src/config/wa-web-version-example-pin.spec.ts src/engine/docs-29-counts.spec.ts src/engine/docs-27-backfill-engine.spec.ts src/engine/engine-inventory-parity.spec.ts src/engine/engine-parity.spec.ts src/engine/inventory-mapping-direction.spec.ts src/engine/adapters/baileys-voice-waveform.spec.ts src/engine/adapters/wwebjs-newsletter-preview.spec.ts src/modules/audit/audit-coverage.spec.ts src/modules/auth/global-route-fence-coverage.spec.ts src/modules/auth/query-session-scope-coverage.spec.ts src/modules/docker/compose-network.spec.ts src/modules/queue/bull-board-mount-coverage.spec.ts src/common/release-gate-parity.spec.ts src/database/migrations/__tests__/migration-drift.spec.ts --testPathIgnorePatterns=/node_modules/","test:watch": "jest --watch","test:cov": "jest --coverage","test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",@@ -163,6 +163,7 @@"src/config/openapi-export-pins\\.spec\\.ts$","src/config/wa-web-version-example-pin\\.spec\\.ts$","src/engine/docs-29-counts\\.spec\\.ts$",+ "src/engine/docs-27-backfill-engine\\.spec\\.ts$","src/engine/engine-inventory-parity\\.spec\\.ts$","src/engine/engine-parity\\.spec\\.ts$","src/engine/inventory-mapping-direction\\.spec\\.ts$",@@ -484,6 +485,7 @@"src/config/openapi-export-pins\\.spec\\.ts$","src/config/wa-web-version-example-pin\\.spec\\.ts$","src/engine/docs-29-counts\\.spec\\.ts$",+ "src/engine/docs-27-backfill-engine\\.spec\\.ts$","src/engine/engine-inventory-parity\\.spec\\.ts$","src/engine/engine-parity\\.spec\\.ts$","src/engine/inventory-mapping-direction\\.spec\\.ts$",diff --git a/src/engine/docs-27-backfill-engine.spec.ts b/src/engine/docs-27-backfill-engine.spec.tsnew file mode 100644index 000000000..1aadb8b9b--- /dev/null+++ b/src/engine/docs-27-backfill-engine.spec.ts@@ -0,0 +1,39 @@+import { readFileSync } from 'node:fs';+import { join } from 'node:path';+import { engineCapabilityMatrix } from './engine-capability-matrix';++/**+ * docs/27 tells search-provider authors to backfill through `ctx.engine.*` calls. Those calls go+ * straight to the active engine adapter, so a method the capability matrix marks not-available on+ * an engine rejects there with `EngineNotSupportedError`. The page must say so next to the method,+ * or an author on that engine follows the documented path and every call fails.+ */+describe('docs/27 names the engines its ctx.engine backfill path does not work on', () => {+ const repoRoot = join(__dirname, '..', '..');+ const doc = readFileSync(join(repoRoot, 'docs', '27-plugin-search-providers.md'), 'utf8');+ const paragraphs = doc.split(/\n\s*\n/);+ const matrix = engineCapabilityMatrix();+ const engineName = { wwjs: 'whatsapp-web.js', baileys: 'Baileys' } as const;++ const methods = [...new Set([...doc.matchAll(/ctx\.engine\.(\w+)\(/g)].map(match => match[1]))];++ it('finds the ctx.engine calls the page documents', () => {+ expect(methods).toEqual(expect.arrayContaining(['getChats', 'getChatHistory']));+ });++ const gaps = methods.flatMap(method =>+ (['wwjs', 'baileys'] as const)+ .filter(engine => matrix[method]?.[engine].status === 'not-available')+ .map(engine => [method, engineName[engine]] as const),+ );++ it.each(gaps)('says `%s` rejects on %s', (method, engine) => {+ const qualified = paragraphs.some(+ paragraph =>+ paragraph.includes(`\`${method}\``) &&+ paragraph.includes(engine) &&+ paragraph.includes('EngineNotSupportedError'),+ );+ expect(qualified).toBe(true);+ });+});