Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(identity): keep a lid's phone when another session finds none
Yudhi Armyndharis1f9f6a548e34
diff --git a/docs/05-database-design.md b/docs/05-database-design.mdindex a4ab1b5ee..d2fa1d5aa 100644--- a/docs/05-database-design.md+++ b/docs/05-database-design.md@@ -202,7 +202,7 @@ Every timestamp column on the PostgreSQL data connection is `timestamp without t- a naive timestamp is parsed back as UTC, through a parser registered for the scalar `timestamp` OID only (the `timestamp[]` OID keeps the driver's array parser; the schema has no such column);- every pooled connection issues `SET TIME ZONE 'UTC'` on connect, through the pool's own connect hook so that a socket failure or a refused statement fails the acquire instead of the process. That is what makes the server-side `DEFAULT now()` behind each `@CreateDateColumn`/`@UpdateDateColumn` write UTC too. Boot reads the effective `TimeZone` back, at two instants six months apart so a zone that merely reads +00 in winter is caught, and fails if it is not UTC; a pin that a pooler or a server-side default overrode cannot pass unnoticed.-Three `@CreateDateColumn`/`@UpdateDateColumn` columns are filled by the app rather than by that default: `lid_mappings.updatedAt`, `chat_states.updatedAt` and `baileys_stored_messages.createdAt` reach the database through an `upsert` that passes the value, so the default behind them never fires and they follow the binding rule above instead of the session zone.+Three `@CreateDateColumn`/`@UpdateDateColumn` columns are filled by the app rather than by that default: `lid_mappings.updatedAt`, `chat_states.updatedAt` and `baileys_stored_messages.createdAt` reach the database through an `upsert` (for a negative `lid_mappings` result, an insert and an update) that passes the value, so the default behind them never fires and they follow the binding rule above instead of the session zone.`messages.createdAt` is a fourth, and split within itself: a live message takes the server default, while a row written by the Baileys history backfill carries the message's own time, stamped in `message-history-projector.ts` so the chat panel orders history correctly. Both conventions are UTC since 0.23.6, and the two cannot be told apart row by row, which is why the 0.23.6 upgrade notes exclude that column from any blanket conversion. A restore writes `createdAt`/`updatedAt` explicitly wherever the archive carries them, so a restored table follows the archive rather than either rule above; the 0.23.6 upgrade notes cover that case separately.diff --git a/src/engine/identity/lid-mapping-store.service.ts b/src/engine/identity/lid-mapping-store.service.tsindex cd548796a..1b791dc72 100644--- a/src/engine/identity/lid-mapping-store.service.ts+++ b/src/engine/identity/lid-mapping-store.service.ts@@ -1,6 +1,6 @@import { Injectable, OnModuleInit } from '@nestjs/common';import { InjectRepository } from '@nestjs/typeorm';-import { In, Repository } from 'typeorm';+import { In, IsNull, Repository } from 'typeorm';import { LidMapping } from './lid-mapping.entity';import { userPart } from './wa-id';import { createLogger } from '../../common/services/logger.service';@@ -49,7 +49,10 @@ export interface LidMappingStore {findPhoneForLid?(jid: string): Promise<string | null>;/** The lids the table maps to these phone digits, read from the database. */findLidsForPhone?(phone: string): Promise<string[]>;- /** Write-through, last-write-wins: update the cache + persist. A `null` phone records a negative result. */+ /**+ * Write-through: update the cache + persist. A phone is last-write-wins. A `null` phone records a negative+ * result, and replaces a stored phone only when the same session (or none) wrote it.+ */remember(lid: string, phone: string | null, sessionId?: string): Promise<void>;}@@ -277,11 +280,13 @@ export class LidMappingStoreService implements LidMappingStore, OnModuleInit {if (!lid || this.lidToPhone.get(lid) === phone) {return; // unseen-or-changed only; a no-op write would just churn updatedAt}- this.index(lid, phone);+ // A negative is not cached until the table has taken it: see clear().+ if (phone !== null) this.index(lid, phone);this.unsettledWrites.set(lid, (this.unsettledWrites.get(lid) ?? 0) + 1);if (this.pendingLookups.has(lid)) this.racedLookups.add(lid);try {- await this.repo.upsert({ lid, phone, sessionId: sessionId ?? null, updatedAt: new Date() }, ['lid']);+ if (phone === null) await this.clear(lid, sessionId ?? null);+ else await this.repo.upsert({ lid, phone, sessionId: sessionId ?? null, updatedAt: new Date() }, ['lid']);} catch (err) {this.logger.warn(`Failed to persist lid->phone mapping for ${lid}: ${err instanceof Error ? err.message : String(err)}`,@@ -293,6 +298,36 @@ export class LidMappingStoreService implements LidMappingStore, OnModuleInit {}}+ /**+ * Record that a session found no phone for a lid. The row is shared by every session, and one account+ * can see a contact's number while another cannot, so a negative replaces the row only when it holds+ * no phone or the phone was written by this session (or by none); another session's mapping stays. The+ * check is part of the update, not a read before it, so a phone stored meanwhile is never overwritten.+ * When the row is cleared the cached entry is dropped rather than set: a phone written while the+ * statements ran may be the newer value, and the next read takes whichever the table holds.+ */+ private async clear(lid: string, sessionId: string | null): Promise<void> {+ const updatedAt = new Date();+ // Creates the row when there is none; an existing one is left to the conditional update.+ await this.repo+ .createQueryBuilder()+ .insert()+ .values({ lid, phone: null, sessionId, updatedAt })+ .orIgnore()+ .execute();+ const replaceable = [+ { lid, phone: IsNull() },+ { lid, sessionId: IsNull() },+ ...(sessionId ? [{ lid, sessionId }] : []),+ ];+ const { affected } = await this.repo.update(replaceable, { phone: null, sessionId, updatedAt });+ if (!affected) return;+ const cached = this.lidToPhone.get(lid);+ this.lidToPhone.delete(lid);+ if (cached) this.unindex(lid, cached);+ this.absentFromTable.delete(lid);+ }+/*** Repository fallback for a cache miss. Rows past the preload cap (or evicted by the LRU) are* still persisted, so a miss is warmed from the table: THIS lookup still returns undefined —diff --git a/src/engine/identity/lid-mapping-store.spec.ts b/src/engine/identity/lid-mapping-store.spec.tsindex c01a97c9f..4e3de992d 100644--- a/src/engine/identity/lid-mapping-store.spec.ts+++ b/src/engine/identity/lid-mapping-store.spec.ts@@ -1,4 +1,4 @@-import { FindOperator, In, Repository } from 'typeorm';+import { DataSource, FindOperator, In, Repository } from 'typeorm';import { LidMappingStoreService } from './lid-mapping-store.service';import { LidMapping } from './lid-mapping.entity';@@ -73,16 +73,7 @@ describe('LidMappingStoreService', () => {);});- it('caches a negative result (lid known-but-unresolved)', async () => {- const repo = makeFakeRepo();- const store = await newStore(repo);- await store.remember('333', null);- expect(store.getCached('333')).toBeNull();- expect(store.lidsForPhone('anything')).toEqual([]);- expect(repo.upsert).toHaveBeenCalledWith(expect.objectContaining({ lid: '333', phone: null }), ['lid']);- });-- it('is last-write-wins and reindexes the reverse map on a phone change', async () => {+ it('a phone is last-write-wins and reindexes the reverse map on a phone change', async () => {const store = await newStore(makeFakeRepo([{ lid: '111', phone: '628999' }]));await store.remember('111', '628000');expect(store.getCached('111')).toBe('628000');@@ -625,9 +616,9 @@ describe('LidMappingStoreService — deterministic persisted lookups (authorizat});it('a stale cached negative never shadows a mapping the table has', async () => {- const repo = makeFakeRepo();- const store = await newStore(repo);- await store.remember('222', null); // this node cached a negative+ const repo = makeFakeRepo([{ lid: '222', phone: null }]);+ const store = await newStore(repo); // this node cached a negative+ expect(store.getCached('222')).toBeNull();repo.rows[0].phone = '628888'; // another node has since mapped it in the tableexpect(await store.findPhoneForLid('222@lid')).toBe('628888');expect(await store.phonesForLidsPersisted(['222'])).toEqual({ 222: '628888' });@@ -784,3 +775,117 @@ describe('LidMappingStoreService — deterministic persisted lookups (authorizatexpect(await store.findLidsForPhone('628999')).toEqual(['111']);});});++describe('LidMappingStoreService: a negative result on the shared table', () => {+ let ds: DataSource;+ let repo: Repository<LidMapping>;+ let store: LidMappingStoreService;+ const row = (lid: string) => repo.findOne({ where: { lid } });++ /** Holds the next call of a repository method until `release` is called; `reached` settles on entry. */+ function hold(method: 'upsert' | 'update') {+ let release!: () => void;+ let entered!: () => void;+ const gate = new Promise<void>(resolve => (release = resolve));+ const reached = new Promise<void>(resolve => (entered = resolve));+ const real = repo[method].bind(repo) as (...args: unknown[]) => Promise<unknown>;+ jest.spyOn(repo, method).mockImplementationOnce((async (...args: unknown[]) => {+ entered();+ await gate;+ return real(...args);+ }) as never);+ return { release, reached };+ }++ beforeEach(async () => {+ ds = await new DataSource({+ type: 'better-sqlite3',+ database: ':memory:',+ entities: [LidMapping],+ synchronize: true,+ }).initialize();+ repo = ds.getRepository(LidMapping);+ store = new LidMappingStoreService(repo);+ await store.onModuleInit();+ });++ afterEach(() => ds.destroy());++ it('records a negative for a lid with no row', async () => {+ await store.remember('333', null, 's1');++ expect(await row('333')).toMatchObject({ phone: null, sessionId: 's1' });+ expect(await store.findPhoneForLid('333@lid')).toBeNull();+ });++ it('leaves the phone another session stored, in the table and in the cache', async () => {+ await store.remember('111', '628111', 's1');+ await store.remember('111', null, 's2');++ expect(await row('111')).toMatchObject({ phone: '628111', sessionId: 's1' });+ expect(store.getCached('111')).toBe('628111');+ expect(await store.lidsForPhonesPersisted(['628111'])).toEqual({ 628111: ['111'] });+ });++ it('clears the phone when the session that stored it answers no phone (#1058)', async () => {+ await store.remember('111', '628111', 's1');+ await store.remember('111', null, 's1');++ expect(await row('111')).toMatchObject({ phone: null, sessionId: 's1' });+ expect(store.lidsForPhone('628111')).toEqual([]);+ expect(store.getCached('111')).not.toBe('628111');+ expect(await store.findPhoneForLid('111@lid')).toBeNull();+ });++ it('clears a phone that no session owns', async () => {+ await repo.save({ lid: '111', phone: '628111', sessionId: null });+ await store.reload();++ await store.remember('111', null, 's2');++ expect(await row('111')).toMatchObject({ phone: null, sessionId: 's2' });+ });++ it('drops a phone cached here once the table holds none, whichever session cleared it', async () => {+ await store.remember('111', '628111', 's1');+ await repo.update({ lid: '111' }, { phone: null }); // s1 answered no phone on another node++ await store.remember('111', null, 's2');++ expect(await row('111')).toMatchObject({ phone: null, sessionId: 's2' });+ expect(store.lidsForPhone('628111')).toEqual([]);+ });++ it('a phone from any session still replaces the row', async () => {+ await store.remember('111', null, 's1');+ await store.remember('111', '628111', 's2');+ await store.remember('111', '628222', 's3');++ expect(await row('111')).toMatchObject({ phone: '628222', sessionId: 's3' });+ });++ it('keeps a phone stored while the negative of another session is being written', async () => {+ const update = hold('update');+ const negative = store.remember('111', null, 's2');+ await update.reached; // the row exists (holding no phone); the conditional update has not run+ await store.remember('111', '628111', 's1');+ update.release();+ await negative;++ expect(await row('111')).toMatchObject({ phone: '628111', sessionId: 's1' });+ expect(store.getCached('111')).toBe('628111');+ });++ it('does not cache a negative over a phone still being written', async () => {+ const upsert = hold('upsert');+ const phone = store.remember('111', '628111', 's1');+ await upsert.reached;+ await store.remember('111', null, 's2'); // lands first: there is no row yet+ upsert.release();+ await phone;++ expect(await row('111')).toMatchObject({ phone: '628111', sessionId: 's1' });+ expect(store.getCached('111')).not.toBeNull();+ expect(await store.lidsForPhonesPersisted(['628111'])).toEqual({ 628111: ['111'] });+ });+});diff --git a/src/engine/identity/lid-mapping.entity.ts b/src/engine/identity/lid-mapping.entity.tsindex 9a2f48e5d..4d7842f16 100644--- a/src/engine/identity/lid-mapping.entity.ts+++ b/src/engine/identity/lid-mapping.entity.ts@@ -5,11 +5,12 @@ import { Column, Entity, Index, PrimaryColumn, UpdateDateColumn } from 'typeorm'* lid (the maintainer wants the mapping shared, not per-session), so a lid resolved in one session/run is* usable everywhere and survives restarts - replacing the per-session, in-memory `lidToPn` map.*- * Last-write-wins: a stored mapping is "best known, not forever" (WhatsApp recycles numbers), so it's a- * cache WhatsApp can correct. `phone` is nullable to record a negative result (a lid we looked up and- * could not resolve) so active lookups stay one-network-call-per-unknown-lid. `sessionId` is provenance- * only (which session last wrote the row) - intentionally NOT a foreign key, since the row outlives any- * one session.+ * A phone is last-write-wins: a stored mapping is "best known, not forever" (WhatsApp recycles numbers),+ * so it's a cache WhatsApp can correct. `phone` is nullable to record a negative result (a lid we looked+ * up and could not resolve) so active lookups stay one-network-call-per-unknown-lid. A negative replaces a+ * phone only when the session that wrote it (or none) answers it, since another account may not see the+ * number. `sessionId` is the session that last wrote the row - intentionally NOT a foreign key, since the+ * row outlives any one session.*/@Entity('lid_mappings')@Index(['phone']) // reverse lookup: phone -> lids, for the message from-filter@@ -22,7 +23,7 @@ export class LidMapping {@Column({ type: 'varchar', nullable: true })phone!: string | null;- /** The session that last wrote this row. Provenance/debugging only; not a foreign key. */+ /** The session that last wrote this row; a negative from another session leaves its phone. Not a foreign key. */@Column({ type: 'varchar', nullable: true })sessionId!: string | null;diff --git a/src/modules/auth/chat-scope.service.spec.ts b/src/modules/auth/chat-scope.service.spec.tsindex 262bd90e3..3a72789a9 100644--- a/src/modules/auth/chat-scope.service.spec.ts+++ b/src/modules/auth/chat-scope.service.spec.ts@@ -1,6 +1,6 @@import { ForbiddenException } from '@nestjs/common';import { ChatScopeService } from './chat-scope.service';-import { FindOperator, Repository } from 'typeorm';+import { DataSource, FindOperator, Repository } from 'typeorm';import { LidMappingStoreService } from '../../engine/identity/lid-mapping-store.service';import { LidMapping } from '../../engine/identity/lid-mapping.entity';@@ -116,3 +116,43 @@ describe('ChatScopeService', () => {expect(await svc.allows({ allowedChats: [`${PHONE}@c.us`] }, `${PHONE}@c.us`)).toBe(true);});});++describe('ChatScopeService over the shared lid table, with two sessions', () => {+ const key = { allowedChats: [`${PHONE}@c.us`] };+ let ds: DataSource;+ let store: LidMappingStoreService;+ let svc: ChatScopeService;+ const admitted = async (): Promise<[boolean, boolean]> => [+ await svc.allows(key, `${LID}@lid`),+ (await svc.scopeForFilter(key))!.allowed.has(`${LID}@lid`),+ ];++ beforeEach(async () => {+ ds = await new DataSource({+ type: 'better-sqlite3',+ database: ':memory:',+ entities: [LidMapping],+ synchronize: true,+ }).initialize();+ store = new LidMappingStoreService(ds.getRepository(LidMapping));+ await store.onModuleInit();+ svc = new ChatScopeService(store);+ await store.remember(LID, PHONE, 's1');+ });++ afterEach(() => ds.destroy());++ it('still admits the lid of a phone after a session that cannot see the number answers no phone', async () => {+ expect(await admitted()).toEqual([true, true]);++ await store.remember(LID, null, 's2');++ expect(await admitted()).toEqual([true, true]);+ });++ it('stops admitting it once the session that stored the phone answers no phone', async () => {+ await store.remember(LID, null, 's1');++ expect(await admitted()).toEqual([false, false]);+ });+});diff --git a/src/modules/session/session-lid-resolver.service.ts b/src/modules/session/session-lid-resolver.service.tsindex 6f610cf1e..80b53ccb4 100644--- a/src/modules/session/session-lid-resolver.service.ts+++ b/src/modules/session/session-lid-resolver.service.ts@@ -107,7 +107,8 @@ export class SessionLidResolver {// when the operator never sent to them (#583 R3 Phase 2). A definitive null is persisted too, so// a phone that later becomes hidden overwrites its stale mapping instead of being served forever// (#1058). Reuses the resolution above — no extra network call — and is fire-and-forget so- // dispatch never blocks/fails on it.+ // dispatch never blocks/fails on it. The store keeps a phone another session stored, since that+ // account may still see the number.if (phone || resolved) {void this.lidMappingStore?.remember(userPart(contactId), phone, sessionId)?.catch(() => {});}