Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(message): fail a lapsed node's batches whenever it is taken over
Yudhi Armyndharis27e8a94d731d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ab4f61714..560155b9d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -161,6 +161,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- A template name over 100 code points, such as one of emoji with variation selectors, is refused with `400` instead of failing with `500` on PostgreSQL.- `POST /api/sessions/:sessionId/messages/send-bulk` answers `429` instead of `400` when the node already runs `BULK_MAX_CONCURRENT_BATCHES` batches, so clients retry it.- A bulk batch whose run failed before processing started, or whose process died first, ends `FAILED` instead of staying `PENDING`, and batches failed by the startup or takeover reap report `completedAt`.+- Starting or stopping a session whose node's lease lapsed, with `POST /api/sessions/:sessionId/start` or `POST /api/sessions/:sessionId/stop`, fails that node's unfinished bulk batches, as the takeover sweep does; before, they stayed `PENDING` or `PROCESSING` until a node restarted.- `SEND_PACING_COLD_DAILY_CAP=0` or `off` turns the cold-reachout cap off; a blank value set in the container environment could not, because the boot drops blank container variables.- A session stopped while it was starting or reconnecting no longer reads `initializing` until the next restart.- A stop and start issued while a reconnect was still tearing down the old engine no longer leaves a second engine running on the same account.diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.mdindex 700dd07c4..6087e24e5 100644--- a/docs/13-horizontal-scaling.md+++ b/docs/13-horizontal-scaling.md@@ -39,7 +39,8 @@> sessions in a running-or-should-be state are adopted; mid-pairing and operator-`failed`> ones are left alone, and a cleanly stopped session releases its claim so it is never> "lapsed". Adopting a session fails its stuck in-flight batches (no auto-resume — the dead-> node's already-sent messages are unknowable). Adopting is gated by the same+> node's already-sent messages are unknowable), and so does an explicit `POST /start` or `POST /stop`+> of a session whose holder's lease lapsed. The sweep's adoption is gated by the same> `AUTO_START_SESSIONS` flag as boot auto-start; the sweep itself is not, and runs on every node,> because it also has a job that starts nothing: a row a vanished node left `ready`, `initializing`,> `authenticating` or `action_required` is marked disconnected once its lease expired more than twodiff --git a/src/modules/message/bulk-message.service.spec.ts b/src/modules/message/bulk-message.service.spec.tsindex 93c4155d8..a158978e4 100644--- a/src/modules/message/bulk-message.service.spec.ts+++ b/src/modules/message/bulk-message.service.spec.ts@@ -163,7 +163,7 @@ describe('BulkMessageService.onApplicationBootstrap', () => {* stuck batches (same no-auto-resume policy as boot — the dead node's already-sent messages are* unknowable), scoped strictly to the one session, with inline media payloads stripped.*/- it('reapProcessingBatches fails only the given session’s PROCESSING batches and strips payloads', async () => {+ it('reapProcessingBatches fails only the given session’s unfinished batches and strips payloads', async () => {const mine = {id: 'b1',sessionId: 'sess-a',@@ -174,7 +174,9 @@ describe('BulkMessageService.onApplicationBootstrap', () => {const reaped = await service.reapProcessingBatches('sess-a', 'session adopted from a lapsed node');- expect(repo.find).toHaveBeenCalledWith({ where: { status: BatchStatus.PROCESSING, sessionId: 'sess-a' } });+ expect(repo.find).toHaveBeenCalledWith({+ where: { status: In([BatchStatus.PENDING, BatchStatus.PROCESSING]), sessionId: 'sess-a' },+ });expect(reaped).toBe(1);expect(repo.update).toHaveBeenCalledWith(...failedWrite('b1'));const written = (repo.update.mock.calls[0] as [unknown, Partial<MessageBatch>])[1];@@ -199,8 +201,8 @@ describe('BulkMessageService.onApplicationBootstrap', () => {});it('reapProcessingBatches leaves alone a PROCESSING batch this process is still running', async () => {- // Adopted session: the claim moved here before the engine finished initializing, and a bulk- // request routed here in that window started a batch of its own before the reap ran.+ // A batch this process still runs for the session, from an earlier stint as its holder or a+ // stale engine, is not the lapsed holder's to fail.const running = { id: 'b-live', sessionId: 'sess-a', status: BatchStatus.PROCESSING, messages: [] };const orphan = { id: 'b-dead', sessionId: 'sess-a', status: BatchStatus.PROCESSING, messages: [] };repo.find.mockResolvedValue([running, orphan]);@@ -213,6 +215,29 @@ describe('BulkMessageService.onApplicationBootstrap', () => {expect(repo.update).toHaveBeenCalledWith(...failedWrite('b-dead'));});+ // An explicit start or stop takes a session over from a lapsed node as much as the takeover sweep+ // does, so the reap is handed to the ownership service all of them go through.+ it('hands the lapsed-holder reap to the ownership service', async () => {+ const onAdoption = jest.fn<void, [(sessionId: string) => Promise<unknown>]>();+ const svc = new BulkMessageService(+ repo as never,+ new EngineRegistry(),+ {} as never,+ {} as never,+ {} as never,+ { onAdoption } as never,+ );+ svc.onModuleInit();+ repo.find.mockResolvedValue([{ id: 'b-dead', sessionId: 'sess-a', status: BatchStatus.PENDING, messages: [] }]);++ await onAdoption.mock.calls[0][0]('sess-a');++ expect(repo.find).toHaveBeenCalledWith({+ where: { status: In([BatchStatus.PENDING, BatchStatus.PROCESSING]), sessionId: 'sess-a' },+ });+ expect(repo.update).toHaveBeenCalledWith(...failedWrite('b-dead'));+ });+/*** A batch is only ever driven by the process holding its session's engine, so on a second replica* "PROCESSING" does not mean "abandoned". Reaping a peer's batch tells the caller their send@@ -416,6 +441,40 @@ describe('BulkMessageService.processBatch', () => {await expect(runProcessBatch()).rejects.toThrow('database unavailable');});+ // A reap after this process takes a session over from a lapsed node also fails PENDING batches, so a+ // batch this process creates must be registered before any reap can read its row.+ it('registers a new batch before its row is written, so a reap leaves it alone', async () => {+ repo.findOne.mockResolvedValueOnce(null).mockReturnValueOnce(new Promise(() => undefined));+ Object.assign(repo, { create: jest.fn((b: Partial<MessageBatch>) => ({ ...b })) });+ let registeredAtWrite: boolean | undefined;+ repo.save.mockImplementation((b: MessageBatch) => {+ registeredAtWrite = inFlightMarkers().get(b.id);+ return Promise.resolve(b);+ });+ const created = await service.createBatch('s1', {+ messages: [{ chatId: '[email protected]', type: 'text', content: { text: 'hi' } }],+ } as never);+ Object.assign(repo, { find: jest.fn().mockResolvedValue([{ ...created, status: BatchStatus.PENDING }]) });++ expect(registeredAtWrite).toBe(true);+ expect(await service.reapProcessingBatches('s1', 'session taken from a lapsed node')).toBe(0);+ expect(repo.update).not.toHaveBeenCalled();+ });++ it('keeps the cancel signal of a cancel that lands while the new batch is being saved', async () => {+ repo.findOne.mockResolvedValueOnce(null).mockReturnValueOnce(new Promise(() => undefined));+ Object.assign(repo, { create: jest.fn((b: Partial<MessageBatch>) => ({ ...b })) });+ repo.save.mockImplementation((b: MessageBatch) => {+ inFlightMarkers().set(b.id, false); // cancelBatch found the committed row first+ return Promise.resolve(b);+ });+ const created = await service.createBatch('s1', {+ messages: [{ chatId: '[email protected]', type: 'text', content: { text: 'hi' } }],+ } as never);++ expect(inFlightMarkers().get(created.id)).toBe(false);+ });+it('rejects a new batch (before persisting) when the concurrent in-flight cap is reached', async () => {const prev = process.env.BULK_MAX_CONCURRENT_BATCHES;process.env.BULK_MAX_CONCURRENT_BATCHES = '2';@@ -1022,16 +1081,18 @@ describe('BulkMessageService.processBatch', () => {expect(inFlightMarkers().has('b1')).toBe(false); // marker still released});- it('sends nothing when a cancel commits between pickup and the guarded start transition', async () => {+ // A run starts only from PENDING: a cancel, or a reap after another node adopted the session, that+ // committed before the start transition leaves the row CANCELLED or FAILED, and nothing is sent.+ it('sends nothing when a cancel or a reap commits between pickup and the guarded start transition', async () => {repo.findOne.mockResolvedValue(makeBatch(3));- repo.update.mockResolvedValueOnce({ affected: 0 }); // CANCELLED won the race to the first write+ repo.update.mockResolvedValueOnce({ affected: 0 }); // the row left PENDING before the first writeawait runProcessBatch();expect(engine.sendTextMessage).not.toHaveBeenCalled();expect(repo.update).toHaveBeenCalledTimes(1); // the guarded start transition onlyexpect(repo.update).toHaveBeenCalledWith(- { id: 'b1', status: Not(BatchStatus.CANCELLED) },+ { id: 'b1', status: BatchStatus.PENDING },expect.objectContaining({ status: BatchStatus.PROCESSING }),);expect(repo.save).not.toHaveBeenCalled(); // nothing is written after losing to the cancel@@ -1445,6 +1506,8 @@ describe('BulkMessageService.createBatch base64 media cap', () => {await expect(service.createBatch('s1', dto)).rejects.toThrow('database unavailable');expect((service as unknown as { inFlightBatches: number }).inFlightBatches).toBe(0);+ // The marker registered before the write is dropped too.+ expect((service as unknown as { processingBatches: Map<string, boolean> }).processingBatches.size).toBe(0);});it.each([diff --git a/src/modules/message/bulk-message.service.ts b/src/modules/message/bulk-message.service.tsindex 25666d002..598a8cdfd 100644--- a/src/modules/message/bulk-message.service.ts+++ b/src/modules/message/bulk-message.service.ts@@ -6,6 +6,7 @@ import {NotFoundException,Optional,OnApplicationBootstrap,+ OnModuleInit,} from '@nestjs/common';import { createLogger } from '../../common/services/logger.service';import { ConfigService } from '@nestjs/config';@@ -104,7 +105,7 @@ interface BatchExecutionState {}@Injectable()-export class BulkMessageService implements OnApplicationBootstrap {+export class BulkMessageService implements OnModuleInit, OnApplicationBootstrap {private readonly logger = createLogger(BulkMessageService.name);private readonly processingBatches = new Map<string, boolean>(); // Track active batches for cancellationprivate inFlightBatches = 0; // count of batches currently in processBatch (memory bound, see cap above)@@ -127,6 +128,15 @@ export class BulkMessageService implements OnApplicationBootstrap {private readonly configService?: ConfigService,) {}+ /**+ * A session this process takes over from a node whose lease lapsed leaves that node's unfinished+ * batches behind, whichever path took it: an explicit start or stop as much as the takeover sweep.+ * Registered here, before any bootstrap hook can start a session.+ */+ onModuleInit(): void {+ this.ownership?.onAdoption(sessionId => this.reapProcessingBatches(sessionId, 'session taken from a lapsed node'));+ }+/*** Transition orphaned batches on startup. A batch still in PENDING or PROCESSING belongs to a* previous (crashed/restarted) process — this fresh process is not driving it, so it would@@ -178,25 +188,29 @@ export class BulkMessageService implements OnApplicationBootstrap {}/**- * Fail a session's stuck PROCESSING batches after the session was adopted from a lapsed node.- * Same policy as the boot reaper and for the same reason: the dead node's already-sent messages- * are unknowable, so resuming risks double-sends — FAILED with the payloads stripped is the+ * Fail a session's unfinished (PENDING or PROCESSING) batches after the session was taken over from a+ * node whose lease lapsed: a batch that node saved but never picked up is as orphaned as one it was+ * running. Same policy as the boot reaper and for the same reason: the dead node's already-sent+ * messages are unknowable, so resuming risks double-sends: FAILED with the payloads stripped is the* honest terminal state, and the caller can re-issue the batch knowingly.*- * A batch this process is running itself is not orphaned: the adoption claims the session before- * its engine finishes initializing, so a bulk request routed here in that window starts a batch- * that is already PROCESSING by the time the caller reaps. Only the dead holder's batches go.+ * Runs after this process claims the session from, or releases the claim of, such a node (see+ * onModuleInit). A batch this process created or is still running is not orphaned, as when it held+ * the session before or still runs a stale engine for it: createBatch registers every batch before+ * its row is written, so only the lapsed node's batches go.*/async reapProcessingBatches(sessionId: string, reason: string): Promise<number> {- const processing = (- await this.batchRepository.find({ where: { status: BatchStatus.PROCESSING, sessionId } })+ const unfinished = (+ await this.batchRepository.find({+ where: { status: In([BatchStatus.PENDING, BatchStatus.PROCESSING]), sessionId },+ })).filter(batch => !this.processingBatches.has(batch.id));let failed = 0;- for (const batch of processing) {+ for (const batch of unfinished) {if (await this.failOrphanedBatch(batch)) failed++;}if (failed > 0) {- this.logger.warn(`Marked ${failed} PROCESSING batch(es) FAILED for session ${sessionId} (${reason})`);+ this.logger.warn(`Marked ${failed} unfinished batch(es) FAILED for session ${sessionId} (${reason})`);}return failed;}@@ -281,6 +295,9 @@ export class BulkMessageService implements OnApplicationBootstrap {};const batch = this.batchRepository.create({+ // Assigned here rather than by the database, so the batch is registered below before its row+ // exists anywhere.+ id: randomUUID(),batchId,sessionId,status: BatchStatus.PENDING,@@ -291,6 +308,10 @@ export class BulkMessageService implements OnApplicationBootstrap {currentIndex: 0,});+ // Registered before the row exists, so a reap for this session (a claim that took the session+ // from another holder) can never read this process's own batch unregistered. A cancel can only+ // find the row once it is committed, and the false it sets then stands for the pickup check.+ this.processingBatches.set(batch.id, true);// Reserve synchronously in the same turn as the cap check. There is deliberately no await between// them, so a burst cannot all observe the same stale count and overshoot the ceiling.this.inFlightBatches++;@@ -298,6 +319,7 @@ export class BulkMessageService implements OnApplicationBootstrap {await this.batchRepository.save(batch);} catch (error) {this.inFlightBatches--;+ this.processingBatches.delete(batch.id);// Two concurrent creates with the same caller-supplied batchId both pass the read above; the// unique index decides, and the loser gets the same 400 as the sequential case.if (isUniqueViolation(error)) {@@ -409,7 +431,7 @@ export class BulkMessageService implements OnApplicationBootstrap {throw error;} finally {if (reserved) this.inFlightBatches--;- if (batch) this.processingBatches.delete(batch.id);+ this.processingBatches.delete(batchDbId);}}@@ -427,19 +449,20 @@ export class BulkMessageService implements OnApplicationBootstrap {await this.finalizeBatch(batch, state);}- /** Returns false when a committed cancel won the guarded start UPDATE — send nothing. */+ /** Returns false when the batch left PENDING before this start UPDATE, so nothing is sent. */private async markBatchProcessing(batch: MessageBatch): Promise<boolean> {- // Transition to PROCESSING with the guard IN the UPDATE: it only lands while the stored status- // is not CANCELLED, so a cancel that already committed (any process) can never be overwritten- // back to PROCESSING. Zero affected rows = cancel-before-start won; send nothing.+ // Transition to PROCESSING with the guard IN the UPDATE: it only lands while the stored status is+ // still PENDING, the one status a run starts from. A cancel that already committed (any process)+ // or a reap that failed the batch after another node adopted its session can never be overwritten+ // back to PROCESSING. Zero affected rows means send nothing.batch.status = BatchStatus.PROCESSING;batch.startedAt = new Date();const started = await this.batchRepository.update(- { id: batch.id, status: Not(BatchStatus.CANCELLED) },+ { id: batch.id, status: BatchStatus.PENDING },{ status: BatchStatus.PROCESSING, startedAt: batch.startedAt },);if (!started.affected) {- this.logger.log(`Batch ${batch.batchId} was cancelled before processing started; nothing was sent`);+ this.logger.log(`Batch ${batch.batchId} was cancelled or failed before processing started; nothing was sent`);return false;}return true;diff --git a/src/modules/session/session-ownership.service.spec.ts b/src/modules/session/session-ownership.service.spec.tsindex 72763a415..4dac9be61 100644--- a/src/modules/session/session-ownership.service.spec.ts+++ b/src/modules/session/session-ownership.service.spec.ts@@ -101,6 +101,90 @@ describe('SessionOwnershipService', () => {});});+ // What a node whose lease lapsed left unfinished (its bulk batches) is failed once this process takes+ // the session over, by claiming it or by releasing that node's claim.+ describe('taking a session over from a lapsed node', () => {+ const settle = (): Promise<void> => new Promise(resolve => setImmediate(resolve));+ const lapsedBy = (nodeId: string): Partial<Session> => ({+ nodeId,+ claimedAt: new Date(Date.now() - 120_000),+ leaseExpiresAt: new Date(Date.now() - 60_000),+ });+ const liveBy = (nodeId: string): Partial<Session> => ({+ nodeId,+ claimedAt: new Date(),+ leaseExpiresAt: new Date(Date.now() + 60_000),+ });++ // A released row is not taken over: the node that released it may still be finishing its own+ // batches, and failing them under it would leave two writers on one batch row.+ it('runs the adoption handler for a claim of a lapsed foreign lease only', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ const node = service('node-b');+ node.onAdoption(handler);+ const own = await seed(lapsedBy('node-b'));+ const live = await seed(liveBy('node-c'));+ const released = await seed();+ const orphaned = await seed(lapsedBy('node-a'));++ await expect(node.claim(own.id)).resolves.toBe(true);+ await expect(node.claim(live.id)).resolves.toBe(false);+ await expect(node.claim(released.id)).resolves.toBe(true);+ await settle();+ expect(handler).not.toHaveBeenCalled();++ await expect(node.claim(orphaned.id)).resolves.toBe(true);+ await settle();+ expect(handler.mock.calls).toEqual([[orphaned.id]]);+ });++ // A stop of a session whose holder died clears that holder's claim, and a later start then finds a+ // released row; the release is where the takeover happens.+ it('runs the adoption handler when a release clears a lapsed foreign claim, and only then', async () => {+ const handler = jest.fn().mockResolvedValue(undefined);+ const node = service('node-b');+ node.onAdoption(handler);+ const own = await seed(liveBy('node-b'));+ const live = await seed(liveBy('node-c'));+ const orphaned = await seed(lapsedBy('node-a'));++ await node.release(own.id);+ await node.release(live.id);+ await settle();+ expect(handler).not.toHaveBeenCalled();+ expect((await sessions.findOneByOrFail({ id: own.id })).nodeId).toBeNull();+ expect((await sessions.findOneByOrFail({ id: live.id })).nodeId).toBe('node-c');++ await node.release(orphaned.id);+ await settle();+ expect(handler.mock.calls).toEqual([[orphaned.id]]);+ expect((await sessions.findOneByOrFail({ id: orphaned.id })).nodeId).toBeNull();+ });++ // Awaiting the handler inside claim() held a start between its claim and the moment it counted as+ // starting; a stop landing then released the claim and the engine launched on a row nobody held.+ it('returns from the claim without waiting for the adoption handler', async () => {+ const node = service('node-b');+ const handler = jest.fn(() => new Promise<void>(() => undefined));+ node.onAdoption(handler);+ const session = await seed(lapsedBy('node-a'));++ await expect(node.claim(session.id)).resolves.toBe(true);+ await settle();+ expect(handler).toHaveBeenCalledWith(session.id);+ });++ it('keeps the claim when the adoption handler fails', async () => {+ const node = service('node-b');+ node.onAdoption(() => Promise.reject(new Error('database unavailable')));+ const session = await seed(lapsedBy('node-a'));++ await expect(node.claim(session.id)).resolves.toBe(true);+ await settle();+ expect((await sessions.findOneByOrFail({ id: session.id })).nodeId).toBe('node-b');+ });+ });+describe('releasing', () => {it('frees the session so a peer can take it without waiting for the lease', async () => {const session = await seed();diff --git a/src/modules/session/session-ownership.service.ts b/src/modules/session/session-ownership.service.tsindex 913276da8..e5bc0d92d 100644--- a/src/modules/session/session-ownership.service.ts+++ b/src/modules/session/session-ownership.service.ts@@ -45,6 +45,8 @@ export class SessionOwnershipService {private readonly claimGen = new Map<string, number>();/** Notified when a renewal proves this process no longer holds sessions it thought it did. */private onLeaseLost?: (sessionIds: string[]) => Promise<void> | void;+ /** Notified when this process takes a session over from a node whose lease lapsed. See onAdoption. */+ private onAdopted?: (sessionId: string) => Promise<unknown>;/*** How many callers are currently telling renew() that an empty/blank result is NOT evidence of@@ -127,6 +129,14 @@ export class SessionOwnershipService {async claim(sessionId: string): Promise<boolean> {const now = new Date();const leaseExpiresAt = new Date(now.getTime() + this.leaseTtlMs);+ // Who held it, read only when an adoption handler is registered. The UPDATE below still decides+ // the claim; this read only tells whether it took a lapsed lease over from another node.+ const previous = this.onAdopted+ ? await this.sessions.findOne({+ where: { id: sessionId },+ select: { id: true, nodeId: true, leaseExpiresAt: true },+ })+ : null;const result = await this.sessions.createQueryBuilder().update(Session)@@ -150,6 +160,14 @@ export class SessionOwnershipService {this.owned.add(sessionId);this.bumpClaimGen(sessionId);this.lastSeenLease.set(sessionId, leaseExpiresAt.getTime());+ // A released row (nodeId NULL) is not an adoption: its holder may still be finishing its own+ // batches, and failing them under it would leave two writers on one batch row.+ const adopted =+ previous?.nodeId != null &&+ previous.nodeId !== this.nodeId &&+ previous.leaseExpiresAt != null &&+ previous.leaseExpiresAt < now;+ if (adopted) this.followAdoption(sessionId);} else this.logger.warn('Session is held by another node', { sessionId, nodeId: this.nodeId });return claimed;}@@ -167,13 +185,29 @@ export class SessionOwnershipService {const now = new Date();this.owned.delete(sessionId);this.bumpClaimGen(sessionId);- await this.sessions+ const cleared = { nodeId: null, claimedAt: null, leaseExpiresAt: null, nodeUrl: null };+ // Its own claim first, in one statement as before. Only when that matched nothing is a lapsed+ // claim of another node cleared, and that is a takeover like a claim: the dead holder's+ // unfinished work is failed too (see onAdoption).+ const own = await this.sessions.createQueryBuilder().update(Session)- .set({ nodeId: null, claimedAt: null, leaseExpiresAt: null, nodeUrl: null })+ .set(cleared).where('id = :id', { id: sessionId })- .andWhere('("nodeId" = :me OR "leaseExpiresAt" < :now)', { me: this.nodeId, now: leaseParam(now) })+ .andWhere('"nodeId" = :me', { me: this.nodeId })+ .execute();+ if ((own.affected ?? 0) > 0) return;+ const lapsed = await this.sessions+ .createQueryBuilder()+ .update(Session)+ .set(cleared)+ .where('id = :id', { id: sessionId })+ .andWhere('"nodeId" IS NOT NULL AND "nodeId" <> :me AND "leaseExpiresAt" < :now', {+ me: this.nodeId,+ now: leaseParam(now),+ }).execute();+ if ((lapsed.affected ?? 0) > 0) this.followAdoption(sessionId);}private bumpClaimGen(sessionId: string): void {@@ -221,6 +255,36 @@ export class SessionOwnershipService {this.onLeaseLost = handler;}+ /**+ * Register what to do when this process takes a session over from a node whose lease lapsed, by+ * claiming it (an explicit start, boot auto-start, the takeover sweep) or by releasing that node's+ * claim (a stop of such a session). Whatever that node left unfinished for the+ * session can no longer finish there. A session its holder released is not taken over: that holder+ * may still be finishing its own work.+ */+ onAdoption(handler: (sessionId: string) => Promise<unknown>): void {+ this.onAdopted = handler;+ }++ /**+ * Run the adoption handler off the caller's path. Awaiting it inside claim() would hold the start+ * between its claim and the moment it counts as starting, and a stop landing in that window would+ * release the claim and let the engine launch on a row nobody holds. A failure is logged; the claim+ * or release stands either way.+ */+ private followAdoption(sessionId: string): void {+ const handler = this.onAdopted;+ if (!handler) return;+ void Promise.resolve()+ .then(() => handler(sessionId))+ .catch((error: unknown) =>+ this.logger.warn('Session adoption follow-up failed', {+ sessionId,+ error: error instanceof Error ? error.message : String(error),+ }),+ );+ }+/*** Tell renew() that "this node no longer holds the row" is currently uninformative, and get back* the release. Held by a replace-all data import across its whole transaction.diff --git a/src/modules/takeover/session-takeover.service.spec.ts b/src/modules/takeover/session-takeover.service.spec.tsindex d90054373..20ddc042f 100644--- a/src/modules/takeover/session-takeover.service.spec.ts+++ b/src/modules/takeover/session-takeover.service.spec.ts@@ -4,7 +4,6 @@ import { SessionStoppedException } from '../session/session-engine-controls';import { Session, SessionStatus } from '../session/entities/session.entity';import type { SessionService } from '../session/session.service';import type { SessionOwnershipService } from '../session/session-ownership.service';-import type { BulkMessageService } from '../message/bulk-message.service';import type { ConfigService } from '@nestjs/config';import type { ShutdownService } from '../../common/services/shutdown.service';@@ -12,7 +11,8 @@ import type { ShutdownService } from '../../common/services/shutdown.service';* The sweep is the retry that boot auto-start never had: both live incidents (a container recreate* landing inside the old identity's lease, and a crashed peer) left sessions sitting disconnected* until a manual POST /start. What has to hold: only lapsed-lease sessions worth resuming are- * started, a lost claim race is a non-event, and adopting a session reconciles its stuck batches.+ * started, and a lost claim race is a non-event. The claim on the start path fails the previous+ * holder's stuck batches (see the ownership and bulk message specs).*/describe('SessionTakeoverService', () => {const lapsed = (over: Partial<Session> = {}): Session =>@@ -31,11 +31,9 @@ describe('SessionTakeoverService', () => {): {svc: SessionTakeoverService;start: jest.Mock;- reap: jest.Mock;markLapsedDisconnected: jest.Mock;} => {const start = opts.startImpl ?? jest.fn().mockResolvedValue({});- const reap = jest.fn().mockResolvedValue(0);const markLapsedDisconnected = jest.fn().mockResolvedValue([]);const config = {get: (key: string, def?: unknown) =>@@ -50,10 +48,9 @@ describe('SessionTakeoverService', () => {lapsedHeldByOthers: jest.fn().mockResolvedValue(rows),leaseTtlMs: 60_000,} as unknown as SessionOwnershipService,- { reapProcessingBatches: reap } as unknown as BulkMessageService,config,);- return { svc, start, reap, markLapsedDisconnected };+ return { svc, start, markLapsedDisconnected };};afterEach(() => {@@ -67,13 +64,12 @@ describe('SessionTakeoverService', () => {// leaving it unclaimable by any peer until the lease lapsed.describe('a sweep racing shutdown', () => {it('adopts nothing once the module is being destroyed', async () => {- const { svc, start, reap } = build([lapsed({ name: 'a' }), lapsed({ name: 'b' })]);+ const { svc, start } = build([lapsed({ name: 'a' }), lapsed({ name: 'b' })]);svc.onModuleDestroy();await svc.sweep();expect(start).not.toHaveBeenCalled();- expect(reap).not.toHaveBeenCalled();});it('stops adopting the rest of the batch when shutdown begins mid-sweep', async () => {@@ -102,13 +98,12 @@ describe('SessionTakeoverService', () => {});});- it('adopts a lapsed authenticated session and reconciles its stuck batches', async () => {- const { svc, start, reap } = build([lapsed({ name: 'a' })]);+ it('adopts a lapsed authenticated session through the ordinary start path', async () => {+ const { svc, start } = build([lapsed({ name: 'a' })]);await svc.sweep();expect(start).toHaveBeenCalledWith('id-a');- expect(reap).toHaveBeenCalledWith('id-a', expect.stringContaining('lapsed'));});it('skips sessions not worth resuming: unauthenticated, mid-pairing, or operator-flagged failed', async () => {@@ -134,28 +129,25 @@ describe('SessionTakeoverService', () => {expect(start.mock.calls).toEqual([['id-dropped']]);});- it('a lost claim race is a non-event: no batch reconcile, and the next candidate still starts', async () => {+ it('a lost claim race is a non-event: the next candidate still starts', async () => {jest.useFakeTimers();const start = jest.fn().mockRejectedValueOnce(new ConflictException('held elsewhere')).mockResolvedValueOnce({});- const { svc, reap } = build([lapsed({ name: 'raced' }), lapsed({ name: 'ours' })], { startImpl: start });+ const { svc } = build([lapsed({ name: 'raced' }), lapsed({ name: 'ours' })], { startImpl: start });const sweep = svc.sweep();await jest.advanceTimersByTimeAsync(2100); // the inter-launch staggerawait sweep;- expect(start).toHaveBeenCalledTimes(2);- expect(reap).toHaveBeenCalledTimes(1);- expect(reap).toHaveBeenCalledWith('id-ours', expect.any(String));+ expect(start.mock.calls).toEqual([['id-raced'], ['id-ours']]);});it('a session stopped after the sweep read it is logged as skipped, not as a lost claim race', async () => {const start = jest.fn().mockRejectedValueOnce(new SessionStoppedException('Session id-halted was stopped'));- const { svc, reap } = build([lapsed({ name: 'halted' })], { startImpl: start });+ const { svc } = build([lapsed({ name: 'halted' })], { startImpl: start });const debug = jest.spyOn((svc as unknown as { logger: { debug: jest.Mock } }).logger, 'debug');await svc.sweep();- expect(reap).not.toHaveBeenCalled();expect(debug.mock.calls.map(([message]) => message as string)).toEqual(['Session halted skipped: stopped by an operator',]);@@ -261,7 +253,6 @@ describe('SessionTakeoverService', () => {const svc = new SessionTakeoverService({ start, markLapsedDisconnected: jest.fn().mockResolvedValue([]) } as unknown as SessionService,{ lapsedHeldByOthers: ownershipCalls, leaseTtlMs: 60_000 } as unknown as SessionOwnershipService,- { reapProcessingBatches: jest.fn().mockResolvedValue(0) } as unknown as BulkMessageService,config,);@@ -302,7 +293,6 @@ describe('SessionTakeoverService', () => {const svc = new SessionTakeoverService({ start } as unknown as SessionService,{ lapsedHeldByOthers: ownershipCalls } as unknown as SessionOwnershipService,- { reapProcessingBatches: jest.fn().mockResolvedValue(0) } as unknown as BulkMessageService,config,{ isShuttingDown: () => true } as unknown as ShutdownService,);@@ -324,7 +314,6 @@ describe('SessionTakeoverService', () => {const svc = new SessionTakeoverService({ start: jest.fn().mockResolvedValue(undefined) } as unknown as SessionService,{ lapsedHeldByOthers: ownershipCalls } as unknown as SessionOwnershipService,- { reapProcessingBatches: jest.fn().mockResolvedValue(0) } as unknown as BulkMessageService,config,);diff --git a/src/modules/takeover/session-takeover.service.ts b/src/modules/takeover/session-takeover.service.tsindex 9f339d8ed..7409d8a0f 100644--- a/src/modules/takeover/session-takeover.service.ts+++ b/src/modules/takeover/session-takeover.service.ts@@ -8,7 +8,6 @@ import { SessionOwnershipService } from '../session/session-ownership.service';import { ShutdownService } from '../../common/services/shutdown.service';import { SessionService } from '../session/session.service';import { SessionStoppedException } from '../session/session-engine-controls';-import { BulkMessageService } from '../message/bulk-message.service';/*** Statuses worth adopting from a lapsed node. They all mean "an engine was (or should be) running".@@ -58,9 +57,9 @@ const STRANDED_LEASE_TTL_MULTIPLE = 2;* for lapsed-lease sessions and starts them here through the ordinary start path, so the claim* stays race-safe against peers doing the same.*- * Lives in its own module (not SessionModule) because adopting a session also reconciles its- * in-flight bulk batches via BulkMessageService — which sits in MessageModule, which imports- * SessionModule; importing it back from SessionModule would close the cycle.+ * The lapsed holder's unfinished bulk batches are failed after the claim, by the adoption handler+ * BulkMessageService registers with SessionOwnershipService, the same as for an explicit POST+ * /start, so the sweep only starts sessions.*/@Injectable()export class SessionTakeoverService implements OnApplicationBootstrap, OnModuleDestroy {@@ -79,7 +78,6 @@ export class SessionTakeoverService implements OnApplicationBootstrap, OnModuleDconstructor(private readonly sessionService: SessionService,private readonly ownership: SessionOwnershipService,- private readonly bulkMessages: BulkMessageService,@Optional()private readonly configService?: ConfigService,// The drain signal, not module destruction. `onModuleDestroy` runs at app.close(), AFTER the@@ -169,9 +167,6 @@ export class SessionTakeoverService implements OnApplicationBootstrap, OnModuleDfromNode: session.nodeId,action: 'session_takeover',});- // The dead node's in-flight batches can never complete; surface them as FAILED now rather- // than leaving them stuck in PROCESSING until some node happens to reboot.- await this.bulkMessages.reapProcessingBatches(session.id, 'session adopted from a lapsed node');} catch (error) {if (error instanceof SessionStoppedException) {// Stopped between the sweep's read and this start; the start refused it, as it should.diff --git a/src/modules/takeover/takeover.module.ts b/src/modules/takeover/takeover.module.tsindex d8a73d584..5d5c88985 100644--- a/src/modules/takeover/takeover.module.ts+++ b/src/modules/takeover/takeover.module.ts@@ -1,15 +1,14 @@import { Module } from '@nestjs/common';import { SessionModule } from '../session/session.module';-import { MessageModule } from '../message/message.module';import { SessionTakeoverService } from './session-takeover.service';/**- * Sits ABOVE both SessionModule and MessageModule on purpose: the takeover sweep starts sessions- * (SessionService) and reconciles their bulk batches (BulkMessageService), and MessageModule- * already imports SessionModule — so this is the lowest place both are reachable without a cycle.+ * The takeover sweep starts sessions through SessionService. A claim on that path has the lapsed+ * holder's unfinished bulk batches failed through the ownership adoption handler, so nothing here+ * needs MessageModule.*/@Module({- imports: [SessionModule, MessageModule],+ imports: [SessionModule],providers: [SessionTakeoverService],})export class TakeoverModule {}