Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(message): fail a lapsed node's batches whenever it is taken over

Yudhi Armyndharis27e8a94d731d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ab4f61714..560155b9d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -161,6 +161,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A template name over 100 code points, such as one of emoji with variation selectors, is refused with `400` instead of failing with `500` on PostgreSQL.
- `POST /api/sessions/:sessionId/messages/send-bulk` answers `429` instead of `400` when the node already runs `BULK_MAX_CONCURRENT_BATCHES` batches, so clients retry it.
- A bulk batch whose run failed before processing started, or whose process died first, ends `FAILED` instead of staying `PENDING`, and batches failed by the startup or takeover reap report `completedAt`.
+- Starting or stopping a session whose node's lease lapsed, with `POST /api/sessions/:sessionId/start` or `POST /api/sessions/:sessionId/stop`, fails that node's unfinished bulk batches, as the takeover sweep does; before, they stayed `PENDING` or `PROCESSING` until a node restarted.
- `SEND_PACING_COLD_DAILY_CAP=0` or `off` turns the cold-reachout cap off; a blank value set in the container environment could not, because the boot drops blank container variables.
- A session stopped while it was starting or reconnecting no longer reads `initializing` until the next restart.
- A stop and start issued while a reconnect was still tearing down the old engine no longer leaves a second engine running on the same account.
diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.md
index 700dd07c4..6087e24e5 100644
--- a/docs/13-horizontal-scaling.md
+++ b/docs/13-horizontal-scaling.md
@@ -39,7 +39,8 @@
> sessions in a running-or-should-be state are adopted; mid-pairing and operator-`failed`
> ones are left alone, and a cleanly stopped session releases its claim so it is never
> "lapsed". Adopting a session fails its stuck in-flight batches (no auto-resume — the dead
-> node's already-sent messages are unknowable). Adopting is gated by the same
+> node's already-sent messages are unknowable), and so does an explicit `POST /start` or `POST /stop`
+> of a session whose holder's lease lapsed. The sweep's adoption is gated by the same
> `AUTO_START_SESSIONS` flag as boot auto-start; the sweep itself is not, and runs on every node,
> because it also has a job that starts nothing: a row a vanished node left `ready`, `initializing`,
> `authenticating` or `action_required` is marked disconnected once its lease expired more than two
diff --git a/src/modules/message/bulk-message.service.spec.ts b/src/modules/message/bulk-message.service.spec.ts
index 93c4155d8..a158978e4 100644
--- a/src/modules/message/bulk-message.service.spec.ts
+++ b/src/modules/message/bulk-message.service.spec.ts
@@ -163,7 +163,7 @@ describe('BulkMessageService.onApplicationBootstrap', () => {
* stuck batches (same no-auto-resume policy as boot — the dead node's already-sent messages are
* unknowable), scoped strictly to the one session, with inline media payloads stripped.
*/
- it('reapProcessingBatches fails only the given session’s PROCESSING batches and strips payloads', async () => {
+ it('reapProcessingBatches fails only the given session’s unfinished batches and strips payloads', async () => {
const mine = {
id: 'b1',
sessionId: 'sess-a',
@@ -174,7 +174,9 @@ describe('BulkMessageService.onApplicationBootstrap', () => {
const reaped = await service.reapProcessingBatches('sess-a', 'session adopted from a lapsed node');
- expect(repo.find).toHaveBeenCalledWith({ where: { status: BatchStatus.PROCESSING, sessionId: 'sess-a' } });
+ expect(repo.find).toHaveBeenCalledWith({
+ where: { status: In([BatchStatus.PENDING, BatchStatus.PROCESSING]), sessionId: 'sess-a' },
+ });
expect(reaped).toBe(1);
expect(repo.update).toHaveBeenCalledWith(...failedWrite('b1'));
const written = (repo.update.mock.calls[0] as [unknown, Partial<MessageBatch>])[1];
@@ -199,8 +201,8 @@ describe('BulkMessageService.onApplicationBootstrap', () => {
});
it('reapProcessingBatches leaves alone a PROCESSING batch this process is still running', async () => {
- // Adopted session: the claim moved here before the engine finished initializing, and a bulk
- // request routed here in that window started a batch of its own before the reap ran.
+ // A batch this process still runs for the session, from an earlier stint as its holder or a
+ // stale engine, is not the lapsed holder's to fail.
const running = { id: 'b-live', sessionId: 'sess-a', status: BatchStatus.PROCESSING, messages: [] };
const orphan = { id: 'b-dead', sessionId: 'sess-a', status: BatchStatus.PROCESSING, messages: [] };
repo.find.mockResolvedValue([running, orphan]);
@@ -213,6 +215,29 @@ describe('BulkMessageService.onApplicationBootstrap', () => {
expect(repo.update).toHaveBeenCalledWith(...failedWrite('b-dead'));
});
+ // An explicit start or stop takes a session over from a lapsed node as much as the takeover sweep
+ // does, so the reap is handed to the ownership service all of them go through.
+ it('hands the lapsed-holder reap to the ownership service', async () => {
+ const onAdoption = jest.fn<void, [(sessionId: string) => Promise<unknown>]>();
+ const svc = new BulkMessageService(
+ repo as never,
+ new EngineRegistry(),
+ {} as never,
+ {} as never,
+ {} as never,
+ { onAdoption } as never,
+ );
+ svc.onModuleInit();
+ repo.find.mockResolvedValue([{ id: 'b-dead', sessionId: 'sess-a', status: BatchStatus.PENDING, messages: [] }]);
+
+ await onAdoption.mock.calls[0][0]('sess-a');
+
+ expect(repo.find).toHaveBeenCalledWith({
+ where: { status: In([BatchStatus.PENDING, BatchStatus.PROCESSING]), sessionId: 'sess-a' },
+ });
+ expect(repo.update).toHaveBeenCalledWith(...failedWrite('b-dead'));
+ });
+
/**
* A batch is only ever driven by the process holding its session's engine, so on a second replica
* "PROCESSING" does not mean "abandoned". Reaping a peer's batch tells the caller their send
@@ -416,6 +441,40 @@ describe('BulkMessageService.processBatch', () => {
await expect(runProcessBatch()).rejects.toThrow('database unavailable');
});
+ // A reap after this process takes a session over from a lapsed node also fails PENDING batches, so a
+ // batch this process creates must be registered before any reap can read its row.
+ it('registers a new batch before its row is written, so a reap leaves it alone', async () => {
+ repo.findOne.mockResolvedValueOnce(null).mockReturnValueOnce(new Promise(() => undefined));
+ Object.assign(repo, { create: jest.fn((b: Partial<MessageBatch>) => ({ ...b })) });
+ let registeredAtWrite: boolean | undefined;
+ repo.save.mockImplementation((b: MessageBatch) => {
+ registeredAtWrite = inFlightMarkers().get(b.id);
+ return Promise.resolve(b);
+ });
+ const created = await service.createBatch('s1', {
+ messages: [{ chatId: '[email protected]', type: 'text', content: { text: 'hi' } }],
+ } as never);
+ Object.assign(repo, { find: jest.fn().mockResolvedValue([{ ...created, status: BatchStatus.PENDING }]) });
+
+ expect(registeredAtWrite).toBe(true);
+ expect(await service.reapProcessingBatches('s1', 'session taken from a lapsed node')).toBe(0);
+ expect(repo.update).not.toHaveBeenCalled();
+ });
+
+ it('keeps the cancel signal of a cancel that lands while the new batch is being saved', async () => {
+ repo.findOne.mockResolvedValueOnce(null).mockReturnValueOnce(new Promise(() => undefined));
+ Object.assign(repo, { create: jest.fn((b: Partial<MessageBatch>) => ({ ...b })) });
+ repo.save.mockImplementation((b: MessageBatch) => {
+ inFlightMarkers().set(b.id, false); // cancelBatch found the committed row first
+ return Promise.resolve(b);
+ });
+ const created = await service.createBatch('s1', {
+ messages: [{ chatId: '[email protected]', type: 'text', content: { text: 'hi' } }],
+ } as never);
+
+ expect(inFlightMarkers().get(created.id)).toBe(false);
+ });
+
it('rejects a new batch (before persisting) when the concurrent in-flight cap is reached', async () => {
const prev = process.env.BULK_MAX_CONCURRENT_BATCHES;
process.env.BULK_MAX_CONCURRENT_BATCHES = '2';
@@ -1022,16 +1081,18 @@ describe('BulkMessageService.processBatch', () => {
expect(inFlightMarkers().has('b1')).toBe(false); // marker still released
});
- it('sends nothing when a cancel commits between pickup and the guarded start transition', async () => {
+ // A run starts only from PENDING: a cancel, or a reap after another node adopted the session, that
+ // committed before the start transition leaves the row CANCELLED or FAILED, and nothing is sent.
+ it('sends nothing when a cancel or a reap commits between pickup and the guarded start transition', async () => {
repo.findOne.mockResolvedValue(makeBatch(3));
- repo.update.mockResolvedValueOnce({ affected: 0 }); // CANCELLED won the race to the first write
+ repo.update.mockResolvedValueOnce({ affected: 0 }); // the row left PENDING before the first write
await runProcessBatch();
expect(engine.sendTextMessage).not.toHaveBeenCalled();
expect(repo.update).toHaveBeenCalledTimes(1); // the guarded start transition only
expect(repo.update).toHaveBeenCalledWith(
- { id: 'b1', status: Not(BatchStatus.CANCELLED) },
+ { id: 'b1', status: BatchStatus.PENDING },
expect.objectContaining({ status: BatchStatus.PROCESSING }),
);
expect(repo.save).not.toHaveBeenCalled(); // nothing is written after losing to the cancel
@@ -1445,6 +1506,8 @@ describe('BulkMessageService.createBatch base64 media cap', () => {
await expect(service.createBatch('s1', dto)).rejects.toThrow('database unavailable');
expect((service as unknown as { inFlightBatches: number }).inFlightBatches).toBe(0);
+ // The marker registered before the write is dropped too.
+ expect((service as unknown as { processingBatches: Map<string, boolean> }).processingBatches.size).toBe(0);
});
it.each([
diff --git a/src/modules/message/bulk-message.service.ts b/src/modules/message/bulk-message.service.ts
index 25666d002..598a8cdfd 100644
--- a/src/modules/message/bulk-message.service.ts
+++ b/src/modules/message/bulk-message.service.ts
@@ -6,6 +6,7 @@ import {
NotFoundException,
Optional,
OnApplicationBootstrap,
+ OnModuleInit,
} from '@nestjs/common';
import { createLogger } from '../../common/services/logger.service';
import { ConfigService } from '@nestjs/config';
@@ -104,7 +105,7 @@ interface BatchExecutionState {
}
@Injectable()
-export class BulkMessageService implements OnApplicationBootstrap {
+export class BulkMessageService implements OnModuleInit, OnApplicationBootstrap {
private readonly logger = createLogger(BulkMessageService.name);
private readonly processingBatches = new Map<string, boolean>(); // Track active batches for cancellation
private inFlightBatches = 0; // count of batches currently in processBatch (memory bound, see cap above)
@@ -127,6 +128,15 @@ export class BulkMessageService implements OnApplicationBootstrap {
private readonly configService?: ConfigService,
) {}
+ /**
+ * A session this process takes over from a node whose lease lapsed leaves that node's unfinished
+ * batches behind, whichever path took it: an explicit start or stop as much as the takeover sweep.
+ * Registered here, before any bootstrap hook can start a session.
+ */
+ onModuleInit(): void {
+ this.ownership?.onAdoption(sessionId => this.reapProcessingBatches(sessionId, 'session taken from a lapsed node'));
+ }
+
/**
* Transition orphaned batches on startup. A batch still in PENDING or PROCESSING belongs to a
* previous (crashed/restarted) process — this fresh process is not driving it, so it would
@@ -178,25 +188,29 @@ export class BulkMessageService implements OnApplicationBootstrap {
}
/**
- * Fail a session's stuck PROCESSING batches after the session was adopted from a lapsed node.
- * Same policy as the boot reaper and for the same reason: the dead node's already-sent messages
- * are unknowable, so resuming risks double-sends — FAILED with the payloads stripped is the
+ * Fail a session's unfinished (PENDING or PROCESSING) batches after the session was taken over from a
+ * node whose lease lapsed: a batch that node saved but never picked up is as orphaned as one it was
+ * running. Same policy as the boot reaper and for the same reason: the dead node's already-sent
+ * messages are unknowable, so resuming risks double-sends: FAILED with the payloads stripped is the
* honest terminal state, and the caller can re-issue the batch knowingly.
*
- * A batch this process is running itself is not orphaned: the adoption claims the session before
- * its engine finishes initializing, so a bulk request routed here in that window starts a batch
- * that is already PROCESSING by the time the caller reaps. Only the dead holder's batches go.
+ * Runs after this process claims the session from, or releases the claim of, such a node (see
+ * onModuleInit). A batch this process created or is still running is not orphaned, as when it held
+ * the session before or still runs a stale engine for it: createBatch registers every batch before
+ * its row is written, so only the lapsed node's batches go.
*/
async reapProcessingBatches(sessionId: string, reason: string): Promise<number> {
- const processing = (
- await this.batchRepository.find({ where: { status: BatchStatus.PROCESSING, sessionId } })
+ const unfinished = (
+ await this.batchRepository.find({
+ where: { status: In([BatchStatus.PENDING, BatchStatus.PROCESSING]), sessionId },
+ })
).filter(batch => !this.processingBatches.has(batch.id));
let failed = 0;
- for (const batch of processing) {
+ for (const batch of unfinished) {
if (await this.failOrphanedBatch(batch)) failed++;
}
if (failed > 0) {
- this.logger.warn(`Marked ${failed} PROCESSING batch(es) FAILED for session ${sessionId} (${reason})`);
+ this.logger.warn(`Marked ${failed} unfinished batch(es) FAILED for session ${sessionId} (${reason})`);
}
return failed;
}
@@ -281,6 +295,9 @@ export class BulkMessageService implements OnApplicationBootstrap {
};
const batch = this.batchRepository.create({
+ // Assigned here rather than by the database, so the batch is registered below before its row
+ // exists anywhere.
+ id: randomUUID(),
batchId,
sessionId,
status: BatchStatus.PENDING,
@@ -291,6 +308,10 @@ export class BulkMessageService implements OnApplicationBootstrap {
currentIndex: 0,
});
+ // Registered before the row exists, so a reap for this session (a claim that took the session
+ // from another holder) can never read this process's own batch unregistered. A cancel can only
+ // find the row once it is committed, and the false it sets then stands for the pickup check.
+ this.processingBatches.set(batch.id, true);
// Reserve synchronously in the same turn as the cap check. There is deliberately no await between
// them, so a burst cannot all observe the same stale count and overshoot the ceiling.
this.inFlightBatches++;
@@ -298,6 +319,7 @@ export class BulkMessageService implements OnApplicationBootstrap {
await this.batchRepository.save(batch);
} catch (error) {
this.inFlightBatches--;
+ this.processingBatches.delete(batch.id);
// Two concurrent creates with the same caller-supplied batchId both pass the read above; the
// unique index decides, and the loser gets the same 400 as the sequential case.
if (isUniqueViolation(error)) {
@@ -409,7 +431,7 @@ export class BulkMessageService implements OnApplicationBootstrap {
throw error;
} finally {
if (reserved) this.inFlightBatches--;
- if (batch) this.processingBatches.delete(batch.id);
+ this.processingBatches.delete(batchDbId);
}
}
@@ -427,19 +449,20 @@ export class BulkMessageService implements OnApplicationBootstrap {
await this.finalizeBatch(batch, state);
}
- /** Returns false when a committed cancel won the guarded start UPDATE — send nothing. */
+ /** Returns false when the batch left PENDING before this start UPDATE, so nothing is sent. */
private async markBatchProcessing(batch: MessageBatch): Promise<boolean> {
- // Transition to PROCESSING with the guard IN the UPDATE: it only lands while the stored status
- // is not CANCELLED, so a cancel that already committed (any process) can never be overwritten
- // back to PROCESSING. Zero affected rows = cancel-before-start won; send nothing.
+ // Transition to PROCESSING with the guard IN the UPDATE: it only lands while the stored status is
+ // still PENDING, the one status a run starts from. A cancel that already committed (any process)
+ // or a reap that failed the batch after another node adopted its session can never be overwritten
+ // back to PROCESSING. Zero affected rows means send nothing.
batch.status = BatchStatus.PROCESSING;
batch.startedAt = new Date();
const started = await this.batchRepository.update(
- { id: batch.id, status: Not(BatchStatus.CANCELLED) },
+ { id: batch.id, status: BatchStatus.PENDING },
{ status: BatchStatus.PROCESSING, startedAt: batch.startedAt },
);
if (!started.affected) {
- this.logger.log(`Batch ${batch.batchId} was cancelled before processing started; nothing was sent`);
+ this.logger.log(`Batch ${batch.batchId} was cancelled or failed before processing started; nothing was sent`);
return false;
}
return true;
diff --git a/src/modules/session/session-ownership.service.spec.ts b/src/modules/session/session-ownership.service.spec.ts
index 72763a415..4dac9be61 100644
--- a/src/modules/session/session-ownership.service.spec.ts
+++ b/src/modules/session/session-ownership.service.spec.ts
@@ -101,6 +101,90 @@ describe('SessionOwnershipService', () => {
});
});
+ // What a node whose lease lapsed left unfinished (its bulk batches) is failed once this process takes
+ // the session over, by claiming it or by releasing that node's claim.
+ describe('taking a session over from a lapsed node', () => {
+ const settle = (): Promise<void> => new Promise(resolve => setImmediate(resolve));
+ const lapsedBy = (nodeId: string): Partial<Session> => ({
+ nodeId,
+ claimedAt: new Date(Date.now() - 120_000),
+ leaseExpiresAt: new Date(Date.now() - 60_000),
+ });
+ const liveBy = (nodeId: string): Partial<Session> => ({
+ nodeId,
+ claimedAt: new Date(),
+ leaseExpiresAt: new Date(Date.now() + 60_000),
+ });
+
+ // A released row is not taken over: the node that released it may still be finishing its own
+ // batches, and failing them under it would leave two writers on one batch row.
+ it('runs the adoption handler for a claim of a lapsed foreign lease only', async () => {
+ const handler = jest.fn().mockResolvedValue(undefined);
+ const node = service('node-b');
+ node.onAdoption(handler);
+ const own = await seed(lapsedBy('node-b'));
+ const live = await seed(liveBy('node-c'));
+ const released = await seed();
+ const orphaned = await seed(lapsedBy('node-a'));
+
+ await expect(node.claim(own.id)).resolves.toBe(true);
+ await expect(node.claim(live.id)).resolves.toBe(false);
+ await expect(node.claim(released.id)).resolves.toBe(true);
+ await settle();
+ expect(handler).not.toHaveBeenCalled();
+
+ await expect(node.claim(orphaned.id)).resolves.toBe(true);
+ await settle();
+ expect(handler.mock.calls).toEqual([[orphaned.id]]);
+ });
+
+ // A stop of a session whose holder died clears that holder's claim, and a later start then finds a
+ // released row; the release is where the takeover happens.
+ it('runs the adoption handler when a release clears a lapsed foreign claim, and only then', async () => {
+ const handler = jest.fn().mockResolvedValue(undefined);
+ const node = service('node-b');
+ node.onAdoption(handler);
+ const own = await seed(liveBy('node-b'));
+ const live = await seed(liveBy('node-c'));
+ const orphaned = await seed(lapsedBy('node-a'));
+
+ await node.release(own.id);
+ await node.release(live.id);
+ await settle();
+ expect(handler).not.toHaveBeenCalled();
+ expect((await sessions.findOneByOrFail({ id: own.id })).nodeId).toBeNull();
+ expect((await sessions.findOneByOrFail({ id: live.id })).nodeId).toBe('node-c');
+
+ await node.release(orphaned.id);
+ await settle();
+ expect(handler.mock.calls).toEqual([[orphaned.id]]);
+ expect((await sessions.findOneByOrFail({ id: orphaned.id })).nodeId).toBeNull();
+ });
+
+ // Awaiting the handler inside claim() held a start between its claim and the moment it counted as
+ // starting; a stop landing then released the claim and the engine launched on a row nobody held.
+ it('returns from the claim without waiting for the adoption handler', async () => {
+ const node = service('node-b');
+ const handler = jest.fn(() => new Promise<void>(() => undefined));
+ node.onAdoption(handler);
+ const session = await seed(lapsedBy('node-a'));
+
+ await expect(node.claim(session.id)).resolves.toBe(true);
+ await settle();
+ expect(handler).toHaveBeenCalledWith(session.id);
+ });
+
+ it('keeps the claim when the adoption handler fails', async () => {
+ const node = service('node-b');
+ node.onAdoption(() => Promise.reject(new Error('database unavailable')));
+ const session = await seed(lapsedBy('node-a'));
+
+ await expect(node.claim(session.id)).resolves.toBe(true);
+ await settle();
+ expect((await sessions.findOneByOrFail({ id: session.id })).nodeId).toBe('node-b');
+ });
+ });
+
describe('releasing', () => {
it('frees the session so a peer can take it without waiting for the lease', async () => {
const session = await seed();
diff --git a/src/modules/session/session-ownership.service.ts b/src/modules/session/session-ownership.service.ts
index 913276da8..e5bc0d92d 100644
--- a/src/modules/session/session-ownership.service.ts
+++ b/src/modules/session/session-ownership.service.ts
@@ -45,6 +45,8 @@ export class SessionOwnershipService {
private readonly claimGen = new Map<string, number>();
/** Notified when a renewal proves this process no longer holds sessions it thought it did. */
private onLeaseLost?: (sessionIds: string[]) => Promise<void> | void;
+ /** Notified when this process takes a session over from a node whose lease lapsed. See onAdoption. */
+ private onAdopted?: (sessionId: string) => Promise<unknown>;
/**
* How many callers are currently telling renew() that an empty/blank result is NOT evidence of
@@ -127,6 +129,14 @@ export class SessionOwnershipService {
async claim(sessionId: string): Promise<boolean> {
const now = new Date();
const leaseExpiresAt = new Date(now.getTime() + this.leaseTtlMs);
+ // Who held it, read only when an adoption handler is registered. The UPDATE below still decides
+ // the claim; this read only tells whether it took a lapsed lease over from another node.
+ const previous = this.onAdopted
+ ? await this.sessions.findOne({
+ where: { id: sessionId },
+ select: { id: true, nodeId: true, leaseExpiresAt: true },
+ })
+ : null;
const result = await this.sessions
.createQueryBuilder()
.update(Session)
@@ -150,6 +160,14 @@ export class SessionOwnershipService {
this.owned.add(sessionId);
this.bumpClaimGen(sessionId);
this.lastSeenLease.set(sessionId, leaseExpiresAt.getTime());
+ // A released row (nodeId NULL) is not an adoption: its holder may still be finishing its own
+ // batches, and failing them under it would leave two writers on one batch row.
+ const adopted =
+ previous?.nodeId != null &&
+ previous.nodeId !== this.nodeId &&
+ previous.leaseExpiresAt != null &&
+ previous.leaseExpiresAt < now;
+ if (adopted) this.followAdoption(sessionId);
} else this.logger.warn('Session is held by another node', { sessionId, nodeId: this.nodeId });
return claimed;
}
@@ -167,13 +185,29 @@ export class SessionOwnershipService {
const now = new Date();
this.owned.delete(sessionId);
this.bumpClaimGen(sessionId);
- await this.sessions
+ const cleared = { nodeId: null, claimedAt: null, leaseExpiresAt: null, nodeUrl: null };
+ // Its own claim first, in one statement as before. Only when that matched nothing is a lapsed
+ // claim of another node cleared, and that is a takeover like a claim: the dead holder's
+ // unfinished work is failed too (see onAdoption).
+ const own = await this.sessions
.createQueryBuilder()
.update(Session)
- .set({ nodeId: null, claimedAt: null, leaseExpiresAt: null, nodeUrl: null })
+ .set(cleared)
.where('id = :id', { id: sessionId })
- .andWhere('("nodeId" = :me OR "leaseExpiresAt" < :now)', { me: this.nodeId, now: leaseParam(now) })
+ .andWhere('"nodeId" = :me', { me: this.nodeId })
+ .execute();
+ if ((own.affected ?? 0) > 0) return;
+ const lapsed = await this.sessions
+ .createQueryBuilder()
+ .update(Session)
+ .set(cleared)
+ .where('id = :id', { id: sessionId })
+ .andWhere('"nodeId" IS NOT NULL AND "nodeId" <> :me AND "leaseExpiresAt" < :now', {
+ me: this.nodeId,
+ now: leaseParam(now),
+ })
.execute();
+ if ((lapsed.affected ?? 0) > 0) this.followAdoption(sessionId);
}
private bumpClaimGen(sessionId: string): void {
@@ -221,6 +255,36 @@ export class SessionOwnershipService {
this.onLeaseLost = handler;
}
+ /**
+ * Register what to do when this process takes a session over from a node whose lease lapsed, by
+ * claiming it (an explicit start, boot auto-start, the takeover sweep) or by releasing that node's
+ * claim (a stop of such a session). Whatever that node left unfinished for the
+ * session can no longer finish there. A session its holder released is not taken over: that holder
+ * may still be finishing its own work.
+ */
+ onAdoption(handler: (sessionId: string) => Promise<unknown>): void {
+ this.onAdopted = handler;
+ }
+
+ /**
+ * Run the adoption handler off the caller's path. Awaiting it inside claim() would hold the start
+ * between its claim and the moment it counts as starting, and a stop landing in that window would
+ * release the claim and let the engine launch on a row nobody holds. A failure is logged; the claim
+ * or release stands either way.
+ */
+ private followAdoption(sessionId: string): void {
+ const handler = this.onAdopted;
+ if (!handler) return;
+ void Promise.resolve()
+ .then(() => handler(sessionId))
+ .catch((error: unknown) =>
+ this.logger.warn('Session adoption follow-up failed', {
+ sessionId,
+ error: error instanceof Error ? error.message : String(error),
+ }),
+ );
+ }
+
/**
* Tell renew() that "this node no longer holds the row" is currently uninformative, and get back
* the release. Held by a replace-all data import across its whole transaction.
diff --git a/src/modules/takeover/session-takeover.service.spec.ts b/src/modules/takeover/session-takeover.service.spec.ts
index d90054373..20ddc042f 100644
--- a/src/modules/takeover/session-takeover.service.spec.ts
+++ b/src/modules/takeover/session-takeover.service.spec.ts
@@ -4,7 +4,6 @@ import { SessionStoppedException } from '../session/session-engine-controls';
import { Session, SessionStatus } from '../session/entities/session.entity';
import type { SessionService } from '../session/session.service';
import type { SessionOwnershipService } from '../session/session-ownership.service';
-import type { BulkMessageService } from '../message/bulk-message.service';
import type { ConfigService } from '@nestjs/config';
import type { ShutdownService } from '../../common/services/shutdown.service';
@@ -12,7 +11,8 @@ import type { ShutdownService } from '../../common/services/shutdown.service';
* The sweep is the retry that boot auto-start never had: both live incidents (a container recreate
* landing inside the old identity's lease, and a crashed peer) left sessions sitting disconnected
* until a manual POST /start. What has to hold: only lapsed-lease sessions worth resuming are
- * started, a lost claim race is a non-event, and adopting a session reconciles its stuck batches.
+ * started, and a lost claim race is a non-event. The claim on the start path fails the previous
+ * holder's stuck batches (see the ownership and bulk message specs).
*/
describe('SessionTakeoverService', () => {
const lapsed = (over: Partial<Session> = {}): Session =>
@@ -31,11 +31,9 @@ describe('SessionTakeoverService', () => {
): {
svc: SessionTakeoverService;
start: jest.Mock;
- reap: jest.Mock;
markLapsedDisconnected: jest.Mock;
} => {
const start = opts.startImpl ?? jest.fn().mockResolvedValue({});
- const reap = jest.fn().mockResolvedValue(0);
const markLapsedDisconnected = jest.fn().mockResolvedValue([]);
const config = {
get: (key: string, def?: unknown) =>
@@ -50,10 +48,9 @@ describe('SessionTakeoverService', () => {
lapsedHeldByOthers: jest.fn().mockResolvedValue(rows),
leaseTtlMs: 60_000,
} as unknown as SessionOwnershipService,
- { reapProcessingBatches: reap } as unknown as BulkMessageService,
config,
);
- return { svc, start, reap, markLapsedDisconnected };
+ return { svc, start, markLapsedDisconnected };
};
afterEach(() => {
@@ -67,13 +64,12 @@ describe('SessionTakeoverService', () => {
// leaving it unclaimable by any peer until the lease lapsed.
describe('a sweep racing shutdown', () => {
it('adopts nothing once the module is being destroyed', async () => {
- const { svc, start, reap } = build([lapsed({ name: 'a' }), lapsed({ name: 'b' })]);
+ const { svc, start } = build([lapsed({ name: 'a' }), lapsed({ name: 'b' })]);
svc.onModuleDestroy();
await svc.sweep();
expect(start).not.toHaveBeenCalled();
- expect(reap).not.toHaveBeenCalled();
});
it('stops adopting the rest of the batch when shutdown begins mid-sweep', async () => {
@@ -102,13 +98,12 @@ describe('SessionTakeoverService', () => {
});
});
- it('adopts a lapsed authenticated session and reconciles its stuck batches', async () => {
- const { svc, start, reap } = build([lapsed({ name: 'a' })]);
+ it('adopts a lapsed authenticated session through the ordinary start path', async () => {
+ const { svc, start } = build([lapsed({ name: 'a' })]);
await svc.sweep();
expect(start).toHaveBeenCalledWith('id-a');
- expect(reap).toHaveBeenCalledWith('id-a', expect.stringContaining('lapsed'));
});
it('skips sessions not worth resuming: unauthenticated, mid-pairing, or operator-flagged failed', async () => {
@@ -134,28 +129,25 @@ describe('SessionTakeoverService', () => {
expect(start.mock.calls).toEqual([['id-dropped']]);
});
- it('a lost claim race is a non-event: no batch reconcile, and the next candidate still starts', async () => {
+ it('a lost claim race is a non-event: the next candidate still starts', async () => {
jest.useFakeTimers();
const start = jest.fn().mockRejectedValueOnce(new ConflictException('held elsewhere')).mockResolvedValueOnce({});
- const { svc, reap } = build([lapsed({ name: 'raced' }), lapsed({ name: 'ours' })], { startImpl: start });
+ const { svc } = build([lapsed({ name: 'raced' }), lapsed({ name: 'ours' })], { startImpl: start });
const sweep = svc.sweep();
await jest.advanceTimersByTimeAsync(2100); // the inter-launch stagger
await sweep;
- expect(start).toHaveBeenCalledTimes(2);
- expect(reap).toHaveBeenCalledTimes(1);
- expect(reap).toHaveBeenCalledWith('id-ours', expect.any(String));
+ expect(start.mock.calls).toEqual([['id-raced'], ['id-ours']]);
});
it('a session stopped after the sweep read it is logged as skipped, not as a lost claim race', async () => {
const start = jest.fn().mockRejectedValueOnce(new SessionStoppedException('Session id-halted was stopped'));
- const { svc, reap } = build([lapsed({ name: 'halted' })], { startImpl: start });
+ const { svc } = build([lapsed({ name: 'halted' })], { startImpl: start });
const debug = jest.spyOn((svc as unknown as { logger: { debug: jest.Mock } }).logger, 'debug');
await svc.sweep();
- expect(reap).not.toHaveBeenCalled();
expect(debug.mock.calls.map(([message]) => message as string)).toEqual([
'Session halted skipped: stopped by an operator',
]);
@@ -261,7 +253,6 @@ describe('SessionTakeoverService', () => {
const svc = new SessionTakeoverService(
{ start, markLapsedDisconnected: jest.fn().mockResolvedValue([]) } as unknown as SessionService,
{ lapsedHeldByOthers: ownershipCalls, leaseTtlMs: 60_000 } as unknown as SessionOwnershipService,
- { reapProcessingBatches: jest.fn().mockResolvedValue(0) } as unknown as BulkMessageService,
config,
);
@@ -302,7 +293,6 @@ describe('SessionTakeoverService', () => {
const svc = new SessionTakeoverService(
{ start } as unknown as SessionService,
{ lapsedHeldByOthers: ownershipCalls } as unknown as SessionOwnershipService,
- { reapProcessingBatches: jest.fn().mockResolvedValue(0) } as unknown as BulkMessageService,
config,
{ isShuttingDown: () => true } as unknown as ShutdownService,
);
@@ -324,7 +314,6 @@ describe('SessionTakeoverService', () => {
const svc = new SessionTakeoverService(
{ start: jest.fn().mockResolvedValue(undefined) } as unknown as SessionService,
{ lapsedHeldByOthers: ownershipCalls } as unknown as SessionOwnershipService,
- { reapProcessingBatches: jest.fn().mockResolvedValue(0) } as unknown as BulkMessageService,
config,
);
diff --git a/src/modules/takeover/session-takeover.service.ts b/src/modules/takeover/session-takeover.service.ts
index 9f339d8ed..7409d8a0f 100644
--- a/src/modules/takeover/session-takeover.service.ts
+++ b/src/modules/takeover/session-takeover.service.ts
@@ -8,7 +8,6 @@ import { SessionOwnershipService } from '../session/session-ownership.service';
import { ShutdownService } from '../../common/services/shutdown.service';
import { SessionService } from '../session/session.service';
import { SessionStoppedException } from '../session/session-engine-controls';
-import { BulkMessageService } from '../message/bulk-message.service';
/**
* Statuses worth adopting from a lapsed node. They all mean "an engine was (or should be) running".
@@ -58,9 +57,9 @@ const STRANDED_LEASE_TTL_MULTIPLE = 2;
* for lapsed-lease sessions and starts them here through the ordinary start path, so the claim
* stays race-safe against peers doing the same.
*
- * Lives in its own module (not SessionModule) because adopting a session also reconciles its
- * in-flight bulk batches via BulkMessageService — which sits in MessageModule, which imports
- * SessionModule; importing it back from SessionModule would close the cycle.
+ * The lapsed holder's unfinished bulk batches are failed after the claim, by the adoption handler
+ * BulkMessageService registers with SessionOwnershipService, the same as for an explicit POST
+ * /start, so the sweep only starts sessions.
*/
@Injectable()
export class SessionTakeoverService implements OnApplicationBootstrap, OnModuleDestroy {
@@ -79,7 +78,6 @@ export class SessionTakeoverService implements OnApplicationBootstrap, OnModuleD
constructor(
private readonly sessionService: SessionService,
private readonly ownership: SessionOwnershipService,
- private readonly bulkMessages: BulkMessageService,
@Optional()
private readonly configService?: ConfigService,
// The drain signal, not module destruction. `onModuleDestroy` runs at app.close(), AFTER the
@@ -169,9 +167,6 @@ export class SessionTakeoverService implements OnApplicationBootstrap, OnModuleD
fromNode: session.nodeId,
action: 'session_takeover',
});
- // The dead node's in-flight batches can never complete; surface them as FAILED now rather
- // than leaving them stuck in PROCESSING until some node happens to reboot.
- await this.bulkMessages.reapProcessingBatches(session.id, 'session adopted from a lapsed node');
} catch (error) {
if (error instanceof SessionStoppedException) {
// Stopped between the sweep's read and this start; the start refused it, as it should.
diff --git a/src/modules/takeover/takeover.module.ts b/src/modules/takeover/takeover.module.ts
index d8a73d584..5d5c88985 100644
--- a/src/modules/takeover/takeover.module.ts
+++ b/src/modules/takeover/takeover.module.ts
@@ -1,15 +1,14 @@
import { Module } from '@nestjs/common';
import { SessionModule } from '../session/session.module';
-import { MessageModule } from '../message/message.module';
import { SessionTakeoverService } from './session-takeover.service';
/**
- * Sits ABOVE both SessionModule and MessageModule on purpose: the takeover sweep starts sessions
- * (SessionService) and reconciles their bulk batches (BulkMessageService), and MessageModule
- * already imports SessionModule — so this is the lowest place both are reachable without a cycle.
+ * The takeover sweep starts sessions through SessionService. A claim on that path has the lapsed
+ * holder's unfinished bulk batches failed through the ownership adoption handler, so nothing here
+ * needs MessageModule.
*/
@Module({
- imports: [SessionModule, MessageModule],
+ imports: [SessionModule],
providers: [SessionTakeoverService],
})
export class TakeoverModule {}