Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(session): refuse engine starts once shutdown begins (#1795)

Yudhi Armyndharis9c236931acd4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e5ab61ad1..f3b73688f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- Refuse session starts with `503` once shutdown begins, and leave a start that shutdown interrupts for the next boot instead of marking it failed.
- Retire the pending webhook outbox record when a redrive delivers, so the sweep does not send the event again.
- Stop dashboard multi-page loads and a pending chat mark-as-read from sending requests after logout.
- Keep a new dashboard login signed in when a request sent with the previous API key fails afterwards.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 742cbacba..982579e1a 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -666,7 +666,7 @@ Returned via `transformSession`. Status typically transitions to `initializing`
A session stopped with `POST /stop` or `POST /force-kill` is marked to run again only once the start gets past the `400` and `409` refusals below; a refused start leaves the stop in place.
-**Errors:** `400` session already started / already starting, or this node already holds `MAX_CONCURRENT_SESSIONS` slots, counting running and starting engines and sessions waiting to relaunch after a failed reconnect (`Maximum concurrent sessions reached (N)`, answered only after the `404` and the `409` for a session running on another node; the session is not started, so retry once a slot frees or on a node with room) · `401` · `403` · `404` not found · `409` credential teardown for the same session name still in flight (retryable; body carries `code: 'SESSION_NAME_TEARDOWN_PENDING'`; no destructive side effect runs before the refusal — a retry after cleanup settles proceeds), or another node currently holds this session's engine (refused before any engine is launched), or, with no `code`, a `POST /stop`, a `POST /force-kill` that found a running engine to kill, or a data import (`stopOrphans`) of this session that began or finished while the start was waiting, or a delete of it still running (the start yields and launches nothing; after a stop or force-kill a new `POST /start` clears it and starts the session, after a delete or an import that removed the session the retry answers `404`) · `504` the engine did not finish starting within its timeout (WhatsApp Web or the network unreachable, a stalled browser or resource limit, or, on whatsapp-web.js, an unreachable `proxyUrl`); the engine is torn down, so the start can be retried once the cause is addressed
+**Errors:** `400` session already started / already starting, or this node already holds `MAX_CONCURRENT_SESSIONS` slots, counting running and starting engines and sessions waiting to relaunch after a failed reconnect (`Maximum concurrent sessions reached (N)`, answered only after the `404` and the `409` for a session running on another node; the session is not started, so retry once a slot frees or on a node with room) · `401` · `403` · `404` not found · `409` credential teardown for the same session name still in flight (retryable; body carries `code: 'SESSION_NAME_TEARDOWN_PENDING'`; no destructive side effect runs before the refusal — a retry after cleanup settles proceeds), or another node currently holds this session's engine (refused before any engine is launched), or, with no `code`, a `POST /stop`, a `POST /force-kill` that found a running engine to kill, or a data import (`stopOrphans`) of this session that began or finished while the start was waiting, or a delete of it still running (the start yields and launches nothing; after a stop or force-kill a new `POST /start` clears it and starts the session, after a delete or an import that removed the session the retry answers `404`) · `503` the gateway is shutting down, or the engine's browser or socket connection died during launch and the one automatic retry failed too; no engine is left running, so the start can be retried · `504` the engine did not finish starting within its timeout (WhatsApp Web or the network unreachable, a stalled browser or resource limit, or, on whatsapp-web.js, an unreachable `proxyUrl`); the engine is torn down, so the start can be retried once the cause is addressed
#### POST /api/sessions/:sessionId/stop
diff --git a/openapi.json b/openapi.json
index 7de2f8a62..cee945e2e 100644
--- a/openapi.json
+++ b/openapi.json
@@ -1278,6 +1278,16 @@
}
}
},
+ "503": {
+ "description": "The gateway is shutting down, or the engine's browser or socket connection died during launch and the one automatic retry failed too; no engine is left running, so the start can be retried.",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ErrorResponse"
+ }
+ }
+ }
+ },
"504": {
"description": "The engine did not finish starting within its timeout (WhatsApp Web or the network unreachable, a stalled browser or resource limit, or, on whatsapp-web.js, an unreachable proxyUrl or the auth timeout); the engine is torn down and the start can be retried.",
"content": {
diff --git a/src/modules/session/session-engine-controls.ts b/src/modules/session/session-engine-controls.ts
index 1b706b16d..7b16d6bda 100644
--- a/src/modules/session/session-engine-controls.ts
+++ b/src/modules/session/session-engine-controls.ts
@@ -4,6 +4,7 @@ import {
ConflictException,
HttpStatus,
NotFoundException,
+ ServiceUnavailableException,
} from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { Repository, DataSource } from 'typeorm';
@@ -42,6 +43,13 @@ import {
/** A start refused because an operator stopped the session. Same 409 on the wire as any conflict. */
export class SessionStoppedException extends ConflictException {}
+/** A start refused because the gateway is shutting down. Retryable once it is back. */
+export class ServerShuttingDownException extends ServiceUnavailableException {
+ constructor() {
+ super('Server is shutting down');
+ }
+}
+
/**
* The stop marks. Each add() stamps the id afresh, even one already marked, so start() can tell a mark
* set while it waited from one that was already there when it began.
@@ -114,6 +122,8 @@ export interface SessionEngineControlsHost {
updateStatus(id: string, status: SessionStatus): Promise<void>;
/** Ownership gate, same contract as SessionEngineWiringHost.ownsSession. */
ownsSession(id: string): boolean;
+ /** True once shutdown() has begun; the lifecycle never reopens. */
+ isClosed(): boolean;
stoppingSessions: StopMarks;
/** The engine each operator-initiated teardown is retiring; see the lifecycle field of the same name. */
operatorTeardowns: Map<string, IWhatsAppEngine>;
@@ -338,8 +348,10 @@ export class SessionEngineControls {
// Fenced on ownership like the engine callbacks: initializeEngine can await a slow
// Chromium launch for minutes, and this node's lease can lapse and be taken over inside
// that window. FAILED is excluded from the boot reset AND from the takeover sweep, so
- // writing it onto a row a peer now owns strands the session on every node.
- if (this.host.ownsSession(id)) {
+ // writing it onto a row a peer now owns strands the session on every node. Skipped during
+ // shutdown too: an init that rejects then usually does so because shutdown destroyed its
+ // engine, and the INITIALIZING row is reset by the next boot, where FAILED would not be.
+ if (this.host.ownsSession(id) && !this.host.isClosed()) {
await this.host.updateStatus(id, SessionStatus.FAILED).catch(() => undefined);
}
}
diff --git a/src/modules/session/session-engine-lifecycle.service.ts b/src/modules/session/session-engine-lifecycle.service.ts
index e00a9c3c9..d7851813f 100644
--- a/src/modules/session/session-engine-lifecycle.service.ts
+++ b/src/modules/session/session-engine-lifecycle.service.ts
@@ -34,7 +34,12 @@ import {
SessionEngineEventWiring,
SessionEngineWiringHost,
} from './session-engine-event-wiring';
-import { SessionEngineControls, StopMarks, type StopHooks } from './session-engine-controls';
+import {
+ SessionEngineControls,
+ ServerShuttingDownException,
+ StopMarks,
+ type StopHooks,
+} from './session-engine-controls';
import { SessionOwnershipService, nodeOwnsSession } from './session-ownership.service';
/**
@@ -226,6 +231,9 @@ export class SessionEngineLifecycle {
private readonly wiringHost: SessionEngineWiringHost;
private readonly controls: SessionEngineControls;
+ // Set by shutdown() in the same tick as its registry snapshot; initializeEngine refuses from then on.
+ private closed = false;
+
// Reconnection state per session
private reconnectStates: Map<string, ReconnectState> = new Map();
@@ -408,6 +416,7 @@ export class SessionEngineLifecycle {
// call-ins are NON-async passthrough closures (the Task-1 delegate rule above).
this.controls = new SessionEngineControls({
ownsSession: (id: string) => this.ownsSession(id),
+ isClosed: () => this.closed,
sessionRepository: this.sessionRepository,
engineFactory: this.engineFactory,
engines: this.engineRegistry,
@@ -468,6 +477,7 @@ export class SessionEngineLifecycle {
/** Delegate: SessionEngineControls.shutdown. */
shutdown(): Promise<void> {
+ this.closed = true;
return this.controls.shutdown();
}
@@ -630,6 +640,12 @@ export class SessionEngineLifecycle {
}
private async initializeEngine(id: string, session: Session): Promise<void> {
+ // shutdown() destroys only the engines registered when it starts. A start or a transient retry
+ // still on its way here would register one nothing tears down, so it fails instead; its caller
+ // releases the claim. A 503 is not a transient launch failure, so it is not retried.
+ if (this.closed) {
+ throw new ServerShuttingDownException();
+ }
// A stop that landed before this engine exists had nothing to tear down and already wrote its
// DISCONNECTED; registering now would overwrite it with INITIALIZING and then retire. start()
// clears its own mark and executeReconnect checks it on entry, so a mark seen here always came
@@ -695,8 +711,10 @@ export class SessionEngineLifecycle {
// engine.initialize() below — an intervening await would re-open the retirement window, and it is
// also why ONE isLiveEngine check is enough: the two guards are separated by a synchronous Set
// lookup, so nothing can swap the engine between them (a second, identical check used to sit
- // after the stop mark and could never disagree with this one).
- if (!this.isLiveEngine(id, engine)) {
+ // after the stop mark and could never disagree with this one). `closed` covers an engine that
+ // shutdown() is already destroying: it stays registered until every destroy settles. Returned,
+ // not thrown: start()'s failure path would force-destroy the engine shutdown is destroying.
+ if (this.closed || !this.isLiveEngine(id, engine)) {
return;
}
if (this.stoppingSessions.has(id)) {
diff --git a/src/modules/session/session.controller.ts b/src/modules/session/session.controller.ts
index eda224a0f..d24f88574 100644
--- a/src/modules/session/session.controller.ts
+++ b/src/modules/session/session.controller.ts
@@ -327,6 +327,12 @@ export class SessionController {
'clears it and starts the session; after a delete, or an import that removed the session, the ' +
'retry answers 404.',
})
+ @ApiResponse({
+ status: 503,
+ description:
+ "The gateway is shutting down, or the engine's browser or socket connection died during launch and " +
+ 'the one automatic retry failed too; no engine is left running, so the start can be retried.',
+ })
@ApiResponse({
status: 504,
description:
diff --git a/src/modules/session/session.service.spec.ts b/src/modules/session/session.service.spec.ts
index f45eedcd7..47ed67d64 100644
--- a/src/modules/session/session.service.spec.ts
+++ b/src/modules/session/session.service.spec.ts
@@ -18,7 +18,7 @@ import { EngineNotSupportedError } from '../../common/errors/engine-not-supporte
import { ConfigService } from '@nestjs/config';
import { SessionService, AUTOSTART_THROTTLE_MS } from './session.service';
import { SessionOwnershipService } from './session-ownership.service';
-import { SessionStoppedException } from './session-engine-controls';
+import { ServerShuttingDownException, SessionStoppedException } from './session-engine-controls';
import { decideReconnect, STABLE_READY_MS } from './reconnect-policy';
import { ACK_RECONCILE_DELAY_MS } from './message-projector.service';
import { SessionEngineLifecycle, type ReconnectState } from './session-engine-lifecycle.service';
@@ -2200,6 +2200,157 @@ describe('SessionService', () => {
expect(await nodeIdOf(row.id)).toBeNull();
});
+ // The retry would re-claim the row after shutdown released it and launch an engine nothing tears down.
+ it('start() does not retry a transient failure once shutdown ran during the delay', async () => {
+ const row = await seed();
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession({ id: row.id }));
+ mockEngine.initialize.mockRejectedValueOnce(new EngineTransportError('Protocol error: Target closed'));
+ const claim = jest.spyOn(ownership, 'claim');
+
+ const outcome = service.start(row.id).catch((error: unknown) => error);
+ await until(() => mockEngine.initialize.mock.calls.length === 1 && !lifecycle.isEngineActive(row.id));
+ await service.onModuleDestroy();
+
+ expect(await outcome).toMatchObject({ status: 503 });
+ expect(claim).toHaveBeenCalledTimes(1);
+ expect(mockEngine.initialize).toHaveBeenCalledTimes(1);
+ expect(lifecycle.isEngineActive(row.id)).toBe(false);
+ expect(await nodeIdOf(row.id)).toBeNull();
+ });
+
+ // The first attempt's FAILED is skipped by the boot reset, auto-start and the takeover sweep,
+ // and shutdown refuses the retry that would have overwritten it.
+ describe('a transient failure whose retry shutdown refuses leaves the session for the next boot', () => {
+ const statusOf = async (id: string): Promise<SessionStatus> => (await sessions.findOneByOrFail({ id })).status;
+ const untilFailed = async (id: string): Promise<void> => {
+ for (let i = 0; i < 200 && (await statusOf(id)) !== SessionStatus.FAILED; i++) {
+ await new Promise(resolve => setImmediate(resolve));
+ }
+ expect(await statusOf(id)).toBe(SessionStatus.FAILED);
+ };
+ const arrange = async (): Promise<Session> => {
+ const row = await seed({ status: SessionStatus.DISCONNECTED, phone: '123' });
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession({ id: row.id }));
+ (repository.update as jest.Mock).mockImplementation((...args: Parameters<Repository<Session>['update']>) =>
+ sessions.update(...args),
+ );
+ mockEngine.initialize.mockRejectedValueOnce(new EngineTransportError('Protocol error: Target closed'));
+ return row;
+ };
+ const originalFlag = process.env.AUTO_START_SESSIONS;
+
+ afterEach(() => {
+ if (originalFlag === undefined) delete process.env.AUTO_START_SESSIONS;
+ else process.env.AUTO_START_SESSIONS = originalFlag;
+ });
+
+ it('for an explicit start refused during the delay', async () => {
+ const row = await arrange();
+
+ const outcome = service.start(row.id, { explicit: true }).catch((error: unknown) => error);
+ await untilFailed(row.id);
+ await service.onModuleDestroy();
+
+ expect(await outcome).toBeInstanceOf(ServerShuttingDownException);
+ expect(mockEngine.initialize).toHaveBeenCalledTimes(1);
+ expect(await statusOf(row.id)).toBe(SessionStatus.DISCONNECTED);
+ expect(await nodeIdOf(row.id)).toBeNull();
+ });
+
+ it('for a boot auto-start refused during the delay', async () => {
+ const row = await arrange();
+ process.env.AUTO_START_SESSIONS = 'true';
+ (repository.find as jest.Mock).mockResolvedValue([createMockSession({ id: row.id })]);
+
+ service.onApplicationBootstrap();
+ await untilFailed(row.id);
+ await service.onModuleDestroy();
+
+ expect(mockEngine.initialize).toHaveBeenCalledTimes(1);
+ expect(await statusOf(row.id)).toBe(SessionStatus.DISCONNECTED);
+ expect(await nodeIdOf(row.id)).toBeNull();
+ });
+
+ it('for a retry whose re-claim was pending when shutdown began', async () => {
+ const row = await arrange();
+ const claimNow = ownership.claim.bind(ownership);
+ let releaseClaim: () => void = () => undefined;
+ const reclaiming = new Promise<void>(reached => {
+ jest
+ .spyOn(ownership, 'claim')
+ .mockImplementationOnce(claimNow)
+ .mockImplementationOnce(id => {
+ reached();
+ return new Promise<boolean>(resolve => (releaseClaim = () => resolve(claimNow(id))));
+ });
+ });
+
+ const startingHooks = (): number =>
+ (hookManager.execute as jest.Mock).mock.calls.filter(([event]) => event === 'session:starting').length;
+
+ const outcome = service.start(row.id, { explicit: true }).catch((error: unknown) => error);
+ await reclaiming;
+ expect(await statusOf(row.id)).toBe(SessionStatus.FAILED);
+ expect(startingHooks()).toBe(1);
+ await service.onModuleDestroy();
+ releaseClaim();
+
+ expect(await outcome).toBeInstanceOf(ServerShuttingDownException);
+ expect(startingHooks()).toBe(1);
+ expect(mockEngine.initialize).toHaveBeenCalledTimes(1);
+ expect(await statusOf(row.id)).toBe(SessionStatus.DISCONNECTED);
+ expect(await nodeIdOf(row.id)).toBeNull();
+ });
+
+ it('but not on a row a peer adopted since, whose FAILED is its own', async () => {
+ const row = await arrange();
+
+ const outcome = service.start(row.id, { explicit: true }).catch((error: unknown) => error);
+ await untilFailed(row.id);
+ await sessions.update(row.id, { nodeId: 'node-b', leaseExpiresAt: new Date(Date.now() + 60_000) });
+ await service.onModuleDestroy();
+
+ expect(await outcome).toBeInstanceOf(ServerShuttingDownException);
+ expect(await statusOf(row.id)).toBe(SessionStatus.FAILED);
+ expect(await nodeIdOf(row.id)).toBe('node-b');
+ });
+ });
+
+ it('start() after shutdown released the claims does not claim the row again', async () => {
+ const row = await seed({ status: SessionStatus.DISCONNECTED });
+ const claim = jest.spyOn(ownership, 'claim');
+ await service.onModuleDestroy();
+
+ await expect(service.start(row.id)).rejects.toMatchObject({ status: 503 });
+ expect(claim).not.toHaveBeenCalled();
+ expect(await nodeIdOf(row.id)).toBeNull();
+ });
+
+ // The claim resolves after shutdown released this node's rows; the start must not clear the stop or run the hook.
+ it('start() still waiting on its claim when shutdown begins neither clears a stop nor runs the hook', async () => {
+ const row = await seed({ status: SessionStatus.DISCONNECTED, desiredState: 'stopped' });
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession({ id: row.id, desiredState: 'stopped' }));
+ const claimNow = ownership.claim.bind(ownership);
+ let releaseClaim: () => void = () => undefined;
+ const claim = jest
+ .spyOn(ownership, 'claim')
+ .mockImplementationOnce(id => new Promise<boolean>(resolve => (releaseClaim = () => resolve(claimNow(id)))));
+
+ const outcome = service.start(row.id, { explicit: true }).catch((error: unknown) => error);
+ await until(() => claim.mock.calls.length === 1);
+ await service.onModuleDestroy();
+ releaseClaim();
+
+ expect(await outcome).toMatchObject({ status: 503 });
+ expect(hookManager.execute).not.toHaveBeenCalled();
+ expect((repository.update as jest.Mock).mock.calls).not.toContainEqual([
+ { id: row.id, desiredState: 'stopped' },
+ { desiredState: null },
+ ]);
+ expect(engineFactory.create).not.toHaveBeenCalled();
+ expect(await nodeIdOf(row.id)).toBeNull();
+ });
+
// The earlier stop's mark is still set when the row read fails, before start() clears it.
it('start() still retries a transient failure when only an earlier stop() left its mark', async () => {
const row = await seed();
@@ -8548,6 +8699,132 @@ describe('SessionService', () => {
expect(mockEngine.destroy).toHaveBeenCalled();
expect(service.isActive('sess-uuid-1')).toBe(false);
});
+
+ it('refuses a start still on its way to the engine when shutdown tears the engines down', async () => {
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession());
+ let releaseHook: () => void = () => undefined;
+ const hookHeld = new Promise<void>(resolve => {
+ (hookManager.execute as jest.Mock).mockImplementationOnce(
+ (_e: string, data: unknown) =>
+ new Promise(done => {
+ releaseHook = () => done({ continue: true, data });
+ resolve();
+ }),
+ );
+ });
+
+ const start = service.start('sess-uuid-1');
+ await hookHeld;
+ await service.onModuleDestroy();
+ releaseHook();
+
+ await expect(start).rejects.toMatchObject({ status: 503 });
+ expect(engineFactory.create).not.toHaveBeenCalled();
+ expect(service.isActive('sess-uuid-1')).toBe(false);
+ });
+
+ it('refuses a start that arrives after shutdown before it clears a stop or runs the hook', async () => {
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession({ desiredState: 'stopped' }));
+ await service.onModuleDestroy();
+
+ await expect(service.start('sess-uuid-1', { explicit: true })).rejects.toMatchObject({ status: 503 });
+ expect(hookManager.execute).not.toHaveBeenCalled();
+ expect((repository.update as jest.Mock).mock.calls).not.toContainEqual([
+ { id: 'sess-uuid-1', desiredState: 'stopped' },
+ { desiredState: null },
+ ]);
+ expect(engineFactory.create).not.toHaveBeenCalled();
+ });
+
+ it('does not initialize an engine whose first status write settles while shutdown destroys it', async () => {
+ const flush = async (): Promise<void> => {
+ for (let i = 0; i < 20; i++) await new Promise(resolve => setImmediate(resolve));
+ };
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession());
+ let releaseWrite: () => void = () => undefined;
+ (repository.update as jest.Mock).mockImplementation((_id: unknown, patch: { status?: SessionStatus }) =>
+ patch?.status === SessionStatus.INITIALIZING
+ ? new Promise(resolve => (releaseWrite = () => resolve({ affected: 1 })))
+ : Promise.resolve({ affected: 1 }),
+ );
+ let releaseDestroy: () => void = () => undefined;
+ mockEngine.destroy.mockImplementationOnce(() => new Promise<void>(resolve => (releaseDestroy = resolve)));
+
+ const start = service.start('sess-uuid-1');
+ await flush();
+ expect(engineFactory.create).toHaveBeenCalledTimes(1);
+ const shutdown = service.onModuleDestroy();
+ await flush();
+ expect(mockEngine.destroy).toHaveBeenCalledTimes(1);
+ releaseWrite();
+ await flush();
+ releaseDestroy();
+ await shutdown;
+ await start;
+
+ expect(mockEngine.initialize).not.toHaveBeenCalled();
+ expect(service.isActive('sess-uuid-1')).toBe(false);
+ });
+
+ it('does not initialize a reconnect engine whose first status write settles while shutdown destroys it', async () => {
+ const flush = async (): Promise<void> => {
+ for (let i = 0; i < 20; i++) await new Promise(resolve => setImmediate(resolve));
+ };
+ let releaseWrite: () => void = () => undefined;
+ (repository.update as jest.Mock).mockImplementation((_id: unknown, patch: { status?: SessionStatus }) =>
+ patch?.status === SessionStatus.INITIALIZING
+ ? new Promise(resolve => (releaseWrite = () => resolve({ affected: 1 })))
+ : Promise.resolve({ affected: 1 }),
+ );
+ let releaseDestroy: () => void = () => undefined;
+ mockEngine.destroy.mockImplementationOnce(() => new Promise<void>(resolve => (releaseDestroy = resolve)));
+ const internals = lifecycle as unknown as {
+ reconnectStates: Map<string, ReconnectState>;
+ executeReconnect: (id: string, session: Session, state: ReconnectState) => Promise<void>;
+ };
+ const state: ReconnectState = { attempts: 1, timer: null, maxAttempts: 5, baseDelay: 5000 };
+ internals.reconnectStates.set('sess-uuid-1', state);
+
+ const reconnect = internals.executeReconnect('sess-uuid-1', createMockSession(), state);
+ await flush();
+ expect(engineFactory.create).toHaveBeenCalledTimes(1);
+ const shutdown = service.onModuleDestroy();
+ await flush();
+ expect(mockEngine.destroy).toHaveBeenCalledTimes(1);
+ releaseWrite();
+ await flush();
+ releaseDestroy();
+ await shutdown;
+ await reconnect;
+
+ expect(mockEngine.initialize).not.toHaveBeenCalled();
+ expect(service.isActive('sess-uuid-1')).toBe(false);
+ });
+
+ // FAILED is left out of the boot reset and auto-start, so the session would stay down after a routine restart.
+ it('does not mark a session FAILED when shutdown destroys its engine mid-initialize', async () => {
+ (repository.findOne as jest.Mock).mockResolvedValue(createMockSession());
+ let rejectInit: (error: Error) => void = () => undefined;
+ mockEngine.initialize.mockImplementationOnce(
+ () => new Promise<void>((_resolve, reject) => (rejectInit = reject)),
+ );
+ mockEngine.destroy.mockImplementationOnce(() => {
+ rejectInit(new Error('Target closed'));
+ return Promise.resolve();
+ });
+ const updateStatus = jest.spyOn(lifecycle, 'updateStatus');
+
+ const start = service.start('sess-uuid-1').catch((error: unknown) => error);
+ for (let i = 0; i < 50 && mockEngine.initialize.mock.calls.length === 0; i++) {
+ await new Promise(resolve => setImmediate(resolve));
+ }
+ expect(mockEngine.initialize).toHaveBeenCalledTimes(1);
+ await service.onModuleDestroy();
+
+ expect(await start).toMatchObject({ message: 'Target closed' });
+ expect(updateStatus).not.toHaveBeenCalledWith('sess-uuid-1', SessionStatus.FAILED);
+ expect(service.isActive('sess-uuid-1')).toBe(false);
+ });
});
// ── operator stop intent ──────────────────────────────────────────
@@ -9111,6 +9388,74 @@ describe('SessionService', () => {
);
});
+ it('reports a launch refused by shutdown as abandoned, not as a failed start', async () => {
+ process.env.AUTO_START_SESSIONS = 'true';
+ (repository.find as jest.Mock).mockResolvedValue([{ id: 'a', name: 'A' }]);
+ jest.spyOn(service, 'start').mockImplementationOnce(() => {
+ (service as unknown as { shuttingDown: boolean }).shuttingDown = true;
+ return Promise.reject(new ServerShuttingDownException());
+ });
+ const logger = (service as unknown as { logger: { log: jest.Mock; error: jest.Mock } }).logger;
+ const logError = jest.spyOn(logger, 'error');
+ const logInfo = jest.spyOn(logger, 'log');
+
+ service.onApplicationBootstrap();
+ await autoStartRun();
+
+ expect(logError).not.toHaveBeenCalled();
+ expect(logInfo).toHaveBeenCalledWith(
+ 'Auto-start abandoned for session A: shutting down',
+ expect.objectContaining({ action: 'auto_start_aborted' }),
+ );
+ });
+
+ // onModuleDestroy awaits this loop, so the inter-launch throttle would only delay the exit.
+ it('ends the run without the inter-launch throttle once a launch is refused by shutdown', async () => {
+ process.env.AUTO_START_SESSIONS = 'true';
+ (repository.find as jest.Mock).mockResolvedValue([
+ { id: 'a', name: 'A' },
+ { id: 'b', name: 'B' },
+ ]);
+ const startSpy = jest.spyOn(service, 'start').mockImplementationOnce(() => {
+ (service as unknown as { shuttingDown: boolean }).shuttingDown = true;
+ return Promise.reject(new ServerShuttingDownException());
+ });
+
+ jest.useFakeTimers();
+ try {
+ service.onApplicationBootstrap();
+ let ended = false;
+ void autoStartRun().then(() => (ended = true));
+ await jest.advanceTimersByTimeAsync(AUTOSTART_THROTTLE_MS - 1);
+
+ expect(ended).toBe(true);
+ expect(startSpy).toHaveBeenCalledTimes(1);
+ } finally {
+ jest.useRealTimers();
+ }
+ });
+
+ // EngineTransportError is a 503 too; a real launch failure stays an error even during shutdown.
+ it('reports a transport failure that coincides with shutdown as a failed start', async () => {
+ process.env.AUTO_START_SESSIONS = 'true';
+ (repository.find as jest.Mock).mockResolvedValue([{ id: 'a', name: 'A' }]);
+ jest.spyOn(service, 'start').mockImplementationOnce(() => {
+ (service as unknown as { shuttingDown: boolean }).shuttingDown = true;
+ return Promise.reject(new EngineTransportError('Protocol error: Target closed'));
+ });
+ const logger = (service as unknown as { logger: { error: jest.Mock } }).logger;
+ const logError = jest.spyOn(logger, 'error');
+
+ service.onApplicationBootstrap();
+ await autoStartRun();
+
+ expect(logError).toHaveBeenCalledWith(
+ 'Auto-start failed for session: A',
+ 'Protocol error: Target closed',
+ expect.objectContaining({ action: 'auto_start_failed' }),
+ );
+ });
+
// Nest binds the HTTP listener only after every bootstrap hook settles, and a launch is a
// Chromium start bounded by resolveEngineInitTimeoutMs (>= 60s) with a 2s throttle between
// sessions. Awaiting the loop here kept the port CLOSED for that whole time, so every probe in
diff --git a/src/modules/session/session.service.ts b/src/modules/session/session.service.ts
index 031f3a869..84b533bfe 100644
--- a/src/modules/session/session.service.ts
+++ b/src/modules/session/session.service.ts
@@ -57,7 +57,7 @@ import { HookManager } from '../../core/hooks';
import { LidMappingStoreService } from '../../engine/identity/lid-mapping-store.service';
import { resolveJidCandidates } from '../../engine/identity/jid-candidates';
import { Message } from '../message/entities/message.entity';
-import { SessionStoppedException } from './session-engine-controls';
+import { ServerShuttingDownException, SessionStoppedException } from './session-engine-controls';
// Type-only: the module binds this class to PLUGIN_SESSION_PORT with a `useExisting` alias, which
// TypeScript does not check, so `implements` is what keeps the two in step.
import type { PluginSessionPort } from '../../core/plugins/plugin-host-ports';
@@ -301,6 +301,14 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat
sessionId: session.id,
action: 'auto_start_skipped',
});
+ } else if (error instanceof ServerShuttingDownException) {
+ // The launch in flight when shutdown began, refused so nothing outlives the teardown.
+ this.logger.log(`Auto-start abandoned for session ${session.name}: shutting down`, {
+ sessionId: session.id,
+ action: 'auto_start_aborted',
+ });
+ // Nothing later in the run would start either, and shutdown waits on this loop.
+ return;
} else {
const errorMessage = error instanceof Error ? error.message : 'Unknown error';
this.logger.error(`Auto-start failed for session: ${session.name}`, errorMessage, {
@@ -602,6 +610,9 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat
* the engine start refuses them a row still marked stopped when it reads it.
*/
async start(id: string, { explicit = false }: { explicit?: boolean } = {}): Promise<Session> {
+ // Refused before the claim, the stop clear and the session:starting hook: none of them should run
+ // for a start the exiting process will not keep.
+ this.refuseIfShuttingDown();
// At the cap, refused before the claim: the engine's own cap check runs after it, and its
// refusal releases the claim to no node. A row nobody holds is never adopted by a peer, while a
// lapsed lease left where it is gets taken over by one with room. The engine check stays the
@@ -631,6 +642,10 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat
}
}
+ private refuseIfShuttingDown(): void {
+ if (this.shuttingDown) throw new ServerShuttingDownException();
+ }
+
private async claimAndStart(id: string, explicit: boolean): Promise<Session> {
// Read before the claim, so a stop that lands while the claim is pending counts against this start.
const stopRequestsBefore = this.stopRequests.get(id);
@@ -684,6 +699,8 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat
if (this.stopRequests.get(id) !== stopRequestsBefore) {
throw new SessionStoppedException(`Session ${id} was stopped`);
}
+ // Shutdown may have begun while the claim was pending; the stop clear and the hook must not run.
+ this.refuseIfShuttingDown();
try {
return await this.engineLifecycle.start(id, { explicit });
} catch (error) {
@@ -702,17 +719,38 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat
if (this.stopRequests.get(id) !== stopRequestsBefore) {
throw new SessionStoppedException(`Session ${id} was stopped`);
}
- // The lease may have lapsed while the first attempt ran; the retry must keep holding the
- // claim, never 409 on the session it already owns.
- if (this.ownership && !(await this.ownership.claim(id))) {
- await this.findOne(id);
- throw new ConflictException(`Session ${id} is running on another node`);
- }
- // Checked again: a stop that landed during the re-claim leaves the claim for claimAndStart's catch to release.
- if (this.stopRequests.get(id) !== stopRequestsBefore) {
- throw new SessionStoppedException(`Session ${id} was stopped`);
+ try {
+ // Shutdown may have released this node's claims during the delay; re-claiming would pin the row.
+ this.refuseIfShuttingDown();
+ // The lease may have lapsed while the first attempt ran; the retry must keep holding the
+ // claim, never 409 on the session it already owns.
+ if (this.ownership && !(await this.ownership.claim(id))) {
+ await this.findOne(id);
+ throw new ConflictException(`Session ${id} is running on another node`);
+ }
+ // Checked again: a stop that landed during the re-claim leaves the claim for claimAndStart's catch to release.
+ if (this.stopRequests.get(id) !== stopRequestsBefore) {
+ throw new SessionStoppedException(`Session ${id} was stopped`);
+ }
+ // And shutdown may have begun during the re-claim; the stop clear and the hook must not run.
+ this.refuseIfShuttingDown();
+ return await this.engineLifecycle.start(id, { explicit });
+ } catch (retryError) {
+ // The first attempt may have left FAILED, which the boot reset, auto-start and the takeover
+ // sweep all skip, and shutdown refused the retry that would have overwritten it. Handed back
+ // as DISCONNECTED so the next boot restores the session. Best-effort: the refusal is the answer.
+ // Fenced like the boot reset: a FAILED on a row a peer has since adopted is the peer's.
+ if (retryError instanceof ServerShuttingDownException) {
+ const claimable = this.ownership?.claimableWhere() ?? [{}];
+ await this.sessionRepository
+ .update(
+ claimable.map(clause => ({ ...clause, id, status: SessionStatus.FAILED })),
+ { status: SessionStatus.DISCONNECTED },
+ )
+ .catch(() => undefined);
+ }
+ throw retryError;
}
- return this.engineLifecycle.start(id, { explicit });
}
}