Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(integration): dead-letter undispatched ingress events on retention (#1786)

Yudhi Armyndharisa36de57b18da
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 230f7907d..fb87a81c9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- Dead-letter undispatched ingress events instead of deleting them when they age out of the dedup window.
- Keep a re-sent ingress delivery redrivable when an earlier dead letter for its id was already redriven.
- Deliver Baileys API edit and revoke events to webhook and WebSocket consumers.
- Preserve chat-scoped WebSocket event order while resolving chat identities.
diff --git a/docs/05-database-design.md b/docs/05-database-design.md
index 1d631930e..d5fe0b19f 100644
--- a/docs/05-database-design.md
+++ b/docs/05-database-design.md
@@ -996,7 +996,7 @@ The other interval-based prunes are the same shape: one prune at startup, then a
- **`webhook_delivery_failures`**: `WebhookService.onModuleInit()` (`src/modules/webhook/webhook.service.ts`), window `WEBHOOK_FAILURE_RETENTION_DAYS` (default 90; ≤ 0 disables the prune and logs that it is off). The replay `payload` has its own, shorter window: `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` (default 0, store none) clears expired payloads at startup and hourly; at 0 one startup pass clears any payload an earlier setting left.
- **`webhook_outbox_events`**: `WebhookOutboxService.onModuleInit()` (`src/modules/webhook/webhook-outbox.service.ts`), window `WEBHOOK_OUTBOX_RETENTION_DAYS` (default 7). Only settled rows are deleted; `pending` and `queued` rows can still be replayed and are never pruned on age. A non-positive value does **not** disable the prune: a settled row carries no payload, so the service warns and falls back to 7 days.
-- **`ingress_events`** and **`integration_delivery_failures`** — `IntegrationRetentionService` (`src/modules/integration/integration-retention.service.ts`) prunes both in one timer on two independent windows. `INGRESS_DEDUP_RETENTION_DAYS` (default 7) bounds the dedup rows; a non-positive value does **not** disable it — an unpruned dedup table grows without bound for no functional gain, so the service warns and falls back to the 7-day default. `INGRESS_RETENTION_DAYS` (default 90) bounds the DLQ rows, where long retention can be a deliberate operator choice, so ≤ 0 disables that prune (and only that prune).
+- **`ingress_events`** and **`integration_delivery_failures`** — `IntegrationRetentionService` (`src/modules/integration/integration-retention.service.ts`) prunes both in one timer on two independent windows. `INGRESS_DEDUP_RETENTION_DAYS` (default 7) bounds the dedup rows; a non-positive value does **not** disable it — an unpruned dedup table grows without bound for no functional gain, so the service warns and falls back to the 7-day default. `INGRESS_RETENTION_DAYS` (default 90) bounds the DLQ rows, where long retention can be a deliberate operator choice, so ≤ 0 disables that prune (and only that prune). An aged `pending` event that still carries its payload is first given a dead-letter row and a `failed` mark, and a later run prunes it; it is kept if that write fails. One whose instance or session was deleted, or whose queue job still owns the delivery, is settled and pruned without a dead-letter row.
- **`messages`** and finished **`message_batches`**: `MessageRetentionService` (`src/modules/message/message-retention.service.ts`), window `MESSAGE_RETENTION_DAYS` (default 0, which keeps everything; boot rejects a value above 36500). Messages are deleted by `createdAt` in batches of 500 ids, at most 200 batches per run, so a run deletes at most 100,000 messages; a run that hits that limit logs a warning and is followed by another a minute later until the backlog is cleared. Pending and processing batches are never pruned. Once enabled, send pacing treats a contact silent for longer than the window as a new (cold) contact again, stats totals and search results shrink, archived media files are reclaimed by the chat-media orphan sweep, and a history sync does not write back messages older than the window. Search plugins get no `message:deleted` for pruned rows, so an external index keeps its copies.
- **`send_idempotency_keys`**: `SendIdempotencyService` (`src/modules/message/idempotency/send-idempotency.service.ts`) deletes rows past `expiresAt` (24 hours after the claim) on an hourly `setInterval`, with no startup run: a claim that collides with an expired row deletes it and claims the key itself, so a late prune never blocks a send.
diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.md
index a5bcf6546..041dd589f 100644
--- a/docs/25-integration-fabric.md
+++ b/docs/25-integration-fabric.md
@@ -265,9 +265,21 @@ empty; nothing logs it, and it stays `pending` until `INGRESS_DEDUP_RETENTION_DA
Table growth is bounded by construction rather than by operator hygiene: the per-instance ingress
throttle caps the row-creation rate, dispatched rows slim to a marker + hash, and the two retention
windows prune what remains — `INGRESS_DEDUP_RETENTION_DAYS` (default 7) for the dedup oracle and
-`INGRESS_RETENTION_DAYS` (default 90, `<= 0` disables) for the DLQ. There is deliberately no
-per-instance row-count cap: eviction under a flood of forged delivery ids would silently drop legit
-dedup rows and re-admit their replays, which is worse than the bounded growth it would prevent.
+`INGRESS_RETENTION_DAYS` (default 90, `<= 0` disables) for the DLQ. A `pending` row that still
+carries its payload is settled before the age prune deletes it: the prune first writes its
+dead-letter row (unless an open one exists) and marks it `failed`. A delivery stranded across
+downtime longer than the dedup window, or held by an instance disabled for that long, is therefore
+not dropped: it stays redrivable until `INGRESS_RETENTION_DAYS` prunes that dead-letter row. A row
+whose dead-letter write fails is kept for the next run. Two cases get no dead-letter row: a
+queue job that still owns the delivery takes the `dispatched` mark (closing any open dead-letter
+row), and a row whose instance or bound session was deleted is dropped, since redrive refuses a
+deleted instance and a session delete purges its dead letters. A delivery whose completed job was
+already removed from the queue, or whose job lookup failed while Redis was down, is dead-lettered
+anyway and may already have been delivered: check before redriving it. The startup prune runs before
+plugins load, so a dead-letter row it writes carries no conversation id and its redrive takes the
+per-instance lane. There is deliberately no per-instance row-count cap: eviction under a flood of
+forged delivery ids would silently drop legit dedup rows and re-admit their replays, which is worse
+than the bounded growth it would prevent.
## 25.8 The Integration SDK (v1)
diff --git a/src/modules/integration/entities/ingress-event.entity.ts b/src/modules/integration/entities/ingress-event.entity.ts
index baf222043..2eb64881f 100644
--- a/src/modules/integration/entities/ingress-event.entity.ts
+++ b/src/modules/integration/entities/ingress-event.entity.ts
@@ -8,8 +8,10 @@ import { DateTransformer } from '../../../common/transformers/date.transformer';
// - 'dispatched' — the event reached the dispatch tier (handed to BullMQ or delivered inline). A
// failure INSIDE the tier (BullMQ attempts exhausted) dead-letters separately, so a
// 'dispatched' row is never the reconciler's concern.
-// - 'failed' — terminal: the reconciler exhausted its replay budget; recovery continues via the
-// integration_delivery_failures row + RedriveService.
+// - 'failed' — terminal: the reconciler exhausted its replay budget or found the queue job failed,
+// or the row aged past INGRESS_DEDUP_RETENTION_DAYS undispatched; recovery continues
+// via the integration_delivery_failures row + RedriveService (none is written when the
+// instance or session was deleted).
// NULL marks rows that predate these columns on synchronize-bootstrapped DBs (no backfill ran there);
// NULL reads as "not watched" — the reconciler never sweeps it, so an upgrade can never mass-replay
// the historical dedup log.
diff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.ts
index a9dc0d312..25dd4bc54 100644
--- a/src/modules/integration/ingress-reconciler.service.ts
+++ b/src/modules/integration/ingress-reconciler.service.ts
@@ -1,9 +1,11 @@
import { Injectable, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
+import { isUUID } from 'class-validator';
import { InjectRepository } from '@nestjs/typeorm';
-import { Repository } from 'typeorm';
+import { IsNull, LessThan, Not, Repository } from 'typeorm';
import { IngressEvent } from './entities/ingress-event.entity';
import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';
import { PluginInstance } from './entities/plugin-instance.entity';
+import { Session } from '../session/entities/session.entity';
import {
EnqueueOutcome,
IngressEnqueueService,
@@ -153,8 +155,9 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
try {
// Re-apply the eligibility oracle the live path checks at the door (IngressService.handle
// 404s an unknown/disabled instance): an instance disabled or deleted AFTER persist must
- // not receive the replay. The row stays 'pending' — a re-enabled instance is replayed by a
- // later sweep; a deleted one ages out via INGRESS_DEDUP_RETENTION_DAYS pruning.
+ // not receive the replay. The row stays 'pending' and a re-enabled instance is replayed by a
+ // later sweep; past INGRESS_DEDUP_RETENTION_DAYS the retention prune dead-letters it instead
+ // (deadLetterAgedPending), while a deleted one ages out without a dead letter.
const instance = await this.instances.resolve(row.pluginId, row.instanceId);
if (!instance || !instance.enabled) {
this.logger.log('Skipping ingress event for a disabled or deleted instance', {
@@ -188,6 +191,141 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
}
}
+ /**
+ * Hand every 'pending' row created before `cutoff` that still carries its payload to the DLQ, then
+ * retire the payload with a 'failed' mark, so the dedup-window prune never deletes an acknowledged
+ * delivery that was never dispatched: one stranded across downtime longer than the window, one of an
+ * instance disabled for that long, or any stranded row while the sweep is disabled. Same
+ * DLQ-before-retire order as the replay budget path. A row whose writes fail stays 'pending' with its
+ * payload, which the prune keeps, and is retried on the next run. Skipped while a sweep is running.
+ * A row whose instance or session was deleted, or whose queue job still owns the delivery, gets no
+ * dead letter. Returns the number of rows dead-lettered.
+ */
+ async deadLetterAgedPending(cutoff: Date, batchSize = 100): Promise<number> {
+ if (this.sweeping) return 0;
+ this.sweeping = true;
+ let retired = 0;
+ try {
+ for (;;) {
+ const rows = await this.events.find({
+ where: { dispatchState: 'pending', createdAt: LessThan(cutoff), payload: Not(IsNull()) },
+ order: { createdAt: 'ASC' },
+ take: batchSize,
+ });
+ let progressed = 0;
+ for (const row of rows) {
+ if (!hasPayload(row)) continue;
+ const meta = { pluginId: row.pluginId, instanceId: row.instanceId, deliveryId: row.providerDeliveryId };
+ try {
+ if (await this.ownerDeleted(row)) {
+ // The redrive endpoint refuses a deleted instance, and a session delete purges its dead
+ // letters on purpose: retire the payload without writing one back, so the prune drops it.
+ await this.events.update(
+ { id: row.id, dispatchState: 'pending' },
+ { dispatchState: 'failed', payload: null },
+ );
+ progressed++;
+ this.logger.warn('Undispatched ingress event of a deleted instance or session dropped', {
+ ...meta,
+ action: 'ingress_event_retention_dropped',
+ });
+ continue;
+ }
+ const jobData = this.jobDataFor(row);
+ // A job still in the queue (or completed) owns the delivery, as in reconcileRow: no dead
+ // letter while a copy can still deliver. A failed one falls through to its existing DLQ row.
+ const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);
+ if (existing && existing !== 'failed') {
+ const settled = await this.events.update(
+ { id: row.id, dispatchState: 'pending' },
+ { dispatchState: 'dispatched', payload: null },
+ );
+ // As in reconcileRow: the job delivers it, so an inline-failure dead letter must close.
+ if (settled.affected) await this.retireOpenDeadLetters(row);
+ progressed++;
+ continue;
+ }
+ const written = await this.ensureDeadLetterRow(
+ jobData,
+ row.dispatchAttempts,
+ 'not dispatched within the ingress dedup retention window',
+ );
+ const marked = await this.events.update(
+ { id: row.id, dispatchState: 'pending' },
+ { dispatchState: 'failed', payload: null },
+ );
+ progressed++;
+ if (written) {
+ // A delivery the dispatch tier made concurrently must not stay redrivable. For a failed
+ // event, another node's hand-off may have written a dead letter too, or skipped its own
+ // because of this one, so a writer retires its row only when an open one with a lower id
+ // exists: the lowest-id open row is never retired, and normally it is the only one left.
+ // Ids are random UUIDs, so a writer whose check runs before a lower-id row is visible
+ // keeps its own as well, and two open rows can remain.
+ const state = marked.affected
+ ? 'failed'
+ : (await this.events.findOne({ where: { id: row.id }, select: { dispatchState: true } }))
+ ?.dispatchState;
+ if (state === 'dispatched' || (state === 'failed' && (await this.hasLowerOpenDeadLetter(row, written)))) {
+ await this.failures.update({ id: written, redriven: false }, { redriven: true });
+ }
+ }
+ if (!marked.affected) continue;
+ retired++;
+ this.logger.warn('Ingress event was never dispatched within the dedup retention window; dead-lettered', {
+ ...meta,
+ action: 'ingress_event_retention_dead_lettered',
+ });
+ } catch (err) {
+ this.logger.error(
+ 'Failed to dead-letter an undispatched ingress event; kept for the next run',
+ err instanceof Error ? err.message : String(err),
+ { ...meta, action: 'ingress_event_retention_dead_letter_failed' },
+ );
+ }
+ }
+ if (rows.length < batchSize || progressed === 0) return retired;
+ }
+ } finally {
+ this.sweeping = false;
+ }
+ }
+
+ // The row's instance is gone, or it was bound to a session that is gone. A wildcard or non-id scope
+ // never names a sessions row, so it is never treated as deleted.
+ private async ownerDeleted(row: IngressEvent): Promise<boolean> {
+ if (!(await this.instances.resolve(row.pluginId, row.instanceId))) return true;
+ const sessionId = row.sessionId;
+ return !!sessionId && isUUID(sessionId) && !(await this.events.manager.existsBy(Session, { id: sessionId }));
+ }
+
+ private async hasLowerOpenDeadLetter(row: IngressEvent, written: string): Promise<boolean> {
+ const lower = await this.failures.count({
+ where: {
+ direction: 'inbound',
+ pluginId: row.pluginId,
+ instanceId: row.instanceId,
+ deliveryId: row.providerDeliveryId,
+ redriven: false,
+ id: LessThan(written),
+ },
+ });
+ return lower > 0;
+ }
+
+ private async retireOpenDeadLetters(row: IngressEvent): Promise<void> {
+ await this.failures.update(
+ {
+ direction: 'inbound',
+ pluginId: row.pluginId,
+ instanceId: row.instanceId,
+ deliveryId: row.providerDeliveryId,
+ redriven: false,
+ },
+ { redriven: true },
+ );
+ }
+
private async reconcileRow(
row: IngressEvent & { payload: NonNullable<IngressEvent['payload']> },
maxAttempts: number,
@@ -242,16 +380,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
// Retire any dead-letter row the live path already wrote for this delivery (the inline-failure
// case): the replay, or the job or re-queued copy still live in the queue, delivers it, so a later
// manual redrive must not deliver it again.
- await this.failures.update(
- {
- direction: 'inbound',
- pluginId: row.pluginId,
- instanceId: row.instanceId,
- deliveryId: row.providerDeliveryId,
- redriven: false,
- },
- { redriven: true },
- );
+ await this.retireOpenDeadLetters(row);
this.logger.log('Replayed stranded ingress event', {
pluginId: row.pluginId,
instanceId: row.instanceId,
diff --git a/src/modules/integration/integration-retention.service.spec.ts b/src/modules/integration/integration-retention.service.spec.ts
index 9161dc1b4..919cddd99 100644
--- a/src/modules/integration/integration-retention.service.spec.ts
+++ b/src/modules/integration/integration-retention.service.spec.ts
@@ -2,6 +2,12 @@ import { DataSource, Repository } from 'typeorm';
import { IngressEvent } from './entities/ingress-event.entity';
import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';
import { IntegrationRetentionService } from './integration-retention.service';
+import { IngressReconcilerService } from './ingress-reconciler.service';
+import { IngressEnqueueService } from './ingress-enqueue.service';
+import { PluginInstanceService } from './plugin-instance.service';
+import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';
+import { Session } from '../session/entities/session.entity';
+import { LoggerService } from '../../common/services/logger.service';
const daysAgo = (d: number): Date => {
const dt = new Date();
@@ -24,6 +30,7 @@ describe('IntegrationRetentionService.pruneOlderThan', () => {
service = new IntegrationRetentionService(
ds.getRepository(IngressEvent),
ds.getRepository(IntegrationDeliveryFailure),
+ { deadLetterAgedPending: jest.fn().mockResolvedValue(0) } as unknown as IngressReconcilerService,
);
const events = ds.getRepository(IngressEvent);
@@ -179,6 +186,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =
failuresDelete,
events: { delete: eventsDelete } as unknown as Repository<IngressEvent>,
failures: { delete: failuresDelete } as unknown as Repository<IntegrationDeliveryFailure>,
+ reconciler: { deadLetterAgedPending: jest.fn().mockResolvedValue(0) } as unknown as IngressReconcilerService,
};
};
@@ -193,11 +201,11 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =
process.env.INGRESS_RETENTION_DAYS = '0';
delete process.env.INGRESS_DEDUP_RETENTION_DAYS;
const repos = mockRepos();
- const svc = new IntegrationRetentionService(repos.events, repos.failures);
+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);
svc.onModuleInit();
// Let the startup prune promise settle (the method runs before the log .then()).
- await Promise.resolve();
+ await new Promise(resolve => setImmediate(resolve));
// Dedup rows are never an audit log — their prune cannot be disabled to infinity.
expect(repos.eventsDelete).toHaveBeenCalled();
expect(repos.failuresDelete).not.toHaveBeenCalled();
@@ -208,7 +216,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =
delete process.env.INGRESS_RETENTION_DAYS;
delete process.env.INGRESS_DEDUP_RETENTION_DAYS;
const repos = mockRepos();
- const svc = new IntegrationRetentionService(repos.events, repos.failures);
+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);
jest.useFakeTimers();
try {
@@ -234,7 +242,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =
delete process.env.INGRESS_RETENTION_DAYS;
process.env.INGRESS_DEDUP_RETENTION_DAYS = '21';
const repos = mockRepos();
- const svc = new IntegrationRetentionService(repos.events, repos.failures);
+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);
const pruneSpy = jest.spyOn(svc, 'pruneOlderThan');
svc.onModuleInit();
@@ -246,7 +254,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =
delete process.env.INGRESS_RETENTION_DAYS;
process.env.INGRESS_DEDUP_RETENTION_DAYS = '0';
const repos = mockRepos();
- const svc = new IntegrationRetentionService(repos.events, repos.failures);
+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);
const pruneSpy = jest.spyOn(svc, 'pruneOlderThan');
svc.onModuleInit();
@@ -255,3 +263,394 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =
svc.onModuleDestroy();
});
});
+
+describe('IntegrationRetentionService.pruneOlderThan with undispatched events', () => {
+ let ds: DataSource;
+ let events: Repository<IngressEvent>;
+ let failures: Repository<IntegrationDeliveryFailure>;
+ let reconciler: IngressReconcilerService;
+ let service: IntegrationRetentionService;
+ let resolveInstance: jest.Mock;
+ let existingJobState: jest.Mock;
+ let logError: jest.SpyInstance;
+ const SESSION_ID = '11111111-1111-4111-8111-111111111111';
+
+ beforeEach(async () => {
+ ds = new DataSource({
+ type: 'better-sqlite3',
+ database: ':memory:',
+ entities: [IngressEvent, IntegrationDeliveryFailure, Session],
+ synchronize: true,
+ });
+ await ds.initialize();
+ events = ds.getRepository(IngressEvent);
+ failures = ds.getRepository(IntegrationDeliveryFailure);
+ await ds.getRepository(Session).save({ id: SESSION_ID, name: 'sess-1' });
+ // A disabled instance, no loaded plugin and the queue off: the hand-off must not depend on the
+ // instance being enabled or its plugin being loaded.
+ resolveInstance = jest.fn().mockResolvedValue({ enabled: false });
+ existingJobState = jest.fn().mockResolvedValue(undefined);
+ reconciler = new IngressReconcilerService(
+ events,
+ failures,
+ { existingJobState } as unknown as IngressEnqueueService,
+ { getPlugin: () => undefined } as unknown as PluginLoaderService,
+ { resolve: resolveInstance } as unknown as PluginInstanceService,
+ );
+ service = new IntegrationRetentionService(events, failures, reconciler);
+ jest.spyOn(LoggerService.prototype, 'warn').mockImplementation(() => undefined);
+ logError = jest.spyOn(LoggerService.prototype, 'error').mockImplementation(() => undefined);
+ });
+
+ afterEach(async () => {
+ jest.restoreAllMocks();
+ if (ds.isInitialized) await ds.destroy();
+ });
+
+ const insertEvent = (id: string, over: Partial<IngressEvent>) =>
+ events
+ .createQueryBuilder()
+ .insert()
+ .values({
+ id,
+ instanceId: 'inst',
+ pluginId: 'plug',
+ providerDeliveryId: `d-${id}`,
+ route: 'chatwoot',
+ payload: { headers: {}, query: {}, body: '{"msg":"hello"}', rawBody: '{"msg":"hello"}', method: 'POST' },
+ sessionId: SESSION_ID,
+ dispatchState: 'pending',
+ createdAt: daysAgo(8),
+ ...over,
+ })
+ .execute();
+
+ it('dead-letters an aged pending event with its payload instead of deleting it unrecoverably', async () => {
+ await insertEvent('stranded', {});
+
+ const first = await service.pruneOlderThan(7, 90);
+
+ // The payload is not lost: it now lives in a redrivable dead-letter row, written before the marker
+ // was settled, and the settled marker goes with the next run.
+ expect(first).toEqual({ events: 0, failures: 0 });
+ expect(await events.findOneByOrFail({ id: 'stranded' })).toMatchObject({ dispatchState: 'failed', payload: null });
+ expect(await service.pruneOlderThan(7, 90)).toEqual({ events: 1, failures: 0 });
+ expect(await events.count()).toBe(0);
+ const [dlq] = await failures.find();
+ expect(dlq).toMatchObject({
+ direction: 'inbound',
+ pluginId: 'plug',
+ instanceId: 'inst',
+ sessionId: SESSION_ID,
+ deliveryId: 'd-stranded',
+ lastError: 'not dispatched within the ingress dedup retention window',
+ redriven: false,
+ });
+ expect(dlq.payload).toMatchObject({
+ route: 'chatwoot',
+ method: 'POST',
+ ingress: { body: '{"msg":"hello"}', rawBody: '{"msg":"hello"}' },
+ });
+ });
+
+ it('retires the payload without a second dead letter when the live path already wrote one', async () => {
+ await insertEvent('inline-failed', { dispatchAttempts: 1 });
+ await failures.save({
+ direction: 'inbound',
+ pluginId: 'plug',
+ instanceId: 'inst',
+ deliveryId: 'd-inline-failed',
+ attempts: 1,
+ lastError: 'sandbox 5xx',
+ payload: { route: 'chatwoot', ingress: { headers: {}, query: {}, body: '{}', rawBody: '{}' } },
+ redriven: false,
+ });
+ const update = jest.spyOn(events, 'update');
+
+ await service.pruneOlderThan(7, 90);
+
+ // Settled by the hand-off: the payload is retired against the existing dead letter.
+ expect(update).toHaveBeenCalledWith(
+ { id: 'inline-failed', dispatchState: 'pending' },
+ { dispatchState: 'failed', payload: null },
+ );
+ const [dlq] = await failures.find();
+ expect(dlq).toMatchObject({ deliveryId: 'd-inline-failed', lastError: 'sandbox 5xx', redriven: false });
+ expect(await failures.count()).toBe(1);
+ });
+
+ it('writes a fresh dead letter when the only one for the delivery id was already redriven', async () => {
+ await insertEvent('resent', {});
+ const old = await failures.save({
+ direction: 'inbound',
+ pluginId: 'plug',
+ instanceId: 'inst',
+ deliveryId: 'd-resent',
+ attempts: 1,
+ lastError: 'sandbox 5xx',
+ payload: null,
+ redriven: true,
+ });
+
+ await service.pruneOlderThan(7, 90);
+
+ const open = await failures.find({ where: { deliveryId: 'd-resent', redriven: false } });
+ expect(open).toHaveLength(1);
+ expect(open[0].id).not.toBe(old.id);
+ expect(open[0].payload).toMatchObject({ ingress: { rawBody: '{"msg":"hello"}' } });
+ });
+
+ it('keeps an aged pending event with its payload when the dead-letter write fails', async () => {
+ await insertEvent('stranded', {});
+ jest.spyOn(failures, 'save').mockRejectedValueOnce(new Error('disk full'));
+
+ const result = await service.pruneOlderThan(7, 90);
+
+ expect(result.events).toBe(0);
+ const kept = await events.findOneByOrFail({ id: 'stranded' });
+ expect(kept.dispatchState).toBe('pending');
+ expect(kept.payload).toMatchObject({ rawBody: '{"msg":"hello"}' });
+ expect(await failures.count()).toBe(0);
+ });
+
+ it('retires its own dead letter when a concurrent dispatch settles the event first', async () => {
+ await insertEvent('raced', {});
+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {
+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'dispatched', payload = NULL WHERE id = 'raced'`);
+ return { affected: 0, raw: [], generatedMaps: [] };
+ });
+
+ await service.pruneOlderThan(7, 90);
+
+ const [dlq] = await failures.find();
+ expect(dlq).toMatchObject({ deliveryId: 'd-raced', redriven: true });
+ });
+
+ it('keeps its own dead letter redrivable when a concurrent failure mark settles the event first', async () => {
+ await insertEvent('raced', {});
+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {
+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'failed', payload = NULL WHERE id = 'raced'`);
+ return { affected: 0, raw: [], generatedMaps: [] };
+ });
+
+ await service.pruneOlderThan(7, 90);
+
+ const [dlq] = await failures.find();
+ expect(dlq).toMatchObject({ deliveryId: 'd-raced', redriven: false });
+ });
+
+ describe('when another node dead-letters the same delivery concurrently', () => {
+ const LOW_ID = '00000000-0000-4000-8000-000000000000';
+ const HIGH_ID = 'ffffffff-ffff-4fff-bfff-ffffffffffff';
+ // The other node's dead letter lands after this node's open-row check, so both write one.
+ const otherNodeDeadLetter = (id: string) =>
+ failures.insert({
+ id,
+ direction: 'inbound',
+ pluginId: 'plug',
+ instanceId: 'inst',
+ deliveryId: 'd-raced',
+ attempts: 0,
+ lastError: 'not dispatched within the ingress dedup retention window',
+ payload: { route: 'chatwoot', ingress: { headers: {}, query: {}, body: '{}', rawBody: '{}' } },
+ redriven: false,
+ });
+ const openIds = async () =>
+ (await failures.find({ where: { deliveryId: 'd-raced', redriven: false } })).map(f => f.id);
+
+ it('retires its own dead letter when it loses the failure mark to a node with a lower-id row', async () => {
+ await insertEvent('raced', {});
+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {
+ await otherNodeDeadLetter(LOW_ID);
+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'failed', payload = NULL WHERE id = 'raced'`);
+ return { affected: 0, raw: [], generatedMaps: [] };
+ });
+
+ await service.pruneOlderThan(7, 90);
+
+ expect(await openIds()).toEqual([LOW_ID]);
+ expect(await failures.count()).toBe(2);
+ });
+
+ it('retires its own dead letter when it wins the failure mark but a lower-id row exists', async () => {
+ await insertEvent('raced', {});
+ const update = events.update.bind(events);
+ jest.spyOn(events, 'update').mockImplementationOnce(async (criteria, partial) => {
+ await otherNodeDeadLetter(LOW_ID);
+ return update(criteria, partial);
+ });
+
+ await service.pruneOlderThan(7, 90);
+
+ expect(await openIds()).toEqual([LOW_ID]);
+ expect(await events.findOneByOrFail({ id: 'raced' })).toMatchObject({ dispatchState: 'failed', payload: null });
+ });
+
+ it('keeps its own dead letter when the other node holds the higher id', async () => {
+ await insertEvent('raced', {});
+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {
+ await otherNodeDeadLetter(HIGH_ID);
+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'failed', payload = NULL WHERE id = 'raced'`);
+ return { affected: 0, raw: [], generatedMaps: [] };
+ });
+
+ await service.pruneOlderThan(7, 90);
+
+ // The other node retires its own row once its check sees this one; this node's row stays.
+ const open = await openIds();
+ expect(open).toHaveLength(2);
+ expect(open).toContain(HIGH_ID);
+ });
+ });
+
+ it('still prunes aged settled markers, legacy rows, and pending rows without a payload', async () => {
+ await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });
+ await insertEvent('failed', { dispatchState: 'failed', payload: null });
+ await insertEvent('legacy', { dispatchState: null });
+ await insertEvent('empty', { payload: null });
+ await insertEvent('recent', { createdAt: daysAgo(1) });
+
+ const result = await service.pruneOlderThan(7, 90);
+
+ expect(result.events).toBe(4);
+ expect((await events.find()).map(e => e.id)).toEqual(['recent']);
+ expect(await failures.count()).toBe(0);
+ });
+
+ it('writes no dead letter for a deleted instance and prunes the row', async () => {
+ await insertEvent('orphan', {});
+ resolveInstance.mockResolvedValue(null);
+
+ await service.pruneOlderThan(7, 90);
+ const result = await service.pruneOlderThan(7, 90);
+
+ expect(result.events).toBe(1);
+ expect(await events.count()).toBe(0);
+ expect(await failures.count()).toBe(0);
+ });
+
+ it('writes no dead letter for a deleted session, but keeps one for a wildcard scope', async () => {
+ await insertEvent('gone', { sessionId: '22222222-2222-4222-8222-222222222222' });
+ await insertEvent('wildcard', { sessionId: '*' });
+
+ await service.pruneOlderThan(7, 90);
+
+ expect(await events.find({ where: { dispatchState: 'failed' } })).toHaveLength(2);
+ expect((await failures.find()).map(f => f.deliveryId)).toEqual(['d-wildcard']);
+ });
+
+ it('writes no dead letter while a queue job still owns the delivery', async () => {
+ await insertEvent('queued', {});
+ existingJobState.mockResolvedValue('waiting');
+ const update = jest.spyOn(events, 'update');
+
+ await service.pruneOlderThan(7, 90);
+
+ expect(update).toHaveBeenCalledWith(
+ { id: 'queued', dispatchState: 'pending' },
+ { dispatchState: 'dispatched', payload: null },
+ );
+ expect(await failures.count()).toBe(0);
+ });
+
+ it('closes an inline-failure dead letter once a queue job owns the delivery', async () => {
+ await insertEvent('queued', { dispatchAttempts: 1 });
+ await failures.save({
+ direction: 'inbound',
+ pluginId: 'plug',
+ instanceId: 'inst',
+ deliveryId: 'd-queued',
+ attempts: 1,
+ lastError: 'sandbox 5xx',
+ payload: { route: 'chatwoot', ingress: { headers: {}, query: {}, body: '{}', rawBody: '{}' } },
+ redriven: false,
+ });
+ existingJobState.mockResolvedValue('waiting');
+
+ await service.pruneOlderThan(7, 90);
+
+ // The job delivers it, so a manual redrive of the old row would deliver it twice.
+ expect(await events.findOneByOrFail({ id: 'queued' })).toMatchObject({ dispatchState: 'dispatched' });
+ expect(await failures.find()).toEqual([expect.objectContaining({ deliveryId: 'd-queued', redriven: true })]);
+ });
+
+ it('dead-letters an event whose queue job failed', async () => {
+ await insertEvent('job-failed', {});
+ existingJobState.mockResolvedValue('failed');
+
+ await service.pruneOlderThan(7, 90);
+
+ // A failed job delivers nothing, so the payload must survive in a redrivable dead letter.
+ expect(await events.findOneByOrFail({ id: 'job-failed' })).toMatchObject({
+ dispatchState: 'failed',
+ payload: null,
+ });
+ const open = await failures.find({ where: { direction: 'inbound', redriven: false } });
+ expect(open).toHaveLength(1);
+ expect(open[0]).toMatchObject({ deliveryId: 'd-job-failed' });
+ expect(open[0].payload).toMatchObject({ ingress: { rawBody: '{"msg":"hello"}' } });
+ });
+
+ it('still prunes aged events when the dead-letter prune fails', async () => {
+ await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });
+ jest.spyOn(failures, 'delete').mockRejectedValueOnce(new Error('locked'));
+
+ const result = await service.pruneOlderThan(7, 90);
+
+ expect(result).toEqual({ events: 1, failures: 0 });
+ expect(await events.count()).toBe(0);
+ });
+
+ it('hands off a backlog larger than one batch', async () => {
+ for (let i = 0; i < 250; i++) await insertEvent(`b${i}`, {});
+
+ await service.pruneOlderThan(7, 90);
+
+ expect(await events.count({ where: { dispatchState: 'failed' } })).toBe(250);
+ expect(await failures.count()).toBe(250);
+ });
+
+ it('prunes aged settled markers while the hand-off is still waiting', async () => {
+ await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });
+ await insertEvent('stranded', {});
+ // A job-state lookup against a queue that never became reachable does not return.
+ const lookedUp = new Promise<void>(resolve =>
+ existingJobState.mockImplementation(() => {
+ resolve();
+ return new Promise(() => {});
+ }),
+ );
+
+ void service.pruneOlderThan(7, 90);
+ await lookedUp;
+
+ expect((await events.find()).map(e => e.id)).toEqual(['stranded']);
+ });
+
+ it('stops and keeps every row when no dead-letter write succeeds', async () => {
+ for (let i = 0; i < 100; i++) await insertEvent(`f${i}`, {});
+ jest.spyOn(failures, 'save').mockRejectedValue(new Error('disk full'));
+
+ const result = await service.pruneOlderThan(7, 90);
+
+ expect(result.events).toBe(0);
+ expect(await events.count({ where: { dispatchState: 'pending' } })).toBe(100);
+ expect(logError).toHaveBeenCalledTimes(100);
+ expect(logError).toHaveBeenCalledWith(
+ 'Failed to dead-letter an undispatched ingress event; kept for the next run',
+ 'disk full',
+ expect.objectContaining({ action: 'ingress_event_retention_dead_letter_failed' }),
+ );
+ });
+
+ it('keeps aged pending rows while a reconcile sweep holds the guard', async () => {
+ await insertEvent('stranded', {});
+ (reconciler as unknown as { sweeping: boolean }).sweeping = true;
+
+ const result = await service.pruneOlderThan(7, 90);
+
+ expect(result.events).toBe(0);
+ expect((await events.findOneByOrFail({ id: 'stranded' })).payload).not.toBeNull();
+ expect(await failures.count()).toBe(0);
+ });
+});
diff --git a/src/modules/integration/integration-retention.service.ts b/src/modules/integration/integration-retention.service.ts
index 1f841e570..20b5c04ba 100644
--- a/src/modules/integration/integration-retention.service.ts
+++ b/src/modules/integration/integration-retention.service.ts
@@ -1,8 +1,9 @@
import { Injectable, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
-import { LessThan, Repository } from 'typeorm';
+import { IsNull, LessThan, Not, Repository } from 'typeorm';
import { IngressEvent } from './entities/ingress-event.entity';
import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';
+import { IngressReconcilerService } from './ingress-reconciler.service';
import { createLogger } from '../../common/services/logger.service';
// Dedup rows are a delivery-id oracle, not an audit log: provider retries arrive within minutes, so
@@ -23,7 +24,8 @@ const DEFAULT_FAILURE_RETENTION_DAYS = 90;
* - `INGRESS_DEDUP_RETENTION_DAYS` (default 7) prunes `ingress_events`. <= 0 does NOT disable it —
* an unpruned dedup table grows without bound for zero functional gain (its payloads are retired
* on dispatch and its audit value is nil), so a non-positive value falls back to the default with
- * a warning instead of silently opting into unbounded growth.
+ * a warning instead of silently opting into unbounded growth. A 'pending' row that still carries
+ * its payload is only deleted after deadLetterAgedPending has settled it.
* - `INGRESS_RETENTION_DAYS` (default 90) prunes `integration_delivery_failures` — the redrive
* payload store, where long retention can be a deliberate operator choice. <= 0 disables that
* prune (and only that prune).
@@ -37,6 +39,7 @@ export class IntegrationRetentionService implements OnModuleInit, OnModuleDestro
@InjectRepository(IngressEvent, 'data') private readonly eventRepository: Repository<IngressEvent>,
@InjectRepository(IntegrationDeliveryFailure, 'data')
private readonly failureRepository: Repository<IntegrationDeliveryFailure>,
+ private readonly reconciler: IngressReconcilerService,
) {}
onModuleInit(): void {
@@ -86,7 +89,8 @@ export class IntegrationRetentionService implements OnModuleInit, OnModuleDestro
/**
* Delete ingress_events rows older than `eventsDays` and integration_delivery_failures rows older
* than `failuresDays` (null skips the failures prune; omitted = same window as events). Returns
- * the number removed from each table.
+ * the number removed from each table. Aged 'pending' events that still carry a payload are kept and
+ * settled by deadLetterAgedPending, so the next run deletes them; one whose hand-off fails stays.
*/
async pruneOlderThan(
eventsDays: number,
@@ -96,12 +100,32 @@ export class IntegrationRetentionService implements OnModuleInit, OnModuleDestro
eventsCutoff.setDate(eventsCutoff.getDate() - eventsDays);
const failuresCutoff = new Date();
if (failuresDays !== null) failuresCutoff.setDate(failuresCutoff.getDate() - failuresDays);
- const [eventsResult, failuresResult] = await Promise.all([
- this.eventRepository.delete({ createdAt: LessThan(eventsCutoff) }),
+ // DLQ first, so an undispatched event whose only dead letter just aged out is handed a fresh one.
+ // The dedup delete keeps every 'pending' row that still carries its payload, so it runs before the
+ // hand-off, which can wait on an unreachable queue for its job-state lookup; the rows the hand-off
+ // settles go on the next run. A failed step is logged and never blocks the next one.
+ const logFailure = (step: string) => (err: unknown) => {
+ this.logger.error(
+ `Integration ingress retention: ${step} failed`,
+ err instanceof Error ? err.stack : String(err),
+ );
+ return { affected: 0 };
+ };
+ const failuresResult =
failuresDays === null
- ? Promise.resolve({ affected: 0 })
- : this.failureRepository.delete({ createdAt: LessThan(failuresCutoff) }),
- ]);
+ ? { affected: 0 }
+ : await this.failureRepository
+ .delete({ createdAt: LessThan(failuresCutoff) })
+ .catch(logFailure('delivery-failure prune'));
+ const aged = LessThan(eventsCutoff);
+ const eventsResult = await this.eventRepository
+ .delete([
+ { createdAt: aged, dispatchState: Not('pending') },
+ { createdAt: aged, dispatchState: IsNull() },
+ { createdAt: aged, payload: IsNull() },
+ ])
+ .catch(logFailure('dedup prune'));
+ await this.reconciler.deadLetterAgedPending(eventsCutoff).catch(logFailure('undispatched event hand-off'));
return { events: eventsResult.affected || 0, failures: failuresResult.affected || 0 };
}
}