Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(integration): dead-letter undispatched ingress events on retention (#1786)
Yudhi Armyndharisa36de57b18da
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 230f7907d..fb87a81c9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- Dead-letter undispatched ingress events instead of deleting them when they age out of the dedup window.- Keep a re-sent ingress delivery redrivable when an earlier dead letter for its id was already redriven.- Deliver Baileys API edit and revoke events to webhook and WebSocket consumers.- Preserve chat-scoped WebSocket event order while resolving chat identities.diff --git a/docs/05-database-design.md b/docs/05-database-design.mdindex 1d631930e..d5fe0b19f 100644--- a/docs/05-database-design.md+++ b/docs/05-database-design.md@@ -996,7 +996,7 @@ The other interval-based prunes are the same shape: one prune at startup, then a- **`webhook_delivery_failures`**: `WebhookService.onModuleInit()` (`src/modules/webhook/webhook.service.ts`), window `WEBHOOK_FAILURE_RETENTION_DAYS` (default 90; ≤ 0 disables the prune and logs that it is off). The replay `payload` has its own, shorter window: `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` (default 0, store none) clears expired payloads at startup and hourly; at 0 one startup pass clears any payload an earlier setting left.- **`webhook_outbox_events`**: `WebhookOutboxService.onModuleInit()` (`src/modules/webhook/webhook-outbox.service.ts`), window `WEBHOOK_OUTBOX_RETENTION_DAYS` (default 7). Only settled rows are deleted; `pending` and `queued` rows can still be replayed and are never pruned on age. A non-positive value does **not** disable the prune: a settled row carries no payload, so the service warns and falls back to 7 days.-- **`ingress_events`** and **`integration_delivery_failures`** — `IntegrationRetentionService` (`src/modules/integration/integration-retention.service.ts`) prunes both in one timer on two independent windows. `INGRESS_DEDUP_RETENTION_DAYS` (default 7) bounds the dedup rows; a non-positive value does **not** disable it — an unpruned dedup table grows without bound for no functional gain, so the service warns and falls back to the 7-day default. `INGRESS_RETENTION_DAYS` (default 90) bounds the DLQ rows, where long retention can be a deliberate operator choice, so ≤ 0 disables that prune (and only that prune).+- **`ingress_events`** and **`integration_delivery_failures`** — `IntegrationRetentionService` (`src/modules/integration/integration-retention.service.ts`) prunes both in one timer on two independent windows. `INGRESS_DEDUP_RETENTION_DAYS` (default 7) bounds the dedup rows; a non-positive value does **not** disable it — an unpruned dedup table grows without bound for no functional gain, so the service warns and falls back to the 7-day default. `INGRESS_RETENTION_DAYS` (default 90) bounds the DLQ rows, where long retention can be a deliberate operator choice, so ≤ 0 disables that prune (and only that prune). An aged `pending` event that still carries its payload is first given a dead-letter row and a `failed` mark, and a later run prunes it; it is kept if that write fails. One whose instance or session was deleted, or whose queue job still owns the delivery, is settled and pruned without a dead-letter row.- **`messages`** and finished **`message_batches`**: `MessageRetentionService` (`src/modules/message/message-retention.service.ts`), window `MESSAGE_RETENTION_DAYS` (default 0, which keeps everything; boot rejects a value above 36500). Messages are deleted by `createdAt` in batches of 500 ids, at most 200 batches per run, so a run deletes at most 100,000 messages; a run that hits that limit logs a warning and is followed by another a minute later until the backlog is cleared. Pending and processing batches are never pruned. Once enabled, send pacing treats a contact silent for longer than the window as a new (cold) contact again, stats totals and search results shrink, archived media files are reclaimed by the chat-media orphan sweep, and a history sync does not write back messages older than the window. Search plugins get no `message:deleted` for pruned rows, so an external index keeps its copies.- **`send_idempotency_keys`**: `SendIdempotencyService` (`src/modules/message/idempotency/send-idempotency.service.ts`) deletes rows past `expiresAt` (24 hours after the claim) on an hourly `setInterval`, with no startup run: a claim that collides with an expired row deletes it and claims the key itself, so a late prune never blocks a send.diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.mdindex a5bcf6546..041dd589f 100644--- a/docs/25-integration-fabric.md+++ b/docs/25-integration-fabric.md@@ -265,9 +265,21 @@ empty; nothing logs it, and it stays `pending` until `INGRESS_DEDUP_RETENTION_DATable growth is bounded by construction rather than by operator hygiene: the per-instance ingressthrottle caps the row-creation rate, dispatched rows slim to a marker + hash, and the two retentionwindows prune what remains — `INGRESS_DEDUP_RETENTION_DAYS` (default 7) for the dedup oracle and-`INGRESS_RETENTION_DAYS` (default 90, `<= 0` disables) for the DLQ. There is deliberately no-per-instance row-count cap: eviction under a flood of forged delivery ids would silently drop legit-dedup rows and re-admit their replays, which is worse than the bounded growth it would prevent.+`INGRESS_RETENTION_DAYS` (default 90, `<= 0` disables) for the DLQ. A `pending` row that still+carries its payload is settled before the age prune deletes it: the prune first writes its+dead-letter row (unless an open one exists) and marks it `failed`. A delivery stranded across+downtime longer than the dedup window, or held by an instance disabled for that long, is therefore+not dropped: it stays redrivable until `INGRESS_RETENTION_DAYS` prunes that dead-letter row. A row+whose dead-letter write fails is kept for the next run. Two cases get no dead-letter row: a+queue job that still owns the delivery takes the `dispatched` mark (closing any open dead-letter+row), and a row whose instance or bound session was deleted is dropped, since redrive refuses a+deleted instance and a session delete purges its dead letters. A delivery whose completed job was+already removed from the queue, or whose job lookup failed while Redis was down, is dead-lettered+anyway and may already have been delivered: check before redriving it. The startup prune runs before+plugins load, so a dead-letter row it writes carries no conversation id and its redrive takes the+per-instance lane. There is deliberately no per-instance row-count cap: eviction under a flood of+forged delivery ids would silently drop legit dedup rows and re-admit their replays, which is worse+than the bounded growth it would prevent.## 25.8 The Integration SDK (v1)diff --git a/src/modules/integration/entities/ingress-event.entity.ts b/src/modules/integration/entities/ingress-event.entity.tsindex baf222043..2eb64881f 100644--- a/src/modules/integration/entities/ingress-event.entity.ts+++ b/src/modules/integration/entities/ingress-event.entity.ts@@ -8,8 +8,10 @@ import { DateTransformer } from '../../../common/transformers/date.transformer';// - 'dispatched' — the event reached the dispatch tier (handed to BullMQ or delivered inline). A// failure INSIDE the tier (BullMQ attempts exhausted) dead-letters separately, so a// 'dispatched' row is never the reconciler's concern.-// - 'failed' — terminal: the reconciler exhausted its replay budget; recovery continues via the-// integration_delivery_failures row + RedriveService.+// - 'failed' — terminal: the reconciler exhausted its replay budget or found the queue job failed,+// or the row aged past INGRESS_DEDUP_RETENTION_DAYS undispatched; recovery continues+// via the integration_delivery_failures row + RedriveService (none is written when the+// instance or session was deleted).// NULL marks rows that predate these columns on synchronize-bootstrapped DBs (no backfill ran there);// NULL reads as "not watched" — the reconciler never sweeps it, so an upgrade can never mass-replay// the historical dedup log.diff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.tsindex a9dc0d312..25dd4bc54 100644--- a/src/modules/integration/ingress-reconciler.service.ts+++ b/src/modules/integration/ingress-reconciler.service.ts@@ -1,9 +1,11 @@import { Injectable, OnModuleDestroy, OnModuleInit } from '@nestjs/common';+import { isUUID } from 'class-validator';import { InjectRepository } from '@nestjs/typeorm';-import { Repository } from 'typeorm';+import { IsNull, LessThan, Not, Repository } from 'typeorm';import { IngressEvent } from './entities/ingress-event.entity';import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';import { PluginInstance } from './entities/plugin-instance.entity';+import { Session } from '../session/entities/session.entity';import {EnqueueOutcome,IngressEnqueueService,@@ -153,8 +155,9 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {try {// Re-apply the eligibility oracle the live path checks at the door (IngressService.handle// 404s an unknown/disabled instance): an instance disabled or deleted AFTER persist must- // not receive the replay. The row stays 'pending' — a re-enabled instance is replayed by a- // later sweep; a deleted one ages out via INGRESS_DEDUP_RETENTION_DAYS pruning.+ // not receive the replay. The row stays 'pending' and a re-enabled instance is replayed by a+ // later sweep; past INGRESS_DEDUP_RETENTION_DAYS the retention prune dead-letters it instead+ // (deadLetterAgedPending), while a deleted one ages out without a dead letter.const instance = await this.instances.resolve(row.pluginId, row.instanceId);if (!instance || !instance.enabled) {this.logger.log('Skipping ingress event for a disabled or deleted instance', {@@ -188,6 +191,141 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {}}+ /**+ * Hand every 'pending' row created before `cutoff` that still carries its payload to the DLQ, then+ * retire the payload with a 'failed' mark, so the dedup-window prune never deletes an acknowledged+ * delivery that was never dispatched: one stranded across downtime longer than the window, one of an+ * instance disabled for that long, or any stranded row while the sweep is disabled. Same+ * DLQ-before-retire order as the replay budget path. A row whose writes fail stays 'pending' with its+ * payload, which the prune keeps, and is retried on the next run. Skipped while a sweep is running.+ * A row whose instance or session was deleted, or whose queue job still owns the delivery, gets no+ * dead letter. Returns the number of rows dead-lettered.+ */+ async deadLetterAgedPending(cutoff: Date, batchSize = 100): Promise<number> {+ if (this.sweeping) return 0;+ this.sweeping = true;+ let retired = 0;+ try {+ for (;;) {+ const rows = await this.events.find({+ where: { dispatchState: 'pending', createdAt: LessThan(cutoff), payload: Not(IsNull()) },+ order: { createdAt: 'ASC' },+ take: batchSize,+ });+ let progressed = 0;+ for (const row of rows) {+ if (!hasPayload(row)) continue;+ const meta = { pluginId: row.pluginId, instanceId: row.instanceId, deliveryId: row.providerDeliveryId };+ try {+ if (await this.ownerDeleted(row)) {+ // The redrive endpoint refuses a deleted instance, and a session delete purges its dead+ // letters on purpose: retire the payload without writing one back, so the prune drops it.+ await this.events.update(+ { id: row.id, dispatchState: 'pending' },+ { dispatchState: 'failed', payload: null },+ );+ progressed++;+ this.logger.warn('Undispatched ingress event of a deleted instance or session dropped', {+ ...meta,+ action: 'ingress_event_retention_dropped',+ });+ continue;+ }+ const jobData = this.jobDataFor(row);+ // A job still in the queue (or completed) owns the delivery, as in reconcileRow: no dead+ // letter while a copy can still deliver. A failed one falls through to its existing DLQ row.+ const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);+ if (existing && existing !== 'failed') {+ const settled = await this.events.update(+ { id: row.id, dispatchState: 'pending' },+ { dispatchState: 'dispatched', payload: null },+ );+ // As in reconcileRow: the job delivers it, so an inline-failure dead letter must close.+ if (settled.affected) await this.retireOpenDeadLetters(row);+ progressed++;+ continue;+ }+ const written = await this.ensureDeadLetterRow(+ jobData,+ row.dispatchAttempts,+ 'not dispatched within the ingress dedup retention window',+ );+ const marked = await this.events.update(+ { id: row.id, dispatchState: 'pending' },+ { dispatchState: 'failed', payload: null },+ );+ progressed++;+ if (written) {+ // A delivery the dispatch tier made concurrently must not stay redrivable. For a failed+ // event, another node's hand-off may have written a dead letter too, or skipped its own+ // because of this one, so a writer retires its row only when an open one with a lower id+ // exists: the lowest-id open row is never retired, and normally it is the only one left.+ // Ids are random UUIDs, so a writer whose check runs before a lower-id row is visible+ // keeps its own as well, and two open rows can remain.+ const state = marked.affected+ ? 'failed'+ : (await this.events.findOne({ where: { id: row.id }, select: { dispatchState: true } }))+ ?.dispatchState;+ if (state === 'dispatched' || (state === 'failed' && (await this.hasLowerOpenDeadLetter(row, written)))) {+ await this.failures.update({ id: written, redriven: false }, { redriven: true });+ }+ }+ if (!marked.affected) continue;+ retired++;+ this.logger.warn('Ingress event was never dispatched within the dedup retention window; dead-lettered', {+ ...meta,+ action: 'ingress_event_retention_dead_lettered',+ });+ } catch (err) {+ this.logger.error(+ 'Failed to dead-letter an undispatched ingress event; kept for the next run',+ err instanceof Error ? err.message : String(err),+ { ...meta, action: 'ingress_event_retention_dead_letter_failed' },+ );+ }+ }+ if (rows.length < batchSize || progressed === 0) return retired;+ }+ } finally {+ this.sweeping = false;+ }+ }++ // The row's instance is gone, or it was bound to a session that is gone. A wildcard or non-id scope+ // never names a sessions row, so it is never treated as deleted.+ private async ownerDeleted(row: IngressEvent): Promise<boolean> {+ if (!(await this.instances.resolve(row.pluginId, row.instanceId))) return true;+ const sessionId = row.sessionId;+ return !!sessionId && isUUID(sessionId) && !(await this.events.manager.existsBy(Session, { id: sessionId }));+ }++ private async hasLowerOpenDeadLetter(row: IngressEvent, written: string): Promise<boolean> {+ const lower = await this.failures.count({+ where: {+ direction: 'inbound',+ pluginId: row.pluginId,+ instanceId: row.instanceId,+ deliveryId: row.providerDeliveryId,+ redriven: false,+ id: LessThan(written),+ },+ });+ return lower > 0;+ }++ private async retireOpenDeadLetters(row: IngressEvent): Promise<void> {+ await this.failures.update(+ {+ direction: 'inbound',+ pluginId: row.pluginId,+ instanceId: row.instanceId,+ deliveryId: row.providerDeliveryId,+ redriven: false,+ },+ { redriven: true },+ );+ }+private async reconcileRow(row: IngressEvent & { payload: NonNullable<IngressEvent['payload']> },maxAttempts: number,@@ -242,16 +380,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {// Retire any dead-letter row the live path already wrote for this delivery (the inline-failure// case): the replay, or the job or re-queued copy still live in the queue, delivers it, so a later// manual redrive must not deliver it again.- await this.failures.update(- {- direction: 'inbound',- pluginId: row.pluginId,- instanceId: row.instanceId,- deliveryId: row.providerDeliveryId,- redriven: false,- },- { redriven: true },- );+ await this.retireOpenDeadLetters(row);this.logger.log('Replayed stranded ingress event', {pluginId: row.pluginId,instanceId: row.instanceId,diff --git a/src/modules/integration/integration-retention.service.spec.ts b/src/modules/integration/integration-retention.service.spec.tsindex 9161dc1b4..919cddd99 100644--- a/src/modules/integration/integration-retention.service.spec.ts+++ b/src/modules/integration/integration-retention.service.spec.ts@@ -2,6 +2,12 @@ import { DataSource, Repository } from 'typeorm';import { IngressEvent } from './entities/ingress-event.entity';import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';import { IntegrationRetentionService } from './integration-retention.service';+import { IngressReconcilerService } from './ingress-reconciler.service';+import { IngressEnqueueService } from './ingress-enqueue.service';+import { PluginInstanceService } from './plugin-instance.service';+import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';+import { Session } from '../session/entities/session.entity';+import { LoggerService } from '../../common/services/logger.service';const daysAgo = (d: number): Date => {const dt = new Date();@@ -24,6 +30,7 @@ describe('IntegrationRetentionService.pruneOlderThan', () => {service = new IntegrationRetentionService(ds.getRepository(IngressEvent),ds.getRepository(IntegrationDeliveryFailure),+ { deadLetterAgedPending: jest.fn().mockResolvedValue(0) } as unknown as IngressReconcilerService,);const events = ds.getRepository(IngressEvent);@@ -179,6 +186,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =failuresDelete,events: { delete: eventsDelete } as unknown as Repository<IngressEvent>,failures: { delete: failuresDelete } as unknown as Repository<IntegrationDeliveryFailure>,+ reconciler: { deadLetterAgedPending: jest.fn().mockResolvedValue(0) } as unknown as IngressReconcilerService,};};@@ -193,11 +201,11 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =process.env.INGRESS_RETENTION_DAYS = '0';delete process.env.INGRESS_DEDUP_RETENTION_DAYS;const repos = mockRepos();- const svc = new IntegrationRetentionService(repos.events, repos.failures);+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);svc.onModuleInit();// Let the startup prune promise settle (the method runs before the log .then()).- await Promise.resolve();+ await new Promise(resolve => setImmediate(resolve));// Dedup rows are never an audit log — their prune cannot be disabled to infinity.expect(repos.eventsDelete).toHaveBeenCalled();expect(repos.failuresDelete).not.toHaveBeenCalled();@@ -208,7 +216,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =delete process.env.INGRESS_RETENTION_DAYS;delete process.env.INGRESS_DEDUP_RETENTION_DAYS;const repos = mockRepos();- const svc = new IntegrationRetentionService(repos.events, repos.failures);+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);jest.useFakeTimers();try {@@ -234,7 +242,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =delete process.env.INGRESS_RETENTION_DAYS;process.env.INGRESS_DEDUP_RETENTION_DAYS = '21';const repos = mockRepos();- const svc = new IntegrationRetentionService(repos.events, repos.failures);+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);const pruneSpy = jest.spyOn(svc, 'pruneOlderThan');svc.onModuleInit();@@ -246,7 +254,7 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =delete process.env.INGRESS_RETENTION_DAYS;process.env.INGRESS_DEDUP_RETENTION_DAYS = '0';const repos = mockRepos();- const svc = new IntegrationRetentionService(repos.events, repos.failures);+ const svc = new IntegrationRetentionService(repos.events, repos.failures, repos.reconciler);const pruneSpy = jest.spyOn(svc, 'pruneOlderThan');svc.onModuleInit();@@ -255,3 +263,394 @@ describe('IntegrationRetentionService.onModuleInit (retention scheduling)', () =svc.onModuleDestroy();});});++describe('IntegrationRetentionService.pruneOlderThan with undispatched events', () => {+ let ds: DataSource;+ let events: Repository<IngressEvent>;+ let failures: Repository<IntegrationDeliveryFailure>;+ let reconciler: IngressReconcilerService;+ let service: IntegrationRetentionService;+ let resolveInstance: jest.Mock;+ let existingJobState: jest.Mock;+ let logError: jest.SpyInstance;+ const SESSION_ID = '11111111-1111-4111-8111-111111111111';++ beforeEach(async () => {+ ds = new DataSource({+ type: 'better-sqlite3',+ database: ':memory:',+ entities: [IngressEvent, IntegrationDeliveryFailure, Session],+ synchronize: true,+ });+ await ds.initialize();+ events = ds.getRepository(IngressEvent);+ failures = ds.getRepository(IntegrationDeliveryFailure);+ await ds.getRepository(Session).save({ id: SESSION_ID, name: 'sess-1' });+ // A disabled instance, no loaded plugin and the queue off: the hand-off must not depend on the+ // instance being enabled or its plugin being loaded.+ resolveInstance = jest.fn().mockResolvedValue({ enabled: false });+ existingJobState = jest.fn().mockResolvedValue(undefined);+ reconciler = new IngressReconcilerService(+ events,+ failures,+ { existingJobState } as unknown as IngressEnqueueService,+ { getPlugin: () => undefined } as unknown as PluginLoaderService,+ { resolve: resolveInstance } as unknown as PluginInstanceService,+ );+ service = new IntegrationRetentionService(events, failures, reconciler);+ jest.spyOn(LoggerService.prototype, 'warn').mockImplementation(() => undefined);+ logError = jest.spyOn(LoggerService.prototype, 'error').mockImplementation(() => undefined);+ });++ afterEach(async () => {+ jest.restoreAllMocks();+ if (ds.isInitialized) await ds.destroy();+ });++ const insertEvent = (id: string, over: Partial<IngressEvent>) =>+ events+ .createQueryBuilder()+ .insert()+ .values({+ id,+ instanceId: 'inst',+ pluginId: 'plug',+ providerDeliveryId: `d-${id}`,+ route: 'chatwoot',+ payload: { headers: {}, query: {}, body: '{"msg":"hello"}', rawBody: '{"msg":"hello"}', method: 'POST' },+ sessionId: SESSION_ID,+ dispatchState: 'pending',+ createdAt: daysAgo(8),+ ...over,+ })+ .execute();++ it('dead-letters an aged pending event with its payload instead of deleting it unrecoverably', async () => {+ await insertEvent('stranded', {});++ const first = await service.pruneOlderThan(7, 90);++ // The payload is not lost: it now lives in a redrivable dead-letter row, written before the marker+ // was settled, and the settled marker goes with the next run.+ expect(first).toEqual({ events: 0, failures: 0 });+ expect(await events.findOneByOrFail({ id: 'stranded' })).toMatchObject({ dispatchState: 'failed', payload: null });+ expect(await service.pruneOlderThan(7, 90)).toEqual({ events: 1, failures: 0 });+ expect(await events.count()).toBe(0);+ const [dlq] = await failures.find();+ expect(dlq).toMatchObject({+ direction: 'inbound',+ pluginId: 'plug',+ instanceId: 'inst',+ sessionId: SESSION_ID,+ deliveryId: 'd-stranded',+ lastError: 'not dispatched within the ingress dedup retention window',+ redriven: false,+ });+ expect(dlq.payload).toMatchObject({+ route: 'chatwoot',+ method: 'POST',+ ingress: { body: '{"msg":"hello"}', rawBody: '{"msg":"hello"}' },+ });+ });++ it('retires the payload without a second dead letter when the live path already wrote one', async () => {+ await insertEvent('inline-failed', { dispatchAttempts: 1 });+ await failures.save({+ direction: 'inbound',+ pluginId: 'plug',+ instanceId: 'inst',+ deliveryId: 'd-inline-failed',+ attempts: 1,+ lastError: 'sandbox 5xx',+ payload: { route: 'chatwoot', ingress: { headers: {}, query: {}, body: '{}', rawBody: '{}' } },+ redriven: false,+ });+ const update = jest.spyOn(events, 'update');++ await service.pruneOlderThan(7, 90);++ // Settled by the hand-off: the payload is retired against the existing dead letter.+ expect(update).toHaveBeenCalledWith(+ { id: 'inline-failed', dispatchState: 'pending' },+ { dispatchState: 'failed', payload: null },+ );+ const [dlq] = await failures.find();+ expect(dlq).toMatchObject({ deliveryId: 'd-inline-failed', lastError: 'sandbox 5xx', redriven: false });+ expect(await failures.count()).toBe(1);+ });++ it('writes a fresh dead letter when the only one for the delivery id was already redriven', async () => {+ await insertEvent('resent', {});+ const old = await failures.save({+ direction: 'inbound',+ pluginId: 'plug',+ instanceId: 'inst',+ deliveryId: 'd-resent',+ attempts: 1,+ lastError: 'sandbox 5xx',+ payload: null,+ redriven: true,+ });++ await service.pruneOlderThan(7, 90);++ const open = await failures.find({ where: { deliveryId: 'd-resent', redriven: false } });+ expect(open).toHaveLength(1);+ expect(open[0].id).not.toBe(old.id);+ expect(open[0].payload).toMatchObject({ ingress: { rawBody: '{"msg":"hello"}' } });+ });++ it('keeps an aged pending event with its payload when the dead-letter write fails', async () => {+ await insertEvent('stranded', {});+ jest.spyOn(failures, 'save').mockRejectedValueOnce(new Error('disk full'));++ const result = await service.pruneOlderThan(7, 90);++ expect(result.events).toBe(0);+ const kept = await events.findOneByOrFail({ id: 'stranded' });+ expect(kept.dispatchState).toBe('pending');+ expect(kept.payload).toMatchObject({ rawBody: '{"msg":"hello"}' });+ expect(await failures.count()).toBe(0);+ });++ it('retires its own dead letter when a concurrent dispatch settles the event first', async () => {+ await insertEvent('raced', {});+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'dispatched', payload = NULL WHERE id = 'raced'`);+ return { affected: 0, raw: [], generatedMaps: [] };+ });++ await service.pruneOlderThan(7, 90);++ const [dlq] = await failures.find();+ expect(dlq).toMatchObject({ deliveryId: 'd-raced', redriven: true });+ });++ it('keeps its own dead letter redrivable when a concurrent failure mark settles the event first', async () => {+ await insertEvent('raced', {});+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'failed', payload = NULL WHERE id = 'raced'`);+ return { affected: 0, raw: [], generatedMaps: [] };+ });++ await service.pruneOlderThan(7, 90);++ const [dlq] = await failures.find();+ expect(dlq).toMatchObject({ deliveryId: 'd-raced', redriven: false });+ });++ describe('when another node dead-letters the same delivery concurrently', () => {+ const LOW_ID = '00000000-0000-4000-8000-000000000000';+ const HIGH_ID = 'ffffffff-ffff-4fff-bfff-ffffffffffff';+ // The other node's dead letter lands after this node's open-row check, so both write one.+ const otherNodeDeadLetter = (id: string) =>+ failures.insert({+ id,+ direction: 'inbound',+ pluginId: 'plug',+ instanceId: 'inst',+ deliveryId: 'd-raced',+ attempts: 0,+ lastError: 'not dispatched within the ingress dedup retention window',+ payload: { route: 'chatwoot', ingress: { headers: {}, query: {}, body: '{}', rawBody: '{}' } },+ redriven: false,+ });+ const openIds = async () =>+ (await failures.find({ where: { deliveryId: 'd-raced', redriven: false } })).map(f => f.id);++ it('retires its own dead letter when it loses the failure mark to a node with a lower-id row', async () => {+ await insertEvent('raced', {});+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {+ await otherNodeDeadLetter(LOW_ID);+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'failed', payload = NULL WHERE id = 'raced'`);+ return { affected: 0, raw: [], generatedMaps: [] };+ });++ await service.pruneOlderThan(7, 90);++ expect(await openIds()).toEqual([LOW_ID]);+ expect(await failures.count()).toBe(2);+ });++ it('retires its own dead letter when it wins the failure mark but a lower-id row exists', async () => {+ await insertEvent('raced', {});+ const update = events.update.bind(events);+ jest.spyOn(events, 'update').mockImplementationOnce(async (criteria, partial) => {+ await otherNodeDeadLetter(LOW_ID);+ return update(criteria, partial);+ });++ await service.pruneOlderThan(7, 90);++ expect(await openIds()).toEqual([LOW_ID]);+ expect(await events.findOneByOrFail({ id: 'raced' })).toMatchObject({ dispatchState: 'failed', payload: null });+ });++ it('keeps its own dead letter when the other node holds the higher id', async () => {+ await insertEvent('raced', {});+ jest.spyOn(events, 'update').mockImplementationOnce(async () => {+ await otherNodeDeadLetter(HIGH_ID);+ await events.query(`UPDATE ingress_events SET "dispatchState" = 'failed', payload = NULL WHERE id = 'raced'`);+ return { affected: 0, raw: [], generatedMaps: [] };+ });++ await service.pruneOlderThan(7, 90);++ // The other node retires its own row once its check sees this one; this node's row stays.+ const open = await openIds();+ expect(open).toHaveLength(2);+ expect(open).toContain(HIGH_ID);+ });+ });++ it('still prunes aged settled markers, legacy rows, and pending rows without a payload', async () => {+ await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });+ await insertEvent('failed', { dispatchState: 'failed', payload: null });+ await insertEvent('legacy', { dispatchState: null });+ await insertEvent('empty', { payload: null });+ await insertEvent('recent', { createdAt: daysAgo(1) });++ const result = await service.pruneOlderThan(7, 90);++ expect(result.events).toBe(4);+ expect((await events.find()).map(e => e.id)).toEqual(['recent']);+ expect(await failures.count()).toBe(0);+ });++ it('writes no dead letter for a deleted instance and prunes the row', async () => {+ await insertEvent('orphan', {});+ resolveInstance.mockResolvedValue(null);++ await service.pruneOlderThan(7, 90);+ const result = await service.pruneOlderThan(7, 90);++ expect(result.events).toBe(1);+ expect(await events.count()).toBe(0);+ expect(await failures.count()).toBe(0);+ });++ it('writes no dead letter for a deleted session, but keeps one for a wildcard scope', async () => {+ await insertEvent('gone', { sessionId: '22222222-2222-4222-8222-222222222222' });+ await insertEvent('wildcard', { sessionId: '*' });++ await service.pruneOlderThan(7, 90);++ expect(await events.find({ where: { dispatchState: 'failed' } })).toHaveLength(2);+ expect((await failures.find()).map(f => f.deliveryId)).toEqual(['d-wildcard']);+ });++ it('writes no dead letter while a queue job still owns the delivery', async () => {+ await insertEvent('queued', {});+ existingJobState.mockResolvedValue('waiting');+ const update = jest.spyOn(events, 'update');++ await service.pruneOlderThan(7, 90);++ expect(update).toHaveBeenCalledWith(+ { id: 'queued', dispatchState: 'pending' },+ { dispatchState: 'dispatched', payload: null },+ );+ expect(await failures.count()).toBe(0);+ });++ it('closes an inline-failure dead letter once a queue job owns the delivery', async () => {+ await insertEvent('queued', { dispatchAttempts: 1 });+ await failures.save({+ direction: 'inbound',+ pluginId: 'plug',+ instanceId: 'inst',+ deliveryId: 'd-queued',+ attempts: 1,+ lastError: 'sandbox 5xx',+ payload: { route: 'chatwoot', ingress: { headers: {}, query: {}, body: '{}', rawBody: '{}' } },+ redriven: false,+ });+ existingJobState.mockResolvedValue('waiting');++ await service.pruneOlderThan(7, 90);++ // The job delivers it, so a manual redrive of the old row would deliver it twice.+ expect(await events.findOneByOrFail({ id: 'queued' })).toMatchObject({ dispatchState: 'dispatched' });+ expect(await failures.find()).toEqual([expect.objectContaining({ deliveryId: 'd-queued', redriven: true })]);+ });++ it('dead-letters an event whose queue job failed', async () => {+ await insertEvent('job-failed', {});+ existingJobState.mockResolvedValue('failed');++ await service.pruneOlderThan(7, 90);++ // A failed job delivers nothing, so the payload must survive in a redrivable dead letter.+ expect(await events.findOneByOrFail({ id: 'job-failed' })).toMatchObject({+ dispatchState: 'failed',+ payload: null,+ });+ const open = await failures.find({ where: { direction: 'inbound', redriven: false } });+ expect(open).toHaveLength(1);+ expect(open[0]).toMatchObject({ deliveryId: 'd-job-failed' });+ expect(open[0].payload).toMatchObject({ ingress: { rawBody: '{"msg":"hello"}' } });+ });++ it('still prunes aged events when the dead-letter prune fails', async () => {+ await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });+ jest.spyOn(failures, 'delete').mockRejectedValueOnce(new Error('locked'));++ const result = await service.pruneOlderThan(7, 90);++ expect(result).toEqual({ events: 1, failures: 0 });+ expect(await events.count()).toBe(0);+ });++ it('hands off a backlog larger than one batch', async () => {+ for (let i = 0; i < 250; i++) await insertEvent(`b${i}`, {});++ await service.pruneOlderThan(7, 90);++ expect(await events.count({ where: { dispatchState: 'failed' } })).toBe(250);+ expect(await failures.count()).toBe(250);+ });++ it('prunes aged settled markers while the hand-off is still waiting', async () => {+ await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });+ await insertEvent('stranded', {});+ // A job-state lookup against a queue that never became reachable does not return.+ const lookedUp = new Promise<void>(resolve =>+ existingJobState.mockImplementation(() => {+ resolve();+ return new Promise(() => {});+ }),+ );++ void service.pruneOlderThan(7, 90);+ await lookedUp;++ expect((await events.find()).map(e => e.id)).toEqual(['stranded']);+ });++ it('stops and keeps every row when no dead-letter write succeeds', async () => {+ for (let i = 0; i < 100; i++) await insertEvent(`f${i}`, {});+ jest.spyOn(failures, 'save').mockRejectedValue(new Error('disk full'));++ const result = await service.pruneOlderThan(7, 90);++ expect(result.events).toBe(0);+ expect(await events.count({ where: { dispatchState: 'pending' } })).toBe(100);+ expect(logError).toHaveBeenCalledTimes(100);+ expect(logError).toHaveBeenCalledWith(+ 'Failed to dead-letter an undispatched ingress event; kept for the next run',+ 'disk full',+ expect.objectContaining({ action: 'ingress_event_retention_dead_letter_failed' }),+ );+ });++ it('keeps aged pending rows while a reconcile sweep holds the guard', async () => {+ await insertEvent('stranded', {});+ (reconciler as unknown as { sweeping: boolean }).sweeping = true;++ const result = await service.pruneOlderThan(7, 90);++ expect(result.events).toBe(0);+ expect((await events.findOneByOrFail({ id: 'stranded' })).payload).not.toBeNull();+ expect(await failures.count()).toBe(0);+ });+});diff --git a/src/modules/integration/integration-retention.service.ts b/src/modules/integration/integration-retention.service.tsindex 1f841e570..20b5c04ba 100644--- a/src/modules/integration/integration-retention.service.ts+++ b/src/modules/integration/integration-retention.service.ts@@ -1,8 +1,9 @@import { Injectable, OnModuleDestroy, OnModuleInit } from '@nestjs/common';import { InjectRepository } from '@nestjs/typeorm';-import { LessThan, Repository } from 'typeorm';+import { IsNull, LessThan, Not, Repository } from 'typeorm';import { IngressEvent } from './entities/ingress-event.entity';import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';+import { IngressReconcilerService } from './ingress-reconciler.service';import { createLogger } from '../../common/services/logger.service';// Dedup rows are a delivery-id oracle, not an audit log: provider retries arrive within minutes, so@@ -23,7 +24,8 @@ const DEFAULT_FAILURE_RETENTION_DAYS = 90;* - `INGRESS_DEDUP_RETENTION_DAYS` (default 7) prunes `ingress_events`. <= 0 does NOT disable it —* an unpruned dedup table grows without bound for zero functional gain (its payloads are retired* on dispatch and its audit value is nil), so a non-positive value falls back to the default with- * a warning instead of silently opting into unbounded growth.+ * a warning instead of silently opting into unbounded growth. A 'pending' row that still carries+ * its payload is only deleted after deadLetterAgedPending has settled it.* - `INGRESS_RETENTION_DAYS` (default 90) prunes `integration_delivery_failures` — the redrive* payload store, where long retention can be a deliberate operator choice. <= 0 disables that* prune (and only that prune).@@ -37,6 +39,7 @@ export class IntegrationRetentionService implements OnModuleInit, OnModuleDestro@InjectRepository(IngressEvent, 'data') private readonly eventRepository: Repository<IngressEvent>,@InjectRepository(IntegrationDeliveryFailure, 'data')private readonly failureRepository: Repository<IntegrationDeliveryFailure>,+ private readonly reconciler: IngressReconcilerService,) {}onModuleInit(): void {@@ -86,7 +89,8 @@ export class IntegrationRetentionService implements OnModuleInit, OnModuleDestro/*** Delete ingress_events rows older than `eventsDays` and integration_delivery_failures rows older* than `failuresDays` (null skips the failures prune; omitted = same window as events). Returns- * the number removed from each table.+ * the number removed from each table. Aged 'pending' events that still carry a payload are kept and+ * settled by deadLetterAgedPending, so the next run deletes them; one whose hand-off fails stays.*/async pruneOlderThan(eventsDays: number,@@ -96,12 +100,32 @@ export class IntegrationRetentionService implements OnModuleInit, OnModuleDestroeventsCutoff.setDate(eventsCutoff.getDate() - eventsDays);const failuresCutoff = new Date();if (failuresDays !== null) failuresCutoff.setDate(failuresCutoff.getDate() - failuresDays);- const [eventsResult, failuresResult] = await Promise.all([- this.eventRepository.delete({ createdAt: LessThan(eventsCutoff) }),+ // DLQ first, so an undispatched event whose only dead letter just aged out is handed a fresh one.+ // The dedup delete keeps every 'pending' row that still carries its payload, so it runs before the+ // hand-off, which can wait on an unreachable queue for its job-state lookup; the rows the hand-off+ // settles go on the next run. A failed step is logged and never blocks the next one.+ const logFailure = (step: string) => (err: unknown) => {+ this.logger.error(+ `Integration ingress retention: ${step} failed`,+ err instanceof Error ? err.stack : String(err),+ );+ return { affected: 0 };+ };+ const failuresResult =failuresDays === null- ? Promise.resolve({ affected: 0 })- : this.failureRepository.delete({ createdAt: LessThan(failuresCutoff) }),- ]);+ ? { affected: 0 }+ : await this.failureRepository+ .delete({ createdAt: LessThan(failuresCutoff) })+ .catch(logFailure('delivery-failure prune'));+ const aged = LessThan(eventsCutoff);+ const eventsResult = await this.eventRepository+ .delete([+ { createdAt: aged, dispatchState: Not('pending') },+ { createdAt: aged, dispatchState: IsNull() },+ { createdAt: aged, payload: IsNull() },+ ])+ .catch(logFailure('dedup prune'));+ await this.reconciler.deadLetterAgedPending(eventsCutoff).catch(logFailure('undispatched event hand-off'));return { events: eventsResult.affected || 0, failures: failuresResult.affected || 0 };}}