Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(webhook): preserve replay data through failures

Yudhi Armyndharisfc2d2c38f212
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9922b7e4a..34afdd5ae 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,7 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
-- `POST /api/webhooks/delivery-failures/redrive` (ADMIN) replays recorded webhook deliveries that still hold their event data, in batches of up to 500, oldest first, with the stored idempotency key so a receiver can dedup a replay; `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` sets how long a terminal failure row keeps that data (default `0`: not kept, nothing to replay), and `GET /api/webhooks/delivery-failures` marks each row `replayable`. All five SDKs gain the call in their next release.
+- Add bounded webhook failure redrive with optional payload retention.
+
+### Fixed
+
+- Preserve webhook outbox payloads until delivery or durable failure storage succeeds.
+- Enforce webhook replay retention and keep later eligible failures moving through batches.
+- Keep one terminal webhook failure per receiver idempotency key.
## [0.24.0] - 2026-10-03
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 0188978ba..6b2d31eed 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -4674,15 +4674,17 @@ A delivery shed because the dispatch queue was full, or refused during shutdown,
**Errors:** `401` missing/invalid API key · `403` key role below ADMIN
+Queued deliveries retain their pre-hook outbox copy until the worker completes delivery or stores a terminal failure. The reconciler checks the current job before replaying and keeps rows while Redis cannot be inspected. If failure persistence is unavailable after the replay budget is spent, it retries the storage handoff without another POST. This recovery boundary is best effort before the outbox row is created; receivers must still deduplicate repeated POSTs.
+
#### POST /api/webhooks/delivery-failures/redrive
-Replay recorded webhook deliveries that still hold their event data, in one bounded batch, oldest first. A row is replayable when it is terminal (`attempts` > 0; an `attempts: 0` row is the outbox's to replay) and was recorded while `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0, until that window passes. With the default `0` nothing is kept and this call replays nothing.
+Replay recorded webhook deliveries that still hold their event data, in one bounded batch, with the lowest attempt counts first, then oldest first. A row is replayable when it is terminal (`attempts` > 0; an `attempts: 0` row is the outbox's to replay) and was recorded while `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0, until that window passes. With the default `0` nothing is kept and this call replays nothing.
-Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempotency-Key`), so a receiver that already handled the event — the POST timed out after it was processed — can dedup the replay instead of acting twice. `webhook:before` hooks run again on the stored pre-hook event data, as on an outbox replay. With the queue disabled each row gets **one** attempt inside the request; with the queue enabled the row is enqueued with the webhook's normal retry policy. A replay that is delivered removes its row; one that fails again keeps the row and raises its `attempts` by one (no second row is filed). Calls on one node run one after another, and every call is written to the audit log with its counts (no payload content).
+Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempotency-Key`), so receivers can deduplicate a POST that timed out after processing. `webhook:before` hooks run again on the stored pre-hook data. Each row gets **one direct POST** inside the request, including when ordinary dispatch uses the queue. Success removes its row; another failure keeps it and raises its attempt count. Calls on one node run serially. The audit log records counts without payload content.
**Auth:** API key (ADMIN) · **Scope:** rows are confined to the calling key's `allowedSessions`; the body's `sessionId` can only narrow that. A row is replayed only to the webhook it was recorded for, and only while that webhook still belongs to the row's session.
-**Request body** (every field optional; an empty body `{}` takes the oldest rows)
+**Request body** (every field optional; an empty body `{}` takes the eligible rows with the fewest attempts)
| Field | Type | Required | Validation | Description |
| ----------- | --------------- | -------- | -------------------------------------------- | ------------------------------------------------------------------------- |
@@ -4696,15 +4698,15 @@ Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempote
```json
{
"redriven": 3,
- "delivered": 2,
- "enqueued": 1,
+ "delivered": 3,
+ "enqueued": 0,
"failed": 0,
"skipped": 0,
"remaining": 0
}
```
-`redriven` is `delivered` + `enqueued`. `skipped` counts rows not replayed because the webhook was removed, disabled or no longer subscribes to the event, or a `webhook:before` hook cancelled it; those rows stay. `remaining` is the number of replayable rows still in scope after the call — call again while it is above `0`.
+`redriven` equals `delivered`; `enqueued` is retained for response compatibility and is always `0`. Removed, disabled, unsubscribed, expired and out-of-scope rows are excluded before applying the limit. `skipped` counts selected rows cancelled by a hook or invalidated during the call. `remaining` counts eligible rows in the same requested session, webhook and IDs after the call. Retry another batch only after resolving persistent failures or hook cancellations; `remaining > 0` alone does not mean another call will succeed.
**Errors:** `400` body fails validation · `401` missing/invalid API key · `403` key role below ADMIN
diff --git a/openapi.json b/openapi.json
index 9ed90cb29..0b51c7b0b 100644
--- a/openapi.json
+++ b/openapi.json
@@ -19247,7 +19247,7 @@
},
"enqueued": {
"type": "number",
- "description": "Handed to the queue; each row is removed when its job delivers and kept if it fails again.",
+ "description": "Reserved for response compatibility; operator redrive uses direct POST and returns zero.",
"example": 1
},
"failed": {
diff --git a/sdk/go/types_webhook.go b/sdk/go/types_webhook.go
index 104843180..3dab58f4f 100644
--- a/sdk/go/types_webhook.go
+++ b/sdk/go/types_webhook.go
@@ -211,7 +211,7 @@ type WebhookDeliveryFailure struct {
}
// RedriveWebhookDeliveriesRequest is the body of RedriveDeliveryFailures.
-// Every field narrows; an empty body takes the oldest rows.
+// Every field narrows; an empty body takes eligible rows with the fewest attempts.
type RedriveWebhookDeliveriesRequest struct {
// SessionID limits the batch to one session (within the key's allowedSessions).
SessionID string `json:"sessionId,omitempty"`
@@ -229,8 +229,7 @@ type WebhookRedriveResult struct {
Redriven int `json:"redriven"`
// Delivered rows went out by a direct POST; their failure rows were removed.
Delivered int `json:"delivered"`
- // Enqueued rows were handed to the queue; each is removed when its job
- // delivers and kept if it fails again.
+ // Enqueued is reserved for compatibility; operator redrive always returns zero.
Enqueued int `json:"enqueued"`
// Failed replays failed again; their rows stay, with attempts raised by one.
Failed int `json:"failed"`
diff --git a/sdk/java/src/main/java/com/rmyndharis/openwa/model/WebhookRedriveResult.java b/sdk/java/src/main/java/com/rmyndharis/openwa/model/WebhookRedriveResult.java
index aa5680ade..1a9253108 100644
--- a/sdk/java/src/main/java/com/rmyndharis/openwa/model/WebhookRedriveResult.java
+++ b/sdk/java/src/main/java/com/rmyndharis/openwa/model/WebhookRedriveResult.java
@@ -5,8 +5,7 @@ package com.rmyndharis.openwa.model;
*
* @param redriven rows replayed by this call: delivered plus enqueued
* @param delivered delivered by a direct POST; their failure rows were removed
- * @param enqueued handed to the queue; each row is removed when its job delivers and kept if it
- * fails again
+ * @param enqueued reserved for compatibility; operator redrive always returns zero
* @param failed replays that failed again; their rows stay, with attempts raised by one
* @param skipped rows not replayed: the webhook was removed, disabled or unsubscribed, or a plugin
* cancelled it
diff --git a/src/database/migrations/1787200000000-DeduplicateTerminalWebhookFailures.ts b/src/database/migrations/1787200000000-DeduplicateTerminalWebhookFailures.ts
new file mode 100644
index 000000000..84af974aa
--- /dev/null
+++ b/src/database/migrations/1787200000000-DeduplicateTerminalWebhookFailures.ts
@@ -0,0 +1,28 @@
+import { MigrationInterface, QueryRunner } from 'typeorm';
+
+/** Keep one terminal record per receiver deduplication identity. */
+export class DeduplicateTerminalWebhookFailures1787200000000 implements MigrationInterface {
+ name = 'DeduplicateTerminalWebhookFailures1787200000000';
+
+ async up(queryRunner: QueryRunner): Promise<void> {
+ if (queryRunner.dataSource.options.type === 'postgres') await queryRunner.query('SET LOCAL statement_timeout = 0');
+ // Prefer a replay copy, then the largest attempt count and most recent record.
+ // Unkeyed failures have no shared identity and remain separate.
+ await queryRunner.query(`DELETE FROM "webhook_delivery_failures" WHERE "id" IN (
+ SELECT "id" FROM (
+ SELECT "id", ROW_NUMBER() OVER (
+ PARTITION BY "webhookId", "idempotencyKey"
+ ORDER BY CASE WHEN "payload" IS NOT NULL THEN 0 ELSE 1 END,
+ "attempts" DESC, "createdAt" DESC, "id" DESC
+ ) AS ordinal FROM "webhook_delivery_failures"
+ WHERE "attempts" > 0 AND "idempotencyKey" IS NOT NULL
+ ) AS ranked WHERE ordinal > 1
+ )`);
+ await queryRunner.query(`CREATE UNIQUE INDEX IF NOT EXISTS "UQ_webhook_delivery_failures_terminal"
+ ON "webhook_delivery_failures" ("webhookId", "idempotencyKey") WHERE "attempts" > 0`);
+ }
+
+ async down(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query('DROP INDEX IF EXISTS "UQ_webhook_delivery_failures_terminal"');
+ }
+}
diff --git a/src/database/migrations/__tests__/1787200000000-DeduplicateTerminalWebhookFailures.spec.ts b/src/database/migrations/__tests__/1787200000000-DeduplicateTerminalWebhookFailures.spec.ts
new file mode 100644
index 000000000..4d5a1ab33
--- /dev/null
+++ b/src/database/migrations/__tests__/1787200000000-DeduplicateTerminalWebhookFailures.spec.ts
@@ -0,0 +1,30 @@
+import { DataSource } from 'typeorm';
+import { DeduplicateTerminalWebhookFailures1787200000000 } from '../1787200000000-DeduplicateTerminalWebhookFailures';
+
+describe('terminal webhook failure identity migration', () => {
+ it('preserves replay data while merging keyed duplicates and allows rollback', async () => {
+ const ds = await new DataSource({ type: 'better-sqlite3', database: ':memory:' }).initialize();
+ try {
+ await ds.query(`CREATE TABLE webhook_delivery_failures (id text PRIMARY KEY, "webhookId" text,
+ "idempotencyKey" text, attempts integer, payload text, "createdAt" text)`);
+ await ds.query(`INSERT INTO webhook_delivery_failures VALUES
+ ('a','w','k',5,NULL,'2026-10-05'), ('b','w','k',3,'{"body":"retained"}','2026-10-04'),
+ ('c','w','k',0,NULL,'2026-10-03'), ('d','w',NULL,3,NULL,'2026-10-05'), ('e','w',NULL,3,NULL,'2026-10-05')`);
+ const migration = new DeduplicateTerminalWebhookFailures1787200000000();
+ const runner = ds.createQueryRunner();
+ await migration.up(runner);
+ await migration.up(runner);
+ const rows = await ds.query<Array<{ id: string }>>('SELECT id FROM webhook_delivery_failures ORDER BY id');
+ expect(rows.map(r => r.id)).toEqual(['b', 'c', 'd', 'e']);
+ await expect(
+ ds.query(`INSERT INTO webhook_delivery_failures VALUES ('f','w','k',1,NULL,'2026-10-05')`),
+ ).rejects.toThrow();
+ await migration.down(runner);
+ await expect(
+ ds.query(`INSERT INTO webhook_delivery_failures VALUES ('f','w','k',1,NULL,'2026-10-05')`),
+ ).resolves.toBeDefined();
+ } finally {
+ await ds.destroy();
+ }
+ });
+});
diff --git a/src/modules/queue/processors/webhook.processor.spec.ts b/src/modules/queue/processors/webhook.processor.spec.ts
index 6da78d869..ba4e99c84 100644
--- a/src/modules/queue/processors/webhook.processor.spec.ts
+++ b/src/modules/queue/processors/webhook.processor.spec.ts
@@ -1,6 +1,7 @@
import { DelayedError, Job } from 'bullmq';
import { FindOperator, Repository } from 'typeorm';
import { ConfigService } from '@nestjs/config';
+import { WebhookOutboxService } from '../../webhook/webhook-outbox.service';
import { WebhookProcessor } from './webhook.processor';
import { Webhook } from '../../webhook/entities/webhook.entity';
import { WebhookDeliveryFailure } from '../../webhook/entities/webhook-delivery-failure.entity';
@@ -24,7 +25,8 @@ jest.mock('undici', () => {
describe('WebhookProcessor', () => {
let processor: WebhookProcessor;
let repo: { update: jest.Mock; findOne: jest.Mock };
- let failureRepo: { insert: jest.Mock; count: jest.Mock; delete: jest.Mock };
+ let failureRepo: { insert: jest.Mock; count: jest.Mock; delete: jest.Mock; update: jest.Mock };
+ let outbox: { close: jest.Mock };
let failureRows: Array<{ webhookId?: string; idempotencyKey?: string | null; attempts?: number }>;
let hookManager: { execute: jest.Mock };
let configService: { get: jest.Mock };
@@ -65,7 +67,9 @@ describe('WebhookProcessor', () => {
// inserts, so a constant would leave that guard unexercised here and let a duplicated row pass.
failureRows = [];
const insertedFailures = failureRows;
+ outbox = { close: jest.fn().mockResolvedValue(undefined) };
failureRepo = {
+ update: jest.fn().mockResolvedValue({ affected: 1 }),
insert: jest.fn().mockImplementation((rowToInsert: { webhookId?: string; idempotencyKey?: string | null }) => {
insertedFailures.push(rowToInsert);
return Promise.resolve({});
@@ -106,6 +110,7 @@ describe('WebhookProcessor', () => {
failureRepo as unknown as Repository<WebhookDeliveryFailure>,
hookManager as unknown as HookManager,
configService as unknown as ConfigService,
+ outbox as unknown as WebhookOutboxService,
);
// The merged delivery path uses withSafeFetch (undici), so mock undici's fetch, not global.fetch.
mockFetch = undiciFetch as jest.Mock;
@@ -187,7 +192,17 @@ describe('WebhookProcessor', () => {
);
});
+ it('keeps queue ownership in the outbox when the terminal database write fails', async () => {
+ failureRepo.insert.mockRejectedValue(new Error('disk unavailable'));
+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Unavailable' });
+ await expect(processor.process(makeJob({}, 2))).rejects.toThrow('HTTP 503');
+ expect(outbox.close).not.toHaveBeenCalled();
+ });
+
it('keeps the job replay copy (pre-hook data) on the final-attempt row, never the sent body', async () => {
+ configService.get.mockImplementation((key: string, def?: unknown) =>
+ key === 'webhook.failurePayloadRetentionHours' ? 24 : def,
+ );
mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });
const job = makeJob({ maxRetries: 3, replayData: { from: '[email protected]', body: 'hi' } }, 2);
job.data.payload.data = { redacted: true };
@@ -199,6 +214,15 @@ describe('WebhookProcessor', () => {
);
});
+ it.each(['terminal', 'stalled'])('drops an old queued replay copy after retention is disabled (%s)', async mode => {
+ const job = makeJob({ replayData: { body: 'private' } }, 2);
+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Unavailable' });
+ if (mode === 'terminal') await expect(processor.process(job)).rejects.toThrow();
+ else await processor.onWorkerFailed(job, new Error('job stalled more than allowable limit'));
+ expect(failureRepo.insert).toHaveBeenCalledTimes(1);
+ expect((failureRepo.insert.mock.calls as unknown[][])[0][0]).not.toHaveProperty('payload');
+ });
+
it('writes no payload for a job enqueued with payload retention off', async () => {
mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });
@@ -505,6 +529,7 @@ describe('WebhookProcessor', () => {
failureRepo as unknown as Repository<WebhookDeliveryFailure>,
hookManager as unknown as HookManager,
configService as unknown as ConfigService,
+ outbox as unknown as WebhookOutboxService,
);
};
const withMoveToDelayed = (job: Job<WebhookJobData>): jest.Mock => {
diff --git a/src/modules/queue/processors/webhook.processor.ts b/src/modules/queue/processors/webhook.processor.ts
index fb539a65c..fd4aadf96 100644
--- a/src/modules/queue/processors/webhook.processor.ts
+++ b/src/modules/queue/processors/webhook.processor.ts
@@ -8,6 +8,7 @@ import { QUEUE_NAMES } from '../queue-names';
import { workerConnectionOptions, webhookWorkerConcurrency } from '../redis-connection';
import { WebhookJobData, WebhookPayload } from '../../webhook/webhook.service';
import { Webhook } from '../../webhook/entities/webhook.entity';
+import { WebhookOutboxService } from '../../webhook/webhook-outbox.service';
import { WebhookDeliveryFailure } from '../../webhook/entities/webhook-delivery-failure.entity';
import {
clearDeliveryFailureRows,
@@ -71,6 +72,7 @@ export class WebhookProcessor extends WorkerHost {
private readonly failureRepository: Repository<WebhookDeliveryFailure>,
private readonly hookManager: HookManager,
private readonly configService: ConfigService,
+ private readonly outbox: WebhookOutboxService,
) {
super();
// WEBHOOK_DEGRADED_SESSION_CONCURRENCY, else a quarter of the worker pool.
@@ -156,6 +158,7 @@ export class WebhookProcessor extends WorkerHost {
action: 'webhook_skipped_stale',
});
this.failingWebhooks.delete(webhookId);
+ await this.outbox.close(webhookId, payload.idempotencyKey, 'dispatched');
return { statusCode: 0, success: false, error: 'webhook removed, disabled or unsubscribed', responseTime: 0 };
}
ctx.url = current.url;
@@ -171,6 +174,7 @@ export class WebhookProcessor extends WorkerHost {
const { status, responseTime } = await this.postToReceiver(ctx, body, requestHeaders);
this.failingWebhooks.delete(webhookId);
+ await this.outbox.close(webhookId, payload.idempotencyKey, 'dispatched');
await this.recordSuccessfulDelivery(ctx, status, responseTime);
return {
statusCode: status,
@@ -323,11 +327,12 @@ export class WebhookProcessor extends WorkerHost {
lastStatusCode: statusCodeFromError(errorMessage),
lastError: clientError,
// The producer attaches the pre-hook event data only while payload retention is on.
- ...(job.data.replayData ? { payload: job.data.replayData } : {}),
+ ...(this.configService.get<number>('webhook.failurePayloadRetentionHours', 0) > 0 && job.data.replayData
+ ? { payload: job.data.replayData }
+ : {}),
});
- if (recorded) {
- incrementWebhookDeliveryFailures();
- }
+ if (recorded !== false) incrementWebhookDeliveryFailures();
+ if (recorded !== null) await this.outbox.close(webhookId, payload.idempotencyKey, 'dispatched');
}
}
@@ -396,13 +401,14 @@ export class WebhookProcessor extends WorkerHost {
url,
idempotencyKey: payload.idempotencyKey,
deliveryId: payload.deliveryId,
- attempts: job.attemptsMade,
+ attempts: Math.max(1, job.attemptsMade),
lastStatusCode: null, // no HTTP exchange completed on the stalled attempts
lastError: error.message,
- ...(job.data.replayData ? { payload: job.data.replayData } : {}),
+ ...(this.configService.get<number>('webhook.failurePayloadRetentionHours', 0) > 0 && job.data.replayData
+ ? { payload: job.data.replayData }
+ : {}),
});
- if (recorded) {
- incrementWebhookDeliveryFailures();
- }
+ if (recorded !== false) incrementWebhookDeliveryFailures();
+ if (recorded !== null) await this.outbox.close(webhookId, payload.idempotencyKey, 'dispatched');
}
}
diff --git a/src/modules/queue/queue.module.ts b/src/modules/queue/queue.module.ts
index 4d64ba0b8..243429d9a 100644
--- a/src/modules/queue/queue.module.ts
+++ b/src/modules/queue/queue.module.ts
@@ -9,6 +9,8 @@ import { IngressProcessor } from './processors/ingress.processor';
import { QUEUE_NAMES } from './queue-names';
import { queueConnectionOptions } from './redis-connection';
import { Webhook } from '../webhook/entities/webhook.entity';
+import { WebhookOutboxService } from '../webhook/webhook-outbox.service';
+import { WebhookOutboxEvent } from '../webhook/entities/webhook-outbox-event.entity';
import { WebhookDeliveryFailure } from '../webhook/entities/webhook-delivery-failure.entity';
import { IntegrationDeliveryFailure } from '../integration/entities/integration-delivery-failure.entity';
import { IngressEvent } from '../integration/entities/ingress-event.entity';
@@ -34,7 +36,10 @@ export const WEBHOOK_QUEUE_JOB_OPTIONS = {
// Required for WebhookProcessor to inject Repository<Webhook> + Repository<WebhookDeliveryFailure>;
// IngressProcessor to inject Repository<IntegrationDeliveryFailure> + Repository<IngressEvent> (all on
// the 'data' connection).
- TypeOrmModule.forFeature([Webhook, WebhookDeliveryFailure, IntegrationDeliveryFailure, IngressEvent], 'data'),
+ TypeOrmModule.forFeature(
+ [Webhook, WebhookDeliveryFailure, WebhookOutboxEvent, IntegrationDeliveryFailure, IngressEvent],
+ 'data',
+ ),
// Required for WebhookProcessor/IngressProcessor to inject HookManager
HooksModule,
// Required for IngressProcessor to inject PluginLoaderService (already @Global(), imported
@@ -69,7 +74,7 @@ export const WEBHOOK_QUEUE_JOB_OPTIONS = {
adapter: BullMQAdapter,
}),
],
- providers: [WebhookProcessor, IngressProcessor],
+ providers: [WebhookProcessor, IngressProcessor, WebhookOutboxService],
exports: [BullModule],
})
export class QueueModule {}
diff --git a/src/modules/webhook/dto/webhook.dto.ts b/src/modules/webhook/dto/webhook.dto.ts
index 78b2e7e03..2b159ea29 100644
--- a/src/modules/webhook/dto/webhook.dto.ts
+++ b/src/modules/webhook/dto/webhook.dto.ts
@@ -428,7 +428,7 @@ export class WebhookDeliveryFailureDto {
replayable!: boolean;
}
-/** Body of `POST /webhooks/delivery-failures/redrive`. Every field narrows; an empty body takes the oldest rows. */
+/** Body of `POST /webhooks/delivery-failures/redrive`. Every field narrows; an empty body takes eligible rows with the fewest attempts. */
export class RedriveWebhookDeliveriesDto {
@ApiPropertyOptional({
description: "Only rows of this session. Narrows within the calling key's allowedSessions, never past it.",
@@ -474,7 +474,7 @@ export class WebhookRedriveResultDto {
delivered!: number;
@ApiProperty({
- description: 'Handed to the queue; each row is removed when its job delivers and kept if it fails again.',
+ description: 'Reserved for response compatibility; operator redrive uses direct POST and returns zero.',
example: 1,
})
enqueued!: number;
diff --git a/src/modules/webhook/entities/webhook-delivery-failure.entity.ts b/src/modules/webhook/entities/webhook-delivery-failure.entity.ts
index 8cf658a0e..58448e8fa 100644
--- a/src/modules/webhook/entities/webhook-delivery-failure.entity.ts
+++ b/src/modules/webhook/entities/webhook-delivery-failure.entity.ts
@@ -17,6 +17,10 @@ import { jsonColumnType } from '../../../common/utils/column-types';
// Backs the before-insert duplicate lookup that keeps one row per lost delivery rather than one per
// reconciler replay. See AddWebhookDeliveryFailureLookupIndex1786300000000.
@Index('IDX_webhook_delivery_failures_delivery', ['webhookId', 'idempotencyKey'])
+@Index('UQ_webhook_delivery_failures_terminal', ['webhookId', 'idempotencyKey'], {
+ unique: true,
+ where: '"attempts" > 0',
+})
export class WebhookDeliveryFailure {
@PrimaryGeneratedColumn('uuid')
id!: string;
diff --git a/src/modules/webhook/entities/webhook-outbox-event.entity.ts b/src/modules/webhook/entities/webhook-outbox-event.entity.ts
index 5056e5197..a1cc54506 100644
--- a/src/modules/webhook/entities/webhook-outbox-event.entity.ts
+++ b/src/modules/webhook/entities/webhook-outbox-event.entity.ts
@@ -2,22 +2,10 @@ import { Column, CreateDateColumn, Entity, Index, PrimaryGeneratedColumn } from
import { dateColumnType, jsonColumnType } from '../../../common/utils/column-types';
import { DateTransformer } from '../../../common/transformers/date.transformer';
-// The dispatch lifecycle of one outbound delivery, mirroring ingress_events on the inbound side:
-// - 'pending' - the row was written before the attempt; nothing durable owns the delivery yet.
-// The reconciler sweeps these.
-// - 'dispatched' - the delivery reached a durable owner: handed to BullMQ, or completed inline in
-// direct mode. A failure INSIDE that owner dead-letters separately through
-// webhook_delivery_failures, so a 'dispatched' row is never the reconciler's
-// concern. Retiring on ENQUEUE rather than on the POST is what stops the
-// reconciler duplicating work BullMQ already owns.
-// - 'failed' - terminal, set only by the reconciler: either the replay budget is spent
-// (recovery continues through the failure row), or the webhook was removed,
-// disabled or unsubscribed from the event before a replay (no failure row;
-// nothing to recover). The gated direct path closes that second case as
-// 'dispatched' instead.
-// NULL marks rows that predate these columns on a synchronize-bootstrapped database. NULL reads as
-// "not watched", so an upgrade can never mass-replay history.
-export type WebhookOutboxState = 'pending' | 'dispatched' | 'failed';
+// pending: recorded before dispatch; queued: BullMQ owns the current job.
+// Both retain replay data. The worker settles queued rows after success or durable failure.
+// dispatched/failed: settled, payload retired. NULL denotes unwatched legacy rows.
+export type WebhookOutboxState = 'pending' | 'queued' | 'dispatched' | 'failed';
/**
* Durable record of an outbound webhook delivery, written before the attempt.
@@ -53,8 +41,7 @@ export class WebhookOutboxEvent {
@Column()
deliveryId!: string;
- // Retired to NULL the moment an outcome is recorded: only 'pending' rows are replayable, and a
- // dispatched or failed row has no reason to keep a payload that can carry a whole message body.
+ // Pending and queued rows keep their payload until delivery or durable failure handoff.
@Column({ type: jsonColumnType(), nullable: true })
payload!: Record<string, unknown> | null;
diff --git a/src/modules/webhook/utils/deliver-once.ts b/src/modules/webhook/utils/deliver-once.ts
index ebf796c6a..008636f0b 100644
--- a/src/modules/webhook/utils/deliver-once.ts
+++ b/src/modules/webhook/utils/deliver-once.ts
@@ -113,7 +113,7 @@ export async function recordTerminalFailure(
failureRepository: Repository<WebhookDeliveryFailure>,
logger: LoggerService,
input: Omit<Parameters<typeof recordWebhookDeliveryFailure>[2], 'lastStatusCode' | 'lastError'> & { error: unknown },
-): Promise<boolean> {
+): Promise<boolean | null> {
const { error, ...row } = input;
const errMessage = redactSsrfError(error);
return recordWebhookDeliveryFailure(failureRepository, logger, {
diff --git a/src/modules/webhook/utils/record-delivery-failure.spec.ts b/src/modules/webhook/utils/record-delivery-failure.spec.ts
index 12f56a161..774615045 100644
--- a/src/modules/webhook/utils/record-delivery-failure.spec.ts
+++ b/src/modules/webhook/utils/record-delivery-failure.spec.ts
@@ -1,291 +1,131 @@
-import { MoreThan, Repository } from 'typeorm';
+import { DataSource, Repository } from 'typeorm';
+import { mkdtempSync, rmSync } from 'fs';
+import { join } from 'path';
+import { tmpdir } from 'os';
import { WebhookDeliveryFailure } from '../entities/webhook-delivery-failure.entity';
import { clearDeliveryFailureRows, recordWebhookDeliveryFailure, statusCodeFromError } from './record-delivery-failure';
-describe('statusCodeFromError', () => {
- it('parses the status from an "HTTP <code>: ..." message', () => {
- expect(statusCodeFromError('HTTP 503: Service Unavailable')).toBe(503);
- expect(statusCodeFromError('HTTP 404: Not Found')).toBe(404);
- });
-
- it('returns null for a non-HTTP error (network / timeout / SSRF)', () => {
- expect(statusCodeFromError('The operation was aborted due to timeout')).toBeNull();
+const input = {
+ webhookId: 'w',
+ sessionId: 's',
+ event: 'message.received',
+ url: 'https://receiver.example',
+ idempotencyKey: 'key',
+ deliveryId: 'd',
+ attempts: 3,
+ lastStatusCode: 503,
+ lastError: 'HTTP 503: down',
+};
+
+describe('webhook failure persistence', () => {
+ let ds: DataSource;
+ let second: DataSource;
+ let directory: string;
+ let repo: Repository<WebhookDeliveryFailure>;
+ const logger = { error: jest.fn() };
+ beforeEach(async () => {
+ directory = mkdtempSync(join(tmpdir(), 'webhook-failures-'));
+ const options = {
+ type: 'better-sqlite3' as const,
+ database: join(directory, 'data.sqlite'),
+ entities: [WebhookDeliveryFailure],
+ };
+ ds = await new DataSource({ ...options, synchronize: true }).initialize();
+ second = await new DataSource(options).initialize();
+ repo = ds.getRepository(WebhookDeliveryFailure);
+ logger.error.mockClear();
+ });
+ afterEach(async () => {
+ jest.restoreAllMocks();
+ await second.destroy();
+ await ds.destroy();
+ rmSync(directory, { recursive: true, force: true });
+ });
+
+ it('parses HTTP status while leaving network errors unclassified', () => {
+ expect(statusCodeFromError('HTTP 503: down')).toBe(503);
expect(statusCodeFromError('fetch failed')).toBeNull();
});
-});
-
-describe('recordWebhookDeliveryFailure', () => {
- const input = {
- webhookId: 'wh-1',
- sessionId: 's1',
- event: 'message.received',
- url: 'https://r.example/h',
- idempotencyKey: 'k',
- deliveryId: 'd',
- attempts: 3,
- lastStatusCode: 503,
- lastError: 'HTTP 503: x',
- };
-
- const repoWith = (insert: jest.Mock, existing = 0): Repository<WebhookDeliveryFailure> =>
- ({
- insert,
- count: jest.fn().mockResolvedValue(existing),
- delete: jest.fn().mockResolvedValue({}),
- update: jest.fn().mockResolvedValue({}),
- }) as unknown as Repository<WebhookDeliveryFailure>;
-
- it('inserts the failure record, defaulting a missing lastStatusCode to null', async () => {
- const insert = jest.fn().mockResolvedValue({});
- const logger = { error: jest.fn() };
-
+ it('keeps a terminal payload and updates attempts and the latest error on repeat', async () => {
+ const payload = { body: 'private' };
+ await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, payload })).resolves.toBe(true);
await expect(
- recordWebhookDeliveryFailure(repoWith(insert), logger, { ...input, lastStatusCode: undefined }),
- ).resolves.toBe(true);
-
- expect(insert).toHaveBeenCalledWith(expect.objectContaining({ webhookId: 'wh-1', lastStatusCode: null }));
- expect(logger.error).not.toHaveBeenCalled();
- });
-
- it('skips a delivery it has already recorded, so one lost event is one row', async () => {
- // The reconciler replays a stranded delivery once per sweep until its budget is spent, and each
- // failed replay lands here with the SAME idempotency key.
- const insert = jest.fn().mockResolvedValue({});
- const logger = { error: jest.fn() };
-
- await expect(recordWebhookDeliveryFailure(repoWith(insert, 1), logger, input)).resolves.toBe(false);
-
- expect(insert).not.toHaveBeenCalled();
- expect(logger.error).not.toHaveBeenCalled();
- });
-
- it('gives the attempts-0 row the reason of an unsent delivery recorded again, without a second row', async () => {
- // A shed delivery whose replay then fails before sending: the operator has to act on the new
- // reason, not on the shed. Only attempts-0 rows take it, and the metric does not count it again.
- const insert = jest.fn().mockResolvedValue({});
- const update = jest.fn().mockResolvedValue({ affected: 1 });
- const repo = {
- insert,
- update,
- count: jest.fn().mockResolvedValue(1),
- } as unknown as Repository<WebhookDeliveryFailure>;
- const logger = { error: jest.fn() };
-
- await expect(
- recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0, lastStatusCode: null, lastError: 'too big' }),
+ recordWebhookDeliveryFailure(repo, logger, {
+ ...input,
+ attempts: 1,
+ lastStatusCode: 504,
+ lastError: 'HTTP 504: timeout',
+ }),
).resolves.toBe(false);
-
- expect(insert).not.toHaveBeenCalled();
- expect(update).toHaveBeenCalledWith(
- { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 },
- { lastError: 'too big', url: 'https://r.example/h', deliveryId: 'd' },
- );
- });
-
- it('leaves the recorded row alone for a terminal failure recorded again', async () => {
- const update = jest.fn();
- const repo = {
- insert: jest.fn(),
- update,
- delete: jest.fn().mockResolvedValue({}),
- count: jest.fn().mockResolvedValue(1),
- } as unknown as Repository<WebhookDeliveryFailure>;
-
- await expect(recordWebhookDeliveryFailure(repo, { error: jest.fn() }, input)).resolves.toBe(false);
-
- expect(update).not.toHaveBeenCalled();
- });
-
- it('removes an attempts-0 row an earlier terminal record failed to clear', async () => {
- // The first terminal record inserted its row, then its attempts-0 delete failed (or the process
- // stopped in between). The next replay finds the terminal row, and must finish that clear, or
- // the lost event stays listed twice.
- const insert = jest.fn();
- const del = jest.fn().mockResolvedValue({});
- const repo = {
- insert,
- update: jest.fn(),
- delete: del,
- count: jest.fn().mockResolvedValue(1),
- } as unknown as Repository<WebhookDeliveryFailure>;
- const logger = { error: jest.fn() };
-
- await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(false);
-
- expect(insert).not.toHaveBeenCalled();
- expect(del).toHaveBeenCalledWith({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
- expect(logger.error).not.toHaveBeenCalled();
- });
-
- it('removes nothing when an unsent delivery is recorded again', async () => {
- const del = jest.fn();
- const repo = {
- insert: jest.fn(),
- update: jest.fn().mockResolvedValue({}),
- delete: del,
- count: jest.fn().mockResolvedValue(1),
- } as unknown as Repository<WebhookDeliveryFailure>;
-
- await expect(recordWebhookDeliveryFailure(repo, { error: jest.fn() }, { ...input, attempts: 0 })).resolves.toBe(
- false,
- );
-
- expect(del).not.toHaveBeenCalled();
- });
-
- it('still reports an unsent duplicate as not recorded when refreshing its reason fails', async () => {
- // A true here would bump the failure metric for a loss that is already counted.
- const repo = {
- insert: jest.fn(),
- update: jest.fn().mockRejectedValue(new Error('db down')),
- count: jest.fn().mockResolvedValue(1),
- } as unknown as Repository<WebhookDeliveryFailure>;
- const logger = { error: jest.fn() };
-
- await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 })).resolves.toBe(false);
-
- expect(logger.error).toHaveBeenCalledTimes(1);
- });
-
- it('still records a failure that carries no idempotency key', async () => {
- // Control: without a key there is no identity to dedupe on, so the guard must not swallow the
- // row. Otherwise "skips a duplicate" would be satisfied by a helper that stopped inserting.
- const insert = jest.fn().mockResolvedValue({});
- const logger = { error: jest.fn() };
-
- await expect(
- recordWebhookDeliveryFailure(repoWith(insert, 1), logger, { ...input, idempotencyKey: undefined }),
- ).resolves.toBe(true);
-
- expect(insert).toHaveBeenCalled();
- });
-
- it('dedupes a terminal row only against terminal rows, then removes the attempts-0 row it replaces', async () => {
- // A shed or shutdown-refused dispatch left an attempts-0 row. Counted as a duplicate, it would
- // keep the capacity reason as the only record and drop the receiver's actual answer.
- const calls: string[] = [];
- const insert = jest.fn().mockImplementation(() => (calls.push('insert'), Promise.resolve({})));
- const count = jest.fn().mockResolvedValue(0);
- const del = jest.fn().mockImplementation(() => (calls.push('delete'), Promise.resolve({})));
- const repo = { insert, count, delete: del } as unknown as Repository<WebhookDeliveryFailure>;
- const logger = { error: jest.fn() };
-
- await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(true);
-
- expect(count).toHaveBeenCalledWith({
- where: { webhookId: 'wh-1', idempotencyKey: 'k', attempts: MoreThan(0) },
+ const rows = await repo.find({
+ select: { id: true, attempts: true, lastStatusCode: true, lastError: true, payload: true },
});
- expect(del).toHaveBeenCalledWith({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
- // Written first, so a crash in between leaves two rows for the event rather than none.
- expect(calls).toEqual(['insert', 'delete']);
- });
-
- it('dedupes an attempts-0 row against every row of the delivery and removes nothing', async () => {
- const insert = jest.fn().mockResolvedValue({});
- const count = jest.fn().mockResolvedValue(0);
- const del = jest.fn().mockResolvedValue({});
- const repo = { insert, count, delete: del } as unknown as Repository<WebhookDeliveryFailure>;
- const logger = { error: jest.fn() };
-
- await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 })).resolves.toBe(true);
-
- expect(count).toHaveBeenCalledWith({ where: { webhookId: 'wh-1', idempotencyKey: 'k' } });
- expect(del).not.toHaveBeenCalled();
- });
-
- it('keeps the terminal row and the result when removing the attempts-0 row fails', async () => {
- const insert = jest.fn().mockResolvedValue({});
- const repo = {
- insert,
- count: jest.fn().mockResolvedValue(0),
- delete: jest.fn().mockRejectedValue(new Error('db down')),
- } as unknown as Repository<WebhookDeliveryFailure>;
- const logger = { error: jest.fn() };
-
- await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(true);
-
- expect(insert).toHaveBeenCalledTimes(1);
- expect(logger.error).toHaveBeenCalledWith(
- expect.any(String),
- 'db down',
- expect.objectContaining({ action: 'webhook_failure_clear_error' }),
- );
- });
-
- it('swallows a repository error so a logging hiccup cannot re-poison the delivery', async () => {
- const insert = jest.fn().mockRejectedValue(new Error('db down'));
- const logger = { error: jest.fn() };
-
- // Reported as recorded even though the row was lost: the delivery really did fail, and the
- // caller's failure metric must count it rather than hide it behind a database problem.
- await expect(recordWebhookDeliveryFailure(repoWith(insert), logger, input)).resolves.toBe(true);
- expect(logger.error).toHaveBeenCalled();
- });
-
- it('keeps the replay payload on a terminal row', async () => {
- const insert = jest.fn().mockResolvedValue({});
- const payload = { id: 'msg-1', body: 'hi' };
-
- await recordWebhookDeliveryFailure(repoWith(insert), { error: jest.fn() }, { ...input, payload });
-
- expect(insert).toHaveBeenCalledWith(expect.objectContaining({ attempts: 3, payload }));
- });
-
- it('never stores a payload on an attempts-0 row: its pending outbox row already holds the data', async () => {
- const insert = jest.fn().mockResolvedValue({});
-
- await recordWebhookDeliveryFailure(
- repoWith(insert),
- { error: jest.fn() },
- { ...input, attempts: 0, payload: { id: 'msg-1' } },
- );
-
- expect(insert).toHaveBeenCalledTimes(1);
- expect((insert.mock.calls as unknown[][])[0][0]).not.toHaveProperty('payload');
+ expect(rows).toEqual([
+ expect.objectContaining({ attempts: 4, lastStatusCode: 504, lastError: 'HTTP 504: timeout', payload }),
+ ]);
});
-
- it('writes no payload column at all when none is given (payload retention off)', async () => {
- const insert = jest.fn().mockResolvedValue({});
-
- await recordWebhookDeliveryFailure(repoWith(insert), { error: jest.fn() }, input);
-
- expect((insert.mock.calls as unknown[][])[0][0]).not.toHaveProperty('payload');
+ it('does not resurrect an expired payload when refreshing a terminal failure', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, input);
+ await repo.update({ idempotencyKey: 'key' }, { payload: null, createdAt: new Date(Date.now() - 25 * 3600000) });
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, payload: { body: 'private' } });
+ const row = await repo.findOneOrFail({ where: { idempotencyKey: 'key' }, select: { id: true, payload: true } });
+ expect(row.payload).toBeNull();
});
-});
-
-describe('clearDeliveryFailureRows', () => {
- const repoWithDelete = (del: jest.Mock): Repository<WebhookDeliveryFailure> =>
- ({ delete: del }) as unknown as Repository<WebhookDeliveryFailure>;
-
- it('deletes every row of one delivery, or only its attempts-0 rows', async () => {
- const del = jest.fn().mockResolvedValue({});
- const logger = { error: jest.fn() };
- await clearDeliveryFailureRows(repoWithDelete(del), logger, 'wh-1', 'k');
- await clearDeliveryFailureRows(repoWithDelete(del), logger, 'wh-1', 'k', true);
-
- expect(del.mock.calls).toEqual([
- [{ webhookId: 'wh-1', idempotencyKey: 'k' }],
- [{ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 }],
+ it('converges simultaneous recorders from two connections on one terminal identity', async () => {
+ const results = await Promise.all([
+ recordWebhookDeliveryFailure(repo, logger, input),
+ recordWebhookDeliveryFailure(second.getRepository(WebhookDeliveryFailure), logger, { ...input, attempts: 1 }),
]);
+ expect(results.sort()).toEqual([false, true]);
+ expect(await repo.count()).toBe(1);
+ expect((await repo.findOneByOrFail({ idempotencyKey: 'key' })).attempts).toBe(4);
+ });
+ it('replaces an unattempted row only after the terminal row is stored', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0, payload: { body: 'unused' } });
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, payload: { body: 'retained' } });
+ expect(await repo.count()).toBe(1);
+ expect((await repo.findOneByOrFail({ idempotencyKey: 'key' })).attempts).toBe(3);
+ });
+ it('keeps an unattempted row when terminal persistence fails', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 });
+ jest.spyOn(repo, 'insert').mockRejectedValueOnce(new Error('disk unavailable'));
+ await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBeNull();
+ expect((await repo.findOneByOrFail({ idempotencyKey: 'key' })).attempts).toBe(0);
+ expect(logger.error).toHaveBeenCalled();
});
-
- it('deletes nothing without an idempotency key', async () => {
- // Matching on webhookId alone would erase the rows of every other lost event of that webhook.
- const del = jest.fn().mockResolvedValue({});
-
- await clearDeliveryFailureRows(repoWithDelete(del), { error: jest.fn() }, 'wh-1', undefined);
-
- expect(del).not.toHaveBeenCalled();
- });
-
- it('logs a failed delete instead of throwing', async () => {
- const logger = { error: jest.fn() };
-
- await expect(
- clearDeliveryFailureRows(repoWithDelete(jest.fn().mockRejectedValue(new Error('db down'))), logger, 'wh-1', 'k'),
- ).resolves.toBeUndefined();
-
- expect(logger.error).toHaveBeenCalledTimes(1);
+ it('returns a persistence failure when refreshing a terminal row fails', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, input);
+ jest.spyOn(repo, 'update').mockRejectedValueOnce(new Error('disk unavailable'));
+ await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBeNull();
+ });
+ it('records unkeyed losses separately and stores no replay payload by default', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, idempotencyKey: undefined });
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, idempotencyKey: undefined });
+ expect(await repo.count()).toBe(2);
+ const rows = await repo.find({ select: { id: true, payload: true } });
+ expect(rows.every(r => r.payload === null)).toBe(true);
+ });
+ it('never keeps a payload on an unattempted row and refreshes its refusal reason', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0, payload: { body: 'unused' } });
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0, lastError: 'too large' });
+ const row = await repo.findOneOrFail({
+ where: { idempotencyKey: 'key' },
+ select: { id: true, lastError: true, payload: true },
+ });
+ expect(row).toMatchObject({ lastError: 'too large', payload: null });
+ });
+ it('clears only the matching identity and never deletes every row for an unkeyed loss', async () => {
+ await recordWebhookDeliveryFailure(repo, logger, input);
+ await recordWebhookDeliveryFailure(repo, logger, { ...input, idempotencyKey: 'other' });
+ await clearDeliveryFailureRows(repo, logger, 'w', undefined);
+ expect(await repo.count()).toBe(2);
+ await clearDeliveryFailureRows(repo, logger, 'w', 'key');
+ expect(await repo.count()).toBe(1);
+ });
+ it('logs a failed clear without changing the delivery outcome', async () => {
+ jest.spyOn(repo, 'delete').mockRejectedValueOnce(new Error('disk unavailable'));
+ await expect(clearDeliveryFailureRows(repo, logger, 'w', 'key')).resolves.toBeUndefined();
+ expect(logger.error).toHaveBeenCalled();
});
});
diff --git a/src/modules/webhook/utils/record-delivery-failure.ts b/src/modules/webhook/utils/record-delivery-failure.ts
index 0591feb08..9bcf79e68 100644
--- a/src/modules/webhook/utils/record-delivery-failure.ts
+++ b/src/modules/webhook/utils/record-delivery-failure.ts
@@ -1,4 +1,5 @@
import { MoreThan, Repository } from 'typeorm';
+import { isUniqueViolation } from '../../../common/utils/db-errors';
import type { QueryDeepPartialEntity } from 'typeorm/query-builder/QueryPartialEntity';
import { WebhookDeliveryFailure } from '../entities/webhook-delivery-failure.entity';
@@ -64,13 +65,30 @@ export async function clearDeliveryFailureRows(
* path's last attempt) or was not sent (attempts 0, from recordUndelivered: shed, refused at shutdown,
* oversize or a preflight failure). Wrapped in its own try/catch: persisting the failure is best-effort
* bookkeeping and must never throw back into (and re-poison) the delivery result or the fire-and-forget
- * dispatch loop.
+ * dispatch loop. Returns true for a new row, false for an existing row, and null
+ * when persistence failed.
*/
export async function recordWebhookDeliveryFailure(
repo: Repository<WebhookDeliveryFailure>,
logger: ErrorLogger,
input: WebhookDeliveryFailureInput,
-): Promise<boolean> {
+): Promise<boolean | null> {
+ const terminal = input.attempts > 0;
+ const refreshTerminal = async (): Promise<boolean | null> => {
+ const updated = await repo.update(
+ { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey, attempts: MoreThan(0) },
+ {
+ url: input.url,
+ lastError: input.lastError,
+ deliveryId: input.deliveryId,
+ attempts: () => `"attempts" + ${input.attempts}`,
+ lastStatusCode: input.lastStatusCode ?? null,
+ },
+ );
+ if (!updated.affected) return null;
+ await clearDeliveryFailureRows(repo, logger, input.webhookId, input.idempotencyKey, true);
+ return false;
+ };
try {
// One row per lost delivery, not one per attempt. The reconciler replays a pending outbox row
// up to its budget and every failed replay arrives here, so without this guard a single lost
@@ -78,7 +96,6 @@ export async function recordWebhookDeliveryFailure(
// metric. An absent idempotencyKey carries no identity to dedupe on, so it is always appended.
// A terminal row (attempts > 0) is deduplicated only against other terminal rows: an attempts-0
// row (a shed or shutdown-refused dispatch) is replaced by it below instead of suppressing it.
- const terminal = input.attempts > 0;
if (input.idempotencyKey) {
const existing = await repo.count({
where: terminal
@@ -114,7 +131,7 @@ export async function recordWebhookDeliveryFailure(
} else {
// A crash or a failed delete right after an earlier terminal insert can leave the
// attempts-0 row behind; nothing else reconciles it, so every repeat finishes the job.
- await clearDeliveryFailureRows(repo, logger, input.webhookId, input.idempotencyKey, true);
+ return await refreshTerminal();
}
return false;
}
@@ -122,11 +139,16 @@ export async function recordWebhookDeliveryFailure(
const { payload, ...row } = input;
// The cast covers `payload` only: TypeORM's insert typing cannot see through a free-form JSON
// record, while simple-json serializes it like the outbox payload.
- await repo.insert({
- ...row,
- lastStatusCode: input.lastStatusCode ?? null,
- ...(terminal && payload ? { payload } : {}),
- } as QueryDeepPartialEntity<WebhookDeliveryFailure>);
+ try {
+ await repo.insert({
+ ...row,
+ lastStatusCode: input.lastStatusCode ?? null,
+ ...(terminal && payload ? { payload } : {}),
+ } as QueryDeepPartialEntity<WebhookDeliveryFailure>);
+ } catch (error) {
+ if (terminal && input.idempotencyKey && isUniqueViolation(error)) return await refreshTerminal();
+ throw error;
+ }
if (terminal) {
// Only after the insert, so a crash in between leaves both rows, never none. A later terminal
// record of the same delivery repeats this clear.
@@ -139,8 +161,7 @@ export async function recordWebhookDeliveryFailure(
err instanceof Error ? err.message : String(err),
{ webhookId: input.webhookId, deliveryId: input.deliveryId, action: 'webhook_failure_record_error' },
);
- // The delivery really did fail; only the bookkeeping did. Report it as recorded so the metric
- // still counts the loss rather than hiding it behind a database problem.
- return true;
+ // No durable failure row owns the event. Callers must retain their replay copy.
+ return null;
}
}
diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.ts
index 631f01a06..9d8496869 100644
--- a/src/modules/webhook/webhook-delivery.service.spec.ts
+++ b/src/modules/webhook/webhook-delivery.service.spec.ts
@@ -75,7 +75,7 @@ describe('WebhookDeliveryService', () => {
let hookManager: jest.Mocked<Partial<HookManager>>;
let webhookQueue: jest.Mocked<Record<string, jest.Mock>>;
let lidStore: { getCached: jest.Mock; resolveLid: jest.Mock };
- let outboxService: { open: jest.Mock; close: jest.Mock };
+ let outboxService: { open: jest.Mock; close: jest.Mock; markQueued: jest.Mock };
let insertedFailures: Array<Partial<WebhookDeliveryFailure>>;
beforeEach(async () => {
@@ -125,9 +125,20 @@ describe('WebhookDeliveryService', () => {
.mockImplementation((where: Partial<WebhookDeliveryFailure>, set: Partial<WebhookDeliveryFailure>) => {
let affected = 0;
insertedFailures = insertedFailures.map(r => {
- if (Object.entries(where).some(([k, v]) => r[k as keyof WebhookDeliveryFailure] !== v)) return r;
+ if (
+ Object.entries(where).some(([k, v]) =>
+ k === 'attempts' ? !attemptsMatch(r.attempts, v) : r[k as keyof WebhookDeliveryFailure] !== v,
+ )
+ )
+ return r;
affected++;
- return { ...r, ...set };
+ return {
+ ...r,
+ ...set,
+ ...(typeof set.attempts === 'function'
+ ? { attempts: (r.attempts ?? 0) + Number((set.attempts as () => string)().split('+')[1]) }
+ : {}),
+ };
});
return Promise.resolve({ affected });
}),
@@ -162,7 +173,11 @@ describe('WebhookDeliveryService', () => {
resolveLid: jest.fn((jid: string) => (lidStore.getCached(userPart(jid)) as string | null | undefined) ?? null),
};
- outboxService = { open: jest.fn().mockResolvedValue(undefined), close: jest.fn().mockResolvedValue(undefined) };
+ outboxService = {
+ open: jest.fn().mockResolvedValue(undefined),
+ close: jest.fn().mockResolvedValue(undefined),
+ markQueued: jest.fn().mockResolvedValue(undefined),
+ };
const module: TestingModule = await Test.createTestingModule({
providers: [
@@ -762,6 +777,34 @@ describe('WebhookDeliveryService', () => {
// dead-letters in place, so nothing reaches the caller as an exception. When this reported
// nothing, a replay that never delivered was retired 'dispatched' on the first sweep and the
// documented WEBHOOK_RECONCILE_MAX_ATTEMPTS budget was unreachable.
+ it('keeps the outbox payload when recording an oversize preflight refusal fails', async () => {
+ const webhook = createMockWebhook();
+ (repository.find as jest.Mock).mockResolvedValue([webhook]);
+ (configService.get as jest.Mock).mockImplementation((key: string, def?: unknown) =>
+ key === 'webhook.maxPayloadBytes' ? 1 : def,
+ );
+ (failureRepository.insert as jest.Mock).mockRejectedValue(new Error('disk unavailable'));
+ (undiciFetch as unknown as jest.Mock).mockClear();
+ await service.dispatch('sess-1', 'message.received', { body: 'retained' });
+ expect(outboxService.close).not.toHaveBeenCalled();
+ expect(undiciFetch).not.toHaveBeenCalled();
+ });
+
+ it('keeps the outbox pending when a terminal failure cannot be persisted', async () => {
+ const webhook = createMockWebhook({ retryCount: 1 });
+ (repository.find as jest.Mock).mockResolvedValue([webhook]);
+ (undiciFetch as unknown as jest.Mock).mockRejectedValue(new Error('receiver down'));
+ (failureRepository.insert as jest.Mock).mockRejectedValue(new Error('disk unavailable'));
+ await service.dispatch('sess-1', 'message.received', { body: 'retained by outbox' });
+ expect(outboxService.open).toHaveBeenCalledWith(
+ expect.objectContaining({ payload: { body: 'retained by outbox' } }),
+ );
+ expect(outboxService.close).not.toHaveBeenCalled();
+ await expect(
+ service.redeliver(webhook, 'sess-1', 'message.received', 'stored', {}, { singleAttempt: true }),
+ ).resolves.toBe('unrecorded');
+ });
+
it('redeliver reports failed when the receiver never accepts, and delivered when it does', async () => {
const webhook = createMockWebhook({ events: ['message.received'], retryCount: 1 });
(hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
@@ -1604,6 +1647,22 @@ describe('WebhookDeliveryService', () => {
expect(JSON.stringify(jobData).length).toBeLessThan(2048);
});
+ it('uses one direct POST for operator redrive even with the queue enabled', async () => {
+ const queueService = await buildQueueService((key: string, def?: unknown) =>
+ key === 'queue.enabled' ? true : def,
+ );
+ const webhook = createMockWebhook({ retryCount: 4 });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ (undiciFetch as unknown as jest.Mock).mockRejectedValue(new Error('receiver down'));
+ for (let i = 0; i < 2; i++) {
+ await expect(
+ queueService.redeliver(webhook, 'sess-1', 'message.received', 'stored-key', {}, { singleAttempt: true }),
+ ).resolves.toBe('failed');
+ }
+ expect(webhookQueue.add).not.toHaveBeenCalled();
+ expect(undiciFetch as unknown as jest.Mock).toHaveBeenCalledTimes(2);
+ });
+
it('keys the queue job by the delivery id, so a retried enqueue cannot double-deliver', async () => {
const queueService = await buildQueueService((key: string, def?: unknown) => {
if (key === 'queue.enabled') return true;
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 94ad5af73..c61ddb80f 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -1,12 +1,12 @@
import { Injectable, Optional, OnModuleInit, OnModuleDestroy } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
-import { Repository } from 'typeorm';
+import { MoreThan, Repository } from 'typeorm';
import { ConfigService } from '@nestjs/config';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { setTimeout } from 'node:timers/promises';
import { Webhook } from './entities/webhook.entity';
-import { WebhookOutboxService } from './webhook-outbox.service';
+import { WebhookOutboxService, type ReplayableDelivery } from './webhook-outbox.service';
import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';
import { clearDeliveryFailureRows, recordWebhookDeliveryFailure } from './utils/record-delivery-failure';
import {
@@ -79,7 +79,7 @@ const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5000;
* dispatch (nothing left the process), or a direct retry found the webhook removed, disabled or
* unsubscribed (an earlier attempt may already have been POSTed).
*/
-export type WebhookDeliveryOutcome = 'delivered' | 'enqueued' | 'cancelled' | 'failed';
+export type WebhookDeliveryOutcome = 'delivered' | 'enqueued' | 'cancelled' | 'failed' | 'unrecorded';
/** Per-event-occurrence context threaded through the dispatch pipeline stages (was closure state). */
interface DispatchEventContext {
@@ -103,6 +103,8 @@ interface DirectDeliveryOptions {
singleAttempt?: boolean;
}
+class UnrecordedWebhookFailure extends Error {}
+
/** The limiter refused or dropped the task because shutdown closed it. */
const isLimiterClosed = (error: unknown): boolean =>
error instanceof Error && error.message === 'ConcurrencyLimiter closed';
@@ -144,7 +146,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
{ webhookId: string; sessionId: string; event: string; idempotencyKey: string; url: string }
>();
/** Late bookkeeping (dead-letter rows) written by tasks the limiter already released — awaited on shutdown. */
- private readonly pendingBookkeeping = new Set<Promise<void>>();
+ private readonly pendingBookkeeping = new Set<Promise<unknown>>();
/**
* Outbox rows this node still owns, by idempotency key and counted (the same key can be dispatched
* twice), from the moment the row is opened until its dispatch settles: parked in the limiter,
@@ -348,7 +350,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
error: unknown,
action: string,
ctx: DispatchEventContext,
- ): Promise<void> {
+ ): Promise<boolean> {
const { sessionId, event } = ctx;
const lastError = redactSsrfError(error, this.logger, 'webhook dispatch');
const recorded = await recordWebhookDeliveryFailure(this.failureRepository, this.logger, {
@@ -362,9 +364,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
lastStatusCode: null,
lastError,
});
- if (recorded) {
- incrementWebhookDeliveryFailures();
- }
+ if (recorded !== false) incrementWebhookDeliveryFailures();
try {
await this.hookManager.execute(
'webhook:error',
@@ -383,6 +383,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
deliveryId,
action,
});
+ return recorded !== null;
}
/**
@@ -395,7 +396,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
deliveryId: string,
idempotencyKey: string,
ctx: DispatchEventContext,
- ): Promise<{ finalPayload: WebhookPayload; body: string } | 'cancelled' | null> {
+ ): Promise<{ finalPayload: WebhookPayload; body: string } | 'cancelled' | 'unrecorded' | null> {
const { sessionId, event, baseData } = ctx;
try {
const payload: WebhookPayload = {
@@ -479,7 +480,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
}
}
if (payloadBytes > maxPayloadBytes) {
- await this.recordUndelivered(
+ const recorded = await this.recordUndelivered(
webhook,
deliveryId,
idempotencyKey,
@@ -489,12 +490,12 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
'webhook_payload_oversize',
ctx,
);
- return null;
+ return recorded ? null : 'unrecorded';
}
return { finalPayload, body };
} catch (error) {
- await this.recordUndelivered(
+ const recorded = await this.recordUndelivered(
webhook,
deliveryId,
idempotencyKey,
@@ -502,7 +503,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
'webhook_dispatch_preflight_failed',
ctx,
);
- return null;
+ return recorded ? null : 'unrecorded';
}
}
@@ -524,6 +525,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
ctx: DispatchEventContext,
): Promise<WebhookDeliveryOutcome> {
const preflight = await this.preflightDelivery(webhook, deliveryId, idempotencyKey, ctx);
+ if (preflight === 'unrecorded') return 'unrecorded';
if (preflight === 'cancelled') {
// A plugin suppressed this dispatch deliberately. There is no failure to record and nothing
// to retry: reporting it as failed made the reconciler replay a deliberately dropped event
@@ -536,7 +538,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
}
const { finalPayload, body } = preflight;
// Use queue if available, otherwise fallback to direct delivery
- if (this.queueEnabled && this.webhookQueue) {
+ if (!ctx.singleAttempt && this.queueEnabled && this.webhookQueue) {
// A replay's attempts-0 row is not cleared here. It stays until the delivery resolves: a
// successful POST (the processor's, or the fallback's when the add fails) clears it, and a
// terminal failure replaces it right after filing its own row. A restart or a lost job at
@@ -583,6 +585,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
},
});
+ await this.outbox.markQueued(webhook.id, idempotencyKey, deliveryId);
+
// Execute hook after successful queue (NOT delivery - that happens in processor)
await this.hookManager.execute(
'webhook:queued',
@@ -648,7 +652,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
webhookId: webhook.id,
action: 'webhook_queue_fallback_failed',
});
- return 'failed';
+ return fallbackError instanceof UnrecordedWebhookFailure ? 'unrecorded' : 'failed';
}
// The queue never took it, but the fallback POST did.
return 'delivered';
@@ -698,7 +702,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
webhookId: webhook.id,
action: 'webhook_delivery_failed',
});
- return 'failed';
+ return error instanceof UnrecordedWebhookFailure ? 'unrecorded' : 'failed';
}
return 'delivered';
}
@@ -851,10 +855,10 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
url: target.url,
});
try {
- await this.deliverOne(target, deliveryId, idempotencyKey, { ...ctx, yieldSlot });
- // Reached a durable owner: handed to the queue, or completed inline. A failure inside
- // either owner dead-letters through the failure row, so this is never replayed.
- await this.outbox.close(webhook.id, idempotencyKey, 'dispatched');
+ const outcome = await this.deliverOne(target, deliveryId, idempotencyKey, { ...ctx, yieldSlot });
+ // Retire only after queue ownership, success, cancellation or a durable terminal row.
+ if (outcome !== 'unrecorded' && outcome !== 'enqueued')
+ await this.outbox.close(webhook.id, idempotencyKey, 'dispatched');
} finally {
this.inFlightDeliveries.delete(deliveryId);
}
@@ -925,6 +929,39 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
});
}
+ /** A live or unreadable job still owns its queued delivery; never replay alongside it. */
+ async isQueueJobPending(deliveryId: string): Promise<boolean> {
+ if (!this.webhookQueue) return false;
+ try {
+ const job = await this.webhookQueue.getJob(deliveryId);
+ if (!job) return false;
+ return !['completed', 'failed', 'unknown'].includes(await job.getState());
+ } catch (error) {
+ this.logger.warn('Could not inspect queued webhook delivery', { deliveryId, error: String(error) });
+ return true;
+ }
+ }
+
+ /** Finish a spent outbox replay budget only after its failure record is durable. */
+ async recordReplayExhaustion(row: ReplayableDelivery, url: string): Promise<boolean> {
+ const existing = await this.failureRepository.count({
+ where: { webhookId: row.webhookId, idempotencyKey: row.idempotencyKey, attempts: MoreThan(0) },
+ });
+ if (existing > 0) return true;
+ const recorded = await recordWebhookDeliveryFailure(this.failureRepository, this.logger, {
+ webhookId: row.webhookId,
+ sessionId: row.sessionId,
+ event: row.event,
+ url,
+ idempotencyKey: row.idempotencyKey,
+ attempts: Math.max(1, row.attempts),
+ lastError: 'Webhook replay budget exhausted',
+ ...(this.keepsFailurePayload() ? { payload: row.payload } : {}),
+ });
+ if (recorded === true) incrementWebhookDeliveryFailures();
+ return recorded !== null;
+ }
+
/**
* WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS > 0: a terminal failure row keeps the event data it was
* built from, so an operator can redrive it. Off (0) by default, which stores nothing new.
@@ -1047,9 +1084,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
error,
...(options.replayData ? { payload: options.replayData } : {}),
});
- if (recorded) {
- incrementWebhookDeliveryFailures();
- }
+ if (recorded !== false) incrementWebhookDeliveryFailures();
+ if (recorded === null) throw new UnrecordedWebhookFailure(redactSsrfError(error));
throw error;
}
}
diff --git a/src/modules/webhook/webhook-outbox.service.spec.ts b/src/modules/webhook/webhook-outbox.service.spec.ts
index f8a2158d1..dcad5b1ef 100644
--- a/src/modules/webhook/webhook-outbox.service.spec.ts
+++ b/src/modules/webhook/webhook-outbox.service.spec.ts
@@ -1,3 +1,4 @@
+import { In } from 'typeorm';
import { WebhookOutboxService } from './webhook-outbox.service';
import { WebhookOutboxEvent } from './entities/webhook-outbox-event.entity';
@@ -58,6 +59,15 @@ describe('WebhookOutboxService', () => {
);
});
+ it('keeps payload when marking queued and excludes already settled rows', async () => {
+ await service.markQueued('wh-1', 'key_wh-1', 'job-1');
+ expect(repo.update).toHaveBeenCalledWith(
+ expect.objectContaining({ state: In(['pending', 'queued']) }),
+ expect.objectContaining({ state: 'queued', deliveryId: 'job-1' }),
+ );
+ expect((repo.update.mock.calls as unknown[][])[0][1]).not.toHaveProperty('payload');
+ });
+
it('returns only rows that still carry a payload: a retired row is not replayable', async () => {
const row = (over: Partial<WebhookOutboxEvent>): WebhookOutboxEvent => ({
id: 'r',
@@ -81,7 +91,7 @@ describe('WebhookOutboxService', () => {
await expect(service.countAttempt('row-1', 2)).resolves.toBe(true);
expect(repo.update).toHaveBeenCalledWith(
- { id: 'row-1', state: 'pending' },
+ { id: 'row-1', state: In(['pending', 'queued']) },
expect.objectContaining({ attempts: 3 }),
);
});
@@ -111,7 +121,7 @@ describe('WebhookOutboxService retention', () => {
];
// Deleting a 'pending' row on age would discard the delivery this table exists to protect.
expect(where.state.type).toBe('not');
- expect(where.state.value).toBe('pending');
+ expect(where.state.value).toEqual(['pending', 'queued']);
expect(where.createdAt.type).toBe('lessThan');
});
diff --git a/src/modules/webhook/webhook-outbox.service.ts b/src/modules/webhook/webhook-outbox.service.ts
index 4a84a4556..a1fb83a9d 100644
--- a/src/modules/webhook/webhook-outbox.service.ts
+++ b/src/modules/webhook/webhook-outbox.service.ts
@@ -1,6 +1,6 @@
import { Injectable, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
-import { LessThan, Not, Repository } from 'typeorm';
+import { In, IsNull, LessThan, Not, Repository } from 'typeorm';
import { WebhookOutboxEvent, WebhookOutboxState } from './entities/webhook-outbox-event.entity';
import { createLogger } from '../../common/services/logger.service';
import { isUniqueViolation } from '../../common/utils/db-errors';
@@ -16,6 +16,8 @@ export interface ReplayableDelivery {
idempotencyKey: string;
payload: Record<string, unknown>;
attempts: number;
+ deliveryId?: string;
+ state?: WebhookOutboxState | null;
}
/**
@@ -77,7 +79,7 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {
async pruneSettled(olderThanDays: number): Promise<number> {
const cutoff = new Date();
cutoff.setDate(cutoff.getDate() - olderThanDays);
- const result = await this.outbox.delete({ state: Not('pending'), createdAt: LessThan(cutoff) });
+ const result = await this.outbox.delete({ state: Not(In(['pending', 'queued'])), createdAt: LessThan(cutoff) });
return result.affected || 0;
}
@@ -110,15 +112,12 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {
}
}
- /**
- * Record the outcome and retire the payload.
- *
- * `dispatched` covers both modes: handed to the queue, or completed inline. A failure INSIDE
- * either owner dead-letters through webhook_delivery_failures, so a dispatched row is never the
- * reconciler's concern, and retiring on ENQUEUE is what stops the reconciler duplicating work
- * BullMQ already holds.
- */
- async close(webhookId: string, idempotencyKey: string, state: Exclude<WebhookOutboxState, 'pending'>): Promise<void> {
+ /** Retire replay data after successful delivery, cancellation or durable failure handoff. */
+ async close(
+ webhookId: string,
+ idempotencyKey: string,
+ state: Exclude<WebhookOutboxState, 'pending' | 'queued'>,
+ ): Promise<void> {
try {
await this.outbox.update({ webhookId, idempotencyKey }, { state, payload: null, lastAttemptAt: new Date() });
} catch (error) {
@@ -126,10 +125,22 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {
}
}
- /** Pending rows older than the staleness window, oldest first. Only these are replayable. */
+ /** Keep the queue's replay copy until the worker has settled the delivery. */
+ async markQueued(webhookId: string, idempotencyKey: string, deliveryId: string): Promise<void> {
+ try {
+ await this.outbox.update(
+ { webhookId, idempotencyKey, state: In(['pending', 'queued']), payload: Not(IsNull()) },
+ { state: 'queued', deliveryId, lastAttemptAt: new Date() },
+ );
+ } catch (error) {
+ this.logger.warn(`Could not record queued outbound delivery: ${String(error)}`);
+ }
+ }
+
+ /** Unsettled rows older than the staleness window; queued jobs are checked before replay. */
async findStale(olderThan: Date, limit: number): Promise<ReplayableDelivery[]> {
const rows = await this.outbox.find({
- where: { state: 'pending', createdAt: LessThan(olderThan) },
+ where: { state: In(['pending', 'queued']), createdAt: LessThan(olderThan) },
order: { createdAt: 'ASC' },
take: limit,
});
@@ -145,6 +156,8 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {
idempotencyKey: r.idempotencyKey,
payload: r.payload,
attempts: r.attempts,
+ deliveryId: r.deliveryId,
+ state: r.state,
}));
}
@@ -158,7 +171,7 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {
async countAttempt(id: string, attempts: number): Promise<boolean> {
try {
const result = await this.outbox.update(
- { id, state: 'pending' },
+ { id, state: In(['pending', 'queued']) },
{ attempts: attempts + 1, lastAttemptAt: new Date() },
);
return (result.affected ?? 0) > 0;
diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.ts
index b99b18547..08bcad39f 100644
--- a/src/modules/webhook/webhook-reconciler.service.spec.ts
+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts
@@ -37,7 +37,12 @@ describe('resolveWebhookReconcilerOptions', () => {
describe('WebhookReconcilerService', () => {
let outbox: { findStale: jest.Mock; close: jest.Mock; countAttempt: jest.Mock };
- let delivery: { redeliver: jest.Mock; isLocallyPending: jest.Mock };
+ let delivery: {
+ redeliver: jest.Mock;
+ isLocallyPending: jest.Mock;
+ recordReplayExhaustion: jest.Mock;
+ isQueueJobPending: jest.Mock;
+ };
let webhooks: { findOne: jest.Mock };
let service: WebhookReconcilerService;
@@ -49,6 +54,8 @@ describe('WebhookReconcilerService', () => {
};
delivery = {
redeliver: jest.fn().mockResolvedValue('delivered'),
+ recordReplayExhaustion: jest.fn().mockResolvedValue(true),
+ isQueueJobPending: jest.fn().mockResolvedValue(false),
isLocallyPending: jest.fn().mockReturnValue(false),
};
webhooks = { findOne: jest.fn().mockResolvedValue({ id: 'wh-1', active: true, events: ['*'] }) };
@@ -92,13 +99,13 @@ describe('WebhookReconcilerService', () => {
expect(stats).toMatchObject({ replayed: 0, failed: 1 });
});
- it('retires the row when the replay was handed to the queue rather than delivered inline', async () => {
+ it('leaves retirement of an enqueued replay to the worker', async () => {
outbox.findStale.mockResolvedValue([row({ attempts: 1 })]);
delivery.redeliver.mockResolvedValue('enqueued');
const stats = await service.sweep(OPTS);
- expect(outbox.close).toHaveBeenCalledWith('wh-1', 'stored-key_wh-1', 'dispatched');
+ expect(outbox.close).not.toHaveBeenCalled();
expect(stats).toMatchObject({ replayed: 1, failed: 0 });
});
@@ -135,6 +142,23 @@ describe('WebhookReconcilerService', () => {
expect(stats).toMatchObject({ failed: 1, replayed: 0 });
});
+ it('keeps its durable payload when recording a spent budget fails, without another POST', async () => {
+ outbox.findStale.mockResolvedValue([row({ attempts: 3 })]);
+ delivery.recordReplayExhaustion.mockResolvedValue(false);
+ await service.sweep(OPTS);
+ expect(delivery.redeliver).not.toHaveBeenCalled();
+ expect(outbox.close).not.toHaveBeenCalled();
+ expect(delivery.recordReplayExhaustion).toHaveBeenCalledTimes(1);
+ });
+
+ it('keeps a queued row while BullMQ still owns an active job', async () => {
+ outbox.findStale.mockResolvedValue([row({ state: 'queued', deliveryId: 'job' })]);
+ delivery.isQueueJobPending.mockResolvedValue(true);
+ await service.sweep(OPTS);
+ expect(delivery.redeliver).not.toHaveBeenCalled();
+ expect(outbox.close).not.toHaveBeenCalled();
+ });
+
it('does not replay to a subscription that is gone or switched off', async () => {
outbox.findStale.mockResolvedValue([row()]);
webhooks.findOne.mockResolvedValue({ id: 'wh-1', active: false });
diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.ts
index 7da047745..1ab49cc2c 100644
--- a/src/modules/webhook/webhook-reconciler.service.ts
+++ b/src/modules/webhook/webhook-reconciler.service.ts
@@ -102,12 +102,6 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
stats.skipped++;
continue;
}
- if (row.attempts >= opts.maxAttempts) {
- // Budget spent: stop replaying and leave the failure row as the recovery path.
- await this.outbox.close(row.webhookId, row.idempotencyKey, 'failed');
- stats.failed++;
- continue;
- }
const webhook = await this.webhooks.findOne({ where: { id: row.webhookId } });
if (!isDeliverableWebhook(webhook, row.event)) {
// The subscription is gone, switched off or no longer lists this event; replaying it would
@@ -117,6 +111,19 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
stats.skipped++;
continue;
}
+ if (row.deliveryId && (await this.delivery.isQueueJobPending(row.deliveryId))) {
+ stats.skipped++;
+ continue;
+ }
+ if (row.attempts >= opts.maxAttempts) {
+ // A database fault may have prevented every terminal failure write. Keep the outbox
+ // payload until this handoff succeeds, without sending beyond the replay budget.
+ if (await this.delivery.recordReplayExhaustion(row, webhook.url)) {
+ await this.outbox.close(row.webhookId, row.idempotencyKey, 'failed');
+ }
+ stats.failed++;
+ continue;
+ }
if (!(await this.outbox.countAttempt(row.id, row.attempts))) {
// Settled since the batch was read, typically a local dispatch that finished while an
// earlier row in this pass was replaying. The copy in hand is stale; replaying it duplicates.
@@ -135,16 +142,15 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
row.idempotencyKey,
row.payload,
);
- if (outcome === 'failed') {
+ if (outcome === 'failed' || outcome === 'unrecorded') {
// Left 'pending' on purpose: the next sweep retries it until the budget is spent.
this.logger.warn(`Replay of ${row.event} to webhook ${row.webhookId} did not deliver`);
stats.failed++;
continue;
}
- // 'delivered', 'enqueued' and 'cancelled' all retire the row: the delivery reached a durable
- // owner, a plugin dropped it on purpose, or a retry found the webhook removed, disabled or
- // unsubscribed. Only 'failed' is worth another sweep.
- await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');
+ // Enqueued rows retain their copy until the worker settles them. Success and deliberate
+ // cancellation can retire immediately.
+ if (outcome !== 'enqueued') await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');
stats.replayed++;
} catch (error) {
// An exception is an unexpected fault rather than a delivery failure; the row stays
diff --git a/src/modules/webhook/webhook-redrive.service.spec.ts b/src/modules/webhook/webhook-redrive.service.spec.ts
index ae7181f10..adef532b6 100644
--- a/src/modules/webhook/webhook-redrive.service.spec.ts
+++ b/src/modules/webhook/webhook-redrive.service.spec.ts
@@ -1,196 +1,336 @@
-import { FindOperator, Repository } from 'typeorm';
+import { ConfigService } from '@nestjs/config';
+import { randomUUID } from 'node:crypto';
+import { DataSource, Repository } from 'typeorm';
+import { Session, SessionStatus } from '../session/entities/session.entity';
import { Webhook } from './entities/webhook.entity';
import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';
import { WebhookDeliveryService } from './webhook-delivery.service';
import { MAX_WEBHOOK_REDRIVE_LIMIT, WebhookRedriveService } from './webhook-redrive.service';
-
-const failure = (overrides: Partial<WebhookDeliveryFailure> = {}): WebhookDeliveryFailure => ({
- id: 'f-1',
- webhookId: 'wh-1',
- sessionId: 'sess-1',
- event: 'message.received',
- url: 'https://r.example/h',
- idempotencyKey: 'key-1',
- deliveryId: 'd-1',
- attempts: 3,
- lastStatusCode: 503,
- lastError: 'HTTP 503: x',
- payload: { id: 'msg-1', body: 'hi' },
- createdAt: new Date('2026-10-01T00:00:00Z'),
- ...overrides,
-});
-
-const webhook = (overrides: Partial<Webhook> = {}): Webhook =>
- ({
- id: 'wh-1',
- sessionId: 'sess-1',
- url: 'https://r.example/h',
- events: ['message.received'],
- active: true,
- retryCount: 3,
- ...overrides,
- }) as Webhook;
+import { recordWebhookDeliveryFailure } from './utils/record-delivery-failure';
describe('WebhookRedriveService', () => {
- let failures: { find: jest.Mock; count: jest.Mock; update: jest.Mock };
- let webhooks: { find: jest.Mock };
- let delivery: { redeliver: jest.Mock };
+ let ds: DataSource;
+ let failures: Repository<WebhookDeliveryFailure>;
+ let webhooks: Repository<Webhook>;
+ let primary: Webhook;
+ let otherSession: Webhook;
+ let retentionHours: number;
+ let delivery: { redeliver: jest.MockedFunction<WebhookDeliveryService['redeliver']> };
let service: WebhookRedriveService;
- beforeEach(() => {
- failures = {
- find: jest.fn().mockResolvedValue([failure()]),
- count: jest.fn().mockResolvedValue(0),
- update: jest.fn().mockResolvedValue({ affected: 1 }),
+ const addFailure = (overrides: Partial<WebhookDeliveryFailure> = {}): Promise<WebhookDeliveryFailure> =>
+ failures.save(
+ failures.create({
+ webhookId: primary.id,
+ sessionId: primary.sessionId,
+ event: 'message.received',
+ url: primary.url,
+ idempotencyKey: randomUUID(),
+ deliveryId: randomUUID(),
+ attempts: 3,
+ lastStatusCode: 503,
+ lastError: 'HTTP 503: receiver unavailable',
+ payload: { id: 'msg-1', body: 'hi' },
+ createdAt: new Date(Date.now() - 60_000),
+ ...overrides,
+ }),
+ );
+
+ beforeEach(async () => {
+ ds = new DataSource({
+ type: 'better-sqlite3',
+ database: ':memory:',
+ entities: [Session, Webhook, WebhookDeliveryFailure],
+ synchronize: true,
+ });
+ await ds.initialize();
+ failures = ds.getRepository(WebhookDeliveryFailure);
+ webhooks = ds.getRepository(Webhook);
+ for (const id of ['sess-1', 'sess-2']) {
+ await ds.getRepository(Session).save({ id, name: id, status: SessionStatus.READY, config: {} });
+ }
+ [primary, otherSession] = await webhooks.save(
+ ['sess-1', 'sess-2'].map(sessionId =>
+ webhooks.create({
+ sessionId,
+ url: `https://${sessionId}.example/h`,
+ events: ['message.received'],
+ active: true,
+ retryCount: 3,
+ }),
+ ),
+ );
+ retentionHours = 24;
+ delivery = {
+ redeliver: jest.fn<
+ ReturnType<WebhookDeliveryService['redeliver']>,
+ Parameters<WebhookDeliveryService['redeliver']>
+ >(async (webhook, sessionId, event, key, data) => {
+ if (data.fail === true) {
+ const recorded = await recordWebhookDeliveryFailure(
+ failures,
+ { error: jest.fn() },
+ {
+ webhookId: webhook.id,
+ sessionId,
+ event,
+ idempotencyKey: key,
+ url: webhook.url,
+ attempts: 1,
+ lastStatusCode: 503,
+ lastError: 'HTTP 503: receiver unavailable',
+ payload: data,
+ },
+ );
+ return recorded === null ? 'unrecorded' : 'failed';
+ }
+ await failures.delete({ webhookId: webhook.id, idempotencyKey: key });
+ return 'delivered';
+ }),
};
- webhooks = { find: jest.fn().mockResolvedValue([webhook()]) };
- delivery = { redeliver: jest.fn().mockResolvedValue('delivered') };
service = new WebhookRedriveService(
- webhooks as unknown as Repository<Webhook>,
- failures as unknown as Repository<WebhookDeliveryFailure>,
+ webhooks,
+ failures,
delivery as unknown as WebhookDeliveryService,
+ { get: () => retentionHours } as unknown as ConfigService,
);
});
- const findArgs = (): {
- select: Record<string, boolean>;
- where: Record<string, unknown>;
- take: number;
- order: Record<string, string>;
- } => (failures.find.mock.calls as unknown[][])[0][0] as ReturnType<typeof findArgs>;
+ afterEach(async () => {
+ jest.restoreAllMocks();
+ await ds.destroy();
+ });
- it('replays a stored row with its STORED idempotency key, one attempt, and reports it delivered', async () => {
- const result = await service.redrive({});
+ it('replays the stored payload and key with one attempt, and removes the delivered row', async () => {
+ const row = await addFailure();
+ expect(await service.redrive({})).toEqual({
+ redriven: 1,
+ delivered: 1,
+ enqueued: 0,
+ failed: 0,
+ skipped: 0,
+ remaining: 0,
+ });
expect(delivery.redeliver).toHaveBeenCalledWith(
- expect.objectContaining({ id: 'wh-1' }),
- 'sess-1',
- 'message.received',
- 'key-1',
- { id: 'msg-1', body: 'hi' },
+ expect.objectContaining({ id: primary.id }),
+ row.sessionId,
+ row.event,
+ row.idempotencyKey,
+ row.payload,
{ singleAttempt: true },
);
- expect(result).toEqual({ redriven: 1, delivered: 1, enqueued: 0, failed: 0, skipped: 0, remaining: 0 });
+ expect(await failures.count()).toBe(0);
+ expect(await service.redrive({})).toMatchObject({ delivered: 0, remaining: 0 });
});
- it('reads the select:false payload explicitly, takes only terminal keyed rows, oldest first', async () => {
- await service.redrive({});
+ it('selects eligible subscriptions before the batch limit, past more than 100 old unavailable rows', async () => {
+ const disabled = await webhooks.save(webhooks.create({ ...primary, id: randomUUID(), active: false }));
+ const unsubscribed = await webhooks.save(
+ webhooks.create({ ...primary, id: randomUUID(), events: ['message.ack'] }),
+ );
+ const unavailable = [
+ { webhookId: randomUUID() },
+ { webhookId: disabled.id },
+ { webhookId: unsubscribed.id },
+ { webhookId: otherSession.id, sessionId: primary.sessionId },
+ ];
+ for (let i = 0; i < 104; i++) {
+ await addFailure({ ...unavailable[i % unavailable.length], createdAt: new Date(Date.now() - 120_000) });
+ }
+ const healthy = await addFailure();
- const args = findArgs();
- expect(args.select.payload).toBe(true);
- expect(args.where.attempts).toBeInstanceOf(FindOperator);
- expect((args.where.attempts as FindOperator<number>).type).toBe('moreThan');
- expect((args.where.payload as FindOperator<unknown>).type).toBe('not');
- expect((args.where.idempotencyKey as FindOperator<unknown>).type).toBe('not');
- expect(args.order).toEqual({ createdAt: 'ASC', id: 'ASC' });
- expect(args.take).toBe(100);
+ expect(await service.redrive({})).toMatchObject({ delivered: 1, skipped: 0, remaining: 0 });
+ expect(delivery.redeliver).toHaveBeenCalledTimes(1);
+ expect(delivery.redeliver.mock.calls[0][3]).toBe(healthy.idempotencyKey);
+ expect(await failures.count()).toBe(104);
});
- it('clamps the limit to the maximum batch', async () => {
- await service.redrive({ limit: 10_000 });
- expect(findArgs().take).toBe(MAX_WEBHOOK_REDRIVE_LIMIT);
+ it('moves a failed batch behind never-replayed rows', async () => {
+ for (let i = 0; i < 100; i++) {
+ await addFailure({ payload: { fail: true }, createdAt: new Date(Date.now() - 120_000) });
+ }
+ await addFailure({ payload: { healthy: 1 } });
+ await addFailure({ payload: { healthy: 2 } });
+ expect(await service.redrive({})).toMatchObject({ failed: 100, delivered: 0, remaining: 102 });
+ expect(await service.redrive({ limit: 2 })).toMatchObject({ failed: 0, delivered: 2, remaining: 100 });
+ expect(await failures.countBy({ attempts: 4 })).toBe(100);
});
- it('confines a scoped key to its allowedSessions, and a sessionId outside them to nothing', async () => {
- await service.redrive({}, ['sess-1', 'sess-2']);
- expect((findArgs().where.sessionId as FindOperator<string[]>).value).toEqual(['sess-1', 'sess-2']);
+ it('requires a terminal keyed payload within the current retention window', async () => {
+ await addFailure({ createdAt: new Date(Date.now() - 25 * 60 * 60 * 1000) });
+ await addFailure({ attempts: 0 });
+ await addFailure({ payload: null });
+ await addFailure({ idempotencyKey: null as never });
+ const fresh = await addFailure();
- failures.find.mockClear();
- const result = await service.redrive({ sessionId: 'sess-9' }, ['sess-1']);
- expect(failures.find).not.toHaveBeenCalled();
- expect(delivery.redeliver).toHaveBeenCalledTimes(1);
- expect(result.redriven).toBe(0);
+ expect(await service.redrive({})).toMatchObject({ delivered: 1, remaining: 0 });
+ expect(delivery.redeliver.mock.calls[0][3]).toBe(fresh.idempotencyKey);
+ expect(await failures.count()).toBe(4);
});
- it('narrows by webhookId and explicit ids', async () => {
- await service.redrive({ webhookId: 'wh-1', ids: ['f-1', 'f-2'] });
- const where = findArgs().where;
- expect(where.webhookId).toBe('wh-1');
- expect((where.id as FindOperator<string[]>).value).toEqual(['f-1', 'f-2']);
+ it('returns no replayable rows when retention is off, even before payload cleanup', async () => {
+ await addFailure();
+ retentionHours = 0;
+
+ expect(await service.redrive({})).toMatchObject({ redriven: 0, remaining: 0 });
+ expect(delivery.redeliver).not.toHaveBeenCalled();
+ expect(await failures.count()).toBe(1);
});
- it('skips a row whose webhook is gone, disabled, unsubscribed or now owned by another session', async () => {
- failures.find.mockResolvedValue([
- failure({ id: 'a', webhookId: 'gone' }),
- failure({ id: 'b', webhookId: 'off' }),
- failure({ id: 'c', webhookId: 'unsub' }),
- failure({ id: 'd', webhookId: 'moved' }),
- ]);
- webhooks.find.mockResolvedValue([
- webhook({ id: 'off', active: false }),
- webhook({ id: 'unsub', events: ['message.ack'] }),
- webhook({ id: 'moved', sessionId: 'sess-2' }),
- ]);
+ it('narrows both delivery and remaining to the same ids, webhook and allowed sessions', async () => {
+ const chosen = await addFailure();
+ await addFailure();
+ const outside = await addFailure({ webhookId: otherSession.id, sessionId: otherSession.sessionId });
- const result = await service.redrive({});
+ expect(
+ await service.redrive({ ids: [chosen.id, outside.id], webhookId: primary.id }, [primary.sessionId]),
+ ).toMatchObject({ delivered: 1, remaining: 0 });
+ expect(delivery.redeliver).toHaveBeenCalledTimes(1);
+ expect(delivery.redeliver.mock.calls[0][3]).toBe(chosen.idempotencyKey);
+ expect(await failures.count()).toBe(2);
+ expect(await service.redrive({ sessionId: otherSession.sessionId }, [primary.sessionId])).toMatchObject({
+ redriven: 0,
+ remaining: 0,
+ });
+ expect(await service.redrive({ webhookId: otherSession.id }, [primary.sessionId])).toMatchObject({
+ redriven: 0,
+ remaining: 0,
+ });
+ expect(await service.redrive({ ids: [] })).toMatchObject({ redriven: 0, remaining: 0 });
+ expect(delivery.redeliver).toHaveBeenCalledTimes(1);
+ expect(await service.redrive({ sessionId: otherSession.sessionId }, null)).toMatchObject({
+ delivered: 1,
+ remaining: 0,
+ });
+ });
+
+ it('allows wildcard subscriptions to replay another event', async () => {
+ await webhooks.update(primary.id, { events: ['*'] });
+ await addFailure({ event: 'session.disconnected' });
+
+ expect(await service.redrive({})).toMatchObject({ delivered: 1, remaining: 0 });
+ expect(delivery.redeliver.mock.calls[0][2]).toBe('session.disconnected');
+ });
+ it('matches exact event members and ignores non-array subscriptions', async () => {
+ await webhooks.update(primary.id, { events: ['message.received.extra'] });
+ await addFailure();
+ const malformed = await webhooks.save(
+ webhooks.create({ ...primary, id: randomUUID(), events: { event: 'message.received' } as never }),
+ );
+ await addFailure({ webhookId: malformed.id });
+
+ expect(await service.redrive({})).toMatchObject({ redriven: 0, remaining: 0 });
expect(delivery.redeliver).not.toHaveBeenCalled();
- expect(result.skipped).toBe(4);
+ const exact = await addFailure({ event: 'message.received.extra' });
+ expect(await service.redrive({})).toMatchObject({ delivered: 1, remaining: 0 });
+ expect(delivery.redeliver.mock.calls[0][3]).toBe(exact.idempotencyKey);
+ });
+
+ it('supports unrestricted redrive with more than 1000 active webhooks', async () => {
+ const ids = Array.from({ length: 1001 }, () => randomUUID());
+ await webhooks
+ .createQueryBuilder()
+ .insert()
+ .values(ids.map(id => ({ id, sessionId: primary.sessionId, url: primary.url, events: ['message.received'] })))
+ .updateEntity(false)
+ .execute();
+ const row = await addFailure({ webhookId: ids.at(-1) });
+
+ expect(await service.redrive({})).toMatchObject({ delivered: 1, remaining: 0 });
+ expect(delivery.redeliver.mock.calls[0][3]).toBe(row.idempotencyKey);
+ });
+
+ it('orders equal-attempt rows by creation time and id', async () => {
+ const older = new Date(Date.now() - 120_000);
+ const first = await addFailure({
+ id: '00000000-0000-4000-8000-000000000001',
+ createdAt: older,
+ payload: { fail: true },
+ });
+ const second = await addFailure({
+ id: '00000000-0000-4000-8000-000000000002',
+ createdAt: older,
+ payload: { fail: true },
+ });
+ await addFailure();
+
+ expect(await service.redrive({ limit: 2 })).toMatchObject({ failed: 2, remaining: 3 });
+ expect(delivery.redeliver.mock.calls.map(call => call[3])).toEqual([first.idempotencyKey, second.idempotencyKey]);
+ });
+
+ it('clamps an internal request to the maximum batch', async () => {
+ for (let i = 0; i <= MAX_WEBHOOK_REDRIVE_LIMIT; i++) await addFailure({ payload: { fail: true } });
+
+ expect(await service.redrive({ limit: 10_000 })).toMatchObject({
+ failed: MAX_WEBHOOK_REDRIVE_LIMIT,
+ remaining: MAX_WEBHOOK_REDRIVE_LIMIT + 1,
+ });
+ expect(delivery.redeliver).toHaveBeenCalledTimes(MAX_WEBHOOK_REDRIVE_LIMIT);
});
- it('counts enqueued and cancelled outcomes, and raises attempts on a replay that fails again', async () => {
- failures.find.mockResolvedValue([
- failure({ id: 'a', attempts: 3 }),
- failure({ id: 'b', attempts: 5 }),
- failure({ id: 'c' }),
- failure({ id: 'd', attempts: 2 }),
- ]);
+ it('counts legacy enqueued and cancelled outcomes, and increments a rejected replay', async () => {
+ await addFailure();
+ await addFailure();
+ await addFailure();
delivery.redeliver
.mockResolvedValueOnce('enqueued')
- .mockResolvedValueOnce('failed')
.mockResolvedValueOnce('cancelled')
- .mockRejectedValueOnce(new Error('boom'));
- failures.count.mockResolvedValue(2);
+ .mockRejectedValueOnce(new Error('receiver failed'));
- const result = await service.redrive({});
-
- expect(result).toEqual({ redriven: 1, delivered: 0, enqueued: 1, failed: 2, skipped: 1, remaining: 2 });
- expect(failures.update).toHaveBeenCalledWith({ id: 'b' }, { attempts: 6 });
- expect(failures.update).toHaveBeenCalledWith({ id: 'd' }, { attempts: 3 });
- expect(failures.update).toHaveBeenCalledTimes(2);
+ expect(await service.redrive({})).toMatchObject({ redriven: 1, enqueued: 1, failed: 1, skipped: 1, remaining: 3 });
+ expect(await failures.countBy({ attempts: 4 })).toBe(1);
});
- it('keeps the batch going when recording a failed replay fails', async () => {
- failures.find.mockResolvedValue([failure({ id: 'a' }), failure({ id: 'b' })]);
- delivery.redeliver.mockResolvedValueOnce('failed').mockResolvedValueOnce('delivered');
- failures.update.mockRejectedValue(new Error('db down'));
-
- const result = await service.redrive({});
+ it('continues the batch when recording a failed replay fails', async () => {
+ await addFailure({ payload: { fail: true }, createdAt: new Date(Date.now() - 120_000) });
+ await addFailure();
+ jest.spyOn(failures, 'update').mockRejectedValueOnce(new Error('database unavailable'));
+ jest.spyOn(failures, 'increment').mockRejectedValueOnce(new Error('database unavailable'));
- expect(result.failed).toBe(1);
- expect(result.delivered).toBe(1);
+ expect(await service.redrive({})).toMatchObject({ failed: 1, delivered: 1, remaining: 1 });
});
- it('runs overlapping calls one after the other, and survives a failed batch', async () => {
+ it('serializes overlapping calls and keeps the chain usable after a failed batch', async () => {
+ await addFailure();
+ jest.spyOn(failures, 'createQueryBuilder').mockImplementationOnce(() => {
+ throw new Error('database unavailable');
+ });
+ await expect(service.redrive({})).rejects.toThrow('database unavailable');
let release!: () => void;
const gate = new Promise<void>(resolve => (release = resolve));
- delivery.redeliver.mockImplementationOnce(async () => {
+ const deliver = delivery.redeliver.getMockImplementation()!;
+ delivery.redeliver.mockImplementationOnce(async (...args) => {
await gate;
- return 'delivered';
+ return deliver(...args);
});
- failures.find.mockRejectedValueOnce(new Error('db down')).mockResolvedValue([failure()]);
const first = service.redrive({});
- await expect(first).rejects.toThrow('db down');
-
const second = service.redrive({});
- const third = service.redrive({});
await new Promise(resolve => setImmediate(resolve));
- // The third call has not read anything while the second is still replaying.
- expect(failures.find).toHaveBeenCalledTimes(2);
+ expect(delivery.redeliver).toHaveBeenCalledTimes(1);
release();
- await expect(second).resolves.toMatchObject({ delivered: 1 });
- await expect(third).resolves.toMatchObject({ delivered: 1 });
- expect(failures.find).toHaveBeenCalledTimes(3);
+ expect(await first).toMatchObject({ delivered: 1, remaining: 0 });
+ expect(await second).toMatchObject({ delivered: 0, remaining: 0 });
+ expect(delivery.redeliver).toHaveBeenCalledTimes(1);
});
- it('does not read webhooks when nothing is replayable', async () => {
- failures.find.mockResolvedValue([]);
- failures.count.mockResolvedValue(0);
-
- const result = await service.redrive({});
+ it('bounds parallel direct replays to four', async () => {
+ for (let i = 0; i < 10; i++) await addFailure();
+ const deliver = delivery.redeliver.getMockImplementation()!;
+ let active = 0;
+ let peak = 0;
+ delivery.redeliver.mockImplementation(async (...args) => {
+ active++;
+ peak = Math.max(peak, active);
+ await new Promise(resolve => setImmediate(resolve));
+ const outcome = await deliver(...args);
+ active--;
+ return outcome;
+ });
- expect(webhooks.find).not.toHaveBeenCalled();
- expect(result).toEqual({ redriven: 0, delivered: 0, enqueued: 0, failed: 0, skipped: 0, remaining: 0 });
+ expect(await service.redrive({})).toMatchObject({ delivered: 10, remaining: 0 });
+ expect(peak).toBe(4);
});
});
diff --git a/src/modules/webhook/webhook-redrive.service.ts b/src/modules/webhook/webhook-redrive.service.ts
index d784057cc..ff1d0ad1c 100644
--- a/src/modules/webhook/webhook-redrive.service.ts
+++ b/src/modules/webhook/webhook-redrive.service.ts
@@ -1,6 +1,7 @@
import { Injectable } from '@nestjs/common';
+import { ConfigService } from '@nestjs/config';
import { InjectRepository } from '@nestjs/typeorm';
-import { FindOptionsWhere, In, IsNull, MoreThan, Not, Repository } from 'typeorm';
+import { In, IsNull, MoreThan, Not, Repository } from 'typeorm';
import { Webhook } from './entities/webhook.entity';
import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';
import { WebhookDeliveryService } from './webhook-delivery.service';
@@ -69,6 +70,7 @@ export class WebhookRedriveService {
@InjectRepository(WebhookDeliveryFailure, 'data')
private readonly failureRepository: Repository<WebhookDeliveryFailure>,
private readonly delivery: WebhookDeliveryService,
+ private readonly configService: ConfigService,
) {}
redrive(request: WebhookRedriveRequest, allowedSessions?: string[] | null): Promise<WebhookRedriveResult> {
@@ -93,38 +95,43 @@ export class WebhookRedriveService {
// The calling key's allowedSessions is authoritative; the body's sessionId may only narrow it.
const sessionScope = resolveSessionScope(allowedSessions, request.sessionId);
if (sessionScope !== null && sessionScope.length === 0) return result;
+ const retentionHours = this.configService.get<number>('webhook.failurePayloadRetentionHours', 0);
+ if (retentionHours <= 0 || request.ids?.length === 0) return result;
- // Replayable: terminal (an attempts-0 row is the outbox's to replay), with a stored payload and
- // the key the receiver dedups on.
- const where: FindOptionsWhere<WebhookDeliveryFailure> = {
- attempts: MoreThan(0),
- payload: Not(IsNull()),
- idempotencyKey: Not(IsNull()),
- ...(sessionScope ? { sessionId: In(sessionScope) } : {}),
- ...(request.webhookId ? { webhookId: request.webhookId } : {}),
- };
+ const cutoff = new Date(Date.now() - retentionHours * 60 * 60 * 1000);
+ // Join eligible subscriptions before taking a batch. A single membership predicate also avoids
+ // an OR expression and repeated ids parameters for every active webhook across all sessions.
+ const subscriptions =
+ this.failureRepository.manager.connection.options.type === 'postgres'
+ ? "json_array_elements_text(CASE WHEN json_typeof(webhook.events::json) = 'array' THEN webhook.events::json ELSE '[]'::json END) AS subscribed(event)"
+ : "json_each(CASE WHEN json_type(webhook.events) = 'array' THEN webhook.events ELSE '[]' END) AS subscribed";
+ const subscribedEvent =
+ this.failureRepository.manager.connection.options.type === 'postgres' ? 'subscribed.event' : 'subscribed.value';
+ const query = this.failureRepository
+ .createQueryBuilder('failure')
+ .innerJoin(Webhook, 'webhook', 'webhook.id = failure.webhookId AND webhook.sessionId = failure.sessionId')
+ .where({
+ attempts: MoreThan(0),
+ payload: Not(IsNull()),
+ idempotencyKey: Not(IsNull()),
+ createdAt: MoreThan(cutoff),
+ ...(request.ids ? { id: In(request.ids) } : {}),
+ ...(sessionScope ? { sessionId: In(sessionScope) } : {}),
+ ...(request.webhookId ? { webhookId: request.webhookId } : {}),
+ })
+ .andWhere('webhook.active = :active', { active: true })
+ .andWhere(`EXISTS (SELECT 1 FROM ${subscriptions} WHERE ${subscribedEvent} IN ('*', failure.event))`);
const limit = Math.min(Math.max(1, request.limit ?? DEFAULT_WEBHOOK_REDRIVE_LIMIT), MAX_WEBHOOK_REDRIVE_LIMIT);
- const rows = await this.failureRepository.find({
- // `payload` is select: false on the entity; name every column so it is read here.
- select: {
- id: true,
- webhookId: true,
- sessionId: true,
- event: true,
- url: true,
- idempotencyKey: true,
- deliveryId: true,
- attempts: true,
- lastStatusCode: true,
- lastError: true,
- payload: true,
- createdAt: true,
- },
- where: request.ids?.length ? { ...where, id: In(request.ids) } : where,
- // Oldest first: the order the events were lost in.
- order: { createdAt: 'ASC', id: 'ASC' },
- take: limit,
- });
+ const rows = await query
+ .clone()
+ // `payload` is select: false on the entity; replay explicitly reads it.
+ .addSelect('failure.payload')
+ // Failed replays move behind rows with fewer attempts, so a bad receiver cannot pin a batch.
+ .orderBy('failure.attempts', 'ASC')
+ .addOrderBy('failure.createdAt', 'ASC')
+ .addOrderBy('failure.id', 'ASC')
+ .take(limit)
+ .getMany();
if (rows.length > 0) {
const webhookIds = [...new Set(rows.map(r => r.webhookId))];
@@ -141,7 +148,7 @@ export class WebhookRedriveService {
}
result.redriven = result.delivered + result.enqueued;
- result.remaining = await this.failureRepository.count({ where });
+ result.remaining = await query.clone().getCount();
return result;
}
@@ -169,7 +176,8 @@ export class WebhookRedriveService {
else if (outcome === 'cancelled') result.skipped++;
else {
result.failed++;
- await this.bumpAttempts(row);
+ // Terminal failures update their own attempt count in the shared recorder.
+ if (outcome !== 'failed') await this.bumpAttempts(row);
}
} catch (error) {
result.failed++;
@@ -185,7 +193,7 @@ export class WebhookRedriveService {
/** Count the failed replay on the row. Best-effort: the row is already on record either way. */
private async bumpAttempts(row: WebhookDeliveryFailure): Promise<void> {
try {
- await this.failureRepository.update({ id: row.id }, { attempts: row.attempts + 1 });
+ await this.failureRepository.increment({ id: row.id }, 'attempts', 1);
} catch (error) {
this.logger.warn('Could not record a failed webhook redrive on its row', {
failureId: row.id,
diff --git a/src/modules/webhook/webhook.service.spec.ts b/src/modules/webhook/webhook.service.spec.ts
index 6814ac64d..1c4e9b12d 100644
--- a/src/modules/webhook/webhook.service.spec.ts
+++ b/src/modules/webhook/webhook.service.spec.ts
@@ -679,6 +679,9 @@ describe('WebhookService', () => {
});
it('flags the rows that still hold a replay payload, without reading any payload', async () => {
+ (configService.get as jest.Mock).mockImplementation((key: string, def?: unknown) =>
+ key === 'webhook.failurePayloadRetentionHours' ? 24 : def,
+ );
(failureRepository.find as jest.Mock)
.mockResolvedValueOnce([{ id: 'f1' }, { id: 'f2' }])
.mockResolvedValueOnce([{ id: 'f2' }]);
@@ -695,6 +698,13 @@ describe('WebhookService', () => {
};
expect(second.select).toEqual({ id: true });
expect(second.where.id).toEqual(In(['f1', 'f2']));
+ expect(second.where.createdAt).toEqual(expect.any(FindOperator));
+ });
+
+ it('reports retained payloads as unavailable when retention is disabled', async () => {
+ (failureRepository.find as jest.Mock).mockResolvedValue([{ id: 'f1' }]);
+ await expect(service.listDeliveryFailures({})).resolves.toEqual([{ id: 'f1', replayable: false }]);
+ expect(failureRepository.find).toHaveBeenCalledTimes(1);
});
it('skips the replayable lookup for an empty page', async () => {
diff --git a/src/modules/webhook/webhook.service.ts b/src/modules/webhook/webhook.service.ts
index 12a9e6971..d4085fc23 100644
--- a/src/modules/webhook/webhook.service.ts
+++ b/src/modules/webhook/webhook.service.ts
@@ -255,7 +255,8 @@ export class WebhookService implements OnModuleInit, OnModuleDestroy {
// `payload` is select: false, so the page above never carries an event body. Which rows hold one
// is a second, id-only read; with payload retention off no row does and it matches nothing.
const replayable = new Set<string>();
- if (rows.length > 0) {
+ const hours = this.configService.get<number>('webhook.failurePayloadRetentionHours', 0);
+ if (rows.length > 0 && hours > 0) {
const withPayload = await this.failureRepository.find({
select: { id: true },
where: {
@@ -263,6 +264,7 @@ export class WebhookService implements OnModuleInit, OnModuleDestroy {
payload: Not(IsNull()),
attempts: MoreThan(0),
idempotencyKey: Not(IsNull()),
+ createdAt: MoreThan(new Date(Date.now() - hours * 60 * 60 * 1000)),
},
});
for (const r of withPayload) replayable.add(r.id);