Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(metrics): say the webhook failure counter counts unsent deliveries

Yudhi Armyndharisa0c4080090f6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ad35e4635..52e3092b6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The webhook outbox replay no longer delivers an event the webhook has since been unsubscribed from.
- A webhook delivery that succeeds removes the delivery-failure rows filed under its idempotency key, so an event the outbox replay delivers after a shed or shutdown refusal is no longer listed as lost.
- The delivery-failure row of a shed or shutdown-refused webhook delivery takes the reason its replay failed with, and the attempt count once the replay was sent.
+- The `openwa_webhook_delivery_failures_total` help text and the metrics reference say it also counts webhook deliveries that were never sent.
- `POST /api/sessions/:sessionId/webhooks/:id/test` sends a fresh `X-OpenWA-Idempotency-Key` on every call, so a deduplicating receiver runs each test.
- A WebSocket `message` frame with no payload answers `INVALID_MESSAGE` instead of a generic exception.
- On PostgreSQL, boot no longer runs FTS schema DDL when the `body_ts` column and its index already exist, so a restart no longer queues every read and write on `messages` behind the open ones.
diff --git a/docs/10-devops-infrastructure.md b/docs/10-devops-infrastructure.md
index 6990ec737..9847dfcdd 100644
--- a/docs/10-devops-infrastructure.md
+++ b/docs/10-devops-infrastructure.md
@@ -949,7 +949,7 @@ export class MetricsService {
| `openwa_sessions` | gauge | `status` | Session count per status |
| `openwa_messages_total` | gauge | `direction` (`incoming`/`outgoing`) | Current stored messages by direction |
| `openwa_messages_failed_total` | gauge | — | Current messages in FAILED state |
-| `openwa_webhook_delivery_failures_total` | counter | — | Webhook deliveries that terminally failed (all retries exhausted) since process start |
+| `openwa_webhook_delivery_failures_total` | counter | — | Webhook delivery failures since process start: retries exhausted, or never sent |
| `openwa_session_reconnect_attempts_total` | counter | — | Reconnect attempts scheduled across all sessions since process start |
| `openwa_session_reconnect_loop_alerts_total` | counter | — | Reconnect-loop alerts emitted since process start |
| `openwa_sessions_restricted` | gauge | — | Sessions whose account WhatsApp is currently restricting |
diff --git a/src/common/metrics/webhook-delivery-metrics.ts b/src/common/metrics/webhook-delivery-metrics.ts
index dd68596ba..7ac409f69 100644
--- a/src/common/metrics/webhook-delivery-metrics.ts
+++ b/src/common/metrics/webhook-delivery-metrics.ts
@@ -1,20 +1,22 @@
/**
- * Process-local monotonic counter of webhook deliveries that terminally failed (every retry
- * exhausted), incremented one-for-one alongside the durable `webhook_delivery_failures` dead-letter
- * rows. Kept as a plain in-process counter rather than a `COUNT(*)` over that table because the table
- * is pruned on a retention schedule, which would make its count non-monotonic and therefore invalid
- * as a Prometheus `counter` (a prune would look like a counter reset to `rate()`/`increase()`). An
+ * Process-local monotonic counter of webhook delivery failures: deliveries whose every retry was
+ * exhausted, and deliveries never sent (shed, refused at shutdown, rejected before sending). It is
+ * incremented once per row inserted into the durable `webhook_delivery_failures` dead-letter table;
+ * a later replay may replace or remove that row, and the counter keeps the failure. Kept as a plain
+ * in-process counter rather than a `COUNT(*)` over that table because the table is pruned on a
+ * retention schedule, which would make its count non-monotonic and therefore invalid as a
+ * Prometheus `counter` (a prune would look like a counter reset to `rate()`/`increase()`). An
* in-process counter only resets on restart, which those functions already handle correctly, and it
* also captures the failure even when persisting the dead-letter row itself fails.
*/
let terminalFailureTotal = 0;
-/** Record one terminal (all-retries-exhausted) webhook delivery failure. */
+/** Record one webhook delivery failure, terminal or never sent. */
export function incrementWebhookDeliveryFailures(): void {
terminalFailureTotal += 1;
}
-/** Current process-lifetime total of terminal webhook delivery failures. */
+/** Current process-lifetime total of recorded webhook delivery failures. */
export function getWebhookDeliveryFailuresTotal(): number {
return terminalFailureTotal;
}
diff --git a/src/modules/metrics/metrics.service.ts b/src/modules/metrics/metrics.service.ts
index e366be2dd..1f731fe7f 100644
--- a/src/modules/metrics/metrics.service.ts
+++ b/src/modules/metrics/metrics.service.ts
@@ -146,7 +146,7 @@ export class MetricsService {
}
lines.push(
- '# HELP openwa_webhook_delivery_failures_total Webhook deliveries that terminally failed (all retries exhausted) since process start.',
+ '# HELP openwa_webhook_delivery_failures_total Webhook delivery failures recorded since process start: retries exhausted, or never sent (shed, refused at shutdown, rejected before sending).',
);
lines.push('# TYPE openwa_webhook_delivery_failures_total counter');
lines.push(`openwa_webhook_delivery_failures_total ${getWebhookDeliveryFailuresTotal()}`);