Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

106 KB
Raw Blame

Notifications Contract

Contract Metadata

{
  "subsystem_id": "notifications",
  "lane": "L6",
  "contract_file": "docs/release-control/v6/internal/subsystems/notifications.md",
  "status_file": "docs/release-control/v6/internal/status.json",
  "registry_file": "docs/release-control/v6/internal/subsystems/registry.json",
  "dependency_subsystem_ids": [
    "security-privacy"
  ]
}

Purpose

Removing a webhook revokes its identified queued work

Before queued firing or recovery delivery, a saved webhook ID must still name an enabled current destination. A missing ID match cancels that retained work; another destination at the same URL cannot authorise its old payload or credentials. This applies to pending work and operator-retried terminal failures, including after reopening the persistent queue. Policy cancellation is not a provider attempt, success receipt or new delivery audit. Existing failed audits and retained configuration/occurrence links are preserved.

Only legacy queued configuration with no webhook ID may use the existing exact URL fallback. Editing the same enabled ID still delivers its previously admitted configuration snapshot; this repair does not retarget old work, cancel requests already in flight, or change global pause, disabled-destination, retry or retention policy. New alerts for a different enabled destination at that URL still deliver.

queue_destination_revocation_test.go checks identity/URL/disabled/legacy controls and real generic/ntfy firing/recovery delivery across deletion, queue reopen, operator retry, cancellation, audit/receipt readback and new-alert delivery. This is synthetic source proof, not evidence of prior exposure, installed-provider recovery or resolution of a reported notification outage.

The DELETE acknowledgement includes success: true only after durable and live removal complete, retaining the legacy status: "success" field. The existing editor requires that explicit boolean before removing the visible destination; a successful mutation must not be displayed as failed or encourage a blind retry. Persistence/live-publication failures retain their existing refusal/rollback behaviour. TestNotificationHandlers/DeleteWebhook pins both success fields; its failure controls still prove that a rejected save never removes live state.

Approved addresses share a bounded connection budget

Webhook and HTTP Apprise transports pass their validated numeric destinations to pkg/securityutil/pinned_dial.go. A stalled first address must leave time for a healthy later address, including another address of the same family. The shared primitive staggers at most two connect workers, prefers the other address family for fallback, and shares one ten-second connection budget among all candidates. A winner cancels other attempts and closes late successful connections. Only the winning connection carries the HTTP request.

Webhook admission and dialing use the same context-aware DNS lookup with a five-second ceiling. Redirect validation retains the request context. Empty DNS results fail closed. Private-address admission, DNS pinning, TLS identity, same-origin redirects and queue retry policy retain their existing owners.

pinned_connectivity_linux_test.go saturates a real owned loopback TCP accept queue while a second address on the same port accepts HTTP. Both secure clients must reach that receiver exactly once. The queued control reopens the persisted sent status, audit and telemetry with one attempt, one delivery and no failure. webhook_dns_context_test.go exercises cancellation through the production resolver path and verifies bounded admission and private-address refusal. These are source controls, not attribution of fleet connectivity counters or proof of installed-destination recovery.

Destination saves use one submitted snapshot without replacing newer input

The existing email, Apprise, watchdog and entitled push writes preserve their payload fields, endpoint order and server-side policy. The editor captures them before the preceding alert-policy request yields, rather than reading later edits between writes. Apprise response normalisation and masking apply only to its untouched submitted draft in the same loaded context. Newer input and unsaved status survive old acknowledgements; context invalidation stops remaining unsent calls, without cancelling or undoing an issued request. Failed multi-endpoint saves can have earlier persisted writes and are not represented as atomic rollback or provider delivery.

The aggregate editor refuses all policy/destination writes until it has loaded the saved policy for its current context. A rejected policy GET is not permission to save empty or disabled destination defaults. Explicit reload restores normal admission only after saved policy and destination loading finish; existing destination-load errors still block saves. The acknowledgement controls require no downstream writes after initial or replacement policy-read failures, without changing provider, queue or retry semantics.

useAlertDestinationsState.test.tsx and alertSettingsSave.acknowledgement.test.tsx use deferred and rejected replies to prove the snapshot, draft, retry and context boundaries. The genuine browser fixture runs the existing surface, state owners and API adapters against local synthetic endpoints, not SMTP/Apprise providers. Credentials, TLS verification, entitlement, activation, ordinary delivery, queue and retry policy are unchanged.

Notification redirects keep the configured receiver's origin

Webhook and HTTP Apprise firing, recovery, Test and admitted queued delivery follow redirects only within the initial HTTP origin: scheme, normalised host and effective port. An SSRF-permitted host, subdomain, DNS alias or another port is not permission to receive private alert bodies, custom authentication headers or signed requests. HTTP/HTTPS changes are also refused. Same-origin paths and queries remain supported, including default-port and equivalent IPv6 spellings; the existing redirect limit, DNS pinning, URL validation and TLS checks remain. Foreign redirects stop before target DNS resolution, validation logging or any request to that receiver. Origin refusal precedes the redirect-count error, so an exhausted same-origin chain cannot expose its final foreign Location or change that refusal into an unknown/retryable error. A fixed configuration failure advises the operator to configure the final destination URL, without copying the receiver-controlled Location.

That typed failure stops both enhanced transport retries and the queue's retry budget. It remains an undelivered dead letter and audit, never a successful firing/recovery receipt; admitted private configuration is not rewritten. Apprise retains the same safe guidance and existing opaque diagnostic policy.

notification_redirect_test.go supplies real loopback controls for all five redirect statuses, foreign ports/hostname aliases, authenticated HTTPS downgrade with and without the existing TLS override, every sender path and the real queue's firing/recovery dead-letter, audit and receipt lifecycle. Same-origin controls verify unchanged methods, payloads, credentials and signatures. Capped chains exercise direct, enhanced-retry and HTTP Apprise paths; the unchanged same-origin redirect-count controls remain. notification_redirect_origin_test.go covers origin equivalence and separation. Existing redirect-chain controls now exercise paths on one receiver, not three different receivers. This is source privacy/compatibility proof, not installed destination acceptance, evidence of prior exposure or resolution of a report.

Webhook diagnostics withhold receiver-controlled text

Ordinary and enhanced firing/recovery senders, including ntfy's plain-text recovery path, drain at most the existing 1 MiB response cap without retaining body bytes. Logs, delivery errors/history and the existing Test response carry structured status, bounded byte counts and incomplete/limit-reached facts, not arbitrary receiver text. Test keeps its status/string/error shape; a nonempty response string now explains that the body was withheld. ResponseLogging enables metadata only. Invalid Retry-After hints retain the fixed fallback diagnostic without copying the receiver's arbitrary header value. Valid hints still report their parsed bounded duration. Reaching the read cap does not claim the body exceeded it.

HTTP status still owns rejection classification and retry decisions, Retry-After and actual attempt/final-status accounting survive, and incomplete 2xx replies remain failures. Body-read errors retain their original cause for errors.Is/As and classification, without formatting its possibly private prose. Requests, credentials, payload/occurrence identity, TLS/SSRF, destination scope, queue policy/persistence and existing operator retry authority are unchanged. Existing historical diagnostics are not rewritten or claimed to be scrubbed.

webhook_response_confidentiality_test.go exercises the real ordinary/enhanced/ ntfy/retry senders with in-memory HTTP responses: echoed credentials/alert text, successful/rejected responses, error-prose disclosure, final delivery history, unchanged requests and response closure. webhook_response_metadata_test.go checks empty/incomplete/at-cap/over-cap bodies and the Test summary. webhook_response_retry_hint_test.go preserves actual retry/fallback behaviour without disclosing invalid hint text. Existing Test/Gotify and ntfy HTTP-error controls retain status and request expectations with the new withheld response. These are synthetic source controls, not native provider acceptance or a cause/fix claim for reported notification failures.

Saved webhook identity is consistent for editing and Test

Create, Update and Test share one resolver for masked header/custom-field and signing-secret values. Reject missing or ambiguous saved placeholders before persisting, changing live destinations or sending a test request. HTTP header identity is case-insensitive across the complete saved map, even where an exact spelling is present; conflicting submitted aliases cannot depend on map order. Custom fields retain exact, case-sensitive identity. A saved signing secret must be non-empty and not itself a placeholder to satisfy a signing mask.

The replacement map still honours explicit add/change/delete and omitted keys; cleared headers are normalized away so the sender's JSON default survives. List/create/update responses remain masked. Create and unsaved Test reject masks instead of silently dropping credentials; a valid literal create still returns masked values. Saved-form Test uses the same resolved edit as Update but does not change live configuration or encrypted persistence. Persist-before-publish, SSRF/TLS policy, routing/scopes, notification pause honesty and already-admitted queue snapshots remain unchanged. Source controls TestWebhookPlaceholderBoundaryAgreement and TestWebhookSavedFormTestMatchesEncryptedEdit complement the existing real ordinary-delivery/restart/grouping controls; none proves native #2540 relief.

Delivery health reconciliation follows the complete send transition

The queue's automatic send worker records the retry/dead-letter outcome and its per-attempt audit before releasing the firing alert's delivery gate and announcing the health change. The monitoring callback can therefore inspect the complete attempt or take the same alert's cancellation gate without blocking delivery on its own held gate. Configuration failures remain immediate dead letters; transient failures keep their existing budget and backoff. Skipped policy delivery remains a cancellation with no invented provider audit. Explicit operator status/retry/dismiss actions still reconcile immediately following their own committed transitions.

queue_delivery_callback_test.go checks configuration, transient and exhausted failures through the real delivery processor, exact committed status, attempt counts/classes, operational links, one callback and re-entrant writer-gate acquisition. Existing disabled-delivery, restart/resolution/retry and queue health tests retain policy cancellation and operator action coverage; the monitor startup test retains saved-destination restoration and its audit check. This changes callback sequencing only: no SQLite driver, schema/SQL, provider policy, credential, SSRF boundary or notification payload is changed.

SMTP authentication and envelopes belong to each delivery

SMTP mechanism negotiation matches complete advertised names, prefers PLAIN over LOGIN and classifies an unsupported nonempty mechanism list as a local configuration failure. It must not select PLAIN from PLAIN-CLIENTTOKEN or LOGIN from XLOGIN, nor spend retry budgets on an unsupported mechanism. The existing PLAIN probe for a missing/empty AUTH advertisement remains available to legacy servers, retaining the underlying EHLO/connection error if that probe fails.

LOGIN sends username followed by password according to exchange order. Server challenges are opaque display text, including empty or localised prompts, and must neither select a credential nor enter diagnostics. A fresh exchange resets its progress. Premature success or a third challenge fails closed without DATA and without another credential response. Genuine SMTP 535 rejection remains an authentication failure, while 454 retains transient retries and its typed reply.

Connection tests and delivery share the implicit-TLS rule: an explicit TLS flag or port 465 selects TLS before SMTP begins. A configured STARTTLS flag must not request another TLS upgrade inside that established TLS session. Saved flags remain the submitted configuration, and certificate verification still applies.

email_authentication_wire_test.go uses real plain/TLS/STARTTLS receivers to check connection tests, ordinary/threaded/attachment delivery, exact mechanism selection, legacy probing, malformed exchanges, certificate refusal and server verdicts. Its persistent queue control reopens accepted and bad-password verdicts with one attempt, retaining the authentication dead letter for bad credentials. The port-465 control maps only the destination to an owned ephemeral listener and retains the real TLS handshake and SMTP session. Protocol-level controls in email_authentication_test.go cover opaque challenges and exchange bounds. These are synthetic source proofs, not native provider acceptance or attribution of the fleet's aggregate authentication counters.

SMTP managers resolve the provider identity and any supported default username before deciding whether credentials require authentication. Explicit usernames take precedence. Validation, connection tests and real sends use the same resolved credentials without replacing the saved username. Invalid credentials retain the existing authentication failure and immediate dead-letter policy. The actual SMTP attempt reports hasAuth from that resolved manager. Queue admission does not infer authentication from the unexpanded saved username.

Each email delivery owns its configuration snapshot, including its sender and recipients. Preparing a concurrent alert or report cannot change an in-flight delivery's envelope or retry policy. Deliveries using the cached SMTP transport share its rate-limit state without mutating its configuration. Saved settings and already-admitted queue snapshots remain independent. This does not refresh retained destinations or credentials after an edit, or add automatic replay.

Delivery verdicts survive session cleanup and interrupted diagnostics

A successful final SMTP DATA reply establishes server acceptance. Plain SMTP, TLS and STARTTLS delivery must not resend an accepted message because QUIT fails, nor turn that cleanup failure into a terminal delivery audit. Cleanup still runs, with only a fixed local diagnostic; the provider's reply text is not copied into it. Missing acknowledgement, certificate failures and rejected DATA replies remain errors with their existing retry policy. A connection test without a completed DATA transaction still reports QUIT failure. Server acceptance is not proof of arrival in a recipient's inbox or exactly-once recovery after a process crash.

The ntfy plain-text recovery path, like the shared firing sender, retains the HTTP rejection class when reading the diagnostic response body fails. It preserves the wrapped read error, existing retry budgets and bounded body read; 401/403/422 remain permanent failures, while transient HTTP verdicts remain retryable. Incomplete 2xx responses retain the existing failure behaviour. Historical audit rows are not rewritten and operator retry stays explicit.

Verification: email_acceptance_test.go uses actual guest-local plain/TLS/ STARTTLS SMTP sessions for clean, missing and rejected QUIT, permanent and transient DATA rejection, missing acknowledgement, threaded/attachment sends, connection-only tests and certificate refusal. Its connected persistent queue control retains one accepted send and zero terminal failures after reopen. ntfy_response_verdict_test.go exercises real interrupted HTTP bodies, then queue/audit/telemetry readback after reopen and an explicit repaired-destination retry without erasing the original rejection. These are synthetic source controls, not a cause attribution for aggregate fleet counters, installed provider acceptance or a reporter's confirmation.

Pending firing groups count occurrences, not callbacks

Before a grouping window expires, repeated firing callbacks with the same full alert ID and start instant occupy one slot. The newest observed snapshot owns the whole payload and operational linkage, including a genuine later severity downgrade. Equal observation times prefer higher severity; an unknown time cannot replace a known later observation. Missing ID/start identity remains separate, as do different occurrences of a reusable ID. The index is scoped to the pending window and discarded on flush, disable, cancellation or stop; callbacks do not extend the original timer or establish delivery.

firing_grouping_occurrence_test.go verifies timestamp/severity ordering, instant equality across time-zone/monotonic representations, cloned payloads, concurrent callbacks, occurrence-qualified cancellation, all-destination and webhook-only disable/zero-window flushes, and notification disable/re-enable. Stopping the manager cancels its firing timer and checkpoints its accepted snapshots as ordinary delayed queue work after workers quiesce. It preserves batch membership, current destination snapshots and the first admission's original deadline (rounded up to queue precision), or a later quiet-hours replay deadline. It does not flush, restart the window, invent a delivery receipt, or fall back to an early direct send if checkpoint storage fails. A restarted manager treats this committed batch as immutable; new admissions start their own window rather than merging into the old process's batch. Queue replay still applies current quiet hours and destination/enable policy. firing_group_checkpoint_test.go verifies exact occurrence/link snapshots, original deadlines after config edits, cancellation, pause/channel selection, tenant-directory isolation, concurrent timer/stop ownership and storage-error silence. Queue schema, drivers and event types are unchanged. TestPendingFiringGroupStopCancelsTimer checks actual timer/HTTP silence after stop. The connected TestPendingFiringGroupHTTPAcrossRestart uses actual timer expiry, persistent queue reopen, the autonomous sender and two loopback receivers: one snapshot per occurrence/destination, truthful counts/latest content, ordinary cooldown and independent firing/recovery receipts. Ungrouped admissions, scheduled escalation, retry budgets, queue schema and delivery policy are unchanged. This is source acceptance, not a native destination, natural schedule cycle or a reproduced cause of the existing flood reports.

Completion order cannot replace a newer occurrence's cooldown

Firing completions still record an independent delivery receipt for their own occurrence and destination. The reusable-ID cooldown, however, belongs to the newest known occurrence: a late older completion or unknown legacy start cannot evict it. Within the same occurrence, successful completion times and delivered severity only advance. A genuinely newer occurrence begins its own severity history. No queue schema, retry budget, recovery selection or schedule changes.

TestNotificationCooldownReceiptOrdering verifies older, unknown and equal-start completion ordering, including legacy zero starts. The connected TestLateFiringHTTPReceiptPreservesCurrentCooldown holds an older HTTP response while the newer firing completes through the persistent sender. Both snapshots are admitted before ordinary queue activation, so one discovered batch can complete concurrently without adding another dispatcher. It verifies that the ordinary repeat adds no queue row, a current severity increase still reaches the receiver, and both occurrence/destination receipts survive queue reopen. This is loopback source acceptance, not native delivery, restart cooldown persistence, a natural schedule cycle or resolution of the original flood reports.

Queue recovery handler ownership after reload

Router monitor replacement must refresh the existing queue/DLQ handler as well as the normal notification handler. A stopped manager intentionally clears its queue; retaining that manager after reload must not leave Retry, Dismiss or queue reads permanently returning Notification queue not initialized while the replacement manager is available. Handler monitor reads and replacement are synchronised; missing monitors/managers remain an explicit 503, not a panic. This changes neither queue state semantics nor notification delivery transport. Requests already in flight during shutdown may still fail and can be retried.

Verification: TestRouterSetMonitorRefreshesNotificationQueue stops the old notifier, replaces the router monitor, and exercises stats, Retry and Dismiss; TestNotificationQueueHandlers_MonitorReplacementConcurrent covers concurrent replacement and unavailable-manager reads with the race detector. This is lifecycle regression proof, not installed SMTP receipt or reporter confirmation.

Manager shutdown is a restart boundary, not a delivery verdict

NotificationManager.Stop first refuses new alerts and stops the firing timer in one critical section, retaining its buffered snapshots for checkpointing. Queue-less recovery buffers keep their existing volatile semantics; persistent recovery grouping is already durable. Work that reads the queue outside the manager lock holds a lease taken in the critical section that reads it. An admitted alert, recovery or timer group is written to the open queue. Cancellation, operator pause and channel disable still run while Stop drains. Direct delivery started by admitted work is owned until it finishes and records its receipt. An attempt made after Stop begins is refused, neither queued nor sent directly, and no direct delivery starts once Stop has finished.

Stop then quiesces the queue. A worker that observes the stop begins no quiet-hours hold, partition or claim: its job stays pending with no attempt, audit or receipt, and the next start delivers it. Work a worker began before that observation finishes while delivery is still permitted, recording its status, audit and durable firing receipt. Only after every worker and lease has ended does Stop withdraw delivery, detach the queue and close storage. Between quiescing workers and closing admission leases, Stop checkpoints the remaining firing group under the manager lock: resolution/pause either removes its buffered snapshot first or cancels its committed row afterwards, never falling between those states. Grouping edits during draining leave this checkpoint intact. Storage failures are logged, not called durable or routed to direct delivery. A recovery after restart still reaches each destination that received the firing. Shutdown must never cancel queued work as "delivery disabled"; that verdict belongs to the pause and destination controls. Resolution cancellation and at-least-once replay of interrupted sends are unchanged. Delivery attempts retain their existing provider timeouts and SMTP retry budget. Stop waits for their durable outcomes instead of closing storage under an active writer. There is no separate aggregate drain deadline. The queue logs the processing stop after refusing new attempts and before waiting, so process-signal proofs can release held provider responses at that boundary.

TestManagerStopKeepsQueuedDeliveryForRestart holds five real HTTP sends in the worker pool while a sixth admitted job waits for a slot, stops the manager, reopens the persistent queue, and checks the kept job, the in-flight outcomes, their durable receipts and a recovery delivered after restart. manager_lifecycle_test.go holds an admitted alert, recovery or timer group at queue serialisation while two concurrent Stop calls run. It checks the write after reopen, the refusal of a later attempt, and pause and channel disable during the drain. It also checks a never-started queue, the discarded buffered group, and direct delivery that Stop waits for and refuses afterwards. These are loopback source proofs, not process-signal, reload or installed-provider acceptance.

Monitor bootstrap also restores the shared installation webhook policy before activating persistent work, including tenant queues. A later router refresh is too late for a due private-webhook job. Missing settings retain default denial; unreadable settings or invalid CIDRs fail construction before queue creation. The monitor startup tests cover delivery with no router, installation revocation against tenant-local grants, and invalid-policy admission. This boundary keeps shutdown-preserved rows from becoming terminal failures during reconstruction.

Own notification delivery transport, provider configuration, queueing, and notification-management API surfaces. The alert schedule selects firing, grouped, and matching recovery delivery through one normalized target (all, email, webhook, or apprise). Escalation delivery remains independently targetable per level, allowing an Apprise/ntfy first notification to escalate through email, or the reverse. Unknown and absent persisted targets preserve historical all-destination behavior, and destination tag filters still apply after target selection. Grouping is an explicit runtime policy: grouping.enabled=false or a zero window delivers each alert independently, and disabling grouping flushes any in-memory pending firing alerts as individual deliveries. Already admitted persistent recovery groups retain their original deadline across a policy edit. Grouped provider payloads must retain every alert, while live ntfy firing deliveries and webhook tests share the same severity-derived title, priority, and tags. Email rendering treats patrol_finding as a finding rather than a numeric threshold alert. Single and grouped HTML/text bodies show the finding message and optional category, omit meaningless zero-value metrics, preserve mixed group headings, and HTML-escape every resource, message, and category field. The alerts-owned append-only event log may record notification dispatch, quiet-hours deferral, and suppression decisions at the alert manager's policy seams. Those events explain why delivery was or was not attempted, but they do not replace the notifications-owned queue, delivery log, retry/DLQ state, or provider receipts and must not be treated as proof that a destination accepted or displayed a notification.

Canonical Files

  1. internal/notifications/notifications.go
  2. internal/notifications/queue.go
  3. internal/notifications/email_enhanced.go
  4. internal/notifications/webhook_enhanced.go
  5. internal/api/alerting/notifications.go
  6. frontend-modern/src/api/notifications.ts
  7. internal/operationaltrust/contracts.go
  8. internal/api/alerting/notification_queue.go
  9. internal/notifications/tag_routing.go
  10. internal/notifications/delivery_health.go
  11. internal/notifications/deadman_config.go
  12. internal/notifications/failure_class.go
  13. internal/notifications/quiet_hours_queue.go
  14. internal/notifications/apprise_diagnostics.go
  15. pkg/securityutil/pinned_dial.go

Shared Boundaries

  1. frontend-modern/src/api/notifications.ts shared with api-contracts: the notifications frontend client is both a notification delivery control surface and a canonical API payload contract boundary.
  2. internal/api/alerting/notification_queue.go shared with api-contracts: the notification queue and DLQ handler is both a notification delivery consequence surface and a canonical API payload boundary for operational transition links.
  3. internal/api/alerting/notifications.go shared with api-contracts: notification handlers are both a notification delivery control surface and a canonical API payload contract boundary.
  4. internal/operationaltrust/contracts.go shared with alerts: the operational trust contract is jointly consumed by canonical alert lifecycle ownership and notification delivery linkage without making delivery state operational truth.

Extension Points

  1. Add or change provider delivery, queue processing, or retry behavior through internal/notifications/
  2. Add or change notification-management request or response handling through internal/api/alerting/notifications.go
  3. Add or change notification-management frontend transport through frontend-modern/src/api/notifications.ts
  4. Add or change the delivery-health verdict through internal/notifications/delivery_health.go. ClassifyQueueHealth is the single rule for whether configured destinations are reaching anyone, and it lives beside the queue that produces the counts so consumers cannot each carry their own copy. Only terminal failed or dead-letter outcomes count as unhealthy; recoverable retry attempts do not. A queue that cannot be read reports unavailable and never healthy, because an unreadable queue must not be mistaken for successful delivery. internal/api/alerting/notifications.go delegates to this rule rather than reimplementing it, and monitoring consumes it to raise the notification-delivery system alert. The queue also emits one process-local health-changed callback after a relevant transition has committed and after its database lock is released. Monitoring installs that callback and owns the alert projection; notification code must not raise or clear alerts directly. Retry and dismissal reconcile even when no row changed, because an already-repaired queue may still have a persisted stale warning from an earlier process.
  5. Change approved-address connection scheduling through pkg/securityutil/pinned_dial.go, owned by security-privacy. Keep webhook allowlisting and provider-specific destination policy in this subsystem.

Forbidden Paths

  1. Reintroducing notification delivery behavior as implicit side effects under alerts or generic monitoring ownership
  2. Duplicating webhook or email delivery safety checks outside internal/notifications/
  3. Letting notification queue, DLQ, or provider test paths drift away from the explicit proof routes in registry.json

Completion Obligations

  1. Update this contract when canonical notification entry points move
  2. Keep notification API transport and backend delivery proofs aligned in registry.json
  3. Preserve explicit queue, webhook-security, and provider-delivery coverage when notification behavior changes

The persistent queue and audit log carry typed NotificationLink entries for every linked alert in a grouped delivery. A link keeps one notification id, operational record id, lifecycle transition id, cause key, destination id, and delivery state across queue retries. Queue state is delivery evidence only: it must never create, resolve, acknowledge, suppress, or count operational records. Resolution notifications link to the recovery-evidence transition. Partial cancellation of a grouped firing delivery removes only the links for the resolved alerts, while retry, failure, cancellation, and dead-letter state remain inspectable in the queue and audit records. Destination identities are stable opaque routing identities and must not expose credentials.

  1. Webhook URLs are credential-bearing (Gotify, ntfy and Telegram all carry tokens in the path or query), so every surface that emits one must pass it through RedactWebhookURLSecrets first. This covers logs and returned errors alike, including the rate-limit drop paths in checkWebhookRateLimit and the enhanced sender, which are the sites most likely to fire repeatedly for a misconfigured destination. Redaction is proven by capturing log output rather than by reading the call sites, because a missed site is invisible to a source scan. Regression coverage: TestWebhookRateLimitLogsRedactURLSecrets and TestRedactWebhookTransportErrorPreservesBehaviorWithoutToken in internal/notifications/webhook_url_redaction_test.go.

Current State

Masked destination edits preserve each submitted value independently

Webhook editing restores each submitted ***REDACTED*** header/custom-field value from that destination, rather than replacing the entire submitted map. Explicit changes, additions, empty values and removals beside another masked credential survive live publication and encrypted reload. Header names match case-insensitively; custom fields retain exact, canonical service-specific names. An unknown, already-masked or ambiguous saved value fails with HTTP 400 before any save or live mutation. Omitted maps retain their existing replacement semantics. A response must not disclose a stored custom-field value restored from a mask; it echoes the canonical submitted value instead.

TestWebhookEditPreservesOnlyMaskedValues and TestWebhookEditRejectsUnrecoverableMasks pin editing, deletion, blank values, case matching, no-mutation rejection and confidentiality. TestWebhookMaskedEditPersistsAndDelivers connects masked form reads/edits, encrypted save/reopen, ordinary grouped firing/recovery and disabling an active group to individual deliveries through the persistent sender. Built-in Telegram and custom Generic bodies retain all members, exact headers/fields and ordinary per-occurrence recovery receipts. This is loopback source proof, not native Telegram acceptance or a reproduction of the reporter's actual stored settings. Already admitted queue jobs keep their configuration snapshots and current pause/disable/removal checks; editing does not rewrite or replay them. The built-in template-header correction remains separate and unchanged.

Apprise diagnostic confidentiality

Apprise CLI targets may carry credentials in arbitrary provider schemes. CLI output and HTTP response/error text may echo target credentials, API keys, configuration keys or private alert content. Firing, recovery, test and queued delivery must not copy those bytes into logs, returned errors, attempt audits, dead letters or delivery-log projections. Configuration-update logs use only normalised mode, counts, presence flags, timeout and TLS-policy flags. Authorised configuration reads/edits and private admitted queue configuration are unchanged.

apprise_diagnostics.go retains structured failure classes and typed causes for errors.Is/errors.As, but formats only fixed safe reasons or a numeric CLI exit. HTTP status remains the authoritative retry verdict; response bodies are drained only to the existing size limit and discarded. CLI diagnostics retain byte and target counts, not output or target values. Shared URL validation and redirect checks use an opaque Apprise diagnostic label, including allowlist debug logs; ordinary webhook diagnostics retain their existing redaction. Endpoint, mounted path, escaped configuration key, target arguments, authentication headers, payloads, TLS verification, SSRF checks, retry budget and delivery state do not change. Existing historical logs/audits are not rewritten or claimed cleared.

Verification: apprise_confidentiality_test.go captures actual CLI/loopback HTTP logs and errors across firing, recovery and test sends; verifies unchanged arguments/requests; exercises malformed URLs, permitted and refused redirects, typed transport failures and the real queue's retry/DLQ/audit/log projection. internal/api/alerting/notifications_test.go covers configuration logging and the real sender's test-error HTTP response. All secrets are synthetic; these controls do not establish installed Apprise acceptance or prior user exposure.

Current-policy quiet-hours replay

The persistent queue checks every due provider attempt against an immutable snapshot from alerts.Manager.QuietHoursNotificationPolicy, installed by NotificationManager.SetQuietHoursPolicyProvider before saved work is activated in the monitor constructor. The snapshot is obtained before queue/database or per-alert delivery locks; its evaluator does not call back into the alert owner. Stored replay timestamps are wake conditions, not evidence that the current schedule permits delivery. Full-day schedules recheck at the retained daily boundary, and late replay into another quiet interval remains held.

quiet_hours_queue.go reloads the pending row after acquiring delivery gates, preserving cancellations that rewrote a grouped row after worker discovery. An all-held batch moves only its wake time. A mixed batch atomically retains the held original and creates one ready child with the admitted destination bytes, occurrence/transition links, creation time and remaining attempt budget. Child IDs bind the parent and exact ready payload; held work does not gain an attempt, audit entry, firing receipt or false success. Prior attempt audits remain on the original batch, while actual child attempts carry their own linked receipts. Already-due recovery groups cannot reopen their grouping window while held. Partition persistence or ambiguous linkage failure leaves the original intact and makes no provider attempt. Due-time refresh prevents stale worker snapshots from repeatedly postponing or claiming a now-held row. Actual retries retain the existing claim/cancellation gates and provider enabled checks.

This does not accelerate a future stored deadline after a schedule edit; the current rule is applied at the next admitted wake. It does not change initial target selection, mutable recovery tag admission, acknowledgement semantics, operator retry budgets or at-least-once delivery after an interrupted send.

quiet_hours_queue_test.go proves continuous/late/DST replay holds, mixed immediate/held delivery, exact admitted destination and operational linkage, budget retention, transaction rollback, durable reopen, cancellation and stale concurrent snapshots, grouped recoveries and all six provider/event families. internal/monitoring/monitor_notification_startup_test.go exercises the actual constructor with due persisted work across two startups. These are local source and HTTP-receipt proofs, not installed provider or natural-day acceptance.

Persistent resolved-alert grouping

SendResolvedAlert applies the configured grouping window to recoveries as well as firings. Event types are never mixed. With the persistent queue available, internal/notifications/resolved_grouping.go persists the first recovery's window and coalesces only untouched, not-yet-due pending jobs with identical notification type, destination configuration and quiet-hours schedule. Queue scheduling metadata identifies these groups; it is not the provider HTTP method. Deadlines round up to queue timestamp precision, never shorten the grouping window, and do not slide with each added occurrence. Retry batches are immutable. No schema or driver modification is needed. Pending groups survive manager shutdown/reopen. The queue-unavailable fallback uses a separate in-memory buffer and does not promise restart durability.

Admission retains destination-specific successful firing receipts. A destination that never received an occurrence, or an unannounced alert, must not receive its recovery. Group extension deduplicates by alert ID and occurrence start time, retains each resolution timestamp and regenerates the group's operational links. Delivery consumes only receipts for the occurrences in the delivered job. Grouping disabled or a zero window keeps new recoveries individual; changing that policy does not rewrite already-admitted persistent jobs.

Generic payloads and built-in/custom recovery summary data retain all grouped alerts and the count, rather than rendering only the first alert. PagerDuty's built-in Events v2 resolve path remains one deduplication key per request; it is explicitly excluded from coalescing to avoid consuming other incidents' receipts. Existing rate limits, retry budgets, destination security checks and firing grouping are unchanged.

Verification: resolved_grouping_contract_test.go establishes ordinary firing receipts then resolves fifteen alerts in one process, after manager restart, and across restart with pending recoveries. Receiver timestamps must respect one second from the first admitted occurrence in each group. An emission burst completed within that window must stay one group; emission/restart work spanning the window is not itself evidence of early delivery. A deliberate late-arrival case checks that an expired first group is not extended. Across groups every occurrence must appear once, in order within its group, with its start time and resolution timestamp intact. The fixture does not change production deadlines, rate limits, or retries. It checks unseen-destination and unannounced-alert suppression, per-occurrence resolution metadata, eleven recovery renderers, individual PagerDuty keys, and the disabled-grouping burst's rate-limit/dead-letter consequence. The HTTP adapter is in-process; queue and receipt persistence are real. Focused and race runs establish this synthetic boundary, not a full notification-suite result, installed monitor recovery, provider acceptance, reporter confirmation or release qualification. The existing singleton delivery test explicitly disables grouping.

Webhook retry delay conversion

parseRetryAfterBackoff bounds parsed integer seconds before multiplying by time.Second, so large provider values cannot overflow into an immediate retry. Signed 64-bit integer parsing makes this independent of native integer width. Positive parsed seconds above the existing WebhookMaxBackoff cap return that cap (30 seconds); non-positive parsed values retain the existing immediate fallback. HTTP-date parsing and invalid-value fallback are unchanged; integers outside signed 64-bit range remain invalid. Negative-value acceptance is compatibility behaviour, not the non-negative delay-seconds grammar of RFC 9110 section 10.2.3.

TestParseRetryAfterBackoff in internal/notifications/webhook_enhanced_test.go pins both large positive and negative overflow cases alongside ordinary seconds, dates, whitespace and invalid inputs. This is pure parser proof: it does not establish elapsed HTTP retry timing, queue persistence, provider acceptance or recipient receipt.

For HTTP 429 and 503 transport retries, a valid Retry-After overrides only the wait following that response. It must not replace the independent exponential backoff schedule. In particular, a zero-delay response cannot cause later headerless 429 or 503 failures to exhaust their remaining retries immediately. The schedule continues doubling up to WebhookMaxBackoff; retry budgets, classification, parsing and the existing header-delay cap remain unchanged. 503 recovery hints use the same bounded parser as 429 rate-limit hints, as described by RFC 9110 section 10.2.3. Invalid or absent hints retain exponential backoff; other status codes do not gain header-based delay handling.

TestSendWebhookWithRetry_ServiceUnavailableRetryAfter in internal/notifications/webhook_enhanced_test.go verifies that a 503 carrying Retry-After: 2 delays the next HTTP request by at least two seconds, rather than the initial one-second backoff, and that a malformed hint retains that initial backoff. It uses an owned loopback destination, not a live provider.

TestSendWebhookWithRetry_ZeroRetryAfterPreservesLaterBackoff in internal/notifications/webhook_enhanced_test.go sends a zero-delay 429 or 503, then a headerless 429 or 503, then success through a local HTTP fixture. It verifies three requests and the two-second exponential wait before the final request. This is transport timing proof, not durable queue retry, installed provider acceptance or recipient receipt.

Webhook transport retry exhaustion

The enhanced retry sender makes one initial HTTP attempt plus RetryCount retries; non-positive counts select WebhookDefaultRetries. Repeated valid zero-delay 429 hints do not reset that budget. When all attempts fail, the sender returns the total attempt count and records one failed webhook history entry with the final status and the number of retries, not one entry per HTTP attempt. Every attempt retains the event ID and payload. A queue delivery that uses this sender has a transport ceiling of effective retry count plus one; layering up to MaxAttempts queue deliveries multiplies that ceiling, but does not guarantee destination receipt. Permanent errors may stop retries earlier.

TestSendWebhookWithRetry_RateLimitExhaustion in internal/notifications/webhook_enhanced_test.go verifies configured counts 1 and 2 and default selection for 0 and -1 against an owned HTTP fixture that always returns 429 with Retry-After: 0. It checks request identity and body, attempt exhaustion, and the single failed history record's status, retry count, payload size and error text. This is transport-budget and local history proof only, not queue persistence, provider receipt or installed qualification.

Notification-management HTTP production and its unit/contract proof now live together under internal/api/alerting/. Router-level scope and integration tests remain in internal/api, while the compatibility aliases there keep the existing extension surface stable. This gives notification qualification a native Go package scheduling boundary without changing routes or payloads.

This subsystem now makes email, webhook, Apprise, queueing, and delivery safety explicit inside the current architecture lane instead of leaving them implied by the broader alerts surface. A later lane split can still promote alerts and notification delivery into their own product lane once the governed floor is ready.

internal/notifications/ is the live delivery engine. It owns provider selection, secure webhook transport, Apprise delivery, queue persistence, DLQ handling, retry policy, and delivery observability for alert-driven notifications. Queue dequeue, direct-delivery fallback, and enqueue-failure recovery now run through one owned notification-delivery executor instead of keeping separate primary send paths. Single-alert, grouped, and resolved webhook delivery now also share one owned rendering path for URL rendering, service-specific enrichment, and template selection. When persistent queue bootstrap fails, the runtime now falls back to an in-memory queue owner instead of dropping into a separate nil-queue direct-send mode. Service-specific webhook compatibility like Pushover app_token / user_token legacy fields is now canonicalized at webhook-config ownership boundaries, so runtime delivery only handles canonical token / user fields instead of injecting aliases mid-flight. That boundary includes config persistence plus API/UI ingress for create, update, and ad hoc test requests; internal/notifications/ may not silently rewrite those legacy keys once webhook state is already live in the runtime.

The webhook template registry is also the canonical source of truth for the alert webhook service set and the metadata the frontend uses to build its service chooser. Frontend presentation may format the options, but it must derive the available services, labels, and descriptions from the backend template registry instead of keeping a second hardcoded service list. Mention field visibility plus mention-placeholder/help copy for supported services must also come from the same backend registry so the editor does not carry a second service-specific presentation map. That same template-registry boundary owns JSON-safe string rendering for the built-in webhook providers. Canonical JSON templates must render runtime strings through the shared notification template helper that JSON-escapes quoted, multi-line, and path-like alert content before validation, instead of injecting raw alert fields directly into JSON bodies. Custom user templates may still choose their own formatting, but the shipped provider templates may not rely on callers to pre-sanitize alert text or resource names just to keep their JSON payloads valid. Teams Adaptive Card templates are part of that same built-in provider boundary: resolved alert titles and resource-name text must pass through the JSON string helper before entering Adaptive TextBlock.text, so a resource name containing quotes or backslashes cannot invalidate the webhook payload. Email single-alert, grouped, resolved, and HTML send paths must follow that same ownership rule: they may expose different calling surfaces, but they must all route through one canonical enhanced email executor instead of rebuilding separate manager/config setup paths. That enhanced email executor owns the production-manager reuse boundary as well as the transport send itself. A test or ad hoc send whose SMTP host, port, username, password, TLS, STARTTLS, or provider differs from the shared manager must build an isolated delivery manager and leave the production manager untouched, so unsaved relay-mode tests cannot inherit stale saved SMTP auth. When the transport identity matches, the shared manager may still update From/To and rate-limit presentation state so grouped and resolved sends keep their persistent limiter continuity. Notification test APIs must follow that same truthfulness rule: test email and webhook paths may keep dedicated top-level entry points, but they must route through canonical error-returning single-delivery executors instead of fire-and-forget wrappers that can report false success. Webhook test service/template synthesis must also stay inside internal/notifications/: API handlers may decode the request and delegate, but service-template selection, safe header copying, and generic test-template fallback may not live as a parallel owner path under internal/api/. Saved-webhook test actions, generic webhook test actions, and ad hoc webhook test actions must also share the same enhanced webhook test executor so service-template ownership, header normalization, and validation cannot drift by entry point. Enhanced webhook test/live delivery must follow that same ownership model: webhook_enhanced.go may expose a richer config shape, but it must bridge back into the canonical webhook render and transport path instead of maintaining a parallel URL-rendering, enrichment, Telegram URL sanitization, or single-send HTTP stack. That same webhook payload boundary also owns tenant identity stamping. Webhook payload data carries the tenant ID and display name of the runtime or organization that fired the alert: environment-provided identity (PULSE_TENANT_ID / PULSE_TENANT_NAME) is the construction-time default for isolated client runtimes, the display name falls back to the ID, and shared-process multi-tenant deployments override it through an org-backed resolver installed by the monitoring subsystem. Templates consume it as {{.TenantID}} / {{.TenantName}}, and the canonical generic template emits its tenant block only when an identity is present so single-tenant payloads keep their existing shape. PSA/ticket-bridge receivers must get tenant routing identity from this payload boundary, not by inferring it from webhook endpoint configuration. That payload boundary also owns the language-neutral MessageKey exposed to custom templates. Canonical alerts derive it from canonicalAlertKind and the alert type (for example metric-threshold.disk); legacy alert paths use the alert type directly. The generic JSON templates must emit the message key, event, resource type, node display name and formatted metric values explicitly so external receivers can translate or reconstruct notifications without parsing English message text or reaching into the metadata map. That same transport boundary also owns outbound delivery integrity. A webhook config may carry an optional signing secret; when present, every JSON delivery through the canonical webhook transport must send X-Pulse-Timestamp and X-Pulse-Signature (v1= + hex HMAC-SHA256 over timestamp + "." + body), computed at the single request-construction choke point and set after custom headers so user-provided header maps cannot shadow them. Alert deliveries also send X-Pulse-Event-ID (alertID:event) as a legacy correlation token; it must be stable across both transport-layer and queue-layer retries of the same alert occurrence. It omits occurrence start, severity and grouped membership and is not a unique incident or delivery identity; it is not covered by the HMAC. Receiver guidance must not prescribe permanent header-only deduplication. Normal queued templates expose every member through .Alerts, whose full- precision StartTime can distinguish later occurrences even when the primary RFC3339 start string has the same whole second. Correlate recovery with that occurrence and preserve severity updates and changed members. The existing wire format and retry stability remain unchanged. webhook_guide_test.go exercises copied templates through the actual signed sender over loopback for retry, escalation, recovery, recurrence, group changes and JSON escaping; it does not establish durable ticket actions or installed notification delivery. The management API must mask a configured signing secret on read and preserve the stored secret when an update echoes the masked placeholder, the same ownership rule it applies to header and custom-field secrets. That same transport boundary also owns webhook request normalization. Rendered webhook URLs must reject userinfo during validation, and request construction must route through a validated absolute URL object instead of reparsing raw URL strings at send time. The same SSRF guard must reject unspecified direct targets such as 0.0.0.0 and :: before delivery and must not allow webhook private-CIDR allowlists to include networks that contain those unspecified addresses. The DNS-rebinding pin in the secure webhook dialer is part of that same guard: hostname dials may connect only to resolver-validated IPs. The shared pinned dialer uses bounded staggered fallback rather than letting a stalled first connect consume the request deadline. Resolver family preference and order within each family survive, while alternate families and later same-family addresses get connection budget. DNS lookups retain cancellation and their five-second ceiling for admission, redirects and the transport itself. That same ownership includes webhook retry classification. The canonical retry gate in webhook_enhanced.go must parse provider failures from both status 429-style and HTTP 429-style error strings before it decides whether to retry, so a non-retryable HTTP 400 result cannot be retried just because the transport changed its error wording. Explicit HTTP status also wins over network-like diagnostic text in the response body: a terminal 403 mentioning "authentication timeout" must stop transport retries, retain the 403 in delivery history, and record zero retries when rejected on the first attempt. The existing transient HTTP exceptions and retryable fallback for network or unclassified failures remain unchanged. TestIsRetryableWebhookError_StatusOverridesBody and TestWebhookRetryRejectsForbiddenTimeoutBody in internal/notifications/webhook_retry_test.go pin this precedence with a status/body matrix and a queue-free loopback receiver that checks request count and delivery history. These are synthetic transport proofs, not installed recipient receipts or a change to persistent-queue retry policy. That same notification transport boundary also owns outbound Apprise HTTP URL normalization. Server URLs must be validated as absolute HTTP(S) endpoints without userinfo before request construction, and the /notify plus optional config-key path must append to the canonical server base instead of being rebuilt through raw string concatenation in local delivery code. That same Apprise boundary also treats ServerURL as a canonical base URL, not a request URL template. The owned runtime must reject query or fragment state on that base, preserve any mounted base path, and resolve /notify plus the optional config-key segment through shared URL helpers so delivery cannot silently drop subpaths or reinterpret appended path segments. That same delivery boundary also owns SMTP mailbox normalization. From, recipient, and Reply-To inputs must be parsed as canonical mailboxes before headers or SMTP envelope commands are constructed, so notification delivery cannot treat raw config strings as header fragments or RCPT TO input. That same SMTP boundary also owns MIME-safe body construction. Text and HTML payloads must be emitted through canonical multipart writers with encoded body parts instead of being concatenated directly into handcrafted message bodies. That same SMTP boundary also owns email threading identity. An email that covers exactly one alert occurrence must carry In-Reply-To and References headers set to a deterministic incident thread ID derived from the alert ID plus firing start time, so mail clients thread the firing, re-notification, and resolved messages of one incident together. The per-send Message-ID must stay unique: re-notified incidents emit multiple emails, and providers that de-duplicate on Message-ID would silently drop repeats, so incident identity may ride only in the threading headers. Grouped emails covering multiple alerts must not carry a group-level thread identity, because firing and resolved batches are not guaranteed to contain the same alert set and a group-level reference would attach messages to the wrong thread. Scheduled report delivery uses that same enhanced email boundary. Report attachments must be emitted as MIME attachment parts by internal/notifications/email_enhanced.go. The scheduler owns report-size validation and the operator's attachment choice. SMTP transport, recipient parsing, headers, body encoding, and attachment encoding remain notification-owned. EnhancedEmailManager.ValidateConfiguration checks the same sender/recipient and provider credential inputs without sending mail. A connection failure while awaiting the completed-message reply is a typed EmailAcceptanceUnconfirmedError. The sender stops its inner retry loop because the server may already have the message. Explicit SMTP replies still use the canonical failure classifier. internal/notifications/email_delivery_test.go checks configuration and a dropped completed-message response with an attachment. Report scheduling consumes that typed uncertainty rather than inferring rejection from a generic connection error. This does not assert inbox receipt or change the persistent alert queue's state vocabulary. That same queue ownership also governs persistent queue storage roots. The notifications queue database must normalize its owned data directory and resolve the fixed notification_queue.db leaf through the shared storage-path helper instead of joining raw caller-provided directory strings. The queue database takes its SQLite page cache from pdb.PageCachePragma in pkg/db/pagecache.go rather than a private cache_size. The modernc driver holds that cache outside the Go heap at about twice its size, where neither the garbage collector nor GOMEMLIMIT reaches it (#1803). TestNotificationQueueUsesSharedPageCacheBound pins the effective value. That same queue owner also governs alert-resolution cancellation policy. Cancelling queued work by alert identifier must remove outstanding firing deliveries for that alert, but it must preserve already-queued resolved notifications so recovery deliveries cannot be dropped just because the alert was resolved before the queue drained. Cancellation must run even when there is no in-memory grouped notification pending, because the persistent queue and the delivery cooldown map are also notification-owned state for the alert occurrence. That same queue boundary also owns processor attachment semantics. The canonical queue may persist pending notifications before a delivery processor is configured, but it must not mark those entries sending, failed, or sent until a processor exists. An explicit processor attachment must wake the pending backlog through the same canonical batch path instead of relying on a separate direct-send shortcut or waiting for an unrelated timer tick.

The monitor constructs its manager before applying saved destinations (webhooks, email, Apprise). It must defer processor installation until those load attempts finish: suppressing only the initial wake would still let the five-second queue tick consume an already-due job against a still-empty destination list and terminally cancel it as ErrNotificationDeliverySkipped. Explicit activation then wakes the pending backlog through the canonical queue path. Ordinary manager constructors retain autonomous processing for callers without a saved configuration bootstrap. TestRestartKeepsDueDeliveryPendingUntilConfigured in resolved_grouping_contract_test.go forces a pre-configuration batch, proves the due row stays pending, then installs the webhook and proves that activation delivers it. This preserves disabled-destination policy at processing time without letting startup timing decide it. Alert delivery cooldown is also owned at this boundary. Normal alert delivery must suppress duplicate sends for the same active alert occurrence when cooldown is disabled or still active. A manager-admitted increase above the highest delivered severity for that occurrence is new information and must pass this gate, while retaining normal destination selection and grouping. Late lower-severity delivery receipts must not lower that high-water mark and permit repeat critical sends. A new occurrence has its own severity history. This exception does not bypass manager acknowledgement, snooze, flapping or hourly budget admission, nor destination routing or rate limits. Scheduled escalation retains its dedicated send path so the alert schedule, not transport cooldown, controls escalation cadence and channel targeting. TestMonitorSeverityDelivery covers observation-to-HTTP delivery and the persisted delivery log for custom sensors, CPU metrics and ZFS health; callback success alone is not delivery evidence. The grouping timer is also notification-owned delivery state. Live alert configuration must apply enabled, window, node, and guest grouping fields as one policy update. Turning grouping off must stop the timer and deliver every already-pending alert separately; service templates must be rendered only after the grouped summary contains every alert, so provider-specific payloads cannot silently collapse to the first alert.

internal/api/alerting/notifications.go and frontend-modern/src/api/notifications.ts are shared boundaries with api-contracts: they are the product-facing control surface for notification-management transport, while canonical payload-shape governance still remains explicit in the shared API contract boundary.

Operational Trust delivery observability

Notification queue rows retain the exact operational record and lifecycle transition IDs through grouping, retry, restart, send, cancellation, and dead letter. Restart during a queued retry must reopen the same durable delivery with the same transition links; it must not synthesize a new operational transition. internal/notifications/queue.go records bounded queue, retry/sent/failed/dead-letter/cancelled outcomes and open-to-enqueue latency without destination, record, resource, or evidence labels. Delivery state remains notification truth only and cannot resolve or reopen the alert lifecycle.

The queue owner may expose a read-only, content-free telemetry aggregate over its retained per-attempt audit rows: total delivery attempts, including retries, plus successful deliveries and terminal failed/dead-letter outcomes since a caller-supplied cutoff. A failed attempt that is returned to pending for retry is not a terminal delivery failure. The aggregate must be computed in storage, must not return or copy notification IDs, alert links, destinations, recipients, endpoint URLs, titles, bodies, error text, or timestamps, and must not mutate queue or alert state. The install-wide monitoring owner may sum that aggregate across provisioned tenants for the outbound seven-day notification outcome counters. Because completed queue rows are retention-bounded, this is a seven-day delivery signal only and must not be presented as lifetime delivery history or as proof that an alert was resolved.

Queue health is retention-bounded delivery truth. GetQueueStats counts every row still retained by the queue: sent, failed, and cancelled rows for seven days, dead-letter rows for 30 days, and nonterminal rows until they complete. The notification health API must report degraded when any retained failed or dlq row exists and unavailable, never healthy, when those counts cannot be read. Pending retries do not degrade health. The response must expose fixed reason codes and retention metadata rather than raw queue errors or notification content.

Operator recovery is owned by the queue, not by database-file deletion. The settings-write retry action returns all retained failed and dlq rows to pending, resets their queue-attempt counters to a fresh retry budget, keeps their operational links, and wakes the processor; it never rewrites existing per-attempt audit rows. The settings-write dismiss action transitions those same rows to cancelled, preserves both queue and audit history, and clears the active health warning. Both actions are transactional across the selected terminal set and report the number of rows actually transitioned.

User-facing delivery log and honest test sends

The queue owner exposes its retained per-attempt audit rows to the local notification-management API as a bounded, newest-first delivery log (GetDeliveryLog in internal/notifications/delivery_log.go). Each entry carries the attempt outcome (sent, retry, failed, dead_letter, cancelled), derived by the same single rule that feeds the delivery outcome metric in RecordAudit, plus alert identifiers, normalized destination identity, attempt count, failure class, error text, and timestamp. Audit rows persist destination identity in a dedicated destination_id column; rows written before that column existed resolve destination identity from their retained operational links. GET /api/notifications/delivery-log (settings-read scope) serves the log as local operator evidence: webhook secrets are redacted from error text at the API boundary, the payload names its retention windows instead of presenting itself as lifetime history, and an unreadable queue is an error, never an empty log. The API queries the longest retained class (30 days) and reports completed_retention_days: 7 plus dead_letter_retention_days: 30, because completed audit rows are removed with their seven-day queue rows while dead-letter audit rows remain with their 30-day queue rows. Reads default to 50 entries and are capped at 200; the operator evidence view requests that bounded maximum. This per-attempt surface is deliberately distinct from the content-free telemetry aggregate above, which remains identity-free. Test sends bypass the queue and must not appear in the delivery log, and the destinations UI says so where the log renders. The frontend client normalizes the entries collection through the shared API collection helper before validating each record, so a malformed collection or row cannot bypass the canonical client boundary or create a module-local fallback shape.

Because test sends also bypass the alert activation gate, a bare success result is exactly how installs come to believe delivery works while every real alert is suppressed. Successful responses from POST /api/notifications/test and POST /api/notifications/webhooks/test must therefore report deliveryPaused: true whenever the notification manager is disabled, and the destinations UI must surface that as a warning instead of a plain success toast.

A webhook test must also send the headers a real delivery sends. When a built-in service has no custom template, Pulse renders that service's body, so the service template's static headers are part of the wire format and override stored copies under any spelling of the same name. prepareWebhookDeliveryContext applies them to every firing delivery and to every recovery except ntfy's, and BuildEnhancedWebhookTestConfig uses the same withWebhookTemplateHeaders rule; other stored headers stay. The stored headers are only a copy the UI seeded at creation, and an edit can blank or replace them. Before this rule, a Telegram webhook with a blank stored Content-Type passed Test while Telegram ignored every real alert's JSON body and answered "message text is empty" (#2540). A custom template means the user owns the body, so stored headers stand. Generic webhooks have no built-in body and keep their stored headers. ntfy recoveries build their own plain-text request and are outside this rule. internal/notifications/webhook_template_headers_test.go drives a real queued Telegram alert with damaged stored Content-Type values, and compares Test, grouped and resolved requests for every built-in service.

NormalizeWebhookConfig drops header entries with a blank name or value at every ingress and on load, so a header cleared in the editor is removed rather than sent empty. An empty Content-Type would otherwise replace the default application/json on generic webhooks, which have no template headers to restore it. The webhook API resolves the editor's masked values per key before normalization, so the ***REDACTED*** mask is never stored or sent (see the api-contracts webhook clause). TestGenericWebhookBlankStoredHeaderKeepsDefaultContentType pins the delivery side.

Occurrence-bound delivery receipts

The notification owner records successful firing delivery by exact alert ID, nanosecond start time, and a normalized destination identity. Email recipients, webhook ID plus URL, and Apprise mode/base/config/targets are part of that identity, so a later occurrence or reconfigured destination cannot inherit an older receipt. A resolved notification is eligible only for destinations that received that exact firing occurrence. Successful recovery delivery deletes the receipt; persistent receipts survive restart and are retention-bounded. Receipt read failure suppresses recovery rather than inventing prior delivery.

Destination alert routing

Email and every enabled alert webhook may define a normalized resource-tag filter and an all or any match mode. Empty filters preserve global delivery. Matching is exact after trimming and case folding, and grouped firing alerts are filtered independently for each destination; Apprise remains global by resource tag. Tags may arrive through the canonical tags, resourceTags, hostTags, or serviceTags alert metadata keys.

Email, every enabled alert webhook, and Apprise may also define a normalized minimum severity of all, warning, or critical. Omitted or unknown values preserve backwards-compatible all-alert delivery. Grouped firing alerts are filtered independently per destination, after both tag and severity policy are applied, so one incident batch may produce different destination payloads. The meanings are exact: all accepts info, warning, and critical; warning accepts warning and critical but excludes info; critical accepts only critical. API normalization, frontend editing, and persisted configuration must round-trip the warning floor rather than reducing it to all-alert delivery.

The delivered payload must preserve the selected alerts' canonical severity. Single and grouped email subjects, summary counts, alert rows, and plain-text fallbacks represent Info separately from Warning. ntfy derives the highest severity in a group without promoting an all-info group: informational delivery uses the INFO title prefix, default priority, and information_source tag; warning and critical retain their higher attention postures. Destination filtering is not complete if the final template relabels or visually promotes the alert.

Resolved delivery deliberately bypasses current tag and severity matching and is filtered by occurrence-bound delivery receipts instead. This guarantees that a destination which received a firing occurrence can receive its recovery even if resource tags or destination policy change. Configuration copies must isolate filter slices, mixed-version updates must preserve omitted routing fields, and explicit empty filters must remain the supported clear operation.

Cooldown state is published only after firing receipts are recorded. Persistent workers atomically claim a still-pending row after taking per-alert delivery gates, and resolution cancellation takes the corresponding exclusive gates. This prevents a stale pending snapshot from being sent after cancellation while preserving grouped-row and operational-link lifecycle semantics. A destination with delivery disabled or failed has no receipt and receives no misleading recovery. These rules remain delivery truth only and cannot resolve the alerts-owned lifecycle.

internal/notifications/delivery_receipts_test.go, internal/notifications/notifications_test.go, and internal/notifications/queue_test.go prove occurrence and destination isolation, persistence, cleanup, and cancellation/claim ordering.

External watchdog transport is a credential-bearing destination boundary

internal/notifications/deadman_config.go owns normalization and validation for healthchecks-compatible success-ping URLs. Possession of the URL can forge healthy state, so configuration is encrypted at rest, exported only inside the passphrase-encrypted configuration bundle, and represented to API clients by a redacted sentinel. URLs are bounded to HTTP(S), exclude userinfo, fragments, localhost, loopback, unspecified, and link-local targets, and must name the base success endpoint rather than /start, /fail, or /log. Endpoint suffix validation inspects the once-decoded URL path, including encoded separators and an encoded trailing slash, so percent-encoded event names cannot bypass this success-only guard. Encoded base tokens and suffix words in query values remain valid; the configured URL is not rewritten. TestValidateDeadManPingURL covers these rejected and accepted forms. TestDeadManRunCycleRejectsEncodedNonSuccessEndpoint proves rejection before transport, a misconfigured state without a success timestamp, and no token in the diagnostic. These are local validation/runtime proofs, not installed watchdog acceptance or recipient delivery evidence. Literal addresses are also compared against every Pulse host interface, while the monitoring dialer repeats that comparison after DNS resolution. Failure to enumerate local interfaces fails closed. A private LAN watchdog remains valid only when its resolved address belongs to a different machine; pointing a hostname or normal LAN address back at Pulse is rejected as same-host.

The monitoring-owned sender uses a dedicated transport that bypasses ambient HTTP proxies, revalidates DNS answers at dial time, never follows redirects, and returns sanitized error classes that cannot disclose the URL or token. Private LAN watchdogs remain valid when separately hosted. Network and 5xx failures receive two bounded retries; permanent response failures do not. A healthy signal is GET, while canonical-loop stall and restart-gap diagnostics are bounded text POSTs containing Pulse health and UTC timing only—never alert content, infrastructure names, destination credentials, or tenant data. This watchdog path is deliberately independent of notification queue activation, quiet hours, grouping, and escalation routing.

Destination configuration publishes only committed state

Email, Apprise, and webhook create, update, and delete handlers treat encrypted configuration persistence as the publication boundary. They serialize destination writes, build complete webhook candidate inventories, persist the candidate first, and only then update the live notification manager. A failed write returns 500 and leaves the live destination inventory unchanged, so an operator cannot receive success for routing state that a restart would undo or have queued delivery cancelled by an uncommitted disable action. The forced failure paths are pinned in internal/api/alerting/notifications_test.go; the API ordering proof lives in internal/api/contract_test.go.

Escalation delivery can select exact logical destinations

Escalation dispatch accepts optional logical destination IDs in addition to the legacy channel target. Exact selection is applied before destination tag and severity policy: email selects the enabled email job, apprise selects the enabled Apprise job, and webhook:<id> selects only that enabled webhook. Unknown or unavailable IDs produce no job and never fall back to a broader channel. A no-job exact selection also leaves notification cooldown state untouched so configuration drift cannot masquerade as successful paging.

Destination IDs never contain endpoint URLs, credentials, or Apprise targets. All retry, queue, receipt, grouping, and recovery semantics remain owned by the selected notification job after routing.

Terminal destination failures distinguish rejection from server failure

The closed notification failure-class contract uses rejected for newly recorded destination HTTP 4xx responses after the dedicated authentication and rate-limit classes, and server_error for newly recorded HTTP 5xx responses. Explicit HTTP 5xx status evidence is classified before generic timeout wording so a 504 remains a destination server failure rather than connectivity. Pre-upgrade audit rows retain their original class until retention removes them. Both classes are recorded only on local audit rows and exported, when telemetry is enabled, as identity-free aggregate terminal failure counts; raw response content and destination identity never leave Pulse.

internal/notifications/queue_test.go pins the local classification order and the terminal-only retry/dead-letter accounting boundary.

Delivery failure class is declared by the sender, never read from a response

The failure class is authoritative where the sender knows it and heuristic only where nobody does. internal/notifications/failure_class.go owns that order: a class declared by the sender through NotificationFailureError wins, then Go's own error types decide (*textproto.Error carries the SMTP reply code, x509 and TLS errors mean tls, while DNS errors, timeouts, failed network operations and propagated EOF/truncation mean connectivity), and only then is the error message text consulted. A connection closing during AUTH or TLS does not prove a credential or certificate verdict. The structured SMTP/TLS verdicts still outrank generic network evidence, and opaque failures remain unknown when no declared or structured evidence identifies them. Explicit EmailAcceptanceUnconfirmedError also remains unknown: loss of the final DATA acknowledgement must not be presented as a definitive delivery or rejection, nor as an ordinary pre-acceptance connection failure.

Two rules follow and are not optional. A destination's response body must never determine its own failure class: every site that builds an error from an HTTP status classifies from the status code, so arbitrary body text is withheld and cannot make a 500 that says "rate limit" record as rate_limited. And an SMTP reply code is not read like an HTTP status: a 5xx reply is the destination refusing the message (rejected, or authentication for 530/534/535/538 and configuration for the 500-504 syntax replies), while only the transient 4xx replies are server_error.

RecordAuditError is the canonical audit entry point because it is the one that preserves a declared class; RecordAudit re-derives from text and is retained only for callers that never had the error value.

internal/notifications/failure_class_test.go pins the precedence order, the SMTP reply-code mapping, and the rule that response-body text cannot steer the recorded class.

The retry ladder is gated on the failure class

Delivery retries are for conditions that can clear. authentication, configuration, and rejected are verdicts about the request itself: the same payload, sent again to the same destination with the same credentials, gets the same answer. Those three dead-letter on the attempt that produced them, without consuming the remaining ladder. connectivity, rate_limited, server_error, tls, and an unclassified failure keep the full ladder, because nothing about them proves a later attempt fails. TLS is deliberately on the retrying side: a handshake can fail transiently during rotation, and the cost of one wasted ladder is lower than the cost of dropping a recoverable notification.

Payload template parse, execution and JSON-validation failures are declared configuration failures at the renderer. Ordinary and enhanced delivery, including plain-text ntfy, share that renderer; ntfy retains plain-text payloads without JSON validation. Malformed retained email, webhook and Apprise config, and unsupported retained notification types, are also configuration failures. These local failures stop on the first attempt rather than repeatedly trying a job that cannot be rendered or decoded. Diagnostics retain a fixed stage message and the wrapped cause for errors.Is/As, without copying configured template text, rendered payloads or private configuration into logs and audits. Retained configuration snapshots and explicit operator retry authority remain unchanged. This does not rewrite historical audit classes or private jobs.

NotificationFailureClass.Retryable is the single owner of that split. Neither the queue nor any destination type may keep its own list. This generalises to every destination the decision webhook delivery already made for HTTP 4xx in isRetryableWebhookError.

The SMTP transport's inner retry loop must apply that same classifier after an unsuccessful send, before sleeping or attempting another connection. Permanent authentication, configuration, and rejection failures return on that attempt; transient failures retain up to MaxRetries + 1 transport attempts. The returned error preserves the structured cause and reports the actual attempt count, not the configured maximum. This applies equally to ordinary, threaded, and attachment email through sendEmailWithOptions.

internal/notifications/email_retry_class_test.go exercises the real sender with in-memory SMTP handshake failures: permanent 550/535/554/501 replies stop at one attempt, while transient 421 retains three configured attempts even when its prose mentions authentication. It checks classification and the reported attempt count. This is transport-only proof, not queue persistence, post-DATA acceptance, or an installed recipient receipt.

Dead-lettering early must not lose the notification: RetryTerminalFailures remains the operator's recovery path, returning eligible retained terminal failures to the queue with a fresh budget once the credentials or configuration are fixed. Resolution removes obsolete firing entries from that eligibility; retry must not resurrect an incident which has already recovered. A dead-letter row records failureClass and a deadLetterReason of failure_class_not_retryable or max_retries_exhausted so the two are distinguishable in local logs.

internal/notifications/failure_class_test.go pins the retryable split and that a deterministic failure dead-letters on its first attempt.

Resolution remains final across terminal retries and restart

Resolution cancellation covers pending, sending, failed, and dead-lettered firing rows. A wholly obsolete row becomes cancelled; a grouped row retains only unrelated firing alerts and their operational links. Recovery jobs are not cancelled by this operation. Only removed pending entries contribute to the pending-suppression return count; terminal or interrupted sends must not be counted as proof that firing was never delivered.

The cancellation verdict persists across queue reopen and bulk operator retry. Per-item retry also rejects cancelled and already-sent rows atomically, so a stale retry request cannot bypass resolution or duplicate a completed delivery. Pending, sending, failed, and dead-lettered rows remain eligible for the existing retry scheduler.

Cancelling a row retains its failed-attempt audit history and announces the changed queue-health verdict only after releasing both the database mutex and per-alert delivery gates. Clearing obsolete retained failures does not prove that a destination has been repaired. Nor does this operation retrospectively identify obsolete rows whose resolution happened before this behaviour was installed; historical backlogs still require incident reconciliation.

internal/notifications/queue_resolution_retry_test.go proves failed and dead-lettered cancellation across durable reopen and actual queue processing, preservation of unrelated grouped firing and recovery jobs, retained failed attempts, callback lock release and committed-state visibility, and the per-item retry eligibility matrix. These are component proofs, not installed receiver receipts or exactly-once delivery guarantees.

Disabled delivery is cancellation, not a receipt

At processing time, globally disabled delivery or a disabled/removed destination returns ErrNotificationDeliverySkipped. The queue persists that job as cancelled with the policy reason and cancelled operational links. It does not write a provider-attempt audit, a successful receipt, or a delivery failure, and operator retry does not replay the cancelled job. Existing attempt history is retained. Queue health reconciliation runs after releasing the database mutex and alert delivery gates.

queue_disabled_delivery_test.go exercises the real manager/queue boundary for email, webhook and Apprise, firing and recovery, and global versus destination disablement. This corrects false successful queue/audit records; it does not establish maintenance-window expiry, stop an already-started provider request, or repair historical false-success records.

SMTP transaction-stage retry regression coverage

TestEmailRetryRespectsSMTPTransactionReplies extends the greeting-failure fixture through MAIL, RCPT, DATA command and completed-message replies. At each stage, structured 550 replies stop after one attempt despite temporary prose; 451 replies retain the three-attempt configured budget, and a 451 followed by acceptance stops after the second attempt. This checks the sender's wrapped errors and retry termination, not only the failure classifier. The in-memory plain-SMTP fixture sends no external mail and does not qualify TLS, installed recipient receipt or queue persistence.

Webhook diagnostic userinfo confidentiality

RedactWebhookURLSecrets masks the entire URL userinfo (including username-only credentials), before its existing Telegram-path and query-secret redaction. Unparseable URLs produce [invalid webhook URL], not a raw credential-bearing fallback. Valid destination host/path and non-secret query fields remain useful for diagnosis. Transport-error redaction copies the URL error and retains its underlying cause without changing the original error or the configured URL. This follows the credential-exclusion principle in the OWASP Logging Cheat Sheet.

Regression tests cover plain/encoded/user-only credentials, malformed URLs, non-authority at signs, combined path/query redaction, error unwrapping and actual rate-limit log output. These queue-free tests establish local diagnostic redaction, not destination receipt, installed recovery or release qualification. No claim is made that arbitrary custom path/query secrets are recognised.

Delivery-log errors use RedactWebhookDiagnosticSecrets so URLs embedded in otherwise useful error text receive the same masking without discarding the surrounding status context. Malformed embedded URLs still fail closed.

Slack webhook diagnostic path confidentiality

The same helper masks paths on the exact hooks.slack.com and hooks.slack-gov.com hosts. /services/ remains as a diagnostic marker; legacy paths become /REDACTED. Matching uses the parsed, case-insensitive hostname and clears the encoded path representation, so ports and escaped path segments do not bypass masking. Other hosts retain their diagnostic paths. Userinfo and known query credentials remain redacted; configured destinations are unchanged.

Slack's incoming-webhook documentation identifies the webhook URL as secret and documents GovSlack's separate domain. Queue-free regression tests cover both hosts, encoded and legacy paths, lookalike/unrelated hosts, transport errors and actual rate-limit log output. This does not establish customer exposure, recipient receipt or recognition of arbitrary custom webhook secrets.

Discord webhook diagnostic path confidentiality

On exact discord.com and legacy discordapp.com hosts, the shared redactor masks the suffix after /webhooks/, including the webhook ID and token. Versioned API prefixes remain visible; encoded paths, host casing and ports cannot bypass masking. Unrelated paths and lookalike hosts are unchanged. Configured destinations and transport error causes are not modified.

Discord's webhook reference identifies the secure webhook token and token-authorised operations. Focused synthetic regressions cover helper output, transport diagnostics and actual rate-limit logs. This is not evidence of customer exposure, recipient receipt, release qualification, or protection of arbitrary custom-host credentials.

Telegram diagnostic path parsing

Telegram bot-path masking operates on the parsed, decoded URL path and clears RawPath after replacement. This covers percent-encoded bot prefixes without mistaking a hostname or a URL inside a query for a bot path. Method suffixes, query diagnostics and fragments remain intact; configured destinations and transport-error causes are unchanged. Host-independent masking is retained for local API servers, which are supported by the Telegram API documentation.

Focused regression tests cover escaped prefixes/tokens, local servers, missing method suffixes, query URLs, fragments, transport errors and rate-limit logs. This is diagnostic containment, not evidence of customer exposure or recipient delivery. Arbitrary path secrets and unrecognised query credentials remain outside this bounded change.

Bounded diagnostic confidentiality: query representations and bypass callers

Recognised query names are token, apikey, api_key, key, secret and password, matched case-insensitively after one URL query decode. Every repeated occurrence is masked, including mixed literal/escaped names and mixed-case spellings. Case folding is limited to diagnostic matching: original query-name spelling is retained, and configured outbound URLs are not normalised. Unrelated names, ordering and values remain intact; invalid name escapes fail closed. This is diagnostic projection, not mutation of configured destinations or a claim to recognise arbitrary secrets.

TestRedactWebhookMixedCaseQuerySecrets covers all six recognised names plus an escaped mixed-case name through URL, embedded-message and transport-error projection. It preserves unrelated lookalikes, the original request URL and wrapped error identity; these synthetic cases do not establish customer exposure.

Resolved ntfy must apply the same transport-error projection before both its error log and returned error. Common HTTP execution preserves payload bytes, event identity and error causes; URLs containing userinfo remain rejected by outbound validation even though historical diagnostic userinfo is masked.

The caller matrix and retained Delivery regression tests exercise URL/message helpers, actual rate-limit logs, common transport and resolved-ntfy transport errors/logs with synthetic secrets. HTTP delivery-log regression verifies encoded and repeated query credentials while retaining diagnostic context and entry identity. Existing exact-output tables bound Slack/GovSlack/legacy, Discord, Telegram/local paths, malformed URLs and non-secret lookalikes. Earlier proof missed decoded query representations and a separate ntfy transport caller: provider-only helper examples were not sufficient sink coverage. This contract does not assert arbitrary response-body/third-party error secrecy, installed recipient delivery, candidate qualification or historical customer exposure.

Saved notification choices at monitor construction

internal/monitoring/monitor_notification_startup_test.go exercises the actual monitor constructor twice against the same saved configuration without applying notification-manager setters in the fixture. It checks initial routing, both resolve choices, enabled/activation gating and encrypted webhook configuration restoration. Email and HTTP Apprise receiver settings are also saved and compared after each construction, including synthetic credentials, recipient lists, severity filters, email tag filters and enabled/disabled choices. No transport is invoked. The cases include webhook, Apprise, email and all destinations.

This boundary uses the configuration persistence API to save choices, not the HTTP or browser save path. It does not start a new operating-system process, drive an alert lifecycle or establish recipient delivery. Those installed acceptance obligations remain separate from constructor restoration coverage.

Interrupted HTTP response bodies retain the delivery verdict

Webhooks and HTTP Apprise share the bounded readWebhookResponse verdict. When response headers establish a non-2xx status, a subsequent body read failure retains that HTTP status in the error and its authoritative notification failure class. Terminal rejections must not become connectivity retries merely because the diagnostic body is truncated. Response headers, including Retry-After, and the underlying read error remain available; partial body text is not added to the error. A 2xx response whose bounded body read fails remains a failure, not proof of recipient receipt. HTTP Apprise firing, recovery and Test must not suppress that error or mark the queued attempt sent. Only a successful bounded response verdict can create or consume a delivery receipt. Reaching the cap retains the reader's existing limit semantics. The response-byte cap, payloads, TLS/SSRF policy and retry budgets remain.

Transport-only TestWebhookTruncatedResponseClassification covers 200, 401, 403, 422, 429 and 503 with interrupted bodies. TestWebhookRetryTruncatedResponse proves a truncated 403 stops after one attempt with terminal history, while a truncated 503 can retry to 204 with its event identity intact. No queue or storage workers are started by these tests; this is not installed delivery acceptance or a change to retry budgets.

HTTP retry classification agrees across delivery layers

HTTP 421 retains a connectivity class; 423 and 425 retain a server-error class for temporary receiver conditions. These are already retryable exceptions in the webhook transport, and must not become terminal rejections when a wrapped transport error reaches the queue. Authentication, configuration and other permanent HTTP rejections still stop early; attempt limits remain unchanged.

TestWebhookHTTPRetryPolicyMatchesQueueClassification exercises every status 400–599 through a synthetic HTTP transport with misleading diagnostic text, wraps its returned error and checks both transport retry policy and the shared class predicate used by the queue. TestClassFromHTTPStatus pins the reason classes. This proves the classification boundary without starting queue/storage workers; it does not establish installed receipt or queue scheduling execution.

Initial routing edits and previously queued firing work

Changing the initial target affects new sends; it does not rewrite a previously queued email or revoke independent per-level escalation routing. Queue delivery uses the admitted destination configuration while checking current global and destination enablement. Resolution independently cancels obsolete firing work. queue_policy_change_test.go verifies retained email routing after a webhook initial-target edit with an in-memory SMTP acceptance, and prevents a previously fetched pending item from sending after cancellation. This does not establish recipient inbox timing or recall mail already accepted by a provider.

Warning-level destination controls

Email settings and webhook edit/create/list controls retain warning as distinct from all and critical. A warning selection must not become all through frontend normalization. Existing backend severity admission and delivery queues remain unchanged. The notification API adapter round trips and WebhookConfig edit/save regression establish frontend control preservation, not provider delivery.

Resolution cancellation is occurrence-qualified

The monitor's delayed recovery path calls CancelResolvedAlert with the captured ID/start-time snapshot. It removes only that occurrence's pending in-memory grouping members and cooldown record. Persistent CancelByAlertOccurrence shares the existing per-ID delivery gates and cancellation machinery, but matches both ID and start time: pending, sending, failed and dead-letter firing rows are cancelled or rewritten without removing a recurring incident or unrelated members/operational links in a mixed row. Missing occurrence identity is a no-op. The explicit identifier-wide API, audit history, destination receipts, retry budget, recovery rows and best-effort in-flight semantics remain unchanged.

queue_occurrence_cancellation_test.go verifies all four row states after a fresh database reopen, nanosecond-distinct occurrences, operational-link and failed-attempt preservation, missing-identity safety, grouping/cooldown retention and the unchanged ID-wide API. Monitor connected controls additionally require HTTP acceptance of the surviving occurrence through the autonomous queue.

The live monitor does not interpret pending-only cancellation counts as recipient acceptance. Per-occurrence/per-destination receipts remain the sole recovery admission proof, including partial delivery and lost RAM markers after restart; an unannounced destination receives no recovery.

A close that was not a recovery says what happened

A resolved alert can carry an alerts.AlertResolution when it closed without recovering. The one reason so far, moved_to_agent, means the metric now belongs to the resource's linked Pulse agent. Delivery is unchanged: the close is still queued, gated on the firing receipt and sent, and PagerDuty still receives resolve for the alert's dedup key. Only the wording changes. resolvedAlertMessage gives the resolution's summary with the alert named ("Memory alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.") in place of " on is now healthy" for every templated webhook .Message. Grouped lists append the short outcome ("moved to pve1 (Host Agent)") to the moved entry. ntfy titles a single moved alert MOVED: <resource>. Email and Apprise title it Pulse alert moved: <resource> and add the summary after the breach message. Recovery visuals follow the same rule: when any close in a resolved delivery was not a recovery, WebhookPayloadData.NotRecovered is set and the built-in templates drop them (Discord's green embed becomes the neutral blue, both Teams cards lose recovery green, Mattermost's check emoji becomes :arrow_right:), and ntfy swaps its green check tag for arrow_right, for a single alert or a batch. Built-in titles still say Resolved, which is accurate for the closed alert; custom templates can read .NotRecovered, and one that hard-codes recovery text keeps it. The resolution rides the Alert JSON, so it survives the delivery queue round trip and appears in the default JSON payload's alerts array. Issue #1682 is the precedent: a recipient called "is now healthy" a false factual claim when an alert closed without recovering. The resolve transition such a close links to has cause ownership_transferred in internal/operationaltrust/contracts.go, not recovery_evidence. A notification link records the transition ID and lifecycle state, never the cause, so delivery, receipts and recovery admission treat it exactly like a recovery's resolve transition. TestOwnershipTransferOnlyClosesWithEvidence pins that the cause may only enter resolved and only with evidence.

Verification: resolved_handover_test.go drives the firing, its receipt and the queued close through the normal entry points for ntfy, and renders Discord, both Teams cards, Mattermost, PagerDuty, a grouped Slack list, a grouped ntfy batch and email/Apprise content, plus the unchanged ordinary recovery wording and visuals.

Default email subjects retain bounded resource and host identity

Single firing subjects retain severity/type and name the resource and its host. Digest subjects retain every severity count and include up to three distinct resource/host labels, sorted and deduplicated before shortening, with +N more for omitted labels (not omitted alerts). A host display name retains its raw host alongside it when they differ; absent node identity is unknown host, never an inferred connection/instance. An empty resource name falls back to its resource ID, then Unknown resource. Each supplied label/type is bounded to 48 UTF-8 bytes, with visible truncation; controls/whitespace become spaces. Full body identities remain unchanged. Subjects are hints, not globally unique resource keys or proof of current host identity.

Email-only recovery subjects carry the same bounded identity while preserving single resolved/moved and batch resolved wording. Occurrence threading, recovery receipt admission, routing, queue policy and other destination titles are unchanged. Both ordinary and attachment MIME construction encode Unicode subjects with RFC 2047 and fold at spaces/encoded-word boundaries; they still sanitize header line breaks. email_subject_identity_test.go checks the three-host collision, display/raw/missing identities, severity, deterministic bounds/overflow, full bodies, MIME decode/injection safety and normal single/grouped/recovery construction through an in-memory transport. This is source acceptance, not SMTP/Gmail delivery or an installed reporter retest. Literal ASCII encoded-word syntax in supplied names is itself MIME-encoded, so decoding the subject once preserves the name rather than interpreting it as another identity. The literal-encoded-word control covers both MIME builders.

Notifications link to the alert they report

With a public URL configured, a notification links to the alert itself rather than to Pulse's front page: alertHistoryURL builds /alerts/history?alert=<id>, the address the frontend opens one alert at (buildAlertHistoryAlertPath); a Patrol finding, which reaches notifications as a synthetic alert History never stores, links to the Patrol page (alertLinkURL), as does a batch made only of findings; and alertsPageURL links any other grouped notification to the Alerts overview while its alerts fire and to Alert History once they resolve. Single emails end with "View this alert in Pulse"; grouped emails link each alert's name and point their footer at the overview. Apprise and resolved text add a "View in Pulse" line per alert next to the existing Dashboard line. Webhook templates get {{.AlertURL}} (the alert link, the grouped page link, or the existing {{.Instance}} fallback without a public URL); the built-in Telegram, Slack, Teams, PagerDuty, Teams Adaptive Card, Gotify, ntfy and Mattermost templates link it and say "View in Pulse" where they said "View in Proxmox", and the default JSON payloads add alertUrl. {{.Instance}} keeps its meaning, so custom templates are unchanged. The link carries only the alert id already present in every payload (alert_links_test.go).

SMTP editor settings follow the public configuration

The existing Email notification form offers only settings accepted by its save/read path. Reply-to, max retries and retry delay were UI-only values: buildEmailConfigPayload and NotificationsAPI.updateEmailConfig omitted them, and the notification manager supplies its own retry policy. They are no longer editable or fabricated by the form defaults. UIEmailConfig extends the public EmailConfig rather than declaring unsupported settings separately.

EmailProviderSelect.supported-settings.test.tsx checks their absence both before and after expanding the form, default/payload parity, and supported SMTP, rate-limit and routing preservation through edits and reloads with email on or off. Backend configuration, transport, retry policy, TLS, queue recovery and notification delivery are unchanged. Browser controls are synthetic, not native SMTP acceptance or a containing release.

Buffered warnings survive graceful process boundaries

TestGroupedWarningGracefulProcessRestartAndReload drives ordinary CheckHost warning callbacks through real alert/notification managers, persists active warning dispatch markers and performs actual SIGTERM/normal-exit process restarts plus same-process manager reload. Zero-repeat and positive-cooldown cases retain accepted pending batches; two loopback destinations receive the two-warning group no earlier than its original window. Another real process restart precedes occurrence/destination receipt-gated recoveries. No queue clock edit or fabricated successful firing receipt is used. TestMonitorReloadKeepsBufferedWarning separately covers the production monitor constructor/stop/reconstruction with saved notification policy and destinations: pending work remains delayed with no attempt or audit. These are source/native process proofs, not installed-service, external provider, stable release or reporter recovery. A crash before graceful checkpointing can still lose volatile firing buffers; this repair makes no crash-durability promise.