Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

352 KB
Raw Blame

Alerts Contract

Contract Metadata

{
  "subsystem_id": "alerts",
  "lane": "L6",
  "contract_file": "docs/release-control/v6/internal/subsystems/alerts.md",
  "status_file": "docs/release-control/v6/internal/status.json",
  "registry_file": "docs/release-control/v6/internal/subsystems/registry.json",
  "dependency_subsystem_ids": []
}

Purpose

Host rows show their own alerts, including Proxmox physical disks (2 November 2026)

The Proxmox node row's alert stripe and tint read the node's own alerts through getUnifiedResourceAlertStyles, the matcher the Docker, Machines and Kubernetes host tables already use, instead of every alert whose node field equals the node name. A guest's alert marks its guest row, not its host's, so a tinted host row is explained by something on that host. The unified matcher (getAlertsForUnifiedResource in frontend-modern/src/utils/alerts.ts) now also admits a Proxmox node's physical-disk health and wearout alerts, raised on <instance>:<node>:disk:<device> (ProxmoxPhysicalDiskAlertResourceID), which only the name match used to catch; it builds that prefix from the node's proxmox.instance and proxmox.nodeName, as the backend does. Node metric alerts keyed on the node source id and agent component alerts under agent:<id>/ are unchanged. alerts.test.ts and ProxmoxNodesTable.test.tsx cover the admitted disk alerts, the excluded guest and sibling-node alerts, and the row accent. No alert evaluation, identity, history or notification behaviour changes.

The matcher also admits a node's own storage alerts (usage, offline and ZFS on local, local-zfs and the like): alerts whose metadata.resourceType is storage, whose node is the node's proxmox.nodeName and whose instance matches its proxmox.instance. Storage sits on its own Proxmox tab, so the node row is the Overview's only sign that its local pool is filling. Storage shared across a cluster is recorded under the node cluster and never tints a single node.

Runtime alert controls describe only supported scope (28 October 2026)

Container Runtime rows expose their existing Offline Alerts control, not an all-alerts switch or metric bulk editor that cannot mutate runtime settings. Visible guidance explains that connectivity settings do not mute container or machine alerts, and directs operators to existing per-container Alerts switches and saving; newly added containers need their own setting. No single-runtime child mute, inheritance, policy evaluation, persistence or machine-wide mute is introduced. Existing global-disabled connectivity precedence remains intact.

The shared resource table can omit the Alerts column for this connectivity-only surface, including row, group, default, delay and empty-state column counts; other tables retain their existing controls by default. Phone cards honour the same choice. ResourceTable.test.tsx and ThresholdsTable.test.tsx verify the omitted controls, remaining connectivity action, independent container drafts and preserved machine/other-runtime settings. Desktop and narrow browser proof must join the real projection, mutation hooks and consuming table, not an Off label unsupported by the producer. This is truthful control presentation, not fulfilment of discussion #3432/#2107 whole-runtime-and-children demand, recipient proof, installed recovery or release availability.

Missing PBS node-status polls restart metric timing

When PBS node-status collection fails, absent CPU and memory observations pass through the shared metric evaluator. Pending activation, explicit intent grace and memory recovery timing restart without changing an open occurrence, its acknowledgement, trusted readings or lifecycle history. A fresh poll must satisfy the existing full delay; missing time cannot establish continuous evidence. Measured zero retains normal recovery, including immediate CPU recovery.

pbs_observation_continuity_test.go covers both metrics, legacy and explicit grace, recovery boundaries, retained low/high values, exact-PBS isolation and live/shadow agreement. Global pause, disabled policy and full-outage precedence remain unchanged. No collector, connectivity, schema or transport changes; synthetic manager/browser proof is not installed PBS or recipient recovery.

Storage capacity warnings require observed byte counters

Static usage and forecast alerts admit a finite percentage in range only with positive capacity and non-negative used space no greater than capacity. An empty-store recovery still requires zero used bytes and free bytes equal to total. Missing or contradictory capacity holds the existing occurrence, acknowledgement and trusted evidence, and interrupts pending static activation, forecast confirmations and explicit intent grace. A valid empty store still recovers; disabling usage policy releases capacity alerts even when the store is unavailable. Connectivity and attached ZFS health remain independently evaluated, and the global alerting switch remains a pause.

storage_capacity_observation_test.go exercises CheckStorage and its trend entry point, static and predictive occurrence retention, false activation, exact-store isolation, confirmation/grace interruption, policy and independent connectivity controls. Its opt-in manager payloads feed the production Alerts API/Overview browser proof at desktop and narrow widths. No collector, guest operation, schema, history rewrite or notification transport changes; this is not installed recovery or inbox evidence.

Unavailable Proxmox node disks are not recovered alerts

The node evaluator admits the same disk observations as the canonical node metric: positive capacity, finite percentage in range, and non-negative used space within capacity. Missing or contradictory readings hold an open disk occurrence with its acknowledgement and trusted evidence, and interrupt pending activation and explicit intent grace. An observed zero still recovers normally. Explicit disabled/off policy and an actually covering linked agent still release the node alert; a missing reading does not claim another owner. Independent CPU, memory and temperature evaluation and the existing offline policy are unchanged.

node_disk_observation_test.go exercises CheckNode, lifecycle events, measured zero, both grace paths, exact-node isolation, explicit policy and real linked-agent handover. Its optional manager frames feed the production Alerts API adapter and Overview at desktop/narrow viewports. This aligns observation admission only: no collector, guest operation, schema, historical rewrite or notification transport changes, and no installed recovery or inbox claim.

Unavailable agent filesystems are not recovered disk alerts

Host filesystem thresholds use the canonical disk-observation admission: a finite percentage in range, positive capacity and non-negative used space no larger than capacity. A listed unavailable reading or an entirely empty report holds the existing occurrence, acknowledgement and last trusted evidence. It cannot raise a new alert, resolve one, or complete activation or explicit intent grace across the gap. A measured zero retains ordinary immediate disk recovery; no recovery delay is introduced.

An established linked-node summary identity keeps ownership only while its held disk alert remains open; interrupting a pending run returns ownership to the node. Current host, inherited and filesystem policy disablement, explicit agent removal and missing rows in a non-empty replacement inventory retain their ordinary cleanup semantics. No collection, guest operation, schema, historical metric, notification transport or provider lease changes.

host_filesystem_observation_test.go exercises the real manager, lifecycle ledger, exact-host isolation, timing, policy, removal and linked-node controls. Its optional exported manager payloads feed the real Alerts API adapter and Overview surface in desktop/narrow browser proof, not a native recovery claim.

Listed backups need an observed success date to resolve age warnings

The backup-age evaluator retains an existing subject's occurrence, severity, acknowledgement and last trusted evidence when its still-listed rollup has no success date, a zero date or a future date. Such a date supplies neither a new age nor recovery. For multiple provider rollups of one subject, the newest usable success date wins regardless of input order; an unobserved date cannot hide an older valid breach or displace a valid recent recovery. A fresh valid date still recovers normally. Explicit policy exclusions, disabled rules and removal from the supplied complete inventory keep their existing semantics.

TestCheckBackupsRequiresObservedSuccessDates exercises the real manager with PVE/PBS linked guests, PBS orphans and PMG subjects, occurrence/event retention, provider ordering, same-VMID cross-instance isolation and policy/removal controls. Its opt-in browser frames come from CheckBackups and GetActiveAlerts. This is age-evidence handling only: no change to backup execution, source inventory readiness, snapshot alerts, native freeze/thaw safety or notification delivery, and no proof of a reported installation's recovery.

A listed snapshot with an unknown creation date is not recovered

Snapshot warning cleanup distinguishes an absent snapshot from a still-listed snapshot whose creation time is missing, non-positive Unix time or in the future. After current guest policy and overrides are applied, an unknown date retains only that snapshot's existing canonical occurrence, including its trusted age/size evidence, acknowledgement and observation time. It neither creates a new occurrence nor refreshes old evidence. A trustworthy dated observation, authoritative removal, exclusion or disablement still follows normal recovery. The global alerting switch remains an evaluation pause, not evidence of recovery.

This is deliberately conservative for both age and size warnings: an incomplete snapshot observation does not independently update either metric. It introduces no guest, snapshot, backup, filesystem or freeze/thaw operation. snapshot_date_observation_test.go checks repeated gaps, epoch conversion, exact occurrence/evidence preservation, lifecycle events, independent instances, policy changes, trustworthy recovery and removal. The real manager's exported frames feed the production Alerts API adapter and Overview/card browser fixture at desktop and narrow viewports; this is not installed guest recovery.

Legacy alert-history compaction releases removed-row storage

Expiry, duplicate-occurrence compaction and ID removal allocate backing storage for the surviving history rows, not the original loaded row count. Unchanged expiry and removal passes keep the current array. This reduces avoidable runtime retention after loading a mostly-expired or noisy history without changing the 30-day cutoff, five-minute duplicate chain, earliest occurrence/latest observation, identity, ordering, evidence or empty-array shape.

history_retention_storage_test.go verifies exact survivor capacity, recurrence and neighbour separation, detached migration snapshots, subsequent appends and concurrent reads/writes. Connected primary, backup and retired-JSON loads compact without rewriting the recovery files or changing their retirement state. JSON recovery and event-log authority remain unchanged. Array-capacity evidence is not whole-process RSS or attribution of a reported installation's memory growth.

Webhook list changes share the editor's acknowledged writer

Enable/Disable, deletion, bulk changes and form saves cannot overlap in one webhook editor. An open add/edit draft also locks list actions so a saved row change cannot invalidate that draft's enabled state or identity. Pending list actions expose a live status and keep the saved inventory until the server acknowledges each change. Deletion requires success: true before removing a row; a resolved but unaccepted response retains it.

Bulk Enable/Disable captures only rows whose state needs changing and sends them sequentially. A rejected or unconfirmed write stops the remaining batch, retains earlier acknowledged changes and gives fixed partial-change guidance. No automatic retry or rollback occurs. Unmount withdraws remaining unsent calls, without cancelling or undoing a request already issued. Existing masks, routing, payloads, endpoint policy and notification delivery remain unchanged. The retired destination-tab mutation owner cannot dispatch another call, publish a late toast or replace shared inventory belonging to a newly loaded context.

WebhookConfig.mutations.test.tsx connects the real editor and destination owner to deferred/rejected synthetic API replies for serial batches, shared pending admission, no-op rows, partial failure, open drafts, deletion acceptance, explicit retry and unmount. WebhookConfig.test.tsx retains service, payload, form and list coverage with acknowledged bulk sequencing. useAlertWebhookDestinationsState.test.tsx checks late create/update/delete replies against replacement inventory and refuses calls from the retired owner. Browser proof uses the real existing section, hooks and API adapters at desktop/narrow widths, with synthetic local endpoints; it does not establish provider delivery.

Existing settings saves retain the pending and newer drafts

One save per loaded editor owns a click-time snapshot of the alert policy and all separately persisted destinations. Save and Discard are unavailable while it is pending; fields stay editable. Completion clears the dirty flag only when no newer edits occurred. If newer edits exist, the acknowledgement says that the submitted settings were accepted and the newer changes remain unsaved. Failed writes stop later calls, retain drafts and restore Save for a deliberate retry. The existing multi-endpoint save is not atomic: earlier acknowledged writes are not rolled back or described as undone.

Entering the notifications tab does not reload an unsaved draft or an active save. Explicit Discard still reloads saved settings when no save is pending. Reload, organisation switch and unmount invalidate stale completions and stop unsent writes; they cannot cancel or undo an already-issued request. The newer context alone owns load results and pending flags. Apprise response masking replaces only the untouched submitted draft, not newer input.

Save admission additionally requires a successful saved-policy read in the current editor context. Initial load and every replacement withdraw that admission before resetting draft defaults. A failed read leaves editing and saving unavailable; an explicit reload must obtain the saved policy and finish destination loading before admitting writes. An older failed read cannot withdraw a newer successful context. Destination partial-load failures retain their existing separate save block. Defaults are not a recovered saved policy.

alertSettingsSave.acknowledgement.test.tsx exercises the real policy and destination owners with deferred/rejected API replies, coherent snapshots, newer edits, duplicate admission, tab navigation, retry, normal masking and organisation invalidation. useAlertDestinationsState.test.tsx bounds the same downstream snapshot; alertConfigPresentation.test.ts distinguishes pending, accepted and newer-unsaved messages. Browser proof mounts the real configuration surface at desktop and narrow widths with synthetic local endpoints. These are source controls, not ordinary notification delivery. The acknowledgement controls also require failed-initial/replacement-read write refusal, deliberate reload recovery and superseded-error ownership.

PMG collection opt-outs are not recovery evidence

Missing node queues (including an entirely disabled collector) cannot resolve aggregate queue-depth or oldest-message alerts: recovery and lower severity require every discovered node's observation. A partial non-negative reading is a lower bound and can still establish the highest configured severity (critical when configured, otherwise warning), never downgrade or resolve it. Observed per-node queues retain their existing independent evaluation. Missing quarantine totals preserve both category alerts and add no zero growth sample; a successful complete zero remains real recovery evidence. Explicit alert-policy disablement and the existing offline policy remain separate and unchanged. pmg_collection_test.go and the corrected nil-quarantine control in alerts_test.go cover missing/partial datasets and complete oldest-message recovery, known-high partial activation and refusal to downgrade a critical alert using an incomplete lower reading.

Connection confirmations cannot span unknown or disabled observations

The existing PVE/PBS/PMG/vSphere/TrueNAS connection-degraded detector requires three consecutive degraded observations to activate and three consecutive healthy observations to recover. Pending or unknown connection state is neither degradation nor recovery: it drops pending activation and intent grace, restarts recovery confirmation, and retains an active occurrence's identity, severity, acknowledgement, trusted metadata and observation time without fire/resolve callbacks. Paused connections, disabled connections and owning-resource offline policy disablement clear active alerts immediately and reset pending activation even when no active alert exists. Healthy evidence also ends pending intent grace.

Resets address exactly the connection's canonical discrete-state key, not another connection, condition or metric. Core and optional shadow reducer observe the same reset. Intent bookkeeping and its persisted checkpoint lose the interrupted run; re-enabled detection must establish fresh confirmations and any explicit grace. No new alert family, threshold, grace policy, route or external probe is added.

connection_confirmation_gap_test.go covers all five platforms, six interruption paths, exact-connection isolation, fresh activation/grace, acknowledged recovery and exactly-once callbacks with shadow parity. TestInterruptDiscreteRun in reducer/discrete_test.go pins exact-key isolation, retained firing state, acknowledgement, no fabricated resolution and idempotent interruption. These synthetic source checks do not establish native outage or destination delivery.

Partial platform reads alert as degraded warnings

A platform connection whose poll answered but left one reading unread (poll error category partial) reaches CheckConnection as state degraded. It is a degraded observation of the same connection-degraded discrete state, at warning severity, never critical: it activates after the usual three observations, keeps an open outage open (a missing reading is not recovery evidence), and moves an already-firing critical outage down to warning once the connection answers again. Only active runs the recovery confirmation. The alert message and metadata keep the failed read (stateReason, lastError, lastErrorCategory: partial). No new alert family, threshold, policy or route is added. TestConnectionDegradedPartialReadAlertsAtWarning in operational_contract_test.go pins this.

Metric timing requires continuous observed evidence

Unknown metric evidence interrupts timing, not the occurrence. Missing or rejected unified values, non-finite current readings, and unavailable, incomplete or invalid rolling windows drop a pending activation run and reset an open incident's recovery run. Explicit metric-intent grace and its durable pending checkpoint are reset as well. Fresh evidence must complete a new full configured delay; unknown time cannot fulfil a sustained-for or recovery rule. Existing alert values, identity, start time, acknowledgement, notification and lifecycle records are not refreshed or resolved by that interruption.

Guest backup/QGA deferrals apply this boundary to retained memory and aggregate and per-filesystem disk readings. Failed or expired empty disk inventory interrupts tracked filesystem runs, including a guest's former node-scoped identities, without implying filesystem removal. Different guests, instances, metrics and independently current CPU/I/O remain independent. Host memory also interrupts timing when its usage is unknown; explicit rule disablement still uses its normal clearing path independently of telemetry. Node temperature's existing interruption rule is unchanged. Typed metric dispatchers must exclude metrics they evaluate separately from their own missing-value checks: node usage dispatch does not infer a temperature gap before the node's independent temperature evaluator observes its actual reading.

metric_observation_continuity_test.go verifies actual legacy/canonical, unified, host and VM entry points, full-delay fresh activation/recovery, explicit grace, empty/migrated filesystem inventory and rejected-value/source isolation. guest_observation_test.go no longer treats a retained-reading gap as elapsed breach evidence. These controls establish source lifecycle truthfulness, not native freeze/thaw safety or external notification delivery.

Node temperature alerts judge the primary CPU reading

checkNodeTemperature evaluates models.PrimaryCPUTemperatureCelsius, the hottest usable package, CPU-maximum or core reading, instead of the package with a CPU-maximum fallback. The alert value is therefore the reading the node's table, drawer headline and history show, and a core running above the package now drives the alert. The interruption rule is unchanged: no usable reading interrupts timing and never clears an open alert. Proof: alerts_test.go (TestCheckNodeTemperatureJudgesPrimaryCPUReading).

Open threshold alerts carry a volatile live status — issue #2068

evaluateCanonicalMetricAlert builds MetricStatus from the evaluator's own inputs on every evaluation, including the hysteresis hold and the recovery run, where Value, Message and LastSeen stay at the last breach and no branch may notify. The phase comes from the trigger, the clear level handed to the reducer (a value at or below it counts toward recovery) and the reducer's recovery run; progress uses that run's monotonic elapsed time. The status is stripped from the active-state checkpoint, the recovery mirror and the degraded recovery projection, so a held alert adds no durable writes, and after a restart it is absent until the next evaluation. Resolved records carry no live status. Stale-alert cleanup and the retention TTLs use the latest observation (lastObservedAt), so an alert held below its trigger for over a day is not auto-resolved as unmonitored. Frontend surfaces format the status through features/alerts/metricAlertPresentation.ts and never re-derive the phase or recovery timing.

The optional metricStatus.lastBreachAt dates the evaluated value retained in Alert.Value, not the newer observedAt of a hold or recovery. A breaching sample (including equality at the trigger) advances it; a held or recovering sample uses the existing alert's LastSeen. Unknown/zero dates are omitted, never replaced by the occurrence start, restart or poll time. It remains volatile with the status, so restart rebuilds it from the saved breach and a hold does not rewrite the recovery mirror. Resolution drops the whole status.

TestMetricBreachTimeTracksTheEvaluatedBreach checks holds, recovery progress and reset, exact-trigger and renewed breaches. The existing live-status and checkpoint controls also assert the breach date; restart controls cover both durable active state and the JSON mirror, unchanged mirror bytes/mtime, and legacy unknown-date recovery through a new breach.

Held breach dates in attention

getMetricAlertLastBreachAt selects a valid timezone-qualified ISO metricStatus.lastBreachAt before a valid legacy Alert.lastSeen. Unknown, malformed, impossible-calendar and Go-zero dates stay undated; the occurrence start, current poll and recovery start are never substitutes. Attention, overview/incident hover and active History share this selection. Relative ages use the caller's clock. Missing live status after restart keeps the existing recorded-message path. The Assistant hand-off that also carried the date as Last Breach At model context was retired with the chat window on 2026-10-22.

Verification: date-selection/attention controls in helpers.test.ts. These are synthetic source controls, not native readings, inference, notification provider delivery or restart recovery acceptance.

Alert card and open incident lead with the live reading — issue #2068

The Alerts overview card leads an open threshold alert with the presentation's summary and clear rule, keeps the last breach as the summary's hover text, and labels the alert and clear levels in the status's unit (temperatures follow the viewer's °C/°F preference, as every frontend reading does). Without a live status it keeps the message and labels an alert level only for metric types whose unit is known; other thresholds (queue ages, counts) are stated in their own message, so the card never guesses a percent. The resource incident panel describes only the open occurrence whose alertIdentifier and openedAt match an active alert's id and startTime (the incident store's occurrence key), from that alert's live status, rechecked on the shared relative-time clock so it turns stale when evaluations stop; closed occurrences keep their recorded message. Every disk alert on a guest carries the guest's name, so the presentation names the disk from metadata.label, as the per-disk message does.

History leads its open row with the live reading — issue #2068

Alerts > History is a log of occurrences, but its open row is the one occurrence still in progress: it already shows an Active badge and a running duration, so a breach message beside them reads as the reading now. An active history row for a threshold alert with a usable live status leads with the presentation's summary; the desktop row carries the clear rule and last breach in its hover, and the phone card shows them as muted lines because it has no hover. buildAlertHistoryItems keeps description as the recorded message, so search and the Assistant handoff still read the record, and attaches the source alert as liveAlert only to active, non-system rows. Closed rows, and open ones without a live status, keep their recorded message.

History row controls name the alert they act on

Every Alerts > History row, in the desktop table and the phone cards, repeats Timeline, Resource and an icon-only Assistant button, and each used to share one name ("Timeline", "Resource", "Ask Pulse Assistant about this alert"), so a screen-reader or speech-input user could not tell rows apart. AlertHistoryItemActions now names Timeline with getAlertHistoryTimelineActionLabel (alert title, resource and the row's full start time, since history repeats one alert over time; "Hide timeline for ..." while open, with aria-expanded) and Resource with getAlertHistoryResourceActionLabel (rows on one resource share it because they open the same panel). The icon-only Investigate button, which replaced the Assistant icon on 2026-10-22, names its alert through the alerts.investigate.iconLabel and alerts.investigate.iconLabelAt catalog keys from the host's subjectLabel and subjectTime, while its hover title stays the generic action name. Each name begins with the visible label (AlertHistoryItemActions.test.tsx, AlertHistoryMobileList.test.tsx, InvestigateAlertButton.test.tsx, alertIncidentPresentation.test.ts).

The delivery switch has one name

The Alerts overview header switch for external notification delivery is labelled Send notifications in every state (alerts.activation.switchLabel, English, German and Spanish), and that visible label is the checkbox's accessible name. It used to read "Notifications enabled" or "Notifications paused" depending on its own position, so an off switch beside "Notifications paused" read as "pausing is off". The switch position alone now carries the state. The delivery-paused banner and the pause toast still say Pulse keeps detecting and showing active alerts, which keeps the 2026-07-18 boundary between detection and delivery. Alerts.readOnly.test.tsx pins the stable name in both states, i18n.test.ts rejects state words in the label in every locale, and spec 01 finds the switch by the new name.

The paused banner claims only that delivery is off

Pending notification delivery (pending_review) covers both a fresh install and someone switching delivery off, because deactivate writes the same state. The config cannot tell them apart: activationTime can arrive in an imported config without local activation, and the activation API can enable delivery without recording one. So the paused banner's cause for pending delivery (alerts.delivery.paused.reason.notActivated) reads "Notification delivery is turned off." in every locale, instead of claiming it was never turned on. OverviewTab.emptystate.test.tsx renders both histories through the real activation store and hook.

Retained guest observations are not alert evidence

Guest memory marked last-known, unavailable or an unknown observation state cannot start, advance, refresh or resolve a memory incident. A VM filesystem read with a nonempty diskStatusReason, including retained prev-* values, or unavailable aggregate usage is likewise not current disk evidence. It cannot evaluate either aggregate or per-filesystem thresholds, replace an aggregate incident with filesystem incidents, or clear filesystem alerts as if a failed or expired inventory were authoritative removal. Metric filters do not match these unavailable memory/disk values, including a fabricated healthy zero.

Existing incidents retain their last trusted value, occurrence, acknowledgement, observation time and lifecycle/delivery evidence until a current observation resumes evaluation. Successful current inventory still resolves removed filesystems. Explicit disablement, suppression and stopped-guest behavior remain effective independently of telemetry, including per-filesystem disable overrides on retained identities. Legacy unannotated memory and independent current PVE/Pulse-agent memory/disks keep their existing policy, even while QGA is deferred; current CPU and I/O continue to evaluate. Producers own freshness; no new age threshold, QGA request, payload, alert family or UI surface is added.

guest_observation_test.go pins typed/pointer VM/LXC observation admission, delayed breaches, canonical incident/acknowledgement/ledger continuity, filters, fresh recovery and removal, independent metrics and deliberate disablement. TestGuestBackupDeferralAlertObservationLifecycle connects actual HTTP client, poll builder, retained snapshot, unified view, History and the production alert call. These are synthetic source proofs, not native backup/thaw, containing release, installed delivery or external recipient acceptance.

Pulse system alerts retain their service scope in the existing UI

The pulse-system- identity prefix or explicit systemAlert: true metadata marks a Pulse service condition, not a monitored resource. Overview retains the warning, acknowledgement, snooze and occurrence timeline but offers no resource link or monitoring-policy action. History preserves this scope for active and retained conditions before any resource-name fallback: a machine named Pulse is not evidence that it owns the service warning. Ordinary resource alerts, including those called Pulse, retain their existing target affordances.

The Assistant explanation keeps the exact alert identity, message and approval-required boundary, without invented metric values, a guest target or resource Patrol. Reconstructed History explanations preserve the system marker; new system types need no frontend allowlist. Connected overview/history/handoff and investigation-button controls verify both authoritative markers, legacy metadata, name collisions and ordinary resource controls. Browser acceptance checks the actual components on desktop and narrow touch views. This changes presentation only, not alert evaluation, delivery, API or operator permissions.

Backup-age evaluation failures use the existing system-alert lifecycle

pulse-system-backup-evaluation is a warning about Pulse's inability to read complete backup rollups, not a claim that a workload's backup failed. It has no resource target and carries the standard system metadata. A stable fingerprint keeps repeated failed polls on one alert without repeat notification; existing backup-age alerts are retained. Only complete evaluation or deliberate disablement clears the warning. The public message names the lost evaluation and points to Pulse logs without including raw database paths or provider errors.

backup_evaluation_system_test.go verifies stable identity, idempotence, node cleanup survival and recovery. Monitoring's connected evaluator tests exercise persisted stale/fresh data and failures through this helper. No new route, navigation entry, destination or monitored-resource health signal is added.

RAID spare-count lifecycle — issue #2369

Canonical host RAID alerts use the shared required-member health assessment, not attached-device arithmetic. Known configured counts appear as raidRequiredDevices metadata alongside unchanged total/active/failed/spare observations. A healthy spare creates no incident; an actual missing member or failed device still activates critical health. Scrub/resync/recovery and transient operation expiry cannot mask a static deficit. Restoring all required members with a spare resolves the existing incident through the same canonical state identity, once, without a duplicate fire or resolve. TestRAIDSpareCanonicalActivationAndRecovery checks event-ledger activation, continuity and resolution; collector and ingestion tests check both health and canonical alert severity. Routine scrub/resync remains silent, and genuine recovery/reshape keeps its warning when no critical deficit exists.

Retained occurrence re-fires preserve incident truth

A reducer re-fire inside the retention window keeps its original occurrence identity and reopens that incident. It is distinct from replaying the initial firing. Canonical resource changes carry alert_started_at separately from the transition timestamp, so history reconstruction also retains one occurrence. Legacy records without that key retain their historical timestamp projection. Checkpointed re-fire timestamps reject earlier resolution replay and historical read repair. A later resolution still closes the occurrence, and replaying the older re-fire cannot reopen it. This does not change alert retention or delivery.

TestMonitorLifecycleRefireReopensRetainedOccurrence covers both history paths, canonical reconstruction without an incident checkpoint, repeated replay and historical read repair. TestIncidentRefireCheckpointPreservesTransitionOrder covers checkpoint reload on either side of the final resolution.

Automatic acknowledgement lifecycle

Cleanup-triggered acknowledgement uses the same canonical retention records and reducer acknowledgement state as manual acknowledgement. It survives subsequent metric evaluation and a short recovery/refire within the existing retention window. Cleanup requests active-state persistence when it acknowledges alerts; callbacks remain outside the manager lock. No retention duration is extended. TestAutoAcknowledgementSurvivesMetricEvaluation in internal/alerts/reducer_parity_ack_test.go verifies cleanup, the next metric sample and a short recovery/refire. This is synthetic lifecycle proof, not installed notification-destination acceptance.

Hourly tracking-map cleanup retains canonical acknowledgement records while their incident is active, including legacy-keyed active snapshots and active recurrences with an older inactive timestamp. A decision does not expire from its acknowledgement age while the condition continues. Recovery starts the existing inactive-retention window; the ordinary cleanup's one-hour expiry, hourly stale-record fallback and explicit unacknowledgement remain unchanged. TestTrackingCleanupPreservesAcknowledgedProviderRecurrence in internal/alerts/reducer_parity_ack_test.go verifies manual and automatic acknowledgement through provider-native pool incident reconciliation, affirmative recovery, short recurrence, dispatch callbacks and JSON/durable checkpoint restart. TestTrackingCleanupCanonicalAckRetentionBounds verifies active/legacy identity, recent inactivity, both existing expiry paths and missing-timestamp fallback. Modeled elapsed time and local callbacks do not prove installed recovery or delivery to an external notification destination.

Mobile incident drawers transfer their exact context to Assistant and close through the shared explicit handoff callback. Keeping the source drawer above Assistant, or dropping its occurrence identity to make navigation work, fails the linked investigation journey. Both timeline and resource handoffs require mounted regression and final-build narrow browser proof.

Incident history preserves unavailable reads and provenance

Timeline and resource-history views use the shared incident projection. A failed canonical read stays visibly unavailable, including after an error toast expires or when cached evidence remains. Empty-history copy is shown only after a successful empty read. Retry or Refresh repeats the read without discarding saved notes. The event card keeps canonical observation/occurrence timestamps, source, actor and record ID behind the same accessible evidence disclosure. Unknown timestamps remain unavailable. Truncation is explicit. The Assistant handoff preserves occurrence identity and the latest bounded event window, and labels history as insufficient to establish current health.

Incident resource error-state ownership

Resource incident errors follow the same per-resource latest-request ownership as history and loading. Superseded or disposed reads cannot set an error; reset clears errors and invalidates pending reads. A retry clears the current error while retaining cached history until the current request succeeds. A stale success cannot clear a newer failure. This reconciles the merged error accessor with lifecycle protection, without changing notification delivery.

Verification: resource hook lifecycle assertions cover ordering and retry; scripts/check-incident-request-ownership.mjs checks the real hook and panel in Chromium at desktop and narrow widths, including error state and retry. Its scripted fixture is not full merged-UI or installed delivery acceptance.

Retained-queue recovery feedback has no reading deadline

Retry and Dismiss failures have a view-local untimed equivalent beside the delivery-health card on Overview and Notifications. A health refresh, including a verified healthy result that removes the warning, does not erase a rejected action's message. Cancellation preserves it. Deliberate clear or the next confirmed action supersedes it; reloading or leaving the view resets it. The existing error toast remains transient.

An accepted queue mutation still emits success if the health read fails. That read owns the existing unavailable warning. A synchronous throw or rejected promise from the optional activity-refresh callback must not claim the mutation failed: retained feedback explicitly distinguishes accepted action from failed activity refresh. Older action/callback completions cannot restore feedback after a clear or overwrite a newer action's message. Neither acceptance nor a healthy queue proves intended-recipient receipt; no retry policy changes.

Proof: useNotificationDeliveryHealth.test.tsx pins expiry-independent state, health reconciliation, cancellation, callback failure and ownership; AlertDeliveryHealthCard.test.tsx pins mounted live-region composition and clear focus. scripts/check-recovery-feedback.mjs exercises the real Overview and destination views with scripted API responses, real toast expiry, keyboard actions/clear, unavailable-to-healthy reconciliation and three viewport widths. This is the main-only named bet “Recovery feedback without a reading deadline” from the current demand ledger, not a backend failure diagnosis or backport.

Overview delivery diagnoses use latest-started refresh ownership. Older bulk responses cannot overwrite newer card notification status, and an empty active alert set invalidates outstanding reads. Disposal also prevents updates. Failed refreshes retain the existing snapshot; this ordering repair does not add a freshness indicator or establish recipient receipt. Verify response overlap in OverviewTab.deliverystatus.test.tsx, empty-set invalidation in useAlertOverviewState.test.tsx, and rendered ordering at three widths using scripts/check-alert-diagnosis-ordering.mjs.

Delivery-attempt and held-event reads in Destinations use latest-started refresh ownership. A delayed mount response must not overwrite evidence from configuration Retry or a queue-action refresh, including a newer unavailable result. Attempt loading reflects only the latest attempt request; held events remain independently asynchronous. Scope disposal abandons both reads. Verification: ordinary overlap, held-event and disposal cases in useNotificationDeliveryLog.test.tsx, registered mount/Retry coverage in useAlertDestinationsTabState.test.tsx, and positive rendered-content assertions in scripts/check-delivery-log-ordering.mjs. Scripted browser evidence is not proof of backend delivery or receipt by an independent recipient.

The shared delivery-health card wraps action groups according to available space, retaining readable explanation width when Review, Retry, Dismiss and Refresh appear together. Its heading uses the opaque semantic foreground, not the translucent palette shades reserved for status backgrounds. Verify light/dark layouts at desktop, intermediate and narrow widths, including unavailable health, pending refresh and recovery.

The alerts overview offers the existing delivery-status refresh control when health is unavailable, including after a successful retained-queue action whose follow-up health read fails. The warning remains until a verified healthy read; a successful queue action alone is not evidence of delivery health. Normal degraded summary presentation continues to omit refresh.

Confirmed canonical metric recovery publishes the clearing evaluation's value, observation time, and resolved metric wording in the snapshot consumed by recent-resolution reads and notification callbacks. It must not reuse the last firing sample, including when the clearing value is zero or guest identity migrates between nodes. Clone the snapshot before refreshing recovery content; preserve incident identity and start time. Missing or malformed telemetry is not a clearing observation. This does not change explicit configuration-disable or resource-removal resolution semantics. TestAlertCharacterizationGuestMetricResolutionUsesCurrentNodeIdentityAfterMove in internal/alerts/migration_characterization_test.go pins canonical identity and clearing content together; TestPBSPartialMetricsWebhookLifecycle in internal/monitoring/monitor_pbs_webhook_test.go verifies low/zero measurements and invalid-telemetry suppression through a local HTTP receiver.

TrueNAS INFO provider incidents remain on the canonical resource but do not enter the actionable active-alert lifecycle. Native NOTICE remains actionable at informational canonical severity; it must not be discarded merely because INFO and NOTICE share the monitor risk level. Native severity is preserved by the provider projection. An existing provider incident transitioning from a non-critical level to critical re-enters canonical notification dispatch, subject to acknowledgement, rate limits, and delivery policy. The transition preserves incident identity and start time. Repeated critical observations and severity downgrades do not re-notify; confirmed recovery retains the same lifecycle identity. TestTrueNASNativeCriticalTransition in internal/alerts/unified_incidents_test.go verifies these boundaries through native TrueNAS WARNING, EMERGENCY, and INFO observations.

Warning and critical incidents retain normal routing; downgrade to information clears an existing actionable incident through normal synchronisation. TestSyncUnifiedResourceIncidentsTrueNASInformation in internal/alerts/unified_incidents_test.go pins information, warning, recovery, and preservation of provider resource evidence. TestSyncUnifiedResourceIncidentsTrueNASNativeNotice additionally exercises native projection, NOTICE activation without severity inflation, and confirmed recovery to INFO.

Own alert identity, alert specs, evaluation, persistence semantics, and operator-facing alert routing behavior for live runtime alerts. Alert-history investigations on narrow layouts render in a shared drawer outside the virtualized history row. The drawer title comes from the canonical alert-incident presentation vocabulary. A resource-incidents panel embedded in that drawer suppresses its standalone panel title so the investigation has one accessible heading, while the desktop inline panel retains its own title. Alert schedule delivery routing is one persisted, normalized contract. schedule.initialNotify accepts all, email, webhook, or apprise; missing, legacy, and unknown values preserve the backward-compatible all target. The selected initial target owns firing, grouped, and recovery delivery, while every escalation level retains its own independently normalized target. Saving the alert configuration must update the live notification manager as well as persistence so delivery does not differ before and after restart. Configuration-change reevaluation may resolve metric-backed alerts against their updated thresholds and may apply explicit resource-disable policies, but it must not treat provider-owned incidents as missing thresholds. Unrelated configuration saves preserve those incidents and their acknowledgement state until their provider evaluator supplies recovery evidence. Storage configuration re-evaluation must use the same ordered resource ID and alias override lookup as polling. Static and forecast capacity alerts retain storage policy aliases in durable metadata, including through JSON and SQLite restore; a configuration reload must not fabricate recovery by substituting global defaults for a still-applicable datastore override. Older PBS snapshots without alias metadata may reconstruct the canonical datastore alias only when the recorded PBS instance, datastore name and complete legacy resource ID agree. Hyphenated names must not be split heuristically, and a same-named datastore on another instance must not inherit the override. Explicit policy changes retain normal resolution semantics. TestPBSDatastoreOverrideLifecycleAcrossRestart and TestStoragePolicyAliasesLegacyIdentity in internal/alerts/canonical_stateful_test.go pin restored incident identity, hysteresis, confirmed recovery, refiring, event counts and instance isolation for both persisted-alias and legacy snapshots.

When a VM or container stops, guest evaluation resolves only metric-threshold alerts whose observations are no longer meaningful. Backup-age and snapshot posture remain owned by their posture evaluator and may stay active while the guest is powered off; stopped state is not backup-recovery evidence. Docker and Podman container CPU thresholds evaluate host-capacity-normalized CPU percent, not Docker's runtime-native per-core percent. Alert metadata may carry the raw per-core value and reporting host CPU count for evidence, but the threshold value and canonical cpuPercent metadata remain normalized. Docker and Podman container health is live evidence only while the container state is running. Runtime APIs retain the last health-check result after exit and suspend health checks in other non-running states, so those stale values must clear rather than raise docker-container-health; the canonical runtime state lifecycle owns the non-running condition. Proxmox VM and LXC CPU thresholds consume the same canonical guest CPU-percent normalizer as unified live state and guest history. Proxmox guest CPU is the authoritative observation for that guest; a Pulse host agent running inside the guest may retain its own agent alert identity, but its host CPU observation must not replace or renormalize the Proxmox guest value. Docker and Podman OOM alerts require authoritative runtime evidence: the container must be stopped (exited or dead) and its reported OOMKilled state must be explicitly true. Exit code 137 alone is only SIGKILL evidence; explicit false and unavailable/legacy OOM state both fail closed without an OOM alert. Recovery clears an existing OOM alert when the authoritative predicate is no longer true. Backup-age alert attribution of an unlinked recovery rollup treats the subject ref's namespace field as a connection label, not a PBS namespace: it may match a candidate guest's instance or node only by exact normalized equality. Suffix matching is reserved for real PBS namespaces inside the shared identity helpers; applying it to a PBS connection name cross-attributes clusters that share a VMID, so an unlinked backup whose label identifies no guest exactly stays on its generic rollup key. The PBS submission-source learner that shares those identity helpers is backup attribution state, not alert identity: it is built per evaluation from positively attributed snapshots, never persists across evaluations, and is inconclusive by default, so it can only narrow which guest an already-linked backup belongs to and can never widen, merge, or move an alert's subject. Availability incident and alert identity belongs to the source-owned network-endpoint check. Correlation may project probe evidence onto a matched machine, but it must not copy the check incident onto that machine or create a second alert lifecycle. Failure, recovery, history, acknowledgement, and notification routing therefore remain stable under relinking and restart. Certificate validity uses that same source-owned incident lifecycle. Codes prefixed certificate_ map to alert type certificate: expiry-window incidents are warning, while expired, not-yet-valid, and untrusted incidents are critical. A self-signed certificate is not an untrusted-chain alert. Certificate alerts reuse canonical confirmation, acknowledgement, history, recovery, schedule, intent, and notification routing and must not create a parallel delivery path. Linux memory thresholds consume only canonical cache-aware usage. An explicit usage-unavailable sample, a non-finite percentage, or contradictory used/free/total evidence must not open or clear a memory alert. If such a sample follows an active alert, the alert remains active with its last trusted value until a subsequent trusted sample proves recovery; missing evidence is not evidence that pressure disappeared. Proxmox guest disk-read, disk-write, network-in, and network-out thresholds consume only monitoring-owned valid rate observations. A valid idle interval is explicit zero and may prove recovery; a first sample, missing/null counter, partial response, or rejected out-of-order sample is unknown and must not start, clear, or match a custom metric filter. Alert units remain MiB/s at the threshold boundary (bytes/s / 1024 / 1024); that display/threshold conversion must not be applied to the upstream cumulative-counter divisor.

Disk wearout is an evidence boundary, not a plain threshold. The field carries two distinct absent states and one real zero: -1 is the canonical unreported sentinel, and 0 is genuine evidence only from a device that reports endurance at all. Rotational disks never do, so a 0 from one is absence. Alert evaluation must gate on storagehealth.WearoutReported rather than carrying its own inline boundary. A wearout arm keyed on > 0 silently exempts the single worst reading a disk can publish, which let a spent SSD read critical on the Physical Disks surface while raising no alert at all.

Proxmox wearout recovery requires three consecutive reported readings above 10% remaining life; a reading of exactly 10% still matches the alert. An absent reading does not prove recovery, and a new low reading resets the recovery run; until recovery is confirmed the existing alert retains its low-life value and occurrence. This bounds the resolved-then-refired notification loop when endurance data briefly looks healthy (#2112), while a sustained replacement/corrected reading can still resolve. TestCheckDiskHealthWearoutFlappingDoesNotRepeatNotifications and TestCheckDiskHealthWearoutRecoveryAlertCleared pin dispatch/history deduplication and confirmed recovery. This is synthetic source proof, not a claim that the reporter's installed image has been repaired.

The Proxmox wearout value is percent life remaining, so its alert message names it that way (SSD life remaining is 7%), matching the host-agent SMART wearout_low summary; the threshold travels in the alert's Threshold field, which the Alerts card and notifications render beside the message. Calling the number "wearout" read as 7% worn and contradicted the low-life warning. CheckDiskHealth logs a failed disk at error and a worn disk at warn only on the evaluation that opens the alert (the canonical activated transition); later polls that still find the disk failed or at or below the wearout threshold, including mock mode's per-tick evaluation, log at debug. Logging on every evaluation repeated one failed disk's error on every poll for as long as it stayed failed. TestCheckDiskHealthLowWearoutCreatesAlert pins the message and TestCheckDiskHealthLogsAlertOpeningOnce pins the log-once boundary at the threshold, including a fresh error when a recovered disk fails again. That test captures logs through a hook installed in the package TestMain, because assigning log.Logger inside a test races with managers earlier tests left running.

Host SMART counter growth is an event boundary rather than a warning on every historical non-zero value. For an agent-only disk, the first reported UDMA CRC error count establishes an in-memory baseline and raises no alert. A later increase contributes crc_errors_increased as a warning reason to that disk's canonical disk-health assessment; the next stable sample resolves the active growth event while normal alert and notification history retain it. A lower counter, including a reset or disk replacement, establishes a new baseline and must not alert. Missing or negative evidence cannot create a baseline or prove growth. Baselines are bounded tracking state, are discarded by a full alert state reset, and expire after the standard stale-tracking window. Linked Proxmox host agents continue to defer SMART risk alert ownership to the provider-backed disk surface rather than creating a duplicate host alert. TestCheckHostAlertsWhenSMARTCRCCountIncreases, TestCheckHostSMARTCRCCounterResetEstablishesNewBaseline, and TestCleanupRemovesOnlyStaleSMARTCounterSnapshots in internal/alerts/alerts_test.go pin the baseline, growth, reset, recovery, and retention boundaries.

Host SMART alert policy is configured through the resolved agent threshold chain, not embedded inside the disk evaluator. Missing SMART rule fields seed the backward-compatible policy: failed health is enabled; reallocated, pending, uncorrectable, media-error, and CRC-growth counters trigger at one; remaining-life warning/critical thresholds are 10/5 percent; and NVMe spare warning/critical thresholds are 20/10 percent. An explicit zero disables that individual rule and must survive normalization, cloning, persistence, and per-machine override resolution. Counter rules trigger when current evidence reaches the configured value, while the CRC rule applies the configured minimum increase to two successive reports. Endurance and spare values remain evidence-aware, and invalid percentage inputs clamp to 0..100 with critical no higher than its warning boundary. The Machines threshold surface must round-trip every rule without changing these defaults. Linked Proxmox host agents continue to defer risk alert ownership even when their resolved SMART policy differs.

Host SMART recovery is field-evidence-aware. A standby report preserves disk identity but carries no health authority, so it must not open or clear either the canonical disk-health or disk-wearout assessment. For an active SMART alert, every still-enabled reason that raised the occurrence must be observed again before recovery evaluation; an omitted counter or endurance field is unknown rather than a healthy zero. Explicitly disabling the owning SMART rule remains authoritative and may clear the alert without waking the disk. TestCheckHostSMARTDiskAlertRequiresRelevantRecoveryEvidence in internal/alerts/alerts_test.go pins the fire, standby hold, partial-evidence hold, and authoritative recovery sequence.

Threshold sections are keyed by override identity, not by resource type. The Virtualization Hosts section reads and writes overrides on the bare resource id, while the Machines section resolves through the agent-derived identity candidates. A resource that appears in both sections therefore has two different override keys, and a threshold saved from the wrong one is written where alerting never reads it, leaving the machine on the global default with nothing on screen to explain it. A resource must appear in exactly one threshold section, the one whose identity alerting honours. Standalone Pulse-agent machines belong to Machines, so the Virtualization Hosts section is fed by provider-owned Proxmox PVE nodes rather than by every resource of type agent. Canonical TrueNAS and vSphere hosts are also represented as agent resources, but they belong only to their platform threshold sections; admitting either to Virtualization Hosts would normalize edits against Proxmox node defaults and could silently discard a valid platform override whose value happens to equal the Proxmox default.

QEMU guest-agent filesystems have their own threshold identity: guest-disk:<stable-guest-key>/disk:<mount-device-key>. The stable guest key must use the same cluster-aware alias chain as guest thresholds so a VM node move does not strand its filesystem settings. A filesystem override may disable that mount or supply its own disk threshold. Once a mount has a dedicated override, it is evaluated only against that override and is excluded from the guest aggregate disk alert; otherwise the aggregate would still fire at the guest default and silently defeat a disabled or relaxed mount. The visible threshold row keeps the live guestID-disk-* alert resource identity for active-state and intent-policy linkage while persisting through the stable guest-disk:* override identity.

Numeric threshold editing is owned by frontend-modern/src/components/Alerts/AlertResourceMetricEditor.tsx. The table and card layouts delegate global defaults and resource overrides to that control, so input bounds, empty-input handling and Off/inherit transitions cannot diverge by layout. Table rows save when focus leaves the owning row. Focus movement between sliders, inputs, toggles and Cancel stays inside that edit, so Cancel cannot commit the value before its click runs. Cards retain explicit Save. The responsive table owner identifies the active layout. Retiring a row during a resize transfers the draft and metric focus instead of saving it. Both layouts focus the first supported numeric metric when editing starts from the resource action. Clearing an override shows the inherited metric state. When that default is Off, the badge remains Off and the table slider stays disabled. Enabling from that state stages the factory value in one toggle and leaves the group default unchanged. Existing Off/inherit serialization remains compatible. Table sliders sit above the numeric control so editing does not widen the table and hide the resource identity. Backup, snapshot and offline controls keep their separate configuration contracts. Global default cells never route backup or snapshot into numeric thresholds. Host cards preserve the canonical cluster context shown by table rows, so identically named hosts remain distinguishable. Every threshold control names itself for assistive tech, because moving between form controls reads a control's own name, not its column header: the editor's input is " default" or " for " and its On/Off badge "... alerts: On"; offline, backup and snapshot toggles and the three-state Warn / Crit / Off control are "Offline alerts for : Warn"; display-mode value buttons are "Edit threshold for : "; and the alert switches are "Alerts for ". Each name ends with the control's visible word or value so voice control can say what is on screen (WCAG 2.5.3). ResourceTable.test.tsx pins the names in both layouts.

A per-metric threshold is off whenever its trigger is <= 0. That boundary is engine truth (internal/alerts/canonical_metric.go, internal/alerts/config_runtime.go), not a display convention: -1 is the value the editors stage, and 0 disables the metric just as completely because earlier builds advertised 0 as the disable value and those overrides are still on disk. Every threshold editor reads that same <= 0 rule through frontend-modern/src/components/Alerts/alertResourceTableModel.ts rather than testing for -1 inline: read cells, the row override editor, the global defaults row, and the bulk edit dialog, on both the desktop and the mobile surface. A surface that recognises only -1 reports On for a metric the engine is not alerting on at all, and an unset global default carries the same obligation, because absent is disabled rather than enabled.

Turning a metric back on must write a value the save path will actually persist. Clearing an override only re-inherits the default, so it re-enables nothing when the inherited default is itself off, and an undefined entry in a bulk edit reads as unchanged rather than as on; in both cases the editor stages the metric's enabled default instead, and only clears the override when the inherited default is already enabled. Editors must also not manufacture an off state out of an empty input: a cleared box is mid-edit, and coercing it to 0 disabled the metric in the engine while the row still showed On. Disabling is the toggle's job, and the toggle stages the canonical -1.

The save path writes the backend's default contract, not the editors' sentinel. Alert config normalization reads a negative trigger on a global default as unset and restores the factory threshold, so buildAlertsConfigurationPayload in frontend-modern/src/features/alerts/alertsConfigurationModel.ts writes every Off global default as trigger 0. Sending the staged -1 through made every Off default with a positive factory threshold come back On at that threshold after save and reload; only guest and node CPU, memory and disk escaped, because normalization leaves them alone. Overrides keep -1, which the override path stores as written. Because -1 and 0 are the same setting, the Global Defaults "Custom" badge compares Off values as equal rather than by raw number, so the factory -1 I/O defaults do not read Custom against a saved 0. An unset factory value stays distinct from Off, since unset can mean the default follows another setting. A row's custom-threshold marker keeps comparing raw values: it drives the row's revert control and override removal, so an explicit -1 override must stay visible against an Off default even though both alert the same way. frontend-modern/src/features/alerts/__tests__/alertsConfigurationModel.test.ts pins the payload for every section, and internal/alerts/config_defaults_off_test.go pins that a zero default stays off through UpdateConfig and raises no alert.

External availability-probe reporting loss owns one canonical external-probe-unavailable alert per assigned agent, not one alert per target. The lifecycle waits for the assignment grace floor before firing, keeps target IDs as bounded supporting metadata, yields to the existing host-offline lifecycle when the agent heartbeat is unhealthy, and resolves only when a fresh result from the currently assigned agent arrives. Notification delivery, acknowledgement, history, and recovery reuse the normal alert pipeline.

Storage capacity forecasting is an alert-grade evidence boundary, not an AI prose feature. It requires at least 24 hours of valid history, hourly median normalization, a fresh terminal sample, agreement between full-window and recent positive slopes, and confidence of at least 0.80 before it can fire. Raw poll count alone must never manufacture confidence. Forecast risk opens only when projected exhaustion is within seven days (critical within one day) and recovers only after the projection moves beyond fourteen days or trusted evidence proves growth stopped. Missing, shallow, stale, or low-confidence history is unknown rather than recovery for an already-active forecast. Predictive and static usage policy share metric-threshold:usage as one canonical occurrence: a forecast that later crosses the percentage threshold keeps its start time, acknowledgement, history, and timeline instead of emitting a forecast recovery plus a second capacity alert. The same contract applies to Proxmox, Ceph, TrueNAS pools/datasets, and vSphere datastores through their existing platform thresholds and disable policy.

Rolling metric evaluation is alert policy over monitoring-owned history, not a second incident family. metricEvaluationWindows stores seconds by canonical resource type and metric, with an explicit zero meaning current-value evaluation. The Thresholds surface exposes both the global all rule and the canonical guest workload fallback; concrete workloads inherit through the same runtime chain (vm and app-container through guest, host-like agents through node, then all) and must label the effective parent duration rather than always presenting the global value. CPU seeds a five-minute default; only CPU and burst-prone disk/network rate metrics may use rolling averages, while memory, capacity, and temperature remain instantaneous evidence boundaries. A window requires at least three samples, at least 80% temporal coverage, and no gap larger than the bounded cadence allowance. Weak, stale, or unavailable history is unknown: it cannot open or recover an incident, and an existing incident retains its last trusted value. Ready windows use time-weighted averaging so polling jitter cannot bias the result. The evaluated value drives trigger, hysteresis, severity, delay, and recovery; the current value, sample count, coverage, and window remain alert metadata and operator-visible message evidence. Window changes never change the canonical metric-threshold:<metric> identity, so start time, acknowledgement, history, timeline, escalation, and recovery stay one occurrence.

Current metric observations must be finite in both instantaneous and rolling evaluation, including a manager without a history provider. After selecting in-range observations and last-duplicate authority, any non-finite History sample or non-finite weighted result makes the whole window unknown. Invalid samples are not dropped to manufacture coverage, substituted with an older duplicate, clamped, or treated as zero. Both legacy and canonical evaluation retain the last trusted incident value, identity, acknowledgement and delivery state; no new lifecycle or notification evidence is emitted. Fresh finite evidence resumes normal evaluation, and explicitly disabling a rule still clears it independently of telemetry. windowed_metric_test.go verifies NaN/infinities, arithmetic overflow, duplicate authority, input isolation, JSON-safe incident snapshots, transition-ledger continuity and local dispatch. These source controls do not prove installed or external recipient acceptance.

Active-alert restore is opt-out at construction. NewManagerWithDataDir accepts ManagerOption values, and WithoutPersistedAlertRestore starts the manager with an empty active-alert set instead of reading active-alerts.json. Mock mode selects it: switching the mock toggle already clears active alerts, but a process that boots with mock mode already enabled never runs that path and would otherwise restore alerts raised against real infrastructure and serve them beside fixture data. The option changes startup restore only. Alert evaluation, persistence, notification, acknowledgement, and history are unaffected, and the default construction path still restores.

Canonical Files

  1. internal/alerts/specs/types.go
  2. internal/alerts/specs/evaluator.go
  3. internal/alerts/canonical_metric.go
  4. internal/alerts/canonical_lifecycle.go
  5. internal/alerts/unified_incidents.go 5a. internal/alerts/incident_synthesis.go
  6. frontend-modern/src/features/alerts/AlertOverviewActiveAlertsSection.tsx
  7. frontend-modern/src/utils/alertOverviewPresentation.ts
  8. frontend-modern/src/utils/alertResourceTablePresentation.ts
  9. frontend-modern/src/utils/alertDestinationsPresentation.ts
  10. frontend-modern/src/utils/alertIncidentPresentation.ts
  11. frontend-modern/src/utils/alertSchedulePresentation.ts
  12. frontend-modern/src/utils/alertWebhookPresentation.ts
  13. frontend-modern/src/utils/alertActivationPresentation.ts
  14. frontend-modern/src/utils/alertAdministrationPresentation.ts
  15. frontend-modern/src/utils/alertBulkEditPresentation.ts
  16. frontend-modern/src/utils/alertConfigPresentation.ts
  17. frontend-modern/src/utils/alertEmailPresentation.ts
  18. frontend-modern/src/utils/alertFrequencyPresentation.ts
  19. frontend-modern/src/utils/alertGroupingPresentation.ts
  20. frontend-modern/src/utils/alertHistoryPresentation.ts
  21. frontend-modern/src/utils/alertSeverityPresentation.ts
  22. frontend-modern/src/utils/alertTabsPresentation.ts
  23. frontend-modern/src/features/alerts/types.ts
  24. frontend-modern/src/utils/alertThresholdsPresentation.ts
  25. frontend-modern/src/utils/alertThresholdsSectionPresentation.ts
  26. internal/alerts/history.go
  27. frontend-modern/src/stores/alertsActivation.ts
  28. frontend-modern/src/utils/alertThresholdDefaults.ts
  29. frontend-modern/src/utils/metricThresholds.ts
  30. internal/alerts/config_facade.go
  31. internal/alerts/constants.go
  32. internal/alerts/model.go
  33. internal/alerts/metric_hooks.go
  34. internal/alerts/manager.go
  35. internal/alerts/default_config.go
  36. internal/alerts/lifecycle.go
  37. internal/alerts/escalation.go
  38. internal/alerts/callbacks.go
  39. internal/alerts/config/types.go
  40. internal/alerts/config/normalize.go
  41. internal/alerts/config/identity.go
  42. internal/alerts/notification_policy.go
  43. internal/alerts/read_model.go
  44. internal/alerts/pmg.go
  45. internal/alerts/docker.go
  46. internal/alerts/pbs.go
  47. internal/alerts/storage.go 47a. internal/alerts/capacity_forecast.go 47b. internal/alerts/windowed_metric.go
  48. internal/alerts/node.go 48a. internal/alerts/host_agent_node_links.go
  49. internal/alerts/host.go
  50. internal/alerts/backup_snapshot.go
  51. internal/alerts/disk_health.go
  52. internal/alerts/metric_runtime.go
  53. internal/alerts/health_assessment.go
  54. internal/alerts/guest.go
  55. internal/alerts/config_runtime.go
  56. internal/alerts/active_persistence.go
  57. internal/alerts/tracking_cleanup.go
  58. internal/alerts/active_lifecycle.go
  59. internal/alerts/active_cleanup.go
  60. frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx
  61. internal/alerts/storage_override_identity.go
  62. internal/alerts/unified_eval.go
  63. frontend-modern/src/components/Alerts/ThresholdsTable.tsx
  64. frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsData.ts
  65. frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsPlatformData.ts
  66. frontend-modern/src/types/alerts.ts
  67. frontend-modern/src/stores/websocket.ts
  68. frontend-modern/src/utils/alerts.ts
  69. frontend-modern/src/utils/alertsActivation.ts
  70. internal/operationaltrust/contracts.go
  71. internal/alerts/operational_contract.go
  72. internal/alerts/issue1497_test.go
  73. internal/alerts/system_alert.go
  74. internal/alerts/event_emission.go
  75. internal/alerts/eventlog/eventlog.go
  76. internal/alerts/history_projection.go
  77. internal/alerts/history_migration.go
  78. internal/alerts/active_state_bootstrap.go
  79. internal/alerts/eventlog/active_state.go
  80. frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
  81. frontend-modern/src/stores/alertAcknowledgementRequests.ts
  82. frontend-modern/src/stores/websocket-global.ts

Shared Boundaries

History reads must not occupy the connection used to commit alert lifecycle transitions. Disk-backed event logs use a bounded read-only WAL pool alongside the serialized durable writer. An open history snapshot cannot block a durable append or acquire mutation authority. In-memory test stores retain their single connection because their private database is connection-local. Every event-log connection, the writer and both readers, takes its SQLite page cache from pdb.PageCachePragma in pkg/db/pagecache.go instead of a private 16 MB cache_size, because each connection holds its own cache. The modernc driver holds that cache outside the Go heap at about twice its size, where neither the garbage collector nor GOMEMLIMIT reaches it (#1803). TestEventLogConnectionsUseSharedPageCacheBound pins all three.

Chronological replay selects bounded event IDs in time-index order before loading snapshots and details. Pages preserve occurrence-time and ID ordering, type filters, durable-ID watermarks and caller limits without sorting all retained payloads for every page. A full history read remains complete, and history reconstruction must not truncate silently at the public query limit.

Normal history polling shares one manager-owned chronological fold and advances it only through newly committed event IDs. Its inclusive replay boundary keeps concurrent appends out of the current pass. The event log remains authoritative: retention removal advances a revision in the same transaction, late historical events force chronological reconstruction, and replacing or disabling the store discards the fold. Clear-history tombstones still remove preceding occurrences. Live active-alert overlays and returned objects are independent clones. Explicit Since queries preserve their existing event-window semantics without evicting the normal full-history fold. Repeated attention requests must not replay all retained snapshots independently or accumulate behind the history connection pool. Since queries are answered from that same fold (#1803). Each occurrence keeps one mark per folded event (time, durable ID and the LastSeen it contributed) and the index of its latest acknowledged snapshot. A window starts at the occurrence's first event that the log's own occurred_at >= since text comparison admits, takes LastSeen and acknowledgement from that point on, and orders occurrences by their first in-window event as a walk of only the window would. A read selects and limits rows before copying, so only returned rows are cloned. TestWindowedHistoryMatchesAWalkOfOnlyTheWindow holds this equal to an independent walk of only the window, and TestLimitedHistoryReadCopiesOnlyReturnedRows, TestWindowedHistoryReusesTheFullHistoryFold and TestHistoryReadResultsDoNotAliasTheSharedFold pin the copy, reuse and isolation rules. The fold keeps only each snapshot's newest evidence envelope (newestHistoryEvidence, by observedAt then position), and history output applies the same rule to live overlays and the JSON fallback. A resolved alert's trail of up to 64 envelopes was about 83% of its snapshot. The event log still stores every snapshot whole. TestHistoryEntriesKeepOnlyTheNewestEvidence, TestHistoryOverlayKeepsOnlyTheNewestEvidence and TestHistoryFallbackKeepsOnlyTheNewestEvidence pin this.

PBS node-status availability is distinct from connectivity. While a connected PBS carries monitoring-owned NodeMetricsUnavailable evidence, CPU and memory evaluation must not treat zero-valued placeholders as recovery. Existing policy suppression and full-outage handling retain precedence. Valid low measurements resume normal recovery, including callbacks and recent history. Regression proof lives in internal/alerts/telemetry_quality_test.go and the HTTP-to-manager lifecycle in internal/monitoring/monitor_pbs_coverage_test.go.

  1. frontend-modern/src/stores/websocket-global.ts shared with performance-and-scalability, unified-resources: the process-wide realtime store owns alert acknowledgement request lifetime, unified-resource state and fleet-scale connection and reconciliation.
  2. frontend-modern/src/stores/websocket.ts shared with performance-and-scalability: the connection-owned realtime store is both the canonical alert truth boundary and the fleet-scale resource reconciliation hot path. That shared store normalizes slimmed broadcast resources at ingestion — expanding capabilitiesRef through the state capabilityCatalog and synthesizing the default policy posture for resources published without one — before any alert consumer resolves alert-to-resource identity or policy display. Alert override and threshold identity matching must consult canonicalIdentity.supersededIds alongside aliases, because broadcast aliases no longer duplicate superseded canonical ids.
  3. internal/operationaltrust/contracts.go shared with notifications: the operational trust contract is jointly consumed by canonical alert lifecycle ownership and notification delivery linkage without making delivery state operational truth.
  4. internal/proxmoxidentity/backup_identity.go shared with monitoring, storage-recovery: Proxmox PBS backup subject identity is a shared runtime boundary for monitoring backup freshness, backup-age alert attribution, and recovery-point guest mapping. Alert multiline field presentation is shared with frontend-primitives: notification, timeline, threshold ignored-prefix, and resource threshold note editors must compose the shared FormTextarea primitive for label/id/help wiring and textarea chrome instead of rendering raw native <textarea> shells in alert-owned runtime components. Alert resource threshold action presentation is also shared with frontend-primitives: row, mobile-card, global-default, and bulk-selection icon-only actions must compose ActionIconButton for shared size, tone, focus, title, and accessible-name behavior instead of rendering local <button> plus inline SVG shells in alert-owned runtime components. The notification-delivery health card is an alerts-owned presentation over notification-owned queue truth. When retained terminal failures exist, the Destinations surface exposes explicit retry and dismiss actions with consequence confirmations, refreshes health and delivery history after either action, and never instructs the operator to delete queue storage. Platform connection-degraded alerts are availability observations of their owning PVE, PBS, PMG, VMware, or TrueNAS resource, not a separate policy surface. They must honor that resource's disabled and connectivity-disabled override, the platform-wide alert and offline-alert switches, offline intent and quiet-hours policy, and must clear immediately when that policy becomes disabled. The connection snapshot must carry the owning monitor resource ID used by registry alias resolution; the ledger's display ID is not a substitute for that policy identity. A second connection detector must never notify around a resource's offline-alert toggle. Alert runtime state has one explicit ownership boundary: AlertConfig.enabled controls detector evaluation and in-product alert visibility, while AlertConfig.activationState controls external notification delivery only. The websocket store, resource-row presentation, navigation counts, and Alerts overview must preserve active alert truth while notification delivery is pending review or snoozed. Notification activation must not clear the browser active-alert store, suppress resource indicators, lock threshold/history configuration, or claim that monitoring has stopped. The websocket store's raw resource-delta baseline is connection-scoped. Closing or replacing a socket invalidates that baseline and its recovery throttle; a delta from the replacement connection must not patch the previous connection's raw snapshot. Only a full snapshot delivered over that socket may establish the new delta baseline. The same lineage rule applies independently to the connected-infrastructure and active-alert keyed projections. Dropping an oversized state frame invalidates all three raw baselines and their queued projection work. Oversized-state REST recovery may refresh the current connection's displayed resources, infrastructure, and alerts, but it is independently built and must remain delta-free; a later keyed delta without a socket-owned baseline is ignored and requests the shared throttled recovery path. A marker or baseline-less delta observed during hydration must coalesce one trailing REST refresh after the throttle window so the latest invalidation is not lost. Alert deltas still apply immediately when their socket baseline exists. Late socket callbacks and REST responses from a retired connection must be ignored, while a current oversized connection remains free to hydrate without waiting for the retired connection's request to settle. Cold active-alert hydration has a narrower recovery path than oversized estate state. Until the canonical store has accepted either a socket-owned alert snapshot or a successful GET /api/alerts/active response, the Alerts overview must expose pending or unavailable truth and must not turn an empty local store into a "No active alerts" all-clear. An unintentional socket close before that first alert snapshot starts one throttled active-alert REST recovery from the canonical store, and an open connection that remains alert-snapshot-free for five seconds starts the same recovery rather than waiting for the ninety-second heartbeat timeout; pages must not own an independent alert fetch or cache. That response may refresh display truth but must leave rawActiveAlerts baseline-free, and an alert revision plus request-generation fence must discard it if newer socket truth, a URL-scope switch, or disposal wins the race. Once a snapshot is known, reconnects retain it as the last confirmed projection rather than replacing known alert state with transport uncertainty. A failed recovery must remain visibly unavailable and provide an explicit operator retry. While the document is hidden, that same connection-scoped baseline must keep accepting resource deltas without reconciling the visible resource store on every message. The store accumulates changed resource IDs (and their per-key change shapes, unioned across the hidden ticks with unknown-shape contamination) and performs one canonical catch-up reconciliation on visibilitychange to visible. Alert and resolved-alert truth continues to update normally while hidden; this resource optimization must not defer, clear, or reinterpret alert lifecycle state. Active operator input applies the same separation: scroll, wheel, pointer press, and key activity may defer resource deltas, reporting projection, and the shared realtime tick token until the input-idle flush, but alert and resolved-alert payloads must still commit on the message that carries them. The input gate must never delay acknowledgement, recovery, navigation counts, resource alert indicators, or notification-facing alert truth merely because the resource projection is waiting for an idle window. The visible-tick resource commit may apply metrics-only rows as per-key subtree writes instead of whole-row reconciles, but alert truth is outside that fast path entirely: active-alert and resolved-alert stores keep their own commit path, resource alerts facets are not in the fast-path allow-list, and a row whose patch touches alert-relevant structure always takes the full canonical merge. That commit path replaces each stored alert with the server's payload for its id instead of merging into it. A field the server omits is removed from the store, whether the payload came from a socket snapshot, a keyed delta, or REST recovery: a threshold alert restored after a restart carries no metricStatus until its next evaluation, so the previous live reading must not keep reading as "now"; an unacknowledge from another session clears ackTime and ackUser; a dropped nodeDisplayName or metadata disappears. Resolved alerts follow the same rule. The pending acknowledgement hold still skips a payload that contradicts an unconfirmed local acknowledge or unacknowledge, and the confirming payload then replaces the optimistic copy. Because a replacement reconciles the stored alert in place, local optimistic updates are stored as copies, so it never rewrites a record a caller keeps for rollback. The store's updateAlert holds the only optimistic copy of an acknowledgement. useAlertAcknowledgementState writes through it and keeps no override of its own, so once the server confirms it, a later server change such as an unacknowledge from another session reaches the open tab. When no hold is pending and the stored alert already shows the local acknowledge or unacknowledge, the store sets no hold and keeps the server's ackTime and ackUser: the request's own broadcast can reach the socket before its HTTP response, and a hold would then wait for a second confirmation and skip the next real change. Every local acknowledgement write supersedes a REST recovery in flight: the store discards that response, which the server may have built before accepting the write, and fetches again once it settles. A hold that gives up after 15 seconds re-syncs from /api/alerts/active, waiting for a recovery already in flight and fetching again if that one applies nothing, because the payloads the hold skipped may not be followed by another alert change. websocket-unified.test.ts, useAlertAcknowledgementState.test.tsx and useAlertOverviewState.test.tsx pin these paths. Operational evidence and lifecycle identity are typed through internal/operationaltrust. Evidence envelopes distinguish completeness, confidence, permissions, freshness, correlation, and bounded provider detail. Evidence and transition identifiers are deterministic under retry. Active and resolved alert compatibility payloads carry an additive canonical operational record, latest transition, and bounded evidence envelopes; legacy alert paths must migrate through internal/alerts/operational_contract.go and name their limited provenance honestly rather than inventing confirmed provider evidence. Acknowledgement remains distinct from resolution, and every resolution transition references recovery evidence separate from its trigger evidence. A resolved alert may carry a typed AlertResolution (internal/alerts/resolution.go) when it closed without its condition being observed to clear. The one reason so far, moved_to_agent, describes a metric alert whose resource's linked Pulse agent now owns the metric. releaseCanonicalMetricAlert is the disabled-threshold release extracted from evaluateCanonicalMetricAlert: a caller releasing a metric for any other reason passes a resolution, and clearAlertWithResolution stamps a copy on the removed alert before it is resolved. The resolved callback snapshot, the history row, the resolved event-log entry (the reason code, with Summary() as its message) and the Alert JSON then carry it. Resolved consumers still receive the close, because integrations close their incident by alert ID. The synthesized closing evidence carries the resolution reason and summary instead of legacy_alert_recovery_projection, and the resolve transition carries the summary as its reason. Its cause is ownership_transferred, an operationaltrust.TransitionCause that may only enter resolved and only with evidence (resolvedTransitionCause in operational_contract.go), so Patrol's timeline never labels a handover "Recovery evidence" and alert-quality telemetry does not count a handover during a snooze as resolved while snoozed. A nil resolution leaves the disabled-threshold path and every recovery on recovery_evidence. TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery and TestResolvedHistoryRowKeepsResolutionWithoutEventLog in internal/alerts/operational_contract_test.go pin this. The in-app readers carry it too. GetRecentlyResolved projects it into models.Alert.Resolution (reason, successor and the Summary() text) for the state snapshot, the websocket and the assistant and Patrol prompts, pinned by TestRecentlyResolvedCarriesHandoverResolution in history_test.go. The alerts history lists such a row as moved to agent instead of resolved, even when it was acknowledged, with the account in the badge title and on the phone card (buildAlertHistoryItems, getAlertResolutionDetail). The resolved lifecycle change written to the resource timeline carries the summary as its reason and the code as alert_resolution metadata, so the row's incident timeline says the alert moved instead of "Alert resolved: ".

Extension Points

  1. Add new alert rule kinds in internal/alerts/specs/
  2. Add typed collector/builders in the resource-specific checker owner or internal/alerts/metric_runtime.go
  3. Add identity/persistence updates through canonical alert helpers only
  4. Add or change alert history persistence through internal/alerts/history.go, internal/alerts/history_projection.go, and internal/alerts/eventlog/ using normalized owned storage roots and fixed storage leaves only. Event-log projection changes must preserve parity with the in-memory JSON-history model, and migration changes must retain a retryable source or backup until every legacy entry is durably imported. The event log also owns per-consumer projection watermarks in alert_store_meta (projection_watermark:<consumer>): replay consumers pass their watermark as the walk's AfterID cursor so boot-time repair visits only the un-projected tail, and lowering a watermark (including to zero) is the supported way to force a full replay after a projection store rebuild.
  5. Add or change locked alert-investigation commercial handoff behavior through frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx while preserving the shared upgrade-navigation contract; the alert surface may route to the canonical destination, but must not emit browser-local upgrade metrics or present Pro-required copy when prompt suppression applies.
  6. Add or change frontend metric color thresholds through frontend-modern/src/utils/metricThresholds.ts, frontend-modern/src/utils/alertThresholdDefaults.ts, and frontend-modern/src/stores/alertsActivation.ts so browser display colors consume the same configured alert thresholds and override identity chain as the alert runtime instead of hard-coded per-surface thresholds.
  7. Add or change alert investigation through frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx. On resource-backed alerts Patrol is the doer: the action runs a manual scoped Patrol trigger through the ai-runtime manual Patrol route contract (POST /api/ai/patrol/run with the alert's resource, identifier, and type), and Patrol's findings land on the Patrol page. The button stays hidden when AI is not configured, when the session cannot start a Patrol run (settingsCapabilities.patrolRun, which mirrors the run route's RequireAdmin + ai:execute gate; auth-free installs pass), and for Pulse system alerts that have no resource to scope. A click while the request is in flight starts no second run. Alerts must not grow a second AI path beside it; the Assistant chat hand-offs that used to sit here were retired on 2026-10-22.
  8. Keep alert incident timelines in place through frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx: the incident panel and timeline cards carry sanitized incident facts, raw command/output details stay in the incident or approval surface, and the timeline must not add chat hand-offs, suggested prompt chips, or route-owned remediation instructions.
  9. Add or change alert target resource types through internal/alerts/specs/types.go, internal/alerts/config/identity.go, and internal/alerts/unified_eval.go. Supported target types must share the unified evaluator, the canonical threshold/override identity chain, and the standard notification delivery path.
  10. Add or change the alert notification destinations catalog through frontend-modern/src/features/alerts/tabs/DestinationsTab.tsx and frontend-modern/src/utils/alertDestinationsPresentation.ts. The destinations surface presents mobile push (Pulse Mobile paired through Relay) alongside email, Apprise, and webhooks so phone delivery is discoverable where alert routing is configured. It may read and update the Relay destination's alert minimum severity, but routes pairing and connectivity setup to the canonical /settings/system-relay Remote Access panel rather than duplicating Relay runtime state. When the relay feature is absent it gates through the shared FeatureGateSection and upgrade-navigation contract, rendering no upgrade call-to-action when prompt suppression applies.
  11. Add or change system-scoped alerts through internal/alerts/system_alert.go. A system-scoped alert reports on Pulse itself rather than on a monitored resource, for conditions the operator cannot observe from outside the product. Notification delivery is the founding case: the channel that would carry the warning is the thing that failed, so the alert list and navigation badge are the only escalation path that does not depend on delivery working. System alerts carry the stable pulse-system- identity prefix, set no ResourceID so resource-linked affordances are skipped rather than pointed at nothing, and stamp systemAlert metadata. Raising must stay idempotent for an unchanged condition so a timer-driven evaluator cannot become a notification storm; only a change of level or message re-notifies. New system alert types must be raised through this helper rather than by constructing bare alerts, so identity and the idempotence guarantee stay in one place.

Forbidden Paths

  1. New ad hoc Check*-local evaluator logic
  2. Reintroducing runtime legacy alert-ID contracts
  3. Reintroducing per-family threshold/override merge logic outside the shared path

Completion Obligations

  1. Update alert spec/evaluator tests when a new rule kind is added
  2. Update this contract if alert truth or identity rules change
  3. Route runtime changes through the explicit alert proof policies in registry.json; default fallback proof routing is not allowed
  4. Tighten or add guardrails when an old alert path is removed
  5. Update the event-log schema upgrade and history-projection parity proofs when lifecycle snapshots, occurrence folding, or alert-history authority changes
  6. When notification admission changes, verify firing, recovery, escalation, queued-escalation revalidation, and quiet-hours replay precedence together.

Attention projection source contract

Canonical alert operational records, evidence envelopes, and lifecycle transitions are the only writable source for Patrol attention. The legacy alert adapter must preserve an existing provider-authored recommended next step, use the canonical incidentAction when present, and otherwise add the safe operator instruction to open the affected resource and verify current state before changing it. This is migration guidance, not action authority.

The attention read model may project and filter alert lifecycle state, but it must not reinterpret acknowledgement as resolution, omit suppressed state from inspectability, or convert missing/stale evidence into health.

Current State

Retired AI Discovery progress frames (2026-10-11)

The websocket store no longer handles ai_discovery_progress frames or emits that event-bus type, because AI Discovery was removed. A server that still sends one is ignored without touching resources or alert state; websocket-unified.test.ts pins that. Alert detection and notification state are unchanged.

Webhook drafts survive unconfirmed persistence

The existing Notifications editor awaits an explicit successful acknowledgement from its add/update owner before clearing a draft. The owner returns success only after the API responds and the returned destination replaces or joins the displayed inventory. A rejected or interrupted save keeps the same inputs, masked fields, template, mention and tag/severity policy. Fixed inline guidance does not claim the write was rejected: an interrupted response can follow a committed create, so the operator checks saved destinations in another tab before explicitly retrying. There is no editor-triggered automatic replay.

One pending editor save locks the draft, Cancel and the conflicting local list controls, and subsequent Save calls do not invoke another mutation. Success closes the editor; failure unlocks it for correction or deliberate cancellation. The API, encrypted persistence/publication ordering, saved-value masking, Test payload parity, delivery policy, SSRF/TLS and existing shared API-client retry rules are unchanged. This is draft recovery, not exactly-once creation or proof that ordinary notifications arrive.

WebhookConfig.persistence.test.tsx exercises the actual editor and mutation owner through deferred create/update responses, failure, explicit retry, canonical accepted response, cancellation and fixed callback-error guidance. The deliberate-retry control defers its response separately: clearing the old warning at retry start is not acknowledgement, and the draft stays locked until the accepted response incorporates the destination and closes the editor. WebhookConfig.test.tsx now acknowledges successful saves before expecting editor dismissal. The local webhook-save-acknowledgement browser fixture checks the same components with synthetic API transport, including desktop and phone keyboard/touch journeys and another-tab reconciliation after a lost response. It sends no real notification and establishes no installed outcome.

Active Docker update pending age survives restart

A positive Docker image-update report reuses the matching active occurrence's StartTime as its pending-age authority. Resource and stable-identity timer maps are process-local; restoring the active alert must not impose another 24/48-hour delay during which positive reports fail to refresh LastSeen. Both cleanup paths must retain the observed occurrence, acknowledgement and notification identity without manufacturing recovery, refiring or duplicate delivery. Only the same resource's active docker-container-update occurrence supplies age. Another host/container and a resolved occurrence cannot lend their timers; affirmative recovery still retires tracking and a new update waits its normal delay. Absent/failed checks remain unknown, not recovery. An unactivated pending timer remains process-local; this does not infer an update's first detection from its cached registry-check timestamp.

TestDockerUpdateRestartRestoresActivePendingAge in update_alerts_test.go exercises the public host checker, real JSON/durable checkpoint and restart, 24/48-hour delays, acknowledged/unacknowledged incidents, stale saved observation, cached positive/absent/error reports, hourly and retention cleanup, lifecycle and delivery callbacks, isolated hosts and affirmative recovery/new-delay controls. The saved observation is aged deliberately; this is deterministic source-level restart/housekeeping proof, not a naturally elapsed day, live registry or destination result, or reporter confirmation of #2353's full daily cycle.

A "no update" report without a local digest is unknown

An image-update clear is a comparison: the agent must have known the local digest it compared. Agents before #2353 reported UpdateAvailable: false with an empty CurrentDigest when an image inspect failed, because an empty digest compares as "not different". The server took that as recovery, resolving the alert (attributed to legacy-alert-recovery-adapter, the default for clears without explicit recovery evidence) and restarting the delay, so the alert re-fired a full delay later with no fresh registry check in between. checkDockerContainerImageUpdate now treats a no-update status without a current digest like an absent or failed check: it refreshes LastSeen and keeps the occurrence and its pending age. Only the agent produces these statuses and it always sends the current digest when it knows it; locally built images, which have none, never assert an update, so they lose nothing. The "no update without a current digest" case of TestCheckDockerContainerImageUpdate pins the guard and that a real comparison still resolves; the retention and restart tests' clear fixtures carry a current digest as real agents do. The agent side is in the monitoring contract.

Continuing unacknowledged alerts survive age-based cleanup

MaxAlertAgeDays removes an unacknowledged alert only when both its occurrence start and last observation predate the configured limit. A still-observed condition keeps its occurrence identity, age and dispatch timestamp; cleanup must not hide it or manufacture another firing on the next poll. Legacy alerts without LastSeen use StartTime for inactivity. Zero disables this retention rule. Acknowledged-alert cleanup, automatic acknowledgement, explicit clears, confirmed recovery and notification eligibility remain unchanged.

TestCleanupRetentionRequiresObservationInactivity in alerts_test.go covers continuing/inactive, legacy, disabled, acknowledged and inconsistent-timestamp controls. TestCleanupContinuingDockerUpdateKeepsOccurrence in cleanup_observation_retention_test.go exercises the real Docker update detector with cached, absent and failed registry observations through four cleanup/next-poll cycles, JSON and durable active-state restart, callback counts, history and affirmative recovery. This is source-fixture lifecycle proof, not an installed registry/destination result or a diagnosis of a reporter's daily resolve/reopen cycle.

A settings save keeps the alert settings it did not send

PUT /api/alerts/config applies the body to the stored configuration through Manager.ApplyConfigUpdate and ApplyAlertConfigUpdate in internal/alerts/config/update.go: a top-level key the client sent replaces that setting whole, a key it left out keeps the stored value, and the result is normalized as any update is. The thresholds page has no control for flapping detection (flappingEnabled and its window, threshold and cooldown), alert TTL cleanup (maxAlertAgeDays, maxAcknowledgedAgeDays, autoAcknowledgeAfterHours) or customRules, so it never sends them. The handler used to decode the body into a zero AlertConfig, so every thresholds save turned flapping detection and TTL cleanup off, reset the flapping tuning to its defaults, dropped custom rules, and persisted that. The page's flapping, aggregation and ioNormalization payload objects are not AlertConfig fields and never enabled anything. A stored config keeps what it holds, including values an earlier save zeroed; nothing rewrites them.

The body is decoded on its own, so sent keys mean exactly what the decoder makes of them, duplicates and case-folded keys included. Unsent fields are copied from a JSON round trip of the stored config, so the result shares no maps or slices with it. ApplyConfigUpdate reads, merges and applies under the manager lock, so two concurrent partial saves cannot revert each other in memory, and returns a JSON-cloned snapshot so that persistence, which normalizes the config it is handed in place, never writes into the live config's maps.

internal/alerts/config/update_test.go covers unsent, explicit-off, case-folded, duplicate-key, malformed and full-body updates, a sent object replacing its stored value whole (a sent truenasDiskDefaults: {} does not inherit the stored temperature) and the no-sharing rule. TestUpdateConfigKeepsSettingsAThresholdsSaveDidNotSend and TestApplyConfigUpdateKeepsConcurrentPartialUpdates in config_validation_test.go run saves through the manager, and TestUpdateAlertConfig_KeepsStoredValuesForUnsentKeys in internal/api/alerting/alerts_test.go drives the handler against a real manager.

Confirmed empty storage is recovery evidence

Static storage capacity evaluation must admit a zero usage observation when Total > 0, Used == 0, and Free == Total together confirm an empty store. Offline or unavailable storage remains ineligible. Zero usage without those consistent byte counters is unknown capacity, not evidence of recovery. The observation passes through the existing canonical capacity lifecycle, including recovery confirmation and any predictive-capacity evidence; this rule does not directly clear an incident or bypass notification eligibility.

TestStorageEmptyCapacityRecovery in internal/alerts/alerts_test.go pins confirmed empty recovery and retention under absent/inconsistent counters and offline/unavailable status. internal/monitoring/monitor_storage_webhook_test.go pins production callback/queue delivery of the correlated zero-value recovery, current observation time, matching history and duplicate avoidance, with no recovery on missing capacity. This is synthetic local receiver proof, not installed PBS API, off-host delivery or release qualification.

Alert-quality telemetry folds only canonical durable lifecycle truth

internal/alerts/telemetry_quality.go folds active alerts and the canonical 30-day history projection into identity-free aggregate counts. Severity totals, active-age buckets, resolution-duration buckets, repeated occurrences, and snooze outcomes are computed locally. Canonical state and occurrence time may be used transiently for folding, but neither leaves the process. The snapshot also exposes tenant-level configuration and event/active-state authority counts so downstream rates have denominators and persistence failures remain visible.

The contract deliberately does not classify automatic versus operator resolution because alert resolution currently records recovery evidence, not a resolution actor or operator-resolve action. It does not claim maintenance suppression effectiveness because an intent policy can suppress a candidate before an alert occurrence exists. It also excludes detected flapping episode counts because that diagnostic event path may drop under pressure. Flapping configuration adoption remains measurable. telemetry_quality_test.go pins occurrence folding, snooze sequencing, and exact bucket boundaries.

The alert webhook editor exposes the delivery contract's language-neutral MessageKey, event, resource type and node display name alongside the existing type, severity and metric fields. Presentation must advertise the backend-owned template data rather than asking operators to parse the English alert message or reach into metadata for canonical alert identity.

Native sensor temperatures alert only on their own threshold

A host agent's native lm-sensors temperature (HostSensorSummary.Native, kind temperature) raises an alert only when its own override sets a threshold: AlertConfig.Overrides["agent:<host ID>/sensor:<hex>"].Temperature, where <hex> is the lower-case hex of the sensor ID's UTF-8 bytes (HostNativeSensorResourceID), so IDs that differ only by case or punctuation stay apart after the lower-casing host child IDs and metric names receive. A sensor has no default and never inherits the host's, a linked node's or a guest's CPU Temperature; a missing, disabled or zero override raises nothing. The alert uses the canonical metric path with metric temperature and spec resource type agent-sensor (a migration bridge type in spec validation), so its ID is agent:<host>/sensor:<hex>::metric-threshold:temperature, it inherits the temperature stability delay, critical escalation and performance quiet-hours category, and its message names the sensor's label. Fan inputs raise nothing.

The spec resource type is also the runtime classification, so agent-sensor delay and intent rules apply on top of the agent ones.

An invalid reading, a reading without a value, a configured sensor missing from the report, an empty inventory and telemetry expiry interrupt the run and hold a firing alert: the inventory carries no completeness state, so absence is not evidence of removal. The hold is not indefinite: like any alert, one with no observation for 24 hours is auto-resolved by the stale sweep as no longer monitored, and retention cleanup applies as usual. Removing, disabling or zeroing the override, switching off the host's or all agents' alerting, and removing the host resolve it and drop its pending run; cleanup walks active, reducer-pending and intent-pending state as well as configured overrides, so removing an override still reaches its run. A confirmed offline host clears the alert with its other child alerts. A configuration save re-judges a firing sensor alert by its own override only, and interrupts a pending run when the save switched off its threshold, its host (directly or through an inherited linked-resource override), all agents or global alerting, using the host link identity recorded with the run; a run without that identity is interrupted. A re-enabled threshold therefore cannot reuse earlier grace. In CheckHost, global alerts-off returns early without clearing, as for the other host children; telemetry expiry with alerting off clears them. Clearing all active alerts also drops native sensor pending runs and their recorded identity, and the stale-map sweep drops identity whose run is no longer pending. As for every host metric, CheckHost reads policy at the start of a report (the sensor's own threshold separately from the host gates, delay and intent policy during evaluation), so a report in flight during a save can act on the old policy. The next report corrects a sensor threshold or host gate; if the save turned global alerting off, CheckHost returns early and the stale-policy alert remains until alerting is on again or the stale sweep resolves it. Overrides are keyed by the reporting host ID and are not migrated when a host's identity is healed to a new ID, as for agent filesystem overrides; the threshold must be set again. The alert belongs to the host for history; operator intent, like the other host child alerts, still resolves under the child ID. Tests in internal/alerts/operational_contract_test.go (TestHostNativeSensorResourceIDIsCollisionFree, TestHostNativeSensorThresholdRaisesOnlyFromItsOwnOverride, TestHostNativeSensorGapsHoldAndPolicyResolves, TestHostNativeSensorConfigSaveKeepsAlertItsOverrideStillHolds, TestHostNativeSensorPendingRunRestartsAfterPolicyOff, TestHostNativeSensorUsesSensorDelayPolicy) and TestResourceAlertSpecValidateAllowsAgentSensorMigrationBridgeType pin this.

Alerts → Thresholds → Machines writes these overrides in its Machine Sensors section (ThresholdsTableAgentSensorsSection). It lists every usable temperature sensor a machine reports, plus every sensor with saved settings, so a configured sensor that reads invalid, has left the report, sits on a machine that is not reporting, or belongs to a machine no longer present stays editable and removable with its reason beside the name; a non-reporting machine shows no current reading. Rows are grouped by the machine's resource ID, not its name. The frontend builds the ID with agentSensorResourceId (frontend-modern/src/features/alerts/agentSensorIdentity.ts), the same UTF-8 hex encoding as HostNativeSensorResourceID, and the overrides projection reads it back as type agentSensor. The section has no Global Defaults row and no Alerts toggle: a row's presentation default is temperature: 0, marked presentationDefaults, so an unconfigured sensor reads Off as the engine treats it, and the limit cell is the only switch. Because that default is display-only, saving any Off value on a sensor row stores no threshold and removes an override left with nothing else, and any stored sensor override, a saved 0 included, shows as Custom with a revert action. Other rows keep an explicit Off over an Off inherited default, so it survives a later change to that default. The overrides projection carries each stored override's note for every resource type, so a reloaded editor keeps it, and useAlertOverridesState re-projects when only a note changes, so a discarded note-only edit or a note removed on the server does not linger on the row and get saved back (useAlertOverridesState.test.tsx). A row's delay action opens the alert intent editor on metric.temperature (shared getAlertResourcePreferredIntentSignal), and the editor keeps a target that is not a listed resource, such as a machine sensor or disk, as its own option under the row's label instead of falling back to the first listed resource; preview is unavailable for such targets. A handoff is applied once per request: a later change to the resource list does not replay it over the operator's own selection. useThresholdsHostData.agentSensors.test.ts, useThresholdsOverrideMutations.test.tsx, AlertIntentPolicyPanel.test.tsx and ResourceTable.test.tsx pin the frontend side.

Agent custom sensors use canonical health-assessment alerts

Typed HostSensorSummary.Custom numeric, boolean, or timestamp readings with warning or critical status map to one canonical custom-sensor alert per host and sensor ID. Alert evidence preserves the optional group, subgroup, kind, and event time alongside the evaluated numeric value. Collection errors map to warning only when the local definition reports alertOnError; healthy reports, removed definitions, disabled host thresholds, host removal, and telemetry expiry clear the same identity. Alert evaluation consumes the agent-authored typed status and never executes or receives the local command or REST URL. TestHostCustomSensorAlertLifecycle and TestHostCustomSensorErrorCanBeReportOnly in internal/alerts/host_unraid_lifecycle_test.go pin creation, recovery, opt-out, and cleanup.

Health-assessment warning→critical transitions dispatch the updated incident through the normal acknowledgement, snooze, activation and flapping policy; unchanged severity and critical→warning transitions do not dispatch. The shared health-assessment caller opts into the configured per-incident hourly limit for initial, refired and escalation notifications, so repeated severity oscillations cannot bypass that budget. Incident severity still updates when delivery is suppressed. This applies to all callers of the shared assessment adapter (custom sensors, host storage/RAID and storage ZFS pool/device health), not other stateful alert families. No delivery-policy bypass or separate sensor callback is introduced. TestHostCustomSensorEscalationDelivery in internal/alerts/host_unraid_lifecycle_test.go pins the running-host callback, stable incident identity and suppression/no-noise cases; TestHealthAssessmentEscalationDelivery in internal/alerts/canonical_stateful_test.go pins shared ZFS escalation and hourly-budget exhaustion across warning/critical oscillation.

The alert resource-incident panel (frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx) dropped its "Open in Infrastructure / Workloads / Storage / Recovery" cross-jump chip strip on 2026-05-16 when the surrounding platform-first migration retired broad surface-link chips. The panel now keeps investigation flow in place through the shared incident-timeline cards; it must not reintroduce a chip strip that links to the retired top-level routes, and the supporting buildResolvedResourceSurfaceLinks helper was deleted from frontend-modern/src/routing/resourceLinks.ts as part of the same pass.

On desktop that panel renders inline under the history table row that opened it (AlertHistoryTableAlertRow.tsx). On phone layouts, Resource and Timeline investigations render through MobileAlertHistoryInvestigationDialog.tsx as one full-height, independently scrollable investigation drawer. They must not expand inside AlertHistoryMobileList.tsx: the history is a fixed-estimate virtualized list, so changing one mounted card to incident-timeline height invalidates its spacer math and makes the app scroll shell jump or pull the operator back up the page. The mobile drawer keeps that list height and scroll position stable, contains overscroll within the evidence surface, supports Escape and explicit close, and returns focus to the originating action or the history list fallback. It must not become a page-level sibling in tabs/HistoryTab.tsx, which would again open away from a reader deep in history (#1687). Because several alerts can share one resource, resourceIncidentPanel still carries the originating rowKey, and the mobile investigation state allows exactly one Timeline or Resource surface at a time. Resource incident expansion and filtering remain reactive accessors inside each keyed incident row; snapshotting the expanded set or filter set would let state update while the visible Events timeline stays closed or stale.

The alert history filter bar carries no saved-views affordance. AlertHistoryFiltersCard.tsx must not pass a savedViewsKey to the shared FilterBar; that prop and the localStorage view library behind it were removed because a saved view was only ever the page's URL query string, which the browser's own bookmarks already capture and share. History narrowing stays URL-owned so a filtered history page remains a shareable, bookmarkable link.

The alert history severity facet derives each option count through useAlertHistoryState.countForSeverity, using the same filterAlertHistoryItems predicate that supplies the rendered list for the current fetched period and search term. It must not count an unfiltered or separately reduced collection that can disagree with the selected chip's result. These counts follow the shared Inventory totals visibility preference; the Period facet remains uncounted because it selects the fetched time scope rather than filtering the already-fetched rows.

Alert severity is one closed, end-to-end vocabulary: info, warning, and critical. Alert creation, restored active state, history projection, incidents, operational-trust projections, API responses, sorting, resource highlighting, overview cards, badges, and history filters must preserve those three meanings. The legacy history value error normalizes to critical; empty or unknown runtime values fail safe to warning and must never become low-urgency information accidentally. Informational alerts use the shared blue presentation and sort below warnings, while the history facet exposes a real Info option whose count and filtered rows use the same predicate as every other severity.

The alert history frequency chart in AlertHistoryFrequencyCard.tsx stacks each period by severity, critical at the base, from the severityBuckets that buildAlertTrends counts in the same pass as its totals. Those counts use alertSeverityCountKey, the closed vocabulary above: error is critical and an empty or unknown value is a warning, never information. Segment fills and legend dots share getAlertSeverityFillClass, each at least 3:1 against the chart surface in both themes (light-theme warnings use amber-700). A bar's height is the square root of its share of the busiest period (4px floor, 40px ceiling), so a burst stands well clear while ordinary periods stay tall enough to read their mix; each bar's tooltip and accessible name carry the exact counts. The legend counts critical, warnings and info (info only when present) over the current window and bar selection (AlertHistoryFrequencyCard.test.tsx, alertHistoryModel.test.ts).

The chart's axis names round local times. Buckets start at the window start (now minus the range), so labelling bucket edges printed times such as 11:50, and an hour without minutes reads as a bare number in 24-hour locales (8 Oct, 11). formatAlertAxisTickLabel writes an hour with its minutes, and buildAlertAxisTicks moves each inner tick to nextAlertAxisBoundary, the next whole bucket step within the day (or the next local midnight for date-only labels), at that time's true position. The first tick keeps the exact window start the range line shows, the last reads Now, and the buckets, bar tooltips and counts are unchanged (alertHistoryModel.test.ts).

The desktop alert history table sizes each column for its content instead of giving all visible columns an equal share. Each density is sized for the narrowest card it serves (600px, 800px, 1024px and 1280px): Time (the column header; the cell is the row's clock time, 08:27 AM in a 12-hour locale), severity, duration, status ("acknowledged" needs 119px with its cell padding), the Resource header, the type badge and node (from 1024px) and the three row actions on one line (177px) get the pixels their content and header need at that width, and Message, the column that says what happened, takes the rest and never falls under 18% of the card; App.architecture.test.ts holds those minimums and the 50rem, 64rem and 80rem switch points. From an 800px card the three actions share one line; below it they stack, which makes those rows taller, and rows there still differ by how many action buttons they stack. Table rows show Timeline and Resource as 24px icon buttons beside the Investigate icon (accessible names and titles say what each opens; cards keep the worded buttons), so the three actions need 100px instead of 177px, and Status uses a compact 11px badge that needs 108px for "moved to agent" instead of 119px. Resource and Message take that width. An alert may carry Summary, its message without the subject the row's Resource cell already names ("Running 1 of 3 desired tasks" for "Docker service 'web' is running 1 of 3 desired tasks", "Offline", "Powered off"). Builders set it beside Message where they name their own resource: Docker service gap and update state, Docker host offline, Docker container state, health, restart loop, OOM kill, memory limit and image update, node, PBS and PMG offline, guest powered off, Pulse agent stopped reporting, storage unavailable, connection state and external probe silence. Both canonical lifecycle evaluators copy it into the alert they build, so it changes with the message; a MessageBuilder clears it because a built message has no matching summary. History rows lead with Summary and keep the full message in their title; cards, notifications and detail views keep the message, and history recorded before alerts carried a summary shows its message. Below a 50rem card, Resource and Message cells wrap inside a line-clamp-2 wrapper (data-alert-history-cell) instead of ending in an ellipsis after one line, so more of what happened reads without hover. A message or name longer than its two lines still ends in an ellipsis with the full text on its title; the table does not claim to expose every value without hover. The wrapper sets whitespace-normal itself because .table-fixed td forces nowrap. Message also keeps a two-line minimum height (min-h-[2lh]) so a message's length never changes its row's height: the shared windowing hook sizes its spacers from a single measured row. Scrolling a 600-row history in 60px wheel steps (414 tracked comparisons) moved a tracked row by exactly the scroll distance with uniform 49px rows, while the same history with the minimum removed (rows of 39px and 49px) showed 46 steps where the row jumped by up to 20px. From a 50rem card (a desktop window of about 1250px beside the sidebar) both cells return to a one-line clamp (@min-[50rem]:line-clamp-1, minimum height released) and rows are 33px like the other tables; the earlier 80rem switch left 1280-1440px windows with 49px two-line rows. Measured on the mock estate in a real browser (the estate regenerates on every run, so the row count differs: 88 before, 90 after), messages that ended in an ellipsis fell from 82 of 88 rows to 24 of 90 at a 1024px window with the rail open (a 600px card), from 54 of 88 to 17 of 90 at 1280px, from 24 to 8 at 1440px and from 32 to 4 at 1700px, and the median row from 75px to 49px at every window from 1280px to 1700px; at 2000px, where the cells keep a one-line clamp as before, 9 of 88 became 7 of 90. No time cell or header ended in an ellipsis at nine windows from 1024px to 2000px in a 12-hour locale. Cards narrower than 600px, which a window of about 900px with the rail open produces, are still cramped; a container-driven switch to the card list is the remaining fix.

The history card's own width now chooses between the table and the cards, not the viewport. AlertHistoryTableSection measures its wrapper with useObservedElementWidth and shows the cards whenever the card is narrower than ALERT_HISTORY_TABLE_MIN_WIDTH (800px, the width from which the three row actions share one line); until the card is measured, the phone breakpoint decides as before. A 1024px to roughly 1240px window with the Alerts rail open, which gave the table a 600px to 800px card with stacked actions and names broken mid-word, now shows the cards; a 1280px window (an 837px card) keeps the table. The 600px density and the stacked-action rule above therefore serve only a card that has not been measured. The cards take active from that decision instead of a 768px media query, so an open timeline or resource view moves into the cards' drawer when the cards take over and back to the inline expansion when the table returns (AlertHistoryTableSection.test.tsx, AlertHistoryMobileList.test.tsx).

/alerts/history?alert=<id> (buildAlertHistoryAlertPath) opens History narrowed to one alert's occurrences, the landing a notification can link to. useAlertHistoryState filters to that id before severity and search, shows every loaded occurrence regardless of the range window, and decides once per id, only after saved history has loaded and the live alert list is ready (saved history holds resolved alerts only): if nothing matches outside "All time" it widens to All time once, so a later range is the operator's; focusedAlertMissing reports an id history no longer holds. It opens the latest occurrence (still firing first) through the shared timeline expansion and hands the cards a one-shot drawer request a remounted list cannot replay. The tab names the alert in a status notice with "Show all alerts", and while the live list is unavailable it says the alert may still be firing and offers Retry instead of a verdict (useAlertHistoryState.test.tsx, HistoryTab.test.tsx, alertOverviewPresentation.test.ts).

Alert history row timestamps render clock time in the viewer's own locale and must carry the absolute date and time as a title. The date otherwise lives only in the day group header, which scrolls out of sight, and a hardcoded en-US format misreported the time of day to everyone outside the US (#1687, #1685). useAlertHistoryState owns both formatters (formatAlertRowTime, formatAlertRowTimestamp) so the desktop table and the mobile list cannot drift apart. The day group fullLabel built in frontend-modern/src/features/alerts/alertHistoryModel.ts follows the same rule, because a header reading "Thursday, August 6, 2026" to a reader whose rows are ordered day-month is the same defect one level up. See the localization boundary in the frontend-primitives contract for the shared formatting rule and the audit that enforces it.

Alert browser surfaces no longer manage their own runtime-capabilities fetch or hasAIAlertsFeature prop chain. frontend-modern/src/pages/Alerts.tsx and the shared alert overview surfaces (OverviewTab.tsx, HistoryTab.tsx, AlertOverviewActiveAlertsSection.tsx, AlertHistoryTableSection.tsx, AlertHistoryTableAlertRow.tsx, AlertOverviewAlertCard.tsx) must source AI alert feature gating from the shared entitlements layer, not from a per-surface loadRuntimeCapabilities fetch. Alert surfaces must not re-introduce their own hasAIAlertsFeature, runtimeCapabilitiesLoading, or direct /api/license/runtime-capabilities reads.

Canonical alert identity and evaluation are the live runtime model. Remaining legacy references should exist only in explicit migration or negative test boundaries. TrueNAS per-resource threshold overrides use the unified resource's current canonical ID as their persistence and evaluation key. API-backed TrueNAS systems are keyed by configured connection, never reported hostname or DMI serial, so repolls, result reordering, missing serials, DR clones, and multiple same-hostname appliances cannot transfer or strand an override. The browser may read provider-declared superseded IDs and metric-target IDs while projecting a legacy alerts.json, but an edit must remove those candidates and persist exactly one row under the current canonical ID. While an input is active, the thresholds projector must retain the edited row across WebSocket resource refreshes; blur commits that retained row into the unsaved configuration, and the global configuration save remains the only API persistence boundary.

The monitoring bridge may migrate only provider-declared, unambiguous canonical-ID successions before evaluation. It must persist the migrated configuration before installing it in the alert manager, let a current-key override win while removing its retired duplicate, and retain unknown or temporarily absent override rows until a provider proves succession. The same persisted override must drive trigger, clear, derived critical severity, and notification dispatch before and after restart. Regression ownership is frontend-modern/src/features/alerts/thresholds/hooks/__tests__/truenasThresholdPersistence.test.tsx, frontend-modern/src/features/alerts/__tests__/useAlertsConfigurationState.test.tsx, internal/alerts/canonical_override_migration_test.go, and internal/monitoring/monitor_alert_override_migration_test.go. All v6 platform alert targets must enter runtime threshold evaluation through UnifiedResourceInput and internal/alerts/unified_eval.go: Proxmox guests/ nodes/storage, Docker hosts/containers/services, Kubernetes clusters/nodes/ namespaces/deployments/pods, TrueNAS systems/pools/datasets/disks, VMware vSphere hosts/VMs/datastores/networks, PBS, PMG, and standalone host agents. Adding a platform-local evaluator branch for these resource families is forbidden. Per-platform defaults, per-resource overrides, global disables, active-alert reevaluation, history persistence, and notification delivery must use the same alert configuration shape rather than a platform-specific sidecar. Notification cadence is part of that runtime contract. Schedule.Cooldown and Schedule.MaxAlertsHour apply to already-active alert re-notifications as well as first-fire creation, including canonical metric alerts, legacy metric paths, and severity-change re-notifications. Accepted alert dispatch must record LastNotified back onto the live active-alert state before persistence, even when a restored or replayed alert is dispatched through a clone, so reloads do not reopen the same alert's notification window. Recently resolved alerts are an operator-facing transition window, not an unbounded history store. recentlyResolved must prune expired entries and cap the newest retained entries on insert as well as during cleanup, so monitor sync and websocket state snapshots remain bounded; durable resolved-alert history belongs in the alert history store, not in this live transition cache. recentlyResolved and resolvedAlias have one lock owner: Manager.resolvedMutex. Every access holds that mutex, including alias-repair lookups, which require the write lock. When an operation needs both manager state and resolved state, the only permitted nested order is Manager.mu then Manager.resolvedMutex; no path may acquire Manager.mu while holding Manager.resolvedMutex. Resolved critical sections are limited to map access and must not dispatch, persist history, invoke callbacks, or perform notification work. Canonical lifecycle and stateful cooldown refires consume resolved state through the shared lock-order-aware helper, preserve the original alert StartTime, and keep the five-minute refire/history semantics. The reducer core owns transitions for the canonical stateful family: health assessments, posture thresholds, and change thresholds, including ZFS pool state, backup and snapshot age, and Docker update-delay state. Legacy pending-since maps are read-only mirrors during migration and must not decide activation. Stateful occurrences retain their established timestamp boundary: the current observation, or a caller-supplied override such as the backup timestamp, rather than the pending-run start. A manual clear must pass its clear time into the reducer, record the firing incident as recently resolved, and start acknowledgement retention so a refire inside five minutes restores the same occurrence without duplicating history. Regression ownership is internal/alerts/canonical_stateful_test.go. The browser thresholds surface is also platform-shaped: Proxmox, Docker, Kubernetes, TrueNAS, vSphere, PBS, PMG, and Systems. Route-backed platform choices must use the shared Subtabs navigation above the shared FilterBar; they are not resource-filter facets and must remain visible when the mobile filter shell is collapsed. Resource filtering must use the shared FilterBar chip and "+ Filter" pattern, and alert tables must use the canonical platform table column-kind alignment helpers from frontend-modern/src/features/platformPage/ rather than hard-coded table alignment classes. Alert filter option semantics stay alert-owned, but FilterBar chip presentation is frontend-primitives-owned: severity filter leading dots must use filterChipStatusDot rather than alert-local span factories. Alert notification and timeline form textareas are also shared-primitive consumers. Email recipient lists, Apprise target lists, webhook payload templates, threshold ignored-prefix input, and incident timeline notes must compose FormTextarea for label/id/help wiring and textarea chrome instead of recreating raw labelled <textarea> shells locally. Alert resource row/mobile note editors now follow the same primitive contract. The Recovery and Snapshot Age threshold sections expose exactly one Global Defaults surface: the always-live editor row (table layout) or card (card layout) that the shared resource table renders when global defaults are supplied. They must not add a synthetic read-only resource row mirroring the same record; a second surface invites edits that no override mutation path persists, which is how #1680's silently-dropped edits happened. Snapshot size thresholds belong to the Snapshot Age section, not Recovery, because BackupAlertConfig has no size dimension. The recovery-defaults records carry the normalized warningSizeGiB and criticalSizeGiB metric keys that the column editor resolves for the size columns, and the configuration snapshot layer must round-trip those fields: applyAlertsConfigToSnapshot and buildAlertsConfigurationPayload carry them between the UI snapshot and /api/alerts/config rather than stripping them. Guests in the VMs & Containers section order by display name (numeric-aware, case-insensitive) with vmid as tiebreaker, because the rows render the name and an invisible sort key reads as an unsorted list. Regression ownership is frontend-modern/src/utils/__tests__/metricThresholds.test.ts and frontend-modern/src/features/alerts/thresholds/hooks/__tests__/useThresholdsRecoveryDefaultsState.test.tsx. Guest metric canonical state remains resource-backed and therefore node-scoped for Proxmox guests, so node moves must not strand active alert state on the previous resource ID. When a guest metric alert survives a node move, alerts runtime must migrate the active alert, history entry, acknowledgment record, suppression/rate-limit/flapping tracking, and guest per-disk metric identity to the current canonical state instead of reopening a duplicate alert or resolving only the stale node-scoped identity. Metric alerts stored under a canonical state key must also clear through that same identity. Hysteresis resolution must derive the active-alert storage key from the existing alert and canonical state ID; removing only the legacy <resourceID>-<metric> ID can emit a resolved notification while leaving the canonical alert active to resolve again on every poll.

internal/alerts/reducer is the deterministic lifecycle core. It accepts an explicit observation time and models enablement, hysteresis, sustained delay, intent gating, acknowledgement, and warning/critical severity without persistence or notification side effects. The canonical lifecycle family and the shared metric-threshold runtime now use reducer incidents as transition truth; the alert manager projects those incidents into active alerts, history, persistence, callbacks, and notification policy. Metric incidents are keyed by the canonical metric spec ID, while the metric name remains the severity classification input. Families not yet cut over continue to use the parity and shadow harnesses; a divergence is a reducer bug unless a focused manager regression first proves otherwise.

The confirmation/discrete-state slice uses the same boundary for connectivity, powered-state, and generic discrete matches. It activates after the resolved number of consecutive matches, clears pending state or firing state on a non-match at the evaluator layer, and re-derives spec-carried severity on every firing observation. Both manager and reducer must date first activation at the first matched observation, not the final confirming poll. Alerts runtime owns that first-match timestamp alongside the confirmation counter and must clear it whenever the run resets or its tracking state is removed. A direct reset of the confirmation counter also begins a new run: the next matched observation must replace any stale first-match timestamp so a later alert cannot be backdated to an earlier outage.

Poll-driven offline recovery composes an additional confirmation gate over that evaluator-layer lifecycle. A firing incident resolves only after consecutive healthy observations (three for nodes, PBS, and PMG; two for storage), and an offline observation resets the healthy run. Pending incidents still clear on the first healthy observation, while disabling the rule bypasses recovery confirmation and resolves immediately. The reducer's recovery state and the manager composition must remain step-for-step equivalent under the parity harness.

A discrete incident that reactivates within the five-minute recently-resolved retention is the same occurrence: it restores the original first-activation time and emits the reducer's EventRefired, matching the manager path that reactivates without adding a second history occurrence. The resolved record is consumed by reactivation; after the retention window, activation starts a new occurrence at the new matched observation. The reducer measures retention on the explicit observation clock. The manager's temporary wall-clock comparison remains reference behavior only until cutover and is covered by a parity harness that aligns the two clocks. Additional alert families and downstream delivery behavior require separate contract slices.

Acknowledgement is canonical incident state across reducer families. It may be set only on a firing incident, remains attached through per-observation alert rebuilds, and is removed explicitly by unacknowledge. Resolution makes the acknowledgement record inactive rather than deleting it, so a reactivation within one hour restores the original user and acknowledgement time; after that inactive retention, reactivation is unacknowledged. The reducer enforces this boundary on its observation clock, matching the manager's one-hour cleanup TTL, and parity must cover acknowledgement, unacknowledgement, rebuild, short resolve/re-fire restoration, and expiry for both metric and discrete families.

The intent-policy gate composes at activation time with both discrete confirmation and shared metric thresholds. An explicit policy grace period or resolved operator suppression keeps a matched condition pending while preserving its first active observation and, for discrete state, clamping its confirmation count at the configured requirement. For metrics, the explicit metric.<name> policy replaces the legacy time-threshold delay and accrues grace only from monotonic runtime ticks; a below-trigger observation clears the pending run and resets grace. Grace and operator suppression accrue concurrently; once both release, activation uses the preserved first observation as the incident start. Maintenance, muted or retired monitoring, and expected-offline state may hold a new activation when the resolved policy honors operator state, but must not clear or suppress an incident that is already firing. The reducer applies caller-resolved intent on its observation clock, and parity must exercise the manager's real intent-policy and maintenance-window composition. Backup-offline deferral remains outside this characterized slice. That same guest-threshold owner also governs guest-derived lifecycle and posture alerts. Snapshot age, backup age, powered-off state, and configuration-change reevaluation must all construct a canonical lightweight guest snapshot and route threshold resolution through the shared guest-defaults → filter-driven custom rules → guest-override → pulse-relaxed chain. That canonical guest context must preserve the live guest name and tags for snapshot and backup posture evaluation. Ignored prefixes, pulse-no-alerts, configured ignored tags, and required-tag filtering must resolve through the same guest alert policy before any guest-derived alert is created; posture pollers may not downgrade that context to a name-only lookup that bypasses the operator's suppression policy. Passing nil guest context or resolving only overrides/defaults is forbidden because it silently bypasses custom guest rules and makes guest lifecycle alerting diverge from running-guest metric truth. The pulse-relaxed step lives in effectiveAlertPolicyNoLock, not in CheckGuest, so polling and config-save reevaluation judge a relaxed guest's alert by one threshold. Reevaluation rebuilds the guest from alert metadata, so guestSnapshotFromAlert reads back the tags the evaluator recorded ([]string live, []interface{} after a restart). Without them a save judged the alert by unrelaxed thresholds, resolved one raised under an unset threshold's relaxed floor with a recovery notification, and the next poll raised it again. Those tags must stay current: metricOptionsWithTags drops the key from an open alert once its resource has no tags, or a removed pulse-relaxed would keep judging the alert by the relaxed trigger. The tag lifts CPU, memory and disk triggers to a 95/92/95 floor and gives an unset threshold that floor, but an Off threshold (trigger <= 0) stays Off: relaxing a guest must never raise an alert its config turned off. A filesystem's own override still replaces the relaxed guest disk threshold for that filesystem, on save as while polling. Proof: TestConfigSaveJudgesRelaxedGuestAlertByRelaxedThresholds, TestConfigSaveKeepsRelaxedGuestFilesystemOverride, TestPulseRelaxedKeepsOffGuestThresholdOff and the Off case of TestApplyRelaxedGuestThresholds in internal/alerts/alerts_test.go, and TestCheckGuestDropsRemovedTagsFromOpenAlerts in internal/alerts/unified_eval_test.go. That same guest-alert owner also has to retire per-disk guest alerts when the guest stops, disk alerting is disabled, or the reported disk set changes. Canonical guest disk identity is only valid while the guest still exposes that disk resource under the current thresholds, so runtime cleanup must remove stale guestID-disk-* state instead of leaving orphaned per-disk incidents in active alerts, resolved history, or later UI projections. That same alerts runtime also owns instance-scoped node display-name resolution. Raw node names are not globally unique across configured infrastructure instances, so cached node display names must key on instance + immutable node identity whenever the alert carries instance context. Current and prior native Proxmox node names may resolve that same entry so an override or native rename updates active alerts, resolved alerts, and history presentation without changing the stored diagnostic native resource name or alert identity. Read-model presentation must snapshot resolved state before acquiring the active-alert lock to preserve the established lock hierarchy. Alert updates, incident rebuilds, and guest-metric migrations may fall back to the legacy name-only cache only for instance-less resources like standalone host agents. Display names never become acknowledgement, suppression, rate-limit, flapping, escalation, or notification routing identity. That same host-alert boundary also owns vendor-managed NAS RAID suppression as an alert-lifecycle concern. Shared storagehealth rules decide which Synology or QNAP md arrays are vendor-managed system volumes rather than customer-facing storage, and alerts runtime must use those shared rules both to suppress new RAID incidents and to clear stale suppressed alert IDs even after monitoring has already normalized those arrays out of canonical host state. Storage alert runtime also owns operator-facing resource labels for storage incidents. ZFS device alert labels must preserve raw device names such as /dev/sda4, but must not join pool and device labels with a raw slash because device paths can already begin with /; browser alert surfaces consume the runtime resourceName as authored rather than patching storage labels locally. ZFS pool and device alerts follow the storage's pool attachment lifecycle. A storage checked without an attached ZFS pool must shed any previously raised zfs-pool-state, zfs-pool-errors, and zfs-device alerts on that check rather than waiting for the stale-alert cleanup, because the health path can only clear its own alerts while the attachment exists (#1731). Ceph pool storage threshold resolution is also source-alias aware. Storage alerts must evaluate the normalized pool storage id while accepting legacy agent:<host>-ceph-pool-<name> override keys as aliases, so operators do not lose saved thresholds when the same physical Ceph pool moves between host-agent-only fallback and Proxmox API canonical discovery. Active alert reevaluation after threshold or config changes must use canonical resource type metadata before the legacy node fallback. Host-agent Ceph pool alerts may carry no-colon resource ids with Instance == Node, but when metadata or resource type says storage they must keep using storage threshold resolution and source-alias overrides instead of node defaults. That reevaluation boundary distinguishes metric-backed alert types from provider-owned incidents. A nil metric threshold may resolve only a known threshold-backed metric; it is not recovery evidence for resource incidents, and must not erase their acknowledgement state during a schedule-only save.

Browser metric severity colors are also alert-backed. Workloads, Infrastructure, Storage, and the platform-page tables (Docker hosts and containers, Proxmox nodes, Kubernetes clusters and nodes, TrueNAS systems and apps, vSphere hosts) may pass resolved display thresholds into their local bars, but threshold selection must flow through the shared alert activation store and frontend-modern/src/utils/metricThresholds.ts, including configured hysteresis, disabled thresholds, storage usage defaults, and guest/Docker override identity candidates. The display resolver mirrors the runtime's per-platform default scopes — guest, node, pbs, agent, docker, storage, plus kubernetes, truenas, and vmware backed by kubernetesDefaults, truenasDefaults, and vmwareDefaults — and unified platform resources resolve overrides through the shared unifiedPlatformOverrideIdCandidates chain in frontend-modern/src/features/alerts/alertOverridesModel.ts (the same candidate set buildProjectedOverrides indexes platform alert rows under). Static metric-color defaults are only fallback presentation behavior for callers that do not have alert configuration in scope. Guest bars follow pulse-relaxed the same way. resolveMetricDisplayThresholds takes the tags the engine reads for a guest-scope resource and, when they include pulse-relaxed, applies the applyRelaxedGuestThresholds step to CPU, memory and disk before mapping to display thresholds: an unset threshold gets the 95/92/95 floor with a clear five below it, a lower trigger is raised to the floor and keeps its own clear point (a missing one filled five below, as ensureHysteresisThreshold does) unless the raised trigger reaches it, and an Off threshold stays Off. A relaxed guest's bar therefore turns critical where its alert fires; 88% memory reads warning, not critical, under the relaxed 92% trigger. The display resolver does not read filter-driven custom rules, which the engine folds in before the override, so a guest a custom rule matches can still colour by thresholds the engine does not use, relaxed or not. Only Proxmox VMs and LXCs carry their tags into alert policy, so getWorkloadAlertPolicyTags in frontend-modern/src/components/Workloads/workloadTopology.ts passes tags for those guests alone, and a vSphere VM or a container keeps its thresholds whatever it is tagged. The Alerts thresholds row for a relaxed Proxmox guest keeps showing the configured values its editor changes and adds one line naming the tag and the thresholds it raises (getAlertResourceRelaxedTriggerSummary in frontend-modern/src/components/Alerts/alertResourceTableModel.ts, fed by pulseRelaxed from useThresholdsGuestData); before it, the tag overrode those settings with no visible sign in the UI (issue #863). Proof: the pulse-relaxed cases in frontend-modern/src/utils/__tests__/metricThresholds.test.ts, the alert policy tag cases in frontend-modern/src/components/Workloads/__tests__/workloadTopology.test.ts, and the pulse-relaxed guest cases in frontend-modern/src/components/Alerts/ResourceTable.test.tsx.

Docker container image-update alerts are lifecycle-governed by the alerts runtime. Disabling Docker update alerts globally, disabling alerts for a specific Docker container, ignoring a Docker container prefix, or disabling all Docker container alerts must clear the active image-update alert plus resource and identity first-seen tracking. Generic threshold reevaluation must not keep or resurrect image-update alerts after their owning Docker alert configuration has disabled them.

Docker container alert overrides key on stable container identity. The canonical override key is docker:{hostID}/{containerName}; container IDs change on every recreate, so runtime-ID keys orphaned each image update and accumulated dead entries in alerts.json (#1601). The evaluator resolves the name key first and falls back to the legacy docker:{hostID}/{containerID} key for pre-migration entries, MigrateDockerContainerOverrideKeys (internal/alerts/docker_override_migration.go) re-homes live legacy and unified-hash keys onto the name key on the monitor sync cadence and prunes orphaned ID-shaped entries while keeping name-keyed overrides for absent containers, and the UI writes only the name key while binding rows through the shared dockerContainerOverrideIdCandidates chain (name first; container-ID, short-ID, unified-hash, and slash-tail forms as trailing lookup candidates). Container override work must not reintroduce a runtime-container-ID persistence key.

Docker alert references come only from the builders in internal/alerts/docker.go: a host's own alert docker:<host ID> (DockerHostResourceID, used only by its connectivity alert), a container docker:<host ID>/<container ID> (DockerContainerResourceID), and a Swarm service docker:<host ID>/service/<service ID> (DockerServiceResourceID). A container or service reported without an ID alerts under its name, marked in the ID position (docker:<host ID>/name:<name>, docker:<host ID>/service/name:<normalized name>). Docker and Swarm IDs never contain a colon, so a name reference never takes its host's reference or another service's ID reference, and one with neither an ID nor a name raises no alert. Report cleanup, host removal and the host's offline alert firing clear children by the docker:<host ID>/ prefix; the container and service policy switches classify by the first path segment after the host (dockerAlertResourcePath), so a container name containing /service/ stays a container; and Patrol's scope aliases and resource history parse the same forms. Service overrides key on the same reference, so an override written for a service without an ID under its old unmarked name no longer applies. A reference can contain the canonical state separator (a Docker host disambiguated as <base>::<suffix>), so splitCanonicalStateID splits a state ID before the alert's recorded spec ID, or else after its resource ID, and only without either at the first ::. Splitting at the first :: gave such a host's alerts the reference of the host named <base>; an alert saved that way recovers its reference on restore because its spec ID is intact. Before name references existed, a container without an ID alerted under its host's reference. Only the connectivity alert belongs there, so the host's next report, or its offline alert firing, clears any such alert left active whose recorded host matches and whose container ID is empty. A pending image update recorded there keeps its age under the container's new reference when that first report still shows the update pending; a first report with missing or failed update evidence clears it, and the delay starts again on the next positive report. Proof: TestDockerAlertsWithoutIDsUseTheirOwnReferences, TestDockerHostIDWithSeparatorKeepsItsAlertReferences, TestDockerAlertsRestoredWithCutShortReferencesRecover, TestDockerContainerWithoutIDKeepsPendingUpdateAgeAcrossUpgrade and TestDockerContainerResourceID in internal/alerts/alerts_test.go, and TestOwnerAlertTimelinesUseCanonicalHistoryIdentity in internal/monitoring/monitor_alert_handling_test.go.

Backup orphan evaluation is also inventory-scoped. The alerts runtime may evaluate recovery rollups for backup age, but unresolved Proxmox PVE backup subjects must not be treated as orphaned until monitoring has supplied the matching per-instance guest-type inventory readiness signal. Known Proxmox template subjects are valid backup subjects, not orphaned workload backups, even though templates remain excluded from normal runtime workload resources.

Alert history persistence is also part of that canonical boundary. The history manager may choose the owned runtime data directory, but it must normalize that directory once and then resolve only the fixed alert-history.json and alert-history.backup.json leaves through the shared storage-path helper before any filesystem read, write, rename, or delete. Future history-persistence changes must not reintroduce raw filepath.Join(dataDir, ...) joins from caller-supplied directories or ad hoc history filenames. Agentless availability incidents now enter alerts through the same unified resource incident bridge as storage, PBS, VM, and host resource incidents. Standalone network-endpoint resources and any canonical resource carrying an attached availability facet must create canonical resource-incident alerts with provider display Availability; availability alerting must not introduce a second endpoint-only evaluator or alert identity family outside internal/alerts/unified_incidents.go. When a resource carries multiple checks, the incident NativeID selects the exact check evidence envelope that is copied into the alert and its OperationalRecord; the singular compatibility summary must never substitute evidence from a different target. An incident with no backing availability check still carries first-class evidence: the bridge derives a complete/confirmed envelope from the directly observed incident payload (provider and collector from the incident, digest payload ref) instead of leaving the alert evidence-less for the legacy partial shim to backfill. Because each sync re-observes the incident, the bridge merges the cycle's freshly observed envelopes into the already-active alert, so raise-time evidence (including availability envelopes with validity windows) does not age out while the condition is still directly observed. The same lifecycle transition then projects into Patrol like every other canonical operational record.

Notification transport, provider delivery, queue safety, and notification API transport now live under the explicit notifications subsystem inside the current architecture lane. The alerts surface still owns operator-facing alert pages and routing UX, but it does not implicitly own the delivery engine. That includes the webhook settings editor: alert UI may present provider setup, but canonical service-field ownership such as Pushover token / user normalization belongs to internal/notifications/ and persistence boundaries, not to alert-surface runtime delivery code.

The alert webhook editor now mirrors that canonical Pushover field rule through frontend-modern/src/utils/alertWebhookPresentation.ts, so the UI shares the same alias, preset, and custom-field input mapping instead of carrying its own local webhook-field normalization fork. The destination editor also exposes resource-tag routing for email and each alert webhook. Alert producers must carry canonical resource tags into alert metadata: Proxmox VM/container tags remain literal, Docker container/service labels become key:value tags (or key for empty values), host-agent tags remain literal, and unified resources forward their canonical Tags. The alerts surface owns this producer metadata and routing UX; destination matching, persistence, and receipt-aware delivery remain notifications-owned. The same editor exposes a minimum-severity policy for email, Apprise, each webhook, and Relay mobile push. The alerts surface owns the coherent routing UX; notifications owns provider filtering and recovery receipts, while Relay owns privacy-safe mobile projection and persisted mobile routing policy. Email and Apprise expose all three destination choices: all alerts, warnings and critical alerts, or critical alerts only. Relay deliberately exposes its smaller persisted policy of all alerts or critical alerts only; hiding the warning option there is protocol honesty, not permission to collapse a stored warning floor on notification-owned destinations. The alert manager callback layer now also has to stay fan-out-safe. Monitor delivery, the unified alert bridge, and Patrol-adjacent AI listeners must compose through additive fired/resolved subscriptions instead of overwriting a single callback slot, and alert-triggered Patrol enqueueing must stay on the canonical unified alert bridge plus trigger-manager path rather than reviving duplicate callback-side Patrol shortcuts. That runtime callback boundary is now factored into internal/alerts/callbacks.go as same-package Manager support code so it can own callback mutexes, subscription IDs, legacy Set* slots, and fan-out snapshot helpers without creating an import cycle around the Alert runtime type. Alert configuration types, pure normalization, and resource-type identity helpers now live under internal/alerts/config/; the parent alerts package may re-export aliases and wrappers for compatibility, but consumer packages must keep importing internal/alerts unless they are explicitly taking ownership of alert configuration internals. Alert configuration runtime now lives in internal/alerts/config_runtime.go. That file owns UpdateConfig normalization and activation-state migration, global disable cleanup, active alert reevaluation after threshold changes, threshold override cloning and merge behavior, and hysteresis defaults; future config-driven runtime behavior should extend that owner rather than expanding the central Manager file. Active-alert persistence now lives in internal/alerts/active_persistence.go. That file owns active-alert save/load, secure active-alert storage leaves, startup restoration and legacy active-alert ID migration, and periodic active alert persistence; persistence changes should extend that owner rather than adding save/load logic back to the central Manager file. Persistent managers establish the SQLite alert store during construction, before escalation, periodic persistence, shadow evaluation, or monitoring can act on restored state. alert_active_state is the restart authority and is independent of event retention, so a still-active occurrence never disappears because it predates the 90-day history window. Fired, refired, acknowledged, unacknowledged, escalated, and resolved lifecycle transitions update that projection in the same transaction as their immutable lifecycle event. A resolution matches alert identity plus occurrence start time so a delayed old resolution cannot delete a newer occurrence that reused the canonical ID. Full checkpoints capture mutable fields such as LastSeen, LastNotified, and escalation state with revision compare-and-swap; a checkpoint that raced a lifecycle transition retries instead of overwriting it. SQLite uses full synchronous durability for this authority, and the recovery mirror fsyncs its temporary file plus a platform-native durable rename barrier (parent-directory sync on Unix and write-through replacement on Windows), so the contract covers host power loss rather than only orderly process restart. An unchanged JSON recovery checkpoint preserves already-correct directory permissions without issuing chmod, avoiding redundant Linux directory metadata mutation. It still repairs unsafe access permissions and special mode bits; permission repair must not replace identical JSON. Directory sync and atomic file replacement durability remain unchanged. This is not a guarantee of zero aggregate process writes. TestActiveMirrorUnchangedRepairsDirectoryPermissions in internal/alerts/alerts_test.go pins permission repair and inode retention; TestActiveMirrorUnchangedPreservesDirectoryMetadata in internal/alerts/active_mirror_metadata_linux_test.go pins stable Linux ctime.

active-alerts.json remains an atomic recovery mirror, not a competing healthy read authority. A new or recreated database imports the readable mirror. A failed SQLite checkpoint writes a durable degraded marker, and the next startup uses the mirror once to repair SQLite before clearing that marker. Without a degraded marker, an initialized SQLite projection wins over a stale mirror. Lifecycle append failure first checkpoints a lock-independent active-state projection to the recovery mirror synchronously, then writes the degraded marker. The projection is updated at the canonical set/remove and lifecycle event seams so event emission remains safe even when its caller holds the manager lock. A checkpoint that overlaps a lifecycle failure must observe the failure epoch and retry before it can restore SQLite authority or clear the marker; stale periodic work can never overwrite the failure recovery snapshot. An unreadable or malformed mirror is write-blocked and preserved for manual recovery even when healthy SQLite can continue serving current state; startup must never silently replace a recoverable source with an empty snapshot. Acknowledgement and incident age are never implicit resolution evidence: restart restores long-running and long-acknowledged occurrences until fresh healthy evidence or an explicit operator action resolves them. Tracking-map cleanup now lives in internal/alerts/tracking_cleanup.go. That file owns stale flapping, suppression, pending-alert, offline-confirmation, Docker tracking, rate-limit, recent-alert, acknowledgement, and stale active alert cleanup; future cleanup rules should extend that owner rather than mixing cleanup into resource evaluators. Active-alert lifecycle now lives in internal/alerts/active_lifecycle.go. That file owns acknowledgement and unacknowledgement, manual active-alert clearing, preserved alert state during rebuilds, poll-confirmed offline recovery clears, resolved-alert registration, and no-lock active-alert removal helpers; future active-alert lifecycle changes should extend that owner. Active-alert cleanup now lives in internal/alerts/active_cleanup.go. That file owns TTL cleanup, auto-acknowledgement cleanup, stale acknowledgement retention cleanup, node-retirement cleanup, and full active-alert state reset; future cleanup policy changes should extend that owner. The old central internal/alerts/alerts.go file is intentionally gone. The residual manager surface is now split by ownership: config_facade.go owns compatibility aliases and wrapper functions for the leaf config package, model.go owns alert runtime data structures and clone semantics, constants.go owns package-wide cleanup and storage constants, metric_hooks.go owns Prometheus integration callbacks, manager.go owns Manager state and construction, default_config.go owns the factory runtime configuration (DefaultAlertConfig), lifecycle.go owns shutdown, and escalation.go owns the escalation loop and escalation state mutation. Future changes must extend the owning file rather than reintroducing a central catch-all manager file. Alert notification policy now lives in internal/alerts/notification_policy.go. That file owns dispatch suppression, flapping suppression, quiet-hours suppression, monitor-only notification suppression, cooldown decisions, and per-alert rate limiting; future notification-gating changes should extend that policy owner rather than burying new checks inside metric or resource-specific evaluators.

Fresh installs load the factory alert configuration

DefaultAlertConfig in internal/alerts/default_config.go is the only declaration of factory alert settings. A new Manager starts from it, and LoadAlertConfig in internal/config/persistence.go returns it while alerts.json does not exist; persistence must not declare a second copy. The copy it used to return left out DockerDefaults, a value struct, so a fresh install loaded container CPU, memory and disk as trigger 0, UpdateConfig kept that as a deliberate Off, the thresholds page called the container Global Defaults Custom, and the startup identity migration wrote those zeros to alerts.json within a second of first boot. The missing-file config carries no activation state, because activation is a stored decision: node updates and imports reapply LoadAlertConfig, and UpdateConfig then keeps a running manager's state, while a new manager starts pending review. An existing alerts.json is never rewritten to factory values, because a stored trigger 0 is how a saved Off looks. The factory leaves AutoAcknowledgeAfterHours at 0: no settings control shows it, and the 24 hours it used to declare never reached an install. It leaves the quiet-hours timezone empty so the settings page offers the viewer's browser zone. Flapping detection and alert TTL cleanup are on in the factory and now reach fresh installs; a thresholds-page save still sends neither field, so it turns them off. TestLoadAlertConfigFreshInstallUsesFactoryConfig, TestLoadAlertConfigMissingFileKeepsRunningActivation and TestLoadAlertConfigKeepsStoredDockerDefaultsOff in internal/config/persistence_alert_fresh_install_test.go, plus TestDefaultAlertConfigLeavesFiringAlertsUnacknowledged and TestDefaultAlertConfigLeavesQuietHoursTimezoneUnset in internal/alerts/alerts_test.go, pin this.

Agent disk temperature alerts clear when their threshold is off

Host-agent SMART disk temperature alerts (diskTemperature, resource agent:<host>/disk_temp:<device>) follow one effective threshold: the resolved agent DiskTemperature, refined by the disk type's DiskTempByType entry unless an explicit host or linked-resource override set it (hostDiskTemperatureThresholdNoLock in internal/alerts/host.go). When the host's resolved DiskTemperature is off, CheckHost clears every disk temperature alert on the host. A disabled DiskTempByType entry clears only the alerts of disks of that type, through the disabled metric spec. The all-agents switch, a disabled host override, agent removal and confirmed agent offline clear them with the host's other disk alerts. Before this, nothing cleared them once the agent-level threshold was off.

A config save judges an open disk temperature alert against the same threshold (resolveHostAlertThresholdsNoLock refines it by the alert's diskType metadata, and metric_runtime.go classifies diskTemperature as a threshold metric), so turning the host's DiskTemperature off resolves the alert immediately. An alert persisted before CheckHost recorded diskType (May 2026) carries no disk type until its next firing evaluation, so a save judges it against the lowest enabled trigger any of the host's disks can use: the agent threshold or a DiskTempByType entry (lowestHostDiskTemperatureThresholdNoLock). A save therefore cannot resolve an alert its disk type would still fire.

For a metric alert that reaches the threshold comparison in reevaluateActiveAlertsLocked (docker-host alerts return before it), a config save resolves an alert whose reading is below the new trigger. That includes readings inside a valid recovery band, which the evaluator would hold. A clear level at or above the trigger is no recovery band (buildCanonicalMetricSpec drops it), and a reading at the trigger stays firing, as it does in the reducer, so a save no longer resolves an alert the next evaluation raises again. TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff, TestHostDiskTemperatureAlertsClearWhenAgentLeaves, TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold, TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType and TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand in internal/alerts/host_unraid_lifecycle_test.go pin these paths.

The disk temperature policy decides disk heat

The policy CheckHost applies to a SMART disk (the agent Disk Temp default, refined by the disk type's DiskTempByType entry, and switched off for every type when that default is off) is the canonical answer to "is this disk too hot". diskTemperatureThresholdForType in internal/alerts/host.go holds it. Manager.HostDiskTemperatureThreshold(host, diskType) resolves it for the host agent that reports a disk exactly as CheckHost does: an explicit Disk Temp override on that host, or one inherited from its linked node or guest, replaces the per-type entry, the first override in that chain decides even when it sets no Disk Temp, and an override that switches the host's alerts off leaves no threshold. TestHostDiskTemperatureThresholdMatchesCheckHostOverrides pins each case against what CheckHost fires, including an override under the canonical resource the agent ID resolves to and one inherited from a linked guest. Manager.DiskTemperatureThreshold(diskType) is the hostless case. Patrol reads the policy through AlertThresholdAdapter.GetDiskTemperatureThreshold. DefaultDiskTemperatureThreshold serves callers with no manager. Disk risk (internal/storagehealth) judges no temperature. The Physical Disks Temp cell and Health verdict mirror the policy in resolveDiskTemperatureDisplayThresholds under the override keys of the machine whose agent reports the disk, in the order CheckHost reads them as far as the browser can see that chain (storage-recovery.md lists where it cannot). That resolver returns null when the agent Disk Temp default is off or an override or agent default switches the host's alerts off. The global alerts switch and the agent alert-type switch stop CheckHost without changing the policy. The displays that read it treat null as off: no threshold colour (getTemperatureTextClass with diskTemperature) and never hot. A DiskTempByType entry cannot be switched off on its own, because normalization restores a non-positive entry to its default. A disk is hot from its trigger. Alerts and Patrol findings stay open until the reading falls to the clear value, or under the trigger when there is no band below it. The Physical Disks verdict and the TrueNAS Health cell judge only the current reading against the trigger. PDF reports colour disk temperatures by the same thresholds. TestDiskTemperatureThresholdMatchesCheckHostPolicy in internal/alerts/threshold_resolution_shared_test.go pins per-type resolution, a raised NVMe trigger that CheckHost also honours, the copy, the disabled default and the nil manager.

TrueNAS disk temperature alerts follow the policy too: trueNASDiskTemperatureDefaultNoLock (internal/alerts/truenas_disk_temperature.go) is the type-default tier of effectiveAlertPolicyNoLock for truenas-disk, so an unset TrueNASDiskDefaults.Temperature resolves per disk type (the alertPolicyQuery.DiskType the unified input carries from PhysicalDisk.DiskType). A TrueNAS-wide value the user saved under TrueNAS Disks replaces it for every TrueNAS disk, as a host Disk Temp override does for an agent's disks, 0 meaning off, and a per-disk override beats both. The alert records diskType so a config save re-judges it per type; one without it is held to the lowest per-type trigger. A retained reading (diskinventory.TemperatureCollected false) is no evidence and raises nothing. The factory value is unset. normalizeTrueNASDiskTemperature drops a stored 55/50 only from a config without TrueNASDiskTemperatureByType, because the thresholds page wrote that flat factory value back on every save; the page writes the marker, so a 55 typed later is kept. Every config GetConfig returns carries the marker, so a client that reads and writes back keeps it. Without the marker (an old saved file, or a page loaded before the upgrade) 55/50 is read as that unchosen factory value; a client building a config from scratch with an explicit TrueNAS-wide 55 has to send the marker. The TrueNAS Disks Global Defaults cell reads By type while unset (globalDefaultFallbacks on ResourceTable, resolveAlertResourceGlobalDefaultCell), or Off when the agent Disk Temp default switches the policy off, and switching it on then stages an explicit TrueNAS-wide 55 because unset would stay off. Each disk row inherits its type's trigger from the unsaved editor state (resolveTrueNASDiskTemperatureDefault). The TrueNAS storage table and disk drawer now judge current disk heat through getTrueNASDiskTemperatureThresholds(disk): the disk's canonical-ID temperature override, then a TrueNAS-wide value, then the per-type policy. Disabled disks or TrueNAS Disks defaults and explicit Off values leave readings unjudged; retained values never count as current heat. The API-only disk does not inherit a host Disk Temp override or the agent alerts switch. Patrol/AI use the disk-specific backend resolver described below; the frontend and backend retain their own verification boundaries. TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy and TestTrueNASDiskTemperatureAlertIgnoresRetainedReading in internal/alerts/unified_eval_test.go pin the tiers and retained readings; TestTrueNASDiskTemperatureAlertsReevaluatePerDiskType and TestTrueNASDiskTemperatureAlertWithoutDiskTypeHeldAtLowestTrigger in internal/alerts/alerts_test.go pin re-judging on a config save; TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues pins the saved-config migration.

Proxmox physical disks alert on temperature under the same policy. CheckProxmoxDiskTemperature (internal/alerts/proxmox_disk_temperature.go) raises a diskTemperature metric alert under the disk's PVE alert reference (ProxmoxPhysicalDiskAlertResourceID, resource type proxmox-disk), judged by proxmoxDiskTemperatureThresholdNoLock: the disk type's DiskTempByType entry under the agent Disk Temp default, which switches it off for every type. It carries the disk's serial, WWN and diskType like the health and wearout alerts, so ProxmoxPhysicalDiskAlertIdentifiers lists its <reference>::metric-threshold:diskTemperature identifier for history ownership, and reevaluateActiveAlertsLocked re-judges it per disk type (the lowest per-type trigger when the type was never recorded) instead of falling through to guest thresholds that have no disk temperature. A disk a still-reporting linked agent lists in its SMART report is the agent's: its alert closes with the moved_to_agent resolution and CheckHost raises the agent's own. The handover happens on the next disk poll, so for that one interval an agent newly linked to a disk with an open PVE alert can hold its own alert beside it. A lapsed agent owns nothing, so the node's own readings keep alerting while its agent is silent. An excluded disk, or a policy switched off, closes the alert; a poll without a current reading holds it. Like the health and wearout alerts, a disk that leaves the inventory keeps its alert until the 24-hour stale sweep. The proxmox-disk alert delay defaults to the factory 5 seconds, so a warning waits out the same 300-second noisy-gauge stability window agent and TrueNAS disk temperature warnings do, and a critical reading bypasses it. TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy, TestProxmoxDiskTemperatureAlertOwnershipAndHolds, TestProxmoxDiskTemperatureWarningWaitsOutStabilityWindow, TestProxmoxDiskTemperatureAlertsReevaluatePerDiskType and TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity in internal/alerts/alerts_test.go pin the policy, ownership, holds, re-judging and identifiers.

Agent disk temperature alerts clear when their disk leaves the report

CheckHost also clears a disk temperature alert, and drops its pending threshold run, once its device is missing from three consecutive non-empty SMART lists (cleanupHostDiskTemperatureAlerts, a seen map keyed by resource ID like cleanupHostCustomSensorAlerts). A removed, replaced or renamed disk never sends the reading that would resolve its alert, and shouldPreserveAlertOutsideNodeCleanup exempts agent: alerts from node cleanup, so before this only the 24-hour stale-alert sweep in cleanupStaleMaps resolved it. One omission is not proof the disk left: the Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose probe fails, so the count restarts whenever the disk is reported again. Three reports in a row of failed probes on such a disk still clear its alert.

An empty SMART list skips this cleanup and does not count toward the three, because the agent omits the list when SMART collection fails or is unsupported. A listed disk in standby or without a temperature also holds its alert, since neither shows the disk cooled: standby reports avoid waking the disk, and most failed Linux probes still list the disk with identity only. SMART health alerts hold through standby the same way. A held alert resolves through the normal recovery path, including any configured recovery delay, once the disk reports cool readings again; with no evaluation for 24 hours, the stale-alert sweep resolves it. Turning the threshold off, or removing the agent, drops the host's pending disk temperature runs and absence counts as well as its alerts. TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport in internal/alerts/host_unraid_lifecycle_test.go pins the clear, the holds, the restarted count and the pending-run and count cleanup.

Proxmox node cleanup keeps platform metric alerts

CleanupAlertsForNodes runs on every Proxmox node poll (and every mock alert tick) and removes alerts whose Node is not a current Proxmox node or PBS instance. Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or platform host in Node and a display label such as TrueNAS Disk or Kubernetes Pod in metadata.resourceType, which the preserve list's platform words never matched. The cleanup used to delete them silently on each poll while their reducer incident stayed firing, so the next evaluation re-created each one and handed it to the alert callback again as a new alert. shouldPreserveAlertOutsideNodeCleanup now keeps any alert carrying metadata.platformType, or whose label maps through CanonicalAlertResourceType to a Kubernetes, TrueNAS or vSphere type. TestCleanupAlertsForNodesKeepsPlatformMetricAlerts in internal/alerts/alerts_test.go pins both signals, and its sibling pins that a guest alert on a removed Proxmox node is still removed.

Agent disk temperature alerts judge the reading the disk shows

CheckHost judges each agent disk on the temperature, collection state and disk type the disk shows, read by unifiedresources.HostDiskTemperatureReadings from the same adapters the registry builds the disks with. A SMART row shows its own reading or, when the agent's probe returned none, the one the host's Unraid inventory reports for that disk (HostSMARTDiskTemperature), and keeps its agent:<host>/disk_temp:<device> resource. An Unraid inventory row whose disk key no SMART row carries, a disk only the inventory lists, shows that row's reading (HostUnraidDiskTemperature) and its own transport as disk type, under its kernel block device token, the resource a plain Linux SMART row for the same device uses. A SMART row and an Unraid row with one key are one registry disk even under different device labels, such as a controller member (0 [megaraid,0]) and its Unraid device (sda), and alert once, under the SMART row's resource. When smartctl and Unraid name one device with different serials the registry shows two disks on it, and the device's one alert judges the reading that stands highest against its own disk type's threshold (judgedHostDiskTemperature): over its trigger first, then inside its recovery band, then the larger margin over the trigger, with a tie keeping the reading listed first. Before this CheckHost read only the SMART row's own field, while the Physical Disks Temp cell, the Running Hot verdict ("at or above its alert threshold") and Patrol judged the Unraid reading against the same trigger, so such a disk could read as at its alert threshold with no alert. Unraid's own warning and critical disk temperature notifications do not make this redundant: Pulse already alerts on the smartctl readings and array state of the same hosts, which Unraid also notifies on, and a user who routes alerts through Pulse sets these thresholds here.

Only a reading collected now fires or resolves (HostDiskTemperatureReading.Collected: no standby row, and diskinventory.TemperatureCollected), so a spun-down disk holds the alert like a listed disk without a temperature. A disk only the inventory lists whose device matches the agent's --disk-exclude patterns (fsfilters.MatchesDeviceExclude) raises nothing, and an alert it raised before the exclusion resolves on the next report (clearHostDiskTemperatureResources), whatever the SMART list holds, unless another listed reading uses the same resource: docs/UNIFIED_AGENT.md documents exclusion as removing the disk from monitoring, and linked Proxmox hosts already judge an excluded disk healthy. A disk only the inventory lists is also judged in a report whose SMART list is empty, because the disk shows the inventory's reading in that report. The absence cleanup above still runs only on a non-empty SMART list, and such a report neither counts toward nor restarts a disk's three, so a disk that leaves the inventory while every report's SMART list stays empty keeps its alert until the 24-hour stale-alert sweep. TestCheckHostJudgesTheUnraidTemperatureADiskShows and TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture in internal/alerts/host_unraid_lifecycle_test.go pin the fallback, the inventory-only disk, the empty SMART list, two disks on one device (also of different types), a controller member under another device label, standby, exclusion, the disk type and host override thresholds, the holds, the handover to a SMART reading, departure, and the exclusion clear with and without SMART rows and beside a listed SMART row. TestJudgedHostDiskTemperatureOutranks pins the ranking.

Configured flapping thresholds remain reachable

Every accepted positive FlappingThreshold, including values above ten, must be reachable by current observations inside FlappingWindowSeconds. Per-key history retains at most max(10, FlappingThreshold) timestamps after window pruning; it must not grow with repeated attempts during an active cooldown. Only the threshold-crossing attempt opens that cooldown. Later attempts keep its original deadline even after the observation window drains, and a served cooldown resets the episode so a later storm can arm it again. Lowering the threshold applies the new history bound and threshold on the next check. The read-only delivery diagnosis must explain the same threshold and suppression actually enforced at dispatch. internal/alerts/flapping_threshold_test.go pins these boundaries through normal configuration updates and notification callbacks, alongside the existing flapping cooldown and one-shot callback tests.

Ordinary retention cleanup and hourly tracking cleanup retire a served suppression deadline, its flapping latch and its observations atomically under the manager lock, using the same expiry rule as dispatch. Neither sweep may leave an active occurrence in unbounded flapping suppression after deleting its deadline. The next burst starts a fresh observation window and can arm a new bounded cooldown and one-shot callback. Unexpired cooldowns, other keys' policy and pending bursts without a deadline remain unchanged; expiry does not clear, acknowledge or change the identity of an active alert. The cleanup regression controls in internal/alerts/flapping_threshold_test.go exercise both sweeps, drained and retained windows, dispatch callbacks and delivery diagnosis. These modeled-time controls are not installed notification-destination acceptance.

Default thresholds keep a positive trigger's clear below it

PBS, node temperature, agent, Kubernetes, TrueNAS and vSphere global defaults go through normalizeThresholdPointer in internal/alerts/config/normalize.go from UpdateConfig. The shared persistence normalization in internal/config/persistence.go applies the same normalizer, through NormalizeHysteresisThreshold, to the agent, node temperature and storage pairs. A missing or negative threshold takes the factory default and a zero trigger is off. A positive trigger always keeps a clear below it: a missing clear sits five points under the trigger, floored at 0, and a clear at or above the trigger is repaired. Guest and node usage, storage and Docker defaults hold the same bound through EnsureValidHysteresis in ValidateHysteresisThresholds, NormalizeStorageDefaults and NormalizeDockerThreshold. The canonical per-type entries (nvme, sata and hdd in DiskFillByType; nvme, sas and sata in DiskTempByType, the types the thresholds page edits) keep a positive trigger and follow the same clear rule; a non-positive trigger resets the entry to its type default, because the page cannot switch a type off on its own. Other per-type keys are stored as written.

The thresholds page sends max(0, trigger - 5), so a 1-5% default arrives with clear 0 and must not fall back to the factory clear. Agent defaults used to do that, storing {trigger: 1, clear: 75} for a 1% Machines CPU default, and per-type disk entries replaced the whole pair, trigger included. The evaluator already ignored a clear at or above the trigger (buildCanonicalMetricSpec drops it and the reducer then clears at the trigger), so those alerts fired and resolved the same way; /api/alerts/config served the factory clear and firing alerts reported it as clearThreshold. A repaired pair above the margin, such as {50, 80} written through the API, now has a real recovery band at 45 where it used to clear at the trigger. internal/alerts/config/normalize_low_trigger_clear_test.go pins every default family and both per-type maps, TestUpdateConfigKeepsLowTriggerClearBelowTrigger in internal/alerts/config_validation_test.go pins the saved config and the firing alert's clear level, and the low-trigger and stored-clear tests in internal/config/persistence_test.go pin the written file and load repair.

Monitor-only delivery is terminal

Monitor-only alerts remain visible, but neither a firing nor a recovery notification may enter the delivery queue. They must not acquire quiet-hours replay metadata or schedule escalation levels and critical repeats. An already queued escalation must be revalidated against the current monitor-only state before delivery; the read-only delivery diagnosis reports monitor-only suppression rather than a replayable quiet-hours deferral.

The same policy owner also exposes the read-only alert delivery diagnosis projection used by /api/alerts/delivery-diagnosis; that projection may explain current gating state, quiet-hours replay timing, cooldown timing, rate-limit counts, and flapping suppression, but it must not dispatch callbacks or mutate flapping/rate-limit tracking maps. Omitting the alert identifier returns one diagnosis for every active alert from one manager read pass, ordered by alert identifier, and an empty active set is encoded as [] rather than null. The Alerts Overview consumes that bulk projection once per refresh and may render its current delivery outcome on an unacknowledged active-alert card. Missing or failed diagnosis reads degrade to no delivery line; they must not hide alert truth or trigger per-card requests. internal/alerts/eventlog/ and internal/alerts/event_emission.go own the canonical append-only alert event record. Persistent managers enable a SQLite-backed store under the alerts data directory; ephemeral managers record nothing unless a store is installed explicitly. Resolution, acknowledgement, unacknowledgement, escalation, flapping detection, dispatch, quiet-hours deferral, and suppression append immutable events without changing lifecycle or delivery behavior. Suppression and deferral are recorded as outcome episodes: the first decision is immutable, identical reevaluations of that unchanged outcome append no duplicate row, and a changed reason, details, resource presentation, intervening lifecycle/dispatch event, or later return to that outcome opens a new episode. Lifecycle, escalation, and dispatch events are never coalesced. This keeps delivery activity explanatory instead of poll-frequency-shaped, prevents unchanged reevaluations from crowding a real outcome change out of the non-blocking diagnostic buffer, and bounds diagnostic growth without erasing a meaningful decision transition. A failed write forgets its admission key so a recovered store can accept the outcome again. Reads apply the same episode projection to redundant diagnostic rows written by older versions, so upgraded installations become readable immediately without rewriting their immutable event records. Snapshot-bearing lifecycle transitions commit synchronously before downstream lifecycle projections run; they never share the droppable diagnostic buffer because alert history is reconstructed from them. High-volume notification decisions remain non-blocking and fail-open for alert evaluation: a full diagnostic buffer counts and drops that delivery evidence. Failed asynchronous batches do not advance the successful-write counter, and Flush must report the failure or timeout rather than falsely claiming the batch landed. An unavailable or failed lifecycle store degrades history reads to the recovery model without hiding live active-alert truth. Events retain for 90 days and prune hourly. Retention alone does not bound disk use: one flapping alert can write thousands of full alert-state snapshots inside the window, so the hourly prune also enforces a total stored snapshot-payload cap and removes the oldest events first when it is exceeded. A volume prune raises the same retention revision as age pruning, so the history projection rebuilds from the remaining log rather than reusing a stale fold. Fired and refired lifecycle events come only from the reducer core's explicit activation events: canonical lifecycle reactivation maps EventRefired separately, while shared metric activation records one fired event when its pending incident becomes firing. Repeated firing observations and persisted-alert restore must not append another activation event, and the active-alert storage funnel must never invent one. Event-log reads accept a caller limit but normalize it to the store's bounded maximum before it reaches SQLite. Result-slice allocation is independent of that caller value: a request-provided limit is a row-count preference, never a memory-allocation hint. This remains defense in depth even when an authenticated API handler validates the query parameter, because non-HTTP manager callers use the same store boundary. Alert-history reconstruction is the explicit exception to a single bounded query: it walks every matching lifecycle event oldest-first through bounded keyset pages, releasing the SQLite rows between pages. The public/API query cap therefore remains intact while a noisy installation with more than 1,000 lifecycle events in the requested window cannot silently lose older history. Lifecycle events now carry a full alert snapshot and internal/alerts/history_projection.go can fold those snapshots into one history row per occurrence. Existing pre-snapshot SQLite stores upgrade in place, and rows without snapshots remain readable but cannot contribute to the projection. The event log becomes the alert-history read authority only after legacy history is absent or its complete import and retirement succeeds. Managers without an authoritative event store, and stores that report a lifecycle write failure, retain the in-memory JSON-history model as their fail-soft read fallback. internal/alerts/history_projection_parity_test.go characterizes the event-log projection against that fallback for active, resolved, acknowledged, multi-resource, and repeated occurrences. Live active-alert state overlays the latest projected snapshot so acknowledgement and current values do not become stale. internal/alerts/history_migration.go imports every legacy JSON entry through the event log's synchronous, non-droppable import path before renaming the fixed history and backup files to their .imported backups. The primary or backup source's presence keeps migration active until both leaves retire. Imported events are retry-idempotent by immutable alert ID, occurrence time, and snapshot, so a committed SQLite transaction followed by a failed or partial file rename cannot duplicate history on the next startup. An import failure leaves JSON history authoritative for the next attempt. An unreadable or malformed source, or any entry that cannot produce an identified complete snapshot, blocks both retirement and further JSON writes so startup cannot silently replace recoverable user history with an empty file. Successful retirement stops further JSON writes. Retired .imported leaves remain bounded recovery sources: startup loads and idempotently replays them if events.db is recreated, so loss or quarantine of the database cannot turn a successful migration into silent empty history. Clearing history appends a history_cleared tombstone rather than deleting log rows; the projection ignores earlier lifecycle events but overlays still-active alerts as current state. Unified-incident and system-alert activation must emit snapshot-bearing fired events so their projected history does not begin at acknowledgement or resolution. SubscribeLifecycleCallback is the delivery-independent projection seam for that canonical stream. It runs for lifecycle transitions regardless of activation, quiet hours, grouping, rate limits, or destination state; delivery callbacks remain policy-controlled consumers. Monitoring uses this seam to materialize canonical resource-history breadcrumbs and incident shells. On startup it replays durable lifecycle transitions and migrated legacy-history snapshots oldest first, then idempotently reconciles restored active alerts that predate the event store. Because monitor construction precedes attachment of the API-owned durable unified-resource store, that attachment repeats the same idempotent replay so canonical resource history is repaired as well as the incident fallback cache. A migrated final snapshot expands only facts it actually carries: the occurrence start, saved acknowledgement, and known end; it does not invent delivery, escalation, analysis, or remediation events. The incident store retains those snapshot events as a fallback only when the canonical resource timeline lacks the equivalent event, so a partial durable projection cannot erase the occurrence start while complete canonical history still wins. The occurrence-qualified incident API performs the same repair from the active/history read model when retention or an older migration left a shell absent or empty. This repairs resolved as well as active incident timelines without replaying any delivery side effect, and ensures an alert already visible in Overview or History cannot keep returning an unavailable timeline merely because its notification was held or it predates the event store. Occurrence lookup allows only one second of start-time precision drift; it must not merge recurring or flapping incidents merely because they share an alert identifier and started within the same former ten-minute window. Snapshot events stored at equal timestamp precision order by lifecycle meaning, with detection first and resolution last, rather than by generated event ID; a malformed legacy end before its own start clamps to the start boundary. The same dispatch policy owns firing-notification evidence on active alerts: any alert that passes notification suppression and enters the fired callback fan-out must carry LastNotified before the callback clone is emitted. Resolved notification gating depends on that field to distinguish alerts whose firing notification actually entered delivery from alerts that were never sent. Alert read-side output now lives in internal/alerts/read_model.go. That file owns active alert projection and sorting, metadata coercion helpers, recently-resolved and history output wrappers, and notify-existing redispatch; future output-ordering or metadata coercion changes should extend that owner rather than adding another read path inside resource-specific evaluators. PMG alert evaluation now lives in internal/alerts/pmg.go. That file owns PMG connectivity evaluation, PMG queue and per-node queue checks, quarantine growth tracking, and mail-rate anomaly detection; future Proxmox Mail Gateway alert behavior should extend that resource checker owner rather than adding more PMG logic to the central Manager file. Docker alert evaluation now lives in internal/alerts/docker.go. That file owns Docker host connectivity, container state and health, container metric projection, service gap/update-state checks, image-update timing, and Docker tracking cleanup; future Docker alert behavior should extend that resource checker owner rather than expanding the central Manager file. It must not keep shadow last-exit-code state or infer an OOM kill from exit 137; the accepted container model's nullable runtime-authored OOMKilled field is the sole OOM classification input. PBS alert evaluation now lives in internal/alerts/pbs.go. That file owns PBS connectivity normalization, PBS metric projection, PBS metric cleanup, and PBS offline lifecycle handling; future PBS alert behavior should extend that resource checker owner rather than expanding the central Manager file. Storage alert evaluation now lives in internal/alerts/storage.go. That file owns storage connectivity handling, storage usage projection, ZFS pool/device health checks, and storage offline lifecycle handling; future storage alert behavior should extend that resource checker owner while shared storage-health assessment helpers remain package-level until host and storage health paths are separated cleanly. Storage connectivity status is trimmed and case-normalised before evaluation. Empty or unknown status is absent connectivity evidence: it must not advance recovery or replace an existing incident's identity/start time. Capacity and pool-health evaluation remain independent and may still use valid observations. Normalised offline and unavailable observations do not feed capacity evaluation, preserving the existing rule that offline capacity is not actionable. The existing non-failing treatment of available, online, active, inactive and disabled statuses remains unchanged; explicit connectivity-disable policy still clears the incident. TestStorageUnknownConnectivityDoesNotRecover and TestStorageKnownConnectivityRecoveryCompatibility in internal/alerts/alerts_test.go pin unknown-observation preservation, independent capacity activation, normalised offline activation and confirmed recovery. Proxmox node alert evaluation now lives in internal/alerts/node.go. That file owns node metric and temperature projection, node offline lifecycle handling, and instance-scoped node display-name cache updates; future Proxmox node alert behavior should extend that resource checker owner rather than expanding the central Manager file. When a reporting host agent is linked to a node (LinkedNodeID, the monitor's automatic or operator-set identity decision), the agent resource owns each CPU, memory or disk usage metric it actually evaluates, and CheckNode releases its own copy of those metrics through the disabled-threshold path every cycle: the pending run is dropped and any node alert still open from before the link is resolved, never left frozen until the agent goes offline. CheckHost applies its link outcome after evaluation in per-agent report order. CPU and memory ownership requires a live threshold plus a usable, finite observation with a ready evaluation window, or a still-open agent alert or reducer run. For disk, that evidence must belong to the summary filesystem models.SummaryDisk picks, which is also what the linked node's disk metric reports. Configuration alone is not coverage: an agent with no observation and no open run leaves the node responsible. Missing readings keep an existing agent alert, without creating a node duplicate; observation-gap interruption and fresh confirmation/recovery remain unchanged. Disabling a usage metric releases canonical/shadow pending runs and explicit intent grace, even without a reading. A disabled agent or host also releases its CPU/memory pending runs. UpdateConfig immediately revokes unsupported link coverage, and in-flight reports recheck current policy before applying their outcome. Newer reports, offline and removal invalidate older link updates and removals. Several linked agents contribute the union of their coverage; stale alert metadata alone never suppresses a node. The accepted short duplicate window remains restart-before-first-report and first-offline-before-confirmed-offline; manually linked sibling agents retain their own alerts. Explicit host/inherited disk overrides beat DiskFillByType, as do direct filesystem overrides. A saved direct filesystem disable revokes summary disk ownership without a new agent report. Per-type defaults retain the existing normalization contract (zero is restored to the type default); this repair does not add a per-type Off setting. A nil effective filesystem threshold drops its pending run.

A node-side close is an ownership handover, not observed recovery: each released metric carries moved_to_agent and the identity/name of an agent actually covering that metric in the same link snapshot. If several agents cover it, the lowest covering agent ID wins deterministically; a sibling covering only another metric must not be named. Resolved consumers still receive the close. internal/alerts/host_agent_node_links.go owns ordered links and per-metric successor selection, without persisting stale coverage across restart. Deduplication keys on that link, never on a hostname match, so a same-named node in another instance keeps its alerts, an agent reporting an FQDN still dedups its node, and an operator unlink hands the alerts back. CPU temperature has no host-agent metric, so the node keeps evaluating it whether or not an agent is linked; the node poll already merges agent sensor readings into node.Temperature, giving one temperature alert per machine. A missing CPU reading is not evidence: with a live trigger, checkNodeTemperature neither feeds 0°C nor resolves the open alert, but interruptMetricRun (reducer InterruptMetricRun) drops a pending activation run and restarts a recovery run, so no sustained-for or recovery delay spans a gap in readings; a disabled threshold still clears. Config-save reevaluation classifies alerts whose metadata resourceType is node as node alerts and judges them against node thresholds; the Instance heuristic only covers legacy alerts without a resource type, because current node alerts keep the PVE instance name in Instance. TestCheckNodeKeepsTemperatureAlertWhenHostAgentMonitorsNode, TestCheckNodeReleasesOpenMetricAlertWhenHostAgentRegisters, TestCheckNodeMissingTemperatureDoesNotResolveOpenAlert, TestCheckNodeMissingTemperatureInterruptsTimingRuns, TestConfigSaveKeepsNodeTemperatureAlertOverTrigger, TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate, TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate, TestCheckNodeKeepsMetricsTheAgentCannotEvaluate, TestDisabledAgentThresholdReleasesPendingRunWithoutReading, TestHostAgentNodeLinkFollowsConfigAndReportOrder, TestCheckNodeTakesBackMetricsWhenAgentUnlinks, TestHostDiskOverrideBeatsDiskFillByType, TestReleaseCanonicalMetricAlertClearsIntentPending in internal/alerts/threshold_resolution_shared_test.go and TestReportedAlertOwnershipComposition in internal/alerts/reported_alert_composition_test.go, plus TestHostAgentDeduplicationFollowsNodeLink in internal/alerts/host_dedup_test.go and TestCheckMetricReportsWhetherObservationWasUsable in internal/alerts/windowed_metric_test.go, pin these rules. The real monitor lifecycle/History route is exercised by TestMonitorReportedAgentHandoverUsesActualMetricOwner in internal/monitoring/monitor_alert_handling_test.go. Host-agent alert evaluation now lives in internal/alerts/host.go. That file owns host identity, host-agent metric projection, host disk/SMART/RAID/Unraid health handling, host cleanup, and host offline lifecycle handling; future host agent alert behavior should extend that resource checker owner while shared health-assessment evaluation remains package-level until all storage-health callers can be separated behind a narrower owner. The host offline lifecycle is raised only by a lapsed agent report, and Pulse cannot tell a dead host from an agent that lost its route to Pulse. Its message therefore names the silent agent ("Pulse agent on '' stopped reporting", using the agent's own name without the linked (Host Agent) qualifier) and never claims the host is offline, because a linked Proxmox node keeps reporting through its API while its agent is silent. The host-offline type, alert identity and lifecycle stay unchanged. TestHandleHostOfflineNamesSilentAgentNotHostOutage in internal/alerts/alerts_test.go pins this wording. Host long-running storage-operation alerts require fresh operation evidence. An accepted Unraid cancellation/completion or terminal Linux RAID report resolves the corresponding operation alert immediately. A normal polling gap does not imply completion; after the monitoring-owned reporting lease expires, alerts must re-evaluate with only transient operation/progress fields cleared, retain alerts backed by static degraded/topology evidence when that last-known topology remains available, and progress the separate confirmed host-connectivity alert. Persisted operation alerts also obey the durable accepted-telemetry lease after server restart, so restart cannot make a stale parity or rebuild alert immortal and missing telemetry is never represented as healthy reporting. Snapshot and backup-age alert evaluation now lives in internal/alerts/backup_snapshot.go. That file owns snapshot age/size evaluation, backup rollup age evaluation, backup inventory readiness, PVE template subject matching, namespace disambiguation, and snapshot/backup active alert cleanup; future backup or snapshot alert behavior should extend that owner rather than expanding the central Manager file.

Backup-age attribution follows live identity

Backup rollup subject refs are historical hints, not unconditional proof of a guest's current placement. An exact instance:node:vmid ref is authoritative only while it resolves to a live guest of the same type. Otherwise backup-age evaluation retries typed VMID attribution and prefers live candidates carrying a resource ID over last-known display metadata. A PVE-owned ref can still name its authoritative orphan after the PVE inventory is ready; a stale PBS ref that cannot be resolved uniquely remains unattributed instead of pinning an alert to an old or wrong PVE node. TestCheckBackupsRemapsStaleSubjectRefToUniqueLiveGuest and the PVE orphan/inventory-readiness tests in internal/alerts/alerts_test.go pin both sides of this boundary. Per-guest backup and snapshot overrides are sparse, not frozen copies. A zero-valued threshold field in an override (warningDays, criticalDays, freshHours, staleHours, and the snapshot size pair) inherits the current global default at evaluation time through the merge helpers in internal/alerts/backup_snapshot.go; explicit non-zero values win, and alertOrphaned plus ignoreVMIDs always resolve from the globals because they are instance-wide filters. NormalizeRecoveryOverrides in internal/alerts/config/normalize.go (called from UpdateConfig in internal/alerts/config_runtime.go and from the shared persistence normalization in internal/config/persistence.go) rewrites stored overrides whose threshold tuple exactly matches the current globals into sparse enabled-only form. Those full copies were artifacts of the legacy per-guest toggle, and the equality condition makes the rewrite behavior-preserving at the moment it runs while letting the guest track later global edits (#1126). Overrides whose thresholds differ from the current globals must be preserved: persisted values alone cannot distinguish deliberate per-guest settings from a legacy copy made before the globals changed. A global-settings screenshot does not establish the effective guest threshold. TestBackupDivergentDefaultsOnLoad in internal/alerts/alerts_test.go verifies JSON-decoded configuration with 33/34-day globals: absent and enabled-only overrides inherit the globals, while a retained 7/14-day override still fires a seven-day warning. This is configuration-path evidence, not a reproduction of a reporter installation or a migration that guesses operator intent. TestGuestBackupOverrideInheritsGlobalThresholds, TestFrozenBackupOverrideMigratesToSparse, TestDeliberateBackupOverridePreserved, and TestMergeSnapshotOverrideInheritsZeroFields in internal/alerts/alerts_test.go pin the merge, migration, and preservation behaviors. Backup-age alerts are guest-workload scoped. A PBS recovery rollup whose subject carries no guest VMID is a host/config backup (backup-type host, keyed by the node name), not a guest workload, and must not raise a guest backup-age alert: the Backups overview already excludes those entries from guest coverage, and a stale host config backup otherwise notified forever under a node-named subject while the node's guests were backed up. Evaluation skips PBS subjects with an empty VMID, retaining their recovery point and rollup so inventory and artifact display are unchanged. PVE host backups already produce no subject and no alert. Genuinely orphaned PBS guest backups (numeric VMID) and PMG node backups still alert. TestCheckBackupsIgnoresPBSHostBackupSubject and TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup in internal/alerts/alerts_test.go pin both sides of this boundary (#2136). Proxmox disk health alert evaluation now lives in internal/alerts/disk_health.go. That file owns Proxmox disk canonical identity, disk health assessment alerts, known-firmware health suppression, and SSD wearout alerts; future Proxmox disk-health behavior should extend that checker owner rather than expanding the central Manager file. PVE disk-health and wearout occurrences retain the existing instance:node:disk:sanitized-device-path resource reference. The shared constructor in unified resources must produce the same identity as the alert checker; changing occurrence IDs to match registry source IDs would reopen historical incidents and can re-notify a worn disk. The registry resolves this reference as an alias of the exact physical disk for operator-state lookups. Muting one physical disk therefore resolves its active wearout alert and blocks later writes and notifications without muting another disk or changing the wearout recovery rules. TestProxmoxDiskCanonicalResourceIDTrimsIdentity pins the persisted identity shape; monitoring's registry-backed TestProxmoxPhysicalDiskMuteResolvesAndSuppressesWearoutAlert pins the policy path. These are source invariants, not evidence of installed field relief. Each PVE disk alert records the evaluated disk's serial and WWN (unifiedresources.MetadataDiskSerial, MetadataDiskWWN), and resource history owns each lifecycle row by that identity rather than by the path. A row without a usable serial or WWN names only an identity-less disk at its path, and a row whose identity is ambiguous, or names no disk but one another resource already holds, stays under the path reference. Reads by the path reference find those rows through the alert identifiers ProxmoxPhysicalDiskAlertIdentifiers lists, so spec IDs and occurrence IDs must keep matching it; TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity in internal/alerts/alerts_test.go pins both. Shared metric threshold runtime now lives in internal/alerts/metric_runtime.go. That file owns metric threshold lookup, per-metric delay and intent resolution, reducer input composition, active-alert projection, delivery side effects, metric runtime options, alert key sanitation, and metric delta helpers. The reducer core owns pending, firing, severity, and hysteresis transitions; future metric-threshold behavior should extend these owners rather than adding shared metric logic back to the central Manager file. Shared storage-health assessment alerting now lives in internal/alerts/health_assessment.go. That file owns storage-health reason normalization, ZFS pool/device reason filtering, canonical health-assessment alert synchronization, and ZFS device assessment construction for host and storage checkers; future shared health-assessment behavior should extend that owner rather than reappearing inside resource-specific evaluators or the central Manager file. Proxmox guest alert evaluation now lives in internal/alerts/guest.go. That file owns guest metric projection, per-disk guest metric alerts, guest powered-off lifecycle alerts, Pulse tag controls, relaxed guest thresholds, and guest suppression cleanup; future guest-specific alert behavior should extend that checker owner rather than expanding the central Manager file. Commercial alert handoffs now follow the same shared navigation boundary. frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx may resolve the canonical ai_alerts destination from the shared license/commercial contract, but it must delegate the actual open behavior to the frontend-primitives typed upgrade-navigation owner instead of reintroducing alert-local window.open(...) or raw external-tab assumptions. That same alert button must also honor the ordinary self-hosted prompt suppression policy: when presentationPolicy.hideUpgrade is true, a locked alert-investigation action may remain visibly unavailable, but it must not show Pro-required tooltip copy, track upgrade clicks, or open the commercial handoff route. Unlocked alert-investigation Assistant handoffs are contextual explanation and triage entries, not autonomous execution grants. InvestigateAlertButton.tsx must pass autonomousMode: false when it opens Pulse Assistant, and it must open the drawer with context only rather than seeding a product-authored prompt or choosing a diagnostic/remediation route. The visible drawer briefing for that same handoff is Alerts-owned presentation context: alert identifier, severity, metric, resource, threshold, duration, node label, and message may be shown, while raw diagnostic or remediation commands remain outside the handoff. Alert-adjacent shared helpers also inherit the runtime-versus-commercial split now carried by the shared licensing stores. Alert pages may consume runtime feature truth from frontend-modern/src/stores/license.ts, but any upgrade/trial posture must come from the dedicated commercial-posture contract, and public-demo suppression must flow from the shared resolved presentationPolicy contract instead of alert-local demo checks or entitlement reads. That same shared read-only presentation contract now also owns the public-demo alerts shell posture. When presentationPolicy.readOnly is true, the alerts page must behave as a reporting surface: overview/history remain available, alert activation controls stay hidden, configuration tabs must not render or remain navigable, and the overview empty state must not tell public demo users to toggle alerting back on when demo mode already blocks write requests. The alerts page also owns its mobile tab-shell presentation directly. frontend-modern/src/pages/Alerts.tsx may keep alert-specific active and disabled tab styling, but horizontal tab scrolling must route through the shared touch-scroll / scrollbar-hide class contract instead of writing inline overflow styles that break CSP on the public shell. The selected tab must expose canonical current-page state and compose the shared active horizontal-rail visibility owner, so direct navigation and viewport changes bring Thresholds, Notifications, or Schedule fully into view instead of leaving the active destination clipped beyond the mobile rail. Alert tab routing is part of that same presentation boundary. frontend-modern/src/features/alerts/types.ts owns the canonical mapping between visible alert tabs and URLs. The operator-facing Notifications tab must use /alerts/notifications as its canonical route because the visible navigation label is Notifications; /alerts/destinations is a legacy alias only and must normalize through tabFromPath / pathForTab instead of being reintroduced as canonical UI vocabulary. The page header for that tab must use Notifications as its title; destination wording may describe concrete email, Apprise, or webhook endpoints, but it must not reappear as the primary tab or page identity. That shared alert presentation boundary now also has explicit alerts ownership. frontend-modern/src/utils/alertWebhookPresentation.ts is the canonical owner for webhook setup copy, service labels, mention-help phrasing, custom-field presets, and add/test/update/delete action wording; frontend-modern/src/utils/alertSchedulePresentation.ts owns quiet-hours day and suppress-category card styling; and frontend-modern/src/utils/alertIncidentPresentation.ts owns incident badge, timeline, filter-chip, note-editor, and resource-incident panel presentation. Future alert presentation work must extend those helpers through the alerts contract instead of leaving alert-facing wording or styling inlined in page or feature shells while the registry treats the helpers as unowned. German and Spanish localization for the Alerts Overview operator journey is owned by the canonical i18n layer plus the alert presentation helpers: frontend-modern/src/i18n/messages.ts and frontend-modern/src/i18n/policy.ts own the catalog and non-translation policy, while frontend-modern/src/utils/alertOverviewPresentation.ts, frontend-modern/src/utils/alertActivationPresentation.ts, and frontend-modern/src/utils/alertTabsPresentation.ts own the translated operator-facing alert overview, activation, tab, timeline, and acknowledgement copy consumed by frontend-modern/src/pages/Alerts.tsx, frontend-modern/src/features/alerts/AlertOverviewStatsCards.tsx, frontend-modern/src/features/alerts/AlertOverviewActiveAlertsSection.tsx, frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx, frontend-modern/src/features/alerts/useAlertAcknowledgementState.ts, frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx, frontend-modern/src/components/Alerts/IncidentEventFilters.tsx, and frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx. Machine-facing alert identifiers, alert types, resource IDs, resource names, node names, source-system messages, commands, command output, event payloads, and log text must remain untranslated; only the user-visible alert controls may use the active app locale.

The Recovery section is a thresholds deep link: /alerts/thresholds/proxmox#backups expands and scrolls to it like #pbs and #pmg (useThresholdsTableState.test.tsx).

The thresholds resource search is route state: useThresholdsTableState reads and writes ?q= (ALERT_THRESHOLDS_QUERY_PARAMS.query) with replace navigation, keeps it across platform tab clicks, section hashes and the legacy-path redirects, so a link from elsewhere can open a tab already searched to one resource (buildAlertThresholdsPath). Host and Docker host rows match the search on their shown, display and raw names (matchesThresholdResourceSearch), because a link carries the full display label while those rows show a shortened name and keep the label as their raw name. Because each search edit navigates within the page, the Alerts page's unsaved-changes guard asks only when the destination path differs (shouldConfirmAlertsLeave in alertConfigPresentation.ts): narrowing the list keeps an unsaved edit without a prompt, leaving the page still asks (useThresholdsTableState.test.tsx, alertConfigPresentation.test.ts, Alerts.helpers.test.ts).

A successful Alerts page save re-reads the shared alertsActivation config in useAlertsConfigurationState.ts (only when the config write landed and the save still owns the editor), because backup shields and metric colours on every page read that config and otherwise kept the old limits until a reload. refreshConfig in frontend-modern/src/stores/alertsActivation.ts installs a read only while it is the latest: each read and each organisation switch takes a new generation, so a slow reply for the previous organisation or an older read cannot overwrite the current config (alertsActivation.test.ts, useAlertsConfigurationState.test.tsx).

The remaining alert configuration and history presentation helpers now also have explicit alerts ownership. frontend-modern/src/utils/alertActivationPresentation.ts, frontend-modern/src/utils/alertAdministrationPresentation.ts, frontend-modern/src/utils/alertBulkEditPresentation.ts, frontend-modern/src/utils/alertConfigPresentation.ts, frontend-modern/src/utils/alertEmailPresentation.ts, frontend-modern/src/utils/alertFrequencyPresentation.ts, frontend-modern/src/utils/alertGroupingPresentation.ts, frontend-modern/src/utils/alertHistoryPresentation.ts, frontend-modern/src/utils/alertSeverityPresentation.ts, frontend-modern/src/utils/alertTabsPresentation.ts, frontend-modern/src/utils/alertThresholdsPresentation.ts, and frontend-modern/src/utils/alertThresholdsSectionPresentation.ts are the canonical owners for alert enablement copy, history administration wording, bulk-edit labels, schedule/configuration text, email-destination field labels, frequency chips, grouping card styling, history source and resource badges, severity badges, tab labels, thresholds empty states, and thresholds section status labels. Overview stat-card labels must also route through the alert overview presentation helper, and user-facing configuration or thresholds copy must use workload, VM, and container vocabulary instead of exposing internal guest override/filter names unless the UI is naming a backend field directly. Alert severity presentation has a split ownership boundary: alerts owns formatAlertSeverityLabel, compact severity labels, legacy alert severity class helpers, severity-bucket-to-StatusIndicatorVariant mapping, and severity-bucket-to-detail-row tone mapping in frontend-modern/src/utils/alertSeverityPresentation.ts, while frontend-primitives owns the visible platform severity badge and dot shells in frontend-modern/src/components/shared/AlertSeverityBadge.tsx. Platform alert tables must consume that split instead of recreating provider-local severity label, text-class, detail-tone, or variant helpers. Platform alert detail field formatting is also a shared alerts presentation boundary. frontend-modern/src/utils/alertDetailPresentation.ts owns provider code labels, provider-specific resource-type labels, vSphere alert entity labels, row timestamp labels, and detail timestamp labels for Docker, Kubernetes, TrueNAS, and vSphere alert tables. Platform alert tables must call those helpers instead of recreating local formatCode, formatResourceType, formatEntityType, formatStartedAt, or detailDateTime helpers. Alert history filter defaults such as the all-time period option must likewise come from the alert overview/history presentation helper and the shared filter-option label primitive rather than hard-coded title-case strings in the history filter card. Alert configuration select options share that same rule: all-channel escalation labels must come from alertConfigPresentation.ts plus the shared filter-option primitive, not a schedule-page-local All Channels string. Thresholds empty states that hand operators to Infrastructure settings must use frontend-modern/src/utils/infrastructureSettingsPresentation.ts for the canonical Settings → Infrastructure label instead of hard-coding generic Settings copy or removed nested settings paths. Future alert configuration or history presentation work should extend those helpers instead of rebuilding alert-specific semantics in pages, dashboard surfaces, feature hooks, or thresholds shells.

Alert history and threshold resource tables also inherit the shared product table subgroup-row contract. AlertHistoryTableGroupRow and grouped rows in AlertResourceTableDesktop must route their date/resource group bands through frontend-modern/src/components/shared/groupedTableRowPresentation.ts instead of local bg-surface-alt fills, so alert subgroup hierarchy stays visually consistent with Infrastructure, Workloads, and Storage tables. Alert history table shells must also rely on the shared TableCard frame and the shared Table primitive for horizontal overflow rather than adding alert-local bordered or overflow-x-auto wrappers inside the history section.

The alert webhook service chooser also now derives its service set from the backend webhook template registry, rather than keeping a second frontend-only list of services, labels, descriptions, and mention-copy metadata. The WebhookConfig editor now imports the shared webhook template API type directly so it does not retain a local duplicate shape for chooser metadata. That webhook editor now also keeps runtime ownership in frontend-modern/src/components/Alerts/useWebhookConfigState.ts, while frontend-modern/src/components/Alerts/WebhookConfigList.tsx owns the existing-webhook list surface and frontend-modern/src/components/Alerts/WebhookConfigForm.tsx owns the add/edit form surface. Future webhook template loading, form normalization, custom-field preset handling, or webhook editor state transitions should land in those owners instead of being rebuilt inline in frontend-modern/src/components/Alerts/WebhookConfig.tsx.

Alert spec validation still accepts the explicit migration-bridge resource types (node, agent-disk, docker-host, backup-subject, proxmox-disk), but any other non-canonical type string is rejected before it can reach alert persistence. That keeps alert routing aligned with the canonical unified resource model instead of silently normalizing legacy type aliases inside the alert layer.

Frontend alert surfaces and backend alert-support files now require explicit registry path-policy coverage, so new alert-owned runtime files must be mapped to a concrete proof route instead of silently inheriting subsystem-default verification.

The alerts schedule surface now also routes quiet-hour suppress-category card styling through frontend-modern/src/utils/alertSchedulePresentation.ts instead of leaving that selectable-card presentation inline in frontend-modern/src/pages/Alerts.tsx. That same schedule/runtime boundary also owns quiet-hours clock semantics. Backend quiet hours are minute-granular user input, so runtime evaluation must treat the configured start minute and end minute as inclusive and therefore keep schedules such as 00:00 to 23:59 active through the full final minute instead of expiring at 23:59:00. Alert quiet-hours proofs should control time through the alert manager clock hook instead of depending on wall clock execution at whatever second the test runner happens to hit. The schedule compares local civil clock minutes on the selected calendar day, not time.Date instants. Both occurrences of a repeated minute have the same quiet-hours policy; nonexistent clock minutes do not shift the configured start or end. Initial queued replay uses the first real minute outside the current non-full-day window, including an unselected day reached at midnight. This does not reinterpret the day toggles as the previous evening's ownership. Full-day windows retain the existing daily revalidation boundary; it is not an invented open minute or permission to send during a continuous schedule. TestQuietHoursDaylightSavingClockAndReplay in internal/alerts/quiet_hours_test.go checks London, New York and Lord Howe (one-hour and half-hour changes), inclusive boundaries and exact UTC replay. TestQuietHoursOvernightSelectedCalendarDays checks overnight day selection. Quiet-hours location fallback is read-only; configuration updates own the location cache, including when public suppression helpers run concurrently. Manager.QuietHoursNotificationPolicy exposes an immutable read-only snapshot to the notification owner before it takes queue or per-alert delivery locks. Initial admission and queued replay share civil-clock/category semantics, including repeated/skipped minutes and inclusive ends. The queue revalidates every due attempt against current policy rather than trusting old replay metadata. Policy snapshots neither mutate alert lifecycle/acknowledgement nor select destinations; cancellation and actual delivery remain queue-owned. TestQuietHoursDeliverySnapshotIsCurrentIndependentAndReadOnly and TestQuietHoursDeliverySnapshotConcurrentConfigUpdates in internal/alerts/quiet_hours_test.go cover snapshot isolation and concurrent configuration reads. This establishes source scheduling, not external receipt. Quiet-hours suppression also applies to alert delivery lifecycle, not only the initial raised notification. Resolved notifications must not fan out when the alert was never notified or was already acknowledged, and monitoring-driven escalation delivery must consult the same quiet-hours suppression path while still letting canonical escalation state reach websocket consumers. Resource-tag filtering must not re-evaluate mutable tags for a resolved event. The notification owner uses the firing delivery receipt to select recovery destinations, so a tag change between firing and recovery cannot suppress a clear from a destination that received the original alert. That schedule surface now also follows the same shell/runtime split as the other feature tabs: frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx stays the render shell, while frontend-modern/src/features/alerts/useAlertScheduleState.ts owns schedule reset behavior, quiet-hours day/category toggles, cooldown/grouping/escalation update policy, and the canonical defaults handoff. Future schedule control-flow work should extend that hook instead of rebuilding those mutations inline in the tab shell. The backend cooldown gate is part of that same schedule contract. A disabled cooldown (0 or negative) means "do not send periodic re-notifications for the same active alert"; it still allows the first notification for a new alert occurrence, while level-escalation delivery remains owned by the separate escalation path. Runtime evaluation must not treat disabled cooldown as "always notify" because the alert loop runs every metric tick.

Incident-event filter chip and filter-action styling now routes through frontend-modern/src/utils/alertIncidentPresentation.ts for both frontend-modern/src/pages/Alerts.tsx and frontend-modern/src/features/alerts/OverviewTab.tsx instead of allowing those alert timeline surfaces to fork their own filter presentation.

Alert incident acknowledged badges, timeline event cards, and note-editor presentation now also route through frontend-modern/src/utils/alertIncidentPresentation.ts instead of remaining duplicated inline across the alerts page and overview timeline surfaces.

Poll-driven connectivity recovery is also part of canonical alert truth. Resources that clear an offline alert from later healthy polls must require repeated healthy confirmations before resolving that alert instead of clearing on the first recovered sample; otherwise transient poll recovery reopens the same regression as false "back online" notifications and missing downtime signal. Nodes, PBS, and PMG require three healthy confirmations before resolution, while storage requires two.

Host-agent threshold ownership now follows the linked resource model. Explicit agent overrides still win, but when no host-agent override exists the alerts runtime must inherit linked node or guest overrides for that agent so metric and connectivity behavior match the logical machine the agent augments. Persisted host alerts must carry enough linked-resource metadata for reevaluation after threshold changes to honor that same inheritance rule.

Alert resource tables, grouped node headers, and alert override reconstruction now route resource-backed names through the shared policy-aware alerts helper so governed resources do not fall back to raw names when the thresholds editor rebuilds, saves, or re-renders override rows. Alert threshold tables now route their visible resource row labels, search labels, and persisted override display names through the same shared helper so governed agent, guest, and storage rows do not leak raw names when the threshold editor saves or re-renders them. That threshold editor data shaping now routes through frontend-modern/src/features/alerts/thresholds/thresholdsResourceModel.ts for shared override-ID compatibility, grouped resource normalization, and storage status policy, while frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsData.ts stays the composition owner for the family-specific threshold projectors in frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsHostData.ts, frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsDockerData.ts, frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsGuestData.ts, and frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsInfrastructureData.ts. backup and snapshot default sanitization plus factory-drift policy now live in frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsRecoveryDefaultsState.ts, while frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsTableState.ts owns threshold-table route sync, section collapse state, search/edit shell state, and bulk-edit dialog control. Pure override upsert and hysteresis-entry helpers now live in frontend-modern/src/features/alerts/thresholds/thresholdsOverrideMutationModel.ts. Threshold edit persistence, bulk threshold application, and backup/snapshot override toggles now route through frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsOverrideMutations.ts, while powered-off/connectivity state transitions plus alert-removal side effects now route through frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsAvailabilityMutations.ts. The backup/snapshot override toggles in that mutations owner write sparse enabled-only overrides (zero-valued threshold fields) rather than copying the current global defaults into the override, because zero-valued fields inherit the globals at evaluation time and a copied value freezes the guest against later global edits (#1126). Toggling a guest that already carries explicit override thresholds must preserve those values while flipping enabled. The Backups and Snapshots global-defaults editors in frontend-modern/src/components/Alerts/ThresholdsTableProxmoxBackupsSection.tsx and frontend-modern/src/components/Alerts/ThresholdsTableProxmoxSnapshotsSection.tsx reconcile warning/critical pairs through reconcileWarningCriticalEdit in frontend-modern/src/features/alerts/thresholds/helpers.ts: a single-field edit that creates a conflict adjusts the field the user did not touch, so a 32-day warning edit raises critical alongside it instead of being silently clamped back down by the sanitizers before save. That same thresholds host-data boundary now treats top-level TrueNAS appliances as canonical agent resources with platformType: 'truenas'. System-disk group headers must use agent-owned header metadata instead of guest/node- friendly header metadata, so appliance labels like TrueNAS Main do not collapse to vendor-only TrueNAS inside thresholds or override surfaces. The thresholds tab adapter contract now lives in frontend-modern/src/features/alerts/thresholds/thresholdsTabModel.ts, so frontend-modern/src/features/alerts/tabs/ThresholdsTab.tsx stays a shell instead of carrying a duplicate table-prop interface and hand-mapped adapter layer. frontend-modern/src/components/Alerts/ThresholdsTable.tsx is now limited to shell composition for search/help/nav plus bulk-edit dialog flow, while the tab render owners live in frontend-modern/src/components/Alerts/ThresholdsTableProxmoxTab.tsx, frontend-modern/src/components/Alerts/ThresholdsTablePMGTab.tsx, frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx, frontend-modern/src/components/Alerts/ThresholdsTableDockerTab.tsx, frontend-modern/src/components/Alerts/ThresholdsTableKubernetesTab.tsx, frontend-modern/src/components/Alerts/ThresholdsTableTrueNASTab.tsx, frontend-modern/src/components/Alerts/ThresholdsTableVMwareTab.tsx, and frontend-modern/src/components/Alerts/ThresholdsTablePBSTab.tsx. New threshold row grouping, override-ID compatibility, resource normalization, thresholds-table controller logic, or per-tab runtime should land in those feature hooks and tab owners rather than being rebuilt inside the shell. The shell-owned thresholds sub-routes are platform-shaped user-facing paths: /alerts/thresholds/proxmox, /alerts/thresholds/docker, /alerts/thresholds/kubernetes, /alerts/thresholds/truenas, /alerts/thresholds/vmware, /alerts/thresholds/pbs, /alerts/thresholds/pmg, and /alerts/thresholds/systems. Legacy neutral links like /alerts/thresholds/infrastructure, /alerts/thresholds/containers, and /alerts/thresholds/mail-gateway must redirect to the matching platform-shaped route. Legacy /alerts/thresholds/agents links must continue to resolve to Systems. frontend-modern/src/components/Alerts/ThresholdsTable.tsx must expose those peer platform routes through the shared Subtabs primitive rather than an inline FilterBar group. Clearing search or override filters must preserve the active platform route. The thresholds shell is overview-first: every resource section starts collapsed, persisted operator expansion choices still win, and Expand all / Collapse all remain available on every platform tab. Because collapsed tables cannot carry the discovery burden, the shell must surface a filter-independent custom-override summary above the sections; choosing a summary item selects the Custom-only filter, opens its owning section, and moves that section into view. Resources not represented there continue to inherit their group defaults. Metric enablement is an explicit On/Off interaction in row, global-default, mobile, and bulk editors. Enabled numeric inputs accept positive trigger values only and user-facing copy must not expose the persisted disable sentinel. Internally, the canonical <= 0 read rule remains intact for legacy data, while new Off actions stage -1 (saved as trigger 0 on global defaults, which the backend otherwise resets to factory), so existing configuration and the API contract continue to round-trip without making 0 a second customer-facing disable path. Within the Proxmox tab, render-heavy ownership now further routes through frontend-modern/src/components/Alerts/ThresholdsTableProxmoxNodesSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableProxmoxPBSSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableProxmoxGuestsSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableProxmoxGuestFilteringSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableProxmoxBackupsSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableProxmoxSnapshotsSection.tsx, and frontend-modern/src/components/Alerts/ThresholdsTableProxmoxStorageSection.tsx with the shared section contract in frontend-modern/src/features/alerts/thresholds/thresholdsTableSectionProps.ts. Future infrastructure-thresholds presentation work should extend those section owners instead of expanding frontend-modern/src/components/Alerts/ThresholdsTableProxmoxTab.tsx back into a mixed render surface. The Docker tab now follows that same section-owner shape through frontend-modern/src/components/Alerts/ThresholdsTableDockerIgnoredPrefixesSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableDockerServiceGapSection.tsx, frontend-modern/src/components/Alerts/ThresholdsTableDockerHostsSection.tsx, and frontend-modern/src/components/Alerts/ThresholdsTableDockerContainersSection.tsx. The containers thresholds surface must consume canonical app-container parents through the shared alert-overrides state rather than assuming docker-host is the only runtime shape. API-backed TrueNAS parents belong in the same Container Runtimes / Containers surface, while Docker-specific controls like ignored prefixes and Swarm service gap settings must stay gated to real Docker runtimes instead of leaking onto non-Docker platforms. At the current support floor, TrueNAS alert support means the shared alert surfaces can evaluate, show, and drill into incidents on TrueNAS-backed systems, disks, and app parents using the canonical resource model and related links into infrastructure, workloads, storage, and recovery. Pulse does not promise a TrueNAS-only alert workflow or provider-specific alert management surface beyond the shared alerts product. At the current locked VMware floor, alert support must mean the same shared alert surfaces can evaluate, show, and drill into vSphere alarm, health, and threshold signals on canonical agent, vm, storage, and network resources, with related event/task context routed through the shared incident and resource links. This is the alerts support floor for admitted vSphere resources; it does not by itself promote the broader VMware platform readiness state beyond the separately governed platform-admission floor. Pulse must not grow a VMware-only alert shell, alarm editor, or direct alarm-control surface in phase 1. That same VMware alert rule now also includes the topology boundary. Alarm context that originates on a datacenter, cluster, folder, or resource pool may inform a shared incident, but it must still resolve onto canonical agent, vm, storage, or network investigation paths rather than creating synthetic top-level VMware incident resources. If that attachment cannot be done honestly for a given signal, the signal should remain supporting context instead of inflating the support claim. That same VMware alert rule now also includes the timeline boundary. Related VMware event and task context may enrich shared alert and incident views, but it must do so through the canonical incident and resource-history paths rather than through a VMware-only history browser, event drill-down route, or alarm management shell. Future Docker thresholds presentation work should extend those section owners instead of expanding frontend-modern/src/components/Alerts/ThresholdsTableDockerTab.tsx back into a mixed render surface. The systems tab now follows that same shell-versus-section pattern through frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx and frontend-modern/src/components/Alerts/ThresholdsTableAgentDisksSection.tsx. Future systems-thresholds presentation work should extend those section owners instead of expanding frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx back into a mixed render surface. The alert resource thresholds editor now follows the same shape: shared metric normalization, bounds, value-resolution, and override-label logic live in frontend-modern/src/components/Alerts/alertResourceTableModel.ts, shared group header presentation lives in frontend-modern/src/components/Alerts/AlertResourceGroupHeader.tsx, desktop table ownership lives in frontend-modern/src/components/Alerts/AlertResourceTableDesktop.tsx, mobile card ownership lives in frontend-modern/src/components/Alerts/AlertResourceTableMobile.tsx, render-heavy desktop row ownership lives in frontend-modern/src/components/Alerts/AlertResourceTableRow.tsx, and selection state, delay-row toggling, and inline metric-input focus live in frontend-modern/src/components/Alerts/useAlertResourceTableState.ts. Shared web-interface launch affordances inside alert resource rows and grouped agent headers must compose frontend-modern/src/components/shared/WebInterfaceLink.tsx; alerts own only the alert/resource data and URL availability decision, not URL safety classification, the adjacent launch-control shell, new-tab safety attributes, row-click containment, invalid-URL warnings, or accessible launch-label semantics. Alert resource names remain inert row identity so opening a saved web interface cannot also toggle selection or the thresholds editor. Resource-table empty states, badge labels, offline-state wording, note placeholders, and metric input titles now route through frontend-modern/src/utils/alertResourceTablePresentation.ts instead of remaining duplicated across the desktop and mobile thresholds surfaces. frontend-modern/src/components/Alerts/ResourceTable.tsx is now limited to the shell boundary for breakpoint selection and bulk-edit composition. Future resource-table threshold semantics should land in those owners instead of being rebuilt inline in the shell.

Alert incident timeline event cards now route through frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx, while their meta-row, heading, detail, command, and output typography still route through frontend-modern/src/utils/alertIncidentPresentation.ts instead of keeping duplicate timeline card structure inline in the alerts page and overview timelines.

Expanded alert incident detail now also routes through frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx and frontend-modern/src/components/Alerts/IncidentEventFilters.tsx so the overview surface and the history table share the same loading/error states, canonical timeline meta row, note editor, and event-filter controls instead of maintaining two independent incident-detail implementations. That shared timeline runtime state now routes through frontend-modern/src/features/alerts/useAlertIncidentTimelineState.ts, which owns incident timeline fetch, expansion state, note-save flow, and shared event-filter state for both frontend-modern/src/features/alerts/OverviewTab.tsx and frontend-modern/src/features/alerts/tabs/HistoryTab.tsx. Future incident timeline control flow should land in that feature hook instead of being forked back into either alert surface. Alert incident timeline handoffs into Pulse Assistant are now owned by the Alerts incident handoff model and carry only sanitized incident facts plus event summaries into both the visible drawer briefing and the backend model-only handoff context; raw command and output details stay in the incident timeline or approval surface. The shared hook may expose Timeline for a displayed mock alert because mock mode now guarantees an occurrence-qualified incident through the same typed API. A mock fixture missing that incident is a broken alert read-model contract, not an empty-state case for the frontend to disguise; genuine unknown real occurrences may still render the existing no-timeline state.

Resource incident panel cards, summary rows, and toggle-button presentation now also route through frontend-modern/src/utils/alertIncidentPresentation.ts instead of remaining inline inside frontend-modern/src/pages/Alerts.tsx.

That same resource incident panel now treats collapsed incident activity as a canonical alert read-model summary rather than a page-local sentence. The collapsed row must summarize filtered incident events by canonical event type order and reuse the shared event-card renderer for expanded incident detail, so the alert history page does not drift away from the overview timeline when canonical lifecycle or remediation events are added.

Active alert card state, acknowledged badge, and primary/secondary action button presentation now route through frontend-modern/src/utils/alertOverviewPresentation.ts instead of remaining inline in frontend-modern/src/features/alerts/OverviewTab.tsx. The canonical shared alert-acknowledgement runtime owner is now frontend-modern/src/features/alerts/useAlertAcknowledgementState.ts, which owns optimistic single/bulk acknowledge control flow, restore behavior, and notification feedback for frontend-modern/src/features/alerts/useAlertOverviewState.ts and the alert overview render shells. frontend-modern/src/features/alerts/useAlertOverviewState.ts now owns the derived alert read-model and Last 24 Hours stat refresh for frontend-modern/src/features/alerts/OverviewTab.tsx, while composing that shared acknowledgement owner instead of keeping its own alert mutation fork. Future overview action behavior should extend that shared acknowledgement hook instead of putting acknowledge mutations back into render shells. Global and group acknowledgement read the bulk endpoint's per-alert results through one path, so a group action reports the alerts it could not acknowledge with the same count notice as the global action, beside the success count of a partial result. Request ownership is app-wide: the alert store creates frontend-modern/src/stores/alertAcknowledgementRequests.ts, which gives each alert to at most one unsettled card, restore, group or global request. Later card, group and global actions leave an owned alert to that request, which reports its outcome. A card shows every alert an unsettled request carries as processing, a global request's included, and keeps the 1.5-second hold only after a card or group request settles, so a failed group or global request can be retried at once. The cards' processing state and the global button's progress derive from the owner: a remounted overview shows a request its predecessor sent, and an older request's hold cannot mark an alert idle while a newer request carries it. The owner resets with the store's alert scope (the organization switch through switchUrl, shutdown and store cleanup). Its pending acknowledgement confirmation guards and their 15-second timers end at that same boundary, so a settled local change cannot suppress another scope's alert with a reused identifier or emit a stale resync notice there. A same-scope reconnect retains both the request owner and confirmation guards. A response from an ended scope writes nothing to the store and shows nothing, since both belong to the next scope, whose alerts can share its identifiers; a rejection is still logged. Within one scope a response still reports once after the overview unmounts. The jsdom no-op store keeps its own owner, reset by its shutdown, and shutdownGlobalWebSocketStore ends that shared app for a test as page unload ends it for the app. alertAcknowledgementRequests.test.ts, websocket.alertAcknowledgementRequests.test.ts, websocket-global.noopLifetime.test.ts, useAlertAcknowledgementState.test.tsx and OverviewTab.acknowledgementbusy.test.tsx pin these paths. Overview ages and bands read the shared clock. AlertOverviewAlertCard.tsx passes its own useRelativeTimeNow reading as now to formatAlertOverviewStartedAgo and getMetricAlertPresentation, and the Last 24 Hours count in useAlertOverviewState.ts reads the same clock, so all three measure from the wall clock. The hook no longer exposes a minute tick: read as "now", that tick measured a card mounted between ticks from up to a minute earlier, so an alert raised 70 seconds ago read "this minute", an alert raised since the last tick stayed out of the 24h count until the next one, a reading already past the 10-minute stale cut-off led the card as live, and the stale flip and 24h drop landed up to a minute late. A mounted card now turns stale, and an alert leaves the 24h count, within one 30-second tick with no data change. The delivery diagnoses refresh is a server read and keeps its own minute interval rather than following the 30-second clock. AlertOverviewAlertCard.clock.test.tsx pins the mid-tick mount, the stale flip and the 24h drop under fake timers, and useAlertOverviewState.test.tsx pins the diagnoses cadence. Render-heavy alert overview ownership now routes through frontend-modern/src/features/alerts/AlertOverviewStatsCards.tsx, frontend-modern/src/features/alerts/AlertOverviewActiveAlertsSection.tsx, and frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx instead of rebuilding stats-card, active-alert, and timeline-card presentation inline inside frontend-modern/src/features/alerts/OverviewTab.tsx. AlertOverviewAlertCard.tsx also owns the compact delivery-status placement, while deliveryDiagnosisPresentation.ts owns the reason-to-copy and tone mapping. Acknowledged cards keep their canonical badge and omit that redundant line; held delivery states that can surprise an operator use the attention tone, while cooldown, quiet-hours, monitor-only, and successful/pending states remain neutral. The card's resource link resolves the owning platform page from alert metadata, never from message wording alone. Provider incidents land on canonical vm, storage, network, and agent resource types rather than a platform-prefixed type, so the card reads the platform from an explicit platformType, then the incident's incidentProvider, then the resource's resourceSources, all through resolvePlatformTypeFromSources in frontend-modern/src/utils/sourcePlatforms.ts: a vCenter alarm whose message is only the alarm's name links to the vSphere overview, a TrueNAS pool incident to TrueNAS, and a PBS incident that Pulse itself raised (provider pulse) to Proxmox by its pbs source. The resolved platform maps to its page through PRIMARY_PLATFORM_NAV_SCOPE_IDS and the routing/resourceLinks.ts path builders, and the same canonical platform id goes to the Monitoring policy menu, so the link and the named inventory owner cannot disagree. Only an alert whose metadata names no platform falls back to its resource type (agent, docker-*, app-container, k8s-*, truenas-*, vmware-*) and finally the message; agent alerts open /standalone/machines, because no /machines route exists. A resource-id prefix or substring and the message may pick the page only when the alert names no resource type at all, because Proxmox guest ids embed the user's cluster and node names and messages name the guest: VM alerts on node docker-01, in a cluster named agent, or for a guest named vmware-test still link to Proxmox and name Proxmox as their owner. Metric alerts put a display label in resourceType (vSphere VM, Kubernetes Pod), so the unified metric and capacity evaluators (internal/alerts/unified_eval.go, internal/alerts/capacity_forecast.go) stamp the canonical platformType beside it, and connection-degraded alerts (internal/alerts/connection.go) stamp it from the connection type; the card reads that ahead of every fallback. The menu's owner copy comes from frontend-modern/src/utils/resourceMonitoringPolicy.ts, where an explicit platform outranks the resource-type guess and the agent-removal copy is kept to the agent's own machine: libvirt VMs and Unraid arrays in resource drawers get the neutral source-system copy. Host-agent alerts (internal/alerts/host.go) stamp platformType agent, and only the machine's own alerts keep resourceType agent: filesystem, disk-temperature and SMART alerts carry agent-disk, RAID and Unraid array alerts agent-storage, and custom-sensor alerts agent-sensor, because the menu writes policy to their child resource id, never to the machine. Each child type keeps agent among its CanonicalResourceTypeKeys, and configuration re-evaluation treats agent policy as final for host-agent alerts (TrueNAS systems and vSphere hosts keep their platform policy), so the storage and guest switches cannot resolve a SMART, RAID or sensor alert that the next agent report would re-raise. A metric alert held between its clear and trigger thresholds still refreshes resourceType and platformType from its producer while its value, message and last-seen time stay at the last breach, so a restored alert that is still firing or held picks up the current classification at its next evaluation; one that resolves on that evaluation keeps the stored classification in its resolved record. Menu sentences open with the owner label capitalised. The retired dashboard recent-alert panel must not be reintroduced as a parallel alert surface. Alert summary/tone copy belongs to the alert overview presentation owner, and any future compact alert surface must compose the shared alert read-model and acknowledgement hook rather than creating a dashboard-only panel. The Alerts Overview description must stay monitor-first: it names active incidents and current coverage across monitored resources, and must not imply that the overview itself owns installation-wide alert activation controls. The exported English fallback in alertOverviewPresentation.ts, localized message catalog, and header metadata proof must remain textually aligned.

The Schedule tab's header title is "Delivery Schedule" (ALERTS_PAGE_SCHEDULE_TITLE, alerts.page.schedule.title; de "Zustellplan", es "Programacion de entregas"). The page sets how alerts are delivered (quiet hours, cooldown, grouping, delivery channels, recovery and escalation); it was titled "Maintenance Schedule", which promised maintenance windows, a separate per-resource feature this page does not hold (alertOverviewPresentation.test.ts, i18n.test.ts).

Alert threshold and schedule surfaces must now also treat discoveryTarget as optional frontend input and keep grouping-card state on the canonical node group-header contract. Frontend alert pages may not assume discovery metadata is always present when deriving override IDs or toggle styling.

The alerts page shell in frontend-modern/src/pages/Alerts.tsx must now keep destinations, history, schedule, and thresholds rendering feature-owned under frontend-modern/src/features/alerts/tabs/. New alert tab surfaces should be extracted as feature modules instead of remaining page-local function blocks, so the page owns navigation and cross-surface routing while tab files own their runtime presentation, tab-local interaction logic, and any history-table presentation or thresholds-table adapter logic that does not belong in a shared primitive.

The history tab itself now follows the same shell-versus-runtime rule. The canonical history runtime owner is frontend-modern/src/features/alerts/useAlertHistoryState.ts, which now owns alert-history fetch, persistent filter state, history-clear flow, and composition of the derived history owners. Resource-incident panel loading, refresh, and expansion state now live in frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts, while the pure analytics model for history-item projection, trend buckets, group labels, axis ticks, and selected bucket detail now lives in frontend-modern/src/features/alerts/alertHistoryModel.ts. The tab shell in frontend-modern/src/features/alerts/tabs/HistoryTab.tsx now composes frontend-modern/src/features/alerts/AlertHistoryFrequencyCard.tsx, frontend-modern/src/features/alerts/AlertHistoryFiltersCard.tsx, frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx, frontend-modern/src/features/alerts/AlertHistoryTableSection.tsx, frontend-modern/src/features/alerts/AlertHistoryTableGroupRow.tsx, frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx, and frontend-modern/src/features/alerts/AlertHistoryAdministrationCard.tsx. Future alert-history control-flow work should extend the feature hook, new grouping or trend semantics should extend the history model, and render-heavy history surfaces should extend those section owners instead of putting fetch, resource-incident state, or table rendering back into the shell. Alert History toolbar state is one route-owned unit. Search, period, and severity are read and written by useAlertHistoryState.ts; its active-filter count includes search, and AlertHistoryFiltersCard.tsx delegates contextual Clear filters to one composite hook mutation instead of letting FilterBar issue sequential route writes. That reset removes all three query parameters in one navigation and clears any transient chart-bucket selection, so a search-only result set remains visibly resettable and an older URL write cannot resurrect another filter. That same history surface also owns the resource-incident panel, and the panel links nowhere. frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx names the selected resource through the unified-resource lookup it is given (getResource, falling back to the history state's lookup) and getPreferredInfrastructureDisplayName, and shows the resource name the selected history row carried while no unified record matches. It imports nothing from frontend-modern/src/routing/resourceLinks.ts, and it offers no resource cross-jump links: investigation continues in place through the incident timeline cards, as the retired cross-jump chip note above describes. Future incident-link work must not reintroduce a chip set, local infrastructure-link assembly, or provider-local route strings inside the alert feature shell; frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts pins the panel free of buildResolvedResourceSurfaceLinks, buildResourceSurfaceLinksForResource, and buildInfrastructureResourceLink.

Alert configuration load/save state, notification config reloads, and threshold override normalization now route through frontend-modern/src/features/alerts/AlertsConfigurationSurface.tsx instead of living inline in frontend-modern/src/pages/Alerts.tsx. The page shell owns navigation, activation chrome, and cross-surface routing; the configuration surface is now a shell that composes the destinations, schedule, and thresholds tabs. The canonical alert-policy runtime owner is now frontend-modern/src/features/alerts/useAlertsConfigurationState.ts for config transport, notification-config reloads, and save/load orchestration, frontend-modern/src/features/alerts/useAlertsConfigurationSnapshotState.ts for default-backed mutable config snapshot state plus apply/capture/reset ownership, frontend-modern/src/features/alerts/alertsConfigurationModel.ts for backend config normalization, factory defaults, docker-gap validation, and save-payload serialization, frontend-modern/src/features/alerts/alertOverridesModel.ts for raw override normalization plus resource-backed override projection. That override owner must canonicalize legacy per-node shared-storage keys and hashed storage resource ids onto the storage metrics target id before the thresholds surface rebuilds, so old Ceph/shared-datastore overrides and newly projected Ceph pool overrides still surface on the live v6 editor instead of disappearing after the feature-shell migration, and frontend-modern/src/features/alerts/useAlertOverridesState.ts for reactive override state, derived resource lists, and overview handoff, and frontend-modern/src/features/alerts/alertDestinationsModel.ts for email and Apprise config normalization plus outbound payload shaping, and frontend-modern/src/features/alerts/useAlertDestinationsState.ts for notification destination reload and persistence orchestration. frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts now owns webhook load/mutate/test flow, frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts now owns destination test actions, retry orchestration, and notification delivery-health loading around that webhook runtime, while frontend-modern/src/features/alerts/tabs/DestinationsTab.tsx stays the destinations render shell and composes frontend-modern/src/features/alerts/AlertDeliveryHealthCard.tsx, frontend-modern/src/features/alerts/AlertEmailDestinationsSection.tsx, frontend-modern/src/features/alerts/AlertAppriseDestinationsSection.tsx, frontend-modern/src/features/alerts/AlertWebhookDestinationsSection.tsx, and the dedicated load/error wrappers. The delivery card must warn for retained terminal failures, present an unavailable queue-health read as unverified rather than healthy, and give the operator a refresh action plus configuration and test guidance. Its warning copy must explain that expired delivery records are removed hourly and that, absent another terminal failure, the warning clears only after the last retained record reaches its configured retention limit. Recoverable retry attempts must not produce this warning. frontend-modern/src/features/alerts/AlertDeliveryHealthCard.test.tsx and frontend-modern/src/features/alerts/__tests__/useAlertDestinationsTabState.test.tsx are the focused render and runtime proofs for that operator surface. Future config cleanup should extend the config transport hook, the config model, the override-projection hook, or the shared frontend-modern/src/utils/alertDestinationsPresentation.ts helper for customer-facing destinations copy instead of reviving inline retry, test, and error text across the feature tabs. Runtime cleanup should extend the destinations hook owned by the subsystem that carries the behavior instead of letting the broader configuration hook absorb all four concerns again. Local Apprise delivery is a shipped server-runtime capability, not an operator-installed optional. The shared presentation helper must show the Telegram chat_id:topic form, the notification backend must preserve each target as one CLI argument, and the deployment-installability contract must keep the pinned apprise executable available in every Pulse server image. The email destination provider picker now follows that same split: frontend-modern/src/components/Alerts/useEmailProviderSelectState.ts owns provider-catalog loading and provider-default application, while frontend-modern/src/components/Alerts/EmailProviderSelect.tsx stays the render shell and consumes the canonical UIEmailConfig feature type instead of keeping a second local email-config interface. The alert scheduling surface now follows the same shell/section split: frontend-modern/src/features/alerts/useAlertScheduleState.ts owns schedule runtime and default/reset policy, while frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx stays the shell and composes the dedicated quiet-hours, cooldown, grouping, recovery, escalation, and summary section owners instead of carrying those panels inline.

Alert filter metadata and grouped header consumers must also preserve the canonical agent and node header boundary when reusing shared filter primitives. Frontend alert tables may not drift back to ad hoc host-key grouping or narrow filter key predicates that drop optional hostname values before alert group metadata is derived. That same shared alert boundary now also owns provider-backed resource-incident alerts beyond storage-only cases. internal/alerts/model.go, internal/alerts/unified_incidents.go, and frontend-modern/src/utils/alertIncidentPresentation.ts must treat VMware- backed host and VM incidents as the same canonical resource-incident vocabulary used everywhere else, with quiet-hours routing derived from the shared incident category and provider context carried only as shared alert and timeline metadata. Alert history may surface VMware alarm, task, and snapshot context inside that shared model, but it must not fork into VMware-only alert types, badges, or incident chrome. That same alert-shell boundary now also treats websocket access as a shared app-runtime dependency rather than an alerts-owned provider. Alert shells such as frontend-modern/src/pages/Alerts.tsx and frontend-modern/src/features/alerts/tabs/HistoryTab.tsx may consume live state only through frontend-modern/src/contexts/appRuntime.ts; they must not import @/App or create reverse dependencies into the root shell chunk, because alerts surfaces must remain lazy-load safe and must not blank the app before auth/bootstrap finishes. Alert commercial handoffs now also follow the runtime-versus-commercial split. frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx may resolve upgrade destinations through the shared commercial-posture store, but alert runtime availability and chat enablement must stay governed by the non-commercial app runtime and assistant state instead of reusing the same commercial payload as feature truth. Alert schedule and incident-timeline surfaces now also keep their browser state typed through one feature-owned contract. Quiet-hour suppress options must be cloned into mutable feature props before crossing section boundaries, quiet-day callbacks must preserve the canonical weekday key union, and incident timeline expansion/note-saving state must remain Set<string>-owned instead of drifting to untyped browser-local collections. That same alerts runtime boundary also owns canonical identity derivation and active-alert persistence. Shared canonical identity helpers may infer resource, spec, and state ids from legacy alerts, but they must do so without mutating live in-memory alert instances unless the caller explicitly backfills that state. Persisted active-alert snapshots must therefore clone alerts under lock, backfill canonical identity on the clone, replace the persisted public ID with the canonical state ID, and serialize that snapshot instead of mutating the live alert map during async saves or incident rebuilds. Restore must preserve acknowledgement fields and rewrite a successfully derived legacy snapshot atomically after the in-memory restore; unrecognized records remain readable. That same ownership also governs acknowledgement and manual-clear cleanup. Clearing an alert through the canonical alerts runtime must remove both legacy public-id tracking and canonical-state acknowledgement records so old aliases cannot keep an alert acknowledged after the canonical alert has been removed.

Operational Trust writable lifecycle

internal/alerts/active_lifecycle.go is the single writable owner for acknowledge, unacknowledge, suppress, unsuppress, collection-stale, collection-unknown, resolving, and collection-restored transitions. Every writer is idempotent under retry, retains new evidence without duplicating a same-state transition, persists through the active-alert store, and survives restart. Explicit stale, unknown, resolving, or suppressed state takes precedence over the legacy Alert.Acknowledged projection. The canonical record's LastObservedAt is the maximum of retained record, alert, and evidence timestamps so a fresh outage observation cannot be rolled back by an older legacy alert timestamp.

Suppression is bounded and reasoned, leaves the default active queue, and remains inspectable. Expiry or explicit unsuppression returns the record to its detector-owned state; it never resolves it. Only fresh sufficient recovery evidence may enter resolving, and only detector recovery, or a handover recorded as ownership_transferred with its closing evidence, may resolve.

Per-alert snooze is the customer-facing bounded suppression contract, not a page-local timer. SnoozeAlert writes the same canonical operational record, requires a future expiry no more than 30 days away, and appends a durable snoozed lifecycle snapshot. While active, notification dispatch, recovery delivery, delivery diagnosis, and escalation all consult that one record. Expiry runs as an explicit unsnoozed transition and restores acknowledged or open according to the underlying alert without resolving the incident. Skipped escalation levels are not replayed in a burst: the remaining schedule resumes from the unsnooze transition while the incident's original start time and duration remain intact. The alerts overview exposes preset bounded holds, the exact local expiry, and an explicit Resume action over this API-owned state.

Canonical threshold-override succession

Alert threshold overrides are persisted under the current canonical resource ID. internal/alerts/canonical_override_migration.go may re-home an override from a retired ID only when the unified-resource owner explicitly publishes that ID in SupersededCanonicalIDs. Display aliases, hostnames, metric targets, connection labels, and other lookup conveniences are not persistence authority. A still-live old ID or one old ID claimed by multiple current resources fails closed and remains untouched; when both keys exist, the current canonical override wins.

Monitoring owns the synchronization point that applies this migration to the active alert configuration and persists it before later reloads. The frontend may read legacy connection-target keys as compatibility candidates, but save, edit, and delete operations target the resource's current canonical ID and retain that ID across refetches or reported-identity changes. This migration changes alert configuration only and does not create customer-infrastructure mutation authority outside canonical Actions.

internal/alerts/canonical_override_migration_test.go proves the unambiguous/live/ambiguous decision matrix, internal/monitoring/monitor_alert_override_migration_test.go proves durable reload behavior, and frontend-modern/src/features/alerts/thresholds/hooks/__tests__/truenasThresholdPersistence.test.tsx proves the browser-side TrueNAS save/refetch contract.

Versioned persisted alert identity

AlertConfig.identitySchemaVersion is the additive persisted identity marker; the current schema is version 1. PlanAlertIdentityMigration is the mandatory dry-run boundary: it operates on a copied config and reports source/target versions, removed and added keys, deferred rows, and unsupported future versions before any write. ApplyAlertIdentityMigration may install that exact plan only while its source version is still current. The monitoring bridge must persist the result before updating the live manager.

Version 1 folds provider-declared canonical succession, Docker container recreation IDs, Proxmox guest and guest-disk aliases, and live storage aliases onto the single write identity already used by the editor/evaluator. A current row wins over retired duplicates. Multiple legacy rows claiming one target, ambiguous live ownership, unknown rows, and temporarily absent resources fail closed and stay in alerts.json for a later resource snapshot that can prove the mapping. Alias readers remain compatibility inputs for those deferred rows, not persistence authority. Storage alias provenance is carried on the unified resource StorageMeta so the plan does not reconstruct ownership from display names alone.

The marker is rollback-safe: older binaries ignore the additive JSON field and may omit it on a later save; after re-upgrade the idempotent planner runs again. Newer binaries preserve an already-held marker when a stale browser payload omits it, while the alert editor explicitly round-trips the marker. A config whose marker is newer than the running binary is never rewritten. Regression ownership is internal/alerts/alert_identity_migration_test.go, internal/monitoring/monitor_alert_override_migration_test.go, and frontend-modern/src/features/alerts/__tests__/alertsConfigurationModel.snapshot.test.ts.

Alert config snapshots are owned copies

Manager.GetConfig() returns AlertConfig.Clone(), a typed deep copy that shares no map, slice or pointer with the live config (custom rule filter values are copied as decoded JSON values). Callers encode, persist and edit that snapshot outside m.mu: the GET config handler, alert activation, the identity migration, backup evaluation and diagnostics. It used to be a shallow copy, so SaveAlertConfig normalizing it in place, or UpdateConfig re-normalizing the overrides of a re-applied snapshot, wrote the live maps while evaluation or another encoder read them, and Go aborted Pulse with a concurrent map write. SaveAlertConfig also clones before normalizing, because a config just handed to UpdateConfig is the live config. Persistence keeps an explicit 0 type-level delay as no delay on save and load, matching NormalizeTimeThresholds. It used to store and load it as the 5-second factory delay, which the noisy-gauge rule stretches to 300 seconds for memory and temperature, and the shared map write was what made the running manager agree. Pulse saves never wrote a 0, so only a newly sent or hand-edited 0 changes meaning. internal/alerts/config/clone_test.go fills every field and fails when a new reference field is not copied. Regression ownership is internal/alerts/resolved_lock_discipline_test.go and internal/config/persistence_alert_ownership_test.go.

Alert evaluation owns the thresholds it normalizes

Host, guest and Docker container evaluation copy what they read from m.config while holding m.mu, and only then fill in defaults. CheckHost and CheckGuest clone a per-disk override's Disk threshold before ensureHysteresisThreshold sets a missing Clear to the trigger minus 5. They used to normalize the live override's pointer after unlocking, so a GetConfig() clone under the read lock raced the write, and the saved override changed to whatever the last poll filled in. The live override now keeps the clear it was saved with, and the config API returns that value. Docker container evaluation resolves its thresholds from cloned DockerDefaults plus the override under the same lock, and reads the restart-loop and memory-limit settings under the lock. It used to point into m.config.DockerDefaults with no lock while UpdateConfig replaced m.config. The test-only getThresholdForMetricFromConfig, which normalized in place, is gone. internal/alerts/alerts_test.go TestAlertEvaluationLeavesTheLiveConfigToTheLock runs host, guest and Docker evaluation against GetConfig() and UpdateConfig(GetConfig()) under -race.

Config-save reevaluation judges a filesystem alert by its own threshold

UpdateConfig re-judges every active alert, and a filesystem usage alert's threshold resolves through the same per-filesystem helper its evaluator uses. hostDiskUsageThresholdNoLock resolves an agent filesystem (agent:<host>/disk:<label>): its own override, then the DiskFillByType threshold for a hardware type inferred from the device while the host's disk alerting is on, then the host's disk threshold. guestDiskUsageThresholdNoLock resolves a guest filesystem (<guestID>-disk-<key>): its guest-disk: override, then the guest's disk threshold. CheckHost and CheckGuest call them under the read lock, and resolveHostAlertThresholdsNoLock and resolveGuestAlertThresholdsNoLock call them for reevaluation. Both return an override or per-type threshold as an owned copy. A guest filesystem alert resolves its guest's threshold, override and identity-based custom rules by the canonical instance:node:vmid guest ID that parseGuestAlertIdentity reads from its resource ID. Custom rules that filter on live metrics still see no readings there, as for every reevaluated guest alert. Reevaluation used to judge agent filesystems by the host threshold, and guest filesystems without their guest-disk: override or the guest's own override, because the filesystem's resource ID names no guest. Any settings save resolved an alert a per-disk threshold of 70 raised at 80 under a default of 90, notifying recovery, and the next poll raised it again as a new alert. internal/alerts/threshold_resolution_shared_test.go TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold covers an agent filesystem override, an NVMe fill threshold, a guest filesystem override and a guest override: an unchanged save must keep the alert, and a save lifting the trigger above the reading must resolve it.

Versioned alert-intent policy

The alerts runtime owns one versioned alert-intent document and its durable pending-condition state. Stable signal keys are *, state.offline, incident.availability, and metric.<name>. Effective fields resolve independently from legacy metric behavior, document defaults, resource-type overrides, and canonical-resource overrides, in that order. Keys are normalized before validation and collisions fail closed; updates use revision compare-and-swap so a stale browser cannot overwrite a newer document.

Intent affects when detector truth becomes eligible for an active alert; it does not mutate the underlying observation or create a second alert identity. Without an explicit applicable rule, established alert behavior remains compatible. With a rule, the first matched time is durable across restart and becomes the canonical alert start time once eligible. Preview evaluates the same resolver but restores pending state before returning, so it is read-only. Invalid documents, persistence failures, or revision conflicts must leave the prior in-memory and durable policy active.

Resource-type inheritance is field-wise from general to specific before the canonical-resource rule: for example guest supplies a VM/LXC default and vm or system-container may replace individual fields. A guest policy must not leak into node or agent connectivity. Omitted grace preserves inherited or factory behavior, explicit zero is a deliberate no-wait policy, and a positive powered-off grace is duration authority rather than a poll-count threshold. The no-policy powered-state path retains the legacy two-confirmation contract for migration compatibility; an applicable powered-state duration activates on the first stopped observation at or after elapsed eligibility.

Runtime eligibility uses server receipt time plus process-monotonic elapsed progress, never client/provider timestamps or poll counts. Duplicate reports advance no time, delayed polls count actual elapsed time, and wall-clock forward/backward changes do not change the decision. Pending state persists accumulated elapsed progress. Restart re-baselines the monotonic clock without counting unobserved process downtime, while legacy timestamp-only pending files are imported once into the elapsed representation. Recovery, explicit guest suppression, disabling offline alerts, and tracking cleanup remove both durable pending state and its transient monotonic baseline before a later outage can start.

Operator maintenance and intentionally-offline state are read only through the canonical unified-resource identity. One-shot and recurring maintenance use the same effective occurrence boundary. Exact-resource maintenance always applies; ancestor maintenance applies only when its persisted scope is resource_and_descendants. Overlapping exact and inherited windows remain suppressed until the latest active end, and every operator-state mutation reconciles the active-alert set immediately so a host window cannot leave child alerts visible until another detector write. Backup-aware offline deferral consumes fresh, matching, active task evidence, applies the configured post-backup grace, and always terminates at its hard cap. Missing, stale, future-skewed, finished, or mismatched backup evidence cannot suppress an outage. This policy changes alert activation only: notification delivery, recovery assurance, and customer-infrastructure mutation retain their existing owners.

internal/alerts/intent_policy_test.go and internal/alerts/powered_off_tolerance_test.go prove field-wise type precedence, normalization, exact zero/positive duration boundaries, VM/LXC coverage, duplicate and delayed reports, wall-clock changes, suppression reset, migration/restart continuity, backup hard-cap behavior, preview immutability, and first-match lifecycle identity.

The thresholds resource rows expose that same policy owner directly through a per-resource alert-delay action. The action carries the canonical resource id and the first supported CPU, memory, or disk signal into the versioned intent-policy editor, expands it, and leaves persistence and inheritance with /api/alerts/intent-policies. Resources without a supported metric open on state.offline. Threshold rows must not create a parallel delay field in the legacy threshold override document or save intent changes through the threshold-config endpoint.

Provider-observed storage and workload incident lifecycle

Provider observations use one stable alert identity composed from provider, connection-local resource identity, native signal identity, and canonical incident code. Synthetic TrueNAS pool, vdev, dataset, disk, app, and container conditions require two consecutive observations before activation and two consecutive healthy observations before recovery. A transient poll therefore does not fire or clear an alert. Active state, acknowledgement, suppression, severity changes, recovery, and history remain owned by the shared alerts runtime and survive restart; an absent resource or unknown collection result is not recovery evidence.

Native TrueNAS alerts remain authoritative for their equivalent condition. Their stable native ID suppresses only the duplicate synthetic pool/vdev state, missing-member, or scan condition; distinct read/write/checksum evidence continues to participate. Pulse does not send a second TrueNAS email stream. Intentional stopped apps and other stable conditions remain suppressible through the canonical per-resource or global alert configuration, and readonly replication targets do not produce incidents when native replication evidence classifies them as healthy.

Ceph contributes one cluster-scoped provider incident only for native HEALTH_WARN or HEALTH_ERR evidence. It shares the same confirmation, recovery, history, and suppression machinery but cannot manufacture a disk incident from cluster health.

internal/alerts/unified_incident_confirmation_test.go, internal/truenas/provider_pool_health_contract_test.go, and internal/unifiedresources/ceph_pool_health_contract_test.go prove the lifecycle and deduplication matrix.

Per-resource monitoring policy is an alert-runtime invariant

The canonical unified-resource operator state is honored without requiring an explicit versioned alert-intent rule. Factory behavior enables operator-state evaluation, and a signal rule cannot weaken a persisted resource monitoring or lifecycle policy. expected_offline suppresses only state.offline and incident.availability; muted, retired, and active maintenance suppress all signals. Unknown alert families fail safe to the default signal and cannot be hidden by expected-offline.

Every active-alert writer passes through the same operator-state gate, and notification delivery consults the same decision before quiet-hours policy. The active-store writer reports admission explicitly. Detectors must not add history, recent-alert state, fired events, rate-limit consumption, or delivery intent after a rejected write. Lifecycle evaluators report suppression without an activation transition. Restore and guest-identity migration must not revive acknowledgement or tracking state for a rejected alert. Repeated observations, policy removal or expiry, and post-mutation reconciliation are exercised across provider incidents, metric and canonical lifecycle writers in internal/alerts/alert_admission_test.go. Operator suppression is never converted into a quiet-hours replay, including for recovery notifications carrying stale replay metadata. The internal alert dispatch path applies this terminal decision even for an already-active alert between a policy save and its reconciliation: it neither invokes a delivery callback nor marks the alert notified, and it removes stale replay metadata. The read-only delivery diagnosis reports the operator reason as suppressed, not as a quiet-hours deferral. TestOperatorPolicySuppressesDirectAlertDispatchWithoutQueueReplay pins muted, expected-offline, maintenance and unaffected performance cases, including the immutable suppression-versus-deferral event distinction. After a persisted policy mutation, ReconcileResourceOperatorState resolves already-active records for that resource immediately; later detector writes cannot recreate them while suppression remains active. Existing prefix, tag, and pulse-no-alerts bulk rules remain compatible inputs and do not create a second per-resource state store. internal/alerts/intent_policy_test.go pins factory operator-state evaluation, writer gating, and active reconciliation.

Object drawers expose the active problem across identity aliases

An object drawer with active alerts must show the exact problem in its default Overview, not only an alert accent or aggregate status already present in the table row. Frontend matching accepts a bounded ordered set containing the canonical resource id and provider-authored, metrics-target, alias, and superseded identities. The same candidate set drives row alert decoration and drawer message selection so opening a highlighted object cannot lose the alert because the provider stream and unified-resource projection use different keys. This is a read-only projection of the existing active-alert map; it does not create alert truth, change acknowledgement, or add a detector or delivery path.

Large alert collections stay continuously scrollable

Resource alert tables, active-alert lists, incident timelines, and resolved history must preserve full filter, group, count, selection, and action semantics without mounting the full estate. Desktop tables use the shared PlatformWindowedRows owner; phone cards use PlatformWindowedList with feature budgets no larger than 32 items. Native scroll extent comes from spacer geometry, and wheel/touch projection must maintain a mounted directional runway so rapid scrolling never exposes a loading-looking blank section. Alert windowing is presentation-only and must not change alert truth, grouping, acknowledgement, resolution, or delivery state.

Destinations tab carries delivery evidence

The alert destinations tab is where the belief "delivery works" is formed, so it must carry delivery evidence, not only configuration. It renders the notifications-owned delivery log (AlertDeliveryLogCard fed by useNotificationDeliveryLog over GET /api/notifications/delivery-log) as a newest-first record of real alert delivery attempts with plain-language outcome labels, destination names resolved from the loaded webhook configs, failure classes, and secret-redacted error text. The card names its retention windows separately: completed attempts remain for seven days while dead-letter attempts can remain for 30 days. Retained terminal failures say that retries stopped, because a permanent verdict can stop the first attempt before its retry budget is exhausted. The description uses the same boundary and never infers attempt exhaustion from dead-letter status alone. It requests the bounded maximum of 200 rows, displays an absolute semantic timestamp for timeline correlation with relative time as secondary hover context, and says that test sends skip the queue. An unreadable log renders as unavailable rather than empty. While delivery needs attention the card sits directly below delivery health and recovery controls on the Notifications route; a degraded Overview warning links to that route so the diagnostic evidence and safe retry/dismiss actions are reachable from the warning. Otherwise the route answers "where do my alerts go" first: webhooks (named by the services they reach), then email and Apprise, then this card. A switched-off email or Apprise destination renders one line saying what turning it on does plus a "Show settings" control that reveals the form without enabling it or marking the page changed. A new webhook opens on the service choice rather than the generic JSON form. useNotificationDeliveryHealth updates the health snapshot and its availability in one batch so the attention state never reads a half-updated pair and remounts the warning mid-refresh. The same card interleaves alert-owned notification_suppressed and notification_deferred evidence from GET /api/alerts/events with those attempts, ordered by the recorded event or attempt time. Held rows name the affected resource, alert type, and plain-language policy reason even when no delivery attempt exists. The event read is independent and fail-soft: an unavailable event log removes held rows without delaying, hiding, or marking the notifications-owned attempt log as unavailable. Refreshing delivery evidence starts both reads from one operator action while preserving that failure isolation. Test-send results that report deliveryPaused surface as a warning toast, never plain success, across the email, Apprise, and webhook test actions in useAlertDestinationsTabState and useAlertWebhookDestinationsState. Delivery evidence remains notification truth: the delivery log card must not resolve, suppress, or re-evaluate alerts, and it does not alter the AlertConfig.enabled versus activationState ownership boundary above.

Overview states paused delivery once and keeps rows scannable

The Alerts overview is where a notification lands, so it answers "what is broken now" before configuration. When notification delivery is gated off, OverviewTab renders the shared AlertDeliveryPausedCard once, driven by useAlertDeliveryPausedReason (the same activation reading the destinations tab uses), with overview wording and a link to the Notifications route. While that banner is shown, per-alert delivery lines for the global holds notifications_inactive and notifications_disabled are suppressed; every alert-specific hold (cooldown, quiet hours, rate limit, flapping, suppression window, monitor-only) still renders on its row. With detection switched off the banner must not claim Pulse is still detecting alerts. Active alerts render as compact rows: severity on the left edge, icon and badge, the alert type, the start time as a locale-aware relative age on the overview's minute clock, Acknowledge, Snooze and Investigate visible, and Timeline plus resource monitoring policy behind a More disclosure that also closes an open timeline. A one-line summary beside the heading counts open alerts by severity and the triggered-in-24h total; threshold override counts are configuration and do not appear on the overview. None of this changes alert lifecycle, acknowledgement, snooze, or delivery evaluation.

Phone alert rows keep one action row

Below the sm breakpoint (640px) an active-alert row on the overview shows Acknowledge (or its reversal), Snooze (or Unsnooze) and More on one row; the Investigate split button moves into the More disclosure beside Timeline and the resource monitoring policy. From sm up the row is unchanged. Four actions had wrapped onto two rows under every alert, so a 375px phone showed about three alerts per screen (average row 208px on the mock estate). Snooze stays on the row at every width, so its dialog is never remounted when a phone rotates across sm; only Investigate moves, by the shared useBreakpoint() width, and renders once. The Acknowledged and Snoozed badges and the snoozed-until line stay on the row. More names its region through aria-controls only while that region is rendered. This is presentation only: acknowledgement, snooze, investigation and policy handlers are unchanged.

A guest alert's name opens that guest

On the overview, a VM or container alert whose platform page is Proxmox or vSphere links its resource name to that page with ?resource= set to the alert's own resource id. The shared workloads table resolves that id (including a Proxmox source id such as <cluster>-<node>-<vmid>) through the resource registry, opens the guest's drawer and reveals it, so a notification reaches the affected guest in one tap instead of the top of a page with every guest on it. Other alerts (hosts, Docker containers, Kubernetes, TrueNAS, Pulse system alerts) keep their platform page link, because their pages do not read ?resource=; Proxmox storage alerts open their pool (below). So do orphaned backup alerts (backup-subject:...), whose ids name no resource the registry can open. A guest's per-filesystem alert (<guest>-disk-<key>, marked by mountpoint or diskIndex metadata) names no registry resource either, so CheckGuest in internal/alerts/guest.go stamps its owning guest as guestResourceId metadata: the same guest ID the guest's own alerts are raised under, on the canonical-spec and legacy checkMetric paths alike (TestGuestFilesystemAlertsNameOwningGuest). The card links a filesystem alert to ?resource=<guestResourceId>, opening the guest whose filesystem filled; a filesystem alert without that key (raised before the stamp) keeps the platform page link and never falls back to its own id. The platform used for the monitoring policy menu is still read from the page path alone.

A Proxmox storage alert's name opens its pool

A storage alert (resourceType storage) whose platform page is Proxmox links its name to /proxmox/storage?resource=, so a full or degraded pool opens on Storage instead of the top of the Proxmox overview. ZFS pool-state, pool-error and device alerts are raised under <storage>/zfs-pool:<pool>[/device:<device>] ids, which the resource registry cannot open, so checkZFSPoolHealth in internal/alerts/storage.go stamps the owning storage as storageResourceId metadata (TestZFSAlertsNameTheirStorage); the card prefers that key, then the alert's own id. An unstamped composite id (raised before the stamp, or a pool-error alert whose counters have not grown since) opens Storage without naming a pool and never passes the composite id. TrueNAS and vSphere storage alerts keep their platform page, which does not read ?resource=.

Thresholds lead with each group's default limits

The Thresholds page answers "what will alert me, and at what limits" before any group is opened. Every collapsed threshold group whose table edits global defaults states them in its header through getAlertThresholdsDefaultsSummary in frontend-modern/src/utils/alertThresholdsSectionPresentation.ts. The helper reads the same column list and defaults object the group's resource table edits (each section hoists its columns into one module constant), and drops every metric isAlertResourceMetricOff reports, so the header can never show a limit the engine is not evaluating. Docker's restart loop (count within its window) and memory-limit warn/critical are stated as whole rules with their effective values: an unset value is the factory value, and a rule or level at 0 reads "off", as the engine evaluates it. Backup age is summarized by its warning and critical days only, because fresh and stale hours colour the dashboard and never alert. A group switched off as a whole shows its Disabled badge and no defaults line. The CPU averaging line counts every configured profile override, including ones only the API sets. CPU averaging is tuning, not the page's answer. It is one collapsed disclosure stating the effective window and the number of platform overrides, with the same per-profile selects inside, each a compact shared FormSelect with its profile name as a visible label. Docker and Machines list their resource groups before Docker-only rules and the SMART card. Threshold values, inheritance, metric evaluation windows and persistence are unchanged.

Off is 0 for Docker and PMG count thresholds, and per-row edits exist only where an override stores them

Docker's restart-loop and memory-limit thresholds and every PMG threshold are plain ints in DockerThresholdConfig and PMGThresholdConfig, and 0 is Off: the PMG checks already skipped a level at 0, and the Docker restart-loop check now turns off when its count or window is 0 while the memory-limit check evaluates warning and critical as independent levels. The settings Off toggle sends -1, so NormalizeDockerDefaults and NormalizePMGDefaults store any negative value as 0 instead of the factory value they used to substitute, which reverted every Off on save. To keep a value an older alerts.json or an import predates from reading as Off, AlertConfig and both threshold types decode omitted keys as ThresholdCountUnset, and the normalizers (also run by ConfigPersistence on load and save) replace that marker with the factory value before the config is used or written. A PMG quarantine growth level needs both its percentage and its minimum growth; either at 0 turns that level off, because passing one alone made the canonical spec invalid and skipped the whole quarantine check.

A Docker container override re-enables state alerts over a disabled Docker default only when it carries its own powered-off severity, the same rule applyThresholdOverride applies to every other resource. DisableConnectivity is a plain bool, so an override saved for a CPU threshold alone used to turn that container's state alerts back on.

The frontend mirrors the backend instead of second-guessing it. Backup and snapshot ages keep a critical of 0 Off, both in the editor's sanitizers (useThresholdsRecoveryDefaultsState) and in the saved payload (normalizeAgeWarningCriticalPair); only snapshot sizes raise a 0 critical to the warning size, as the backend does. Per-resource rows offer only metrics ThresholdConfig stores: alertResourceSupportsMetric gives Docker containers CPU and memory, and PMG servers nothing, and the PMG section has no bulk edit, because restart, memory-limit and PMG thresholds exist only as defaults and the backend dropped per-container and per-server edits of them on save.

Not covered here: AI Patrol's threshold adapter (internal/ai/alert_threshold_adapter.go) still maps an Off alert threshold to its factory value, so Patrol keeps flagging a metric whose alerts are off; honouring Off there is a Patrol product decision, because a zero Patrol level would flag every reading.

External watchdog is Pulse-availability evidence, not notification delivery

The Alerts destinations surface owns the operator contract for an external dead-man watchdog that detects loss of Pulse itself. It presents masked configuration, live heartbeat state, canonical-monitor progress, consecutive delivery failures, and the last restart interruption. The credential-bearing ping URL must never return to the browser after save; the configured state uses an explicit replacement placeholder and removal action. The watchdog remains active independently of alert activation, snooze, quiet hours, and notification delivery pause because those policies must not disable observation of Pulse. Its Refresh and Show/Hide actions are the shared outline Button (xs and settingsAction sizes), not a copied border shell, so they match the other destination controls and keep a 44px touch target on phones. Last success and Monitor progress are ages, so they read the frontend-primitives shared relative-time clock and keep moving while the panel stays open, and the panel re-reads GET /api/alerts/deadman/status in the background every 30 seconds. A moving age over a status read once at mount would claim the heartbeat stopped while Pulse kept pinging every minute, and a status that is never re-read kept its read-time age through a real stall, which is the one case this panel exists to show. A background re-read leaves the Refresh control alone, and a failed one keeps the last status and marks it unavailable. The status read has no timeout, so reads carry a sequence number and a read's outcome applies only when it is newer than the last read that settled: a success shows its status and clears the unavailable mark, a failure sets it. A slow or hung read can therefore neither overwrite a newer outcome nor block later reads, and reads slower than the poll still land in order. While the panel's poll runs, the next background read goes out at the first poll once the newest read has been pending at least 29 seconds (one second of slack absorbs timer jitter), and a Refresh is released by its own read, by any newer read that settles, or by the first poll at which it has been pending that long, so a hung Refresh frees within about two intervals of the click. The delivery log's relative hover time on each row reads the same clock: a logged attempt's time never changes, so its age is correct over a log that is only re-read on demand.

Watchdog transport and monitoring progress remain notifications- and monitoring-owned respectively. Alerts owns the system-alert projection: delivery failure, canonical-loop stall, restart interruption, and durable-state failure use stable system-alert identities, normal lifecycle/event history, and idempotent fingerprints. A restart interruption is raised into history even when the first successful external report immediately resolves it, so recovery does not erase the outage record. The watchdog must never invent resource identity or become a second lifecycle for infrastructure alerts.

Escalation targets are exact and critical repeats are bounded

An escalation level may address credential-free logical destination IDs: email, apprise, or webhook:<id>. Once exact IDs are present they are the authority for that level; a deleted or unknown ID fails closed and must never widen to a channel or every webhook. The legacy notify channel remains a mixed-version fallback for configurations that do not yet carry exact IDs.

Optional critical-repeat policy applies only after the final escalation level. It sends at most one repeat per checker pass, uses a normalized 5–180 minute interval, and stops while the incident is acknowledged, resolved, snoozed, or global detection or delivery controls are paused. A repeat is durable escalation evidence with an explicit repeat marker rather than a new incident or a replay of missed intervals.

Every escalation level delay uses the same 5–180 minute safety boundary at the frontend state owner and the backend configuration-normalization boundary. Malformed API payloads and manually entered values therefore cannot turn a level into an immediate escalation or an unreachable timer beyond the supported schedule range.

The Schedule surface loads the same email, Apprise, and webhook catalog as the Destinations surface. It shows disabled and deleted selections explicitly, prevents an escalation level from becoming destinationless, and explains the critical-repeat stop conditions before save.

Recovery authority and operator qualification fail closed

If a lifecycle append or active-state checkpoint cannot reach SQLite, the degradation marker is the crash-safe restart authority. The marker is an atomically replaced, file-synced, directory-synced envelope containing the complete active-alert snapshot; the JSON mirror remains a compatible recovery aid but is not required for a new marker. While the marker exists, later saves refresh that envelope and must not make SQLite authoritative again in the same process. Bootstrap repairs SQLite from the envelope and only returns authority after durably removing the marker. Legacy text markers may use a readable JSON mirror, while malformed or source-less markers defer authority rather than silently trusting a potentially stale projection.

tests/integration/scripts/run-alert-qualification.mjs is the alerts-owned operator qualification entrypoint. One run must cover crash-safe active-state recovery, snooze and escalation policy, destination severity and exact-target routing, a signed real HTTP webhook, durable delivery receipts and recovery, dead-man progress and restart-gap reporting, plus the production frontend against a managed local backend. The browser phase must exercise active-alert snooze, delivery diagnosis, restart persistence, unsnooze, incident evidence, history migration, virtualization, and clear tombstones without replacing the backend APIs with route mocks.

Noisy gauges use one trust-stable lifecycle

Memory, temperature, and disk-temperature warnings use one continuous stability window on both lifecycle edges: an unchanged factory configuration requires five minutes above the trigger to open and five minutes at or below the recovery threshold to close. A return to the hysteresis band or a renewed breach resets the recovery run. Critical metric evidence may bypass only this factory or legacy activation delay; an explicit alert-intent policy remains authoritative. A metric-specific delay, including zero, remains the simple operator override and becomes the matching recovery window for these gauges.

Recovery timing uses the reducer's monotonic runtime evidence when supplied, so wall-clock changes cannot prematurely close an incident. Missing or unknown observations do not constitute healthy evidence and therefore cannot advance a recovery run. Restarts conservatively restart an in-progress recovery window rather than manufacturing elapsed health.

For guest disk capacity, valid per-filesystem evidence owns the alert identity and suppresses the less actionable guest-wide aggregate. This guarantees one incident for one full filesystem while retaining the aggregate only when no usable filesystem evidence exists. OS-managed transient mounts, including macOS Gatekeeper App Translocation volumes, are removed at the shared filesystem collection boundary before they can enter capacity inventory or alert evaluation.

Shared-system correlation never merges detector lifecycles

An active alert may carry the optional typed correlation object with a shared-system kind, stable key, primary or supporting role, and an auditable reason. This is presentation-only incident context. Every correlated signal retains its own canonical spec, confirmation state, acknowledgement, recovery evidence, history, and notification decision; recovery of one signal must never clear another.

Proxmox connection availability is the primary signal for its pve:<instance> system. PVE node connectivity is a supporting signal through provider-owned instance membership. A linked host-agent connectivity alert is supporting only when monitoring proves the reciprocal node ID and agent ID relationship and the node supplies the same non-empty instance. Missing, one-sided, duplicate, or conflicting identity fails open and leaves alerts separate. Resource-path truncation, hostname similarity, and timing coincidence are forbidden as correlation evidence.

The overview groups exact-resource detector signals and explicit shared-system signals. It selects a declared primary when one is present, describes the rest as linked signals, and acknowledges each underlying alert individually. Older alerts and clients remain compatible because the field is additive and optional. internal/alerts/correlation_test.go, internal/monitoring/alert_correlation_test.go, and frontend-modern/src/features/alerts/__tests__/useAlertOverviewState.test.tsx pin lifecycle independence, fail-open identity, and client grouping.

Infrastructure incident synthesis is deterministic and evidence preserving

The same optional alert correlation boundary also admits the infrastructure-incident kind. It adds one of six typed failure classes (runtime, network-path, application-response, certificate, dependency, or evidence-coverage), an inference state (supported-cause or observation-set), the primary alert and resource IDs, sorted affected resource IDs, and a bounded observation list carrying alert, resource, severity, observation time, and evidence IDs. It never copies provider response bodies, credentials, arbitrary metadata, or error payloads.

internal/alerts/incident_synthesis.go derives this context from the complete post-recovery active detector set plus active canonical relationships from ResourceRelationshipsWithCanonicalParent. It may traverse at most eight dependency edges and records at most 100 observations. Exact relationship direction, confidence of at least 0.8, a current primary failure, and bounded event ordering are required for supported-cause. Healthy or disagreeing primary resource state, weaker topology, late timing, cycles, or incomplete support must remain an observation-set. Name similarity, resource-ID prefix truncation, message parsing, and coincident timestamps remain forbidden causal evidence.

Every grouped alert still owns its canonical spec, acknowledgement, history, timeline, evidence, and recovery. Synthesis is recomputed on each unified incident reconciliation and clears stale synthesized links, so primary-first recovery leaves the still-failing symptom visible as standalone immediately. Existing shared-system correlations remain authoritative and are not overwritten. Resource maintenance and monitoring intent continue through the canonical resource-policy and intent gates before synthesis; no group-local mute or maintenance state is allowed.

Only a supporting member of a supported-cause group suppresses duplicate notification delivery, with the typed correlated_primary reason and primary alert ID in the event log. The primary remains the one delivery owner. Supporting members also skip escalation levels and critical repeats while that supported cause is current; an already queued escalation is revalidated before delivery. If the supported primary recovers or the causal link is lost while its symptom still fails, the symptom regains independent delivery without a new occurrence. It receives its first notification only if it was never dispatched before grouping; an already-notified symptom does not duplicate on group dissolution. Acknowledgement, rate limits and other ordinary delivery gates still apply. observation-set members continue to notify independently because Pulse has not established causality. incident_synthesis_test.go pins classification, contradiction downgrade, bounded evidence, duplicate-delivery suppression, partial-recovery delivery and no duplicate after prior dispatch, escalation suppression, and queued-callback revalidation without muting observation sets.

Alert hydration is not resource admission

REST active-alert recovery may complete before a resource snapshot. Neither active incidents nor alert hydration completion establishes an empty resource estate or replaces platform navigation admission. Incident access must remain available during socket reconnect; this does not assert notification delivery.

Qualification additionally injects admission HTTP 503 on reconnect with populated inventory and checks incident access on desktop and the 390px mobile rail. Enabled acknowledgement controls prove access only, not persisted acknowledgement or external notification receipt.

Issue explanation follows the customer-journey qualification boundary, which separates scripted interaction proof from provider reasoning, remediation and repeatable customer outcomes.

Delivery health requests have latest-started ownership

Overlapping mount, configuration Retry and post-queue-action reads must not allow an older completion to replace newer delivery health. Success, failure, first-load completion and the refreshing flag belong only to the most recently started health request. A stale healthy response cannot hide degraded attention; a stale error cannot invent unavailability after recovery. Queue actions still refresh from the server, not from their affected count. This changes no provider acceptance, incident lifecycle or delivery guarantee.

The hook and destinations caller regressions in useNotificationDeliveryHealth.test.tsx and useAlertDestinationsTabState.test.tsx pin ordering and loading ownership. scripts/check-delivery-health-ordering.mjs exercises the real caller and card in Chromium with scripted API completions; it is not installed delivery proof.

Alert status distinguishes dispatch from destination evidence

The active alert card renders a valid diagnosis lastNotified timestamp as “Dispatch requested”, never “Notified”: the alert manager records this field before invoking delivery callbacks. A cooldown's nextEligibleAt is labelled “next eligible”, not a promised send time. Neither field proves destination acceptance or recipient receipt; that evidence must not be inferred from the muted presentation tone. Missing or invalid timestamps retain the existing pending/cooldown fallback; acknowledged alerts retain their badge without a second status line. No API field, notification policy or shared primitive changes.

The existing wrapping status text must remain readable at desktop and phone widths despite the longer labels. The presentation and Overview delivery-status tests cover the evidence boundary; scripts/check-alert-dispatch-copy.mjs qualifies the real Overview with scripted API data in Chromium, not installed notification delivery.

Unchanged active-alert JSON recovery checkpoints

The recovery-mirror writer canonicalises complete records before comparing bounded existing bytes. An identical regular file with mode 0600 is not replaced; acknowledgement, metadata and resolved/empty state changes still use the atomic synced replacement path. Missing, corrupt, insecure or symlink destinations are not accepted as unchanged. Failed writes remain retryable and the unchanged path retains directory sync so a prior post-rename sync failure is not silently accepted. Windows retains replacement when its reported mode does not match POSIX 0600.

TestActiveMirror* isolates this JSON path without database workers and covers ordering, fresh-manager checkpoints, state changes, file loss/corruption, failed rename retry and Unix destination hardening. This addresses one write source in #1966, not total installed write amplification: directory metadata handling, intent snapshots, changing alert histories and database writes remain separate. It changes neither alert latency nor the selected release candidate.

Resource incident reads retain lifecycle ownership

The resource incident hook gives each started read a unique per-resource owner. Only that owner may publish history, report a failure or clear loading. Reset invalidates all pending owners before clearing state; disposal invalidates them and prevents new loads. Overlapping reads for different resources remain independent. Closing a row still permits its in-flight result to populate the existing cache; reopening cached history and explicit refresh are unchanged. Requests are not transport-cancelled. No API, retention or notification-delivery policy changes.

The hook's ten ordinary regression/control cases cover success, catch and finally writes, reset/reopen and disposal. The existing panel tests cover its presentation. scripts/check-incident-request-ownership.mjs exercises the real hook and panel in Chromium at desktop and phone widths with scripted responses and fixture reset, overlap and unmount controls. It is component lifecycle acceptance, not an installed full-page or notification-delivery receipt. PR1973's proposed resourceIncidentError accessor is absent here; if introduced, its writes must obey the same owner check and gain a stale-error regression.

Destination server-error presentation

Delivery activity preserves the queue's server_error classification as “Destination server error”, rather than treating it as unclassified. When that class dominates retained terminal failures, the full health description directs operators to destination service status and server logs, then to retry retained deliveries once the service is available. Unknown classes still use the generic fallback; this presentation does not change retry scheduling or imply receipt.

The presentation unit tests cover the label and terminal guidance. scripts/check-delivery-server-error-copy.mjs renders the real health and log cards in Chromium at desktop, tablet and phone widths, asserting the copy, absence of page errors and horizontal overflow. Scripted props are component evidence only, not installed notification delivery.

Webhook test/save custom-field parity

The webhook form applies the same service-specific custom-field normalisation when testing unsaved data as when saving it. In particular, manually entered Pushover app_token/user_token aliases become token/user with the existing canonical-value precedence. Existing-field editing remains normalised on load. This changes neither delivery scheduling nor the meaning of a successful test. The registered WebhookConfig regression covers alias test/save payload equality; the browser fixture exercises the real form at desktop and phone widths with synthetic callbacks, not a hosted Pushover destination or installed delivery.

PBS capacity has one alert policy owner

PBS datastore capacity alerts are evaluated by CheckStorageWithCapacityTrend from fresh PBS polls. Storage defaults, canonical datastore aliases, per-resource overrides, hysteresis and predictive capacity policy govern that lifecycle. The fixed 90/95% PBS topology assessment remains resource risk evidence; its Pulse-generated capacity_runway_low incidents must not independently enter active alerts on either the datastore or the parent backup server. Existing copies retire through normal policy reconciliation, without deleting state or claiming that capacity itself recovered. Other datastore state/error incidents and native provider incidents retain their existing lifecycle.

TestPBSCapacityUsesStoragePolicyNotTopologyBands in internal/alerts/unified_incidents_test.go uses real registry projection to pin both duplicate symptoms at 97.9%, canonical-alias 99% versus 90% policy, existing alert retirement, threshold recovery, and preservation of datastore failure. The posture and roll-up tests use datastore state failures, independent of capacity. This relies on the live PBS poll evaluator; a release adaptation must include that evaluator rather than remove the topology alerts in isolation. TestPBSPolledCapacityRequiresObservedRecovery in internal/monitoring/monitor_pbs_coverage_test.go additionally exercises the 97.9% policy transition through synthetic PBS HTTP polling, storage conversion and unified alert synchronisation, including absence of duplicate parent alerts.

Alert history clear/read ordering

History read failure is not an empty result. The existing History tab announces an error and offers an explicit read-only retry. Pending or failed reads expose neither the frequency chart nor severity totals, and failure suppresses the ordinary empty-result claim. Same-range saved entries remain available with an incomplete/out-of-date warning, alongside live alerts. A snapshot loaded for another range is retained internally but excluded from that range's rows; switching range is not deletion. Search, severity and range remain URL-owned.

Only the latest request may settle rows, failure or loading. An explicit retry admits one pending logical read; existing API transport policy is unchanged. Organisation replacement clears the saved snapshot and read error; retired reads and clears cannot change the replacement view. Disposal prevents another retry or a late completion from modifying view state. An already-dispatched clear is not cancelled, and deletion/retention, active alerts, acknowledgement and notification delivery are unchanged.

useAlertHistoryState.test.tsx and HistoryTab.loadFailure.test.tsx exercise initial failure, retry, pending/accepted empty results, same-range retention, range isolation with live alerts, obsolete replies, context replacement and disposal. The parent-bound browser receipt exercises the real existing tab, hook, API adapter and stylesheet at desktop and phone widths with synthetic HTTP failures and deferred replies. These are source presentation/ownership controls, not installed backend recovery or destination delivery.

A successful history clear invalidates reads started before its completion and settles their loading state, so delayed responses cannot repopulate deleted history rows. Failed clearing leaves the pending read valid; later range refreshes remain available. This is view-state ordering, not a change to retention, active alerts, acknowledgement or notification recovery semantics.

The deferred-response cases in useAlertHistoryState.test.tsx verify successful and failed clearing plus subsequent range refresh. The registered scripts/check-incident-request-ownership.mjs browser proof also exercises the real history hook and administration card: load a row, start a pending range read, confirm clear, then release the obsolete response at desktop and phone widths. Scripted API responses establish component behaviour, not installed backend deletion or destination delivery.

Escalation callback admission uses current policy and occurrence

PrepareEscalationNotification rejects asynchronous snapshots after escalation or global alert disablement, inactive activation, recovery, acknowledgement, snooze, supported-cause symptom correlation, or replacement by another occurrence of the same alert ID. It returns the current alert and detached current per-level routing, not mutable manager state. The monitoring callback must use this admission before enqueueing. This check does not recall provider-accepted messages or make the subsequent notification enqueue atomic with alert resolution.

escalation_policy_change_test.go pins a short-lived alert and disabled policy before the former 180-minute deadline, plus current payload/routing selection, copy isolation, snooze and invalid-level rejection. These deterministic component tests establish neither installed delivery timing nor the cause of a delayed email.

Unchanged intent-pending checkpoints

The serialized active-alert checkpoint also compares the complete, sorted pending-policy JSON against a bounded regular file with mode 0600 before replacement. Empty and nonempty unchanged pending state leave the file intact; changed or cleared state, missing/stale content, unsafe destinations and failed writes retain replacement/retry semantics. This does not change intent timing, retention, alert freshness or the existing persistence durability contract. TestIntentCheckpoint* exercises the complete JSON-only checkpoint without starting database workers, including restart loading and Unix hardening. The active mirror's unchanged fast path alone does not cover this companion file. This is a narrow write contributor, not aggregate installed-write acceptance.

Coalesced asynchronous active-alert checkpoints

active_persistence.go admits one asynchronous checkpoint worker per Manager, not one goroutine and full serialization per mutation. Admission is protected by the shutdown lock. A worker starts immediately, takes fresh state through SaveActiveAlerts, and retains one follow-up request for mutations arriving during its current pass. Returning to idle and accepting another request share the same lock, so requests cannot be stranded at the idle boundary. The worker never holds the shutdown lock while acquiring persistence or manager-state locks; callers may request a checkpoint while holding the manager-state lock.

This coalesces derived snapshots, not lifecycle events or notification work. Synchronous lifecycle durability, checkpoint revision/failure-epoch checks, recovery-mirror replacement, intent timing and periodic saves are unchanged. An error or recovered panic releases admission after any requested follow-up; there is no automatic busy-loop retry. Once shutdown closes admission, the worker finishes only its in-flight pass; Stop joins it and writes the final current snapshot synchronously instead of draining redundant queued snapshots.

TestAsyncActiveCheckpointBurstIsBounded in internal/alerts/alerts_test.go blocks the real checkpoint writer, bounds a 512-request burst to one worker, then verifies current acknowledgement, incident age, metadata and pending intent through actual JSON persistence and a fresh Manager. The scheduler regressions in internal/alerts/async_checkpoint_test.go pin one follow-up for 10,000 in-flight requests, idle restart, error/panic recovery, retry after a real file failure, and shutdown admission/draining. These checks establish bounded checkpoint work, not the installed CPU/write rate or any reporter's recovery.

Warning-level destination preference persistence

Email API decoding and encoding, plus webhook edit/create and list presentation, preserve all three supported minimum-severity values: all, warning, critical. Unknown values still fall back to all. The email adapter round-trip matrix and webhook edit/save regression cover issue #2069 without altering delivery filters or sending provider notifications. Component proof is not installed delivery.

SMTP rate-limit preference persistence

The existing SMTP editor's rate limit is included by alertDestinationsModel.buildEmailConfigPayload on every destination save. Unrelated email edits and switching email off must preserve the loaded limit; an explicitly edited limit must reach the existing Notifications API unchanged. Zero remains the backend's default-limit sentinel, not an omitted or unlimited value. Missing saved limits still use the existing UI default of 60 per minute. SMTP masks, TLS settings, recipients, tags and severity preferences are unchanged. This repairs a missing UI-to-API field, not the schema or transport policy. Revealing switched-off email settings permits pointer and keyboard editing without enabling email. The status toggle remains off and Send test email stays disabled.

emailRateLimit.persistence.test.tsx connects the actual SMTP editor, destination mutation owner, payload builder and email API to synthetic wire assertions. The two payload-shape suites retain strict key checks including rateLimit. browser-tests/email-rate-limit.cjs checks save and readback on desktop and phone, including unrelated edits, disabled email and legacy zero. These fixtures do not send notifications or establish native SMTP throughput, release availability or the cause of an existing delivery failure.

Pulse Mobile push destination without an upsell

The Alerts destinations page renders the mobile push panel only when the instance has the relay feature. Community installs see no panel and no upgrade prompt, and never load or save relay push settings. Licensed instances keep the minimum-severity control and the link to Pulse Mobile settings, with the 31 March 2027 retirement date in the copy. Webhook, email, Apprise and dead-man destinations are unaffected.

Occurrence-qualified asynchronous recovery delivery

Every resolution funnel passes the actual ResolvedAlert to the callback bus. SetResolvedAlertCallback captures a deep-cloned alert plus its resolution time before asynchronous execution, rather than reconstructing an occurrence from a reusable ID later. Legacy ID-only subscribers retain their public IDs; a panic in one recovery consumer does not prevent the other consumers from running.

The live monitor uses this snapshot for AI recovery and notification delivery. CancelResolvedAlert removes only matching ID/start-time grouping entries and cooldown state. CancelByAlertOccurrence applies the same match to persistent pending, sending, failed and dead-letter firing rows, preserving newer occurrences, unrelated grouped members and their operational links. Missing start-time identity is a no-op, never a request for identifier-wide cancellation. The existing explicit ID-wide cancellation API remains available. Delivery gates, pending-only suppression counts, destination receipts, attempt history, quiet hours and best-effort treatment of already-started sends are unchanged.

operational_contract_test.go verifies snapshot isolation across a blocked callback and changed ID-indexed history, legacy identity and panic isolation. queue_occurrence_cancellation_test.go verifies nanosecond-distinct recurrences, grouping/cooldown preservation, missing identity and restart readback for all four cancellable row states without deleting failed audits or new-occurrence destination receipts. The monitor's connected tests use real PBS evaluation, normal callbacks, the persistent queue and a local HTTP destination; these receipts are not native provider or installed-release acceptance.

Pending cancellation is not proof that every destination missed the firing. The monitor delegates recovery eligibility to persisted occurrence/destination receipts even when an old retry remains or restart has lost RAM cooldown state. The partial-destination monitor control establishes a real local HTTP 200/503 split and preserves the newer firing and the old recipient's recovery.

Notification activity reports each evidence source independently

The existing Notifications activity card distinguishes an initial or pending read from a confirmed empty window. Held/deferred event reads have their own loading and unavailable state; a failed read withdraws their prior rows and keeps the warning until a later successful read. Only the current refresh owns those states, and disposed readers cannot publish late completions. A slow held-event read does not delay delivery attempts or disable their refresh.

A delivery-attempt failure withdraws only attempt rows, not independently readable held events. Conversely, missing held evidence does not hide readable attempts or imply that no notifications were held. The combined empty state requires both sources to be readable and no refresh pending. Retention, event limits, queue actions, permissions and notification delivery policy are unchanged.

useNotificationDeliveryLog.test.tsx, useAlertDestinationsTabState.test.tsx and AlertDeliveryLogCard.test.tsx exercise independent failure, recovery, initial loading and current-request ownership; the production Notifications tab browser fixture verifies the connected read/retry states at desktop and phone widths. These are synthetic frontend controls, not recipient delivery or containing-release acceptance.

Resource drawers read open alerts from the active-alert map

getAlertsForUnifiedResource in frontend-modern/src/utils/alerts.ts is the canonical lookup from one unified resource to its open alerts in the websocket activeAlerts map. It matches the alert keys the backend writes for that resource (unified id, canonical primary and superseded ids, Proxmox source id, metrics target, and for agent and Docker hosts agent:<id> and docker:<host source id> plus the component alerts nested under them) and orders them most severe first. A Docker container answers to the key the engine writes for it, docker:<host source id>/<container id> (or docker:<host source id>/name:<name> without an ID, per DockerContainerResourceID), so its drawer lists its own alerts; until 2026-10-14 the container drawer matched none of them. getOwnAlertsForUnifiedResource returns only the exact-key alerts, without the nested component alerts, and is what drawer charts use for dashed triggers: a host's Needs attention lists its containers, but a container's memory trigger is not the host's. Those rows use getAlertAttentionCopy, so a latched or recovering alert, whose message keeps its breach-time value, reads as "Memory 70% now, recovering" beside a chart showing 70%. The Machines/Infrastructure and Docker host drawers build their "Needs attention" rows from it; resources no longer embed an alert list, so no drawer can silently read an always-empty copy. getUnifiedResourceAlertStyles computes row highlighting from that same set, so the Docker hosts and Machines table rows tint for exactly the unacknowledged alerts their drawers list. Docker host rows previously matched the row id and display name against getAlertStyles; Docker alerts are keyed docker:<host source id> and carry the hostname as their node, so a host whose display name differs from its hostname was never highlighted, and Machines rows had no alert highlighting at all. getAlertStyles keeps its id-plus-node-name matching for the platform tables whose alerts are keyed that way.

Threshold rows mark the limits whose alerts are open

On Alerts > Thresholds a metric value carries a pulsing "Active alert" dot while an alert on that metric is open. useThresholdsTableState indexes the websocket activeAlerts map by each alert's resourceId and type (indexActiveMetricAlerts in frontend-modern/src/features/alerts/thresholds/thresholdsResourceModel.ts) and asks thresholdRowHasActiveAlert per cell. The map is keyed by alert ID (<resource>::metric-threshold:<metric> for threshold alerts), so a row never rebuilds a key; until 2026-11-01 it looked up <row id>-<metric>, a key no alert has carried since canonical metric IDs, and no row ever showed the dot. A row matches on alertResourceIds when its data projector sets them, else on its own id. Machine rows answer to agent:<host ID> (CheckHost's hostResourceID, while the row and its override keep the bare host ID) and to their disks' usage and SMART temperature alerts nested under it (agent:<host>/disk: and agent:<host>/disk_temp:), because the machine's Disk % and Disk Temp limits are what those disks inherit; sensor and RAID children have limits of their own. Machine disk and sensor rows are keyed on the alert resource ID already. Proxmox node rows answer only to their Proxmox node ID: a node merged with its Pulse agent hands CPU, memory and disk to the agent, and those alerts mark the Machines row whose limits judge them, never the node row's unused limits. Guest rows answer to getUnifiedResourceAlertResourceIds (frontend-modern/src/utils/alerts.ts), the exact keys getOwnAlertsForUnifiedResource matches.

SMART temperature continuity and pending-only policy intervals

Disk temperature activation/grace and recovery require continuous accepted observations. Empty SMART inventory, one disk omission, standby, non-positive readings, explicit missing/unavailable/unsupported collection and reporting-lease expiry interrupt pending/recovery timing without changing an open alert, its acknowledgement or lifecycle events. Positive legacy readings without temperature provenance remain accepted; an expired last-known positive value is not live health. Canonical and shadow reducers plus explicit intent grace are interrupted together. A non-empty inventory still resolves a departed disk only after three consecutive omissions; empty/expired inventory breaks that confirmation sequence.

Configuration saves reset pending-only runs when global agent policy, host or inherited node/guest overrides, or the applicable disk-type threshold disables evaluation, even if no report arrives before re-enablement. Existing known-type default normalisation is unchanged; the zero-type save control uses a custom type whose zero threshold the normaliser preserves. Only live pending identity/link/type context is retained; unrelated saves preserve a known enabled run and explicit overrides retain precedence over type defaults. Restored intent without that context restarts conservatively on a save rather than inventing continuous enablement. Cleared intent grace is checkpointed. Existing active-alert per-type/save/offline/removal resolution remains unchanged.

metric_observation_continuity_test.go and host_unraid_lifecycle_test.go cover valid/legacy/provenance controls, missing/rejected/expiry gaps on both timing edges, explicit grace, acknowledgement/event preservation, host isolation, configuration intervals, override precedence and departure. These deterministic source tests do not establish installed temperatures or native guest recovery.

SMART risk inventory gaps and disabled rules

Agent disk-health and disk-wearout alerts follow the SMART inventory, not filesystem usage cleanup. Empty inventory, standby and unobserved risk fields hold the existing occurrence and acknowledgement without recovery/refiring. Departure requires three consecutive non-empty reports omitting the disk; listing it again, an empty inventory or reporting-lease expiry restarts that confirmation. Both risk families on the same disk count once per report. Filesystem absence still resolves usage on its first missing report. Temperature pending/grace/recovery interruption and departure semantics remain unchanged.

Disabling every recorded cause releases that risk alert without a reading, both on a configuration save and on an empty report. Partially enabled causes, missing legacy cause metadata and unknown codes remain conservative. Re-enabling cannot raise an alert from absent evidence; fresh bad evidence uses the existing stateful refire/cooldown history policy. Node linking retains existing node ownership, including when the agent's SMART inventory is empty; host removal and policy disablement release agent risk state and stale absence counts. Counts are ephemeral and restart conservatively after a process restart.

TestHostSMARTRisk* in host_unraid_lifecycle_test.go exercises actual CheckHost, expiry, configuration save, firing/recovery callbacks and event history with synthetic telemetry. It covers both families sharing a disk, interruption/reconfirmation, partial rule disablement, re-enablement, legacy/unknown causes, filesystem and host isolation, removal and node-link ownership. These are source lifecycle controls, not collector, smartctl, installed notification or native disk-recovery proof.

Storage observation gaps restart confirmation and timing

An absent or unknown storage connectivity status interrupts activation, recovery confirmations and pending offline intent grace without resolving an existing outage or changing its acknowledgement. Capacity is independently observable while connectivity is unknown. Missing/unconfirmed-zero capacity, negative usage and offline/unavailable storage interrupt capacity activation, and intent grace timing; they do not supply healthy capacity evidence. Capacity recovery retains its existing immediate measured-clear behaviour, not the recovery stability window used by memory and temperature gauges. A confirmed empty store still supplies a genuine zero-usage recovery reading. Fresh observations must satisfy the existing full confirmation/delay after a gap. Interruption is scoped to the storage's canonical identity and applied to both live and shadow reducers; interrupted intent is checkpointed for restart.

TestStorageConnectivityObservationGaps and TestStorageConnectivityGapRestartsIntentGrace exercise the real storage entry point, acknowledged occurrence retention, neighbour isolation, fresh reconfirmation and empty intent checkpoint. Storage routes in TestMetricObservationGapRestartsActivation covers capacity gaps and explicit intent grace; TestStorageCapacityGapHoldsOccurrence verifies retained, acknowledged capacity incidents and genuine immediate empty-store recovery. Existing restart, empty-capacity and predictive-capacity controls remain required. These source controls are not native storage recovery or installed destination-delivery evidence.

TestStorageConnectivityGapWebhook also exercises the production Monitor callbacks, ordinary notification queue and guest-local HTTP receiver: the gap and first fresh healthy poll send no recovery; confirmed recovery sends one resolved receipt with the original occurrence identity, and the queue drains without failed/DLQ work. This is synthetic callback/receiver acceptance, not a native appliance or external webhook-provider result.

Composed host filesystem threshold ownership

The shared filesystem resolver preserves the explicit host or linked-resource Disk override ahead of per-hardware-type defaults, for both host reports and configuration-save reevaluation. A filesystem-specific override still takes precedence. Disabling that filesystem releases its canonical usage metric and pending intent, retaining the reviewed agent/node ownership semantics. TestHostDiskOverrideBeatsDiskFillByType checks agent and inherited overrides through a configuration save and the next report without losing the occurrence; TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold retains per-filesystem and per-type controls. This is source composition, not installed delivery proof.

Shared disk-temperature composition and pending ownership

Host temperature evaluation retains the upstream shared SMART/Unraid reading and per-type threshold judgement, including Unraid-only disks and competing readings on one device. It also retains reviewed observation-gap interruption and pending host/link/type context for configuration saves. A live Unraid-only row is not an observation gap merely because SMART is empty; absent or uncollected rows still restart confirmation. TestUnraidOnlyTemperatureRetainsPendingContinuity checks continuous, spun-down, omitted and type-off intervals under both grace paths; existing SMART pending, override, departure and upstream Unraid controls remain.

Applied configuration input ownership

Manager.UpdateConfig clones the supplied configuration before normalisation and retention. Callers keep ownership of its JSON-shaped maps, slices and threshold pointers, just as they own GetConfig snapshots. Subsequent edits do not alter running policy until explicitly reapplied; normalisation cannot rewrite an input concurrently encoded for persistence or a response. The existing typed clone's non-JSON custom filter-value limitation is unchanged. ApplyConfigUpdate already decodes an owned configuration under the manager lock and keeps its atomic partial-update semantics.

config_validation_test.go requires unchanged caller inputs, normalised live thresholds, isolated edits, ordinary guest CPU evaluation, explicit resubmission and concurrent response encoding/update under the race detector. Existing snapshot, persistence, partial-update and threshold tests remain required. This is source ownership acceptance, not installed notification or guest-agent recovery evidence.

TrueNAS disk heat policy for Patrol and Assistant

Manager.TrueNASDiskTemperatureThreshold(resourceID, diskType) resolves the truenas-disk tiers of effectiveAlertPolicyNoLock: the disk's canonical or identity-resolved override, then TrueNAS Disks defaults, then disk-type policy. A disabled disk/default or non-positive trigger leaves no heat policy. Global/platform alert switches silence alert evaluation without changing this policy, matching existing agent-disk semantics. Callers receive a copy; config normalisation, saved Off/custom/inherited values and alert severity are unchanged. IsTrueNASDiskResource uses the evaluator's classification, including merged agent/TrueNAS disks. Patrol/Assistant do not walk those disks to a synthetic TrueNAS system agent and accidentally apply an unrelated host override. Retained temperature readings remain unjudged, not affirmative recovery.

Verification: TestTrueNASDiskTemperatureThresholdMatchesTrueNASDiskAlerts and TestIsTrueNASDiskResourceMatchesTheUnifiedEvaluator in internal/alerts/threshold_resolution_shared_test.go cover tier, identity, Off/disabled, untyped, clone and nil-manager compatibility. TestAlertThresholdAdapter_TrueNASDiskTemperatureFollowsItsAlertTiers and TestPatrolJudgesTrueNASDisksByTheirAlertTiers cover adapter, Patrol triage, verification and Assistant context while preserving agent-host overrides. Frontend table/drawer tiers use the Web resolver described below; these backend tests do not establish browser or native appliance acceptance.

TrueNAS disk display reconciliation (8 October 2026)

The TrueNAS storage table and inline resource drawer resolve temperature by the disk resource's own temperature override, then truenasDiskDefaults, then the per-type disk policy. Explicit zero/negative thresholds or a disabled disk/TrueNAS default leave readings visible but unjudged. Host diskTemperature overrides and the agent alerts switch do not control API-only TrueNAS disks. The shared agent disk resolver and Proxmox pulse-relaxed guest resolution remain unchanged. Existing Off-save normalization and inherited TrueNAS threshold editors on main are retained, not replaced by the older PR stack.

Verification: src/utils/__tests__/metricThresholds.test.ts, src/features/truenas/__tests__/TrueNASStorageTopologyTable.test.tsx and the production table/drawer fixture browser-tests/truenas-disk-thresholds.cjs. The containing backend also resolves the disk-specific tiers for Patrol. Frontend checks do not establish native appliance or notification recovery.

Existing SMTP form exposes only persisted settings

The Email destination editor follows the public EmailConfig: SMTP connection, credentials, From/recipients, security, rate limit, tags and minimum severity. Reply-to, max retries and retry delay are not offered or fabricated by the UI; the save path never accepted them. UIEmailConfig extends that public type with concrete normalized password/rate-limit values, and the advanced form explains that Pulse manages retries. Existing server retry behaviour is unchanged.

EmailProviderSelect.supported-settings.test.tsx requires absence of all three ineffective controls, default/payload parity and supported edits/reloads when email is on or off. Existing component and destination-state tests retain provider, mask, routing and off-state behaviour; src/utils/__tests__/alertEmailPresentation.test.ts covers the retry explanation. Browser proof uses the production email panel, destination state, API and CSS with synthetic local endpoints. It is not native SMTP delivery or a release.

Notification recovery belongs to its mounted view

Overview and Notifications retire their delivery-health/recovery owner on unmount. Late health success, failure and loading completion cannot update that owner. Late Retry/Dismiss completion cannot publish a global toast, change its feedback or start health/activity readbacks after the view retires. A callback scheduled before retirement is checked again before invocation; already-started activity failures cannot restore retired feedback. Calls to retired health/action/clear handlers do nothing, including no confirmation.

An already-dispatched mutation or read is not cancelled, undone or declared failed by navigation. Active views retain the existing confirmation, acknowledgement, read ordering and persistent failure feedback. A replacement view owns its independent read and any deliberately confirmed action; source checks do not establish notification delivery or installed queue recovery.

useNotificationDeliveryHealth.lifecycle.test.tsx covers pending reads and both actions, callback scheduling/completion and retired entry points. useAlertDestinationsTabState.test.tsx connects both actions to the real Notifications owner, proving a late result cannot notify or start readbacks in its replacement while the replacement's own action still works. Browser controls use the actual Overview/Notifications components, API adapters, toast and CSS with synthetic local endpoints at desktop/phone widths.

Mounted held readings follow the shared clock (10 October 2026)

Node, guest and Mail Gateway drawer attention, and the Proxmox node temperature hover, age an unchanged metric evaluation on the existing shared clock. The metric presentation requires an explicit clock; drawer copy reads an accessor only for usable live status, leaving non-metric rows unchanged. The ten-minute stale boundary, evaluated breach date, legacy fallback, clear rule and operator memory basis remain independent. This does not infer collection or recovery. Mounted drawer controls and lazy-copy tests cover this boundary.

Guest filesystem settings do not assert current readings (12 October 2026)

The existing Proxmox Guest Filesystems settings rows use getWorkloadGuestDiskRead for the same provider read/power-state admission as Workloads. A prev- read reason labels retained usage Last known; an unavailable read or explicitly stopped VM withdraws used bytes and percentage, without deleting mount identity, saved overrides or threshold controls. Native runtimeStatus takes precedence over a canonical resource held online by another source. Independent LXC platform storage does not inherit VM-only read reasons. A new accepted reading clears the explanation, including zero.

The phone card displays the optional readingMessage as wrapping text beside the reading. The desktop row puts it on the name's line, ending in an ellipsis with the whole note in its title, so the row stays one line instead of wrapping a paragraph under the name (rows had reached 65px). Desktop rows are 33px: the 24px alert toggles and the 26px Global Defaults threshold inputs overhang the row instead of setting a 35px height. Missing filesystem samples keep saved settings editable with an explicit unavailable label. Existing numeric serialization, identity, persistence, active-alert linkage, edit-time row stabilisation and threshold evaluation remain unchanged. This presentation does not change collection, issue guest commands, prove thaw/writes/liveness, or establish installed/released recovery.

Verification: useThresholdsGuestData.test.ts covers every fixed read reason, power/read replacement, LXC and source independence, unknown-reason privacy and saved-override retention. ResourceTable.test.tsx covers visible/reactive text in both layouts. browser-tests/guest-filesystem-thresholds.cjs connects the production hook, table and editors on desktop Chromium and phone WebKit, including scoped refusal text, custom settings and same-VMID/mount separation.

Frontend alert matching for PBS and PMG resources also keys on their instanceId (getUnifiedResourceAlertKeys in frontend-modern/src/utils/alerts.ts), the id internal/alerts writes as ResourceID for both (pbs.ID, pmg.ID). The platform tables already matched it; the drawers no longer depend on a metrics target being set to find the same alerts. Inside a guest drawer, a physical disk's agent temperature alerts (agent:<id>/disk_temp:<device>) are matched through the guest's own agent id. No alert is created, cleared or retyped by this; it changes which open alerts a drawer lists.

Alert Investigate became a Patrol run on 2026-10-22, when the browser Assistant chat window was retired. Active alert cards and History rows keep one Investigate action (InvestigateAlertButton), which starts a targeted Patrol check through POST /api/ai/patrol/run with the alert's resource ID, identifier, and type, then confirms with a toast that points at the Patrol page. It is offered only to sessions that route accepts, read from the new settingsCapabilities.patrolRun security-status field. The "Ask Pulse Assistant" icon, the incident-timeline Assistant hand-off, and their hand-off models (alertAssistantHandoffModel.ts, incidentAssistantHandoffModel.ts, IncidentAssistantHandoffButton.tsx) are gone, as is the chat-only alertTargetTypes.ts resolver. The alerts.assistant.* catalog keys became alerts.investigate.* in English, German, and Spanish.