Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Alerts Contract
Contract Metadata
{
"subsystem_id": "alerts",
"lane": "L6",
"contract_file": "docs/release-control/v6/internal/subsystems/alerts.md",
"status_file": "docs/release-control/v6/internal/status.json",
"registry_file": "docs/release-control/v6/internal/subsystems/registry.json",
"dependency_subsystem_ids": []
}
Purpose
Host rows show their own alerts, including Proxmox physical disks (2 November 2026)
The Proxmox node row's alert stripe and tint read the node's own alerts
through getUnifiedResourceAlertStyles, the matcher the Docker, Machines and
Kubernetes host tables already use, instead of every alert whose node field
equals the node name. A guest's alert marks its guest row, not its host's, so
a tinted host row is explained by something on that host. The unified matcher
(getAlertsForUnifiedResource in frontend-modern/src/utils/alerts.ts) now
also admits a Proxmox node's physical-disk health and wearout alerts, raised on
<instance>:<node>:disk:<device> (ProxmoxPhysicalDiskAlertResourceID), which
only the name match used to catch; it builds that prefix from the node's
proxmox.instance and proxmox.nodeName, as the backend does. Node metric
alerts keyed on the node source id and agent component alerts under
agent:<id>/ are unchanged. alerts.test.ts and ProxmoxNodesTable.test.tsx
cover the admitted disk alerts, the excluded guest and sibling-node alerts,
and the row accent. No alert evaluation, identity, history or notification
behaviour changes.
The matcher also admits a node's own storage alerts (usage, offline and ZFS on
local, local-zfs and the like): alerts whose metadata.resourceType is
storage, whose node is the node's proxmox.nodeName and whose instance
matches its proxmox.instance. Storage sits on its own Proxmox tab, so the
node row is the Overview's only sign that its local pool is filling. Storage
shared across a cluster is recorded under the node cluster and never tints a
single node.
Runtime alert controls describe only supported scope (28 October 2026)
Container Runtime rows expose their existing Offline Alerts control, not an all-alerts switch or metric bulk editor that cannot mutate runtime settings. Visible guidance explains that connectivity settings do not mute container or machine alerts, and directs operators to existing per-container Alerts switches and saving; newly added containers need their own setting. No single-runtime child mute, inheritance, policy evaluation, persistence or machine-wide mute is introduced. Existing global-disabled connectivity precedence remains intact.
The shared resource table can omit the Alerts column for this connectivity-only
surface, including row, group, default, delay and empty-state column counts;
other tables retain their existing controls by default. Phone cards honour the
same choice. ResourceTable.test.tsx and ThresholdsTable.test.tsx verify the
omitted controls, remaining connectivity action, independent container drafts
and preserved machine/other-runtime settings. Desktop and narrow browser proof
must join the real projection, mutation hooks and consuming table, not an Off
label unsupported by the producer. This is truthful control presentation, not
fulfilment of discussion #3432/#2107 whole-runtime-and-children demand, recipient
proof, installed recovery or release availability.
Missing PBS node-status polls restart metric timing
When PBS node-status collection fails, absent CPU and memory observations pass through the shared metric evaluator. Pending activation, explicit intent grace and memory recovery timing restart without changing an open occurrence, its acknowledgement, trusted readings or lifecycle history. A fresh poll must satisfy the existing full delay; missing time cannot establish continuous evidence. Measured zero retains normal recovery, including immediate CPU recovery.
pbs_observation_continuity_test.go covers both metrics, legacy and explicit
grace, recovery boundaries, retained low/high values, exact-PBS isolation and
live/shadow agreement. Global pause, disabled policy and full-outage precedence
remain unchanged. No collector, connectivity, schema or transport changes;
synthetic manager/browser proof is not installed PBS or recipient recovery.
Storage capacity warnings require observed byte counters
Static usage and forecast alerts admit a finite percentage in range only with positive capacity and non-negative used space no greater than capacity. An empty-store recovery still requires zero used bytes and free bytes equal to total. Missing or contradictory capacity holds the existing occurrence, acknowledgement and trusted evidence, and interrupts pending static activation, forecast confirmations and explicit intent grace. A valid empty store still recovers; disabling usage policy releases capacity alerts even when the store is unavailable. Connectivity and attached ZFS health remain independently evaluated, and the global alerting switch remains a pause.
storage_capacity_observation_test.go exercises CheckStorage and its trend
entry point, static and predictive occurrence retention, false activation,
exact-store isolation, confirmation/grace interruption, policy and independent
connectivity controls. Its opt-in manager payloads feed the production Alerts
API/Overview browser proof at desktop and narrow widths. No collector, guest
operation, schema, history rewrite or notification transport changes; this is
not installed recovery or inbox evidence.
Unavailable Proxmox node disks are not recovered alerts
The node evaluator admits the same disk observations as the canonical node metric: positive capacity, finite percentage in range, and non-negative used space within capacity. Missing or contradictory readings hold an open disk occurrence with its acknowledgement and trusted evidence, and interrupt pending activation and explicit intent grace. An observed zero still recovers normally. Explicit disabled/off policy and an actually covering linked agent still release the node alert; a missing reading does not claim another owner. Independent CPU, memory and temperature evaluation and the existing offline policy are unchanged.
node_disk_observation_test.go exercises CheckNode, lifecycle events, measured
zero, both grace paths, exact-node isolation, explicit policy and real linked-agent
handover. Its optional manager frames feed the production Alerts API adapter and
Overview at desktop/narrow viewports. This aligns observation admission only: no
collector, guest operation, schema, historical rewrite or notification transport
changes, and no installed recovery or inbox claim.
Unavailable agent filesystems are not recovered disk alerts
Host filesystem thresholds use the canonical disk-observation admission: a finite percentage in range, positive capacity and non-negative used space no larger than capacity. A listed unavailable reading or an entirely empty report holds the existing occurrence, acknowledgement and last trusted evidence. It cannot raise a new alert, resolve one, or complete activation or explicit intent grace across the gap. A measured zero retains ordinary immediate disk recovery; no recovery delay is introduced.
An established linked-node summary identity keeps ownership only while its held disk alert remains open; interrupting a pending run returns ownership to the node. Current host, inherited and filesystem policy disablement, explicit agent removal and missing rows in a non-empty replacement inventory retain their ordinary cleanup semantics. No collection, guest operation, schema, historical metric, notification transport or provider lease changes.
host_filesystem_observation_test.go exercises the real manager, lifecycle
ledger, exact-host isolation, timing, policy, removal and linked-node controls.
Its optional exported manager payloads feed the real Alerts API adapter and
Overview surface in desktop/narrow browser proof, not a native recovery claim.
Listed backups need an observed success date to resolve age warnings
The backup-age evaluator retains an existing subject's occurrence, severity, acknowledgement and last trusted evidence when its still-listed rollup has no success date, a zero date or a future date. Such a date supplies neither a new age nor recovery. For multiple provider rollups of one subject, the newest usable success date wins regardless of input order; an unobserved date cannot hide an older valid breach or displace a valid recent recovery. A fresh valid date still recovers normally. Explicit policy exclusions, disabled rules and removal from the supplied complete inventory keep their existing semantics.
TestCheckBackupsRequiresObservedSuccessDates exercises the real manager with
PVE/PBS linked guests, PBS orphans and PMG subjects, occurrence/event retention,
provider ordering, same-VMID cross-instance isolation and policy/removal controls.
Its opt-in browser frames come from CheckBackups and GetActiveAlerts.
This is age-evidence handling only: no change to backup execution, source
inventory readiness, snapshot alerts, native freeze/thaw safety or notification
delivery, and no proof of a reported installation's recovery.
A listed snapshot with an unknown creation date is not recovered
Snapshot warning cleanup distinguishes an absent snapshot from a still-listed snapshot whose creation time is missing, non-positive Unix time or in the future. After current guest policy and overrides are applied, an unknown date retains only that snapshot's existing canonical occurrence, including its trusted age/size evidence, acknowledgement and observation time. It neither creates a new occurrence nor refreshes old evidence. A trustworthy dated observation, authoritative removal, exclusion or disablement still follows normal recovery. The global alerting switch remains an evaluation pause, not evidence of recovery.
This is deliberately conservative for both age and size warnings: an incomplete
snapshot observation does not independently update either metric. It introduces
no guest, snapshot, backup, filesystem or freeze/thaw operation.
snapshot_date_observation_test.go checks repeated gaps, epoch conversion,
exact occurrence/evidence preservation, lifecycle events, independent instances,
policy changes, trustworthy recovery and removal. The real manager's exported
frames feed the production Alerts API adapter and Overview/card browser fixture
at desktop and narrow viewports; this is not installed guest recovery.
Legacy alert-history compaction releases removed-row storage
Expiry, duplicate-occurrence compaction and ID removal allocate backing storage for the surviving history rows, not the original loaded row count. Unchanged expiry and removal passes keep the current array. This reduces avoidable runtime retention after loading a mostly-expired or noisy history without changing the 30-day cutoff, five-minute duplicate chain, earliest occurrence/latest observation, identity, ordering, evidence or empty-array shape.
history_retention_storage_test.go verifies exact survivor capacity, recurrence
and neighbour separation, detached migration snapshots, subsequent appends and
concurrent reads/writes. Connected primary, backup and retired-JSON loads compact
without rewriting the recovery files or changing their retirement state. JSON
recovery and event-log authority remain unchanged. Array-capacity evidence is not
whole-process RSS or attribution of a reported installation's memory growth.
Webhook list changes share the editor's acknowledged writer
Enable/Disable, deletion, bulk changes and form saves cannot overlap in one
webhook editor. An open add/edit draft also locks list actions so a saved
row change cannot invalidate that draft's enabled state or identity. Pending
list actions expose a live status and keep the saved inventory until the
server acknowledges each change. Deletion requires success: true before
removing a row; a resolved but unaccepted response retains it.
Bulk Enable/Disable captures only rows whose state needs changing and sends them sequentially. A rejected or unconfirmed write stops the remaining batch, retains earlier acknowledged changes and gives fixed partial-change guidance. No automatic retry or rollback occurs. Unmount withdraws remaining unsent calls, without cancelling or undoing a request already issued. Existing masks, routing, payloads, endpoint policy and notification delivery remain unchanged. The retired destination-tab mutation owner cannot dispatch another call, publish a late toast or replace shared inventory belonging to a newly loaded context.
WebhookConfig.mutations.test.tsx connects the real editor and destination
owner to deferred/rejected synthetic API replies for serial batches, shared
pending admission, no-op rows, partial failure, open drafts, deletion acceptance,
explicit retry and unmount. WebhookConfig.test.tsx retains service, payload,
form and list coverage with acknowledged bulk sequencing.
useAlertWebhookDestinationsState.test.tsx checks late create/update/delete
replies against replacement inventory and refuses calls from the retired owner.
Browser proof uses
the real existing section, hooks and API adapters at desktop/narrow widths,
with synthetic local endpoints; it does not establish provider delivery.
Existing settings saves retain the pending and newer drafts
One save per loaded editor owns a click-time snapshot of the alert policy and all separately persisted destinations. Save and Discard are unavailable while it is pending; fields stay editable. Completion clears the dirty flag only when no newer edits occurred. If newer edits exist, the acknowledgement says that the submitted settings were accepted and the newer changes remain unsaved. Failed writes stop later calls, retain drafts and restore Save for a deliberate retry. The existing multi-endpoint save is not atomic: earlier acknowledged writes are not rolled back or described as undone.
Entering the notifications tab does not reload an unsaved draft or an active save. Explicit Discard still reloads saved settings when no save is pending. Reload, organisation switch and unmount invalidate stale completions and stop unsent writes; they cannot cancel or undo an already-issued request. The newer context alone owns load results and pending flags. Apprise response masking replaces only the untouched submitted draft, not newer input.
Save admission additionally requires a successful saved-policy read in the current editor context. Initial load and every replacement withdraw that admission before resetting draft defaults. A failed read leaves editing and saving unavailable; an explicit reload must obtain the saved policy and finish destination loading before admitting writes. An older failed read cannot withdraw a newer successful context. Destination partial-load failures retain their existing separate save block. Defaults are not a recovered saved policy.
alertSettingsSave.acknowledgement.test.tsx exercises the real policy and
destination owners with deferred/rejected API replies, coherent snapshots,
newer edits, duplicate admission, tab navigation, retry, normal masking and
organisation invalidation. useAlertDestinationsState.test.tsx bounds the
same downstream snapshot; alertConfigPresentation.test.ts distinguishes
pending, accepted and newer-unsaved messages. Browser proof mounts the real
configuration surface at desktop and narrow widths with synthetic local
endpoints. These are source controls, not ordinary notification delivery.
The acknowledgement controls also require failed-initial/replacement-read
write refusal, deliberate reload recovery and superseded-error ownership.
PMG collection opt-outs are not recovery evidence
Missing node queues (including an entirely disabled collector) cannot resolve
aggregate queue-depth or oldest-message alerts: recovery and lower severity
require every discovered node's observation. A partial non-negative reading
is a lower bound and can still establish the highest configured severity
(critical when configured, otherwise warning), never downgrade or resolve it.
Observed per-node queues retain their existing independent evaluation. Missing quarantine totals preserve both category alerts
and add no zero growth sample; a successful complete zero remains real recovery
evidence. Explicit alert-policy disablement and the existing offline policy
remain separate and unchanged. pmg_collection_test.go and the corrected
nil-quarantine control in alerts_test.go cover missing/partial datasets and
complete oldest-message recovery, known-high partial activation and refusal
to downgrade a critical alert using an incomplete lower reading.
Connection confirmations cannot span unknown or disabled observations
The existing PVE/PBS/PMG/vSphere/TrueNAS connection-degraded detector requires three consecutive degraded observations to activate and three consecutive healthy observations to recover. Pending or unknown connection state is neither degradation nor recovery: it drops pending activation and intent grace, restarts recovery confirmation, and retains an active occurrence's identity, severity, acknowledgement, trusted metadata and observation time without fire/resolve callbacks. Paused connections, disabled connections and owning-resource offline policy disablement clear active alerts immediately and reset pending activation even when no active alert exists. Healthy evidence also ends pending intent grace.
Resets address exactly the connection's canonical discrete-state key, not another connection, condition or metric. Core and optional shadow reducer observe the same reset. Intent bookkeeping and its persisted checkpoint lose the interrupted run; re-enabled detection must establish fresh confirmations and any explicit grace. No new alert family, threshold, grace policy, route or external probe is added.
connection_confirmation_gap_test.go covers all five platforms, six interruption
paths, exact-connection isolation, fresh activation/grace, acknowledged recovery
and exactly-once callbacks with shadow parity. TestInterruptDiscreteRun in
reducer/discrete_test.go pins exact-key isolation, retained firing state,
acknowledgement, no fabricated resolution and idempotent interruption. These
synthetic source checks do not establish native outage or destination delivery.
Partial platform reads alert as degraded warnings
A platform connection whose poll answered but left one reading unread (poll
error category partial) reaches CheckConnection as state degraded. It is
a degraded observation of the same connection-degraded discrete state, at
warning severity, never critical: it activates after the usual three
observations, keeps an open outage open (a missing reading is not recovery
evidence), and moves an already-firing critical outage down to warning once the
connection answers again. Only active runs the recovery confirmation. The
alert message and metadata keep the failed read (stateReason, lastError,
lastErrorCategory: partial). No new alert family, threshold, policy or route
is added. TestConnectionDegradedPartialReadAlertsAtWarning in
operational_contract_test.go pins this.
Metric timing requires continuous observed evidence
Unknown metric evidence interrupts timing, not the occurrence. Missing or rejected unified values, non-finite current readings, and unavailable, incomplete or invalid rolling windows drop a pending activation run and reset an open incident's recovery run. Explicit metric-intent grace and its durable pending checkpoint are reset as well. Fresh evidence must complete a new full configured delay; unknown time cannot fulfil a sustained-for or recovery rule. Existing alert values, identity, start time, acknowledgement, notification and lifecycle records are not refreshed or resolved by that interruption.
Guest backup/QGA deferrals apply this boundary to retained memory and aggregate and per-filesystem disk readings. Failed or expired empty disk inventory interrupts tracked filesystem runs, including a guest's former node-scoped identities, without implying filesystem removal. Different guests, instances, metrics and independently current CPU/I/O remain independent. Host memory also interrupts timing when its usage is unknown; explicit rule disablement still uses its normal clearing path independently of telemetry. Node temperature's existing interruption rule is unchanged. Typed metric dispatchers must exclude metrics they evaluate separately from their own missing-value checks: node usage dispatch does not infer a temperature gap before the node's independent temperature evaluator observes its actual reading.
metric_observation_continuity_test.go verifies actual legacy/canonical,
unified, host and VM entry points, full-delay fresh activation/recovery,
explicit grace, empty/migrated filesystem inventory and rejected-value/source
isolation. guest_observation_test.go no longer treats a retained-reading gap
as elapsed breach evidence. These controls establish source lifecycle
truthfulness, not native freeze/thaw safety or external notification delivery.
Node temperature alerts judge the primary CPU reading
checkNodeTemperature evaluates models.PrimaryCPUTemperatureCelsius, the
hottest usable package, CPU-maximum or core reading, instead of the package
with a CPU-maximum fallback. The alert value is therefore the reading the
node's table, drawer headline and history show, and a core running above the
package now drives the alert. The interruption rule is unchanged: no usable
reading interrupts timing and never clears an open alert. Proof:
alerts_test.go (TestCheckNodeTemperatureJudgesPrimaryCPUReading).
Open threshold alerts carry a volatile live status — issue #2068
evaluateCanonicalMetricAlert builds MetricStatus from the evaluator's own
inputs on every evaluation, including the hysteresis hold and the recovery
run, where Value, Message and LastSeen stay at the last breach and no
branch may notify. The phase comes from the trigger, the clear level handed to
the reducer (a value at or below it counts toward recovery) and the reducer's
recovery run; progress uses that run's monotonic elapsed time. The status is
stripped from the active-state checkpoint, the recovery mirror and the
degraded recovery projection, so a held alert adds no durable writes, and
after a restart it is absent until the next evaluation. Resolved records carry
no live status. Stale-alert cleanup and the retention TTLs use the latest
observation (lastObservedAt), so an alert held below its trigger for over a
day is not auto-resolved as unmonitored. Frontend surfaces format the status
through features/alerts/metricAlertPresentation.ts and never re-derive the
phase or recovery timing.
The optional metricStatus.lastBreachAt dates the evaluated value retained in
Alert.Value, not the newer observedAt of a hold or recovery. A breaching
sample (including equality at the trigger) advances it; a held or recovering
sample uses the existing alert's LastSeen. Unknown/zero dates are omitted,
never replaced by the occurrence start, restart or poll time. It remains
volatile with the status, so restart rebuilds it from the saved breach and a
hold does not rewrite the recovery mirror. Resolution drops the whole status.
TestMetricBreachTimeTracksTheEvaluatedBreach checks holds, recovery progress
and reset, exact-trigger and renewed breaches. The existing live-status and
checkpoint controls also assert the breach date; restart controls cover both
durable active state and the JSON mirror, unchanged mirror bytes/mtime, and
legacy unknown-date recovery through a new breach.
Held breach dates in attention
getMetricAlertLastBreachAt selects a valid timezone-qualified ISO
metricStatus.lastBreachAt before a valid legacy Alert.lastSeen.
Unknown, malformed, impossible-calendar and Go-zero dates stay undated; the
occurrence start, current poll and recovery start are never substitutes.
Attention, overview/incident hover and active History share this selection.
Relative ages use the caller's clock. Missing live status after restart
keeps the existing recorded-message path. The Assistant hand-off that also
carried the date as Last Breach At model context was retired with the chat
window on 2026-10-22.
Verification: date-selection/attention controls in helpers.test.ts. These
are synthetic source controls, not native readings, inference, notification
provider delivery or restart recovery acceptance.
Alert card and open incident lead with the live reading — issue #2068
The Alerts overview card leads an open threshold alert with the presentation's
summary and clear rule, keeps the last breach as the summary's hover text, and
labels the alert and clear levels in the status's unit (temperatures follow
the viewer's °C/°F preference, as every frontend reading does). Without a live
status it keeps the message and labels an alert level only for metric types
whose unit is known; other thresholds (queue ages, counts) are stated in their
own message, so the card never guesses a percent. The resource incident panel
describes only the open occurrence whose alertIdentifier and openedAt
match an active alert's id and startTime (the incident store's occurrence
key), from that alert's live status, rechecked on the shared relative-time
clock so it turns stale when evaluations stop; closed occurrences keep their
recorded message. Every disk alert on a guest carries the guest's name, so the
presentation names the disk from metadata.label, as the per-disk message
does.
History leads its open row with the live reading — issue #2068
Alerts > History is a log of occurrences, but its open row is the one
occurrence still in progress: it already shows an Active badge and a running
duration, so a breach message beside them reads as the reading now. An active
history row for a threshold alert with a usable live status leads with the
presentation's summary; the desktop row carries the clear rule and last
breach in its hover, and the phone card shows them as muted lines because it
has no hover. buildAlertHistoryItems keeps description as the recorded
message, so search and the Assistant handoff still read the record, and
attaches the source alert as liveAlert only to active, non-system rows.
Closed rows, and open ones without a live status, keep their recorded message.
History row controls name the alert they act on
Every Alerts > History row, in the desktop table and the phone cards, repeats
Timeline, Resource and an icon-only Assistant button, and each used to share
one name ("Timeline", "Resource", "Ask Pulse Assistant about this alert"), so
a screen-reader or speech-input user could not tell rows apart.
AlertHistoryItemActions now names Timeline with
getAlertHistoryTimelineActionLabel (alert title, resource and the row's full
start time, since history repeats one alert over time; "Hide timeline for
..." while open, with aria-expanded) and Resource with
getAlertHistoryResourceActionLabel (rows on one resource share it because
they open the same panel). The icon-only Investigate button, which replaced
the Assistant icon on 2026-10-22, names its alert through the
alerts.investigate.iconLabel and alerts.investigate.iconLabelAt catalog
keys from the host's subjectLabel and subjectTime, while its hover title
stays the generic action name. Each name begins with
the visible label (AlertHistoryItemActions.test.tsx,
AlertHistoryMobileList.test.tsx, InvestigateAlertButton.test.tsx,
alertIncidentPresentation.test.ts).
The delivery switch has one name
The Alerts overview header switch for external notification delivery is
labelled Send notifications in every state
(alerts.activation.switchLabel, English, German and Spanish), and that
visible label is the checkbox's accessible name. It used to read
"Notifications enabled" or "Notifications paused" depending on its own
position, so an off switch beside "Notifications paused" read as "pausing is
off". The switch position alone now carries the state. The delivery-paused
banner and the pause toast still say Pulse keeps detecting and showing active
alerts, which keeps the 2026-07-18 boundary between detection and delivery.
Alerts.readOnly.test.tsx pins the stable name in both states, i18n.test.ts
rejects state words in the label in every locale, and spec 01 finds the switch
by the new name.
The paused banner claims only that delivery is off
Pending notification delivery (pending_review) covers both a fresh install
and someone switching delivery off, because deactivate writes the same state.
The config cannot tell them apart: activationTime can arrive in an imported
config without local activation, and the activation API can enable delivery
without recording one. So the paused banner's cause for pending delivery
(alerts.delivery.paused.reason.notActivated) reads "Notification delivery is
turned off." in every locale, instead of claiming it was never turned on.
OverviewTab.emptystate.test.tsx renders both histories through the real
activation store and hook.
Retained guest observations are not alert evidence
Guest memory marked last-known, unavailable or an unknown observation state
cannot start, advance, refresh or resolve a memory incident. A VM filesystem
read with a nonempty diskStatusReason, including retained prev-* values,
or unavailable aggregate usage is likewise not current disk evidence. It cannot
evaluate either aggregate or per-filesystem thresholds, replace an aggregate
incident with filesystem incidents, or clear filesystem alerts as if a failed
or expired inventory were authoritative removal. Metric filters do not match
these unavailable memory/disk values, including a fabricated healthy zero.
Existing incidents retain their last trusted value, occurrence, acknowledgement, observation time and lifecycle/delivery evidence until a current observation resumes evaluation. Successful current inventory still resolves removed filesystems. Explicit disablement, suppression and stopped-guest behavior remain effective independently of telemetry, including per-filesystem disable overrides on retained identities. Legacy unannotated memory and independent current PVE/Pulse-agent memory/disks keep their existing policy, even while QGA is deferred; current CPU and I/O continue to evaluate. Producers own freshness; no new age threshold, QGA request, payload, alert family or UI surface is added.
guest_observation_test.go pins typed/pointer VM/LXC observation admission,
delayed breaches, canonical incident/acknowledgement/ledger continuity, filters,
fresh recovery and removal, independent metrics and deliberate disablement.
TestGuestBackupDeferralAlertObservationLifecycle connects actual HTTP client,
poll builder, retained snapshot, unified view, History and the production alert
call. These are synthetic source proofs, not native backup/thaw, containing
release, installed delivery or external recipient acceptance.
Pulse system alerts retain their service scope in the existing UI
The pulse-system- identity prefix or explicit systemAlert: true metadata
marks a Pulse service condition, not a monitored resource. Overview retains
the warning, acknowledgement, snooze and occurrence timeline but offers no
resource link or monitoring-policy action. History preserves this scope for
active and retained conditions before any resource-name fallback: a machine
named Pulse is not evidence that it owns the service warning. Ordinary resource
alerts, including those called Pulse, retain their existing target affordances.
The Assistant explanation keeps the exact alert identity, message and approval-required boundary, without invented metric values, a guest target or resource Patrol. Reconstructed History explanations preserve the system marker; new system types need no frontend allowlist. Connected overview/history/handoff and investigation-button controls verify both authoritative markers, legacy metadata, name collisions and ordinary resource controls. Browser acceptance checks the actual components on desktop and narrow touch views. This changes presentation only, not alert evaluation, delivery, API or operator permissions.
Backup-age evaluation failures use the existing system-alert lifecycle
pulse-system-backup-evaluation is a warning about Pulse's inability to read
complete backup rollups, not a claim that a workload's backup failed. It has no
resource target and carries the standard system metadata. A stable fingerprint
keeps repeated failed polls on one alert without repeat notification; existing
backup-age alerts are retained. Only complete evaluation or deliberate disablement
clears the warning. The public message names the lost evaluation and points to
Pulse logs without including raw database paths or provider errors.
backup_evaluation_system_test.go verifies stable identity, idempotence, node
cleanup survival and recovery. Monitoring's connected evaluator tests exercise
persisted stale/fresh data and failures through this helper. No new route,
navigation entry, destination or monitored-resource health signal is added.
RAID spare-count lifecycle — issue #2369
Canonical host RAID alerts use the shared required-member health assessment,
not attached-device arithmetic. Known configured counts appear as
raidRequiredDevices metadata alongside unchanged total/active/failed/spare
observations. A healthy spare creates no incident; an actual missing member or
failed device still activates critical health. Scrub/resync/recovery and transient
operation expiry cannot mask a static deficit. Restoring all required members
with a spare resolves the existing incident through the same canonical state
identity, once, without a duplicate fire or resolve.
TestRAIDSpareCanonicalActivationAndRecovery checks event-ledger activation,
continuity and resolution; collector and ingestion tests check both health and
canonical alert severity. Routine scrub/resync remains silent, and genuine
recovery/reshape keeps its warning when no critical deficit exists.
Retained occurrence re-fires preserve incident truth
A reducer re-fire inside the retention window keeps its original occurrence
identity and reopens that incident. It is distinct from replaying the initial
firing. Canonical resource changes carry alert_started_at separately from the
transition timestamp, so history reconstruction also retains one occurrence.
Legacy records without that key retain their historical timestamp projection.
Checkpointed re-fire timestamps reject earlier resolution replay and historical
read repair. A later resolution still closes the occurrence, and replaying the
older re-fire cannot reopen it. This does not change alert retention or delivery.
TestMonitorLifecycleRefireReopensRetainedOccurrence covers both history paths,
canonical reconstruction without an incident checkpoint, repeated replay and
historical read repair. TestIncidentRefireCheckpointPreservesTransitionOrder
covers checkpoint reload on either side of the final resolution.
Automatic acknowledgement lifecycle
Cleanup-triggered acknowledgement uses the same canonical retention records and
reducer acknowledgement state as manual acknowledgement. It survives subsequent
metric evaluation and a short recovery/refire within the existing retention
window. Cleanup requests active-state persistence when it acknowledges alerts;
callbacks remain outside the manager lock. No retention duration is extended.
TestAutoAcknowledgementSurvivesMetricEvaluation in
internal/alerts/reducer_parity_ack_test.go verifies cleanup, the next metric
sample and a short recovery/refire. This is synthetic lifecycle proof, not
installed notification-destination acceptance.
Hourly tracking-map cleanup retains canonical acknowledgement records while
their incident is active, including legacy-keyed active snapshots and active
recurrences with an older inactive timestamp. A decision does not expire from
its acknowledgement age while the condition continues. Recovery starts the
existing inactive-retention window; the ordinary cleanup's one-hour expiry,
hourly stale-record fallback and explicit unacknowledgement remain unchanged.
TestTrackingCleanupPreservesAcknowledgedProviderRecurrence in
internal/alerts/reducer_parity_ack_test.go verifies manual and automatic
acknowledgement through provider-native pool incident reconciliation, affirmative
recovery, short recurrence, dispatch callbacks and JSON/durable checkpoint restart.
TestTrackingCleanupCanonicalAckRetentionBounds verifies active/legacy identity,
recent inactivity, both existing expiry paths and missing-timestamp fallback.
Modeled elapsed time and local callbacks do not prove installed recovery or
delivery to an external notification destination.
Mobile incident drawers transfer their exact context to Assistant and close through the shared explicit handoff callback. Keeping the source drawer above Assistant, or dropping its occurrence identity to make navigation work, fails the linked investigation journey. Both timeline and resource handoffs require mounted regression and final-build narrow browser proof.
Incident history preserves unavailable reads and provenance
Timeline and resource-history views use the shared incident projection. A failed canonical read stays visibly unavailable, including after an error toast expires or when cached evidence remains. Empty-history copy is shown only after a successful empty read. Retry or Refresh repeats the read without discarding saved notes. The event card keeps canonical observation/occurrence timestamps, source, actor and record ID behind the same accessible evidence disclosure. Unknown timestamps remain unavailable. Truncation is explicit. The Assistant handoff preserves occurrence identity and the latest bounded event window, and labels history as insufficient to establish current health.
Incident resource error-state ownership
Resource incident errors follow the same per-resource latest-request ownership as history and loading. Superseded or disposed reads cannot set an error; reset clears errors and invalidates pending reads. A retry clears the current error while retaining cached history until the current request succeeds. A stale success cannot clear a newer failure. This reconciles the merged error accessor with lifecycle protection, without changing notification delivery.
Verification: resource hook lifecycle assertions cover ordering and retry;
scripts/check-incident-request-ownership.mjs checks the real hook and panel
in Chromium at desktop and narrow widths, including error state and retry.
Its scripted fixture is not full merged-UI or installed delivery acceptance.
Retained-queue recovery feedback has no reading deadline
Retry and Dismiss failures have a view-local untimed equivalent beside the delivery-health card on Overview and Notifications. A health refresh, including a verified healthy result that removes the warning, does not erase a rejected action's message. Cancellation preserves it. Deliberate clear or the next confirmed action supersedes it; reloading or leaving the view resets it. The existing error toast remains transient.
An accepted queue mutation still emits success if the health read fails. That read owns the existing unavailable warning. A synchronous throw or rejected promise from the optional activity-refresh callback must not claim the mutation failed: retained feedback explicitly distinguishes accepted action from failed activity refresh. Older action/callback completions cannot restore feedback after a clear or overwrite a newer action's message. Neither acceptance nor a healthy queue proves intended-recipient receipt; no retry policy changes.
Proof: useNotificationDeliveryHealth.test.tsx pins expiry-independent state, health reconciliation, cancellation, callback failure and ownership; AlertDeliveryHealthCard.test.tsx pins mounted live-region composition and clear focus. scripts/check-recovery-feedback.mjs exercises the real Overview and destination views with scripted API responses, real toast expiry, keyboard actions/clear, unavailable-to-healthy reconciliation and three viewport widths. This is the main-only named bet “Recovery feedback without a reading deadline” from the current demand ledger, not a backend failure diagnosis or backport.
Overview delivery diagnoses use latest-started refresh ownership. Older bulk
responses cannot overwrite newer card notification status, and an empty active
alert set invalidates outstanding reads. Disposal also prevents updates. Failed
refreshes retain the existing snapshot; this ordering repair does not add a
freshness indicator or establish recipient receipt. Verify response overlap in
OverviewTab.deliverystatus.test.tsx, empty-set invalidation in
useAlertOverviewState.test.tsx, and rendered ordering at three widths using
scripts/check-alert-diagnosis-ordering.mjs.
Delivery-attempt and held-event reads in Destinations use latest-started
refresh ownership. A delayed mount response must not overwrite evidence from
configuration Retry or a queue-action refresh, including a newer unavailable
result. Attempt loading reflects only the latest attempt request; held events
remain independently asynchronous. Scope disposal abandons both reads.
Verification: ordinary overlap, held-event and disposal cases in
useNotificationDeliveryLog.test.tsx, registered mount/Retry coverage in
useAlertDestinationsTabState.test.tsx, and positive rendered-content assertions
in scripts/check-delivery-log-ordering.mjs. Scripted browser evidence is not
proof of backend delivery or receipt by an independent recipient.
The shared delivery-health card wraps action groups according to available space, retaining readable explanation width when Review, Retry, Dismiss and Refresh appear together. Its heading uses the opaque semantic foreground, not the translucent palette shades reserved for status backgrounds. Verify light/dark layouts at desktop, intermediate and narrow widths, including unavailable health, pending refresh and recovery.
The alerts overview offers the existing delivery-status refresh control when health is unavailable, including after a successful retained-queue action whose follow-up health read fails. The warning remains until a verified healthy read; a successful queue action alone is not evidence of delivery health. Normal degraded summary presentation continues to omit refresh.
Confirmed canonical metric recovery publishes the clearing evaluation's value,
observation time, and resolved metric wording in the snapshot consumed by
recent-resolution reads and notification callbacks. It must not reuse the last
firing sample, including when the clearing value is zero or guest identity
migrates between nodes. Clone the snapshot before refreshing recovery content;
preserve incident identity and start time. Missing or malformed telemetry is
not a clearing observation. This does not change explicit configuration-disable
or resource-removal resolution semantics.
TestAlertCharacterizationGuestMetricResolutionUsesCurrentNodeIdentityAfterMove
in internal/alerts/migration_characterization_test.go pins canonical identity
and clearing content together; TestPBSPartialMetricsWebhookLifecycle in
internal/monitoring/monitor_pbs_webhook_test.go verifies low/zero measurements
and invalid-telemetry suppression through a local HTTP receiver.
TrueNAS INFO provider incidents remain on the canonical resource but do not
enter the actionable active-alert lifecycle. Native NOTICE remains actionable
at informational canonical severity; it must not be discarded merely because
INFO and NOTICE share the monitor risk level. Native severity is preserved by
the provider projection.
An existing provider incident transitioning from a non-critical level to
critical re-enters canonical notification dispatch, subject to acknowledgement,
rate limits, and delivery policy. The transition preserves incident identity
and start time. Repeated critical observations and severity downgrades do not
re-notify; confirmed recovery retains the same lifecycle identity.
TestTrueNASNativeCriticalTransition in
internal/alerts/unified_incidents_test.go verifies these boundaries through
native TrueNAS WARNING, EMERGENCY, and INFO observations.
Warning and critical incidents retain normal routing; downgrade to information
clears an existing actionable incident through normal synchronisation.
TestSyncUnifiedResourceIncidentsTrueNASInformation in
internal/alerts/unified_incidents_test.go pins information, warning, recovery,
and preservation of provider resource evidence.
TestSyncUnifiedResourceIncidentsTrueNASNativeNotice additionally exercises
native projection, NOTICE activation without severity inflation, and confirmed
recovery to INFO.
Own alert identity, alert specs, evaluation, persistence semantics, and
operator-facing alert routing behavior for live runtime alerts.
Alert-history investigations on narrow layouts render in a shared drawer
outside the virtualized history row. The drawer title comes from the canonical
alert-incident presentation vocabulary. A resource-incidents panel embedded in
that drawer suppresses its standalone panel title so the investigation has one
accessible heading, while the desktop inline panel retains its own title.
Alert schedule delivery routing is one persisted, normalized contract.
schedule.initialNotify accepts all, email, webhook, or apprise;
missing, legacy, and unknown values preserve the backward-compatible all
target. The selected initial target owns firing, grouped, and recovery delivery,
while every escalation level retains its own independently normalized target.
Saving the alert configuration must update the live notification manager as
well as persistence so delivery does not differ before and after restart.
Configuration-change reevaluation may resolve metric-backed alerts against
their updated thresholds and may apply explicit resource-disable policies, but
it must not treat provider-owned incidents as missing thresholds. Unrelated
configuration saves preserve those incidents and their acknowledgement state
until their provider evaluator supplies recovery evidence.
Storage configuration re-evaluation must use the same ordered resource ID and
alias override lookup as polling. Static and forecast capacity alerts retain
storage policy aliases in durable metadata, including through JSON and SQLite
restore; a configuration reload must not fabricate recovery by substituting
global defaults for a still-applicable datastore override. Older PBS snapshots
without alias metadata may reconstruct the canonical datastore alias only when
the recorded PBS instance, datastore name and complete legacy resource ID agree.
Hyphenated names must not be split heuristically, and a same-named datastore on
another instance must not inherit the override. Explicit policy changes retain
normal resolution semantics.
TestPBSDatastoreOverrideLifecycleAcrossRestart and
TestStoragePolicyAliasesLegacyIdentity in
internal/alerts/canonical_stateful_test.go pin restored incident identity,
hysteresis, confirmed recovery, refiring, event counts and instance isolation
for both persisted-alias and legacy snapshots.
When a VM or container stops, guest evaluation resolves only metric-threshold
alerts whose observations are no longer meaningful. Backup-age and snapshot
posture remain owned by their posture evaluator and may stay active while the
guest is powered off; stopped state is not backup-recovery evidence.
Docker and Podman container CPU thresholds evaluate host-capacity-normalized
CPU percent, not Docker's runtime-native per-core percent. Alert metadata may
carry the raw per-core value and reporting host CPU count for evidence, but the
threshold value and canonical cpuPercent metadata remain normalized.
Docker and Podman container health is live evidence only while the container
state is running. Runtime APIs retain the last health-check result after exit
and suspend health checks in other non-running states, so those stale values
must clear rather than raise docker-container-health; the canonical runtime
state lifecycle owns the non-running condition.
Proxmox VM and LXC CPU thresholds consume the same canonical guest CPU-percent
normalizer as unified live state and guest history. Proxmox guest CPU is the
authoritative observation for that guest; a Pulse host agent running inside the
guest may retain its own agent alert identity, but its host CPU observation
must not replace or renormalize the Proxmox guest value.
Docker and Podman OOM alerts require authoritative runtime evidence: the
container must be stopped (exited or dead) and its reported OOMKilled
state must be explicitly true. Exit code 137 alone is only SIGKILL evidence;
explicit false and unavailable/legacy OOM state both fail closed without an OOM
alert. Recovery clears an existing OOM alert when the authoritative predicate
is no longer true.
Backup-age alert attribution of an unlinked recovery rollup treats the
subject ref's namespace field as a connection label, not a PBS namespace: it
may match a candidate guest's instance or node only by exact normalized
equality. Suffix matching is reserved for real PBS namespaces inside the
shared identity helpers; applying it to a PBS connection name
cross-attributes clusters that share a VMID, so an unlinked backup whose
label identifies no guest exactly stays on its generic rollup key. The PBS
submission-source learner that shares those identity helpers is backup
attribution state, not alert identity: it is built per evaluation from
positively attributed snapshots, never persists across evaluations, and is
inconclusive by default, so it can only narrow which guest an already-linked
backup belongs to and can never widen, merge, or move an alert's subject.
Availability incident and alert identity belongs to the source-owned
network-endpoint check. Correlation may project probe evidence onto a matched
machine, but it must not copy the check incident onto that machine or create a
second alert lifecycle. Failure, recovery, history, acknowledgement, and
notification routing therefore remain stable under relinking and restart.
Certificate validity uses that same source-owned incident lifecycle. Codes
prefixed certificate_ map to alert type certificate: expiry-window incidents
are warning, while expired, not-yet-valid, and untrusted incidents are critical.
A self-signed certificate is not an untrusted-chain alert. Certificate alerts
reuse canonical confirmation, acknowledgement, history, recovery, schedule,
intent, and notification routing and must not create a parallel delivery path.
Linux memory thresholds consume only canonical cache-aware usage. An explicit
usage-unavailable sample, a non-finite percentage, or contradictory
used/free/total evidence must not open or clear a memory alert. If such a
sample follows an active alert, the alert remains active with its last trusted
value until a subsequent trusted sample proves recovery; missing evidence is
not evidence that pressure disappeared.
Proxmox guest disk-read, disk-write, network-in, and network-out thresholds
consume only monitoring-owned valid rate observations. A valid idle interval
is explicit zero and may prove recovery; a first sample, missing/null counter,
partial response, or rejected out-of-order sample is unknown and must not
start, clear, or match a custom metric filter. Alert units remain MiB/s at the
threshold boundary (bytes/s / 1024 / 1024); that display/threshold conversion
must not be applied to the upstream cumulative-counter divisor.
Disk wearout is an evidence boundary, not a plain threshold. The field carries
two distinct absent states and one real zero: -1 is the canonical unreported
sentinel, and 0 is genuine evidence only from a device that reports endurance
at all. Rotational disks never do, so a 0 from one is absence. Alert
evaluation must gate on storagehealth.WearoutReported rather than carrying its
own inline boundary. A wearout arm keyed on > 0 silently exempts the single
worst reading a disk can publish, which let a spent SSD read critical on the
Physical Disks surface while raising no alert at all.
Proxmox wearout recovery requires three consecutive reported readings above
10% remaining life; a reading of exactly 10% still matches the alert. An
absent reading does not prove recovery, and a new low reading resets the
recovery run; until recovery is confirmed the
existing alert retains its low-life value and occurrence. This bounds the
resolved-then-refired notification loop when endurance data briefly looks
healthy (#2112), while a sustained replacement/corrected reading can still
resolve. TestCheckDiskHealthWearoutFlappingDoesNotRepeatNotifications and
TestCheckDiskHealthWearoutRecoveryAlertCleared pin dispatch/history
deduplication and confirmed recovery. This is synthetic source proof, not a
claim that the reporter's installed image has been repaired.
The Proxmox wearout value is percent life remaining, so its alert message names
it that way (SSD life remaining is 7%), matching the host-agent SMART
wearout_low summary; the threshold travels in the alert's Threshold field,
which the Alerts card and notifications render beside the message. Calling the
number "wearout" read as 7% worn and contradicted the low-life warning.
CheckDiskHealth logs a failed disk at error and a worn disk at warn only on
the evaluation that opens the alert (the canonical activated transition);
later polls that still find the disk failed or at or below the wearout
threshold, including mock mode's per-tick evaluation, log at debug. Logging on
every evaluation repeated one failed disk's error on every poll for as long as
it stayed failed. TestCheckDiskHealthLowWearoutCreatesAlert pins the message
and TestCheckDiskHealthLogsAlertOpeningOnce pins the log-once boundary at the
threshold, including a fresh error when a recovered disk fails again. That test
captures logs through a hook installed in the package TestMain, because
assigning log.Logger inside a test races with managers earlier tests left
running.
Host SMART counter growth is an event boundary rather than a warning on every
historical non-zero value. For an agent-only disk, the first reported UDMA CRC
error count establishes an in-memory baseline and raises no alert. A later
increase contributes crc_errors_increased as a warning reason to that disk's
canonical disk-health assessment; the next stable sample resolves the active
growth event while normal alert and notification history retain it. A lower
counter, including a reset or disk replacement, establishes a new baseline and
must not alert. Missing or negative evidence cannot create a baseline or prove
growth. Baselines are bounded tracking state, are discarded by a full alert
state reset, and expire after the standard stale-tracking window. Linked
Proxmox host agents continue to defer SMART risk alert ownership to the
provider-backed disk surface rather than creating a duplicate host alert.
TestCheckHostAlertsWhenSMARTCRCCountIncreases,
TestCheckHostSMARTCRCCounterResetEstablishesNewBaseline, and
TestCleanupRemovesOnlyStaleSMARTCounterSnapshots in
internal/alerts/alerts_test.go pin the baseline, growth, reset, recovery, and
retention boundaries.
Host SMART alert policy is configured through the resolved agent threshold chain, not embedded inside the disk evaluator. Missing SMART rule fields seed the backward-compatible policy: failed health is enabled; reallocated, pending, uncorrectable, media-error, and CRC-growth counters trigger at one; remaining-life warning/critical thresholds are 10/5 percent; and NVMe spare warning/critical thresholds are 20/10 percent. An explicit zero disables that individual rule and must survive normalization, cloning, persistence, and per-machine override resolution. Counter rules trigger when current evidence reaches the configured value, while the CRC rule applies the configured minimum increase to two successive reports. Endurance and spare values remain evidence-aware, and invalid percentage inputs clamp to 0..100 with critical no higher than its warning boundary. The Machines threshold surface must round-trip every rule without changing these defaults. Linked Proxmox host agents continue to defer risk alert ownership even when their resolved SMART policy differs.
Host SMART recovery is field-evidence-aware. A standby report preserves disk
identity but carries no health authority, so it must not open or clear either
the canonical disk-health or disk-wearout assessment. For an active SMART
alert, every still-enabled reason that raised the occurrence must be observed
again before recovery evaluation; an omitted counter or endurance field is
unknown rather than a healthy zero. Explicitly disabling the owning SMART rule
remains authoritative and may clear the alert without waking the disk.
TestCheckHostSMARTDiskAlertRequiresRelevantRecoveryEvidence in
internal/alerts/alerts_test.go pins the fire, standby hold, partial-evidence
hold, and authoritative recovery sequence.
Threshold sections are keyed by override identity, not by resource type. The
Virtualization Hosts section reads and writes overrides on the bare resource id,
while the Machines section resolves through the agent-derived identity
candidates. A resource that appears in both sections therefore has two different
override keys, and a threshold saved from the wrong one is written where
alerting never reads it, leaving the machine on the global default with nothing
on screen to explain it. A resource must appear in exactly one threshold section,
the one whose identity alerting honours. Standalone Pulse-agent machines belong
to Machines, so the Virtualization Hosts section is fed by provider-owned
Proxmox PVE nodes rather than by every resource of type agent. Canonical
TrueNAS and vSphere hosts are also represented as agent resources, but they
belong only to their platform threshold sections; admitting either to
Virtualization Hosts would normalize edits against Proxmox node defaults and
could silently discard a valid platform override whose value happens to equal
the Proxmox default.
QEMU guest-agent filesystems have their own threshold identity:
guest-disk:<stable-guest-key>/disk:<mount-device-key>. The stable guest key
must use the same cluster-aware alias chain as guest thresholds so a VM node
move does not strand its filesystem settings. A filesystem override may disable
that mount or supply its own disk threshold. Once a mount has a dedicated
override, it is evaluated only against that override and is excluded from the
guest aggregate disk alert; otherwise the aggregate would still fire at the
guest default and silently defeat a disabled or relaxed mount. The visible
threshold row keeps the live guestID-disk-* alert resource identity for
active-state and intent-policy linkage while persisting through the stable
guest-disk:* override identity.
Numeric threshold editing is owned by
frontend-modern/src/components/Alerts/AlertResourceMetricEditor.tsx.
The table and card layouts delegate global defaults and resource overrides to
that control, so input bounds, empty-input handling and Off/inherit transitions
cannot diverge by layout. Table rows save when focus leaves the owning row.
Focus movement between sliders, inputs, toggles and Cancel stays inside that
edit, so Cancel cannot commit the value before its click runs. Cards retain
explicit Save. The responsive table owner identifies the active layout.
Retiring a row during a resize transfers the draft and metric focus instead
of saving it. Both layouts focus the first supported numeric metric when
editing starts from the resource action.
Clearing an override shows the inherited metric state. When that default is
Off, the badge remains Off and the table slider stays disabled. Enabling
from that state stages the factory value in one toggle and leaves the group
default unchanged. Existing Off/inherit serialization remains compatible.
Table sliders sit above the numeric control so editing does not widen the table
and hide the resource identity. Backup, snapshot and offline controls keep their
separate configuration contracts. Global default cells never route backup or
snapshot into numeric thresholds. Host cards preserve the canonical cluster
context shown by table rows, so identically named hosts remain distinguishable.
Every threshold control names itself for assistive tech, because moving
between form controls reads a control's own name, not its column header:
the editor's input is " default" or " for " and its
On/Off badge "... alerts: On"; offline, backup and snapshot toggles and the
three-state Warn / Crit / Off control are "Offline alerts for : Warn";
display-mode value buttons are "Edit threshold for : ";
and the alert switches are "Alerts for ". Each name ends with the
control's visible word or value so voice control can say what is on screen
(WCAG 2.5.3). ResourceTable.test.tsx pins the names in both layouts.
A per-metric threshold is off whenever its trigger is <= 0. That boundary is
engine truth (internal/alerts/canonical_metric.go,
internal/alerts/config_runtime.go), not a display convention: -1 is the
value the editors stage, and 0 disables the metric just as completely because
earlier builds advertised 0 as the disable value and those overrides are still
on disk. Every threshold editor reads that same <= 0 rule through
frontend-modern/src/components/Alerts/alertResourceTableModel.ts rather than
testing for -1 inline: read cells, the row override editor, the global
defaults row, and the bulk edit dialog, on both the desktop and the mobile
surface. A surface that recognises only -1 reports On for a metric the engine
is not alerting on at all, and an unset global default carries the same
obligation, because absent is disabled rather than enabled.
Turning a metric back on must write a value the save path will actually
persist. Clearing an override only re-inherits the default, so it re-enables
nothing when the inherited default is itself off, and an undefined entry in a
bulk edit reads as unchanged rather than as on; in both cases the editor stages
the metric's enabled default instead, and only clears the override when the
inherited default is already enabled. Editors must also not manufacture an off
state out of an empty input: a cleared box is mid-edit, and coercing it to 0
disabled the metric in the engine while the row still showed On. Disabling is
the toggle's job, and the toggle stages the canonical -1.
The save path writes the backend's default contract, not the editors' sentinel.
Alert config normalization reads a negative trigger on a global default as
unset and restores the factory threshold, so buildAlertsConfigurationPayload
in frontend-modern/src/features/alerts/alertsConfigurationModel.ts writes
every Off global default as trigger 0. Sending the staged -1 through made
every Off default with a positive factory threshold come back On at that
threshold after save and reload; only guest and node CPU, memory and disk
escaped, because normalization leaves them alone. Overrides keep -1, which
the override path stores as written. Because -1 and 0 are the same
setting, the Global Defaults "Custom" badge compares Off values as equal
rather than by raw number, so the factory -1 I/O defaults do not read
Custom against a saved 0. An unset factory value stays distinct from Off,
since unset can mean the default follows another setting. A row's
custom-threshold marker keeps comparing raw values: it drives the row's revert
control and override removal, so an explicit -1 override must stay visible
against an Off default even though both alert the same way.
frontend-modern/src/features/alerts/__tests__/alertsConfigurationModel.test.ts
pins the payload for every section, and
internal/alerts/config_defaults_off_test.go pins that a zero default stays
off through UpdateConfig and raises no alert.
External availability-probe reporting loss owns one canonical
external-probe-unavailable alert per assigned agent, not one alert per target.
The lifecycle waits for the assignment grace floor before firing, keeps target
IDs as bounded supporting metadata, yields to the existing host-offline
lifecycle when the agent heartbeat is unhealthy, and resolves only when a
fresh result from the currently assigned agent arrives. Notification delivery,
acknowledgement, history, and recovery reuse the normal alert pipeline.
Storage capacity forecasting is an alert-grade evidence boundary, not an AI
prose feature. It requires at least 24 hours of valid history, hourly median
normalization, a fresh terminal sample, agreement between full-window and
recent positive slopes, and confidence of at least 0.80 before it can fire.
Raw poll count alone must never manufacture confidence. Forecast risk opens
only when projected exhaustion is within seven days (critical within one day)
and recovers only after the projection moves beyond fourteen days or trusted
evidence proves growth stopped. Missing, shallow, stale, or low-confidence
history is unknown rather than recovery for an already-active forecast.
Predictive and static usage policy share metric-threshold:usage as one
canonical occurrence: a forecast that later crosses the percentage threshold
keeps its start time, acknowledgement, history, and timeline instead of
emitting a forecast recovery plus a second capacity alert. The same contract
applies to Proxmox, Ceph, TrueNAS pools/datasets, and vSphere datastores through
their existing platform thresholds and disable policy.
Rolling metric evaluation is alert policy over monitoring-owned history, not a
second incident family. metricEvaluationWindows stores seconds by canonical
resource type and metric, with an explicit zero meaning current-value
evaluation. The Thresholds surface exposes both the global all rule and the
canonical guest workload fallback; concrete workloads inherit through the
same runtime chain (vm and app-container through guest, host-like agents
through node, then all) and must label the effective parent duration rather
than always presenting the global value. CPU seeds a five-minute default; only
CPU and burst-prone disk/network rate metrics may use
rolling averages, while memory, capacity, and temperature remain instantaneous
evidence boundaries. A window requires at least three samples, at least 80%
temporal coverage, and no gap larger than the bounded cadence allowance. Weak,
stale, or unavailable history is unknown: it cannot open or recover an
incident, and an existing incident retains its last trusted value. Ready
windows use time-weighted averaging so polling jitter cannot bias the result.
The evaluated value drives trigger, hysteresis, severity, delay, and recovery;
the current value, sample count, coverage, and window remain alert metadata and
operator-visible message evidence. Window changes never change the canonical
metric-threshold:<metric> identity, so start time, acknowledgement, history,
timeline, escalation, and recovery stay one occurrence.
Current metric observations must be finite in both instantaneous and rolling
evaluation, including a manager without a history provider. After selecting
in-range observations and last-duplicate authority, any non-finite History
sample or non-finite weighted result makes the whole window unknown. Invalid
samples are not dropped to manufacture coverage, substituted with an older
duplicate, clamped, or treated as zero. Both legacy and canonical evaluation
retain the last trusted incident value, identity, acknowledgement and delivery
state; no new lifecycle or notification evidence is emitted. Fresh finite
evidence resumes normal evaluation, and explicitly disabling a rule still
clears it independently of telemetry. windowed_metric_test.go verifies
NaN/infinities, arithmetic overflow, duplicate authority, input isolation,
JSON-safe incident snapshots, transition-ledger continuity and local dispatch.
These source controls do not prove installed or external recipient acceptance.
Active-alert restore is opt-out at construction. NewManagerWithDataDir accepts
ManagerOption values, and WithoutPersistedAlertRestore starts the manager
with an empty active-alert set instead of reading active-alerts.json. Mock
mode selects it: switching the mock toggle already clears active alerts, but a
process that boots with mock mode already enabled never runs that path and would
otherwise restore alerts raised against real infrastructure and serve them
beside fixture data. The option changes startup restore only. Alert evaluation,
persistence, notification, acknowledgement, and history are unaffected, and the
default construction path still restores.
Canonical Files
internal/alerts/specs/types.gointernal/alerts/specs/evaluator.gointernal/alerts/canonical_metric.gointernal/alerts/canonical_lifecycle.gointernal/alerts/unified_incidents.go5a.internal/alerts/incident_synthesis.gofrontend-modern/src/features/alerts/AlertOverviewActiveAlertsSection.tsxfrontend-modern/src/utils/alertOverviewPresentation.tsfrontend-modern/src/utils/alertResourceTablePresentation.tsfrontend-modern/src/utils/alertDestinationsPresentation.tsfrontend-modern/src/utils/alertIncidentPresentation.tsfrontend-modern/src/utils/alertSchedulePresentation.tsfrontend-modern/src/utils/alertWebhookPresentation.tsfrontend-modern/src/utils/alertActivationPresentation.tsfrontend-modern/src/utils/alertAdministrationPresentation.tsfrontend-modern/src/utils/alertBulkEditPresentation.tsfrontend-modern/src/utils/alertConfigPresentation.tsfrontend-modern/src/utils/alertEmailPresentation.tsfrontend-modern/src/utils/alertFrequencyPresentation.tsfrontend-modern/src/utils/alertGroupingPresentation.tsfrontend-modern/src/utils/alertHistoryPresentation.tsfrontend-modern/src/utils/alertSeverityPresentation.tsfrontend-modern/src/utils/alertTabsPresentation.tsfrontend-modern/src/features/alerts/types.tsfrontend-modern/src/utils/alertThresholdsPresentation.tsfrontend-modern/src/utils/alertThresholdsSectionPresentation.tsinternal/alerts/history.gofrontend-modern/src/stores/alertsActivation.tsfrontend-modern/src/utils/alertThresholdDefaults.tsfrontend-modern/src/utils/metricThresholds.tsinternal/alerts/config_facade.gointernal/alerts/constants.gointernal/alerts/model.gointernal/alerts/metric_hooks.gointernal/alerts/manager.gointernal/alerts/default_config.gointernal/alerts/lifecycle.gointernal/alerts/escalation.gointernal/alerts/callbacks.gointernal/alerts/config/types.gointernal/alerts/config/normalize.gointernal/alerts/config/identity.gointernal/alerts/notification_policy.gointernal/alerts/read_model.gointernal/alerts/pmg.gointernal/alerts/docker.gointernal/alerts/pbs.gointernal/alerts/storage.go47a.internal/alerts/capacity_forecast.go47b.internal/alerts/windowed_metric.gointernal/alerts/node.go48a.internal/alerts/host_agent_node_links.gointernal/alerts/host.gointernal/alerts/backup_snapshot.gointernal/alerts/disk_health.gointernal/alerts/metric_runtime.gointernal/alerts/health_assessment.gointernal/alerts/guest.gointernal/alerts/config_runtime.gointernal/alerts/active_persistence.gointernal/alerts/tracking_cleanup.gointernal/alerts/active_lifecycle.gointernal/alerts/active_cleanup.gofrontend-modern/src/components/Alerts/InvestigateAlertButton.tsxinternal/alerts/storage_override_identity.gointernal/alerts/unified_eval.gofrontend-modern/src/components/Alerts/ThresholdsTable.tsxfrontend-modern/src/features/alerts/thresholds/hooks/useThresholdsData.tsfrontend-modern/src/features/alerts/thresholds/hooks/useThresholdsPlatformData.tsfrontend-modern/src/types/alerts.tsfrontend-modern/src/stores/websocket.tsfrontend-modern/src/utils/alerts.tsfrontend-modern/src/utils/alertsActivation.tsinternal/operationaltrust/contracts.gointernal/alerts/operational_contract.gointernal/alerts/issue1497_test.gointernal/alerts/system_alert.gointernal/alerts/event_emission.gointernal/alerts/eventlog/eventlog.gointernal/alerts/history_projection.gointernal/alerts/history_migration.gointernal/alerts/active_state_bootstrap.gointernal/alerts/eventlog/active_state.gofrontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsxfrontend-modern/src/stores/alertAcknowledgementRequests.tsfrontend-modern/src/stores/websocket-global.ts
Shared Boundaries
History reads must not occupy the connection used to commit alert lifecycle
transitions. Disk-backed event logs use a bounded read-only WAL pool alongside
the serialized durable writer. An open history snapshot cannot block a durable
append or acquire mutation authority. In-memory test stores retain their single
connection because their private database is connection-local.
Every event-log connection, the writer and both readers, takes its SQLite page
cache from pdb.PageCachePragma in pkg/db/pagecache.go instead of a private
16 MB cache_size, because each connection holds its own cache. The modernc driver holds that cache outside the Go heap at about twice its size, where neither the garbage collector nor GOMEMLIMIT reaches it (#1803).
TestEventLogConnectionsUseSharedPageCacheBound pins all three.
Chronological replay selects bounded event IDs in time-index order before loading snapshots and details. Pages preserve occurrence-time and ID ordering, type filters, durable-ID watermarks and caller limits without sorting all retained payloads for every page. A full history read remains complete, and history reconstruction must not truncate silently at the public query limit.
Normal history polling shares one manager-owned chronological fold and advances
it only through newly committed event IDs. Its inclusive replay boundary keeps
concurrent appends out of the current pass. The event log remains authoritative:
retention removal advances a revision in the same transaction, late historical
events force chronological reconstruction, and replacing or disabling the store
discards the fold. Clear-history tombstones still remove preceding occurrences.
Live active-alert overlays and returned objects are independent clones. Explicit
Since queries preserve their existing event-window semantics without evicting
the normal full-history fold. Repeated attention requests must not replay all
retained snapshots independently or accumulate behind the history connection pool.
Since queries are answered from that same fold (#1803). Each occurrence keeps one
mark per folded event (time, durable ID and the LastSeen it contributed) and the
index of its latest acknowledged snapshot. A window starts at the occurrence's
first event that the log's own occurred_at >= since text comparison admits,
takes LastSeen and acknowledgement from that point on, and orders occurrences by
their first in-window event as a walk of only the window would. A read selects
and limits rows before copying, so only returned rows are cloned.
TestWindowedHistoryMatchesAWalkOfOnlyTheWindow holds this equal to an
independent walk of only the window, and
TestLimitedHistoryReadCopiesOnlyReturnedRows,
TestWindowedHistoryReusesTheFullHistoryFold and
TestHistoryReadResultsDoNotAliasTheSharedFold pin the copy, reuse and
isolation rules.
The fold keeps only each snapshot's newest evidence envelope
(newestHistoryEvidence, by observedAt then position), and history output
applies the same rule to live overlays and the JSON fallback. A resolved
alert's trail of up to 64 envelopes was about 83% of its snapshot. The event log
still stores every snapshot whole. TestHistoryEntriesKeepOnlyTheNewestEvidence,
TestHistoryOverlayKeepsOnlyTheNewestEvidence and
TestHistoryFallbackKeepsOnlyTheNewestEvidence pin this.
PBS node-status availability is distinct from connectivity. While a connected
PBS carries monitoring-owned NodeMetricsUnavailable evidence, CPU and memory
evaluation must not treat zero-valued placeholders as recovery. Existing
policy suppression and full-outage handling retain precedence. Valid low
measurements resume normal recovery, including callbacks and recent history.
Regression proof lives in internal/alerts/telemetry_quality_test.go and the
HTTP-to-manager lifecycle in internal/monitoring/monitor_pbs_coverage_test.go.
frontend-modern/src/stores/websocket-global.tsshared withperformance-and-scalability,unified-resources: the process-wide realtime store owns alert acknowledgement request lifetime, unified-resource state and fleet-scale connection and reconciliation.frontend-modern/src/stores/websocket.tsshared withperformance-and-scalability: the connection-owned realtime store is both the canonical alert truth boundary and the fleet-scale resource reconciliation hot path. That shared store normalizes slimmed broadcast resources at ingestion — expandingcapabilitiesRefthrough the statecapabilityCatalogand synthesizing the default policy posture for resources published without one — before any alert consumer resolves alert-to-resource identity or policy display. Alert override and threshold identity matching must consultcanonicalIdentity.supersededIdsalongside aliases, because broadcast aliases no longer duplicate superseded canonical ids.internal/operationaltrust/contracts.goshared withnotifications: the operational trust contract is jointly consumed by canonical alert lifecycle ownership and notification delivery linkage without making delivery state operational truth.internal/proxmoxidentity/backup_identity.goshared withmonitoring,storage-recovery: Proxmox PBS backup subject identity is a shared runtime boundary for monitoring backup freshness, backup-age alert attribution, and recovery-point guest mapping. Alert multiline field presentation is shared with frontend-primitives: notification, timeline, threshold ignored-prefix, and resource threshold note editors must compose the sharedFormTextareaprimitive for label/id/help wiring and textarea chrome instead of rendering raw native<textarea>shells in alert-owned runtime components. Alert resource threshold action presentation is also shared with frontend-primitives: row, mobile-card, global-default, and bulk-selection icon-only actions must composeActionIconButtonfor shared size, tone, focus, title, and accessible-name behavior instead of rendering local<button>plus inline SVG shells in alert-owned runtime components. The notification-delivery health card is an alerts-owned presentation over notification-owned queue truth. When retained terminal failures exist, the Destinations surface exposes explicit retry and dismiss actions with consequence confirmations, refreshes health and delivery history after either action, and never instructs the operator to delete queue storage. Platformconnection-degradedalerts are availability observations of their owning PVE, PBS, PMG, VMware, or TrueNAS resource, not a separate policy surface. They must honor that resource's disabled and connectivity-disabled override, the platform-wide alert and offline-alert switches, offline intent and quiet-hours policy, and must clear immediately when that policy becomes disabled. The connection snapshot must carry the owning monitor resource ID used by registry alias resolution; the ledger's display ID is not a substitute for that policy identity. A second connection detector must never notify around a resource's offline-alert toggle. Alert runtime state has one explicit ownership boundary:AlertConfig.enabledcontrols detector evaluation and in-product alert visibility, whileAlertConfig.activationStatecontrols external notification delivery only. The websocket store, resource-row presentation, navigation counts, and Alerts overview must preserve active alert truth while notification delivery is pending review or snoozed. Notification activation must not clear the browser active-alert store, suppress resource indicators, lock threshold/history configuration, or claim that monitoring has stopped. The websocket store's raw resource-delta baseline is connection-scoped. Closing or replacing a socket invalidates that baseline and its recovery throttle; a delta from the replacement connection must not patch the previous connection's raw snapshot. Only a full snapshot delivered over that socket may establish the new delta baseline. The same lineage rule applies independently to the connected-infrastructure and active-alert keyed projections. Dropping an oversized state frame invalidates all three raw baselines and their queued projection work. Oversized-state REST recovery may refresh the current connection's displayed resources, infrastructure, and alerts, but it is independently built and must remain delta-free; a later keyed delta without a socket-owned baseline is ignored and requests the shared throttled recovery path. A marker or baseline-less delta observed during hydration must coalesce one trailing REST refresh after the throttle window so the latest invalidation is not lost. Alert deltas still apply immediately when their socket baseline exists. Late socket callbacks and REST responses from a retired connection must be ignored, while a current oversized connection remains free to hydrate without waiting for the retired connection's request to settle. Cold active-alert hydration has a narrower recovery path than oversized estate state. Until the canonical store has accepted either a socket-owned alert snapshot or a successfulGET /api/alerts/activeresponse, the Alerts overview must exposependingorunavailabletruth and must not turn an empty local store into a "No active alerts" all-clear. An unintentional socket close before that first alert snapshot starts one throttled active-alert REST recovery from the canonical store, and an open connection that remains alert-snapshot-free for five seconds starts the same recovery rather than waiting for the ninety-second heartbeat timeout; pages must not own an independent alert fetch or cache. That response may refresh display truth but must leaverawActiveAlertsbaseline-free, and an alert revision plus request-generation fence must discard it if newer socket truth, a URL-scope switch, or disposal wins the race. Once a snapshot is known, reconnects retain it as the last confirmed projection rather than replacing known alert state with transport uncertainty. A failed recovery must remain visibly unavailable and provide an explicit operator retry. While the document is hidden, that same connection-scoped baseline must keep accepting resource deltas without reconciling the visible resource store on every message. The store accumulates changed resource IDs (and their per-key change shapes, unioned across the hidden ticks with unknown-shape contamination) and performs one canonical catch-up reconciliation onvisibilitychangeto visible. Alert and resolved-alert truth continues to update normally while hidden; this resource optimization must not defer, clear, or reinterpret alert lifecycle state. Active operator input applies the same separation: scroll, wheel, pointer press, and key activity may defer resource deltas, reporting projection, and the shared realtime tick token until the input-idle flush, but alert and resolved-alert payloads must still commit on the message that carries them. The input gate must never delay acknowledgement, recovery, navigation counts, resource alert indicators, or notification-facing alert truth merely because the resource projection is waiting for an idle window. The visible-tick resource commit may apply metrics-only rows as per-key subtree writes instead of whole-row reconciles, but alert truth is outside that fast path entirely: active-alert and resolved-alert stores keep their own commit path, resourcealertsfacets are not in the fast-path allow-list, and a row whose patch touches alert-relevant structure always takes the full canonical merge. That commit path replaces each stored alert with the server's payload for its id instead of merging into it. A field the server omits is removed from the store, whether the payload came from a socket snapshot, a keyed delta, or REST recovery: a threshold alert restored after a restart carries nometricStatusuntil its next evaluation, so the previous live reading must not keep reading as "now"; an unacknowledge from another session clearsackTimeandackUser; a droppednodeDisplayNameormetadatadisappears. Resolved alerts follow the same rule. The pending acknowledgement hold still skips a payload that contradicts an unconfirmed local acknowledge or unacknowledge, and the confirming payload then replaces the optimistic copy. Because a replacement reconciles the stored alert in place, local optimistic updates are stored as copies, so it never rewrites a record a caller keeps for rollback. The store'supdateAlertholds the only optimistic copy of an acknowledgement.useAlertAcknowledgementStatewrites through it and keeps no override of its own, so once the server confirms it, a later server change such as an unacknowledge from another session reaches the open tab. When no hold is pending and the stored alert already shows the local acknowledge or unacknowledge, the store sets no hold and keeps the server'sackTimeandackUser: the request's own broadcast can reach the socket before its HTTP response, and a hold would then wait for a second confirmation and skip the next real change. Every local acknowledgement write supersedes a REST recovery in flight: the store discards that response, which the server may have built before accepting the write, and fetches again once it settles. A hold that gives up after 15 seconds re-syncs from/api/alerts/active, waiting for a recovery already in flight and fetching again if that one applies nothing, because the payloads the hold skipped may not be followed by another alert change.websocket-unified.test.ts,useAlertAcknowledgementState.test.tsxanduseAlertOverviewState.test.tsxpin these paths. Operational evidence and lifecycle identity are typed throughinternal/operationaltrust. Evidence envelopes distinguish completeness, confidence, permissions, freshness, correlation, and bounded provider detail. Evidence and transition identifiers are deterministic under retry. Active and resolved alert compatibility payloads carry an additive canonical operational record, latest transition, and bounded evidence envelopes; legacy alert paths must migrate throughinternal/alerts/operational_contract.goand name their limited provenance honestly rather than inventing confirmed provider evidence. Acknowledgement remains distinct from resolution, and every resolution transition references recovery evidence separate from its trigger evidence. A resolved alert may carry a typedAlertResolution(internal/alerts/resolution.go) when it closed without its condition being observed to clear. The one reason so far,moved_to_agent, describes a metric alert whose resource's linked Pulse agent now owns the metric.releaseCanonicalMetricAlertis the disabled-threshold release extracted fromevaluateCanonicalMetricAlert: a caller releasing a metric for any other reason passes a resolution, andclearAlertWithResolutionstamps a copy on the removed alert before it is resolved. The resolved callback snapshot, the history row, the resolved event-log entry (the reason code, withSummary()as its message) and the Alert JSON then carry it. Resolved consumers still receive the close, because integrations close their incident by alert ID. The synthesized closing evidence carries the resolution reason and summary instead oflegacy_alert_recovery_projection, and the resolve transition carries the summary as itsreason. Its cause isownership_transferred, anoperationaltrust.TransitionCausethat may only enterresolvedand only with evidence (resolvedTransitionCauseinoperational_contract.go), so Patrol's timeline never labels a handover "Recovery evidence" and alert-quality telemetry does not count a handover during a snooze as resolved while snoozed. A nil resolution leaves the disabled-threshold path and every recovery onrecovery_evidence.TestReleasedMetricAlertCarriesResolutionInsteadOfRecoveryandTestResolvedHistoryRowKeepsResolutionWithoutEventLogininternal/alerts/operational_contract_test.gopin this. The in-app readers carry it too.GetRecentlyResolvedprojects it intomodels.Alert.Resolution(reason, successor and theSummary()text) for the state snapshot, the websocket and the assistant and Patrol prompts, pinned byTestRecentlyResolvedCarriesHandoverResolutioninhistory_test.go. The alerts history lists such a row asmoved to agentinstead ofresolved, even when it was acknowledged, with the account in the badge title and on the phone card (buildAlertHistoryItems,getAlertResolutionDetail). The resolved lifecycle change written to the resource timeline carries the summary as its reason and the code asalert_resolutionmetadata, so the row's incident timeline says the alert moved instead of "Alert resolved: ".
Extension Points
- Add new alert rule kinds in
internal/alerts/specs/ - Add typed collector/builders in the resource-specific checker owner or
internal/alerts/metric_runtime.go - Add identity/persistence updates through canonical alert helpers only
- Add or change alert history persistence through
internal/alerts/history.go,internal/alerts/history_projection.go, andinternal/alerts/eventlog/using normalized owned storage roots and fixed storage leaves only. Event-log projection changes must preserve parity with the in-memory JSON-history model, and migration changes must retain a retryable source or backup until every legacy entry is durably imported. The event log also owns per-consumer projection watermarks inalert_store_meta(projection_watermark:<consumer>): replay consumers pass their watermark as the walk'sAfterIDcursor so boot-time repair visits only the un-projected tail, and lowering a watermark (including to zero) is the supported way to force a full replay after a projection store rebuild. - Add or change locked alert-investigation commercial handoff behavior through
frontend-modern/src/components/Alerts/InvestigateAlertButton.tsxwhile preserving the shared upgrade-navigation contract; the alert surface may route to the canonical destination, but must not emit browser-local upgrade metrics or present Pro-required copy when prompt suppression applies. - Add or change frontend metric color thresholds through
frontend-modern/src/utils/metricThresholds.ts,frontend-modern/src/utils/alertThresholdDefaults.ts, andfrontend-modern/src/stores/alertsActivation.tsso browser display colors consume the same configured alert thresholds and override identity chain as the alert runtime instead of hard-coded per-surface thresholds. - Add or change alert investigation through
frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx. On resource-backed alerts Patrol is the doer: the action runs a manual scoped Patrol trigger through theai-runtimemanual Patrol route contract (POST /api/ai/patrol/runwith the alert's resource, identifier, and type), and Patrol's findings land on the Patrol page. The button stays hidden when AI is not configured, when the session cannot start a Patrol run (settingsCapabilities.patrolRun, which mirrors the run route's RequireAdmin +ai:executegate; auth-free installs pass), and for Pulse system alerts that have no resource to scope. A click while the request is in flight starts no second run. Alerts must not grow a second AI path beside it; the Assistant chat hand-offs that used to sit here were retired on 2026-10-22. - Keep alert incident timelines in place through
frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx: the incident panel and timeline cards carry sanitized incident facts, raw command/output details stay in the incident or approval surface, and the timeline must not add chat hand-offs, suggested prompt chips, or route-owned remediation instructions. - Add or change alert target resource types through
internal/alerts/specs/types.go,internal/alerts/config/identity.go, andinternal/alerts/unified_eval.go. Supported target types must share the unified evaluator, the canonical threshold/override identity chain, and the standard notification delivery path. - Add or change the alert notification destinations catalog through
frontend-modern/src/features/alerts/tabs/DestinationsTab.tsxandfrontend-modern/src/utils/alertDestinationsPresentation.ts. The destinations surface presents mobile push (Pulse Mobile paired through Relay) alongside email, Apprise, and webhooks so phone delivery is discoverable where alert routing is configured. It may read and update the Relay destination's alert minimum severity, but routes pairing and connectivity setup to the canonical/settings/system-relayRemote Access panel rather than duplicating Relay runtime state. When therelayfeature is absent it gates through the sharedFeatureGateSectionand upgrade-navigation contract, rendering no upgrade call-to-action when prompt suppression applies. - Add or change system-scoped alerts through
internal/alerts/system_alert.go. A system-scoped alert reports on Pulse itself rather than on a monitored resource, for conditions the operator cannot observe from outside the product. Notification delivery is the founding case: the channel that would carry the warning is the thing that failed, so the alert list and navigation badge are the only escalation path that does not depend on delivery working. System alerts carry the stablepulse-system-identity prefix, set noResourceIDso resource-linked affordances are skipped rather than pointed at nothing, and stampsystemAlertmetadata. Raising must stay idempotent for an unchanged condition so a timer-driven evaluator cannot become a notification storm; only a change of level or message re-notifies. New system alert types must be raised through this helper rather than by constructing bare alerts, so identity and the idempotence guarantee stay in one place.
Forbidden Paths
- New ad hoc
Check*-local evaluator logic - Reintroducing runtime legacy alert-ID contracts
- Reintroducing per-family threshold/override merge logic outside the shared path
Completion Obligations
- Update alert spec/evaluator tests when a new rule kind is added
- Update this contract if alert truth or identity rules change
- Route runtime changes through the explicit alert proof policies in
registry.json; default fallback proof routing is not allowed - Tighten or add guardrails when an old alert path is removed
- Update the event-log schema upgrade and history-projection parity proofs when lifecycle snapshots, occurrence folding, or alert-history authority changes
- When notification admission changes, verify firing, recovery, escalation, queued-escalation revalidation, and quiet-hours replay precedence together.
Attention projection source contract
Canonical alert operational records, evidence envelopes, and lifecycle
transitions are the only writable source for Patrol attention. The legacy alert
adapter must preserve an existing provider-authored recommended next step, use
the canonical incidentAction when present, and otherwise add the safe
operator instruction to open the affected resource and verify current state
before changing it. This is migration guidance, not action authority.
The attention read model may project and filter alert lifecycle state, but it must not reinterpret acknowledgement as resolution, omit suppressed state from inspectability, or convert missing/stale evidence into health.
Current State
Retired AI Discovery progress frames (2026-10-11)
The websocket store no longer handles ai_discovery_progress frames or emits
that event-bus type, because AI Discovery was removed. A server that still
sends one is ignored without touching resources or alert state;
websocket-unified.test.ts pins that. Alert detection and notification state
are unchanged.
Webhook drafts survive unconfirmed persistence
The existing Notifications editor awaits an explicit successful acknowledgement from its add/update owner before clearing a draft. The owner returns success only after the API responds and the returned destination replaces or joins the displayed inventory. A rejected or interrupted save keeps the same inputs, masked fields, template, mention and tag/severity policy. Fixed inline guidance does not claim the write was rejected: an interrupted response can follow a committed create, so the operator checks saved destinations in another tab before explicitly retrying. There is no editor-triggered automatic replay.
One pending editor save locks the draft, Cancel and the conflicting local list controls, and subsequent Save calls do not invoke another mutation. Success closes the editor; failure unlocks it for correction or deliberate cancellation. The API, encrypted persistence/publication ordering, saved-value masking, Test payload parity, delivery policy, SSRF/TLS and existing shared API-client retry rules are unchanged. This is draft recovery, not exactly-once creation or proof that ordinary notifications arrive.
WebhookConfig.persistence.test.tsx exercises the actual editor and mutation
owner through deferred create/update responses, failure, explicit retry,
canonical accepted response, cancellation and fixed callback-error guidance.
The deliberate-retry control defers its response separately: clearing the old
warning at retry start is not acknowledgement, and the draft stays locked until
the accepted response incorporates the destination and closes the editor.
WebhookConfig.test.tsx now acknowledges successful saves before expecting
editor dismissal. The local webhook-save-acknowledgement browser fixture
checks the same components with synthetic API transport, including desktop and
phone keyboard/touch journeys and another-tab reconciliation after a lost
response. It sends no real notification and establishes no installed outcome.
Active Docker update pending age survives restart
A positive Docker image-update report reuses the matching active occurrence's
StartTime as its pending-age authority. Resource and stable-identity timer maps
are process-local; restoring the active alert must not impose another 24/48-hour
delay during which positive reports fail to refresh LastSeen. Both cleanup
paths must retain the observed occurrence, acknowledgement and notification
identity without manufacturing recovery, refiring or duplicate delivery.
Only the same resource's active docker-container-update occurrence supplies
age. Another host/container and a resolved occurrence cannot lend their timers;
affirmative recovery still retires tracking and a new update waits its normal
delay. Absent/failed checks remain unknown, not recovery. An unactivated pending
timer remains process-local; this does not infer an update's first detection
from its cached registry-check timestamp.
TestDockerUpdateRestartRestoresActivePendingAge in update_alerts_test.go
exercises the public host checker, real JSON/durable checkpoint and restart,
24/48-hour delays, acknowledged/unacknowledged incidents, stale saved observation,
cached positive/absent/error reports, hourly and retention cleanup, lifecycle and
delivery callbacks, isolated hosts and affirmative recovery/new-delay controls.
The saved observation is aged deliberately; this is deterministic source-level
restart/housekeeping proof, not a naturally elapsed day, live registry or
destination result, or reporter confirmation of #2353's full daily cycle.
A "no update" report without a local digest is unknown
An image-update clear is a comparison: the agent must have known the local
digest it compared. Agents before #2353 reported UpdateAvailable: false with
an empty CurrentDigest when an image inspect failed, because an empty digest
compares as "not different". The server took that as recovery, resolving the
alert (attributed to legacy-alert-recovery-adapter, the default for clears
without explicit recovery evidence) and restarting the delay, so the alert
re-fired a full delay later with no fresh registry check in between.
checkDockerContainerImageUpdate now treats a no-update status without a
current digest like an absent or failed check: it refreshes LastSeen and
keeps the occurrence and its pending age. Only the agent produces these
statuses and it always sends the current digest when it knows it; locally
built images, which have none, never assert an update, so they lose nothing.
The "no update without a current digest" case of
TestCheckDockerContainerImageUpdate pins the guard and that a real
comparison still resolves; the retention and restart tests' clear fixtures
carry a current digest as real agents do. The agent side is in the monitoring
contract.
Continuing unacknowledged alerts survive age-based cleanup
MaxAlertAgeDays removes an unacknowledged alert only when both its occurrence
start and last observation predate the configured limit. A still-observed
condition keeps its occurrence identity, age and dispatch timestamp; cleanup
must not hide it or manufacture another firing on the next poll. Legacy alerts
without LastSeen use StartTime for inactivity. Zero disables this retention
rule. Acknowledged-alert cleanup, automatic acknowledgement, explicit clears,
confirmed recovery and notification eligibility remain unchanged.
TestCleanupRetentionRequiresObservationInactivity in alerts_test.go covers
continuing/inactive, legacy, disabled, acknowledged and inconsistent-timestamp
controls. TestCleanupContinuingDockerUpdateKeepsOccurrence in
cleanup_observation_retention_test.go exercises the real Docker update
detector with cached, absent and failed registry observations through four
cleanup/next-poll cycles, JSON and durable active-state restart, callback counts,
history and affirmative recovery. This is source-fixture lifecycle proof, not
an installed registry/destination result or a diagnosis of a reporter's daily
resolve/reopen cycle.
A settings save keeps the alert settings it did not send
PUT /api/alerts/config applies the body to the stored configuration through
Manager.ApplyConfigUpdate and ApplyAlertConfigUpdate in
internal/alerts/config/update.go: a top-level key the client sent replaces
that setting whole, a key it left out keeps the stored value, and the result
is normalized as any update is. The thresholds page has no control for
flapping detection (flappingEnabled and its window, threshold and cooldown),
alert TTL cleanup (maxAlertAgeDays, maxAcknowledgedAgeDays,
autoAcknowledgeAfterHours) or customRules, so it never sends them. The
handler used to decode the body into a zero AlertConfig, so every thresholds
save turned flapping detection and TTL cleanup off, reset the flapping tuning
to its defaults, dropped custom rules, and persisted that. The page's
flapping, aggregation and ioNormalization payload objects are not
AlertConfig fields and never enabled anything. A stored config keeps what it
holds, including values an earlier save zeroed; nothing rewrites them.
The body is decoded on its own, so sent keys mean exactly what the decoder
makes of them, duplicates and case-folded keys included. Unsent fields are
copied from a JSON round trip of the stored config, so the result shares no
maps or slices with it. ApplyConfigUpdate reads, merges and applies under
the manager lock, so two concurrent partial saves cannot revert each other in
memory, and returns a JSON-cloned snapshot so that persistence, which
normalizes the config it is handed in place, never writes into the live
config's maps.
internal/alerts/config/update_test.go covers unsent, explicit-off,
case-folded, duplicate-key, malformed and full-body updates, a sent object
replacing its stored value whole (a sent truenasDiskDefaults: {} does not
inherit the stored temperature) and the no-sharing rule.
TestUpdateConfigKeepsSettingsAThresholdsSaveDidNotSend and
TestApplyConfigUpdateKeepsConcurrentPartialUpdates in
config_validation_test.go run saves through the manager, and
TestUpdateAlertConfig_KeepsStoredValuesForUnsentKeys in
internal/api/alerting/alerts_test.go drives the handler against a real
manager.
Confirmed empty storage is recovery evidence
Static storage capacity evaluation must admit a zero usage observation when
Total > 0, Used == 0, and Free == Total together confirm an empty store.
Offline or unavailable storage remains ineligible. Zero usage without those
consistent byte counters is unknown capacity, not evidence of recovery.
The observation passes through the existing canonical capacity lifecycle,
including recovery confirmation and any predictive-capacity evidence; this
rule does not directly clear an incident or bypass notification eligibility.
TestStorageEmptyCapacityRecovery in internal/alerts/alerts_test.go pins
confirmed empty recovery and retention under absent/inconsistent counters and
offline/unavailable status. internal/monitoring/monitor_storage_webhook_test.go
pins production callback/queue delivery of the correlated zero-value recovery,
current observation time, matching history and duplicate avoidance, with no
recovery on missing capacity. This is synthetic local receiver proof, not
installed PBS API, off-host delivery or release qualification.
Alert-quality telemetry folds only canonical durable lifecycle truth
internal/alerts/telemetry_quality.go folds active alerts and the canonical
30-day history projection into identity-free aggregate counts. Severity totals,
active-age buckets, resolution-duration buckets, repeated occurrences, and
snooze outcomes are computed locally. Canonical state and occurrence time may
be used transiently for folding, but neither leaves the process. The snapshot
also exposes tenant-level configuration and event/active-state authority counts
so downstream rates have denominators and persistence failures remain visible.
The contract deliberately does not classify automatic versus operator
resolution because alert resolution currently records recovery evidence, not a
resolution actor or operator-resolve action. It does not claim maintenance
suppression effectiveness because an intent policy can suppress a candidate
before an alert occurrence exists. It also excludes detected flapping episode
counts because that diagnostic event path may drop under pressure. Flapping
configuration adoption remains measurable. telemetry_quality_test.go pins
occurrence folding, snooze sequencing, and exact bucket boundaries.
The alert webhook editor exposes the delivery contract's language-neutral
MessageKey, event, resource type and node display name alongside the existing
type, severity and metric fields. Presentation must advertise the backend-owned
template data rather than asking operators to parse the English alert message
or reach into metadata for canonical alert identity.
Native sensor temperatures alert only on their own threshold
A host agent's native lm-sensors temperature (HostSensorSummary.Native, kind
temperature) raises an alert only when its own override sets a threshold:
AlertConfig.Overrides["agent:<host ID>/sensor:<hex>"].Temperature, where
<hex> is the lower-case hex of the sensor ID's UTF-8 bytes
(HostNativeSensorResourceID), so IDs that differ only by case or
punctuation stay apart after the lower-casing host child IDs and metric names
receive. A sensor has no default and never inherits the host's, a linked
node's or a guest's CPU Temperature; a missing, disabled or zero override
raises nothing. The alert uses the canonical metric path with metric
temperature and spec resource type agent-sensor (a migration bridge type
in spec validation), so its ID is
agent:<host>/sensor:<hex>::metric-threshold:temperature, it inherits the
temperature stability delay, critical escalation and performance quiet-hours
category, and its message names the sensor's label. Fan inputs raise nothing.
The spec resource type is also the runtime classification, so
agent-sensor delay and intent rules apply on top of the agent ones.
An invalid reading, a reading without a value, a configured sensor missing
from the report, an empty inventory and telemetry expiry interrupt the run and
hold a firing alert: the inventory carries no completeness state, so absence is
not evidence of removal. The hold is not indefinite: like any alert, one with
no observation for 24 hours is auto-resolved by the stale sweep as no longer
monitored, and retention cleanup applies as usual. Removing, disabling or
zeroing the override, switching off the host's or all agents' alerting, and
removing the host resolve it and drop its pending run; cleanup walks active,
reducer-pending and intent-pending state as well as configured overrides, so
removing an override still reaches its run. A confirmed offline host clears
the alert with its other child alerts. A configuration save re-judges a firing
sensor alert by its own override only, and interrupts a pending run when the
save switched off its threshold, its host (directly or through an inherited
linked-resource override), all agents or global alerting, using the host link
identity recorded with the run; a run without that identity is interrupted.
A re-enabled threshold therefore cannot reuse earlier grace. In CheckHost,
global alerts-off returns early without clearing, as for the other host
children; telemetry expiry with alerting off clears them. Clearing all active
alerts also drops native sensor pending runs and their recorded identity, and
the stale-map sweep drops identity whose run is no longer pending. As for every
host metric, CheckHost reads policy at the start of a report (the sensor's
own threshold separately from the host gates, delay and intent policy during
evaluation), so a report in flight during a save can act on the old policy.
The next report corrects a sensor threshold or host gate; if the save turned
global alerting off, CheckHost returns early and the stale-policy alert
remains until alerting is on again or the stale sweep resolves it. Overrides are
keyed by the reporting host ID and are not migrated when a host's identity is
healed to a new ID, as for agent filesystem overrides; the threshold must be
set again. The alert belongs to the host for history; operator intent, like
the other host child alerts, still resolves under the child ID. Tests in
internal/alerts/operational_contract_test.go
(TestHostNativeSensorResourceIDIsCollisionFree,
TestHostNativeSensorThresholdRaisesOnlyFromItsOwnOverride,
TestHostNativeSensorGapsHoldAndPolicyResolves,
TestHostNativeSensorConfigSaveKeepsAlertItsOverrideStillHolds,
TestHostNativeSensorPendingRunRestartsAfterPolicyOff,
TestHostNativeSensorUsesSensorDelayPolicy) and
TestResourceAlertSpecValidateAllowsAgentSensorMigrationBridgeType pin this.
Alerts → Thresholds → Machines writes these overrides in its Machine Sensors
section (ThresholdsTableAgentSensorsSection). It lists every usable
temperature sensor a machine reports, plus every sensor with saved settings,
so a configured sensor that reads invalid, has left the report, sits on a
machine that is not reporting, or belongs to a machine no longer present stays
editable and removable with its reason beside the name; a non-reporting
machine shows no current reading. Rows are grouped by the machine's resource
ID, not its name. The frontend builds the ID with agentSensorResourceId
(frontend-modern/src/features/alerts/agentSensorIdentity.ts), the same
UTF-8 hex encoding as HostNativeSensorResourceID, and the overrides
projection reads it back as type agentSensor. The section has no Global
Defaults row and no Alerts toggle: a row's presentation default is
temperature: 0, marked presentationDefaults, so an unconfigured sensor
reads Off as the engine treats it, and the limit cell is the only switch.
Because that default is display-only, saving any Off value on a sensor row
stores no threshold and removes an override left with nothing else, and any stored sensor override,
a saved 0 included, shows as Custom with a revert action. Other rows keep an
explicit Off over an Off inherited default, so it survives a later change to
that default. The overrides projection carries each stored override's note
for every resource type, so a reloaded editor keeps it, and
useAlertOverridesState re-projects when only a note changes, so a discarded
note-only edit or a note removed on the server does not linger on the row and
get saved back (useAlertOverridesState.test.tsx). A row's delay action opens the alert intent editor on
metric.temperature (shared getAlertResourcePreferredIntentSignal), and
the editor keeps a target that is not a listed resource, such as a machine
sensor or disk, as its own option under the row's label instead of falling
back to the first listed resource; preview is unavailable for such targets.
A handoff is applied once per request: a later change to the resource list
does not replay it over the operator's own selection.
useThresholdsHostData.agentSensors.test.ts,
useThresholdsOverrideMutations.test.tsx, AlertIntentPolicyPanel.test.tsx
and ResourceTable.test.tsx pin the frontend side.
Agent custom sensors use canonical health-assessment alerts
Typed HostSensorSummary.Custom numeric, boolean, or timestamp readings with
warning or critical status map to one canonical custom-sensor alert per
host and sensor ID. Alert evidence preserves the optional group, subgroup,
kind, and event time alongside the evaluated numeric value.
Collection errors map to warning only when the local definition reports
alertOnError; healthy reports, removed definitions, disabled host thresholds,
host removal, and telemetry expiry clear the same identity. Alert evaluation
consumes the agent-authored typed status and never executes or receives the
local command or REST URL. TestHostCustomSensorAlertLifecycle and
TestHostCustomSensorErrorCanBeReportOnly in
internal/alerts/host_unraid_lifecycle_test.go pin creation, recovery,
opt-out, and cleanup.
Health-assessment warning→critical transitions dispatch the updated incident
through the normal acknowledgement, snooze, activation and flapping policy;
unchanged severity and critical→warning transitions do not dispatch. The shared
health-assessment caller opts into the configured per-incident hourly limit for
initial, refired and escalation notifications, so repeated severity oscillations
cannot bypass that budget. Incident severity still updates when delivery is
suppressed. This applies to all callers of the shared assessment adapter
(custom sensors, host storage/RAID and storage ZFS pool/device health), not other
stateful alert families. No delivery-policy bypass or separate sensor callback
is introduced.
TestHostCustomSensorEscalationDelivery in
internal/alerts/host_unraid_lifecycle_test.go pins the running-host callback,
stable incident identity and suppression/no-noise cases;
TestHealthAssessmentEscalationDelivery in
internal/alerts/canonical_stateful_test.go pins shared ZFS escalation and
hourly-budget exhaustion across warning/critical oscillation.
The alert resource-incident panel
(frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx)
dropped its "Open in Infrastructure / Workloads / Storage / Recovery"
cross-jump chip strip on 2026-05-16 when the surrounding platform-first
migration retired broad surface-link chips. The panel now keeps
investigation flow in place through the shared incident-timeline cards; it must not reintroduce a chip strip that
links to the retired top-level routes, and the supporting
buildResolvedResourceSurfaceLinks helper was deleted from
frontend-modern/src/routing/resourceLinks.ts as part of the same pass.
On desktop that panel renders inline under the history table row that opened
it (AlertHistoryTableAlertRow.tsx). On phone layouts, Resource and Timeline
investigations render through
MobileAlertHistoryInvestigationDialog.tsx as one full-height, independently
scrollable investigation drawer. They must not expand inside
AlertHistoryMobileList.tsx: the history is a fixed-estimate virtualized list,
so changing one mounted card to incident-timeline height invalidates its spacer
math and makes the app scroll shell jump or pull the operator back up the page.
The mobile drawer keeps that list height and scroll position stable, contains
overscroll within the evidence surface, supports Escape and explicit close,
and returns focus to the originating action or the history list fallback.
It must not become a page-level sibling in tabs/HistoryTab.tsx, which would
again open away from a reader deep in history (#1687). Because several alerts
can share one resource, resourceIncidentPanel still carries the originating
rowKey, and the mobile investigation state allows exactly one Timeline or
Resource surface at a time. Resource incident expansion and filtering remain
reactive accessors inside each keyed incident row; snapshotting the expanded
set or filter set would let state update while the visible Events timeline
stays closed or stale.
The alert history filter bar carries no saved-views affordance.
AlertHistoryFiltersCard.tsx must not pass a savedViewsKey to the shared
FilterBar; that prop and the localStorage view library behind it were
removed because a saved view was only ever the page's URL query string, which
the browser's own bookmarks already capture and share. History narrowing stays
URL-owned so a filtered history page remains a shareable, bookmarkable link.
The alert history severity facet derives each option count through
useAlertHistoryState.countForSeverity, using the same
filterAlertHistoryItems predicate that supplies the rendered list for the
current fetched period and search term. It must not count an unfiltered or
separately reduced collection that can disagree with the selected chip's
result. These counts follow the shared Inventory totals visibility preference;
the Period facet remains uncounted because it selects the fetched time scope
rather than filtering the already-fetched rows.
Alert severity is one closed, end-to-end vocabulary: info, warning, and
critical. Alert creation, restored active state, history projection,
incidents, operational-trust projections, API responses, sorting, resource
highlighting, overview cards, badges, and history filters must preserve those
three meanings. The legacy history value error normalizes to critical;
empty or unknown runtime values fail safe to warning and must never become
low-urgency information accidentally. Informational alerts use the shared blue
presentation and sort below warnings, while the history facet exposes a real
Info option whose count and filtered rows use the same predicate as every
other severity.
The alert history frequency chart in AlertHistoryFrequencyCard.tsx stacks
each period by severity, critical at the base, from the severityBuckets that
buildAlertTrends counts in the same pass as its totals. Those counts use
alertSeverityCountKey, the closed vocabulary above: error is critical and
an empty or unknown value is a warning, never information. Segment fills and
legend dots share getAlertSeverityFillClass, each at least 3:1 against the
chart surface in both themes (light-theme warnings use amber-700). A bar's
height is the square root of its share of the busiest period (4px floor, 40px
ceiling), so a burst stands well clear while ordinary periods stay tall
enough to read their mix; each bar's tooltip and accessible name carry the
exact counts. The legend counts critical, warnings and info (info only when
present) over the current window and bar selection
(AlertHistoryFrequencyCard.test.tsx, alertHistoryModel.test.ts).
The chart's axis names round local times. Buckets start at the window start
(now minus the range), so labelling bucket edges printed times such as
11:50, and an hour without minutes reads as a bare number in 24-hour
locales (8 Oct, 11). formatAlertAxisTickLabel writes an hour with its
minutes, and buildAlertAxisTicks moves each inner tick to
nextAlertAxisBoundary, the next whole bucket step within the day (or the
next local midnight for date-only labels), at that time's true position. The
first tick keeps the exact window start the range line shows, the last reads
Now, and the buckets, bar tooltips and counts are unchanged
(alertHistoryModel.test.ts).
The desktop alert history table sizes each column for its content instead of
giving all visible columns an equal share. Each density is sized for the
narrowest card it serves (600px, 800px, 1024px and 1280px): Time (the column
header; the cell is the row's clock time, 08:27 AM in a 12-hour locale),
severity, duration, status ("acknowledged" needs 119px with its cell padding),
the Resource header, the type badge and node (from 1024px) and the three row
actions on one line (177px) get the pixels their content and header need at
that width, and Message, the column that says what happened, takes the rest
and never falls under 18% of the card; App.architecture.test.ts holds those
minimums and the 50rem, 64rem and 80rem switch points. From an 800px card the
three actions share one line; below it they stack, which makes those rows
taller, and rows there still differ by how many action buttons they stack.
Table rows show Timeline and Resource as 24px icon buttons beside the
Investigate icon (accessible names and titles say what each opens; cards keep
the worded buttons), so the three actions need 100px instead of 177px, and
Status uses a compact 11px badge that needs 108px for "moved to agent"
instead of 119px. Resource and Message take that width.
An alert may carry Summary, its message without the subject the row's
Resource cell already names ("Running 1 of 3 desired tasks" for "Docker
service 'web' is running 1 of 3 desired tasks", "Offline", "Powered off").
Builders set it beside Message where they name their own resource: Docker
service gap and update state, Docker host offline, Docker container state,
health, restart loop, OOM kill, memory limit and image update, node, PBS and
PMG offline, guest powered off, Pulse agent stopped reporting, storage
unavailable, connection state and external probe silence. Both canonical
lifecycle evaluators copy it into the alert they build, so it changes with
the message; a MessageBuilder clears it because a built message has no
matching summary. History rows lead with Summary and keep the full message
in their title; cards, notifications and detail views keep the message, and
history recorded before alerts carried a summary shows its message.
Below a 50rem card, Resource and Message cells wrap inside a line-clamp-2
wrapper (data-alert-history-cell) instead of ending in an ellipsis after one
line, so more of what happened reads without hover. A message or name longer
than its two lines still ends in an ellipsis with the full text on its title;
the table does not claim to expose every value without hover. The wrapper sets
whitespace-normal itself because .table-fixed td forces nowrap. Message
also keeps a two-line minimum height (min-h-[2lh]) so a message's length
never changes its row's height: the shared windowing hook sizes its spacers
from a single measured row. Scrolling a 600-row history in 60px wheel steps
(414 tracked comparisons) moved a tracked row by exactly the scroll distance
with uniform 49px rows, while the same history with the minimum removed (rows
of 39px and 49px) showed 46 steps where the row jumped by up to 20px. From a
50rem card (a desktop window of about 1250px beside the sidebar) both cells
return to a one-line clamp (@min-[50rem]:line-clamp-1, minimum height
released) and rows are 33px like the other tables; the earlier 80rem switch
left 1280-1440px windows with 49px two-line rows.
Measured on the mock estate in a real browser (the estate regenerates on
every run, so the row count differs: 88 before, 90 after), messages that ended
in an ellipsis fell from 82 of 88 rows to 24 of 90 at a 1024px window with the
rail open (a 600px card), from 54 of 88 to 17 of 90 at 1280px, from 24 to 8 at
1440px and from 32 to 4 at 1700px, and the median row from 75px to 49px at
every window from 1280px to 1700px; at 2000px, where the cells keep a
one-line clamp as before, 9 of 88 became 7 of 90. No time cell or header
ended in an ellipsis at nine windows from 1024px to 2000px in a 12-hour
locale. Cards narrower than 600px, which a window of about 900px with
the rail open produces, are still cramped; a container-driven switch to the
card list is the remaining fix.
The history card's own width now chooses between the table and the cards,
not the viewport. AlertHistoryTableSection measures its wrapper with
useObservedElementWidth and shows the cards whenever the card is narrower
than ALERT_HISTORY_TABLE_MIN_WIDTH (800px, the width from which the three
row actions share one line); until the card is measured, the phone
breakpoint decides as before. A 1024px to roughly 1240px window with the
Alerts rail open, which gave the table a 600px to 800px card with stacked
actions and names broken mid-word, now shows the cards; a 1280px window
(an 837px card) keeps the table. The 600px density and the stacked-action
rule above therefore serve only a card that has not been measured. The cards
take active from that decision instead of a 768px media query, so an open
timeline or resource view moves into the cards' drawer when the cards take
over and back to the inline expansion when the table returns
(AlertHistoryTableSection.test.tsx, AlertHistoryMobileList.test.tsx).
/alerts/history?alert=<id> (buildAlertHistoryAlertPath) opens History
narrowed to one alert's occurrences, the landing a notification can link to.
useAlertHistoryState filters to that id before severity and search, shows
every loaded occurrence regardless of the range window, and decides once per
id, only after saved history has loaded and the live alert list is ready
(saved history holds resolved alerts only): if nothing matches outside "All
time" it widens to All time once, so a later range is the operator's;
focusedAlertMissing reports an id history no longer holds. It opens the
latest occurrence (still firing first) through the shared timeline expansion
and hands the cards a one-shot drawer request a remounted list cannot replay.
The tab names the alert in a status notice with "Show all alerts", and while
the live list is unavailable it says the alert may still be firing and
offers Retry instead of a verdict (useAlertHistoryState.test.tsx,
HistoryTab.test.tsx, alertOverviewPresentation.test.ts).
Alert history row timestamps render clock time in the viewer's own locale and
must carry the absolute date and time as a title. The date otherwise lives
only in the day group header, which scrolls out of sight, and a hardcoded
en-US format misreported the time of day to everyone outside the US
(#1687, #1685). useAlertHistoryState owns both formatters
(formatAlertRowTime, formatAlertRowTimestamp) so the desktop table and the
mobile list cannot drift apart. The day group fullLabel built in
frontend-modern/src/features/alerts/alertHistoryModel.ts follows the same
rule, because a header reading "Thursday, August 6, 2026" to a reader whose
rows are ordered day-month is the same defect one level up. See the
localization boundary in the frontend-primitives contract for the shared
formatting rule and the audit that enforces it.
Alert browser surfaces no longer manage their own runtime-capabilities fetch or
hasAIAlertsFeature prop chain. frontend-modern/src/pages/Alerts.tsx and the
shared alert overview surfaces (OverviewTab.tsx, HistoryTab.tsx,
AlertOverviewActiveAlertsSection.tsx, AlertHistoryTableSection.tsx,
AlertHistoryTableAlertRow.tsx, AlertOverviewAlertCard.tsx) must source AI
alert feature gating from the shared entitlements layer, not from a per-surface
loadRuntimeCapabilities fetch. Alert surfaces must not re-introduce their own
hasAIAlertsFeature, runtimeCapabilitiesLoading, or direct
/api/license/runtime-capabilities reads.
Canonical alert identity and evaluation are the live runtime model. Remaining
legacy references should exist only in explicit migration or negative test
boundaries.
TrueNAS per-resource threshold overrides use the unified resource's current
canonical ID as their persistence and evaluation key. API-backed TrueNAS
systems are keyed by configured connection, never reported hostname or DMI
serial, so repolls, result reordering, missing serials, DR clones, and multiple
same-hostname appliances cannot transfer or strand an override. The browser
may read provider-declared superseded IDs and metric-target IDs while
projecting a legacy alerts.json, but an edit must remove those candidates
and persist exactly one row under the current canonical ID. While an input is
active, the thresholds projector must retain the edited row across WebSocket
resource refreshes; blur commits that retained row into the unsaved
configuration, and the global configuration save remains the only API
persistence boundary.
The monitoring bridge may migrate only provider-declared, unambiguous
canonical-ID successions before evaluation. It must persist the migrated
configuration before installing it in the alert manager, let a current-key
override win while removing its retired duplicate, and retain unknown or
temporarily absent override rows until a provider proves succession. The same
persisted override must drive trigger, clear, derived critical severity, and
notification dispatch before and after restart. Regression ownership is
frontend-modern/src/features/alerts/thresholds/hooks/__tests__/truenasThresholdPersistence.test.tsx,
frontend-modern/src/features/alerts/__tests__/useAlertsConfigurationState.test.tsx,
internal/alerts/canonical_override_migration_test.go, and
internal/monitoring/monitor_alert_override_migration_test.go.
All v6 platform alert targets must enter runtime threshold evaluation through
UnifiedResourceInput and internal/alerts/unified_eval.go: Proxmox guests/
nodes/storage, Docker hosts/containers/services, Kubernetes clusters/nodes/
namespaces/deployments/pods, TrueNAS systems/pools/datasets/disks, VMware
vSphere hosts/VMs/datastores/networks, PBS, PMG, and standalone host agents.
Adding a platform-local evaluator branch for these resource families is
forbidden.
Per-platform defaults, per-resource overrides, global disables, active-alert
reevaluation, history persistence, and notification delivery must use the same
alert configuration shape rather than a platform-specific sidecar.
Notification cadence is part of that runtime contract. Schedule.Cooldown and
Schedule.MaxAlertsHour apply to already-active alert re-notifications as well
as first-fire creation, including canonical metric alerts, legacy metric paths,
and severity-change re-notifications. Accepted alert dispatch must record
LastNotified back onto the live active-alert state before persistence, even
when a restored or replayed alert is dispatched through a clone, so reloads do
not reopen the same alert's notification window.
Recently resolved alerts are an operator-facing transition window, not an
unbounded history store. recentlyResolved must prune expired entries and cap
the newest retained entries on insert as well as during cleanup, so monitor
sync and websocket state snapshots remain bounded; durable resolved-alert
history belongs in the alert history store, not in this live transition cache.
recentlyResolved and resolvedAlias have one lock owner:
Manager.resolvedMutex. Every access holds that mutex, including alias-repair
lookups, which require the write lock. When an operation needs both manager
state and resolved state, the only permitted nested order is
Manager.mu then Manager.resolvedMutex; no path may acquire Manager.mu
while holding Manager.resolvedMutex. Resolved critical sections are limited
to map access and must not dispatch, persist history, invoke callbacks, or
perform notification work. Canonical lifecycle and stateful cooldown refires
consume resolved state through the shared lock-order-aware helper, preserve the
original alert StartTime, and keep the five-minute refire/history semantics.
The reducer core owns transitions for the canonical stateful family: health
assessments, posture thresholds, and change thresholds, including ZFS pool
state, backup and snapshot age, and Docker update-delay state. Legacy
pending-since maps are read-only mirrors during migration and must not decide
activation. Stateful occurrences retain their established timestamp boundary:
the current observation, or a caller-supplied override such as the backup
timestamp, rather than the pending-run start. A manual clear must pass its
clear time into the reducer, record the firing incident as recently resolved,
and start acknowledgement retention so a refire inside five minutes restores
the same occurrence without duplicating history. Regression ownership is
internal/alerts/canonical_stateful_test.go.
The browser thresholds surface is also platform-shaped: Proxmox, Docker,
Kubernetes, TrueNAS, vSphere, PBS, PMG, and Systems. Route-backed platform
choices must use the shared Subtabs navigation above the shared FilterBar;
they are not resource-filter facets and must remain visible when the mobile
filter shell is collapsed. Resource filtering must use the shared FilterBar
chip and "+ Filter" pattern, and alert tables must use the canonical platform
table column-kind alignment helpers from
frontend-modern/src/features/platformPage/ rather than hard-coded table
alignment classes.
Alert filter option semantics stay alert-owned, but FilterBar chip
presentation is frontend-primitives-owned: severity filter leading dots must
use filterChipStatusDot rather than alert-local span factories.
Alert notification and timeline form textareas are also shared-primitive
consumers. Email recipient lists, Apprise target lists, webhook payload
templates, threshold ignored-prefix input, and incident timeline notes must
compose FormTextarea for label/id/help wiring and textarea chrome instead of
recreating raw labelled <textarea> shells locally. Alert resource row/mobile
note editors now follow the same primitive contract.
The Recovery and Snapshot Age threshold sections expose exactly one Global
Defaults surface: the always-live editor row (table layout) or card (card
layout) that the shared resource table renders when global defaults are
supplied. They must not add a synthetic read-only resource row mirroring the
same record; a second surface invites edits that no override mutation path
persists, which is how #1680's silently-dropped edits happened. Snapshot size
thresholds belong to the Snapshot Age section, not Recovery, because
BackupAlertConfig has no size dimension. The recovery-defaults records carry
the normalized warningSizeGiB and criticalSizeGiB metric keys that the
column editor resolves for the size columns, and the configuration snapshot
layer must round-trip those fields: applyAlertsConfigToSnapshot and
buildAlertsConfigurationPayload carry them between the UI snapshot and
/api/alerts/config rather than stripping them. Guests in the VMs &
Containers section order by display name (numeric-aware, case-insensitive)
with vmid as tiebreaker, because the rows render the name and an invisible
sort key reads as an unsorted list. Regression ownership is
frontend-modern/src/utils/__tests__/metricThresholds.test.ts and
frontend-modern/src/features/alerts/thresholds/hooks/__tests__/useThresholdsRecoveryDefaultsState.test.tsx.
Guest metric canonical state remains resource-backed and therefore node-scoped
for Proxmox guests, so node moves must not strand active alert state on the
previous resource ID. When a guest metric alert survives a node move, alerts
runtime must migrate the active alert, history entry, acknowledgment record,
suppression/rate-limit/flapping tracking, and guest per-disk metric identity
to the current canonical state instead of reopening a duplicate alert or
resolving only the stale node-scoped identity.
Metric alerts stored under a canonical state key must also clear through that
same identity. Hysteresis resolution must derive the active-alert storage key
from the existing alert and canonical state ID; removing only the legacy
<resourceID>-<metric> ID can emit a resolved notification while leaving the
canonical alert active to resolve again on every poll.
internal/alerts/reducer is the deterministic lifecycle core. It accepts an
explicit observation time and models enablement, hysteresis, sustained delay,
intent gating, acknowledgement, and warning/critical severity without
persistence or notification side effects. The canonical lifecycle family and
the shared metric-threshold runtime now use reducer incidents as transition
truth; the alert manager projects those incidents into active alerts, history,
persistence, callbacks, and notification policy. Metric incidents are keyed by
the canonical metric spec ID, while the metric name remains the severity
classification input. Families not yet cut over continue to use the parity and
shadow harnesses; a divergence is a reducer bug unless a focused manager
regression first proves otherwise.
The confirmation/discrete-state slice uses the same boundary for connectivity, powered-state, and generic discrete matches. It activates after the resolved number of consecutive matches, clears pending state or firing state on a non-match at the evaluator layer, and re-derives spec-carried severity on every firing observation. Both manager and reducer must date first activation at the first matched observation, not the final confirming poll. Alerts runtime owns that first-match timestamp alongside the confirmation counter and must clear it whenever the run resets or its tracking state is removed. A direct reset of the confirmation counter also begins a new run: the next matched observation must replace any stale first-match timestamp so a later alert cannot be backdated to an earlier outage.
Poll-driven offline recovery composes an additional confirmation gate over that evaluator-layer lifecycle. A firing incident resolves only after consecutive healthy observations (three for nodes, PBS, and PMG; two for storage), and an offline observation resets the healthy run. Pending incidents still clear on the first healthy observation, while disabling the rule bypasses recovery confirmation and resolves immediately. The reducer's recovery state and the manager composition must remain step-for-step equivalent under the parity harness.
A discrete incident that reactivates within the five-minute recently-resolved
retention is the same occurrence: it restores the original first-activation
time and emits the reducer's EventRefired, matching the manager path that
reactivates without adding a second history occurrence. The resolved record is
consumed by reactivation; after the retention window, activation starts a new
occurrence at the new matched observation. The reducer measures retention on
the explicit observation clock. The manager's temporary wall-clock comparison
remains reference behavior only until cutover and is covered by a parity
harness that aligns the two clocks. Additional alert families and downstream
delivery behavior require separate contract slices.
Acknowledgement is canonical incident state across reducer families. It may be set only on a firing incident, remains attached through per-observation alert rebuilds, and is removed explicitly by unacknowledge. Resolution makes the acknowledgement record inactive rather than deleting it, so a reactivation within one hour restores the original user and acknowledgement time; after that inactive retention, reactivation is unacknowledged. The reducer enforces this boundary on its observation clock, matching the manager's one-hour cleanup TTL, and parity must cover acknowledgement, unacknowledgement, rebuild, short resolve/re-fire restoration, and expiry for both metric and discrete families.
The intent-policy gate composes at activation time with both discrete
confirmation and shared metric thresholds. An explicit policy grace period or
resolved operator suppression keeps a matched condition pending while
preserving its first active observation and, for discrete state, clamping its
confirmation count at the configured requirement. For metrics, the explicit
metric.<name> policy replaces the legacy time-threshold delay and accrues
grace only from monotonic runtime ticks; a below-trigger observation clears the
pending run and resets grace. Grace and operator suppression accrue
concurrently; once both release, activation uses the preserved first
observation as the incident start. Maintenance, muted or retired monitoring,
and expected-offline state may hold a new activation when the resolved policy
honors operator state, but must not clear or suppress an incident that is
already firing. The reducer applies caller-resolved intent on its observation
clock, and parity must exercise the manager's real intent-policy and
maintenance-window composition. Backup-offline deferral remains outside this
characterized slice.
That same guest-threshold owner also governs guest-derived lifecycle and
posture alerts. Snapshot age, backup age, powered-off state, and
configuration-change reevaluation must all construct a canonical lightweight
guest snapshot and route threshold resolution through the shared
guest-defaults → filter-driven custom rules → guest-override → pulse-relaxed
chain.
That canonical guest context must preserve the live guest name and tags for
snapshot and backup posture evaluation. Ignored prefixes, pulse-no-alerts,
configured ignored tags, and required-tag filtering must resolve through the
same guest alert policy before any guest-derived alert is created; posture
pollers may not downgrade that context to a name-only lookup that bypasses the
operator's suppression policy.
Passing nil guest context or resolving only overrides/defaults is forbidden
because it silently bypasses custom guest rules and makes guest lifecycle
alerting diverge from running-guest metric truth.
The pulse-relaxed step lives in effectiveAlertPolicyNoLock, not in
CheckGuest, so polling and config-save reevaluation judge a relaxed guest's
alert by one threshold. Reevaluation rebuilds the guest from alert metadata,
so guestSnapshotFromAlert reads back the tags the evaluator recorded
([]string live, []interface{} after a restart). Without them a save judged
the alert by unrelaxed thresholds, resolved one raised under an unset
threshold's relaxed floor with a recovery notification, and the next poll
raised it again. Those tags must stay current: metricOptionsWithTags drops
the key from an open alert once its resource has no tags, or a removed
pulse-relaxed would keep judging the alert by the relaxed trigger. The tag
lifts CPU, memory and disk triggers to a 95/92/95 floor and gives an unset
threshold that floor, but an Off threshold (trigger <= 0) stays Off: relaxing
a guest must never raise an alert its config turned off. A filesystem's own
override still replaces the relaxed guest disk threshold for that filesystem,
on save as while polling. Proof:
TestConfigSaveJudgesRelaxedGuestAlertByRelaxedThresholds,
TestConfigSaveKeepsRelaxedGuestFilesystemOverride,
TestPulseRelaxedKeepsOffGuestThresholdOff and the Off case of
TestApplyRelaxedGuestThresholds in internal/alerts/alerts_test.go, and
TestCheckGuestDropsRemovedTagsFromOpenAlerts in
internal/alerts/unified_eval_test.go.
That same guest-alert owner also has to retire per-disk guest alerts when the
guest stops, disk alerting is disabled, or the reported disk set changes.
Canonical guest disk identity is only valid while the guest still exposes that
disk resource under the current thresholds, so runtime cleanup must remove
stale guestID-disk-* state instead of leaving orphaned per-disk incidents in
active alerts, resolved history, or later UI projections.
That same alerts runtime also owns instance-scoped node display-name
resolution. Raw node names are not globally unique across configured
infrastructure instances, so cached node display names must key on instance +
immutable node identity whenever the alert carries instance context. Current
and prior native Proxmox node names may resolve that same entry so an override
or native rename updates active alerts, resolved alerts, and history
presentation without changing the stored diagnostic native resource name or
alert identity. Read-model presentation must snapshot resolved state before
acquiring the active-alert lock to preserve the established lock hierarchy.
Alert updates, incident rebuilds, and guest-metric migrations may fall back to
the legacy name-only cache only for instance-less resources like standalone
host agents. Display names never become acknowledgement, suppression,
rate-limit, flapping, escalation, or notification routing identity.
That same host-alert boundary also owns vendor-managed NAS RAID suppression as
an alert-lifecycle concern. Shared storagehealth rules decide which Synology
or QNAP md arrays are vendor-managed system volumes rather than customer-facing
storage, and alerts runtime must use those shared rules both to suppress new
RAID incidents and to clear stale suppressed alert IDs even after monitoring
has already normalized those arrays out of canonical host state.
Storage alert runtime also owns operator-facing resource labels for storage
incidents. ZFS device alert labels must preserve raw device names such as
/dev/sda4, but must not join pool and device labels with a raw slash because
device paths can already begin with /; browser alert surfaces consume the
runtime resourceName as authored rather than patching storage labels locally.
ZFS pool and device alerts follow the storage's pool attachment lifecycle. A
storage checked without an attached ZFS pool must shed any previously raised
zfs-pool-state, zfs-pool-errors, and zfs-device alerts on that check rather
than waiting for the stale-alert cleanup, because the health path can only
clear its own alerts while the attachment exists (#1731).
Ceph pool storage threshold resolution is also source-alias aware. Storage
alerts must evaluate the normalized pool storage id while accepting legacy
agent:<host>-ceph-pool-<name> override keys as aliases, so operators do not
lose saved thresholds when the same physical Ceph pool moves between
host-agent-only fallback and Proxmox API canonical discovery.
Active alert reevaluation after threshold or config changes must use canonical
resource type metadata before the legacy node fallback. Host-agent Ceph pool
alerts may carry no-colon resource ids with Instance == Node, but when
metadata or resource type says storage they must keep using storage threshold
resolution and source-alias overrides instead of node defaults.
That reevaluation boundary distinguishes metric-backed alert types from
provider-owned incidents. A nil metric threshold may resolve only a known
threshold-backed metric; it is not recovery evidence for resource incidents,
and must not erase their acknowledgement state during a schedule-only save.
Browser metric severity colors are also alert-backed. Workloads,
Infrastructure, Storage, and the platform-page tables (Docker hosts and
containers, Proxmox nodes, Kubernetes clusters and nodes, TrueNAS systems and
apps, vSphere hosts) may pass resolved display thresholds into their local
bars, but threshold selection must flow through the shared alert activation
store and frontend-modern/src/utils/metricThresholds.ts, including configured
hysteresis, disabled thresholds, storage usage defaults, and guest/Docker
override identity candidates. The display resolver mirrors the runtime's
per-platform default scopes — guest, node, pbs, agent, docker, storage, plus
kubernetes, truenas, and vmware backed by kubernetesDefaults,
truenasDefaults, and vmwareDefaults — and unified platform resources
resolve overrides through the shared
unifiedPlatformOverrideIdCandidates chain in
frontend-modern/src/features/alerts/alertOverridesModel.ts (the same
candidate set buildProjectedOverrides indexes platform alert rows under).
Static metric-color defaults are only fallback presentation behavior for
callers that do not have alert configuration in scope.
Guest bars follow pulse-relaxed the same way. resolveMetricDisplayThresholds
takes the tags the engine reads for a guest-scope resource and, when they
include pulse-relaxed, applies the applyRelaxedGuestThresholds step to
CPU, memory and disk before mapping to display thresholds: an unset threshold
gets the 95/92/95 floor with a clear five below it, a lower trigger is raised
to the floor and keeps its own clear point (a missing one filled five below,
as ensureHysteresisThreshold does) unless the raised trigger reaches it, and
an Off threshold stays Off. A relaxed guest's bar therefore turns critical
where its alert fires; 88% memory reads warning, not critical, under the
relaxed 92% trigger. The display resolver does not read filter-driven custom
rules, which the engine folds in before the override, so a guest a custom
rule matches can still colour by thresholds the engine does not use, relaxed
or not. Only Proxmox VMs and LXCs carry their tags into alert
policy, so getWorkloadAlertPolicyTags in
frontend-modern/src/components/Workloads/workloadTopology.ts passes tags for
those guests alone, and a vSphere VM or a container keeps its thresholds
whatever it is tagged. The Alerts thresholds row for a relaxed Proxmox guest
keeps showing the configured values its editor changes and adds one line
naming the tag and the thresholds it raises
(getAlertResourceRelaxedTriggerSummary in
frontend-modern/src/components/Alerts/alertResourceTableModel.ts, fed by
pulseRelaxed from useThresholdsGuestData); before it, the tag overrode
those settings with no visible sign in the UI (issue #863). Proof: the
pulse-relaxed cases in frontend-modern/src/utils/__tests__/metricThresholds.test.ts,
the alert policy tag cases in
frontend-modern/src/components/Workloads/__tests__/workloadTopology.test.ts,
and the pulse-relaxed guest cases in
frontend-modern/src/components/Alerts/ResourceTable.test.tsx.
Docker container image-update alerts are lifecycle-governed by the alerts runtime. Disabling Docker update alerts globally, disabling alerts for a specific Docker container, ignoring a Docker container prefix, or disabling all Docker container alerts must clear the active image-update alert plus resource and identity first-seen tracking. Generic threshold reevaluation must not keep or resurrect image-update alerts after their owning Docker alert configuration has disabled them.
Docker container alert overrides key on stable container identity. The
canonical override key is docker:{hostID}/{containerName}; container IDs
change on every recreate, so runtime-ID keys orphaned each image update and
accumulated dead entries in alerts.json (#1601). The evaluator resolves the
name key first and falls back to the legacy docker:{hostID}/{containerID}
key for pre-migration entries, MigrateDockerContainerOverrideKeys
(internal/alerts/docker_override_migration.go) re-homes live legacy and
unified-hash keys onto the name key on the monitor sync cadence and prunes
orphaned ID-shaped entries while keeping name-keyed overrides for absent
containers, and the UI writes only the name key while binding rows through the
shared dockerContainerOverrideIdCandidates chain (name first; container-ID,
short-ID, unified-hash, and slash-tail forms as trailing lookup candidates).
Container override work must not reintroduce a runtime-container-ID
persistence key.
Docker alert references come only from the builders in
internal/alerts/docker.go: a host's own alert docker:<host ID>
(DockerHostResourceID, used only by its connectivity alert), a container
docker:<host ID>/<container ID> (DockerContainerResourceID), and a Swarm
service docker:<host ID>/service/<service ID> (DockerServiceResourceID).
A container or service reported without an ID alerts under its name, marked in
the ID position (docker:<host ID>/name:<name>,
docker:<host ID>/service/name:<normalized name>). Docker and Swarm IDs never
contain a colon, so a name reference never takes its host's reference or
another service's ID reference, and one with neither an ID nor a name raises
no alert. Report cleanup, host removal and the host's offline alert firing
clear children by the docker:<host ID>/ prefix; the container and service
policy switches classify by the first path segment after the host
(dockerAlertResourcePath), so a container name containing /service/ stays
a container; and Patrol's scope aliases and resource history parse the same
forms. Service overrides key on the same reference, so an override written for
a service without an ID under its old unmarked name no longer applies. A
reference can contain the canonical state separator (a Docker host
disambiguated as <base>::<suffix>), so splitCanonicalStateID splits a
state ID before the alert's recorded spec ID, or else after its resource ID,
and only without either at the first ::. Splitting at the first :: gave
such a host's alerts the reference of the host named <base>; an alert saved
that way recovers its reference on restore because its spec ID is intact.
Before name references existed, a container without an ID alerted under its
host's reference. Only the connectivity alert belongs there, so the host's
next report, or its offline alert firing, clears any such alert
left active whose recorded host matches and whose container ID is empty. A
pending image update recorded there keeps its age under the container's new
reference when that first report still shows the update pending; a first
report with missing or failed update evidence clears it, and the delay starts
again on the next positive report. Proof:
TestDockerAlertsWithoutIDsUseTheirOwnReferences,
TestDockerHostIDWithSeparatorKeepsItsAlertReferences,
TestDockerAlertsRestoredWithCutShortReferencesRecover,
TestDockerContainerWithoutIDKeepsPendingUpdateAgeAcrossUpgrade and
TestDockerContainerResourceID in internal/alerts/alerts_test.go, and
TestOwnerAlertTimelinesUseCanonicalHistoryIdentity in
internal/monitoring/monitor_alert_handling_test.go.
Backup orphan evaluation is also inventory-scoped. The alerts runtime may evaluate recovery rollups for backup age, but unresolved Proxmox PVE backup subjects must not be treated as orphaned until monitoring has supplied the matching per-instance guest-type inventory readiness signal. Known Proxmox template subjects are valid backup subjects, not orphaned workload backups, even though templates remain excluded from normal runtime workload resources.
Alert history persistence is also part of that canonical boundary. The history
manager may choose the owned runtime data directory, but it must normalize that
directory once and then resolve only the fixed alert-history.json and
alert-history.backup.json leaves through the shared storage-path helper
before any filesystem read, write, rename, or delete. Future history-persistence
changes must not reintroduce raw filepath.Join(dataDir, ...) joins from
caller-supplied directories or ad hoc history filenames.
Agentless availability incidents now enter alerts through the same unified
resource incident bridge as storage, PBS, VM, and host resource incidents.
Standalone network-endpoint resources and any canonical resource carrying an
attached availability facet must create canonical resource-incident alerts
with provider display Availability; availability alerting must not introduce
a second endpoint-only evaluator or alert identity family outside
internal/alerts/unified_incidents.go. When a resource carries multiple
checks, the incident NativeID selects the exact check evidence envelope that
is copied into the alert and its OperationalRecord; the singular
compatibility summary must never substitute evidence from a different target.
An incident with no backing availability check still carries first-class
evidence: the bridge derives a complete/confirmed envelope from the directly
observed incident payload (provider and collector from the incident, digest
payload ref) instead of leaving the alert evidence-less for the legacy partial
shim to backfill. Because each sync re-observes the incident, the bridge merges
the cycle's freshly observed envelopes into the already-active alert, so
raise-time evidence (including availability envelopes with validity windows)
does not age out while the condition is still directly observed.
The same lifecycle transition then projects into Patrol like every other
canonical operational record.
Notification transport, provider delivery, queue safety, and notification API
transport now live under the explicit notifications subsystem inside the
current architecture lane. The alerts surface still owns operator-facing alert
pages and routing UX, but it does not implicitly own the delivery engine.
That includes the webhook settings editor: alert UI may present provider setup,
but canonical service-field ownership such as Pushover token / user
normalization belongs to internal/notifications/ and persistence boundaries,
not to alert-surface runtime delivery code.
The alert webhook editor now mirrors that canonical Pushover field rule through
frontend-modern/src/utils/alertWebhookPresentation.ts, so the UI shares the
same alias, preset, and custom-field input mapping instead of carrying its own
local webhook-field normalization fork.
The destination editor also exposes resource-tag routing for email and each
alert webhook. Alert producers must carry canonical resource tags into alert
metadata: Proxmox VM/container tags remain literal, Docker container/service
labels become key:value tags (or key for empty values), host-agent tags
remain literal, and unified resources forward their canonical Tags. The
alerts surface owns this producer metadata and routing UX; destination
matching, persistence, and receipt-aware delivery remain notifications-owned.
The same editor exposes a minimum-severity policy for email, Apprise, each
webhook, and Relay mobile push. The alerts surface owns the coherent routing UX;
notifications owns provider filtering and recovery receipts, while Relay owns
privacy-safe mobile projection and persisted mobile routing policy.
Email and Apprise expose all three destination choices: all alerts, warnings
and critical alerts, or critical alerts only. Relay deliberately exposes its
smaller persisted policy of all alerts or critical alerts only; hiding the
warning option there is protocol honesty, not permission to collapse a stored
warning floor on notification-owned destinations.
The alert manager callback layer now also has to stay fan-out-safe. Monitor
delivery, the unified alert bridge, and Patrol-adjacent AI listeners must
compose through additive fired/resolved subscriptions instead of overwriting a
single callback slot, and alert-triggered Patrol enqueueing must stay on the
canonical unified alert bridge plus trigger-manager path rather than reviving
duplicate callback-side Patrol shortcuts.
That runtime callback boundary is now factored into
internal/alerts/callbacks.go as same-package Manager support code so it can
own callback mutexes, subscription IDs, legacy Set* slots, and fan-out snapshot
helpers without creating an import cycle around the Alert runtime type.
Alert configuration types, pure normalization, and resource-type identity
helpers now live under internal/alerts/config/; the parent alerts package
may re-export aliases and wrappers for compatibility, but consumer packages
must keep importing internal/alerts unless they are explicitly taking
ownership of alert configuration internals.
Alert configuration runtime now lives in internal/alerts/config_runtime.go.
That file owns UpdateConfig normalization and activation-state migration,
global disable cleanup, active alert reevaluation after threshold changes,
threshold override cloning and merge behavior, and hysteresis defaults; future
config-driven runtime behavior should extend that owner rather than expanding
the central Manager file.
Active-alert persistence now lives in internal/alerts/active_persistence.go.
That file owns active-alert save/load, secure active-alert storage leaves,
startup restoration and legacy active-alert ID migration, and periodic active
alert persistence; persistence changes should extend that owner rather than
adding save/load logic back to the central Manager file.
Persistent managers establish the SQLite alert store during construction,
before escalation, periodic persistence, shadow evaluation, or monitoring can
act on restored state. alert_active_state is the restart authority and is
independent of event retention, so a still-active occurrence never disappears
because it predates the 90-day history window. Fired, refired, acknowledged,
unacknowledged, escalated, and resolved lifecycle transitions update that
projection in the same transaction as their immutable lifecycle event. A
resolution matches alert identity plus occurrence start time so a delayed old
resolution cannot delete a newer occurrence that reused the canonical ID.
Full checkpoints capture mutable fields such as LastSeen, LastNotified, and
escalation state with revision compare-and-swap; a checkpoint that raced a
lifecycle transition retries instead of overwriting it. SQLite uses full
synchronous durability for this authority, and the recovery mirror fsyncs its
temporary file plus a platform-native durable rename barrier (parent-directory
sync on Unix and write-through replacement on Windows), so the contract covers
host power loss rather than only orderly process restart.
An unchanged JSON recovery checkpoint preserves already-correct directory
permissions without issuing chmod, avoiding redundant Linux directory metadata
mutation. It still repairs unsafe access permissions and special mode bits;
permission repair must not replace identical JSON. Directory sync and atomic
file replacement durability remain unchanged. This is not a guarantee of zero
aggregate process writes. TestActiveMirrorUnchangedRepairsDirectoryPermissions
in internal/alerts/alerts_test.go pins permission repair and inode retention;
TestActiveMirrorUnchangedPreservesDirectoryMetadata in
internal/alerts/active_mirror_metadata_linux_test.go pins stable Linux ctime.
active-alerts.json remains an atomic recovery mirror, not a competing healthy
read authority. A new or recreated database imports the readable mirror. A
failed SQLite checkpoint writes a durable degraded marker, and the next startup
uses the mirror once to repair SQLite before clearing that marker. Without a
degraded marker, an initialized SQLite projection wins over a stale mirror.
Lifecycle append failure first checkpoints a lock-independent active-state
projection to the recovery mirror synchronously, then writes the degraded
marker. The projection is updated at the canonical set/remove and lifecycle
event seams so event emission remains safe even when its caller holds the
manager lock. A checkpoint that overlaps a lifecycle failure must observe the
failure epoch and retry before it can restore SQLite authority or clear the
marker; stale periodic work can never overwrite the failure recovery snapshot.
An unreadable or malformed mirror is write-blocked and preserved for manual
recovery even when healthy SQLite can continue serving current state; startup
must never silently replace a recoverable source with an empty snapshot.
Acknowledgement and incident age are never implicit resolution evidence:
restart restores long-running and long-acknowledged occurrences until fresh
healthy evidence or an explicit operator action resolves them.
Tracking-map cleanup now lives in internal/alerts/tracking_cleanup.go. That
file owns stale flapping, suppression, pending-alert, offline-confirmation,
Docker tracking, rate-limit, recent-alert, acknowledgement, and stale active
alert cleanup; future cleanup rules should extend that owner rather than
mixing cleanup into resource evaluators.
Active-alert lifecycle now lives in internal/alerts/active_lifecycle.go.
That file owns acknowledgement and unacknowledgement, manual active-alert
clearing, preserved alert state during rebuilds, poll-confirmed offline
recovery clears, resolved-alert registration, and no-lock active-alert removal
helpers; future active-alert lifecycle changes should extend that owner.
Active-alert cleanup now lives in internal/alerts/active_cleanup.go. That
file owns TTL cleanup, auto-acknowledgement cleanup, stale acknowledgement
retention cleanup, node-retirement cleanup, and full active-alert state reset;
future cleanup policy changes should extend that owner.
The old central internal/alerts/alerts.go file is intentionally gone. The
residual manager surface is now split by ownership: config_facade.go owns
compatibility aliases and wrapper functions for the leaf config package,
model.go owns alert runtime data structures and clone semantics,
constants.go owns package-wide cleanup and storage constants,
metric_hooks.go owns Prometheus integration callbacks, manager.go owns
Manager state and construction, default_config.go owns the factory runtime
configuration (DefaultAlertConfig), lifecycle.go owns shutdown, and escalation.go owns
the escalation loop and escalation state mutation. Future changes must extend
the owning file rather than reintroducing a central catch-all manager file.
Alert notification policy now lives in internal/alerts/notification_policy.go.
That file owns dispatch suppression, flapping suppression, quiet-hours
suppression, monitor-only notification suppression, cooldown decisions, and
per-alert rate limiting; future notification-gating changes should extend that
policy owner rather than burying new checks inside metric or resource-specific
evaluators.
Fresh installs load the factory alert configuration
DefaultAlertConfig in internal/alerts/default_config.go is the only
declaration of factory alert settings. A new Manager starts from it, and
LoadAlertConfig in internal/config/persistence.go returns it while
alerts.json does not exist; persistence must not declare a second copy. The
copy it used to return left out DockerDefaults, a value struct, so a fresh
install loaded container CPU, memory and disk as trigger 0, UpdateConfig
kept that as a deliberate Off, the thresholds page called the container Global
Defaults Custom, and the startup identity migration wrote those zeros to
alerts.json within a second of first boot. The missing-file config carries
no activation state, because activation is a stored decision: node updates and
imports reapply LoadAlertConfig, and UpdateConfig then keeps a running
manager's state, while a new manager starts pending review. An existing
alerts.json is never rewritten to factory values, because a stored trigger 0
is how a saved Off looks. The factory leaves AutoAcknowledgeAfterHours at 0: no settings
control shows it, and the 24 hours it used to declare never reached an
install. It leaves the quiet-hours timezone empty so the settings page offers
the viewer's browser zone. Flapping detection and alert TTL cleanup are on in
the factory and now reach fresh installs; a thresholds-page save still sends
neither field, so it turns them off. TestLoadAlertConfigFreshInstallUsesFactoryConfig,
TestLoadAlertConfigMissingFileKeepsRunningActivation and
TestLoadAlertConfigKeepsStoredDockerDefaultsOff in
internal/config/persistence_alert_fresh_install_test.go, plus
TestDefaultAlertConfigLeavesFiringAlertsUnacknowledged and
TestDefaultAlertConfigLeavesQuietHoursTimezoneUnset in
internal/alerts/alerts_test.go, pin this.
Agent disk temperature alerts clear when their threshold is off
Host-agent SMART disk temperature alerts (diskTemperature, resource
agent:<host>/disk_temp:<device>) follow one effective threshold: the
resolved agent DiskTemperature, refined by the disk type's DiskTempByType
entry unless an explicit host or linked-resource override set it
(hostDiskTemperatureThresholdNoLock in internal/alerts/host.go). When the
host's resolved DiskTemperature is off, CheckHost clears every disk
temperature alert on the host. A disabled DiskTempByType entry clears only
the alerts of disks of that type, through the disabled metric spec. The
all-agents switch, a disabled host override, agent removal and confirmed
agent offline clear them with the host's other disk alerts. Before this,
nothing cleared them once the agent-level threshold was off.
A config save judges an open disk temperature alert against the same
threshold (resolveHostAlertThresholdsNoLock refines it by the alert's
diskType metadata, and metric_runtime.go classifies diskTemperature as a
threshold metric), so turning the host's DiskTemperature off resolves the
alert immediately. An alert persisted before CheckHost recorded diskType (May
2026) carries no disk type until its next firing evaluation, so a save judges
it against the lowest enabled trigger any of the host's disks can use: the
agent threshold or a DiskTempByType entry
(lowestHostDiskTemperatureThresholdNoLock). A save therefore cannot resolve
an alert its disk type would still fire.
For a metric alert that reaches the threshold comparison in
reevaluateActiveAlertsLocked (docker-host alerts return before it), a config
save resolves an alert whose reading is
below the new trigger. That includes readings inside a valid recovery band,
which the evaluator would hold. A clear level at or above the trigger is no
recovery band (buildCanonicalMetricSpec drops it), and a reading at the
trigger stays firing, as it does in the reducer, so a save no longer resolves
an alert the next evaluation raises again.
TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff,
TestHostDiskTemperatureAlertsClearWhenAgentLeaves,
TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold,
TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType and
TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand in
internal/alerts/host_unraid_lifecycle_test.go pin these paths.
The disk temperature policy decides disk heat
The policy CheckHost applies to a SMART disk (the agent Disk Temp default,
refined by the disk type's DiskTempByType entry, and switched off for every
type when that default is off) is the canonical answer to "is this disk too
hot". diskTemperatureThresholdForType in internal/alerts/host.go holds it.
Manager.HostDiskTemperatureThreshold(host, diskType) resolves it for the host
agent that reports a disk exactly as CheckHost does: an explicit Disk Temp
override on that host, or one inherited from its linked node or guest, replaces
the per-type entry, the first override in that chain decides even when it sets
no Disk Temp, and an override that switches the host's alerts off leaves no
threshold. TestHostDiskTemperatureThresholdMatchesCheckHostOverrides pins each
case against what CheckHost fires, including an override under the canonical
resource the agent ID resolves to and one inherited from a linked guest. Manager.DiskTemperatureThreshold(diskType)
is the hostless case. Patrol reads the policy through
AlertThresholdAdapter.GetDiskTemperatureThreshold.
DefaultDiskTemperatureThreshold serves callers with no manager. Disk risk
(internal/storagehealth) judges no temperature. The Physical Disks Temp cell
and Health verdict mirror the policy in resolveDiskTemperatureDisplayThresholds
under the override keys of the machine whose agent reports the disk, in the
order CheckHost reads them as far as the browser can see that chain
(storage-recovery.md lists where it cannot). That resolver returns null when the agent Disk
Temp default is off or an override or agent default switches the host's alerts
off. The global alerts switch and the agent alert-type switch stop CheckHost
without changing the policy.
The displays that read it treat null as off: no threshold colour
(getTemperatureTextClass with diskTemperature) and never hot. A
DiskTempByType entry cannot be switched off on its own, because
normalization restores a non-positive entry to its default.
A disk is hot from its trigger. Alerts and Patrol findings stay open until the
reading falls to the clear value, or under the trigger when there is no band
below it. The Physical Disks verdict and the TrueNAS Health cell judge only the
current reading against the trigger. PDF reports colour disk temperatures by
the same thresholds.
TestDiskTemperatureThresholdMatchesCheckHostPolicy in
internal/alerts/threshold_resolution_shared_test.go pins per-type resolution, a raised
NVMe trigger that CheckHost also honours, the copy, the disabled default and
the nil manager.
TrueNAS disk temperature alerts follow the policy too:
trueNASDiskTemperatureDefaultNoLock
(internal/alerts/truenas_disk_temperature.go) is the type-default tier of
effectiveAlertPolicyNoLock for truenas-disk, so an unset
TrueNASDiskDefaults.Temperature resolves per disk type (the
alertPolicyQuery.DiskType the unified input carries from
PhysicalDisk.DiskType). A TrueNAS-wide value the user saved under TrueNAS
Disks replaces it for every TrueNAS disk, as a host Disk Temp override does for
an agent's disks, 0 meaning off, and a per-disk override beats both. The alert
records diskType so a config save re-judges it per type; one without it is
held to the lowest per-type trigger. A retained reading
(diskinventory.TemperatureCollected false) is no evidence and raises nothing.
The factory value is unset. normalizeTrueNASDiskTemperature drops a stored
55/50 only from a config without TrueNASDiskTemperatureByType, because the
thresholds page wrote that flat factory value back on every save; the page
writes the marker, so a 55 typed later is kept. Every config GetConfig
returns carries the marker, so a client that reads and writes back keeps it.
Without the marker (an old saved file, or a page loaded before the upgrade)
55/50 is read as that unchosen factory value; a client building a config from
scratch with an explicit TrueNAS-wide 55 has to send the marker. The TrueNAS
Disks Global Defaults cell reads By type while unset
(globalDefaultFallbacks on ResourceTable,
resolveAlertResourceGlobalDefaultCell), or Off when the agent Disk Temp
default switches the policy off, and switching it on then stages an explicit
TrueNAS-wide 55 because unset would stay off. Each disk row inherits its type's
trigger from the unsaved editor state (resolveTrueNASDiskTemperatureDefault).
The TrueNAS storage table and disk drawer now judge current disk heat through
getTrueNASDiskTemperatureThresholds(disk): the disk's canonical-ID temperature
override, then a TrueNAS-wide value, then the per-type policy. Disabled disks
or TrueNAS Disks defaults and explicit Off values leave readings unjudged;
retained values never count as current heat. The API-only disk does not inherit
a host Disk Temp override or the agent alerts switch. Patrol/AI use the disk-specific backend resolver described below; the
frontend and backend retain their own verification boundaries.
TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy and
TestTrueNASDiskTemperatureAlertIgnoresRetainedReading in
internal/alerts/unified_eval_test.go pin the tiers and retained readings;
TestTrueNASDiskTemperatureAlertsReevaluatePerDiskType and
TestTrueNASDiskTemperatureAlertWithoutDiskTypeHeldAtLowestTrigger in
internal/alerts/alerts_test.go pin re-judging on a config save;
TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues pins the
saved-config migration.
Proxmox physical disks alert on temperature under the same policy.
CheckProxmoxDiskTemperature (internal/alerts/proxmox_disk_temperature.go)
raises a diskTemperature metric alert under the disk's PVE alert reference
(ProxmoxPhysicalDiskAlertResourceID, resource type proxmox-disk), judged by
proxmoxDiskTemperatureThresholdNoLock: the disk type's DiskTempByType entry
under the agent Disk Temp default, which switches it off for every type. It
carries the disk's serial, WWN and diskType like the health and wearout
alerts, so ProxmoxPhysicalDiskAlertIdentifiers lists its
<reference>::metric-threshold:diskTemperature identifier for history
ownership, and reevaluateActiveAlertsLocked re-judges it per disk type (the
lowest per-type trigger when the type was never recorded) instead of falling
through to guest thresholds that have no disk temperature. A disk a
still-reporting linked agent lists in its SMART report is the agent's: its
alert closes with the moved_to_agent resolution and CheckHost raises the
agent's own. The handover happens on the next disk poll, so for that one
interval an agent newly linked to a disk with an open PVE alert can hold its
own alert beside it. A lapsed agent owns nothing, so the node's own readings
keep alerting while its agent is silent. An
excluded disk, or a policy switched off, closes the alert; a poll without a
current reading holds it. Like the health and wearout alerts, a disk that
leaves the inventory keeps its alert until the 24-hour stale sweep. The
proxmox-disk alert delay defaults to the factory 5 seconds, so a warning
waits out the same 300-second noisy-gauge stability window agent and TrueNAS
disk temperature warnings do, and a critical reading bypasses it.
TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy,
TestProxmoxDiskTemperatureAlertOwnershipAndHolds,
TestProxmoxDiskTemperatureWarningWaitsOutStabilityWindow,
TestProxmoxDiskTemperatureAlertsReevaluatePerDiskType and
TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity in
internal/alerts/alerts_test.go pin the policy, ownership, holds, re-judging
and identifiers.
Agent disk temperature alerts clear when their disk leaves the report
CheckHost also clears a disk temperature alert, and drops its pending
threshold run, once its device is missing from three consecutive non-empty
SMART lists (cleanupHostDiskTemperatureAlerts, a seen map keyed by resource
ID like cleanupHostCustomSensorAlerts). A removed, replaced or renamed disk
never sends the reading that would resolve its alert, and
shouldPreserveAlertOutsideNodeCleanup exempts agent: alerts from node
cleanup, so before this only the 24-hour stale-alert sweep in
cleanupStaleMaps resolved it. One omission is not proof the disk left: the
Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose
probe fails, so the count restarts whenever the disk is reported again. Three
reports in a row of failed probes on such a disk still clear its alert.
An empty SMART list skips this cleanup and does not count toward the three,
because the agent omits the list when SMART collection fails or is
unsupported. A listed disk in standby or without a temperature also holds its
alert, since neither shows the disk cooled: standby reports avoid waking the
disk, and most failed Linux probes still list the disk with identity only.
SMART health alerts hold through standby the same way. A held alert resolves
through the normal recovery path, including any configured recovery delay,
once the disk reports cool readings again; with no evaluation for 24 hours,
the stale-alert sweep resolves it. Turning the threshold off, or removing the
agent, drops the host's pending disk temperature runs and absence counts as
well as its alerts.
TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport in
internal/alerts/host_unraid_lifecycle_test.go pins the clear, the holds, the
restarted count and the pending-run and count cleanup.
Proxmox node cleanup keeps platform metric alerts
CleanupAlertsForNodes runs on every Proxmox node poll (and every mock alert
tick) and removes alerts whose Node is not a current Proxmox node or PBS
instance. Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or
platform host in Node and a display label such as TrueNAS Disk or
Kubernetes Pod in metadata.resourceType, which the preserve list's
platform words never matched. The cleanup used to delete them silently on
each poll while their reducer incident stayed firing, so the next evaluation
re-created each one and handed it to the alert callback again as a new
alert. shouldPreserveAlertOutsideNodeCleanup now keeps any alert carrying
metadata.platformType, or whose label maps through
CanonicalAlertResourceType to a Kubernetes, TrueNAS or vSphere type.
TestCleanupAlertsForNodesKeepsPlatformMetricAlerts in
internal/alerts/alerts_test.go pins both signals, and its
sibling pins that a guest alert on a removed Proxmox node is still removed.
Agent disk temperature alerts judge the reading the disk shows
CheckHost judges each agent disk on the temperature, collection state and
disk type the disk shows, read by unifiedresources.HostDiskTemperatureReadings
from the same adapters the registry builds the disks with. A SMART row shows its
own reading or, when the agent's probe returned none, the one the host's Unraid
inventory reports for that disk (HostSMARTDiskTemperature), and keeps its
agent:<host>/disk_temp:<device> resource. An Unraid inventory row whose disk
key no SMART row carries, a disk only the inventory lists, shows that row's
reading (HostUnraidDiskTemperature) and its own transport as disk type, under
its kernel block device token, the resource a plain Linux SMART row for the
same device uses. A SMART row and an Unraid row with one key are one registry
disk even under different device labels, such as a controller member
(0 [megaraid,0]) and its Unraid device (sda), and alert once, under the
SMART row's resource. When smartctl and Unraid name
one device with different serials the registry shows two disks on it, and the
device's one alert judges the reading that stands highest against its own disk
type's threshold (judgedHostDiskTemperature): over its trigger first, then
inside its recovery band, then the larger margin over the trigger, with a tie
keeping the reading listed first.
Before this CheckHost read only the SMART row's own field, while the Physical
Disks Temp cell, the Running Hot verdict ("at or above its alert threshold")
and Patrol judged the Unraid reading against the same trigger, so such a disk
could read as at its alert threshold with no alert. Unraid's own warning and
critical disk temperature notifications do not make this redundant: Pulse
already alerts on the smartctl readings and array state of the same hosts,
which Unraid also notifies on, and a user who routes alerts through Pulse sets
these thresholds here.
Only a reading collected now fires or resolves
(HostDiskTemperatureReading.Collected: no standby row, and
diskinventory.TemperatureCollected), so a spun-down disk holds the alert
like a listed disk without a temperature. A disk only the
inventory lists whose device matches the agent's --disk-exclude patterns
(fsfilters.MatchesDeviceExclude) raises nothing, and an alert it raised before
the exclusion resolves on the next report (clearHostDiskTemperatureResources),
whatever the SMART list holds, unless another listed reading uses the same
resource: docs/UNIFIED_AGENT.md documents exclusion as
removing the disk from monitoring, and linked Proxmox hosts already judge an
excluded disk healthy. A disk only the inventory lists is also judged in a
report whose SMART list is empty, because the disk shows the inventory's
reading in that report. The absence cleanup above still runs only on a
non-empty SMART list, and such a report neither counts toward nor restarts a
disk's three, so a disk that leaves the inventory while every report's SMART
list stays empty keeps its alert until the 24-hour stale-alert sweep.
TestCheckHostJudgesTheUnraidTemperatureADiskShows and
TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture in
internal/alerts/host_unraid_lifecycle_test.go pin the fallback, the
inventory-only disk, the empty SMART list, two disks on one device (also of
different types), a controller member under another device label, standby,
exclusion, the disk type
and host override thresholds, the holds, the handover to a SMART reading,
departure, and the exclusion clear with and without SMART rows and beside a
listed SMART row. TestJudgedHostDiskTemperatureOutranks pins the ranking.
Configured flapping thresholds remain reachable
Every accepted positive FlappingThreshold, including values above ten, must
be reachable by current observations inside FlappingWindowSeconds. Per-key
history retains at most max(10, FlappingThreshold) timestamps after window
pruning; it must not grow with repeated attempts during an active cooldown.
Only the threshold-crossing attempt opens that cooldown. Later attempts keep
its original deadline even after the observation window drains, and a served
cooldown resets the episode so a later storm can arm it again. Lowering the
threshold applies the new history bound and threshold on the next check. The
read-only delivery diagnosis must explain the same threshold and suppression
actually enforced at dispatch. internal/alerts/flapping_threshold_test.go
pins these boundaries through normal configuration updates and notification
callbacks, alongside the existing flapping cooldown and one-shot callback tests.
Ordinary retention cleanup and hourly tracking cleanup retire a served
suppression deadline, its flapping latch and its observations atomically under
the manager lock, using the same expiry rule as dispatch. Neither sweep may
leave an active occurrence in unbounded flapping suppression after deleting its
deadline. The next burst starts a fresh observation window and can arm a new
bounded cooldown and one-shot callback. Unexpired cooldowns, other keys' policy
and pending bursts without a deadline remain unchanged; expiry does not clear,
acknowledge or change the identity of an active alert. The cleanup regression
controls in internal/alerts/flapping_threshold_test.go exercise both sweeps,
drained and retained windows, dispatch callbacks and delivery diagnosis. These
modeled-time controls are not installed notification-destination acceptance.
Default thresholds keep a positive trigger's clear below it
PBS, node temperature, agent, Kubernetes, TrueNAS and vSphere global defaults
go through normalizeThresholdPointer in internal/alerts/config/normalize.go
from UpdateConfig. The shared persistence normalization in
internal/config/persistence.go applies the same normalizer, through
NormalizeHysteresisThreshold, to the agent, node temperature and storage
pairs. A missing or negative threshold takes the factory default and a zero
trigger is off. A positive trigger always keeps a clear below it: a missing
clear sits five points under the trigger, floored at 0, and a clear at or
above the trigger is repaired. Guest and node usage, storage and Docker
defaults hold the same bound through EnsureValidHysteresis in
ValidateHysteresisThresholds, NormalizeStorageDefaults and
NormalizeDockerThreshold. The canonical per-type entries (nvme, sata and
hdd in DiskFillByType; nvme, sas and sata in DiskTempByType, the
types the thresholds page edits) keep a positive trigger and follow the same
clear rule; a non-positive trigger resets the entry to its type default,
because the page cannot switch a type off on its own. Other per-type keys are
stored as written.
The thresholds page sends max(0, trigger - 5), so a 1-5% default arrives
with clear 0 and must not fall back to the factory clear. Agent defaults used
to do that, storing {trigger: 1, clear: 75} for a 1% Machines CPU default,
and per-type disk entries replaced the whole pair, trigger included. The
evaluator already ignored a clear at or above the trigger
(buildCanonicalMetricSpec drops it and the reducer then clears at the
trigger), so those alerts fired and resolved the same way; /api/alerts/config
served the factory clear and firing alerts reported it as clearThreshold. A
repaired pair above the margin, such as {50, 80} written through the API,
now has a real recovery band at 45 where it used to clear at the trigger.
internal/alerts/config/normalize_low_trigger_clear_test.go pins every
default family and both per-type maps, TestUpdateConfigKeepsLowTriggerClearBelowTrigger in
internal/alerts/config_validation_test.go pins the saved config and the
firing alert's clear level, and the low-trigger and stored-clear tests in
internal/config/persistence_test.go pin the written file and load repair.
Monitor-only delivery is terminal
Monitor-only alerts remain visible, but neither a firing nor a recovery notification may enter the delivery queue. They must not acquire quiet-hours replay metadata or schedule escalation levels and critical repeats. An already queued escalation must be revalidated against the current monitor-only state before delivery; the read-only delivery diagnosis reports monitor-only suppression rather than a replayable quiet-hours deferral.
The same policy owner also exposes the read-only alert delivery diagnosis
projection used by /api/alerts/delivery-diagnosis; that projection may explain
current gating state, quiet-hours replay timing, cooldown timing, rate-limit
counts, and flapping suppression, but it must not dispatch callbacks or mutate
flapping/rate-limit tracking maps.
Omitting the alert identifier returns one diagnosis for every active alert from
one manager read pass, ordered by alert identifier, and an empty active set is
encoded as [] rather than null. The Alerts Overview consumes that bulk
projection once per refresh and may render its current delivery outcome on an
unacknowledged active-alert card. Missing or failed diagnosis reads degrade to
no delivery line; they must not hide alert truth or trigger per-card requests.
internal/alerts/eventlog/ and internal/alerts/event_emission.go own the
canonical append-only alert event record. Persistent managers enable a
SQLite-backed store under the alerts data directory; ephemeral managers record
nothing unless a store is installed explicitly. Resolution, acknowledgement,
unacknowledgement, escalation, flapping detection, dispatch, quiet-hours
deferral, and suppression append immutable events without changing lifecycle
or delivery behavior. Suppression and deferral are recorded as outcome
episodes: the first decision is immutable, identical reevaluations of that
unchanged outcome append no duplicate row, and a changed reason, details,
resource presentation, intervening lifecycle/dispatch event, or later return
to that outcome opens a new episode. Lifecycle, escalation, and dispatch
events are never coalesced. This keeps delivery activity explanatory instead
of poll-frequency-shaped, prevents unchanged reevaluations from crowding a
real outcome change out of the non-blocking diagnostic buffer, and bounds
diagnostic growth without erasing a meaningful decision transition. A failed
write forgets its admission key so a recovered store can accept the outcome
again. Reads apply the same episode projection to
redundant diagnostic rows written by older versions, so upgraded installations
become readable immediately without rewriting their immutable event records.
Snapshot-bearing lifecycle transitions commit
synchronously before downstream lifecycle projections run; they never share
the droppable diagnostic buffer because alert history is reconstructed from
them. High-volume notification decisions remain non-blocking and fail-open for
alert evaluation: a full diagnostic buffer counts and drops that delivery
evidence. Failed asynchronous batches do not advance the successful-write
counter, and Flush must report the failure or timeout rather than falsely
claiming the batch landed. An unavailable or failed lifecycle store degrades
history reads to the recovery model without hiding live active-alert truth.
Events retain for 90 days and prune hourly. Retention alone does not bound disk
use: one flapping alert can write thousands of full alert-state snapshots inside
the window, so the hourly prune also enforces a total stored snapshot-payload
cap and removes the oldest events first when it is exceeded. A volume prune
raises the same retention revision as age pruning, so the history projection
rebuilds from the remaining log rather than reusing a stale fold. Fired and
refired lifecycle events come only from the reducer core's explicit activation
events: canonical lifecycle reactivation maps EventRefired separately, while
shared metric activation records one fired event when its pending incident
becomes firing. Repeated firing observations and persisted-alert restore must
not append another activation event, and the active-alert storage funnel must
never invent one.
Event-log reads accept a caller limit but normalize it to the store's bounded
maximum before it reaches SQLite. Result-slice allocation is independent of
that caller value: a request-provided limit is a row-count preference, never a
memory-allocation hint. This remains defense in depth even when an authenticated
API handler validates the query parameter, because non-HTTP manager callers use
the same store boundary.
Alert-history reconstruction is the explicit exception to a single bounded
query: it walks every matching lifecycle event oldest-first through bounded
keyset pages, releasing the SQLite rows between pages. The public/API query cap
therefore remains intact while a noisy installation with more than 1,000
lifecycle events in the requested window cannot silently lose older history.
Lifecycle events now carry a full alert snapshot and
internal/alerts/history_projection.go can fold those snapshots into one
history row per occurrence. Existing pre-snapshot SQLite stores upgrade in
place, and rows without snapshots remain readable but cannot contribute to the
projection. The event log becomes the alert-history read authority only after
legacy history is absent or its complete import and retirement succeeds.
Managers without an authoritative event store, and stores that report a
lifecycle write failure, retain the in-memory JSON-history model as their
fail-soft read fallback.
internal/alerts/history_projection_parity_test.go characterizes the event-log
projection against that fallback for active, resolved, acknowledged,
multi-resource, and repeated occurrences. Live active-alert state overlays the
latest projected snapshot so acknowledgement and current values do not become
stale.
internal/alerts/history_migration.go imports every legacy JSON entry through
the event log's synchronous, non-droppable import path before renaming the
fixed history and backup files to their .imported backups. The primary or
backup source's presence keeps migration active until both leaves retire.
Imported events are retry-idempotent by immutable alert ID, occurrence time,
and snapshot, so a committed SQLite transaction followed by a failed or
partial file rename cannot duplicate history on the next startup. An import
failure leaves JSON history authoritative for the next attempt. An unreadable
or malformed source, or any entry that cannot produce an identified complete
snapshot, blocks both retirement and further JSON writes so startup cannot
silently replace recoverable user history with an empty file. Successful
retirement stops further JSON writes. Retired .imported leaves remain bounded
recovery sources: startup loads and idempotently replays them if events.db is
recreated, so loss or quarantine of the database cannot turn a successful
migration into silent empty history. Clearing history appends a
history_cleared tombstone rather than deleting log rows; the projection
ignores earlier lifecycle events but overlays still-active alerts as current
state. Unified-incident and system-alert activation must emit snapshot-bearing
fired events so their projected history does not begin at acknowledgement or
resolution.
SubscribeLifecycleCallback is the delivery-independent projection seam for
that canonical stream. It runs for lifecycle transitions regardless of
activation, quiet hours, grouping, rate limits, or destination state; delivery
callbacks remain policy-controlled consumers. Monitoring uses this seam to
materialize canonical resource-history breadcrumbs and incident shells. On
startup it replays durable lifecycle transitions and migrated legacy-history
snapshots oldest first, then idempotently reconciles restored active alerts
that predate the event store. Because monitor construction precedes attachment
of the API-owned durable unified-resource store, that attachment repeats the
same idempotent replay so canonical resource history is repaired as well as the
incident fallback cache. A migrated final snapshot expands only facts it
actually carries: the occurrence start, saved acknowledgement, and known end;
it does not invent delivery, escalation, analysis, or remediation events.
The incident store retains those snapshot events as a fallback only when the
canonical resource timeline lacks the equivalent event, so a partial durable
projection cannot erase the occurrence start while complete canonical history
still wins. The occurrence-qualified incident API performs the same repair
from the active/history read model when retention or an older migration left a
shell absent or empty. This repairs resolved as well as active incident
timelines without replaying any delivery side effect, and ensures an alert
already visible in Overview or History cannot keep returning an unavailable
timeline merely because its notification was held or it predates the event
store. Occurrence lookup allows only one second of start-time precision drift;
it must not merge recurring or flapping incidents merely because they share an
alert identifier and started within the same former ten-minute window.
Snapshot events stored at equal timestamp precision order by lifecycle meaning,
with detection first and resolution last, rather than by generated event ID;
a malformed legacy end before its own start clamps to the start boundary.
The same dispatch policy owns firing-notification evidence on active alerts:
any alert that passes notification suppression and enters the fired callback
fan-out must carry LastNotified before the callback clone is emitted. Resolved
notification gating depends on that field to distinguish alerts whose firing
notification actually entered delivery from alerts that were never sent.
Alert read-side output now lives in internal/alerts/read_model.go. That file
owns active alert projection and sorting, metadata coercion helpers,
recently-resolved and history output wrappers, and notify-existing redispatch;
future output-ordering or metadata coercion changes should extend that owner
rather than adding another read path inside resource-specific evaluators.
PMG alert evaluation now lives in internal/alerts/pmg.go. That file owns PMG
connectivity evaluation, PMG queue and per-node queue checks, quarantine growth
tracking, and mail-rate anomaly detection; future Proxmox Mail Gateway alert
behavior should extend that resource checker owner rather than adding more PMG
logic to the central Manager file.
Docker alert evaluation now lives in internal/alerts/docker.go. That file
owns Docker host connectivity, container state and health, container metric
projection, service gap/update-state checks, image-update timing, and Docker
tracking cleanup; future Docker alert behavior should extend that resource
checker owner rather than expanding the central Manager file. It must not keep
shadow last-exit-code state or infer an OOM kill from exit 137; the accepted
container model's nullable runtime-authored OOMKilled field is the sole OOM
classification input.
PBS alert evaluation now lives in internal/alerts/pbs.go. That file owns PBS
connectivity normalization, PBS metric projection, PBS metric cleanup, and PBS
offline lifecycle handling; future PBS alert behavior should extend that
resource checker owner rather than expanding the central Manager file.
Storage alert evaluation now lives in internal/alerts/storage.go. That file
owns storage connectivity handling, storage usage projection, ZFS pool/device
health checks, and storage offline lifecycle handling; future storage alert
behavior should extend that resource checker owner while shared storage-health
assessment helpers remain package-level until host and storage health paths are
separated cleanly.
Storage connectivity status is trimmed and case-normalised before evaluation.
Empty or unknown status is absent connectivity evidence: it must not advance
recovery or replace an existing incident's identity/start time. Capacity and
pool-health evaluation remain independent and may still use valid observations.
Normalised offline and unavailable observations do not feed capacity
evaluation, preserving the existing rule that offline capacity is not actionable.
The existing non-failing treatment of available, online, active, inactive and
disabled statuses remains unchanged; explicit connectivity-disable policy still
clears the incident. TestStorageUnknownConnectivityDoesNotRecover and
TestStorageKnownConnectivityRecoveryCompatibility in
internal/alerts/alerts_test.go pin unknown-observation preservation, independent
capacity activation, normalised offline activation and confirmed recovery.
Proxmox node alert evaluation now lives in internal/alerts/node.go. That file
owns node metric and temperature projection, node offline lifecycle handling,
and instance-scoped node display-name cache updates; future Proxmox node alert behavior should extend that resource
checker owner rather than expanding the central Manager file.
When a reporting host agent is linked to a node (LinkedNodeID, the monitor's
automatic or operator-set identity decision), the agent resource owns each
CPU, memory or disk usage metric it actually evaluates, and CheckNode releases
its own copy of those metrics through the disabled-threshold path every cycle:
the pending run is dropped and any node alert still open from before the link is
resolved, never left frozen until the agent goes offline. CheckHost applies
its link outcome after evaluation in per-agent report order. CPU and memory
ownership requires a live threshold plus a usable, finite observation with a
ready evaluation window, or a still-open agent alert or reducer run. For disk,
that evidence must belong to the summary filesystem models.SummaryDisk picks,
which is also what the linked node's disk metric reports. Configuration alone
is not coverage: an agent with no observation and no open run leaves the node
responsible. Missing readings keep an existing agent alert, without creating a
node duplicate; observation-gap interruption and fresh confirmation/recovery
remain unchanged. Disabling a usage metric releases canonical/shadow pending
runs and explicit intent grace, even without a reading. A disabled agent or host
also releases its CPU/memory pending runs. UpdateConfig immediately revokes
unsupported link coverage, and in-flight reports recheck current policy before
applying their outcome. Newer reports, offline and removal invalidate older
link updates and removals. Several linked agents contribute the union of their
coverage; stale alert metadata alone never suppresses a node. The accepted
short duplicate window remains restart-before-first-report and
first-offline-before-confirmed-offline; manually linked sibling agents retain
their own alerts. Explicit host/inherited disk overrides beat DiskFillByType,
as do direct filesystem overrides. A saved direct filesystem disable revokes
summary disk ownership without a new agent report. Per-type defaults retain
the existing normalization contract (zero is restored to the type default);
this repair does not add a per-type Off setting. A nil effective filesystem
threshold drops its pending run.
A node-side close is an ownership handover, not observed recovery: each released
metric carries moved_to_agent and the identity/name of an agent actually
covering that metric in the same link snapshot. If several agents cover it,
the lowest covering agent ID wins deterministically; a sibling covering only
another metric must not be named. Resolved consumers still receive the close.
internal/alerts/host_agent_node_links.go owns ordered links and per-metric
successor selection, without persisting stale coverage across restart.
Deduplication keys on that link,
never on a hostname match, so a same-named node in another instance keeps its
alerts, an agent reporting an FQDN still dedups its node, and an operator unlink
hands the alerts back. CPU temperature has no host-agent metric, so the node
keeps evaluating it whether or not an agent is linked; the node poll already
merges agent sensor readings into node.Temperature, giving one temperature
alert per machine. A missing CPU reading is not evidence: with a live trigger,
checkNodeTemperature neither feeds 0°C nor resolves the open alert, but
interruptMetricRun (reducer InterruptMetricRun) drops a pending activation
run and restarts a recovery run, so no sustained-for or recovery delay spans a
gap in readings; a disabled threshold still clears. Config-save reevaluation
classifies alerts whose metadata resourceType is node as node alerts and
judges them against node thresholds; the Instance heuristic only covers legacy
alerts without a resource type, because current node alerts keep the PVE
instance name in Instance.
TestCheckNodeKeepsTemperatureAlertWhenHostAgentMonitorsNode,
TestCheckNodeReleasesOpenMetricAlertWhenHostAgentRegisters,
TestCheckNodeMissingTemperatureDoesNotResolveOpenAlert,
TestCheckNodeMissingTemperatureInterruptsTimingRuns,
TestConfigSaveKeepsNodeTemperatureAlertOverTrigger,
TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate,
TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate,
TestCheckNodeKeepsMetricsTheAgentCannotEvaluate,
TestDisabledAgentThresholdReleasesPendingRunWithoutReading,
TestHostAgentNodeLinkFollowsConfigAndReportOrder,
TestCheckNodeTakesBackMetricsWhenAgentUnlinks,
TestHostDiskOverrideBeatsDiskFillByType,
TestReleaseCanonicalMetricAlertClearsIntentPending in
internal/alerts/threshold_resolution_shared_test.go and
TestReportedAlertOwnershipComposition in
internal/alerts/reported_alert_composition_test.go, plus
TestHostAgentDeduplicationFollowsNodeLink in
internal/alerts/host_dedup_test.go and
TestCheckMetricReportsWhetherObservationWasUsable in
internal/alerts/windowed_metric_test.go, pin these rules.
The real monitor lifecycle/History route is exercised by
TestMonitorReportedAgentHandoverUsesActualMetricOwner in
internal/monitoring/monitor_alert_handling_test.go.
Host-agent alert evaluation now lives in internal/alerts/host.go. That file
owns host identity, host-agent metric projection, host disk/SMART/RAID/Unraid
health handling, host cleanup, and host offline lifecycle handling; future host
agent alert behavior should extend that resource checker owner while shared
health-assessment evaluation remains package-level until all storage-health
callers can be separated behind a narrower owner.
The host offline lifecycle is raised only by a lapsed agent report, and
Pulse cannot tell a dead host from an agent that lost its route to Pulse.
Its message therefore names the silent agent ("Pulse agent on ''
stopped reporting", using the agent's own name without the linked
(Host Agent) qualifier) and never claims the host is offline, because a
linked Proxmox node keeps reporting through its API while its agent is
silent. The host-offline type, alert identity and lifecycle stay
unchanged. TestHandleHostOfflineNamesSilentAgentNotHostOutage in
internal/alerts/alerts_test.go pins this wording.
Host long-running storage-operation alerts require fresh operation evidence.
An accepted Unraid cancellation/completion or terminal Linux RAID report
resolves the corresponding operation alert immediately. A normal polling gap
does not imply completion; after the monitoring-owned reporting lease expires,
alerts must re-evaluate with only transient operation/progress fields cleared,
retain alerts backed by static degraded/topology evidence when that last-known
topology remains available, and progress the separate confirmed
host-connectivity alert. Persisted operation alerts also obey the durable
accepted-telemetry lease after server restart, so restart cannot make a stale
parity or rebuild alert immortal and missing telemetry is never represented as
healthy reporting.
Snapshot and backup-age alert evaluation now lives in
internal/alerts/backup_snapshot.go. That file owns snapshot age/size
evaluation, backup rollup age evaluation, backup inventory readiness, PVE
template subject matching, namespace disambiguation, and snapshot/backup active
alert cleanup; future backup or snapshot alert behavior should extend that
owner rather than expanding the central Manager file.
Backup-age attribution follows live identity
Backup rollup subject refs are historical hints, not unconditional proof of a
guest's current placement. An exact instance:node:vmid ref is authoritative
only while it resolves to a live guest of the same type. Otherwise backup-age
evaluation retries typed VMID attribution and prefers live candidates carrying
a resource ID over last-known display metadata. A PVE-owned ref can still name
its authoritative orphan after the PVE inventory is ready; a stale PBS ref that
cannot be resolved uniquely remains unattributed instead of pinning an alert to
an old or wrong PVE node. TestCheckBackupsRemapsStaleSubjectRefToUniqueLiveGuest
and the PVE orphan/inventory-readiness tests in internal/alerts/alerts_test.go
pin both sides of this boundary.
Per-guest backup and snapshot overrides are sparse, not frozen copies. A
zero-valued threshold field in an override (warningDays, criticalDays,
freshHours, staleHours, and the snapshot size pair) inherits the current
global default at evaluation time through the merge helpers in
internal/alerts/backup_snapshot.go; explicit non-zero values win, and
alertOrphaned plus ignoreVMIDs always resolve from the globals because they
are instance-wide filters. NormalizeRecoveryOverrides in
internal/alerts/config/normalize.go (called from UpdateConfig in
internal/alerts/config_runtime.go and from the shared persistence
normalization in internal/config/persistence.go) rewrites stored overrides
whose threshold tuple exactly matches the current globals into sparse
enabled-only form. Those full copies were artifacts of the legacy per-guest
toggle, and the equality condition makes the rewrite behavior-preserving at
the moment it runs while letting the guest track later global edits (#1126).
Overrides whose thresholds differ from the current globals must be preserved:
persisted values alone cannot distinguish deliberate per-guest settings from a
legacy copy made before the globals changed. A global-settings screenshot does
not establish the effective guest threshold.
TestBackupDivergentDefaultsOnLoad in internal/alerts/alerts_test.go verifies
JSON-decoded configuration with 33/34-day globals: absent and enabled-only
overrides inherit the globals, while a retained 7/14-day override still fires
a seven-day warning. This is configuration-path evidence, not a reproduction
of a reporter installation or a migration that guesses operator intent.
TestGuestBackupOverrideInheritsGlobalThresholds,
TestFrozenBackupOverrideMigratesToSparse,
TestDeliberateBackupOverridePreserved, and
TestMergeSnapshotOverrideInheritsZeroFields in
internal/alerts/alerts_test.go pin the merge, migration, and preservation
behaviors.
Backup-age alerts are guest-workload scoped. A PBS recovery rollup whose subject
carries no guest VMID is a host/config backup (backup-type host, keyed by the
node name), not a guest workload, and must not raise a guest backup-age alert:
the Backups overview already excludes those entries from guest coverage, and a
stale host config backup otherwise notified forever under a node-named subject
while the node's guests were backed up. Evaluation skips PBS subjects with an
empty VMID, retaining their recovery point and rollup so inventory and artifact
display are unchanged. PVE host backups already produce no subject and no alert.
Genuinely orphaned PBS guest backups (numeric VMID) and PMG node backups still
alert. TestCheckBackupsIgnoresPBSHostBackupSubject and
TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup in
internal/alerts/alerts_test.go pin both sides of this boundary (#2136).
Proxmox disk health alert evaluation now lives in
internal/alerts/disk_health.go. That file owns Proxmox disk canonical
identity, disk health assessment alerts, known-firmware health suppression, and
SSD wearout alerts; future Proxmox disk-health behavior should extend that
checker owner rather than expanding the central Manager file.
PVE disk-health and wearout occurrences retain the existing
instance:node:disk:sanitized-device-path resource reference. The shared
constructor in unified resources must produce the same identity as the alert
checker; changing occurrence IDs to match registry source IDs would reopen
historical incidents and can re-notify a worn disk. The registry resolves this
reference as an alias of the exact physical disk for operator-state lookups.
Muting one physical disk therefore resolves its active wearout alert and blocks
later writes and notifications without muting another disk or changing the
wearout recovery rules. TestProxmoxDiskCanonicalResourceIDTrimsIdentity pins
the persisted identity shape; monitoring's registry-backed
TestProxmoxPhysicalDiskMuteResolvesAndSuppressesWearoutAlert pins the policy
path. These are source invariants, not evidence of installed field relief.
Each PVE disk alert records the evaluated disk's serial and WWN
(unifiedresources.MetadataDiskSerial, MetadataDiskWWN), and resource
history owns each lifecycle row by that identity rather than by the path. A
row without a usable serial or WWN names only an identity-less disk at its
path, and a row whose identity is ambiguous, or names no disk but one another
resource already holds, stays under the path reference.
Reads by the path reference find those rows through the alert identifiers
ProxmoxPhysicalDiskAlertIdentifiers lists, so spec IDs and occurrence IDs
must keep matching it;
TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity in
internal/alerts/alerts_test.go pins both.
Shared metric threshold runtime now lives in
internal/alerts/metric_runtime.go. That file owns metric threshold lookup,
per-metric delay and intent resolution, reducer input composition, active-alert
projection, delivery side effects, metric runtime options, alert key sanitation,
and metric delta helpers. The reducer core owns pending, firing, severity, and
hysteresis transitions; future metric-threshold behavior should extend these
owners rather than adding shared metric logic back to the central Manager file.
Shared storage-health assessment alerting now lives in
internal/alerts/health_assessment.go. That file owns storage-health reason
normalization, ZFS pool/device reason filtering, canonical health-assessment
alert synchronization, and ZFS device assessment construction for host and
storage checkers; future shared health-assessment behavior should extend that
owner rather than reappearing inside resource-specific evaluators or the
central Manager file.
Proxmox guest alert evaluation now lives in internal/alerts/guest.go. That
file owns guest metric projection, per-disk guest metric alerts, guest
powered-off lifecycle alerts, Pulse tag controls, relaxed guest thresholds, and
guest suppression cleanup; future guest-specific alert behavior should extend
that checker owner rather than expanding the central Manager file.
Commercial alert handoffs now follow the same shared navigation boundary.
frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx may resolve
the canonical ai_alerts destination from the shared license/commercial
contract, but it must delegate the actual open behavior to the
frontend-primitives typed upgrade-navigation owner instead of reintroducing
alert-local window.open(...) or raw external-tab assumptions.
That same alert button must also honor the ordinary self-hosted prompt
suppression policy: when presentationPolicy.hideUpgrade is true, a locked
alert-investigation action may remain visibly unavailable, but it must not
show Pro-required tooltip copy, track upgrade clicks, or open the commercial
handoff route.
Unlocked alert-investigation Assistant handoffs are contextual explanation and
triage entries, not autonomous execution grants. InvestigateAlertButton.tsx
must pass autonomousMode: false when it opens Pulse Assistant, and it must
open the drawer with context only rather than seeding a product-authored prompt
or choosing a diagnostic/remediation route. The
visible drawer briefing for that same handoff is Alerts-owned presentation
context: alert identifier, severity, metric, resource, threshold, duration,
node label, and message may be shown, while raw diagnostic or remediation
commands remain outside the handoff.
Alert-adjacent shared helpers also inherit the runtime-versus-commercial split
now carried by the shared licensing stores. Alert pages may consume runtime
feature truth from frontend-modern/src/stores/license.ts, but any
upgrade/trial posture must come from the dedicated commercial-posture
contract, and public-demo suppression must flow from the shared resolved
presentationPolicy contract instead of alert-local demo checks or
entitlement reads.
That same shared read-only presentation contract now also owns the public-demo
alerts shell posture. When presentationPolicy.readOnly is true, the alerts
page must behave as a reporting surface: overview/history remain available,
alert activation controls stay hidden, configuration tabs must not render or
remain navigable, and the overview empty state must not tell public demo users
to toggle alerting back on when demo mode already blocks write requests.
The alerts page also owns its mobile tab-shell presentation directly.
frontend-modern/src/pages/Alerts.tsx may keep alert-specific active and
disabled tab styling, but horizontal tab scrolling must route through the
shared touch-scroll / scrollbar-hide class contract instead of writing
inline overflow styles that break CSP on the public shell. The selected tab
must expose canonical current-page state and compose the shared active
horizontal-rail visibility owner, so direct navigation and viewport changes
bring Thresholds, Notifications, or Schedule fully into view instead of
leaving the active destination clipped beyond the mobile rail.
Alert tab routing is part of that same presentation boundary.
frontend-modern/src/features/alerts/types.ts owns the canonical mapping
between visible alert tabs and URLs. The operator-facing Notifications tab
must use /alerts/notifications as its canonical route because the visible
navigation label is Notifications; /alerts/destinations is a legacy alias
only and must normalize through tabFromPath / pathForTab instead of being
reintroduced as canonical UI vocabulary. The page header for that tab must use
Notifications as its title; destination wording may describe concrete email,
Apprise, or webhook endpoints, but it must not reappear as the primary tab or
page identity.
That shared alert presentation boundary now also has explicit alerts ownership.
frontend-modern/src/utils/alertWebhookPresentation.ts is the canonical owner
for webhook setup copy, service labels, mention-help phrasing, custom-field
presets, and add/test/update/delete action wording;
frontend-modern/src/utils/alertSchedulePresentation.ts owns quiet-hours day
and suppress-category card styling; and
frontend-modern/src/utils/alertIncidentPresentation.ts owns incident badge,
timeline, filter-chip, note-editor, and resource-incident panel presentation.
Future alert presentation work must extend those helpers through the alerts
contract instead of leaving alert-facing wording or styling inlined in page or
feature shells while the registry treats the helpers as unowned.
German and Spanish localization for the Alerts Overview operator journey is
owned by the canonical i18n layer plus the alert presentation helpers:
frontend-modern/src/i18n/messages.ts and
frontend-modern/src/i18n/policy.ts own the catalog and non-translation
policy, while frontend-modern/src/utils/alertOverviewPresentation.ts,
frontend-modern/src/utils/alertActivationPresentation.ts, and
frontend-modern/src/utils/alertTabsPresentation.ts own the translated
operator-facing alert overview, activation, tab, timeline, and acknowledgement
copy consumed by frontend-modern/src/pages/Alerts.tsx,
frontend-modern/src/features/alerts/AlertOverviewStatsCards.tsx,
frontend-modern/src/features/alerts/AlertOverviewActiveAlertsSection.tsx,
frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx,
frontend-modern/src/features/alerts/useAlertAcknowledgementState.ts,
frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx,
frontend-modern/src/components/Alerts/IncidentEventFilters.tsx, and
frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx.
Machine-facing alert identifiers, alert types, resource IDs, resource names,
node names, source-system messages, commands, command output, event payloads,
and log text must remain untranslated; only the user-visible alert controls
may use the active app locale.
The Recovery section is a thresholds deep link: /alerts/thresholds/proxmox#backups
expands and scrolls to it like #pbs and #pmg (useThresholdsTableState.test.tsx).
The thresholds resource search is route state: useThresholdsTableState
reads and writes ?q= (ALERT_THRESHOLDS_QUERY_PARAMS.query) with replace
navigation, keeps it across platform tab clicks, section hashes and the
legacy-path redirects, so a link from elsewhere can open a tab already
searched to one resource (buildAlertThresholdsPath). Host and Docker host
rows match the search on their shown, display and raw names
(matchesThresholdResourceSearch), because a link carries the full display
label while those rows show a shortened name and keep the label as their raw
name. Because each search edit navigates within the
page, the Alerts page's unsaved-changes guard asks only when the destination
path differs (shouldConfirmAlertsLeave in alertConfigPresentation.ts):
narrowing the list keeps an unsaved edit without a prompt, leaving the page
still asks (useThresholdsTableState.test.tsx,
alertConfigPresentation.test.ts, Alerts.helpers.test.ts).
A successful Alerts page save re-reads the shared alertsActivation config in
useAlertsConfigurationState.ts (only when the config write landed and the
save still owns the editor), because backup shields and metric colours on
every page read that config and otherwise kept the old limits until a reload.
refreshConfig in frontend-modern/src/stores/alertsActivation.ts installs a
read only while it is the latest: each read and each organisation switch takes
a new generation, so a slow reply for the previous organisation or an older
read cannot overwrite the current config (alertsActivation.test.ts,
useAlertsConfigurationState.test.tsx).
The remaining alert configuration and history presentation helpers now also
have explicit alerts ownership. frontend-modern/src/utils/alertActivationPresentation.ts,
frontend-modern/src/utils/alertAdministrationPresentation.ts,
frontend-modern/src/utils/alertBulkEditPresentation.ts,
frontend-modern/src/utils/alertConfigPresentation.ts,
frontend-modern/src/utils/alertEmailPresentation.ts,
frontend-modern/src/utils/alertFrequencyPresentation.ts,
frontend-modern/src/utils/alertGroupingPresentation.ts,
frontend-modern/src/utils/alertHistoryPresentation.ts,
frontend-modern/src/utils/alertSeverityPresentation.ts,
frontend-modern/src/utils/alertTabsPresentation.ts,
frontend-modern/src/utils/alertThresholdsPresentation.ts, and
frontend-modern/src/utils/alertThresholdsSectionPresentation.ts are the
canonical owners for alert enablement copy, history administration wording,
bulk-edit labels, schedule/configuration text, email-destination field labels,
frequency chips, grouping card styling, history source and resource badges,
severity badges, tab labels, thresholds empty states, and thresholds section
status labels. Overview stat-card labels must also route through the alert
overview presentation helper, and user-facing configuration or thresholds copy
must use workload, VM, and container vocabulary instead of exposing internal
guest override/filter names unless the UI is naming a backend field directly.
Alert severity presentation has a split ownership boundary: alerts owns
formatAlertSeverityLabel, compact severity labels, legacy alert severity
class helpers, severity-bucket-to-StatusIndicatorVariant mapping, and
severity-bucket-to-detail-row tone mapping in
frontend-modern/src/utils/alertSeverityPresentation.ts, while
frontend-primitives owns the visible platform severity badge and dot shells in
frontend-modern/src/components/shared/AlertSeverityBadge.tsx. Platform alert
tables must consume that split instead of recreating provider-local severity
label, text-class, detail-tone, or variant helpers.
Platform alert detail field formatting is also a shared alerts presentation
boundary. frontend-modern/src/utils/alertDetailPresentation.ts owns provider
code labels, provider-specific resource-type labels, vSphere alert entity
labels, row timestamp labels, and detail timestamp labels for Docker,
Kubernetes, TrueNAS, and vSphere alert tables. Platform alert tables must call
those helpers instead of recreating local formatCode, formatResourceType,
formatEntityType, formatStartedAt, or detailDateTime helpers.
Alert history filter defaults such as the all-time period option must likewise
come from the alert overview/history presentation helper and the shared
filter-option label primitive rather than hard-coded title-case strings in the
history filter card.
Alert configuration select options share that same rule: all-channel
escalation labels must come from alertConfigPresentation.ts plus the shared
filter-option primitive, not a schedule-page-local All Channels string.
Thresholds empty states that hand operators to Infrastructure settings must use
frontend-modern/src/utils/infrastructureSettingsPresentation.ts for the
canonical Settings → Infrastructure label instead of hard-coding generic
Settings copy or removed nested settings paths.
Future alert configuration or history presentation work should
extend those helpers instead of rebuilding alert-specific semantics in pages,
dashboard surfaces, feature hooks, or thresholds shells.
Alert history and threshold resource tables also inherit the shared product
table subgroup-row contract. AlertHistoryTableGroupRow and grouped rows in
AlertResourceTableDesktop must route their date/resource group bands through
frontend-modern/src/components/shared/groupedTableRowPresentation.ts instead
of local bg-surface-alt fills, so alert subgroup hierarchy stays visually
consistent with Infrastructure, Workloads, and Storage tables.
Alert history table shells must also rely on the shared TableCard frame and
the shared Table primitive for horizontal overflow rather than adding
alert-local bordered or overflow-x-auto wrappers inside the history section.
The alert webhook service chooser also now derives its service set from the
backend webhook template registry, rather than keeping a second frontend-only
list of services, labels, descriptions, and mention-copy metadata.
The WebhookConfig editor now imports the shared webhook template API type
directly so it does not retain a local duplicate shape for chooser metadata.
That webhook editor now also keeps runtime ownership in
frontend-modern/src/components/Alerts/useWebhookConfigState.ts, while
frontend-modern/src/components/Alerts/WebhookConfigList.tsx owns the
existing-webhook list surface and
frontend-modern/src/components/Alerts/WebhookConfigForm.tsx owns the
add/edit form surface. Future webhook template loading, form normalization,
custom-field preset handling, or webhook editor state transitions should land
in those owners instead of being rebuilt inline in
frontend-modern/src/components/Alerts/WebhookConfig.tsx.
Alert spec validation still accepts the explicit migration-bridge resource
types (node, agent-disk, docker-host, backup-subject,
proxmox-disk), but any other non-canonical type string is rejected before
it can reach alert persistence. That keeps alert routing aligned with the
canonical unified resource model instead of silently normalizing legacy type
aliases inside the alert layer.
Frontend alert surfaces and backend alert-support files now require explicit registry path-policy coverage, so new alert-owned runtime files must be mapped to a concrete proof route instead of silently inheriting subsystem-default verification.
The alerts schedule surface now also routes quiet-hour suppress-category card
styling through frontend-modern/src/utils/alertSchedulePresentation.ts
instead of leaving that selectable-card presentation inline in
frontend-modern/src/pages/Alerts.tsx.
That same schedule/runtime boundary also owns quiet-hours clock semantics.
Backend quiet hours are minute-granular user input, so runtime evaluation must
treat the configured start minute and end minute as inclusive and therefore
keep schedules such as 00:00 to 23:59 active through the full final
minute instead of expiring at 23:59:00. Alert quiet-hours proofs should
control time through the alert manager clock hook instead of depending on wall
clock execution at whatever second the test runner happens to hit.
The schedule compares local civil clock minutes on the selected calendar day,
not time.Date instants. Both occurrences of a repeated minute have the same
quiet-hours policy; nonexistent clock minutes do not shift the configured
start or end. Initial queued replay uses the first real minute outside the
current non-full-day window, including an unselected day reached at midnight.
This does not reinterpret the day toggles as the previous evening's ownership.
Full-day windows retain the existing daily revalidation boundary; it is not an
invented open minute or permission to send during a continuous schedule.
TestQuietHoursDaylightSavingClockAndReplay in
internal/alerts/quiet_hours_test.go checks London, New York and Lord Howe
(one-hour and half-hour changes), inclusive boundaries and exact UTC replay.
TestQuietHoursOvernightSelectedCalendarDays checks overnight day selection.
Quiet-hours location fallback is read-only; configuration updates own the
location cache, including when public suppression helpers run concurrently.
Manager.QuietHoursNotificationPolicy exposes an immutable read-only snapshot
to the notification owner before it takes queue or per-alert delivery locks.
Initial admission and queued replay share civil-clock/category semantics,
including repeated/skipped minutes and inclusive ends. The queue revalidates
every due attempt against current policy rather than trusting old replay
metadata. Policy snapshots neither mutate alert lifecycle/acknowledgement nor
select destinations; cancellation and actual delivery remain queue-owned.
TestQuietHoursDeliverySnapshotIsCurrentIndependentAndReadOnly and
TestQuietHoursDeliverySnapshotConcurrentConfigUpdates in
internal/alerts/quiet_hours_test.go cover snapshot isolation and concurrent
configuration reads. This establishes source scheduling, not external receipt.
Quiet-hours suppression also applies to alert delivery lifecycle, not only the
initial raised notification. Resolved notifications must not fan out when the
alert was never notified or was already acknowledged, and monitoring-driven
escalation delivery must consult the same quiet-hours suppression path while
still letting canonical escalation state reach websocket consumers.
Resource-tag filtering must not re-evaluate mutable tags for a resolved event.
The notification owner uses the firing delivery receipt to select recovery
destinations, so a tag change between firing and recovery cannot suppress a
clear from a destination that received the original alert.
That schedule surface now also follows the same shell/runtime split as the
other feature tabs: frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx
stays the render shell, while
frontend-modern/src/features/alerts/useAlertScheduleState.ts owns schedule
reset behavior, quiet-hours day/category toggles, cooldown/grouping/escalation
update policy, and the canonical defaults handoff. Future schedule control-flow
work should extend that hook instead of rebuilding those mutations inline in
the tab shell.
The backend cooldown gate is part of that same schedule contract. A disabled
cooldown (0 or negative) means "do not send periodic re-notifications for
the same active alert"; it still allows the first notification for a new alert
occurrence, while level-escalation delivery remains owned by the separate
escalation path. Runtime evaluation must not treat disabled cooldown as
"always notify" because the alert loop runs every metric tick.
Incident-event filter chip and filter-action styling now routes through
frontend-modern/src/utils/alertIncidentPresentation.ts for both
frontend-modern/src/pages/Alerts.tsx and
frontend-modern/src/features/alerts/OverviewTab.tsx instead of allowing
those alert timeline surfaces to fork their own filter presentation.
Alert incident acknowledged badges, timeline event cards, and note-editor
presentation now also route through
frontend-modern/src/utils/alertIncidentPresentation.ts instead of remaining
duplicated inline across the alerts page and overview timeline surfaces.
Poll-driven connectivity recovery is also part of canonical alert truth. Resources that clear an offline alert from later healthy polls must require repeated healthy confirmations before resolving that alert instead of clearing on the first recovered sample; otherwise transient poll recovery reopens the same regression as false "back online" notifications and missing downtime signal. Nodes, PBS, and PMG require three healthy confirmations before resolution, while storage requires two.
Host-agent threshold ownership now follows the linked resource model. Explicit agent overrides still win, but when no host-agent override exists the alerts runtime must inherit linked node or guest overrides for that agent so metric and connectivity behavior match the logical machine the agent augments. Persisted host alerts must carry enough linked-resource metadata for reevaluation after threshold changes to honor that same inheritance rule.
Alert resource tables, grouped node headers, and alert override reconstruction
now route resource-backed names through the shared policy-aware alerts helper
so governed resources do not fall back to raw names when the thresholds editor
rebuilds, saves, or re-renders override rows.
Alert threshold tables now route their visible resource row labels, search
labels, and persisted override display names through the same shared helper
so governed agent, guest, and storage rows do not leak raw names when the
threshold editor saves or re-renders them.
That threshold editor data shaping now routes through
frontend-modern/src/features/alerts/thresholds/thresholdsResourceModel.ts
for shared override-ID compatibility, grouped resource normalization, and
storage status policy, while
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsData.ts
stays the composition owner for the family-specific threshold projectors in
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsHostData.ts,
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsDockerData.ts,
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsGuestData.ts,
and
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsInfrastructureData.ts.
backup and snapshot default sanitization plus factory-drift policy now live in
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsRecoveryDefaultsState.ts,
while frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsTableState.ts
owns threshold-table route sync, section collapse state, search/edit shell
state, and bulk-edit dialog control. Pure override upsert and hysteresis-entry
helpers now live in
frontend-modern/src/features/alerts/thresholds/thresholdsOverrideMutationModel.ts.
Threshold edit persistence, bulk threshold application, and backup/snapshot
override toggles now route through
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsOverrideMutations.ts,
while powered-off/connectivity state transitions plus alert-removal side
effects now route through
frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsAvailabilityMutations.ts.
The backup/snapshot override toggles in that mutations owner write sparse
enabled-only overrides (zero-valued threshold fields) rather than copying the
current global defaults into the override, because zero-valued fields inherit
the globals at evaluation time and a copied value freezes the guest against
later global edits (#1126). Toggling a guest that already carries explicit
override thresholds must preserve those values while flipping enabled.
The Backups and Snapshots global-defaults editors in
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxBackupsSection.tsx
and
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxSnapshotsSection.tsx
reconcile warning/critical pairs through reconcileWarningCriticalEdit in
frontend-modern/src/features/alerts/thresholds/helpers.ts: a single-field
edit that creates a conflict adjusts the field the user did not touch, so a
32-day warning edit raises critical alongside it instead of being silently
clamped back down by the sanitizers before save.
That same thresholds host-data boundary now treats top-level TrueNAS appliances
as canonical agent resources with platformType: 'truenas'. System-disk
group headers must use agent-owned header metadata instead of guest/node-
friendly header metadata, so appliance labels like TrueNAS Main do not
collapse to vendor-only TrueNAS inside thresholds or override surfaces.
The thresholds tab adapter contract now lives in
frontend-modern/src/features/alerts/thresholds/thresholdsTabModel.ts, so
frontend-modern/src/features/alerts/tabs/ThresholdsTab.tsx stays a shell
instead of carrying a duplicate table-prop interface and hand-mapped adapter
layer.
frontend-modern/src/components/Alerts/ThresholdsTable.tsx is now limited to
shell composition for search/help/nav plus bulk-edit dialog flow, while the
tab render owners live in
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxTab.tsx,
frontend-modern/src/components/Alerts/ThresholdsTablePMGTab.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableDockerTab.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableKubernetesTab.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableTrueNASTab.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableVMwareTab.tsx, and
frontend-modern/src/components/Alerts/ThresholdsTablePBSTab.tsx. New
threshold row grouping, override-ID compatibility, resource normalization,
thresholds-table controller logic, or per-tab runtime should land in those
feature hooks and tab owners rather than being rebuilt inside the shell.
The shell-owned thresholds sub-routes are platform-shaped user-facing paths:
/alerts/thresholds/proxmox, /alerts/thresholds/docker,
/alerts/thresholds/kubernetes, /alerts/thresholds/truenas,
/alerts/thresholds/vmware, /alerts/thresholds/pbs,
/alerts/thresholds/pmg, and /alerts/thresholds/systems. Legacy neutral
links like /alerts/thresholds/infrastructure,
/alerts/thresholds/containers, and /alerts/thresholds/mail-gateway must
redirect to the matching platform-shaped route. Legacy
/alerts/thresholds/agents links must continue to resolve to Systems.
frontend-modern/src/components/Alerts/ThresholdsTable.tsx must expose those
peer platform routes through the shared Subtabs primitive rather than an
inline FilterBar group. Clearing search or override filters must preserve the
active platform route.
The thresholds shell is overview-first: every resource section starts
collapsed, persisted operator expansion choices still win, and Expand all /
Collapse all remain available on every platform tab. Because collapsed tables
cannot carry the discovery burden, the shell must surface a filter-independent
custom-override summary above the sections; choosing a summary item selects
the Custom-only filter, opens its owning section, and moves that section into
view. Resources not represented there continue to inherit their group defaults.
Metric enablement is an explicit On/Off interaction in row, global-default,
mobile, and bulk editors. Enabled numeric inputs accept positive trigger values
only and user-facing copy must not expose the persisted disable sentinel.
Internally, the canonical <= 0 read rule remains intact for legacy data, while
new Off actions stage -1 (saved as trigger 0 on global defaults, which the
backend otherwise resets to factory), so existing configuration and the API
contract continue to round-trip without making 0 a second customer-facing
disable path.
Within the Proxmox tab, render-heavy ownership now further routes through
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxNodesSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxPBSSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxGuestsSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxGuestFilteringSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxBackupsSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableProxmoxSnapshotsSection.tsx,
and frontend-modern/src/components/Alerts/ThresholdsTableProxmoxStorageSection.tsx
with the shared section contract in
frontend-modern/src/features/alerts/thresholds/thresholdsTableSectionProps.ts.
Future infrastructure-thresholds presentation work should extend those section owners
instead of expanding frontend-modern/src/components/Alerts/ThresholdsTableProxmoxTab.tsx
back into a mixed render surface.
The Docker tab now follows that same section-owner shape through
frontend-modern/src/components/Alerts/ThresholdsTableDockerIgnoredPrefixesSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableDockerServiceGapSection.tsx,
frontend-modern/src/components/Alerts/ThresholdsTableDockerHostsSection.tsx,
and frontend-modern/src/components/Alerts/ThresholdsTableDockerContainersSection.tsx.
The containers thresholds surface must consume canonical app-container
parents through the shared alert-overrides state rather than assuming
docker-host is the only runtime shape. API-backed TrueNAS parents belong in
the same Container Runtimes / Containers surface, while Docker-specific
controls like ignored prefixes and Swarm service gap settings must stay gated
to real Docker runtimes instead of leaking onto non-Docker platforms.
At the current support floor, TrueNAS alert support means the shared alert
surfaces can evaluate, show, and drill into incidents on TrueNAS-backed
systems, disks, and app parents using the canonical resource model and related
links into infrastructure, workloads, storage, and recovery. Pulse does not
promise a TrueNAS-only alert workflow or provider-specific alert management
surface beyond the shared alerts product.
At the current locked VMware floor, alert support must mean the same shared
alert surfaces can evaluate, show, and drill into vSphere alarm, health, and
threshold signals on canonical agent, vm, storage, and network
resources, with related event/task context routed through the shared incident
and resource links. This is the alerts support floor for admitted vSphere
resources; it does not by itself promote the broader VMware platform readiness
state beyond the separately governed platform-admission floor. Pulse must not
grow a VMware-only alert shell, alarm editor, or direct alarm-control surface
in phase 1.
That same VMware alert rule now also includes the topology boundary. Alarm
context that originates on a datacenter, cluster, folder, or resource pool may
inform a shared incident, but it must still resolve onto canonical agent,
vm, storage, or network investigation paths rather than creating synthetic
top-level VMware incident resources. If that attachment cannot be done
honestly for a given signal, the signal should remain supporting context
instead of inflating the support claim.
That same VMware alert rule now also includes the timeline boundary. Related
VMware event and task context may enrich shared alert and incident views, but
it must do so through the canonical incident and resource-history paths rather
than through a VMware-only history browser, event drill-down route, or alarm
management shell.
Future Docker thresholds presentation work should extend those section owners
instead of expanding frontend-modern/src/components/Alerts/ThresholdsTableDockerTab.tsx
back into a mixed render surface.
The systems tab now follows that same shell-versus-section pattern through
frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx
and frontend-modern/src/components/Alerts/ThresholdsTableAgentDisksSection.tsx.
Future systems-thresholds presentation work should extend those section owners
instead of expanding frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx
back into a mixed render surface.
The alert resource thresholds editor now follows the same shape: shared metric
normalization, bounds, value-resolution, and override-label logic live in
frontend-modern/src/components/Alerts/alertResourceTableModel.ts, shared group
header presentation lives in
frontend-modern/src/components/Alerts/AlertResourceGroupHeader.tsx, desktop
table ownership lives in
frontend-modern/src/components/Alerts/AlertResourceTableDesktop.tsx, mobile
card ownership lives in
frontend-modern/src/components/Alerts/AlertResourceTableMobile.tsx, render-heavy
desktop row ownership lives in
frontend-modern/src/components/Alerts/AlertResourceTableRow.tsx, and selection
state, delay-row toggling, and inline metric-input focus live in
frontend-modern/src/components/Alerts/useAlertResourceTableState.ts. Shared
web-interface launch affordances inside alert resource rows and grouped agent
headers must compose
frontend-modern/src/components/shared/WebInterfaceLink.tsx; alerts own only
the alert/resource data and URL availability decision, not URL safety
classification, the adjacent launch-control shell, new-tab safety attributes,
row-click containment, invalid-URL warnings, or accessible launch-label
semantics. Alert resource names remain inert row identity so opening a saved
web interface cannot also toggle selection or the thresholds editor.
Resource-table empty states, badge labels, offline-state wording, note
placeholders, and metric input titles now route through
frontend-modern/src/utils/alertResourceTablePresentation.ts instead of
remaining duplicated across the desktop and mobile thresholds surfaces.
frontend-modern/src/components/Alerts/ResourceTable.tsx is now limited to the
shell boundary for breakpoint selection and bulk-edit composition. Future
resource-table threshold semantics should land in those owners instead of
being rebuilt inline in the shell.
Alert incident timeline event cards now route through
frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx,
while their meta-row, heading, detail, command, and output typography still
route through frontend-modern/src/utils/alertIncidentPresentation.ts
instead of keeping duplicate timeline card structure inline in the alerts
page and overview timelines.
Expanded alert incident detail now also routes through
frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx and
frontend-modern/src/components/Alerts/IncidentEventFilters.tsx so the
overview surface and the history table share the same loading/error states,
canonical timeline meta row, note editor, and event-filter controls instead
of maintaining two independent incident-detail implementations.
That shared timeline runtime state now routes through
frontend-modern/src/features/alerts/useAlertIncidentTimelineState.ts, which
owns incident timeline fetch, expansion state, note-save flow, and shared
event-filter state for both frontend-modern/src/features/alerts/OverviewTab.tsx
and frontend-modern/src/features/alerts/tabs/HistoryTab.tsx. Future incident
timeline control flow should land in that feature hook instead of being forked
back into either alert surface. Alert incident timeline handoffs into Pulse
Assistant are now owned by the Alerts incident handoff model and carry only
sanitized incident facts plus event summaries into both the visible drawer
briefing and the backend model-only handoff context; raw command and output
details stay in the incident timeline or approval surface.
The shared hook may expose Timeline for a displayed mock alert because mock
mode now guarantees an occurrence-qualified incident through the same typed
API. A mock fixture missing that incident is a broken alert read-model contract,
not an empty-state case for the frontend to disguise; genuine unknown real
occurrences may still render the existing no-timeline state.
Resource incident panel cards, summary rows, and toggle-button presentation
now also route through frontend-modern/src/utils/alertIncidentPresentation.ts
instead of remaining inline inside frontend-modern/src/pages/Alerts.tsx.
That same resource incident panel now treats collapsed incident activity as a canonical alert read-model summary rather than a page-local sentence. The collapsed row must summarize filtered incident events by canonical event type order and reuse the shared event-card renderer for expanded incident detail, so the alert history page does not drift away from the overview timeline when canonical lifecycle or remediation events are added.
Active alert card state, acknowledged badge, and primary/secondary action
button presentation now route through
frontend-modern/src/utils/alertOverviewPresentation.ts instead of remaining
inline in frontend-modern/src/features/alerts/OverviewTab.tsx.
The canonical shared alert-acknowledgement runtime owner is now
frontend-modern/src/features/alerts/useAlertAcknowledgementState.ts, which
owns optimistic single/bulk acknowledge control flow, restore behavior, and
notification feedback for frontend-modern/src/features/alerts/useAlertOverviewState.ts
and the alert overview render shells.
frontend-modern/src/features/alerts/useAlertOverviewState.ts now owns the
derived alert read-model and Last 24 Hours stat refresh for
frontend-modern/src/features/alerts/OverviewTab.tsx, while composing that
shared acknowledgement owner instead of keeping its own alert mutation fork.
Future overview action behavior should extend that shared acknowledgement hook
instead of putting acknowledge mutations back into render shells.
Global and group acknowledgement read the bulk endpoint's per-alert results
through one path, so a group action reports the alerts it could not
acknowledge with the same count notice as the global action, beside the
success count of a partial result. Request ownership is app-wide: the alert
store creates frontend-modern/src/stores/alertAcknowledgementRequests.ts,
which gives each alert to at most one unsettled card, restore, group or
global request. Later card, group and global actions leave an owned alert to
that request, which reports its outcome. A card shows every alert an
unsettled request carries as processing, a global request's included, and
keeps the 1.5-second hold only after a card or group request settles, so a
failed group or global request can be retried at once. The cards' processing
state and the global button's progress derive from the owner: a remounted
overview shows a request its predecessor sent, and an older request's hold
cannot mark an alert idle while a newer request carries it. The owner resets
with the store's alert scope (the organization switch through switchUrl,
shutdown and store cleanup). Its pending acknowledgement confirmation guards
and their 15-second timers end at that same boundary, so a settled local change
cannot suppress another scope's alert with a reused identifier or emit a stale
resync notice there. A same-scope reconnect retains both the request owner and
confirmation guards. A response from an ended scope writes nothing
to the store and shows nothing, since both belong to the next scope, whose
alerts can share its identifiers; a rejection is still logged. Within one
scope a response still reports once after the overview unmounts. The jsdom
no-op store keeps its own owner, reset by its shutdown, and
shutdownGlobalWebSocketStore ends that shared app for a test as page unload
ends it for the app. alertAcknowledgementRequests.test.ts,
websocket.alertAcknowledgementRequests.test.ts,
websocket-global.noopLifetime.test.ts, useAlertAcknowledgementState.test.tsx
and OverviewTab.acknowledgementbusy.test.tsx pin these paths.
Overview ages and bands read the shared clock. AlertOverviewAlertCard.tsx
passes its own useRelativeTimeNow reading as now to
formatAlertOverviewStartedAgo and getMetricAlertPresentation, and the Last
24 Hours count in useAlertOverviewState.ts reads the same clock, so all three
measure from the wall clock. The hook no longer exposes a minute tick: read
as "now", that tick measured a card mounted between ticks from up to a minute
earlier, so an alert raised 70 seconds ago read "this minute", an alert
raised since the last tick stayed out of the 24h count until the next one, a
reading already past the 10-minute stale cut-off led the card as live, and the
stale flip and 24h drop landed up to a minute late. A mounted card now turns stale,
and an alert leaves the 24h count, within one 30-second tick with no data
change. The delivery diagnoses refresh is a server read and keeps its own
minute interval rather than following the 30-second clock.
AlertOverviewAlertCard.clock.test.tsx pins the mid-tick mount, the stale flip
and the 24h drop under fake timers, and useAlertOverviewState.test.tsx pins
the diagnoses cadence.
Render-heavy alert overview ownership now routes through
frontend-modern/src/features/alerts/AlertOverviewStatsCards.tsx,
frontend-modern/src/features/alerts/AlertOverviewActiveAlertsSection.tsx,
and frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx instead
of rebuilding stats-card, active-alert, and timeline-card presentation inline
inside frontend-modern/src/features/alerts/OverviewTab.tsx.
AlertOverviewAlertCard.tsx also owns the compact delivery-status placement,
while deliveryDiagnosisPresentation.ts owns the reason-to-copy and tone
mapping. Acknowledged cards keep their canonical badge and omit that redundant
line; held delivery states that can surprise an operator use the attention
tone, while cooldown, quiet-hours, monitor-only, and successful/pending states
remain neutral.
The card's resource link resolves the owning platform page from alert
metadata, never from message wording alone. Provider incidents land on
canonical vm, storage, network, and agent resource types rather than a
platform-prefixed type, so the card reads the platform from an explicit
platformType, then the incident's incidentProvider, then the resource's
resourceSources, all through resolvePlatformTypeFromSources in
frontend-modern/src/utils/sourcePlatforms.ts: a vCenter alarm whose message
is only the alarm's name links to the vSphere overview, a TrueNAS pool
incident to TrueNAS, and a PBS incident that Pulse itself raised (provider
pulse) to Proxmox by its pbs source. The resolved platform maps to its
page through PRIMARY_PLATFORM_NAV_SCOPE_IDS and the routing/resourceLinks.ts
path builders, and the same canonical platform id goes to the Monitoring
policy menu, so the link and the named inventory owner cannot disagree. Only
an alert whose metadata names no platform falls back to its resource type
(agent, docker-*, app-container, k8s-*, truenas-*, vmware-*)
and finally the message; agent alerts open /standalone/machines, because no
/machines route exists. A resource-id prefix or substring and the
message may pick the page only when the alert names no resource type at all,
because Proxmox guest ids embed the user's cluster and node names and messages
name the guest: VM alerts on node docker-01, in a cluster named agent, or
for a guest named vmware-test still link to Proxmox and name Proxmox as their
owner. Metric alerts put a display label in resourceType (vSphere VM,
Kubernetes Pod), so the unified metric and capacity evaluators
(internal/alerts/unified_eval.go, internal/alerts/capacity_forecast.go)
stamp the canonical platformType beside it, and connection-degraded alerts
(internal/alerts/connection.go) stamp it from the connection type; the card
reads that ahead of every fallback. The menu's owner copy comes from
frontend-modern/src/utils/resourceMonitoringPolicy.ts, where an explicit
platform outranks the resource-type guess and the agent-removal copy is kept
to the agent's own machine: libvirt VMs and Unraid arrays in resource drawers
get the neutral source-system copy. Host-agent alerts
(internal/alerts/host.go) stamp platformType agent, and only the
machine's own alerts keep resourceType agent: filesystem,
disk-temperature and SMART alerts carry agent-disk, RAID and Unraid array
alerts agent-storage, and custom-sensor alerts agent-sensor, because the
menu writes policy to their child resource id, never to the machine. Each
child type keeps agent among its CanonicalResourceTypeKeys, and
configuration re-evaluation treats agent policy as final for host-agent
alerts (TrueNAS systems and vSphere hosts keep their platform policy), so the
storage and guest switches cannot resolve a SMART, RAID or sensor alert that
the next agent report would re-raise. A metric alert held between its clear
and trigger thresholds still refreshes resourceType and platformType from
its producer while its value, message and last-seen time stay at the last
breach, so a restored alert that is still firing or held picks up the current
classification at its next evaluation; one that resolves on that evaluation
keeps the stored classification in its resolved record. Menu sentences open
with the owner label capitalised.
The retired dashboard recent-alert panel must not be reintroduced as a
parallel alert surface. Alert summary/tone copy belongs to the alert overview
presentation owner, and any future compact alert surface must compose the
shared alert read-model and acknowledgement hook rather than creating a
dashboard-only panel.
The Alerts Overview description must stay monitor-first: it names active
incidents and current coverage across monitored resources, and must not imply
that the overview itself owns installation-wide alert activation controls.
The exported English fallback in alertOverviewPresentation.ts, localized
message catalog, and header metadata proof must remain textually aligned.
The Schedule tab's header title is "Delivery Schedule" (ALERTS_PAGE_SCHEDULE_TITLE,
alerts.page.schedule.title; de "Zustellplan", es "Programacion de entregas").
The page sets how alerts are delivered (quiet hours, cooldown, grouping,
delivery channels, recovery and escalation); it was titled "Maintenance
Schedule", which promised maintenance windows, a separate per-resource
feature this page does not hold (alertOverviewPresentation.test.ts,
i18n.test.ts).
Alert threshold and schedule surfaces must now also treat
discoveryTarget as optional frontend input and keep grouping-card state on
the canonical node group-header contract. Frontend alert pages may not
assume discovery metadata is always present when deriving override IDs or
toggle styling.
The alerts page shell in frontend-modern/src/pages/Alerts.tsx must now keep
destinations, history, schedule, and thresholds rendering feature-owned under
frontend-modern/src/features/alerts/tabs/. New alert tab surfaces should be
extracted as feature modules instead of remaining page-local function blocks,
so the page owns navigation and cross-surface routing while tab files own their
runtime presentation, tab-local interaction logic, and any history-table
presentation or thresholds-table adapter logic that does not belong in a shared
primitive.
The history tab itself now follows the same shell-versus-runtime rule. The
canonical history runtime owner is
frontend-modern/src/features/alerts/useAlertHistoryState.ts, which now owns
alert-history fetch, persistent filter state, history-clear flow, and
composition of the derived history owners. Resource-incident panel loading,
refresh, and expansion state now live in
frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts, while
the pure analytics model for history-item projection, trend buckets, group
labels, axis ticks, and selected bucket detail now lives in
frontend-modern/src/features/alerts/alertHistoryModel.ts. The tab shell in
frontend-modern/src/features/alerts/tabs/HistoryTab.tsx now composes
frontend-modern/src/features/alerts/AlertHistoryFrequencyCard.tsx,
frontend-modern/src/features/alerts/AlertHistoryFiltersCard.tsx,
frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx,
frontend-modern/src/features/alerts/AlertHistoryTableSection.tsx,
frontend-modern/src/features/alerts/AlertHistoryTableGroupRow.tsx,
frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx, and
frontend-modern/src/features/alerts/AlertHistoryAdministrationCard.tsx.
Future alert-history control-flow work should extend the feature hook, new
grouping or trend semantics should extend the history model, and render-heavy
history surfaces should extend those section owners instead of putting fetch,
resource-incident state, or table rendering back into the shell.
Alert History toolbar state is one route-owned unit. Search, period, and
severity are read and written by useAlertHistoryState.ts; its active-filter
count includes search, and AlertHistoryFiltersCard.tsx delegates contextual
Clear filters to one composite hook mutation instead of letting FilterBar
issue sequential route writes. That reset removes all three query parameters
in one navigation and clears any transient chart-bucket selection, so a
search-only result set remains visibly resettable and an older URL write
cannot resurrect another filter.
That same history surface also owns the resource-incident panel, and the panel
links nowhere. frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx
names the selected resource through the unified-resource lookup it is given
(getResource, falling back to the history state's lookup) and
getPreferredInfrastructureDisplayName, and shows the resource name the
selected history row carried while no unified record matches. It imports nothing from
frontend-modern/src/routing/resourceLinks.ts, and it offers no resource
cross-jump links: investigation continues in place through the incident
timeline cards, as the retired cross-jump chip note above describes. Future incident-link work
must not reintroduce a chip set, local infrastructure-link assembly, or
provider-local route strings inside the alert feature shell;
frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts pins the panel
free of buildResolvedResourceSurfaceLinks, buildResourceSurfaceLinksForResource,
and buildInfrastructureResourceLink.
Alert configuration load/save state, notification config reloads, and threshold
override normalization now route through
frontend-modern/src/features/alerts/AlertsConfigurationSurface.tsx instead of
living inline in frontend-modern/src/pages/Alerts.tsx. The page shell owns
navigation, activation chrome, and cross-surface routing; the configuration
surface is now a shell that composes the destinations, schedule, and thresholds
tabs. The canonical alert-policy runtime owner is now
frontend-modern/src/features/alerts/useAlertsConfigurationState.ts for
config transport, notification-config reloads, and save/load orchestration,
frontend-modern/src/features/alerts/useAlertsConfigurationSnapshotState.ts
for default-backed mutable config snapshot state plus apply/capture/reset
ownership,
frontend-modern/src/features/alerts/alertsConfigurationModel.ts for backend
config normalization, factory defaults, docker-gap validation, and save-payload
serialization,
frontend-modern/src/features/alerts/alertOverridesModel.ts for raw override
normalization plus resource-backed override projection. That override owner
must canonicalize legacy per-node shared-storage keys and hashed storage
resource ids onto the storage metrics target id before the thresholds surface
rebuilds, so old Ceph/shared-datastore overrides and newly projected Ceph pool
overrides still surface on the live v6 editor instead of disappearing after
the feature-shell migration, and
frontend-modern/src/features/alerts/useAlertOverridesState.ts for reactive
override state, derived resource lists, and overview handoff, and
frontend-modern/src/features/alerts/alertDestinationsModel.ts for email and
Apprise config normalization plus outbound payload shaping, and
frontend-modern/src/features/alerts/useAlertDestinationsState.ts for
notification destination reload and persistence orchestration.
frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts now
owns webhook load/mutate/test flow,
frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts now owns
destination test actions, retry orchestration, and notification delivery-health
loading around that webhook runtime,
while
frontend-modern/src/features/alerts/tabs/DestinationsTab.tsx stays the
destinations render shell and composes
frontend-modern/src/features/alerts/AlertDeliveryHealthCard.tsx,
frontend-modern/src/features/alerts/AlertEmailDestinationsSection.tsx,
frontend-modern/src/features/alerts/AlertAppriseDestinationsSection.tsx,
frontend-modern/src/features/alerts/AlertWebhookDestinationsSection.tsx, and
the dedicated load/error wrappers. The delivery card must warn for retained
terminal failures, present an unavailable queue-health read as unverified
rather than healthy, and give the operator a refresh action plus configuration
and test guidance. Its warning copy must explain that expired delivery records
are removed hourly and that, absent another terminal failure, the warning
clears only after the last retained record reaches its configured retention
limit. Recoverable retry attempts must not produce this warning.
frontend-modern/src/features/alerts/AlertDeliveryHealthCard.test.tsx and
frontend-modern/src/features/alerts/__tests__/useAlertDestinationsTabState.test.tsx
are the focused render and runtime proofs for that operator surface.
Future config cleanup should extend the
config transport hook, the config model, the override-projection hook, or the
shared frontend-modern/src/utils/alertDestinationsPresentation.ts helper for
customer-facing destinations copy instead of reviving inline retry, test, and
error text across the feature tabs. Runtime cleanup should extend the
destinations hook owned by the subsystem that carries the behavior instead of
letting the broader configuration hook absorb all four concerns again.
Local Apprise delivery is a shipped server-runtime capability, not an
operator-installed optional. The shared presentation helper must show the
Telegram chat_id:topic form, the notification backend must preserve each
target as one CLI argument, and the deployment-installability contract must
keep the pinned apprise executable available in every Pulse server image.
The email destination provider picker now follows that same split:
frontend-modern/src/components/Alerts/useEmailProviderSelectState.ts owns
provider-catalog loading and provider-default application, while
frontend-modern/src/components/Alerts/EmailProviderSelect.tsx stays the
render shell and consumes the canonical UIEmailConfig feature type instead of
keeping a second local email-config interface.
The alert scheduling surface now follows the same shell/section split:
frontend-modern/src/features/alerts/useAlertScheduleState.ts owns schedule
runtime and default/reset policy, while
frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx stays the shell and
composes the dedicated quiet-hours, cooldown, grouping, recovery, escalation,
and summary section owners instead of carrying those panels inline.
Alert filter metadata and grouped header consumers must also preserve the
canonical agent and node header boundary when reusing shared filter
primitives. Frontend alert tables may not drift back to ad hoc host-key
grouping or narrow filter key predicates that drop optional hostname values
before alert group metadata is derived.
That same shared alert boundary now also owns provider-backed resource-incident
alerts beyond storage-only cases. internal/alerts/model.go,
internal/alerts/unified_incidents.go, and
frontend-modern/src/utils/alertIncidentPresentation.ts must treat VMware-
backed host and VM incidents as the same canonical resource-incident
vocabulary used everywhere else, with quiet-hours routing derived from the
shared incident category and provider context carried only as shared alert and
timeline metadata. Alert history may surface VMware alarm, task, and snapshot
context inside that shared model, but it must not fork into VMware-only alert
types, badges, or incident chrome.
That same alert-shell boundary now also treats websocket access as a shared
app-runtime dependency rather than an alerts-owned provider. Alert shells such
as frontend-modern/src/pages/Alerts.tsx and
frontend-modern/src/features/alerts/tabs/HistoryTab.tsx may consume live
state only through frontend-modern/src/contexts/appRuntime.ts; they must not
import @/App or create reverse dependencies into the root shell chunk,
because alerts surfaces must remain lazy-load safe and must not blank the app
before auth/bootstrap finishes.
Alert commercial handoffs now also follow the runtime-versus-commercial split.
frontend-modern/src/components/Alerts/InvestigateAlertButton.tsx may resolve
upgrade destinations through the shared commercial-posture store, but alert
runtime availability and chat enablement must stay governed by the
non-commercial app runtime and assistant state instead of reusing the same
commercial payload as feature truth.
Alert schedule and incident-timeline surfaces now also keep their browser state
typed through one feature-owned contract. Quiet-hour suppress options must be
cloned into mutable feature props before crossing section boundaries, quiet-day
callbacks must preserve the canonical weekday key union, and incident timeline
expansion/note-saving state must remain Set<string>-owned instead of drifting
to untyped browser-local collections.
That same alerts runtime boundary also owns canonical identity derivation and
active-alert persistence. Shared canonical identity helpers may infer resource,
spec, and state ids from legacy alerts, but they must do so without mutating
live in-memory alert instances unless the caller explicitly backfills that
state. Persisted active-alert snapshots must therefore clone alerts under lock,
backfill canonical identity on the clone, replace the persisted public ID with
the canonical state ID, and serialize that snapshot instead of mutating the
live alert map during async saves or incident rebuilds. Restore must preserve
acknowledgement fields and rewrite a successfully derived legacy snapshot
atomically after the in-memory restore; unrecognized records remain readable.
That same ownership also governs acknowledgement and manual-clear cleanup.
Clearing an alert through the canonical alerts runtime must remove both legacy
public-id tracking and canonical-state acknowledgement records so old aliases
cannot keep an alert acknowledged after the canonical alert has been removed.
Operational Trust writable lifecycle
internal/alerts/active_lifecycle.go is the single writable owner for
acknowledge, unacknowledge, suppress, unsuppress, collection-stale,
collection-unknown, resolving, and collection-restored transitions. Every
writer is idempotent under retry, retains new evidence without duplicating a
same-state transition, persists through the active-alert store, and survives
restart. Explicit stale, unknown, resolving, or suppressed state takes
precedence over the legacy Alert.Acknowledged projection. The canonical
record's LastObservedAt is the maximum of retained record, alert, and evidence
timestamps so a fresh outage observation cannot be rolled back by an older
legacy alert timestamp.
Suppression is bounded and reasoned, leaves the default active queue, and
remains inspectable. Expiry or explicit unsuppression returns the record to its
detector-owned state; it never resolves it. Only fresh sufficient recovery
evidence may enter resolving, and only detector recovery, or a handover
recorded as ownership_transferred with its closing evidence, may resolve.
Per-alert snooze is the customer-facing bounded suppression contract, not a
page-local timer. SnoozeAlert writes the same canonical operational record,
requires a future expiry no more than 30 days away, and appends a durable
snoozed lifecycle snapshot. While active, notification dispatch, recovery
delivery, delivery diagnosis, and escalation all consult that one record.
Expiry runs as an explicit unsnoozed transition and restores acknowledged
or open according to the underlying alert without resolving the incident.
Skipped escalation levels are not replayed in a burst: the remaining schedule
resumes from the unsnooze transition while the incident's original start time
and duration remain intact. The alerts overview exposes preset bounded holds,
the exact local expiry, and an explicit Resume action over this API-owned state.
Canonical threshold-override succession
Alert threshold overrides are persisted under the current canonical resource
ID. internal/alerts/canonical_override_migration.go may re-home an override
from a retired ID only when the unified-resource owner explicitly publishes
that ID in SupersededCanonicalIDs. Display aliases, hostnames, metric
targets, connection labels, and other lookup conveniences are not persistence
authority. A still-live old ID or one old ID claimed by multiple current
resources fails closed and remains untouched; when both keys exist, the current
canonical override wins.
Monitoring owns the synchronization point that applies this migration to the active alert configuration and persists it before later reloads. The frontend may read legacy connection-target keys as compatibility candidates, but save, edit, and delete operations target the resource's current canonical ID and retain that ID across refetches or reported-identity changes. This migration changes alert configuration only and does not create customer-infrastructure mutation authority outside canonical Actions.
internal/alerts/canonical_override_migration_test.go proves the
unambiguous/live/ambiguous decision matrix,
internal/monitoring/monitor_alert_override_migration_test.go proves durable
reload behavior, and
frontend-modern/src/features/alerts/thresholds/hooks/__tests__/truenasThresholdPersistence.test.tsx
proves the browser-side TrueNAS save/refetch contract.
Versioned persisted alert identity
AlertConfig.identitySchemaVersion is the additive persisted identity marker;
the current schema is version 1. PlanAlertIdentityMigration is the mandatory
dry-run boundary: it operates on a copied config and reports source/target
versions, removed and added keys, deferred rows, and unsupported future
versions before any write. ApplyAlertIdentityMigration may install that exact
plan only while its source version is still current. The monitoring bridge must
persist the result before updating the live manager.
Version 1 folds provider-declared canonical succession, Docker container
recreation IDs, Proxmox guest and guest-disk aliases, and live storage aliases
onto the single write identity already used by the editor/evaluator. A current
row wins over retired duplicates. Multiple legacy rows claiming one target,
ambiguous live ownership, unknown rows, and temporarily absent resources fail
closed and stay in alerts.json for a later resource snapshot that can prove
the mapping. Alias readers remain compatibility inputs for those deferred rows,
not persistence authority. Storage alias provenance is carried on the unified
resource StorageMeta so the plan does not reconstruct ownership from display
names alone.
The marker is rollback-safe: older binaries ignore the additive JSON field and
may omit it on a later save; after re-upgrade the idempotent planner runs again.
Newer binaries preserve an already-held marker when a stale browser payload
omits it, while the alert editor explicitly round-trips the marker. A config
whose marker is newer than the running binary is never rewritten. Regression
ownership is internal/alerts/alert_identity_migration_test.go,
internal/monitoring/monitor_alert_override_migration_test.go, and
frontend-modern/src/features/alerts/__tests__/alertsConfigurationModel.snapshot.test.ts.
Alert config snapshots are owned copies
Manager.GetConfig() returns AlertConfig.Clone(), a typed deep copy that
shares no map, slice or pointer with the live config (custom rule filter
values are copied as decoded JSON values). Callers encode, persist
and edit that snapshot outside m.mu: the GET config handler, alert
activation, the identity migration, backup evaluation and diagnostics. It used
to be a shallow copy, so SaveAlertConfig normalizing it in place, or
UpdateConfig re-normalizing the overrides of a re-applied snapshot, wrote the
live maps while evaluation or another encoder read them, and Go aborted Pulse
with a concurrent map write. SaveAlertConfig also clones before normalizing,
because a config just handed to UpdateConfig is the live config. Persistence
keeps an explicit 0 type-level delay as no delay on save and load, matching
NormalizeTimeThresholds. It used to store and load it as the 5-second factory
delay, which the noisy-gauge rule stretches to 300 seconds for memory and
temperature, and the shared map write was what made the running manager agree.
Pulse saves never wrote a 0, so only a newly sent or hand-edited 0 changes
meaning. internal/alerts/config/clone_test.go
fills every field and fails when a new reference field is not copied.
Regression ownership is internal/alerts/resolved_lock_discipline_test.go and
internal/config/persistence_alert_ownership_test.go.
Alert evaluation owns the thresholds it normalizes
Host, guest and Docker container evaluation copy what they read from
m.config while holding m.mu, and only then fill in defaults.
CheckHost and CheckGuest clone a per-disk
override's Disk threshold before ensureHysteresisThreshold sets a missing
Clear to the trigger minus 5. They used to normalize the live override's
pointer after unlocking, so a GetConfig() clone under the read lock raced
the write, and the saved override changed to whatever the last poll filled in.
The live override now keeps the clear it was saved with, and the config API
returns that value. Docker container evaluation resolves its thresholds from
cloned DockerDefaults plus the override under the same lock, and reads the
restart-loop and memory-limit settings under the lock. It used to point into
m.config.DockerDefaults with no lock while UpdateConfig replaced
m.config. The test-only getThresholdForMetricFromConfig, which normalized
in place, is gone. internal/alerts/alerts_test.go
TestAlertEvaluationLeavesTheLiveConfigToTheLock runs host, guest and Docker
evaluation against GetConfig() and UpdateConfig(GetConfig()) under
-race.
Config-save reevaluation judges a filesystem alert by its own threshold
UpdateConfig re-judges every active alert, and a filesystem usage alert's
threshold resolves through the same per-filesystem helper its evaluator uses.
hostDiskUsageThresholdNoLock resolves an agent filesystem
(agent:<host>/disk:<label>): its own override, then the DiskFillByType
threshold for a hardware type inferred from the device while the host's disk
alerting is on, then the host's disk threshold. guestDiskUsageThresholdNoLock
resolves a guest filesystem (<guestID>-disk-<key>): its guest-disk:
override, then the guest's disk threshold. CheckHost and CheckGuest call
them under the read lock, and resolveHostAlertThresholdsNoLock and
resolveGuestAlertThresholdsNoLock call them for reevaluation. Both return an
override or per-type threshold as an owned copy. A guest filesystem alert
resolves its guest's threshold, override and identity-based custom rules by the
canonical instance:node:vmid guest ID that parseGuestAlertIdentity reads
from its resource ID. Custom rules that filter on live metrics still see no
readings there, as for every reevaluated guest alert. Reevaluation used to
judge agent filesystems by the host threshold, and guest filesystems without
their guest-disk: override or the guest's own override, because the
filesystem's resource ID names no guest. Any settings save resolved an alert a
per-disk threshold of 70 raised at 80 under a default of 90, notifying
recovery, and the next poll raised it again as a new alert.
internal/alerts/threshold_resolution_shared_test.go
TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold covers an agent
filesystem override, an NVMe fill threshold, a guest filesystem override and a
guest override: an unchanged save must keep the alert, and a save lifting the
trigger above the reading must resolve it.
Versioned alert-intent policy
The alerts runtime owns one versioned alert-intent document and its durable
pending-condition state. Stable signal keys are *, state.offline,
incident.availability, and metric.<name>. Effective fields resolve
independently from legacy metric behavior, document defaults, resource-type
overrides, and canonical-resource overrides, in that order. Keys are
normalized before validation and collisions fail closed; updates use revision
compare-and-swap so a stale browser cannot overwrite a newer document.
Intent affects when detector truth becomes eligible for an active alert; it does not mutate the underlying observation or create a second alert identity. Without an explicit applicable rule, established alert behavior remains compatible. With a rule, the first matched time is durable across restart and becomes the canonical alert start time once eligible. Preview evaluates the same resolver but restores pending state before returning, so it is read-only. Invalid documents, persistence failures, or revision conflicts must leave the prior in-memory and durable policy active.
Resource-type inheritance is field-wise from general to specific before the
canonical-resource rule: for example guest supplies a VM/LXC default and
vm or system-container may replace individual fields. A guest policy must
not leak into node or agent connectivity. Omitted grace preserves inherited or
factory behavior, explicit zero is a deliberate no-wait policy, and a positive
powered-off grace is duration authority rather than a poll-count threshold.
The no-policy powered-state path retains the legacy two-confirmation contract
for migration compatibility; an applicable powered-state duration activates
on the first stopped observation at or after elapsed eligibility.
Runtime eligibility uses server receipt time plus process-monotonic elapsed progress, never client/provider timestamps or poll counts. Duplicate reports advance no time, delayed polls count actual elapsed time, and wall-clock forward/backward changes do not change the decision. Pending state persists accumulated elapsed progress. Restart re-baselines the monotonic clock without counting unobserved process downtime, while legacy timestamp-only pending files are imported once into the elapsed representation. Recovery, explicit guest suppression, disabling offline alerts, and tracking cleanup remove both durable pending state and its transient monotonic baseline before a later outage can start.
Operator maintenance and intentionally-offline state are read only through the
canonical unified-resource identity. One-shot and recurring maintenance use
the same effective occurrence boundary. Exact-resource maintenance always
applies; ancestor maintenance applies only when its persisted scope is
resource_and_descendants. Overlapping exact and inherited windows remain
suppressed until the latest active end, and every operator-state mutation
reconciles the active-alert set immediately so a host window cannot leave child
alerts visible until another detector write. Backup-aware offline deferral consumes
fresh, matching, active task evidence, applies the configured post-backup grace,
and always terminates at its hard cap. Missing, stale, future-skewed,
finished, or mismatched backup evidence cannot suppress an outage. This policy
changes alert activation only: notification delivery, recovery assurance, and
customer-infrastructure mutation retain their existing owners.
internal/alerts/intent_policy_test.go and
internal/alerts/powered_off_tolerance_test.go prove field-wise type
precedence, normalization, exact zero/positive duration boundaries, VM/LXC
coverage, duplicate and delayed reports, wall-clock changes, suppression
reset, migration/restart continuity, backup hard-cap behavior, preview
immutability, and first-match lifecycle identity.
The thresholds resource rows expose that same policy owner directly through a
per-resource alert-delay action. The action carries the canonical resource id
and the first supported CPU, memory, or disk signal into the versioned
intent-policy editor, expands it, and leaves persistence and inheritance with
/api/alerts/intent-policies. Resources without a supported metric open on
state.offline. Threshold rows must not create a parallel delay field in the
legacy threshold override document or save intent changes through the
threshold-config endpoint.
Provider-observed storage and workload incident lifecycle
Provider observations use one stable alert identity composed from provider, connection-local resource identity, native signal identity, and canonical incident code. Synthetic TrueNAS pool, vdev, dataset, disk, app, and container conditions require two consecutive observations before activation and two consecutive healthy observations before recovery. A transient poll therefore does not fire or clear an alert. Active state, acknowledgement, suppression, severity changes, recovery, and history remain owned by the shared alerts runtime and survive restart; an absent resource or unknown collection result is not recovery evidence.
Native TrueNAS alerts remain authoritative for their equivalent condition. Their stable native ID suppresses only the duplicate synthetic pool/vdev state, missing-member, or scan condition; distinct read/write/checksum evidence continues to participate. Pulse does not send a second TrueNAS email stream. Intentional stopped apps and other stable conditions remain suppressible through the canonical per-resource or global alert configuration, and readonly replication targets do not produce incidents when native replication evidence classifies them as healthy.
Ceph contributes one cluster-scoped provider incident only for native
HEALTH_WARN or HEALTH_ERR evidence. It shares the same confirmation,
recovery, history, and suppression machinery but cannot manufacture a disk
incident from cluster health.
internal/alerts/unified_incident_confirmation_test.go,
internal/truenas/provider_pool_health_contract_test.go, and
internal/unifiedresources/ceph_pool_health_contract_test.go prove the
lifecycle and deduplication matrix.
Per-resource monitoring policy is an alert-runtime invariant
The canonical unified-resource operator state is honored without requiring an
explicit versioned alert-intent rule. Factory behavior enables operator-state
evaluation, and a signal rule cannot weaken a persisted resource monitoring or
lifecycle policy. expected_offline suppresses only state.offline and
incident.availability; muted, retired, and active maintenance suppress all
signals. Unknown alert families fail safe to the default signal and cannot be
hidden by expected-offline.
Every active-alert writer passes through the same operator-state gate, and
notification delivery consults the same decision before quiet-hours policy.
The active-store writer reports admission explicitly. Detectors must not add
history, recent-alert state, fired events, rate-limit consumption, or delivery
intent after a rejected write. Lifecycle evaluators report suppression without
an activation transition. Restore and guest-identity migration must not revive
acknowledgement or tracking state for a rejected alert. Repeated observations,
policy removal or expiry, and post-mutation reconciliation are exercised across
provider incidents, metric and canonical lifecycle writers in
internal/alerts/alert_admission_test.go.
Operator suppression is never converted into a quiet-hours replay, including
for recovery notifications carrying stale replay metadata. The internal alert
dispatch path applies this terminal decision even for an already-active alert
between a policy save and its reconciliation: it neither invokes a delivery
callback nor marks the alert notified, and it removes stale replay metadata.
The read-only delivery diagnosis reports the operator reason as suppressed,
not as a quiet-hours deferral. TestOperatorPolicySuppressesDirectAlertDispatchWithoutQueueReplay
pins muted, expected-offline, maintenance and unaffected performance cases,
including the immutable suppression-versus-deferral event distinction.
After a persisted policy mutation, ReconcileResourceOperatorState resolves
already-active records for that resource immediately; later detector writes
cannot recreate them while suppression remains active. Existing prefix, tag,
and pulse-no-alerts bulk rules remain compatible inputs and do not create a
second per-resource state store. internal/alerts/intent_policy_test.go pins
factory operator-state evaluation, writer gating, and active reconciliation.
Object drawers expose the active problem across identity aliases
An object drawer with active alerts must show the exact problem in its default Overview, not only an alert accent or aggregate status already present in the table row. Frontend matching accepts a bounded ordered set containing the canonical resource id and provider-authored, metrics-target, alias, and superseded identities. The same candidate set drives row alert decoration and drawer message selection so opening a highlighted object cannot lose the alert because the provider stream and unified-resource projection use different keys. This is a read-only projection of the existing active-alert map; it does not create alert truth, change acknowledgement, or add a detector or delivery path.
Large alert collections stay continuously scrollable
Resource alert tables, active-alert lists, incident timelines, and resolved
history must preserve full filter, group, count, selection, and action semantics
without mounting the full estate. Desktop tables use the shared
PlatformWindowedRows owner; phone cards use PlatformWindowedList with
feature budgets no larger than 32 items. Native scroll extent comes from spacer
geometry, and wheel/touch projection must maintain a mounted directional runway
so rapid scrolling never exposes a loading-looking blank section. Alert
windowing is presentation-only and must not change alert truth, grouping,
acknowledgement, resolution, or delivery state.
Destinations tab carries delivery evidence
The alert destinations tab is where the belief "delivery works" is formed, so
it must carry delivery evidence, not only configuration. It renders the
notifications-owned delivery log (AlertDeliveryLogCard fed by
useNotificationDeliveryLog over GET /api/notifications/delivery-log) as a
newest-first record of real alert delivery attempts with plain-language
outcome labels, destination names resolved from the loaded webhook configs,
failure classes, and secret-redacted error text. The card names its retention
windows separately: completed attempts remain for seven days while
dead-letter attempts can remain for 30 days. Retained terminal failures say
that retries stopped, because a permanent verdict can stop the first attempt
before its retry budget is exhausted. The description uses the same boundary
and never infers attempt exhaustion from dead-letter status alone.
It requests the bounded maximum
of 200 rows, displays an absolute semantic timestamp for timeline correlation
with relative time as secondary hover context, and says that test sends skip
the queue. An unreadable log renders as unavailable rather than empty. While
delivery needs attention the card sits directly below delivery health and
recovery controls on the Notifications route; a degraded Overview warning
links to that route so the diagnostic evidence and safe retry/dismiss actions
are reachable from the warning. Otherwise the route answers "where do my
alerts go" first: webhooks (named by the services they reach), then email and
Apprise, then this card. A switched-off email or Apprise destination renders
one line saying what turning it on does plus a "Show settings" control that
reveals the form without enabling it or marking the page changed. A new
webhook opens on the service choice rather than the generic JSON form.
useNotificationDeliveryHealth updates the health snapshot and its
availability in one batch so the attention state never reads a half-updated
pair and remounts the warning mid-refresh. The same card interleaves alert-owned
notification_suppressed and notification_deferred evidence from
GET /api/alerts/events with those attempts, ordered by the recorded event or
attempt time. Held rows name the affected resource, alert type, and
plain-language policy reason even when no delivery attempt exists. The event
read is independent and fail-soft: an unavailable event log removes held rows
without delaying, hiding, or marking the notifications-owned attempt log as
unavailable. Refreshing delivery evidence starts both reads from one operator
action while preserving that failure isolation. Test-send results that report
deliveryPaused surface as a warning toast, never plain success, across the
email, Apprise, and webhook test actions in useAlertDestinationsTabState
and useAlertWebhookDestinationsState. Delivery evidence remains
notification truth: the delivery log card must not resolve, suppress, or
re-evaluate alerts, and it does not alter the AlertConfig.enabled versus
activationState ownership boundary above.
Overview states paused delivery once and keeps rows scannable
The Alerts overview is where a notification lands, so it answers "what is
broken now" before configuration. When notification delivery is gated off,
OverviewTab renders the shared AlertDeliveryPausedCard once, driven by
useAlertDeliveryPausedReason (the same activation reading the destinations
tab uses), with overview wording and a link to the Notifications route. While
that banner is shown, per-alert delivery lines for the global holds
notifications_inactive and notifications_disabled are suppressed; every
alert-specific hold (cooldown, quiet hours, rate limit, flapping, suppression
window, monitor-only) still renders on its row. With detection switched off
the banner must not claim Pulse is still detecting alerts. Active alerts
render as compact rows: severity on the left edge, icon and badge, the alert
type, the start time as a locale-aware relative age on the overview's minute
clock, Acknowledge, Snooze and Investigate visible, and Timeline plus
resource monitoring policy behind a More disclosure that also closes an open
timeline. A one-line summary beside the heading counts open alerts by
severity and the triggered-in-24h total; threshold override counts are
configuration and do not appear on the overview. None of this changes alert
lifecycle, acknowledgement, snooze, or delivery evaluation.
Phone alert rows keep one action row
Below the sm breakpoint (640px) an active-alert row on the overview shows
Acknowledge (or its reversal), Snooze (or Unsnooze) and More on one row; the
Investigate split button moves into the More disclosure beside Timeline and
the resource monitoring policy. From sm up the row is unchanged. Four
actions had wrapped onto two rows under every alert, so a 375px phone showed
about three alerts per screen (average row 208px on the mock estate). Snooze
stays on the row at every width, so its dialog is never remounted when a
phone rotates across sm; only Investigate moves, by the shared
useBreakpoint() width, and renders once. The Acknowledged and Snoozed
badges and the snoozed-until line stay on the row. More names its region
through aria-controls only while that region is rendered. This is
presentation only: acknowledgement, snooze, investigation and policy
handlers are unchanged.
A guest alert's name opens that guest
On the overview, a VM or container alert whose platform page is Proxmox or
vSphere links its resource name to that page with ?resource= set to the
alert's own resource id. The shared workloads table resolves that id
(including a Proxmox source id such as <cluster>-<node>-<vmid>) through
the resource registry, opens the guest's drawer and reveals it, so a
notification reaches the affected guest in one tap instead of the top of a
page with every guest on it. Other alerts (hosts, Docker
containers, Kubernetes, TrueNAS, Pulse system alerts) keep their platform
page link, because their pages do not read ?resource=; Proxmox storage
alerts open their pool (below). So do orphaned
backup alerts (backup-subject:...), whose ids name no resource the
registry can open. A guest's per-filesystem alert (<guest>-disk-<key>,
marked by mountpoint or diskIndex metadata) names no registry resource
either, so CheckGuest in internal/alerts/guest.go stamps its owning
guest as guestResourceId metadata: the same guest ID the guest's own
alerts are raised under, on the canonical-spec and legacy checkMetric
paths alike (TestGuestFilesystemAlertsNameOwningGuest). The card links a
filesystem alert to ?resource=<guestResourceId>, opening the guest whose
filesystem filled; a filesystem alert without that key (raised before the
stamp) keeps the platform page link and never falls back to its own id.
The platform used for the monitoring policy menu is still read from the
page path alone.
A Proxmox storage alert's name opens its pool
A storage alert (resourceType storage) whose platform page is Proxmox
links its name to /proxmox/storage?resource=, so a full or degraded pool
opens on Storage instead of the top of the Proxmox overview. ZFS pool-state,
pool-error and device alerts are raised under
<storage>/zfs-pool:<pool>[/device:<device>] ids, which the resource
registry cannot open, so checkZFSPoolHealth in internal/alerts/storage.go
stamps the owning storage as storageResourceId metadata
(TestZFSAlertsNameTheirStorage); the card prefers that key, then the
alert's own id. An unstamped composite id (raised before the stamp, or a
pool-error alert whose counters have not grown since) opens Storage without
naming a pool and never passes the composite id. TrueNAS and vSphere storage
alerts keep their platform page, which does not read ?resource=.
Thresholds lead with each group's default limits
The Thresholds page answers "what will alert me, and at what limits" before
any group is opened. Every collapsed threshold group whose table edits global
defaults states them in its header through
getAlertThresholdsDefaultsSummary in
frontend-modern/src/utils/alertThresholdsSectionPresentation.ts. The helper
reads the same column list and defaults object the group's resource table
edits (each section hoists its columns into one module constant), and drops
every metric isAlertResourceMetricOff reports, so the header can never show
a limit the engine is not evaluating. Docker's restart loop (count within its
window) and memory-limit warn/critical are stated as whole rules with their
effective values: an unset value is the factory value, and a rule or level at
0 reads "off", as the engine evaluates it. Backup age is
summarized by its warning and critical days only, because fresh and stale
hours colour the dashboard and never alert. A group switched off as a whole
shows its Disabled badge and no defaults line. The CPU averaging line counts
every configured profile override, including ones only the API sets. CPU averaging is tuning, not the page's
answer. It is one collapsed disclosure stating the effective window and the
number of platform overrides, with the same per-profile selects inside, each
a compact shared FormSelect with its profile name as a visible label. Docker
and Machines list their resource groups before Docker-only rules and the
SMART card. Threshold values, inheritance, metric evaluation windows and
persistence are unchanged.
Off is 0 for Docker and PMG count thresholds, and per-row edits exist only where an override stores them
Docker's restart-loop and memory-limit thresholds and every PMG threshold are
plain ints in DockerThresholdConfig and PMGThresholdConfig, and 0 is Off:
the PMG checks already skipped a level at 0, and the Docker restart-loop check
now turns off when its count or window is 0 while the memory-limit check
evaluates warning and critical as independent levels. The settings Off toggle
sends -1, so NormalizeDockerDefaults and NormalizePMGDefaults store any
negative value as 0 instead of the factory value they used to substitute, which
reverted every Off on save. To keep a value an older alerts.json or an import
predates from reading as Off, AlertConfig and both threshold types decode
omitted keys as ThresholdCountUnset, and the normalizers (also run by
ConfigPersistence on load and save) replace that marker with the factory
value before the config is used or written. A PMG quarantine growth level needs
both its percentage and its minimum growth; either at 0 turns that level off,
because passing one alone made the canonical spec invalid and skipped the whole
quarantine check.
A Docker container override re-enables state alerts over a disabled Docker
default only when it carries its own powered-off severity, the same rule
applyThresholdOverride applies to every other resource. DisableConnectivity
is a plain bool, so an override saved for a CPU threshold alone used to turn
that container's state alerts back on.
The frontend mirrors the backend instead of second-guessing it. Backup and
snapshot ages keep a critical of 0 Off, both in the editor's sanitizers
(useThresholdsRecoveryDefaultsState) and in the saved payload
(normalizeAgeWarningCriticalPair); only snapshot sizes raise a 0 critical to
the warning size, as the backend does. Per-resource rows offer only metrics ThresholdConfig stores:
alertResourceSupportsMetric gives Docker containers CPU and memory, and PMG
servers nothing, and the PMG section has no bulk edit, because restart,
memory-limit and PMG thresholds exist only as defaults and the backend dropped
per-container and per-server edits of them on save.
Not covered here: AI Patrol's threshold adapter (internal/ai/alert_threshold_adapter.go)
still maps an Off alert threshold to its factory value, so Patrol keeps flagging
a metric whose alerts are off; honouring Off there is a Patrol product decision,
because a zero Patrol level would flag every reading.
External watchdog is Pulse-availability evidence, not notification delivery
The Alerts destinations surface owns the operator contract for an external
dead-man watchdog that detects loss of Pulse itself. It presents masked
configuration, live heartbeat state, canonical-monitor progress, consecutive
delivery failures, and the last restart interruption. The credential-bearing
ping URL must never return to the browser after save; the configured state uses
an explicit replacement placeholder and removal action. The watchdog remains
active independently of alert activation, snooze, quiet hours, and notification
delivery pause because those policies must not disable observation of Pulse.
Its Refresh and Show/Hide actions are the shared outline Button (xs and
settingsAction sizes), not a copied border shell, so they match the other
destination controls and keep a 44px touch target on phones.
Last success and Monitor progress are ages, so they read the frontend-primitives
shared relative-time clock and keep moving while the panel stays open, and the
panel re-reads GET /api/alerts/deadman/status in the background every 30
seconds. A moving age over a status read once at mount would claim the
heartbeat stopped while Pulse kept pinging every minute, and a status that is
never re-read kept its read-time age through a real stall, which is the one
case this panel exists to show. A background re-read leaves the Refresh control
alone, and a failed one keeps the last status and marks it unavailable. The
status read has no timeout, so reads carry a sequence number and a read's
outcome applies only when it is newer than the last read that settled: a
success shows its status and clears the unavailable mark, a failure sets it. A
slow or hung read can therefore neither overwrite a newer outcome nor block
later reads, and reads slower than the poll still land in order. While the
panel's poll runs, the next background read goes out at the first poll once the
newest read has been pending at least 29 seconds (one second of slack absorbs
timer jitter), and a Refresh is released by its own read, by any newer read that
settles, or by the first poll at which it has been pending that long, so a hung
Refresh frees within about two intervals of the click.
The delivery log's relative hover time on each row reads the same clock: a
logged attempt's time never changes, so its age is correct over a log that is
only re-read on demand.
Watchdog transport and monitoring progress remain notifications- and monitoring-owned respectively. Alerts owns the system-alert projection: delivery failure, canonical-loop stall, restart interruption, and durable-state failure use stable system-alert identities, normal lifecycle/event history, and idempotent fingerprints. A restart interruption is raised into history even when the first successful external report immediately resolves it, so recovery does not erase the outage record. The watchdog must never invent resource identity or become a second lifecycle for infrastructure alerts.
Escalation targets are exact and critical repeats are bounded
An escalation level may address credential-free logical destination IDs:
email, apprise, or webhook:<id>. Once exact IDs are present they are the
authority for that level; a deleted or unknown ID fails closed and must never
widen to a channel or every webhook. The legacy notify channel remains a
mixed-version fallback for configurations that do not yet carry exact IDs.
Optional critical-repeat policy applies only after the final escalation level. It sends at most one repeat per checker pass, uses a normalized 5–180 minute interval, and stops while the incident is acknowledged, resolved, snoozed, or global detection or delivery controls are paused. A repeat is durable escalation evidence with an explicit repeat marker rather than a new incident or a replay of missed intervals.
Every escalation level delay uses the same 5–180 minute safety boundary at the frontend state owner and the backend configuration-normalization boundary. Malformed API payloads and manually entered values therefore cannot turn a level into an immediate escalation or an unreachable timer beyond the supported schedule range.
The Schedule surface loads the same email, Apprise, and webhook catalog as the Destinations surface. It shows disabled and deleted selections explicitly, prevents an escalation level from becoming destinationless, and explains the critical-repeat stop conditions before save.
Recovery authority and operator qualification fail closed
If a lifecycle append or active-state checkpoint cannot reach SQLite, the degradation marker is the crash-safe restart authority. The marker is an atomically replaced, file-synced, directory-synced envelope containing the complete active-alert snapshot; the JSON mirror remains a compatible recovery aid but is not required for a new marker. While the marker exists, later saves refresh that envelope and must not make SQLite authoritative again in the same process. Bootstrap repairs SQLite from the envelope and only returns authority after durably removing the marker. Legacy text markers may use a readable JSON mirror, while malformed or source-less markers defer authority rather than silently trusting a potentially stale projection.
tests/integration/scripts/run-alert-qualification.mjs is the alerts-owned
operator qualification entrypoint. One run must cover crash-safe active-state
recovery, snooze and escalation policy, destination severity and exact-target
routing, a signed real HTTP webhook, durable delivery receipts and recovery,
dead-man progress and restart-gap reporting, plus the production frontend
against a managed local backend. The browser phase must exercise active-alert
snooze, delivery diagnosis, restart persistence, unsnooze, incident evidence,
history migration, virtualization, and clear tombstones without replacing the
backend APIs with route mocks.
Noisy gauges use one trust-stable lifecycle
Memory, temperature, and disk-temperature warnings use one continuous stability window on both lifecycle edges: an unchanged factory configuration requires five minutes above the trigger to open and five minutes at or below the recovery threshold to close. A return to the hysteresis band or a renewed breach resets the recovery run. Critical metric evidence may bypass only this factory or legacy activation delay; an explicit alert-intent policy remains authoritative. A metric-specific delay, including zero, remains the simple operator override and becomes the matching recovery window for these gauges.
Recovery timing uses the reducer's monotonic runtime evidence when supplied, so wall-clock changes cannot prematurely close an incident. Missing or unknown observations do not constitute healthy evidence and therefore cannot advance a recovery run. Restarts conservatively restart an in-progress recovery window rather than manufacturing elapsed health.
For guest disk capacity, valid per-filesystem evidence owns the alert identity and suppresses the less actionable guest-wide aggregate. This guarantees one incident for one full filesystem while retaining the aggregate only when no usable filesystem evidence exists. OS-managed transient mounts, including macOS Gatekeeper App Translocation volumes, are removed at the shared filesystem collection boundary before they can enter capacity inventory or alert evaluation.
Shared-system correlation never merges detector lifecycles
An active alert may carry the optional typed correlation object with a
shared-system kind, stable key, primary or supporting role, and an
auditable reason. This is presentation-only incident context. Every correlated
signal retains its own canonical spec, confirmation state, acknowledgement,
recovery evidence, history, and notification decision; recovery of one signal
must never clear another.
Proxmox connection availability is the primary signal for its pve:<instance>
system. PVE node connectivity is a supporting signal through provider-owned
instance membership. A linked host-agent connectivity alert is supporting only
when monitoring proves the reciprocal node ID and agent ID relationship and
the node supplies the same non-empty instance. Missing, one-sided, duplicate,
or conflicting identity fails open and leaves alerts separate. Resource-path
truncation, hostname similarity, and timing coincidence are forbidden as
correlation evidence.
The overview groups exact-resource detector signals and explicit shared-system
signals. It selects a declared primary when one is present, describes the rest
as linked signals, and acknowledges each underlying alert individually. Older
alerts and clients remain compatible because the field is additive and
optional. internal/alerts/correlation_test.go,
internal/monitoring/alert_correlation_test.go, and
frontend-modern/src/features/alerts/__tests__/useAlertOverviewState.test.tsx
pin lifecycle independence, fail-open identity, and client grouping.
Infrastructure incident synthesis is deterministic and evidence preserving
The same optional alert correlation boundary also admits the
infrastructure-incident kind. It adds one of six typed failure classes
(runtime, network-path, application-response, certificate,
dependency, or evidence-coverage), an inference state
(supported-cause or observation-set), the primary alert and resource IDs,
sorted affected resource IDs, and a bounded observation list carrying alert,
resource, severity, observation time, and evidence IDs. It never copies
provider response bodies, credentials, arbitrary metadata, or error payloads.
internal/alerts/incident_synthesis.go derives this context from the complete
post-recovery active detector set plus active canonical relationships from
ResourceRelationshipsWithCanonicalParent. It may traverse at most eight
dependency edges and records at most 100 observations. Exact relationship
direction, confidence of at least 0.8, a current primary failure, and bounded
event ordering are required for supported-cause. Healthy or disagreeing
primary resource state, weaker topology, late timing, cycles, or incomplete
support must remain an observation-set. Name similarity, resource-ID prefix
truncation, message parsing, and coincident timestamps remain forbidden causal
evidence.
Every grouped alert still owns its canonical spec, acknowledgement, history,
timeline, evidence, and recovery. Synthesis is recomputed on each unified
incident reconciliation and clears stale synthesized links, so primary-first
recovery leaves the still-failing symptom visible as standalone immediately.
Existing shared-system correlations remain authoritative and are not
overwritten. Resource maintenance and monitoring intent continue through the
canonical resource-policy and intent gates before synthesis; no group-local
mute or maintenance state is allowed.
Only a supporting member of a supported-cause group suppresses duplicate
notification delivery, with the typed correlated_primary reason and primary
alert ID in the event log. The primary remains the one delivery owner. Supporting
members also skip escalation levels and critical repeats while that supported
cause is current; an already queued escalation is revalidated before delivery.
If the supported primary recovers or the causal link is lost while its symptom
still fails, the symptom regains independent delivery without a new occurrence.
It receives its first notification only if it was never dispatched before
grouping; an already-notified symptom does not duplicate on group dissolution.
Acknowledgement, rate limits and other ordinary delivery gates still apply.
observation-set members continue to notify independently because Pulse has
not established causality. incident_synthesis_test.go pins classification,
contradiction downgrade, bounded evidence, duplicate-delivery suppression,
partial-recovery delivery and no duplicate after prior dispatch, escalation
suppression, and queued-callback revalidation without muting observation sets.
Alert hydration is not resource admission
REST active-alert recovery may complete before a resource snapshot. Neither active incidents nor alert hydration completion establishes an empty resource estate or replaces platform navigation admission. Incident access must remain available during socket reconnect; this does not assert notification delivery.
Qualification additionally injects admission HTTP 503 on reconnect with populated inventory and checks incident access on desktop and the 390px mobile rail. Enabled acknowledgement controls prove access only, not persisted acknowledgement or external notification receipt.
Issue explanation follows the customer-journey qualification boundary, which separates scripted interaction proof from provider reasoning, remediation and repeatable customer outcomes.
Delivery health requests have latest-started ownership
Overlapping mount, configuration Retry and post-queue-action reads must not allow an older completion to replace newer delivery health. Success, failure, first-load completion and the refreshing flag belong only to the most recently started health request. A stale healthy response cannot hide degraded attention; a stale error cannot invent unavailability after recovery. Queue actions still refresh from the server, not from their affected count. This changes no provider acceptance, incident lifecycle or delivery guarantee.
The hook and destinations caller regressions in
useNotificationDeliveryHealth.test.tsx and
useAlertDestinationsTabState.test.tsx pin ordering and loading ownership.
scripts/check-delivery-health-ordering.mjs exercises the real caller and card
in Chromium with scripted API completions; it is not installed delivery proof.
Alert status distinguishes dispatch from destination evidence
The active alert card renders a valid diagnosis lastNotified timestamp as
“Dispatch requested”, never “Notified”: the alert manager records this field
before invoking delivery callbacks. A cooldown's nextEligibleAt is labelled
“next eligible”, not a promised send time. Neither field proves destination
acceptance or recipient receipt; that evidence must not be inferred from the
muted presentation tone. Missing or invalid timestamps retain the existing
pending/cooldown fallback; acknowledged alerts retain their badge without a
second status line. No API field, notification policy or shared primitive changes.
The existing wrapping status text must remain readable at desktop and phone
widths despite the longer labels. The presentation and Overview delivery-status
tests cover the evidence boundary; scripts/check-alert-dispatch-copy.mjs
qualifies the real Overview with scripted API data in Chromium, not installed
notification delivery.
Unchanged active-alert JSON recovery checkpoints
The recovery-mirror writer canonicalises complete records before comparing bounded existing bytes. An identical regular file with mode 0600 is not replaced; acknowledgement, metadata and resolved/empty state changes still use the atomic synced replacement path. Missing, corrupt, insecure or symlink destinations are not accepted as unchanged. Failed writes remain retryable and the unchanged path retains directory sync so a prior post-rename sync failure is not silently accepted. Windows retains replacement when its reported mode does not match POSIX 0600.
TestActiveMirror* isolates this JSON path without database workers and covers
ordering, fresh-manager checkpoints, state changes, file loss/corruption, failed
rename retry and Unix destination hardening. This addresses one write source
in #1966, not total installed write amplification: directory metadata handling,
intent snapshots, changing alert histories and database writes remain separate.
It changes neither alert latency nor the selected release candidate.
Resource incident reads retain lifecycle ownership
The resource incident hook gives each started read a unique per-resource owner. Only that owner may publish history, report a failure or clear loading. Reset invalidates all pending owners before clearing state; disposal invalidates them and prevents new loads. Overlapping reads for different resources remain independent. Closing a row still permits its in-flight result to populate the existing cache; reopening cached history and explicit refresh are unchanged. Requests are not transport-cancelled. No API, retention or notification-delivery policy changes.
The hook's ten ordinary regression/control cases cover success, catch and
finally writes, reset/reopen and disposal. The existing panel tests cover its
presentation. scripts/check-incident-request-ownership.mjs exercises the real
hook and panel in Chromium at desktop and phone widths with scripted responses
and fixture reset, overlap and unmount controls. It is component lifecycle
acceptance, not an installed full-page or notification-delivery receipt.
PR1973's proposed resourceIncidentError accessor is absent here; if introduced,
its writes must obey the same owner check and gain a stale-error regression.
Destination server-error presentation
Delivery activity preserves the queue's server_error classification as
“Destination server error”, rather than treating it as unclassified. When that
class dominates retained terminal failures, the full health description directs
operators to destination service status and server logs, then to retry retained
deliveries once the service is available. Unknown classes still use the generic
fallback; this presentation does not change retry scheduling or imply receipt.
The presentation unit tests cover the label and terminal guidance.
scripts/check-delivery-server-error-copy.mjs renders the real health and log
cards in Chromium at desktop, tablet and phone widths, asserting the copy,
absence of page errors and horizontal overflow. Scripted props are component
evidence only, not installed notification delivery.
Webhook test/save custom-field parity
The webhook form applies the same service-specific custom-field normalisation when testing unsaved data as when saving it. In particular, manually entered Pushover app_token/user_token aliases become token/user with the existing canonical-value precedence. Existing-field editing remains normalised on load. This changes neither delivery scheduling nor the meaning of a successful test. The registered WebhookConfig regression covers alias test/save payload equality; the browser fixture exercises the real form at desktop and phone widths with synthetic callbacks, not a hosted Pushover destination or installed delivery.
PBS capacity has one alert policy owner
PBS datastore capacity alerts are evaluated by CheckStorageWithCapacityTrend
from fresh PBS polls. Storage defaults, canonical datastore aliases, per-resource
overrides, hysteresis and predictive capacity policy govern that lifecycle.
The fixed 90/95% PBS topology assessment remains resource risk evidence; its
Pulse-generated capacity_runway_low incidents must not independently enter
active alerts on either the datastore or the parent backup server. Existing
copies retire through normal policy reconciliation, without deleting state or
claiming that capacity itself recovered. Other datastore state/error incidents
and native provider incidents retain their existing lifecycle.
TestPBSCapacityUsesStoragePolicyNotTopologyBands in
internal/alerts/unified_incidents_test.go uses real registry projection to pin
both duplicate symptoms at 97.9%, canonical-alias 99% versus 90% policy, existing
alert retirement, threshold recovery, and preservation of datastore failure.
The posture and roll-up tests use datastore state failures, independent of
capacity. This relies on the live PBS poll evaluator; a release adaptation must
include that evaluator rather than remove the topology alerts in isolation.
TestPBSPolledCapacityRequiresObservedRecovery in
internal/monitoring/monitor_pbs_coverage_test.go additionally exercises the
97.9% policy transition through synthetic PBS HTTP polling, storage conversion
and unified alert synchronisation, including absence of duplicate parent alerts.
Alert history clear/read ordering
History read failure is not an empty result. The existing History tab announces an error and offers an explicit read-only retry. Pending or failed reads expose neither the frequency chart nor severity totals, and failure suppresses the ordinary empty-result claim. Same-range saved entries remain available with an incomplete/out-of-date warning, alongside live alerts. A snapshot loaded for another range is retained internally but excluded from that range's rows; switching range is not deletion. Search, severity and range remain URL-owned.
Only the latest request may settle rows, failure or loading. An explicit retry admits one pending logical read; existing API transport policy is unchanged. Organisation replacement clears the saved snapshot and read error; retired reads and clears cannot change the replacement view. Disposal prevents another retry or a late completion from modifying view state. An already-dispatched clear is not cancelled, and deletion/retention, active alerts, acknowledgement and notification delivery are unchanged.
useAlertHistoryState.test.tsx and HistoryTab.loadFailure.test.tsx exercise
initial failure, retry, pending/accepted empty results, same-range retention,
range isolation with live alerts, obsolete replies, context replacement and
disposal. The parent-bound browser receipt exercises the real existing tab,
hook, API adapter and stylesheet at desktop and phone widths with synthetic
HTTP failures and deferred replies. These are source presentation/ownership
controls, not installed backend recovery or destination delivery.
A successful history clear invalidates reads started before its completion and settles their loading state, so delayed responses cannot repopulate deleted history rows. Failed clearing leaves the pending read valid; later range refreshes remain available. This is view-state ordering, not a change to retention, active alerts, acknowledgement or notification recovery semantics.
The deferred-response cases in useAlertHistoryState.test.tsx verify successful and failed clearing plus subsequent range refresh. The registered scripts/check-incident-request-ownership.mjs browser proof also exercises the real history hook and administration card: load a row, start a pending range read, confirm clear, then release the obsolete response at desktop and phone widths. Scripted API responses establish component behaviour, not installed backend deletion or destination delivery.
Escalation callback admission uses current policy and occurrence
PrepareEscalationNotification rejects asynchronous snapshots after escalation
or global alert disablement, inactive activation, recovery, acknowledgement,
snooze, supported-cause symptom correlation, or replacement by another
occurrence of the same alert ID. It returns
the current alert and detached current per-level routing, not mutable manager
state. The monitoring callback must use this admission before enqueueing.
This check does not recall provider-accepted messages or make the subsequent
notification enqueue atomic with alert resolution.
escalation_policy_change_test.go pins a short-lived alert and disabled policy
before the former 180-minute deadline, plus current payload/routing selection,
copy isolation, snooze and invalid-level rejection. These deterministic component tests
establish neither installed delivery timing nor the cause of a delayed email.
Unchanged intent-pending checkpoints
The serialized active-alert checkpoint also compares the complete, sorted
pending-policy JSON against a bounded regular file with mode 0600 before
replacement. Empty and nonempty unchanged pending state leave the file intact;
changed or cleared state, missing/stale content, unsafe destinations and failed
writes retain replacement/retry semantics. This does not change intent timing,
retention, alert freshness or the existing persistence durability contract.
TestIntentCheckpoint* exercises the complete JSON-only checkpoint without
starting database workers, including restart loading and Unix hardening. The
active mirror's unchanged fast path alone does not cover this companion file.
This is a narrow write contributor, not aggregate installed-write acceptance.
Coalesced asynchronous active-alert checkpoints
active_persistence.go admits one asynchronous checkpoint worker per Manager,
not one goroutine and full serialization per mutation. Admission is protected
by the shutdown lock. A worker starts immediately, takes fresh state through
SaveActiveAlerts, and retains one follow-up request for mutations arriving
during its current pass. Returning to idle and accepting another request share
the same lock, so requests cannot be stranded at the idle boundary. The worker
never holds the shutdown lock while acquiring persistence or manager-state
locks; callers may request a checkpoint while holding the manager-state lock.
This coalesces derived snapshots, not lifecycle events or notification work.
Synchronous lifecycle durability, checkpoint revision/failure-epoch checks,
recovery-mirror replacement, intent timing and periodic saves are unchanged.
An error or recovered panic releases admission after any requested follow-up;
there is no automatic busy-loop retry. Once shutdown closes admission, the
worker finishes only its in-flight pass; Stop joins it and writes the final
current snapshot synchronously instead of draining redundant queued snapshots.
TestAsyncActiveCheckpointBurstIsBounded in internal/alerts/alerts_test.go
blocks the real checkpoint writer, bounds a 512-request burst to one worker,
then verifies current acknowledgement, incident age, metadata and pending intent
through actual JSON persistence and a fresh Manager. The scheduler regressions
in internal/alerts/async_checkpoint_test.go pin one follow-up for 10,000
in-flight requests, idle restart, error/panic recovery, retry after a real file
failure, and shutdown admission/draining. These checks establish bounded
checkpoint work, not the installed CPU/write rate or any reporter's recovery.
Warning-level destination preference persistence
Email API decoding and encoding, plus webhook edit/create and list presentation, preserve all three supported minimum-severity values: all, warning, critical. Unknown values still fall back to all. The email adapter round-trip matrix and webhook edit/save regression cover issue #2069 without altering delivery filters or sending provider notifications. Component proof is not installed delivery.
SMTP rate-limit preference persistence
The existing SMTP editor's rate limit is included by
alertDestinationsModel.buildEmailConfigPayload on every destination save.
Unrelated email edits and switching email off must preserve the loaded limit;
an explicitly edited limit must reach the existing Notifications API unchanged.
Zero remains the backend's default-limit sentinel, not an omitted or unlimited
value. Missing saved limits still use the existing UI default of 60 per minute.
SMTP masks, TLS settings, recipients, tags and severity preferences are unchanged.
This repairs a missing UI-to-API field, not the schema or transport policy.
Revealing switched-off email settings permits pointer and keyboard editing without
enabling email. The status toggle remains off and Send test email stays disabled.
emailRateLimit.persistence.test.tsx connects the actual SMTP editor, destination
mutation owner, payload builder and email API to synthetic wire assertions.
The two payload-shape suites retain strict key checks including rateLimit.
browser-tests/email-rate-limit.cjs checks save and readback on desktop and phone,
including unrelated edits, disabled email and legacy zero. These fixtures do not
send notifications or establish native SMTP throughput, release availability or
the cause of an existing delivery failure.
Pulse Mobile push destination without an upsell
The Alerts destinations page renders the mobile push panel only when the
instance has the relay feature. Community installs see no panel and no
upgrade prompt, and never load or save relay push settings. Licensed instances
keep the minimum-severity control and the link to Pulse Mobile settings, with
the 31 March 2027 retirement date in the copy. Webhook, email, Apprise and
dead-man destinations are unaffected.
Occurrence-qualified asynchronous recovery delivery
Every resolution funnel passes the actual ResolvedAlert to the callback bus.
SetResolvedAlertCallback captures a deep-cloned alert plus its resolution time
before asynchronous execution, rather than reconstructing an occurrence from a
reusable ID later. Legacy ID-only subscribers retain their public IDs; a panic
in one recovery consumer does not prevent the other consumers from running.
The live monitor uses this snapshot for AI recovery and notification delivery.
CancelResolvedAlert removes only matching ID/start-time grouping entries and
cooldown state. CancelByAlertOccurrence applies the same match to persistent
pending, sending, failed and dead-letter firing rows, preserving newer
occurrences, unrelated grouped members and their operational links. Missing
start-time identity is a no-op, never a request for identifier-wide cancellation.
The existing explicit ID-wide cancellation API remains available. Delivery
gates, pending-only suppression counts, destination receipts, attempt history,
quiet hours and best-effort treatment of already-started sends are unchanged.
operational_contract_test.go verifies snapshot isolation across a
blocked callback and changed ID-indexed history, legacy identity and panic
isolation. queue_occurrence_cancellation_test.go verifies nanosecond-distinct
recurrences, grouping/cooldown preservation, missing identity and restart
readback for all four cancellable row states without deleting failed audits or
new-occurrence destination receipts. The monitor's connected tests use real PBS
evaluation, normal callbacks, the persistent queue and a local HTTP destination;
these receipts are not native provider or installed-release acceptance.
Pending cancellation is not proof that every destination missed the firing. The monitor delegates recovery eligibility to persisted occurrence/destination receipts even when an old retry remains or restart has lost RAM cooldown state. The partial-destination monitor control establishes a real local HTTP 200/503 split and preserves the newer firing and the old recipient's recovery.
Notification activity reports each evidence source independently
The existing Notifications activity card distinguishes an initial or pending read from a confirmed empty window. Held/deferred event reads have their own loading and unavailable state; a failed read withdraws their prior rows and keeps the warning until a later successful read. Only the current refresh owns those states, and disposed readers cannot publish late completions. A slow held-event read does not delay delivery attempts or disable their refresh.
A delivery-attempt failure withdraws only attempt rows, not independently readable held events. Conversely, missing held evidence does not hide readable attempts or imply that no notifications were held. The combined empty state requires both sources to be readable and no refresh pending. Retention, event limits, queue actions, permissions and notification delivery policy are unchanged.
useNotificationDeliveryLog.test.tsx, useAlertDestinationsTabState.test.tsx
and AlertDeliveryLogCard.test.tsx exercise independent failure, recovery,
initial loading and current-request ownership; the production Notifications
tab browser fixture verifies the connected read/retry states at desktop and
phone widths. These are synthetic frontend controls, not recipient delivery
or containing-release acceptance.
Resource drawers read open alerts from the active-alert map
getAlertsForUnifiedResource in frontend-modern/src/utils/alerts.ts is the
canonical lookup from one unified resource to its open alerts in the websocket
activeAlerts map. It matches the alert keys the backend writes for that
resource (unified id, canonical primary and superseded ids, Proxmox source id,
metrics target, and for agent and Docker hosts agent:<id> and
docker:<host source id> plus the component alerts nested under them) and
orders them most severe first. A Docker container answers to the key the
engine writes for it, docker:<host source id>/<container id> (or
docker:<host source id>/name:<name> without an ID, per
DockerContainerResourceID), so its drawer lists its own alerts; until
2026-10-14 the container drawer matched none of them.
getOwnAlertsForUnifiedResource returns only the exact-key alerts, without
the nested component alerts, and is what drawer charts use for dashed
triggers: a host's Needs attention lists its containers, but a container's
memory trigger is not the host's. Those rows use getAlertAttentionCopy, so a
latched or recovering alert, whose message keeps its breach-time value, reads
as "Memory 70% now, recovering" beside a chart showing 70%. The
Machines/Infrastructure and Docker host
drawers build their "Needs attention" rows from it; resources no longer embed
an alert list, so no drawer can silently read an always-empty copy.
getUnifiedResourceAlertStyles computes row highlighting from that same set,
so the Docker hosts and Machines table rows tint for exactly the unacknowledged
alerts their drawers list. Docker host rows previously matched the row id and
display name against getAlertStyles; Docker alerts are keyed
docker:<host source id> and carry the hostname as their node, so a host whose
display name differs from its hostname was never highlighted, and Machines rows
had no alert highlighting at all. getAlertStyles keeps its
id-plus-node-name matching for the platform tables whose alerts are keyed that
way.
Threshold rows mark the limits whose alerts are open
On Alerts > Thresholds a metric value carries a pulsing "Active alert" dot
while an alert on that metric is open. useThresholdsTableState indexes the
websocket activeAlerts map by each alert's resourceId and type
(indexActiveMetricAlerts in
frontend-modern/src/features/alerts/thresholds/thresholdsResourceModel.ts)
and asks thresholdRowHasActiveAlert per cell. The map is keyed by alert ID
(<resource>::metric-threshold:<metric> for threshold alerts), so a row never
rebuilds a key; until 2026-11-01 it looked up <row id>-<metric>, a key no
alert has carried since canonical metric IDs, and no row ever showed the dot.
A row matches on alertResourceIds when its data projector sets them, else on
its own id. Machine rows answer to agent:<host ID> (CheckHost's
hostResourceID, while the row and its override keep the bare host ID) and to
their disks' usage and SMART temperature alerts nested under it
(agent:<host>/disk: and agent:<host>/disk_temp:), because the machine's
Disk % and Disk Temp limits are what those disks inherit; sensor and RAID
children have limits of their own. Machine disk and sensor rows are keyed on
the alert resource ID already. Proxmox node rows answer only to their Proxmox
node ID: a node merged with its Pulse agent hands CPU, memory and disk to the
agent, and those alerts mark the Machines row whose limits judge them, never
the node row's unused limits. Guest rows answer to
getUnifiedResourceAlertResourceIds (frontend-modern/src/utils/alerts.ts),
the exact keys getOwnAlertsForUnifiedResource matches.
SMART temperature continuity and pending-only policy intervals
Disk temperature activation/grace and recovery require continuous accepted observations. Empty SMART inventory, one disk omission, standby, non-positive readings, explicit missing/unavailable/unsupported collection and reporting-lease expiry interrupt pending/recovery timing without changing an open alert, its acknowledgement or lifecycle events. Positive legacy readings without temperature provenance remain accepted; an expired last-known positive value is not live health. Canonical and shadow reducers plus explicit intent grace are interrupted together. A non-empty inventory still resolves a departed disk only after three consecutive omissions; empty/expired inventory breaks that confirmation sequence.
Configuration saves reset pending-only runs when global agent policy, host or inherited node/guest overrides, or the applicable disk-type threshold disables evaluation, even if no report arrives before re-enablement. Existing known-type default normalisation is unchanged; the zero-type save control uses a custom type whose zero threshold the normaliser preserves. Only live pending identity/link/type context is retained; unrelated saves preserve a known enabled run and explicit overrides retain precedence over type defaults. Restored intent without that context restarts conservatively on a save rather than inventing continuous enablement. Cleared intent grace is checkpointed. Existing active-alert per-type/save/offline/removal resolution remains unchanged.
metric_observation_continuity_test.go and host_unraid_lifecycle_test.go cover
valid/legacy/provenance controls, missing/rejected/expiry gaps on both timing
edges, explicit grace, acknowledgement/event preservation, host isolation,
configuration intervals, override precedence and departure. These deterministic
source tests do not establish installed temperatures or native guest recovery.
SMART risk inventory gaps and disabled rules
Agent disk-health and disk-wearout alerts follow the SMART inventory, not
filesystem usage cleanup. Empty inventory, standby and unobserved risk fields
hold the existing occurrence and acknowledgement without recovery/refiring.
Departure requires three consecutive non-empty reports omitting the disk;
listing it again, an empty inventory or reporting-lease expiry restarts that
confirmation. Both risk families on the same disk count once per report.
Filesystem absence still resolves usage on its first missing report. Temperature
pending/grace/recovery interruption and departure semantics remain unchanged.
Disabling every recorded cause releases that risk alert without a reading, both on a configuration save and on an empty report. Partially enabled causes, missing legacy cause metadata and unknown codes remain conservative. Re-enabling cannot raise an alert from absent evidence; fresh bad evidence uses the existing stateful refire/cooldown history policy. Node linking retains existing node ownership, including when the agent's SMART inventory is empty; host removal and policy disablement release agent risk state and stale absence counts. Counts are ephemeral and restart conservatively after a process restart.
TestHostSMARTRisk* in host_unraid_lifecycle_test.go exercises actual CheckHost, expiry, configuration
save, firing/recovery callbacks and event history with synthetic telemetry.
It covers both families sharing a disk, interruption/reconfirmation, partial
rule disablement, re-enablement, legacy/unknown causes, filesystem and host
isolation, removal and node-link ownership. These are source lifecycle controls,
not collector, smartctl, installed notification or native disk-recovery proof.
Storage observation gaps restart confirmation and timing
An absent or unknown storage connectivity status interrupts activation,
recovery confirmations and pending offline intent grace without resolving an
existing outage or changing its acknowledgement. Capacity is independently
observable while connectivity is unknown. Missing/unconfirmed-zero capacity,
negative usage and offline/unavailable storage interrupt capacity activation,
and intent grace timing; they do not supply healthy capacity evidence.
Capacity recovery retains its existing immediate measured-clear behaviour,
not the recovery stability window used by memory and temperature gauges.
A confirmed empty store still supplies a genuine zero-usage recovery reading.
Fresh observations must satisfy the existing full confirmation/delay after a
gap. Interruption is scoped to the storage's canonical identity and applied to
both live and shadow reducers; interrupted intent is checkpointed for restart.
TestStorageConnectivityObservationGaps and
TestStorageConnectivityGapRestartsIntentGrace exercise the real storage
entry point, acknowledged occurrence retention, neighbour isolation, fresh
reconfirmation and empty intent checkpoint. Storage routes in
TestMetricObservationGapRestartsActivation covers capacity gaps and explicit
intent grace; TestStorageCapacityGapHoldsOccurrence verifies retained,
acknowledged capacity incidents and genuine immediate empty-store recovery. Existing restart, empty-capacity and predictive-capacity controls
remain required. These source controls are not native storage recovery or
installed destination-delivery evidence.
TestStorageConnectivityGapWebhook also exercises the production Monitor
callbacks, ordinary notification queue and guest-local HTTP receiver: the gap
and first fresh healthy poll send no recovery; confirmed recovery sends one
resolved receipt with the original occurrence identity, and the queue drains
without failed/DLQ work. This is synthetic callback/receiver acceptance, not a
native appliance or external webhook-provider result.
Composed host filesystem threshold ownership
The shared filesystem resolver preserves the explicit host or linked-resource
Disk override ahead of per-hardware-type defaults, for both host reports and
configuration-save reevaluation. A filesystem-specific override still takes
precedence. Disabling that filesystem releases its canonical usage metric and
pending intent, retaining the reviewed agent/node ownership semantics.
TestHostDiskOverrideBeatsDiskFillByType checks agent and inherited overrides
through a configuration save and the next report without losing the occurrence;
TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold retains per-filesystem
and per-type controls. This is source composition, not installed delivery proof.
Shared disk-temperature composition and pending ownership
Host temperature evaluation retains the upstream shared SMART/Unraid reading and
per-type threshold judgement, including Unraid-only disks and competing readings
on one device. It also retains reviewed observation-gap interruption and pending
host/link/type context for configuration saves. A live Unraid-only row is not an
observation gap merely because SMART is empty; absent or uncollected rows still
restart confirmation. TestUnraidOnlyTemperatureRetainsPendingContinuity checks
continuous, spun-down, omitted and type-off intervals under both grace paths;
existing SMART pending, override, departure and upstream Unraid controls remain.
Applied configuration input ownership
Manager.UpdateConfig clones the supplied configuration before normalisation
and retention. Callers keep ownership of its JSON-shaped maps, slices and
threshold pointers, just as they own GetConfig snapshots. Subsequent edits
do not alter running policy until explicitly reapplied; normalisation cannot
rewrite an input concurrently encoded for persistence or a response. The
existing typed clone's non-JSON custom filter-value limitation is unchanged.
ApplyConfigUpdate already decodes an owned configuration under the manager
lock and keeps its atomic partial-update semantics.
config_validation_test.go requires unchanged caller inputs, normalised
live thresholds, isolated edits, ordinary guest CPU evaluation, explicit
resubmission and concurrent response encoding/update under the race detector.
Existing snapshot, persistence, partial-update and threshold tests remain
required. This is source ownership acceptance, not installed notification or
guest-agent recovery evidence.
TrueNAS disk heat policy for Patrol and Assistant
Manager.TrueNASDiskTemperatureThreshold(resourceID, diskType) resolves the
truenas-disk tiers of effectiveAlertPolicyNoLock: the disk's canonical or
identity-resolved override, then TrueNAS Disks defaults, then disk-type policy.
A disabled disk/default or non-positive trigger leaves no heat policy.
Global/platform alert switches silence alert evaluation without changing this
policy, matching existing agent-disk semantics. Callers receive a copy; config
normalisation, saved Off/custom/inherited values and alert severity are unchanged.
IsTrueNASDiskResource uses the evaluator's classification, including merged
agent/TrueNAS disks. Patrol/Assistant do not walk those disks to a synthetic
TrueNAS system agent and accidentally apply an unrelated host override.
Retained temperature readings remain unjudged, not affirmative recovery.
Verification: TestTrueNASDiskTemperatureThresholdMatchesTrueNASDiskAlerts
and TestIsTrueNASDiskResourceMatchesTheUnifiedEvaluator in
internal/alerts/threshold_resolution_shared_test.go cover tier, identity,
Off/disabled, untyped, clone and nil-manager compatibility.
TestAlertThresholdAdapter_TrueNASDiskTemperatureFollowsItsAlertTiers and
TestPatrolJudgesTrueNASDisksByTheirAlertTiers cover adapter, Patrol triage,
verification and Assistant context while preserving agent-host overrides.
Frontend table/drawer tiers use the Web resolver described below; these
backend tests do not establish browser or native appliance acceptance.
TrueNAS disk display reconciliation (8 October 2026)
The TrueNAS storage table and inline resource drawer resolve temperature by
the disk resource's own temperature override, then truenasDiskDefaults,
then the per-type disk policy. Explicit zero/negative thresholds or a disabled
disk/TrueNAS default leave readings visible but unjudged. Host diskTemperature
overrides and the agent alerts switch do not control API-only TrueNAS disks.
The shared agent disk resolver and Proxmox pulse-relaxed guest resolution
remain unchanged. Existing Off-save normalization and inherited TrueNAS
threshold editors on main are retained, not replaced by the older PR stack.
Verification: src/utils/__tests__/metricThresholds.test.ts,
src/features/truenas/__tests__/TrueNASStorageTopologyTable.test.tsx and the
production table/drawer fixture browser-tests/truenas-disk-thresholds.cjs.
The containing backend also resolves the disk-specific tiers for Patrol.
Frontend checks do not establish native appliance or notification recovery.
Existing SMTP form exposes only persisted settings
The Email destination editor follows the public EmailConfig: SMTP connection,
credentials, From/recipients, security, rate limit, tags and minimum severity.
Reply-to, max retries and retry delay are not offered or fabricated by the UI;
the save path never accepted them. UIEmailConfig extends that public type
with concrete normalized password/rate-limit values, and the advanced form
explains that Pulse manages retries. Existing server retry behaviour is unchanged.
EmailProviderSelect.supported-settings.test.tsx requires absence of all three
ineffective controls, default/payload parity and supported edits/reloads when
email is on or off. Existing component and destination-state tests retain
provider, mask, routing and off-state behaviour;
src/utils/__tests__/alertEmailPresentation.test.ts covers the retry explanation.
Browser proof uses the production email panel, destination state, API and CSS
with synthetic local endpoints. It is not native SMTP delivery or a release.
Notification recovery belongs to its mounted view
Overview and Notifications retire their delivery-health/recovery owner on unmount. Late health success, failure and loading completion cannot update that owner. Late Retry/Dismiss completion cannot publish a global toast, change its feedback or start health/activity readbacks after the view retires. A callback scheduled before retirement is checked again before invocation; already-started activity failures cannot restore retired feedback. Calls to retired health/action/clear handlers do nothing, including no confirmation.
An already-dispatched mutation or read is not cancelled, undone or declared failed by navigation. Active views retain the existing confirmation, acknowledgement, read ordering and persistent failure feedback. A replacement view owns its independent read and any deliberately confirmed action; source checks do not establish notification delivery or installed queue recovery.
useNotificationDeliveryHealth.lifecycle.test.tsx covers pending reads and
both actions, callback scheduling/completion and retired entry points.
useAlertDestinationsTabState.test.tsx connects both actions to the real
Notifications owner, proving a late result cannot notify or start readbacks
in its replacement while the replacement's own action still works. Browser
controls use the actual Overview/Notifications components, API adapters,
toast and CSS with synthetic local endpoints at desktop/phone widths.
Mounted held readings follow the shared clock (10 October 2026)
Node, guest and Mail Gateway drawer attention, and the Proxmox node temperature hover, age an unchanged metric evaluation on the existing shared clock. The metric presentation requires an explicit clock; drawer copy reads an accessor only for usable live status, leaving non-metric rows unchanged. The ten-minute stale boundary, evaluated breach date, legacy fallback, clear rule and operator memory basis remain independent. This does not infer collection or recovery. Mounted drawer controls and lazy-copy tests cover this boundary.
Guest filesystem settings do not assert current readings (12 October 2026)
The existing Proxmox Guest Filesystems settings rows use
getWorkloadGuestDiskRead for the same provider read/power-state admission as
Workloads. A prev- read reason labels retained usage Last known; an
unavailable read or explicitly stopped VM withdraws used bytes and percentage,
without deleting mount identity, saved overrides or threshold controls. Native
runtimeStatus takes precedence over a canonical resource held online by
another source. Independent LXC platform storage does not inherit VM-only
read reasons. A new accepted reading clears the explanation, including zero.
The phone card displays the optional readingMessage as wrapping text beside
the reading. The desktop row puts it on the name's line, ending in an
ellipsis with the whole note in its title, so the row stays one line instead
of wrapping a paragraph under the name (rows had reached 65px). Desktop rows
are 33px: the 24px alert toggles and the 26px Global Defaults threshold
inputs overhang the row instead of setting a 35px height.
Missing filesystem samples keep saved settings editable with an explicit
unavailable label. Existing numeric serialization, identity, persistence,
active-alert linkage, edit-time row stabilisation and threshold evaluation
remain unchanged. This presentation does not change collection, issue guest
commands, prove thaw/writes/liveness, or establish installed/released recovery.
Verification: useThresholdsGuestData.test.ts covers every fixed read reason,
power/read replacement, LXC and source independence, unknown-reason privacy and
saved-override retention. ResourceTable.test.tsx covers visible/reactive text
in both layouts. browser-tests/guest-filesystem-thresholds.cjs connects the
production hook, table and editors on desktop Chromium and phone WebKit,
including scoped refusal text, custom settings and same-VMID/mount separation.
Frontend alert matching for PBS and PMG resources also keys on their
instanceId (getUnifiedResourceAlertKeys in frontend-modern/src/utils/alerts.ts),
the id internal/alerts writes as ResourceID for both (pbs.ID, pmg.ID).
The platform tables already matched it; the drawers no longer depend on a
metrics target being set to find the same alerts. Inside a guest drawer, a
physical disk's agent temperature alerts (agent:<id>/disk_temp:<device>) are
matched through the guest's own agent id. No alert is created, cleared or
retyped by this; it changes which open alerts a drawer lists.
Alert Investigate became a Patrol run on 2026-10-22, when the browser
Assistant chat window was retired. Active alert cards and History rows keep one
Investigate action (InvestigateAlertButton), which starts a targeted Patrol
check through POST /api/ai/patrol/run with the alert's resource ID,
identifier, and type, then confirms with a toast that points at the Patrol
page. It is offered only to sessions that route accepts, read from the new
settingsCapabilities.patrolRun security-status field. The "Ask Pulse Assistant" icon, the incident-timeline Assistant hand-off,
and their hand-off models (alertAssistantHandoffModel.ts,
incidentAssistantHandoffModel.ts, IncidentAssistantHandoffButton.tsx) are
gone, as is the chat-only alertTargetTypes.ts resolver. The alerts.assistant.*
catalog keys became alerts.investigate.* in English, German, and Spanish.