Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep watchdog Refresh working when a status read hangs
rcourtman32ef1d366d38
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.mdindex 2cf5789a5a..d580a5e488 100644--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@ -3048,7 +3048,11 @@ seconds. A moving age over a status read once at mount would claim theheartbeat stopped while Pulse kept pinging every minute, and a status that isnever re-read kept its read-time age through a real stall, which is the onecase this panel exists to show. A background re-read leaves the Refresh control-alone, and a failed one keeps the last status and marks it unavailable.+alone, and a failed one keeps the last status and marks it unavailable. The+status read has no timeout, so reads carry a sequence number and only the+newest one applies: a read that never settles cannot hold the panel on an old+status or swallow a later Refresh, the next background read goes out once it has+been pending a full interval, and a late answer from it is discarded.The delivery log's relative hover time on each row reads the same clock: alogged attempt's time never changes, so its age is correct over a log that isonly re-read on demand.diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex efe7ea8740..35397fac76 100644--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@ -5218,7 +5218,9 @@ not re-read stays the age at read time, because a moving age over a snapshotthat never refreshes claims the reading stopped when it may not have. Such asurface either re-reads the snapshot in the background, as Proxmox replicationand the external watchdog panel do, or keeps the read-time age, as the Patrol-attention detail does for Last seen. An immutable event time (when a delivery+attention detail does for Last seen. A background re-read applies only its+newest answer, so a read that never settles cannot pin the surface to an older+snapshot while its ages keep moving. An immutable event time (when a deliverywas attempted, a transition happened, a policy was set) ages correctly over anysnapshot and reads the clock.Read-only metadata badges follow the same primitive-owned shell rule.diff --git a/frontend-modern/browser-verification/e24ba99586e7ea96.json b/frontend-modern/browser-verification/e24ba99586e7ea96.jsonnew file mode 100644index 0000000000..9b91d5a3cd--- /dev/null+++ b/frontend-modern/browser-verification/e24ba99586e7ea96.json@@ -0,0 +1,36 @@+{+ "version": 1,+ "base_sha": "278c403d9674cb2eb2066707637e88770636658b",+ "verified_at": "2026-10-06T16:44:58Z",+ "result": "passed",+ "changed_paths": [+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx"+ ],+ "content_sha256": {+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx": "8df8585fbe2838b19227a80af8304cdfdd022b31f73f9deada8cb235bc73d8a2"+ },+ "routes": [+ "/alerts/destinations"+ ],+ "viewports": [+ {+ "width": 1440,+ "height": 900+ },+ {+ "width": 390,+ "height": 844+ }+ ],+ "states": [+ "Private mock stack (backend :7731 built from this clone, Vite :5251 serving this branch) with Playwright page.clock installed before load and /api/alerts/deadman/status intercepted: the mount read answers a healthy status 10 minutes old, the first background read hangs until released and then answers a status 2 hours old, every later read answers a fresh status",+ "After one 31-second poll the hung background read is pending; Refresh still issues a third read and applies it: Last success 10 mins ago -> 34s ago, Refresh enabled",+ "Releasing the hung read's 2-hour-old answer leaves Last success at 34s ago (stale answer discarded), and the next poll issues a fourth read",+ "Control with this file swapped back to the previous commit under the running Vite: Refresh did nothing (no third read, still 10 mins ago) and the hung read's stale answer won (2 hours ago)",+ "No page errors at either viewport"+ ],+ "interactions": [+ "Loaded /alerts/destinations at 1440x900 and 390x844, ran the page clock 31 seconds, clicked Refresh, released the hung read, ran another 30 seconds",+ "Repeated with the previous file as the control"+ ]+}diff --git a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsxindex b6e5f53ee1..fb916c129a 100644--- a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx+++ b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx@@ -58,22 +58,43 @@ export function AlertDeadManDestinationSection(props: AlertDeadManDestinationSecconst [unavailable, setUnavailable] = createSignal(false);const [showUrl, setShowUrl] = createSignal(false);- let backgroundLoad = false;+ // The status read has no timeout, so every read gets a sequence number and+ // only the newest one applies: a read that never settles cannot hold the+ // panel on an old status or swallow a later Refresh. A background read+ // waits while the newest read is younger than one poll interval and leaves+ // the Refresh button alone; a failed read keeps the last status and marks it+ // unavailable. Nothing applies after the panel unmounts.+ let latestRequest = 0;+ let pendingSince: number | undefined;+ let disposed = false;+ onCleanup(() => {+ disposed = true;+ });const loadStatus = async (options: { background?: boolean } = {}) => {- if (loading() || backgroundLoad) return;- // A background re-read leaves the Refresh button alone; a failed one keeps- // the last status and marks it unavailable.- if (options.background) backgroundLoad = true;- else setLoading(true);+ if (options.background) {+ if (pendingSince !== undefined && Date.now() - pendingSince < DEAD_MAN_STATUS_POLL_MS) {+ return;+ }+ } else if (loading()) {+ return;+ }+ const request = ++latestRequest;+ pendingSince = Date.now();+ if (!options.background) setLoading(true);try {- setStatus(await AlertsAPI.getDeadManStatus());+ const next = await AlertsAPI.getDeadManStatus();+ if (disposed || request !== latestRequest) return;+ setStatus(next);setUnavailable(false);} catch (error) {+ if (disposed || request !== latestRequest) return;logger.error('Failed to load external watchdog status', error);setUnavailable(true);} finally {- if (options.background) backgroundLoad = false;- else setLoading(false);+ if (!disposed && request === latestRequest) {+ pendingSince = undefined;+ setLoading(false);+ }}};diff --git a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsxindex 17ba5f8165..8cc27ad159 100644--- a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx+++ b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx@@ -154,4 +154,56 @@ describe('AlertDeadManDestinationSection', () => {vi.useRealTimers();}});++ it('keeps Refresh working and ignores a stale answer when a background read hangs', async () => {+ const start = Date.parse('2026-08-27T12:00:30Z');+ vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });+ const base = await AlertsAPI.getDeadManStatus();+ vi.mocked(AlertsAPI.getDeadManStatus).mockClear();+ let releaseHungRead: (value: typeof base) => void = () => undefined;+ vi.mocked(AlertsAPI.getDeadManStatus)+ .mockResolvedValueOnce(base)+ .mockImplementationOnce(+ () =>+ new Promise((resolve) => {+ releaseHungRead = resolve;+ }),+ )+ .mockResolvedValue({ ...base, lastSuccessAt: '2026-08-27T12:00:55Z' });+ const [pingUrl, setPingUrl] = createSignal('***REDACTED***');++ try {+ render(() => (+ <AlertDeadManDestinationSection+ pingUrl={pingUrl}+ setPingUrl={setPingUrl}+ setHasUnsavedChanges={vi.fn()}+ />+ ));+ const lastSuccess = () => screen.getByText('Last success').nextElementSibling;+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('30s ago'));++ // The first background read never settles.+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(2));++ // Refresh still reads and applies the newer status.+ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(3));+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('5s ago'));+ expect(screen.getByRole('button', { name: 'Refresh' })).not.toBeDisabled();++ // The hung read finally answers with the older status; it must not win.+ releaseHungRead(base);+ await Promise.resolve();+ await Promise.resolve();+ expect(lastSuccess()).toHaveTextContent('5s ago');++ // Background reads keep going after the hang.+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(4));+ } finally {+ vi.useRealTimers();+ }+ });});