Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Apply watchdog reads in order and free a stalled Refresh
rcourtmanb81f0478d3d7
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.mdindex 4e1d591548..ca02ea2373 100644--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@ -3174,10 +3174,12 @@ heartbeat stopped while Pulse kept pinging every minute, and a status that isnever re-read kept its read-time age through a real stall, which is the onecase this panel exists to show. A background re-read leaves the Refresh controlalone, and a failed one keeps the last status and marks it unavailable. The-status read has no timeout, so reads carry a sequence number and only the-newest one applies: a read that never settles cannot hold the panel on an old-status or swallow a later Refresh, the next background read goes out once it has-been pending a full interval, and a late answer from it is discarded.+status read has no timeout, so reads carry a sequence number and an answer+applies only when it is newer than the last one applied: a slow or hung read can+neither overwrite a newer status nor block later reads, reads slower than the+poll still land in order, the next background read goes out at the first poll+once the newest has been pending a full interval, and a Refresh still pending at+such a poll frees the button (within two intervals of the click).The delivery log's relative hover time on each row reads the same clock: alogged attempt's time never changes, so its age is correct over a log that isonly re-read on demand.diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex 9e3afd89f9..9ba9f2d3e8 100644--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@ -5219,9 +5219,10 @@ not re-read stays the age at read time, because a moving age over a snapshotthat never refreshes claims the reading stopped when it may not have. Such asurface either re-reads the snapshot in the background, as Proxmox replicationand the external watchdog panel do, or keeps the read-time age, as the Patrol-attention detail does for Last seen. A background re-read applies only its-newest answer, so a read that never settles cannot pin the surface to an older-snapshot while its ages keep moving. An immutable event time (when a delivery+attention detail does for Last seen. A background re-read applies an answer+only when it is newer than the last one applied, so a slow or hung read can+neither pin the surface to an older snapshot nor overwrite a newer one while+its ages keep moving. An immutable event time (when a deliverywas attempted, a transition happened, a policy was set) ages correctly over anysnapshot and reads the clock.Read-only metadata badges follow the same primitive-owned shell rule.diff --git a/frontend-modern/browser-verification/0b7a94ad4faa1eb7.json b/frontend-modern/browser-verification/0b7a94ad4faa1eb7.jsonnew file mode 100644index 0000000000..0886986878--- /dev/null+++ b/frontend-modern/browser-verification/0b7a94ad4faa1eb7.json@@ -0,0 +1,38 @@+{+ "version": 1,+ "base_sha": "337e908b3577351ece5c4ffdf51ff17380e09b4e",+ "verified_at": "2026-10-06T19:43:25Z",+ "result": "passed",+ "changed_paths": [+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx"+ ],+ "content_sha256": {+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx": "022a2f4b9412890a66420f18bda6105400c29c6deedf185dc102eccc063323d0"+ },+ "routes": [+ "/alerts/destinations"+ ],+ "viewports": [+ {+ "width": 1440,+ "height": 900+ },+ {+ "width": 390,+ "height": 844+ }+ ],+ "states": [+ "Private mock stack (backend :7731, Vite :5251 serving this branch, merged with main 9aaac4d3e5) with Playwright page.clock installed before load and every /api/alerts/deadman/status read held until the script answers it",+ "Reads slower than the poll: each poll starts a read while the previous is pending; answers applied in order (10 mins ago -> 3 mins ago -> 1 min ago), and an older read answered after a newer one did not overwrite it (stayed 1 min ago); every poll tick issued a read (no tick skipped to timer jitter)",+ "A Refresh that never settles, followed by a background read that also never settles: Refresh showed Refreshing\u2026 disabled, then was enabled again at the first poll after it had been pending a full interval, and a second Refresh applied the fresh status",+ "Earlier hang case still holds: with a hung background read, Refresh applies the fresh status and the hung read's late 2-hour-old answer is discarded",+ "Control with the previous loader under the running Vite: slow reads never showed a status (every answer discarded) and the hung Refresh stayed disabled",+ "No page errors at either viewport"+ ],+ "interactions": [+ "Loaded /alerts/destinations at 1440x900 and 390x844, ran the page clock in 30-second steps answering held reads in and out of order",+ "Clicked Refresh with the read held, ran the clock 61 seconds, clicked Refresh again and answered it",+ "Repeated with the previous loader as the control"+ ]+}diff --git a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsxindex fb916c129a..8450336946 100644--- a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx+++ b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx@@ -18,6 +18,10 @@ const REDACTED_PING_URL = '***REDACTED***';// Pulse pings the watchdog every minute, so a status read once at mount goes// stale while the panel stays open.export const DEAD_MAN_STATUS_POLL_MS = 30_000;+// A read pending this long at a poll tick counts as a full interval old. The+// slack absorbs timer jitter: the next tick can land a few milliseconds short+// of a full interval after the read the previous tick started.+const DEAD_MAN_STATUS_STALE_MS = DEAD_MAN_STATUS_POLL_MS - 1_000;const statusPresentation: Record<DeadManStatus['state'], { label: string; class: string }> = {disabled: {@@ -58,42 +62,58 @@ export function AlertDeadManDestinationSection(props: AlertDeadManDestinationSecconst [unavailable, setUnavailable] = createSignal(false);const [showUrl, setShowUrl] = createSignal(false);- // The status read has no timeout, so every read gets a sequence number and- // only the newest one applies: a read that never settles cannot hold the- // panel on an old status or swallow a later Refresh. A background read- // waits while the newest read is younger than one poll interval and leaves- // the Refresh button alone; a failed read keeps the last status and marks it- // unavailable. Nothing applies after the panel unmounts.- let latestRequest = 0;+ // The status read has no timeout. Every read gets a sequence number and an+ // answer applies only when it is newer than the last one applied, so a slow+ // or hung read can neither overwrite a newer status nor block later reads,+ // and reads slower than the poll interval still land in order. A background+ // read waits while the newest read is younger than one poll interval; a+ // Refresh still pending at such a poll counts as stalled and frees the+ // button. Only+ // the newest read's failure marks the status unavailable, and nothing+ // applies after the panel unmounts.+ let latestStarted = 0;+ let latestApplied = 0;let pendingSince: number | undefined;+ let foregroundRequest = 0;+ let foregroundSince = 0;let disposed = false;onCleanup(() => {disposed = true;});const loadStatus = async (options: { background?: boolean } = {}) => {+ const startedAt = Date.now();if (options.background) {- if (pendingSince !== undefined && Date.now() - pendingSince < DEAD_MAN_STATUS_POLL_MS) {+ if (loading() && startedAt - foregroundSince >= DEAD_MAN_STATUS_STALE_MS) {+ setLoading(false);+ }+ if (pendingSince !== undefined && startedAt - pendingSince < DEAD_MAN_STATUS_STALE_MS) {return;}} else if (loading()) {return;}- const request = ++latestRequest;- pendingSince = Date.now();- if (!options.background) setLoading(true);+ const request = ++latestStarted;+ pendingSince = startedAt;+ if (!options.background) {+ foregroundRequest = request;+ foregroundSince = startedAt;+ setLoading(true);+ }try {const next = await AlertsAPI.getDeadManStatus();- if (disposed || request !== latestRequest) return;+ if (disposed || request < latestApplied) return;+ latestApplied = request;setStatus(next);setUnavailable(false);+ if (request >= foregroundRequest) setLoading(false);} catch (error) {- if (disposed || request !== latestRequest) return;+ if (disposed || request !== latestStarted) return;logger.error('Failed to load external watchdog status', error);setUnavailable(true);} finally {- if (!disposed && request === latestRequest) {- pendingSince = undefined;- setLoading(false);+ if (!disposed) {+ if (request === latestStarted) pendingSince = undefined;+ if (request === foregroundRequest) setLoading(false);}}};diff --git a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsxindex 8cc27ad159..1740cab1aa 100644--- a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx+++ b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx@@ -206,4 +206,98 @@ describe('AlertDeadManDestinationSection', () => {vi.useRealTimers();}});++ it('applies reads slower than the poll in order and never lets an older answer win', async () => {+ const start = Date.parse('2026-08-27T12:10:00Z');+ vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });+ const base = await AlertsAPI.getDeadManStatus();+ vi.mocked(AlertsAPI.getDeadManStatus).mockReset();+ const pending: Array<(value: typeof base) => void> = [];+ vi.mocked(AlertsAPI.getDeadManStatus).mockImplementation(+ () =>+ new Promise((resolve) => {+ pending.push(resolve);+ }),+ );+ const at = (time: string) => ({ ...base, lastSuccessAt: time });+ const [pingUrl, setPingUrl] = createSignal('***REDACTED***');++ try {+ render(() => (+ <AlertDeadManDestinationSection+ pingUrl={pingUrl}+ setPingUrl={setPingUrl}+ setHasUnsavedChanges={vi.fn()}+ />+ ));+ await waitFor(() => expect(pending).toHaveLength(1));++ // Every read takes longer than one poll interval.+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);+ await waitFor(() => expect(pending).toHaveLength(2));+ pending[0](at('2026-08-27T12:00:00Z'));+ const lastSuccess = () => screen.getByText('Last success').nextElementSibling;+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('10 mins ago'));++ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);+ await waitFor(() => expect(pending).toHaveLength(3));+ pending[1](at('2026-08-27T12:10:00Z'));+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('1 min ago'));++ // A newer read answers before an older one: the older answer is dropped.+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);+ await waitFor(() => expect(pending).toHaveLength(4));+ pending[3](at('2026-08-27T12:11:30Z'));+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('0s ago'));+ pending[2](at('2026-08-27T11:00:00Z'));+ await Promise.resolve();+ await Promise.resolve();+ expect(lastSuccess()).toHaveTextContent('0s ago');+ } finally {+ vi.useRealTimers();+ }+ });++ it('frees a Refresh that hangs for a whole poll interval', async () => {+ const start = Date.parse('2026-08-27T12:00:30Z');+ vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });+ const base = await AlertsAPI.getDeadManStatus();+ vi.mocked(AlertsAPI.getDeadManStatus).mockReset();+ vi.mocked(AlertsAPI.getDeadManStatus)+ .mockResolvedValueOnce(base)+ .mockImplementationOnce(() => new Promise(() => undefined))+ .mockImplementationOnce(() => new Promise(() => undefined))+ .mockResolvedValue({ ...base, lastSuccessAt: '2026-08-27T12:00:55Z' });+ const [pingUrl, setPingUrl] = createSignal('***REDACTED***');++ try {+ render(() => (+ <AlertDeadManDestinationSection+ pingUrl={pingUrl}+ setPingUrl={setPingUrl}+ setHasUnsavedChanges={vi.fn()}+ />+ ));+ const lastSuccess = () => screen.getByText('Last success').nextElementSibling;+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('30s ago'));++ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));+ await waitFor(() =>+ expect(screen.getByRole('button', { name: 'Refreshing…' })).toBeDisabled(),+ );++ // The Refresh read never settles, and neither does the background read+ // that the next poll starts. The button is usable again after one poll+ // interval, and a second Refresh brings the newer status.+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(3));+ await waitFor(() =>+ expect(screen.getByRole('button', { name: 'Refresh' })).not.toBeDisabled(),+ );+ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('5s ago'));+ } finally {+ vi.useRealTimers();+ }+ });});