Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep watchdog Refresh working when a status read hangs

rcourtman32ef1d366d38
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md
index 2cf5789a5a..d580a5e488 100644
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@ -3048,7 +3048,11 @@ seconds. A moving age over a status read once at mount would claim the
heartbeat stopped while Pulse kept pinging every minute, and a status that is
never re-read kept its read-time age through a real stall, which is the one
case this panel exists to show. A background re-read leaves the Refresh control
-alone, and a failed one keeps the last status and marks it unavailable.
+alone, and a failed one keeps the last status and marks it unavailable. The
+status read has no timeout, so reads carry a sequence number and only the
+newest one applies: a read that never settles cannot hold the panel on an old
+status or swallow a later Refresh, the next background read goes out once it has
+been pending a full interval, and a late answer from it is discarded.
The delivery log's relative hover time on each row reads the same clock: a
logged attempt's time never changes, so its age is correct over a log that is
only re-read on demand.
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
index efe7ea8740..35397fac76 100644
--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md
+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
@@ -5218,7 +5218,9 @@ not re-read stays the age at read time, because a moving age over a snapshot
that never refreshes claims the reading stopped when it may not have. Such a
surface either re-reads the snapshot in the background, as Proxmox replication
and the external watchdog panel do, or keeps the read-time age, as the Patrol
-attention detail does for Last seen. An immutable event time (when a delivery
+attention detail does for Last seen. A background re-read applies only its
+newest answer, so a read that never settles cannot pin the surface to an older
+snapshot while its ages keep moving. An immutable event time (when a delivery
was attempted, a transition happened, a policy was set) ages correctly over any
snapshot and reads the clock.
Read-only metadata badges follow the same primitive-owned shell rule.
diff --git a/frontend-modern/browser-verification/e24ba99586e7ea96.json b/frontend-modern/browser-verification/e24ba99586e7ea96.json
new file mode 100644
index 0000000000..9b91d5a3cd
--- /dev/null
+++ b/frontend-modern/browser-verification/e24ba99586e7ea96.json
@@ -0,0 +1,36 @@
+{
+ "version": 1,
+ "base_sha": "278c403d9674cb2eb2066707637e88770636658b",
+ "verified_at": "2026-10-06T16:44:58Z",
+ "result": "passed",
+ "changed_paths": [
+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx"
+ ],
+ "content_sha256": {
+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx": "8df8585fbe2838b19227a80af8304cdfdd022b31f73f9deada8cb235bc73d8a2"
+ },
+ "routes": [
+ "/alerts/destinations"
+ ],
+ "viewports": [
+ {
+ "width": 1440,
+ "height": 900
+ },
+ {
+ "width": 390,
+ "height": 844
+ }
+ ],
+ "states": [
+ "Private mock stack (backend :7731 built from this clone, Vite :5251 serving this branch) with Playwright page.clock installed before load and /api/alerts/deadman/status intercepted: the mount read answers a healthy status 10 minutes old, the first background read hangs until released and then answers a status 2 hours old, every later read answers a fresh status",
+ "After one 31-second poll the hung background read is pending; Refresh still issues a third read and applies it: Last success 10 mins ago -> 34s ago, Refresh enabled",
+ "Releasing the hung read's 2-hour-old answer leaves Last success at 34s ago (stale answer discarded), and the next poll issues a fourth read",
+ "Control with this file swapped back to the previous commit under the running Vite: Refresh did nothing (no third read, still 10 mins ago) and the hung read's stale answer won (2 hours ago)",
+ "No page errors at either viewport"
+ ],
+ "interactions": [
+ "Loaded /alerts/destinations at 1440x900 and 390x844, ran the page clock 31 seconds, clicked Refresh, released the hung read, ran another 30 seconds",
+ "Repeated with the previous file as the control"
+ ]
+}
diff --git a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
index b6e5f53ee1..fb916c129a 100644
--- a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
+++ b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
@@ -58,22 +58,43 @@ export function AlertDeadManDestinationSection(props: AlertDeadManDestinationSec
const [unavailable, setUnavailable] = createSignal(false);
const [showUrl, setShowUrl] = createSignal(false);
- let backgroundLoad = false;
+ // The status read has no timeout, so every read gets a sequence number and
+ // only the newest one applies: a read that never settles cannot hold the
+ // panel on an old status or swallow a later Refresh. A background read
+ // waits while the newest read is younger than one poll interval and leaves
+ // the Refresh button alone; a failed read keeps the last status and marks it
+ // unavailable. Nothing applies after the panel unmounts.
+ let latestRequest = 0;
+ let pendingSince: number | undefined;
+ let disposed = false;
+ onCleanup(() => {
+ disposed = true;
+ });
const loadStatus = async (options: { background?: boolean } = {}) => {
- if (loading() || backgroundLoad) return;
- // A background re-read leaves the Refresh button alone; a failed one keeps
- // the last status and marks it unavailable.
- if (options.background) backgroundLoad = true;
- else setLoading(true);
+ if (options.background) {
+ if (pendingSince !== undefined && Date.now() - pendingSince < DEAD_MAN_STATUS_POLL_MS) {
+ return;
+ }
+ } else if (loading()) {
+ return;
+ }
+ const request = ++latestRequest;
+ pendingSince = Date.now();
+ if (!options.background) setLoading(true);
try {
- setStatus(await AlertsAPI.getDeadManStatus());
+ const next = await AlertsAPI.getDeadManStatus();
+ if (disposed || request !== latestRequest) return;
+ setStatus(next);
setUnavailable(false);
} catch (error) {
+ if (disposed || request !== latestRequest) return;
logger.error('Failed to load external watchdog status', error);
setUnavailable(true);
} finally {
- if (options.background) backgroundLoad = false;
- else setLoading(false);
+ if (!disposed && request === latestRequest) {
+ pendingSince = undefined;
+ setLoading(false);
+ }
}
};
diff --git a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx
index 17ba5f8165..8cc27ad159 100644
--- a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx
+++ b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx
@@ -154,4 +154,56 @@ describe('AlertDeadManDestinationSection', () => {
vi.useRealTimers();
}
});
+
+ it('keeps Refresh working and ignores a stale answer when a background read hangs', async () => {
+ const start = Date.parse('2026-08-27T12:00:30Z');
+ vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });
+ const base = await AlertsAPI.getDeadManStatus();
+ vi.mocked(AlertsAPI.getDeadManStatus).mockClear();
+ let releaseHungRead: (value: typeof base) => void = () => undefined;
+ vi.mocked(AlertsAPI.getDeadManStatus)
+ .mockResolvedValueOnce(base)
+ .mockImplementationOnce(
+ () =>
+ new Promise((resolve) => {
+ releaseHungRead = resolve;
+ }),
+ )
+ .mockResolvedValue({ ...base, lastSuccessAt: '2026-08-27T12:00:55Z' });
+ const [pingUrl, setPingUrl] = createSignal('***REDACTED***');
+
+ try {
+ render(() => (
+ <AlertDeadManDestinationSection
+ pingUrl={pingUrl}
+ setPingUrl={setPingUrl}
+ setHasUnsavedChanges={vi.fn()}
+ />
+ ));
+ const lastSuccess = () => screen.getByText('Last success').nextElementSibling;
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('30s ago'));
+
+ // The first background read never settles.
+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);
+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(2));
+
+ // Refresh still reads and applies the newer status.
+ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));
+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(3));
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('5s ago'));
+ expect(screen.getByRole('button', { name: 'Refresh' })).not.toBeDisabled();
+
+ // The hung read finally answers with the older status; it must not win.
+ releaseHungRead(base);
+ await Promise.resolve();
+ await Promise.resolve();
+ expect(lastSuccess()).toHaveTextContent('5s ago');
+
+ // Background reads keep going after the hang.
+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);
+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(4));
+ } finally {
+ vi.useRealTimers();
+ }
+ });
});