Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Apply watchdog reads in order and free a stalled Refresh

rcourtmanb81f0478d3d7
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md
index 4e1d591548..ca02ea2373 100644
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@ -3174,10 +3174,12 @@ heartbeat stopped while Pulse kept pinging every minute, and a status that is
never re-read kept its read-time age through a real stall, which is the one
case this panel exists to show. A background re-read leaves the Refresh control
alone, and a failed one keeps the last status and marks it unavailable. The
-status read has no timeout, so reads carry a sequence number and only the
-newest one applies: a read that never settles cannot hold the panel on an old
-status or swallow a later Refresh, the next background read goes out once it has
-been pending a full interval, and a late answer from it is discarded.
+status read has no timeout, so reads carry a sequence number and an answer
+applies only when it is newer than the last one applied: a slow or hung read can
+neither overwrite a newer status nor block later reads, reads slower than the
+poll still land in order, the next background read goes out at the first poll
+once the newest has been pending a full interval, and a Refresh still pending at
+such a poll frees the button (within two intervals of the click).
The delivery log's relative hover time on each row reads the same clock: a
logged attempt's time never changes, so its age is correct over a log that is
only re-read on demand.
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
index 9e3afd89f9..9ba9f2d3e8 100644
--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md
+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
@@ -5219,9 +5219,10 @@ not re-read stays the age at read time, because a moving age over a snapshot
that never refreshes claims the reading stopped when it may not have. Such a
surface either re-reads the snapshot in the background, as Proxmox replication
and the external watchdog panel do, or keeps the read-time age, as the Patrol
-attention detail does for Last seen. A background re-read applies only its
-newest answer, so a read that never settles cannot pin the surface to an older
-snapshot while its ages keep moving. An immutable event time (when a delivery
+attention detail does for Last seen. A background re-read applies an answer
+only when it is newer than the last one applied, so a slow or hung read can
+neither pin the surface to an older snapshot nor overwrite a newer one while
+its ages keep moving. An immutable event time (when a delivery
was attempted, a transition happened, a policy was set) ages correctly over any
snapshot and reads the clock.
Read-only metadata badges follow the same primitive-owned shell rule.
diff --git a/frontend-modern/browser-verification/0b7a94ad4faa1eb7.json b/frontend-modern/browser-verification/0b7a94ad4faa1eb7.json
new file mode 100644
index 0000000000..0886986878
--- /dev/null
+++ b/frontend-modern/browser-verification/0b7a94ad4faa1eb7.json
@@ -0,0 +1,38 @@
+{
+ "version": 1,
+ "base_sha": "337e908b3577351ece5c4ffdf51ff17380e09b4e",
+ "verified_at": "2026-10-06T19:43:25Z",
+ "result": "passed",
+ "changed_paths": [
+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx"
+ ],
+ "content_sha256": {
+ "frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx": "022a2f4b9412890a66420f18bda6105400c29c6deedf185dc102eccc063323d0"
+ },
+ "routes": [
+ "/alerts/destinations"
+ ],
+ "viewports": [
+ {
+ "width": 1440,
+ "height": 900
+ },
+ {
+ "width": 390,
+ "height": 844
+ }
+ ],
+ "states": [
+ "Private mock stack (backend :7731, Vite :5251 serving this branch, merged with main 9aaac4d3e5) with Playwright page.clock installed before load and every /api/alerts/deadman/status read held until the script answers it",
+ "Reads slower than the poll: each poll starts a read while the previous is pending; answers applied in order (10 mins ago -> 3 mins ago -> 1 min ago), and an older read answered after a newer one did not overwrite it (stayed 1 min ago); every poll tick issued a read (no tick skipped to timer jitter)",
+ "A Refresh that never settles, followed by a background read that also never settles: Refresh showed Refreshing\u2026 disabled, then was enabled again at the first poll after it had been pending a full interval, and a second Refresh applied the fresh status",
+ "Earlier hang case still holds: with a hung background read, Refresh applies the fresh status and the hung read's late 2-hour-old answer is discarded",
+ "Control with the previous loader under the running Vite: slow reads never showed a status (every answer discarded) and the hung Refresh stayed disabled",
+ "No page errors at either viewport"
+ ],
+ "interactions": [
+ "Loaded /alerts/destinations at 1440x900 and 390x844, ran the page clock in 30-second steps answering held reads in and out of order",
+ "Clicked Refresh with the read held, ran the clock 61 seconds, clicked Refresh again and answered it",
+ "Repeated with the previous loader as the control"
+ ]
+}
diff --git a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
index fb916c129a..8450336946 100644
--- a/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
+++ b/frontend-modern/src/features/alerts/AlertDeadManDestinationSection.tsx
@@ -18,6 +18,10 @@ const REDACTED_PING_URL = '***REDACTED***';
// Pulse pings the watchdog every minute, so a status read once at mount goes
// stale while the panel stays open.
export const DEAD_MAN_STATUS_POLL_MS = 30_000;
+// A read pending this long at a poll tick counts as a full interval old. The
+// slack absorbs timer jitter: the next tick can land a few milliseconds short
+// of a full interval after the read the previous tick started.
+const DEAD_MAN_STATUS_STALE_MS = DEAD_MAN_STATUS_POLL_MS - 1_000;
const statusPresentation: Record<DeadManStatus['state'], { label: string; class: string }> = {
disabled: {
@@ -58,42 +62,58 @@ export function AlertDeadManDestinationSection(props: AlertDeadManDestinationSec
const [unavailable, setUnavailable] = createSignal(false);
const [showUrl, setShowUrl] = createSignal(false);
- // The status read has no timeout, so every read gets a sequence number and
- // only the newest one applies: a read that never settles cannot hold the
- // panel on an old status or swallow a later Refresh. A background read
- // waits while the newest read is younger than one poll interval and leaves
- // the Refresh button alone; a failed read keeps the last status and marks it
- // unavailable. Nothing applies after the panel unmounts.
- let latestRequest = 0;
+ // The status read has no timeout. Every read gets a sequence number and an
+ // answer applies only when it is newer than the last one applied, so a slow
+ // or hung read can neither overwrite a newer status nor block later reads,
+ // and reads slower than the poll interval still land in order. A background
+ // read waits while the newest read is younger than one poll interval; a
+ // Refresh still pending at such a poll counts as stalled and frees the
+ // button. Only
+ // the newest read's failure marks the status unavailable, and nothing
+ // applies after the panel unmounts.
+ let latestStarted = 0;
+ let latestApplied = 0;
let pendingSince: number | undefined;
+ let foregroundRequest = 0;
+ let foregroundSince = 0;
let disposed = false;
onCleanup(() => {
disposed = true;
});
const loadStatus = async (options: { background?: boolean } = {}) => {
+ const startedAt = Date.now();
if (options.background) {
- if (pendingSince !== undefined && Date.now() - pendingSince < DEAD_MAN_STATUS_POLL_MS) {
+ if (loading() && startedAt - foregroundSince >= DEAD_MAN_STATUS_STALE_MS) {
+ setLoading(false);
+ }
+ if (pendingSince !== undefined && startedAt - pendingSince < DEAD_MAN_STATUS_STALE_MS) {
return;
}
} else if (loading()) {
return;
}
- const request = ++latestRequest;
- pendingSince = Date.now();
- if (!options.background) setLoading(true);
+ const request = ++latestStarted;
+ pendingSince = startedAt;
+ if (!options.background) {
+ foregroundRequest = request;
+ foregroundSince = startedAt;
+ setLoading(true);
+ }
try {
const next = await AlertsAPI.getDeadManStatus();
- if (disposed || request !== latestRequest) return;
+ if (disposed || request < latestApplied) return;
+ latestApplied = request;
setStatus(next);
setUnavailable(false);
+ if (request >= foregroundRequest) setLoading(false);
} catch (error) {
- if (disposed || request !== latestRequest) return;
+ if (disposed || request !== latestStarted) return;
logger.error('Failed to load external watchdog status', error);
setUnavailable(true);
} finally {
- if (!disposed && request === latestRequest) {
- pendingSince = undefined;
- setLoading(false);
+ if (!disposed) {
+ if (request === latestStarted) pendingSince = undefined;
+ if (request === foregroundRequest) setLoading(false);
}
}
};
diff --git a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx
index 8cc27ad159..1740cab1aa 100644
--- a/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx
+++ b/frontend-modern/src/features/alerts/__tests__/AlertDeadManDestinationSection.test.tsx
@@ -206,4 +206,98 @@ describe('AlertDeadManDestinationSection', () => {
vi.useRealTimers();
}
});
+
+ it('applies reads slower than the poll in order and never lets an older answer win', async () => {
+ const start = Date.parse('2026-08-27T12:10:00Z');
+ vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });
+ const base = await AlertsAPI.getDeadManStatus();
+ vi.mocked(AlertsAPI.getDeadManStatus).mockReset();
+ const pending: Array<(value: typeof base) => void> = [];
+ vi.mocked(AlertsAPI.getDeadManStatus).mockImplementation(
+ () =>
+ new Promise((resolve) => {
+ pending.push(resolve);
+ }),
+ );
+ const at = (time: string) => ({ ...base, lastSuccessAt: time });
+ const [pingUrl, setPingUrl] = createSignal('***REDACTED***');
+
+ try {
+ render(() => (
+ <AlertDeadManDestinationSection
+ pingUrl={pingUrl}
+ setPingUrl={setPingUrl}
+ setHasUnsavedChanges={vi.fn()}
+ />
+ ));
+ await waitFor(() => expect(pending).toHaveLength(1));
+
+ // Every read takes longer than one poll interval.
+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);
+ await waitFor(() => expect(pending).toHaveLength(2));
+ pending[0](at('2026-08-27T12:00:00Z'));
+ const lastSuccess = () => screen.getByText('Last success').nextElementSibling;
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('10 mins ago'));
+
+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);
+ await waitFor(() => expect(pending).toHaveLength(3));
+ pending[1](at('2026-08-27T12:10:00Z'));
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('1 min ago'));
+
+ // A newer read answers before an older one: the older answer is dropped.
+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);
+ await waitFor(() => expect(pending).toHaveLength(4));
+ pending[3](at('2026-08-27T12:11:30Z'));
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('0s ago'));
+ pending[2](at('2026-08-27T11:00:00Z'));
+ await Promise.resolve();
+ await Promise.resolve();
+ expect(lastSuccess()).toHaveTextContent('0s ago');
+ } finally {
+ vi.useRealTimers();
+ }
+ });
+
+ it('frees a Refresh that hangs for a whole poll interval', async () => {
+ const start = Date.parse('2026-08-27T12:00:30Z');
+ vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });
+ const base = await AlertsAPI.getDeadManStatus();
+ vi.mocked(AlertsAPI.getDeadManStatus).mockReset();
+ vi.mocked(AlertsAPI.getDeadManStatus)
+ .mockResolvedValueOnce(base)
+ .mockImplementationOnce(() => new Promise(() => undefined))
+ .mockImplementationOnce(() => new Promise(() => undefined))
+ .mockResolvedValue({ ...base, lastSuccessAt: '2026-08-27T12:00:55Z' });
+ const [pingUrl, setPingUrl] = createSignal('***REDACTED***');
+
+ try {
+ render(() => (
+ <AlertDeadManDestinationSection
+ pingUrl={pingUrl}
+ setPingUrl={setPingUrl}
+ setHasUnsavedChanges={vi.fn()}
+ />
+ ));
+ const lastSuccess = () => screen.getByText('Last success').nextElementSibling;
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('30s ago'));
+
+ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));
+ await waitFor(() =>
+ expect(screen.getByRole('button', { name: 'Refreshing…' })).toBeDisabled(),
+ );
+
+ // The Refresh read never settles, and neither does the background read
+ // that the next poll starts. The button is usable again after one poll
+ // interval, and a second Refresh brings the newer status.
+ vi.advanceTimersByTime(DEAD_MAN_STATUS_POLL_MS);
+ await waitFor(() => expect(AlertsAPI.getDeadManStatus).toHaveBeenCalledTimes(3));
+ await waitFor(() =>
+ expect(screen.getByRole('button', { name: 'Refresh' })).not.toBeDisabled(),
+ );
+ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));
+ await waitFor(() => expect(lastSuccess()).toHaveTextContent('5s ago'));
+ } finally {
+ vi.useRealTimers();
+ }
+ });
});