Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Settle slow polled reads and keep replication links honest

rcourtmanba26b3581e2b
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
index 710f4ef07e..2bcd810d94 100644
--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md
+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
@@ -4649,6 +4649,21 @@ Each query run receives an `AbortSignal`; changing the source, resetting the
query, or unmounting its owner must abort the superseded browser request before
starting replacement work. Consumers must forward that signal through their
API/cache layer when the transport supports cancellation.
+A poll (`pollMs`) does not replace a read that an earlier poll started while
+that read is still in flight; it skips the tick. The API answers 429 with
+`Retry-After: 60` and `apiFetch` honours up to two minutes, so a polled read
+slower than the poll interval would otherwise be discarded on every tick and
+never settle. Any other read (a refresh, a source change, a remount
+revalidation) stays replaceable at the next tick, so a slow or hung refresh is
+replaced at most once before polling protects its replacement. A polled read
+still in flight at the first poll tick at or after `STALLED_QUERY_READ_MS` (150
+seconds) is presumed stalled: that tick replaces it and settles it as a failed
+read, clearing `loading`, setting `resolvedOnce` and publishing a timed-out
+`error`, so no consumer is left waiting on a read that never returns. Each
+consumer renders that failure as it renders any other (the Proxmox Replication
+table shows its error and Retry; workload sparklines keep retained series). A
+transport that always takes longer than that deadline plus one poll interval
+never publishes.
The settings reporting shell now also owns a deliberate split between
historical performance reports and current-state VM inventory export.
`frontend-modern/src/components/Settings/ReportingPanel.tsx`,
@@ -5210,9 +5225,10 @@ cells: a value formatted through `formatPlatformTableRelativeTimeValue` or
and tooltips, an open drawer's summary rows and annotations) passes `now` from
the owning component's `useRelativeTimeNow`, and a countdown or freshness band
derived from the same time (replication Next sync, backup age bands) reads it
-too. The clock can trail the wall clock by up to one tick, so a check that
-treats a future time as invalid (the backup age band) measures from the later
-of the two.
+too. Every read of the clock returns the wall clock; the 30-second tick only
+tells readers to re-read, so a cell that mounts between ticks never measures
+from a stale time and a timestamp from the last few seconds never reads as a
+future time.
Read-only metadata badges follow the same primitive-owned shell rule.
`frontend-modern/src/components/shared/MetadataBadge.tsx` owns filled and
outlined appearances, compact sizing, shape, typed tone vocabulary, fit
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 626b08f763..6d712201d4 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -986,15 +986,21 @@ composition, and visible source/state chips must render through `MetadataBadge`
instead of restoring local rounded-sm xs badge spans.
Backup ages read the frontend-primitives shared relative-time clock:
`ProxmoxBackupAgeText` measures the full and compact age text and its
-current, aging, or stale band from `useRelativeTimeNow`, never earlier than the
-wall clock, so a backup row that stays mounted for days moves from current to
-aging without new inventory, and a backup that finished inside the clock's last
-tick never bands as a future, unknown age. `ProxmoxPageSurface` reads
+current, aging, or stale band from `useRelativeTimeNow`, so a backup row that
+stays mounted for days moves from current to aging without new inventory, and
+because every clock read returns the wall clock, a backup that finished since
+the last tick never bands as a future, unknown age. `ProxmoxPageSurface` reads
replication jobs through `createNonSuspendingQuery` with a 30-second background
poll, because the Replication table's Last sync and Next sync move on the same
-clock: a failed read reports the error but keeps the last jobs and the tab, and
-a direct replication link holds until the first read resolves instead of
-falling back to Overview. The poll re-reads the existing read-only
+clock. Each read forwards the query's abort signal, so a replaced or unmounted
+read is cancelled. Whether replication exists is unknown until a read
+succeeds, counted from the last reset (an organization switch): a failed read
+before then holds a direct replication link on the error and its Retry instead
+of falling back to Overview, and once a read has succeeded a failed poll
+reports the error but keeps that answer (the jobs and the tab, or the Overview
+fallback for an empty answer) rather than reopening the route. An access
+denial (401 or 403) withdraws the jobs and the tab, and a direct link holds on
+the denial so the operator sees why. The poll re-reads the existing read-only
`/api/replication/jobs` snapshot and opens no new storage or recovery path.
Proxmox backup-health and recoverable-artifact summary rows follow the same
platform-table density contract as every other provider surface. Storage and
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 5c1dfdb1bb..d8f30a735c 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -1274,9 +1274,14 @@ tooltip, and the Proxmox replication Last sync and Next sync in the row, phone
projection and disclosure. A silent agent or a stalled pvesr scheduler is the
case whose age matters most, so a replication Next sync turns overdue on the
clock without new data. Replication jobs bypass the unified-resource stream,
-so `ProxmoxPageSurface` re-reads them in the background every 30 seconds;
-moving ages over a snapshot read once at mount would age a job that keeps
-syncing and count it overdue.
+so `ProxmoxPageSurface` polls them in the background every 30 seconds,
+skipping a tick while the previous poll's read is still in flight unless that
+read has stalled past `STALLED_QUERY_READ_MS`; moving ages over a snapshot read
+once at mount would age a job that keeps syncing and count it overdue.
+`fetchReplicationJobs` takes the query's abort signal and passes it to
+`apiFetch`, so a replaced or unmounted replication read stops instead of
+running on in the background, and it throws `apiErrorFromResponse` errors that
+keep the HTTP status, so a 401 or 403 withdraws the jobs as an access failure.
Kubernetes name columns hold the chevron, status dot and name inside one
track, so their md widths leave the name room for a typical node or service
name at a 768px viewport: Nodes 20 percent (with Capacity at 16 so its
diff --git a/frontend-modern/browser-verification/0a3f56b9810b4db5.json b/frontend-modern/browser-verification/0a3f56b9810b4db5.json
new file mode 100644
index 0000000000..70d8f76336
--- /dev/null
+++ b/frontend-modern/browser-verification/0a3f56b9810b4db5.json
@@ -0,0 +1,52 @@
+{
+ "version": 1,
+ "base_sha": "9aaac4d3e502c01913fb3637d0997b0ce03a8b7b",
+ "verified_at": "2026-10-06T19:14:07Z",
+ "result": "passed",
+ "changed_paths": [
+ "frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx",
+ "frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx",
+ "frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx",
+ "frontend-modern/src/hooks/createNonSuspendingQuery.ts",
+ "frontend-modern/src/utils/relativeTimeClock.ts"
+ ],
+ "content_sha256": {
+ "frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx": "306a0f3a95585efd4d980549f33a41538a00975f62aff39873a75145e4d9bf5f",
+ "frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx": "e56e51e972457dbfe34f5695b69423f1ac916df3213a61d028e05cbcecfd12bd",
+ "frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx": "33fd51603010d13040faa6fe6b3690d3e4bbe1c941c32ceb1aa0c972ec47370c",
+ "frontend-modern/src/hooks/createNonSuspendingQuery.ts": "6780323e7d9055b0839da57b6204fd68bfce9696e622362ea34a02d0907554bd",
+ "frontend-modern/src/utils/relativeTimeClock.ts": "0ff094e2dc98bf07b3792af478bfa90d3156143ddc5aef0e233de80868ec4dba"
+ },
+ "routes": [
+ "/proxmox/replication",
+ "/proxmox/backups",
+ "/docker/overview",
+ "/standalone/machines"
+ ],
+ "viewports": [
+ {
+ "width": 1600,
+ "height": 1000
+ },
+ {
+ "width": 390,
+ "height": 844
+ }
+ ],
+ "states": [
+ "Private mock stack built from this tree on origin/main 9aaac4d3e5; Playwright drove it with /api/replication/jobs answers shaped per scenario through page.route",
+ "Direct /proxmox/replication link whose first read returns 500, at 1600px and 390px: stays on the route with \"Could not load replication jobs\" and Refresh (main's code fell back to Overview with no error); Refresh loads all 8 jobs",
+ "Jobs loaded, then a poll read answers 403: the Replication tab is withdrawn and the link shows the server's \"Insufficient permissions to read replication jobs\" with Refresh (main kept the tab advertised and a generic message)",
+ "Empty first answer, then a failed poll, then an empty answer: the link stays on Overview throughout and the route never reopens",
+ "page.clock installed, a polled replication read held open: the next two 30s poll ticks start no new request and abort none; releasing it lets it finish and the following tick starts the next read (main started reads 3 and 4 over the held read and never cancelled it)",
+ "Leaving the Proxmox page for Docker while a replication read is held aborts it (net::ERR_ABORTED); main let it run",
+ "Regression: with /ws frames held and keep-alive pongs, Machines Seen cells move from \"just now\" to \"10m ago\" over ten one-minute clock steps on one socket",
+ "Proxmox Backups at 1600px and 390px after the clock change retired the band's Math.max: 36 age cells each, none Unknown, every band colour matching its Current/Aging/Stale title, no page errors",
+ "Same-session control: the five runtime files swapped back to main under the running Vite reproduced every failure above, then restored and compared byte for byte"
+ ],
+ "interactions": [
+ "Clicked Refresh on the replication error state at both widths",
+ "Clicked the Docker nav link while a replication read was in flight",
+ "page.clock.runFor poll steps with data-free pong frames sent before each step"
+ ]
+}
diff --git a/frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailSummary.relativeAge.test.tsx b/frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailSummary.relativeAge.test.tsx
index bf17540823..786914a867 100644
--- a/frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailSummary.relativeAge.test.tsx
+++ b/frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailSummary.relativeAge.test.tsx
@@ -4,6 +4,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest';
import { formatPlatformTableDateTimeValue } from '@/features/platformPage/sharedPlatformPage';
import type { Resource } from '@/types/resource';
+import { RELATIVE_TIME_TICK_MS } from '@/utils/relativeTimeClock';
import { InlineResourceSummaryTables } from '../ResourceDetailSummary';
import { useResourceDetailDrawerDerivedState } from '../useResourceDetailDrawerDerivedState';
import type { UseResourceDetailDrawerStateResult } from '../useResourceDetailDrawerState';
@@ -47,6 +48,12 @@ afterEach(() => {
vi.useRealTimers();
});
+// Time passes with no new data; the next clock tick re-reads the ages.
+const hoursLater = (hours: number) => {
+ vi.setSystemTime(Date.now() + hours * 60 * 60 * 1000);
+ vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
+};
+
describe('resource drawer summary relative ages', () => {
it('keeps an open Kubernetes controller summary age moving without new data', () => {
vi.useFakeTimers({ now: new Date('2026-05-24T13:31:00Z') });
@@ -73,7 +80,7 @@ describe('resource drawer summary relative ages', () => {
within(section()).getByText(`${absoluteTime('2026-05-24T13:00:00Z')} (31m ago)`),
).toBeInTheDocument();
- vi.advanceTimersByTime(2 * 60 * 60 * 1000);
+ hoursLater(2);
// The absolute time is unchanged; only the age after it moves.
expect(
@@ -95,6 +102,7 @@ describe('resource drawer summary relative ages', () => {
containerState: 'running',
createdAt: '2026-05-24T11:31:00Z',
startedAt: '2026-05-24T13:21:00Z',
+ finishedAt: '2026-05-24T12:31:00Z',
},
})}
drawer={drawerState}
@@ -108,14 +116,16 @@ describe('resource drawer summary relative ages', () => {
.closest('tr');
expect(row('Created')).toHaveTextContent('2 hours ago');
expect(row('Started')).toHaveTextContent('10 mins ago');
+ expect(row('Finished')).toHaveTextContent('1 hour ago');
- vi.advanceTimersByTime(3 * 60 * 60 * 1000);
+ hoursLater(3);
expect(row('Created')).toHaveTextContent('5 hours ago');
expect(row('Started')).toHaveTextContent('3 hours ago');
+ expect(row('Finished')).toHaveTextContent('4 hours ago');
});
- it('keeps an open drawer Last seen moving for a resource that stops reporting', () => {
+ it('keeps an open drawer Last seen and platform freshness ages moving without new data', () => {
vi.useFakeTimers({ now: new Date('2026-05-24T13:31:00Z') });
const { state, dispose } = createRoot((dispose) => ({
@@ -124,7 +134,11 @@ describe('resource drawer summary relative ages', () => {
id: 'node-1',
type: 'agent',
lastSeen: Date.parse('2026-05-24T13:30:00Z'),
- }),
+ platformData: {
+ docker: { updatesLastCheckedAt: '2026-05-24T13:01:00Z' },
+ pmg: { lastUpdated: '2026-05-24T13:21:00Z' },
+ },
+ } as Partial<Resource> & Pick<Resource, 'id' | 'type'>),
debugEnabled: () => false,
discoveryFeatureEnabled: () => false,
resourceIntelligence: () => null,
@@ -133,11 +147,15 @@ describe('resource drawer summary relative ages', () => {
}));
expect(state.lastSeen()).toBe('1 min ago');
+ expect(state.dockerUpdatesCheckedRelative()).toBe('30 mins ago');
+ expect(state.pmgUpdatedRelative()).toBe('10 mins ago');
- // The resource stays silent: its lastSeen never changes again.
- vi.advanceTimersByTime(2 * 60 * 60 * 1000);
+ // The resource stays silent: none of its timestamps change again.
+ hoursLater(2);
expect(state.lastSeen()).toBe('2 hours ago');
+ expect(state.dockerUpdatesCheckedRelative()).toBe('2 hours ago');
+ expect(state.pmgUpdatedRelative()).toBe('2 hours ago');
dispose();
});
});
diff --git a/frontend-modern/src/features/kubernetes/__tests__/KubernetesControllersTable.relativeAge.test.tsx b/frontend-modern/src/features/kubernetes/__tests__/KubernetesControllersTable.relativeAge.test.tsx
index 7b547bf664..4f740ca6e7 100644
--- a/frontend-modern/src/features/kubernetes/__tests__/KubernetesControllersTable.relativeAge.test.tsx
+++ b/frontend-modern/src/features/kubernetes/__tests__/KubernetesControllersTable.relativeAge.test.tsx
@@ -17,6 +17,7 @@ vi.mock('@/api/resources', () => ({
}));
import type { Resource } from '@/types/resource';
+import { RELATIVE_TIME_TICK_MS } from '@/utils/relativeTimeClock';
import { KubernetesControllersTable } from '../KubernetesControllersTable';
const makeResource = ({
@@ -85,7 +86,8 @@ describe('KubernetesControllersTable relative ages', () => {
// No data changes: the timestamps stay put and only the clock moves. A
// detail computed once at render would still read "31m ago" here.
- vi.advanceTimersByTime(3 * 60 * 60 * 1000);
+ vi.setSystemTime(Date.parse('2026-05-24T16:31:00Z'));
+ vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
expect(screen.getByText('Completed 3h ago')).toBeInTheDocument();
expect(screen.getByText('Last success 3h ago')).toBeInTheDocument();
diff --git a/frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx b/frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx
index 118fd48185..0cf4d4c4ba 100644
--- a/frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx
+++ b/frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx
@@ -52,6 +52,7 @@ import { ProxmoxMailGatewayTable } from './ProxmoxMailGatewayTable';
import { ProxmoxNodesTable } from './ProxmoxNodesTable';
import { ProxmoxReplicationTable, fetchReplicationJobs } from './ProxmoxReplicationTable';
import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery';
+import { getAPIReadAccessErrorMessage } from '@/utils/apiAccessError';
import { useUnifiedResources } from '@/hooks/useUnifiedResources';
import type { ReplicationJob } from '@/types/api';
import type { Resource } from '@/types/resource';
@@ -127,11 +128,22 @@ export function ProxmoxPageSurface() {
// last jobs, so the tab does not vanish under the user.
const replicationJobs = createNonSuspendingQuery({
source: () => 'proxmox-pve',
- fetcher: () => fetchReplicationJobs(),
+ fetcher: (_source, signal) => fetchReplicationJobs(signal),
initialValue: NO_REPLICATION_JOBS,
pollMs: REPLICATION_JOBS_POLL_MS,
});
const replicationJobCount = createMemo(() => replicationJobs.value().length);
+ // Replication presence is known once a read has succeeded. A failed read
+ // before that says nothing, so a direct link holds on its error and Retry
+ // instead of falling to Overview; a failed poll after that keeps the last
+ // answer rather than reopening a route an empty read already closed. An
+ // access denial withdraws that answer, so the link holds on the denial.
+ const replicationJobsConfirmed = createMemo<boolean>((confirmed) => {
+ if (!replicationJobs.resolvedOnce()) return false;
+ const error = replicationJobs.error();
+ if (getAPIReadAccessErrorMessage(error)) return false;
+ return confirmed || !error;
+ }, false);
const visibleTabs = createMemo(() => {
// An unknown snapshot is not evidence that every optional integration is
// present. Never use estate-wide aggregations here: unrelated VMware VMs
@@ -149,7 +161,7 @@ export function ProxmoxPageSurface() {
// Do not discard a direct link while counts are still unknown: its own
// resource query must be allowed to hydrate before deciding it is absent.
const countsUnknown =
- !tabEvidence.facets?.() || (requested === 'replication' && !replicationJobs.resolvedOnce());
+ !tabEvidence.facets?.() || (requested === 'replication' && !replicationJobsConfirmed());
return countsUnknown || visibleTabIds().has(requested) ? requested : 'overview';
});
const shouldHydrateTab = (tab: ProxmoxPageTabId) => activeTab() === tab;
diff --git a/frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx b/frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx
index 918f4d8486..5a7e714cf4 100644
--- a/frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx
+++ b/frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx
@@ -5,7 +5,7 @@ import { InlineDetailTableRow } from '@/components/shared/InlineDetailTableRow';
import { StatusDot } from '@/components/shared/StatusDot';
import type { StatusIndicatorVariant } from '@/utils/status';
import { TableCell, TableHead, TableRow } from '@/components/shared/Table';
-import { apiFetch } from '@/utils/apiClient';
+import { apiErrorFromResponse, apiFetch } from '@/utils/apiClient';
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
import {
PlatformTableToolbar,
@@ -264,10 +264,15 @@ export const compactReplicationNextSyncText = (text: string, tone: NextSyncTone)
return text.replace(/^in /, '');
};
-export async function fetchReplicationJobs(): Promise<ReplicationJob[]> {
- const response = await apiFetch('/api/replication/jobs?platform=proxmox-pve');
+export async function fetchReplicationJobs(signal?: AbortSignal): Promise<ReplicationJob[]> {
+ const response = await apiFetch('/api/replication/jobs?platform=proxmox-pve', { signal });
if (!response.ok) {
- throw new Error(`Failed to load replication jobs (${response.status})`);
+ // A structured error keeps the HTTP status, so a 401/403 withdraws the
+ // jobs as an access failure instead of being retained as an outage.
+ throw await apiErrorFromResponse(
+ response,
+ `Failed to load replication jobs (${response.status})`,
+ );
}
const payload = (await response.json()) as ReplicationJobsResponse;
return Array.isArray(payload?.data) ? payload.data : [];
diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxPageSurface.contract.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxPageSurface.contract.test.tsx
index 30583297ab..0979b3a487 100644
--- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxPageSurface.contract.test.tsx
+++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxPageSurface.contract.test.tsx
@@ -1,4 +1,4 @@
-import { cleanup, render, screen, waitFor } from '@solidjs/testing-library';
+import { cleanup, fireEvent, render, screen, waitFor } from '@solidjs/testing-library';
import { Route, Router } from '@solidjs/router';
import { createSignal } from 'solid-js';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
@@ -21,7 +21,9 @@ const mockWorkloadSearch = vi.hoisted(() => vi.fn(() => ''));
const mockSelectedNode = vi.hoisted(() => vi.fn<() => string | null>(() => null));
const mockHandleNodeSelect = vi.hoisted(() => vi.fn());
const mockWorkloadsOptions = vi.hoisted(() => vi.fn());
-const mockFetchReplicationJobs = vi.hoisted(() => vi.fn(() => Promise.resolve([] as unknown[])));
+const mockFetchReplicationJobs = vi.hoisted(() =>
+ vi.fn((_signal?: AbortSignal) => Promise.resolve([] as unknown[])),
+);
const makeResource = (resource: Partial<Resource> & Pick<Resource, 'id' | 'type'>): Resource =>
({
@@ -153,14 +155,18 @@ vi.mock('../ProxmoxNodesTable', () => ({
}));
vi.mock('../ProxmoxReplicationTable', () => ({
- ProxmoxReplicationTable: (props: { jobs?: unknown[]; error?: unknown }) => (
+ ProxmoxReplicationTable: (props: { jobs?: unknown[]; error?: unknown; onRetry: () => void }) => (
<div
data-testid="replication-table"
data-jobs={props.jobs === undefined ? 'loading' : props.jobs.length}
data-error={props.error ? 'yes' : 'no'}
- />
+ >
+ <button type="button" onClick={() => props.onRetry()}>
+ Retry replication
+ </button>
+ </div>
),
- fetchReplicationJobs: () => mockFetchReplicationJobs(),
+ fetchReplicationJobs: (signal?: AbortSignal) => mockFetchReplicationJobs(signal),
}));
const renderSurface = () =>
@@ -698,6 +704,141 @@ describe('ProxmoxPageSurface contract', () => {
}
});
+ it('holds a direct replication link on a failed first read and recovers on retry', async () => {
+ mockPathname.mockReturnValue('/proxmox/replication');
+ const node = makeResource({ id: 'node:pve1', type: 'agent', name: 'pve1' });
+ mockUseUnifiedResources.mockReturnValue({
+ resources: () => [node],
+ facets: () => ({ incidentCount: 0, byType: {} }),
+ loading: () => false,
+ error: () => null,
+ refetch: vi.fn(async () => []),
+ });
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockRejectedValueOnce(new Error('replication read failed'));
+ mockFetchReplicationJobs.mockResolvedValueOnce([{ id: 'job-1' }]);
+
+ try {
+ renderSurface();
+ const tabs = screen.getByTestId('platform-section-tabs');
+ // A failed read says nothing about whether replication exists, so the
+ // link keeps its error and Retry instead of falling back to Overview.
+ await waitFor(() =>
+ expect(screen.getByTestId('replication-table')).toHaveAttribute('data-error', 'yes'),
+ );
+ expect(tabs).toHaveAttribute('data-active', 'replication');
+ expect(tabs).toHaveAttribute('data-tabs', 'overview');
+
+ fireEvent.click(screen.getByRole('button', { name: 'Retry replication' }));
+ await waitFor(() =>
+ expect(screen.getByTestId('replication-table')).toHaveAttribute('data-jobs', '1'),
+ );
+ expect(screen.getByTestId('replication-table')).toHaveAttribute('data-error', 'no');
+ expect(tabs).toHaveAttribute('data-tabs', 'overview,replication');
+ expect(tabs).toHaveAttribute('data-active', 'replication');
+ } finally {
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockImplementation(() => Promise.resolve([]));
+ }
+ });
+
+ it('keeps an empty replication answer through a failed poll instead of reopening the route', async () => {
+ vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
+ mockPathname.mockReturnValue('/proxmox/replication');
+ const node = makeResource({ id: 'node:pve1', type: 'agent', name: 'pve1' });
+ mockUseUnifiedResources.mockReturnValue({
+ resources: () => [node],
+ facets: () => ({ incidentCount: 0, byType: {} }),
+ loading: () => false,
+ error: () => null,
+ refetch: vi.fn(async () => []),
+ });
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockResolvedValueOnce([]);
+ mockFetchReplicationJobs.mockRejectedValueOnce(new Error('replication read failed'));
+ mockFetchReplicationJobs.mockResolvedValueOnce([]);
+
+ try {
+ renderSurface();
+ const tabs = screen.getByTestId('platform-section-tabs');
+ // An empty answer means no replication: the link falls back to Overview.
+ await waitFor(() => expect(tabs).toHaveAttribute('data-active', 'overview'));
+
+ vi.advanceTimersByTime(30_000);
+ await waitFor(() => expect(mockFetchReplicationJobs).toHaveBeenCalledTimes(2));
+ await Promise.resolve();
+ expect(tabs).toHaveAttribute('data-active', 'overview');
+ expect(screen.queryByTestId('replication-table')).not.toBeInTheDocument();
+
+ vi.advanceTimersByTime(30_000);
+ await waitFor(() => expect(mockFetchReplicationJobs).toHaveBeenCalledTimes(3));
+ expect(tabs).toHaveAttribute('data-active', 'overview');
+ } finally {
+ vi.useRealTimers();
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockImplementation(() => Promise.resolve([]));
+ }
+ });
+
+ it('holds a direct replication link on an access denial that withdraws loaded jobs', async () => {
+ vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
+ mockPathname.mockReturnValue('/proxmox/replication');
+ const node = makeResource({ id: 'node:pve1', type: 'agent', name: 'pve1' });
+ mockUseUnifiedResources.mockReturnValue({
+ resources: () => [node],
+ facets: () => ({ incidentCount: 0, byType: {} }),
+ loading: () => false,
+ error: () => null,
+ refetch: vi.fn(async () => []),
+ });
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockResolvedValueOnce([{ id: 'job-1' }]);
+ mockFetchReplicationJobs.mockRejectedValueOnce(
+ Object.assign(new Error('Insufficient permissions'), { status: 403 }),
+ );
+
+ try {
+ renderSurface();
+ const tabs = screen.getByTestId('platform-section-tabs');
+ await waitFor(() =>
+ expect(screen.getByTestId('replication-table')).toHaveAttribute('data-jobs', '1'),
+ );
+
+ vi.advanceTimersByTime(30_000);
+ await waitFor(() =>
+ expect(screen.getByTestId('replication-table')).toHaveAttribute('data-error', 'yes'),
+ );
+ // The denied jobs are withdrawn, so the tab goes; the link shows why.
+ expect(tabs).toHaveAttribute('data-tabs', 'overview');
+ expect(tabs).toHaveAttribute('data-active', 'replication');
+ } finally {
+ vi.useRealTimers();
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockImplementation(() => Promise.resolve([]));
+ }
+ });
+
+ it('cancels an in-flight replication read when the page unmounts', async () => {
+ mockPathname.mockReturnValue('/proxmox/replication');
+ setResources([makeResource({ id: 'node:pve1', type: 'agent', name: 'pve1' })]);
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockReturnValueOnce(new Promise<unknown[]>(() => undefined));
+
+ try {
+ const view = renderSurface();
+ expect(mockFetchReplicationJobs).toHaveBeenCalledTimes(1);
+ const signal = mockFetchReplicationJobs.mock.calls[0][0];
+ expect(signal).toBeInstanceOf(AbortSignal);
+ expect(signal?.aborted).toBe(false);
+
+ view.unmount();
+ expect(signal?.aborted).toBe(true);
+ } finally {
+ mockFetchReplicationJobs.mockReset();
+ mockFetchReplicationJobs.mockImplementation(() => Promise.resolve([]));
+ }
+ });
+
it('does not surface stale-agent notices for development builds without an agent target', () => {
mockVersionInfo.mockReturnValue({
version: '6.0.0-rc.6+git.172.g2c360f779.dirty',
diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxReplicationTable.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxReplicationTable.test.tsx
index 8e9f45cdec..938f07c21f 100644
--- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxReplicationTable.test.tsx
+++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxReplicationTable.test.tsx
@@ -7,6 +7,7 @@ import {
REPLICATION_MOBILE_COLUMNS,
REPLICATION_MOBILE_COLUMN_WIDTHS,
compactReplicationNextSyncText,
+ fetchReplicationJobs,
formatMobileReplicationGuestLabel,
} from '../ProxmoxReplicationTable';
import replicationTableSource from '../ProxmoxReplicationTable.tsx?raw';
@@ -134,6 +135,30 @@ describe('ProxmoxReplicationTable', () => {
});
});
+describe('fetchReplicationJobs', () => {
+ it('passes the caller abort signal to the network request', async () => {
+ const fetchSpy = vi
+ .spyOn(globalThis, 'fetch')
+ .mockResolvedValue(new Response(JSON.stringify({ data: [], meta: { total: 0 } })));
+ const controller = new AbortController();
+
+ await expect(fetchReplicationJobs(controller.signal)).resolves.toEqual([]);
+
+ const init = fetchSpy.mock.calls.find(([url]) =>
+ String(url).includes('/api/replication/jobs'),
+ )?.[1];
+ expect(init?.signal).toBe(controller.signal);
+ });
+
+ it('keeps the HTTP status on a failed read so access denials are recognised', async () => {
+ vi.spyOn(globalThis, 'fetch').mockResolvedValue(
+ new Response(JSON.stringify({ message: 'Insufficient permissions' }), { status: 403 }),
+ );
+
+ await expect(fetchReplicationJobs()).rejects.toMatchObject({ status: 403 });
+ });
+});
+
describe('ProxmoxReplicationTable relative times', () => {
const start = Date.parse('2026-10-04T12:00:00Z');
// Last synced 5 minutes before the table opens, next sync due 10 minutes after.
diff --git a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx
index 2d6ec3ca25..ab23ae24f3 100644
--- a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx
+++ b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx
@@ -242,6 +242,27 @@ describe('backup age cell on the shared clock', () => {
);
});
+ it('keeps the full age text and its band in step between clock ticks', () => {
+ vi.useFakeTimers({ now: new Date('2026-10-04T12:00:00Z') });
+ // Another mounted cell starts the shared clock.
+ render(() => <ProxmoxBackupAgeText artifact={artifact()} />);
+
+ // Before the next tick, a backup crosses seven days old as its row mounts.
+ vi.setSystemTime(new Date('2026-10-11T12:00:10Z'));
+ const createdAt = '2026-10-04T12:00:00Z';
+ render(() => (
+ <span data-testid="boundary">
+ <ProxmoxBackupAgeText
+ artifact={artifact({ createdAt, createdMs: Date.parse(createdAt) })}
+ />
+ </span>
+ ));
+
+ const cell = within(screen.getByTestId('boundary')).getByTitle(/backup age/);
+ expect(cell).toHaveTextContent('7d ago');
+ expect(cell).toHaveClass('text-amber-600');
+ });
+
it('does not band a backup that finished after the last clock tick as unknown', () => {
vi.useFakeTimers({ now: new Date('2026-10-04T12:00:00Z') });
// Another mounted cell starts the shared clock at 12:00:00.
diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx b/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx
index 4828f3f754..099548660f 100644
--- a/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx
+++ b/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx
@@ -233,11 +233,8 @@ export function formatCompactBackupAge(createdAt: string, now?: number): string
export function ProxmoxBackupAgeText(props: { artifact: RecoverableArtifact; compact?: boolean }) {
// Backup rows stay mounted while an artifact's creation time never changes,
- // so the age and its freshness band both read the shared clock. The clock
- // can trail the wall clock by up to one tick, and a backup that finished
- // inside that window must not band as a future (unknown) age.
- const tickNow = useRelativeTimeNow();
- const now = () => Math.max(tickNow(), Date.now());
+ // so the age and its freshness band both read the shared clock.
+ const now = useRelativeTimeNow();
const band = () => getRecoveryAgeBand(props.artifact.createdMs, now());
const title = () => {
if (band() === 'unknown') return recoveryAgeTitleByBand.unknown;
diff --git a/frontend-modern/src/hooks/__tests__/createNonSuspendingQuery.test.tsx b/frontend-modern/src/hooks/__tests__/createNonSuspendingQuery.test.tsx
index ce8b4af4b4..95d0cc4c82 100644
--- a/frontend-modern/src/hooks/__tests__/createNonSuspendingQuery.test.tsx
+++ b/frontend-modern/src/hooks/__tests__/createNonSuspendingQuery.test.tsx
@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest';
import { cleanup, fireEvent, render, screen, waitFor } from '@solidjs/testing-library';
import { createSignal } from 'solid-js';
import {
+ STALLED_QUERY_READ_MS,
createNonSuspendingQuery,
getCreateNonSuspendingQueryCacheDiagnosticsForTest,
resetCreateNonSuspendingQueryCacheForTest,
@@ -312,6 +313,111 @@ it.each(['success', 'failure'] as const)(
},
);
+it('lets a background read slower than the poll interval settle', async () => {
+ vi.useFakeTimers();
+ // The second read takes 40s, for example while apiFetch waits out a 429
+ // Retry-After, and the query polls every 30s.
+ const fetcher = vi
+ .fn()
+ .mockResolvedValueOnce('first')
+ .mockImplementationOnce(
+ () => new Promise<string>((resolve) => setTimeout(() => resolve('second'), 40_000)),
+ )
+ .mockResolvedValue('third');
+ const Probe = () => {
+ const query = createNonSuspendingQuery({
+ source: () => 'replication',
+ fetcher,
+ initialValue: '',
+ pollMs: 30_000,
+ });
+ return <output data-testid="query">{query.value()}</output>;
+ };
+ render(() => <Probe />);
+ await vi.advanceTimersByTimeAsync(0);
+ expect(screen.getByTestId('query')).toHaveTextContent('first');
+
+ // The 30s poll starts the slow read; the 60s poll must not discard it.
+ await vi.advanceTimersByTimeAsync(60_000);
+ expect(fetcher).toHaveBeenCalledTimes(2);
+ expect(fetcher.mock.calls[1][1].aborted).toBe(false);
+
+ await vi.advanceTimersByTimeAsync(10_000);
+ expect(screen.getByTestId('query')).toHaveTextContent('second');
+
+ // Polling resumes on the next tick once nothing is in flight.
+ await vi.advanceTimersByTimeAsync(20_000);
+ expect(fetcher).toHaveBeenCalledTimes(3);
+ expect(screen.getByTestId('query')).toHaveTextContent('third');
+});
+
+it('drops a stalled read loading state even when its replacement also hangs', async () => {
+ vi.useFakeTimers();
+ const hang = () => new Promise<string>(() => undefined);
+ const fetcher = vi.fn().mockResolvedValueOnce('first').mockImplementation(hang);
+ const Probe = () => {
+ const query = createNonSuspendingQuery({
+ source: () => 'pbs-host',
+ fetcher,
+ initialValue: '',
+ pollMs: 30_000,
+ });
+ return (
+ <>
+ <button onClick={() => void query.refetch()}>Refresh</button>
+ <output data-testid="query">{`${query.value()}|loading:${query.loading()}`}</output>
+ </>
+ );
+ };
+ render(() => <Probe />);
+ await vi.advanceTimersByTimeAsync(0);
+ fireEvent.click(screen.getByRole('button', { name: 'Refresh' }));
+ expect(screen.getByTestId('query')).toHaveTextContent('first|loading:true');
+
+ // The refresh never returns, so the next poll replaces it; that read never
+ // returns either, and later polls leave it alone until it has stalled.
+ await vi.advanceTimersByTimeAsync(30_000);
+ expect(fetcher).toHaveBeenCalledTimes(3);
+ expect(fetcher.mock.calls[1][1].aborted).toBe(true);
+ await vi.advanceTimersByTimeAsync(STALLED_QUERY_READ_MS - 30_000);
+ expect(fetcher).toHaveBeenCalledTimes(3);
+ expect(screen.getByTestId('query')).toHaveTextContent('first|loading:true');
+
+ await vi.advanceTimersByTimeAsync(30_000);
+ expect(fetcher).toHaveBeenCalledTimes(4);
+ expect(fetcher.mock.calls[2][1].aborted).toBe(true);
+ expect(screen.getByTestId('query')).toHaveTextContent('first|loading:false');
+});
+
+it('reports a never-loaded query as failed once its polled read stalls', async () => {
+ vi.useFakeTimers();
+ // Every read hangs, from the first one on.
+ const fetcher = vi.fn().mockImplementation(() => new Promise<string>(() => undefined));
+ const Probe = () => {
+ const query = createNonSuspendingQuery({
+ source: () => 'replication',
+ fetcher,
+ initialValue: '',
+ pollMs: 30_000,
+ });
+ return (
+ <output data-testid="query">{`resolved:${query.resolvedOnce()}|error:${
+ (query.error() as Error | null)?.message ?? ''
+ }|loading:${query.loading()}`}</output>
+ );
+ };
+ render(() => <Probe />);
+ await vi.advanceTimersByTimeAsync(30_000);
+ expect(fetcher).toHaveBeenCalledTimes(2);
+ expect(screen.getByTestId('query')).toHaveTextContent('resolved:false|error:|loading:true');
+
+ await vi.advanceTimersByTimeAsync(STALLED_QUERY_READ_MS);
+ expect(fetcher).toHaveBeenCalledTimes(3);
+ expect(screen.getByTestId('query')).toHaveTextContent(
+ 'resolved:true|error:The request timed out.|loading:false',
+ );
+});
+
const settle = async () => {
await Promise.resolve();
await Promise.resolve();
diff --git a/frontend-modern/src/hooks/createNonSuspendingQuery.ts b/frontend-modern/src/hooks/createNonSuspendingQuery.ts
index 920e0de316..0ef3a1a43c 100644
--- a/frontend-modern/src/hooks/createNonSuspendingQuery.ts
+++ b/frontend-modern/src/hooks/createNonSuspendingQuery.ts
@@ -26,6 +26,15 @@ interface QueryRunOptions {
*/
const RETAINED_QUERY_CACHE_MAX_ENTRIES = 64;
const RETAINED_QUERY_CACHE_MAX_AGE_MS = 5 * 60_000;
+// A poll does not replace a read an earlier poll started while it is still in
+// flight: a read slower than the poll interval (the API answers 429 with
+// Retry-After: 60, and apiFetch honours up to two minutes) would otherwise be
+// discarded on every tick and never settle. Other reads stay replaceable at
+// the next tick, so a slow refresh is replaced at most once. Past this age a
+// polled read is presumed stalled: the next poll replaces it and reports it as
+// a failed read, so a consumer that never loaded shows its error and Retry
+// rather than spinning while every read hangs.
+export const STALLED_QUERY_READ_MS = 150_000;
type RetainedQueryCacheEntry = {
cachedAt: number;
@@ -179,6 +188,8 @@ export function createNonSuspendingQuery<T, K>(options: CreateNonSuspendingQuery
let latestRequestId = 0;
let activeAbortController: AbortController | null = null;
+ let activeRequestStartedAt = 0;
+ let activeRequestIsPoll = false;
const cancelActiveRequest = () => {
activeAbortController?.abort();
@@ -209,10 +220,12 @@ export function createNonSuspendingQuery<T, K>(options: CreateNonSuspendingQuery
});
onCleanup(unsubscribeLiveQueryOrgSwitch);
- const run = async (key: K, runOptions: QueryRunOptions = {}): Promise<T> => {
+ const run = async (key: K, runOptions: QueryRunOptions = {}, poll = false): Promise<T> => {
cancelActiveRequest();
const abortController = new AbortController();
activeAbortController = abortController;
+ activeRequestStartedAt = Date.now();
+ activeRequestIsPoll = poll;
const requestId = ++latestRequestId;
const retainedCacheKey = getRetainedCacheKey(key);
const cacheGeneration = retainedQueryCacheGeneration;
@@ -296,7 +309,23 @@ export function createNonSuspendingQuery<T, K>(options: CreateNonSuspendingQuery
if (typeof options.pollMs === 'number' && options.pollMs > 0) {
const interval = setInterval(() => {
- void refetch({ background: true });
+ if (activeAbortController !== null && activeRequestIsPoll) {
+ if (Date.now() - activeRequestStartedAt < STALLED_QUERY_READ_MS) return;
+ // The stalled read is abandoned and settles as a failure: its
+ // replacement runs in the background and may hang too, so nothing
+ // else would clear a spinner or a never-loaded state.
+ batch(() => {
+ setLoading(false);
+ setError(new Error('The request timed out.'));
+ setResolvedOnce(true);
+ });
+ }
+ const key = options.source();
+ if (key === null) {
+ reset();
+ return;
+ }
+ void run(key, { background: true }, true);
}, options.pollMs);
onCleanup(() => clearInterval(interval));
}
diff --git a/frontend-modern/src/utils/__tests__/relativeTimeClock.test.ts b/frontend-modern/src/utils/__tests__/relativeTimeClock.test.ts
index 5a92bbd0a3..ae65a87f5c 100644
--- a/frontend-modern/src/utils/__tests__/relativeTimeClock.test.ts
+++ b/frontend-modern/src/utils/__tests__/relativeTimeClock.test.ts
@@ -21,6 +21,22 @@ describe('useRelativeTimeNow', () => {
expect(vi.getTimerCount()).toBe(0);
});
+ it('reads the wall clock between ticks, so a late reader never trails it', () => {
+ vi.useFakeTimers({ now: new Date('2026-10-06T10:00:00Z') });
+
+ const first = createRoot((dispose) => ({ now: useRelativeTimeNow(), dispose }));
+ // A second cell mounts 20s after the clock started, before the next tick.
+ vi.setSystemTime(new Date('2026-10-06T10:00:20Z'));
+ const second = createRoot((dispose) => ({ now: useRelativeTimeNow(), dispose }));
+
+ expect(second.now()).toBe(Date.parse('2026-10-06T10:00:20Z'));
+ expect(first.now()).toBe(second.now());
+
+ first.dispose();
+ second.dispose();
+ expect(vi.getTimerCount()).toBe(0);
+ });
+
it('reads the time directly outside a reactive owner', () => {
vi.useFakeTimers({ now: new Date('2026-10-06T10:00:00Z') });
const now = useRelativeTimeNow();
diff --git a/frontend-modern/src/utils/relativeTimeClock.ts b/frontend-modern/src/utils/relativeTimeClock.ts
index 43cd334e7e..0957a98ca1 100644
--- a/frontend-modern/src/utils/relativeTimeClock.ts
+++ b/frontend-modern/src/utils/relativeTimeClock.ts
@@ -7,19 +7,26 @@ import { createSignal, getOwner, onCleanup } from 'solid-js';
// when nothing does.
export const RELATIVE_TIME_TICK_MS = 30_000;
-const [now, setNow] = createSignal(Date.now());
+// The tick only tells readers to re-read; every read returns the wall clock.
+// A cell that mounts between ticks would otherwise measure from up to one tick
+// ago, so a timestamp from the last few seconds could read as in the future
+// and two cells formatted from the same time could disagree.
+const [tick, setTick] = createSignal(0);
+const now = (): number => {
+ tick();
+ return Date.now();
+};
let readers = 0;
let timer: ReturnType<typeof setInterval> | undefined;
-// Inside a component or reactive root, the returned accessor is the shared
-// ticking signal. Outside one there is nothing to stop a timer, so it reads the
-// time directly.
+// Inside a component or reactive root, the returned accessor tracks the shared
+// tick. Outside one there is nothing to stop a timer, so it reads the time
+// directly.
export function useRelativeTimeNow(): () => number {
if (!getOwner()) return () => Date.now();
readers += 1;
if (timer === undefined) {
- setNow(Date.now());
- timer = setInterval(() => setNow(Date.now()), RELATIVE_TIME_TICK_MS);
+ timer = setInterval(() => setTick((count) => count + 1), RELATIVE_TIME_TICK_MS);
}
onCleanup(() => {
readers -= 1;