Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Withdraw cached live history when agent reporting stops
rcourtmanbc5491b6d1ac
diff --cc docs/release-control/v6/internal/subsystems/api-contracts.mdindex abfe9702f0,6ce20ef1c3..86758f19ed--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@@ -47,6 -51,6 +51,14 @@@ HTTP transport, production fleet handleresource/finding fixtures. It covers true/false/omission, composed filters andan empty intersection; this is not released-binary or customer-estate proof.++### Unreported agent delivery supplies no live chart fallback++++A saved enrollment awaiting its first accepted report, including after its++reporting lease expires, remains queryable by its existing metrics target.++History and summary chart routes retain stored series++but cannot synthesize a now-stamped live sample from that enrollment's absent++metrics. An accepted report restores normal live fallback behaviour.++### Proxmox setup guide names the command-access option by its shared label (21 October 2026)The Proxmox node setup guide's Docker-in-LXC note now points to the Pulsediff --cc docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex a126ddd80c,38d99d7e52..6da432bdc2--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@@ -20,18 -20,6 +20,22 @@@## Purpose+### Agent first-report status++Machines, resource detail drawers and agent connection rows use the shared+"Waiting for first report" label for monitoring's bounded pending startup+state. Machines treats that informational state as unknown rather than an outage.+The collapsed machine row shows the wording beside its name and suppresses+cached metric bars until accepted telemetry arrives. Consumers must not infer+the wait from a browser timer or hide a provider failure behind it. The expanded+drawer presents the wait as information and does not synthesize a current+chart reading from absent or cached metrics. Stored history remains available.++The shared history query immediately withdraws cached live replies and current++fallbacks when the owning source loses reporting authority. That transition++starts a fresh read without waiting for the poll, while retaining actual stored++series and rejecting a late live reply from the previous reporting state.+Pending API polls retain their own wording.+### Backup posture failure and retry stay source-local (24 October 2026)The Backups posture notice offers a keyboard-operable retry with the existingdiff --cc docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex 9a4f339938,a769962262..5d5d3692fb--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@@ -15,6 -15,6 +15,14 @@@## Purpose++### History reads follow reporting transitions++++The shared drawer history query refreshes once when source reporting authority++changes, in addition to its existing bounded polling. Ordinary snapshot object++replacement with unchanged target, range, access and reporting state does not++start a read. Stored series keep their shared resource/range cache; live-only++replies cannot remain current while their canonical source is unreported.++### Workloads aggregate disk readings keep their units and presence (14 October 2026)Workload aggregate metrics are percentage points, including values from zerodiff --cc docs/release-control/v6/internal/subsystems/unified-resources.mdindex 145a210e8f,5852f4af2c..b9004a204e--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@@ -23,6 -23,6 +23,15 @@@ and sort the complete canonical change## Purpose++### Drawer history follows current source authority++++Resource drawers withdraw current chart fallbacks and cached live-only replies++when a standalone agent is offline or awaiting its first accepted report.++The delivery transition refreshes the shared history query; actual stored++observations retain their original timestamps and remain readable while that++read settles. The browser does not infer a new lease or reporting verdict, and++guest power state does not override the provider's metric admission rules.++### Websocket-first routes always obtain their REST snapshot (21 October 2026)Action readiness, scoped facets and other REST-only enrichment arrive onlydiff --cc frontend-modern/src/components/Infrastructure/ResourceDetailDrawerOverviewTab.tsxindex b499345e33,ceb7593639..75a188aac7--- a/frontend-modern/src/components/Infrastructure/ResourceDetailDrawerOverviewTab.tsx+++ b/frontend-modern/src/components/Infrastructure/ResourceDetailDrawerOverviewTab.tsx@@@ -429,6 -420,6 +429,10 @@@ export const ResourceDetailDrawerOvervionRangeChange={drawer.setMetricsHistoryRange}enabled={drawer.activeTab() === 'overview'}currentMetrics={drawer.metricsHistoryCurrentMetrics()}++ liveReadingsAvailable={++ !(resource().type === 'agent' && resource().status === 'offline') &&++ !getAgentFirstReportStatusIndicator(resource())++ }deferredMetrics={drawer.metricsHistoryDeferredMetrics()}groups={drawer.metricsHistoryGroups()}alerts={chartAlerts()}diff --cc frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailDrawer.history.test.tsxindex da3472401d,da3472401d..4980b8dfe4--- a/frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailDrawer.history.test.tsx+++ b/frontend-modern/src/components/Infrastructure/__tests__/ResourceDetailDrawer.history.test.tsx@@@ -7,7 -7,7 +7,7 @@@ importwaitFor,} from '@solidjs/testing-library';import { createSignal, Suspense } from 'solid-js';--import { ChartsAPI } from '@/api/charts';++import { ChartsAPI, type AllMetricsHistoryResponse } from '@/api/charts';import { resetCreateNonSuspendingQueryCacheForTest } from '@/hooks/createNonSuspendingQuery';import resourceDetailDrawerShellSource from '@/components/Infrastructure/ResourceDetailDrawer.tsx?raw';@@@ -162,6 -162,6 +162,57 @@@ const baseResource = (overrides: Partia});describe('ResourceDetailDrawer change history section', () => {++ it('withdraws an open drawer live history reply on restart while preserving real stored history', async () => {++ resetCreateNonSuspendingQueryCacheForTest();++ const source = baseResource({++ id: 'restart-history',++ metricsTarget: { resourceType: 'agent', resourceId: 'restart-history' },++ sourceStatus: { agent: { status: 'online' } },++ cpu: { current: 12 },++ });++ const live: AllMetricsHistoryResponse = {++ resourceType: 'agent',++ resourceId: 'restart-history',++ range: '24h',++ start: 1_700_000_000_000,++ end: 1_700_000_060_000,++ source: 'live',++ metrics: { cpu: [{ timestamp: 1_700_000_060_000, value: 12, min: 12, max: 12 }] },++ };++ let resolveHistory!: (value: AllMetricsHistoryResponse) => void;++ const nextHistory = new Promise<AllMetricsHistoryResponse>((resolve) => {++ resolveHistory = resolve;++ });++ const read = vi++ .spyOn(ChartsAPI, 'getMetricsHistory')++ .mockResolvedValueOnce(live)++ .mockReturnValueOnce(nextHistory);++ const [resource, setResource] = createSignal(source);++ const view = render(() => <ResourceDetailDrawer resource={resource()} />);++ try {++ const history = view.container.querySelector('[data-testid="resource-metrics-history"]')!;++ await waitFor(() => expect(history).toHaveTextContent('Current'));++ setResource({++ ...source,++ status: 'unknown',++ sourceStatus: { agent: { status: 'pending' } },++ cpu: undefined,++ });++ await waitFor(() => expect(read).toHaveBeenCalledTimes(2));++ expect(screen.getAllByText('Waiting for first report').length).toBeGreaterThan(0);++ expect(history).not.toHaveTextContent('Current');++ expect(history.querySelector('[data-history-current]')).toBeNull();++ resolveHistory({ ...live, source: 'store' });++ await waitFor(() => expect(history).toHaveTextContent('Latest stored'));++ expect(history).toHaveTextContent('12.0%');++ expect(history).not.toHaveTextContent('Current');++ } finally {++ view.unmount();++ read.mockRestore();++ resetCreateNonSuspendingQueryCacheForTest();++ }++ });++it('keeps retained guest memory and paused filesystems out of current chart fallbacks', async () => {resetCreateNonSuspendingQueryCacheForTest();const source = baseResource({diff --cc frontend-modern/src/components/Workloads/GuestDrawerHistory.tsxindex 93122846a1,93122846a1..f2ea90a95f--- a/frontend-modern/src/components/Workloads/GuestDrawerHistory.tsx+++ b/frontend-modern/src/components/Workloads/GuestDrawerHistory.tsx@@@ -59,6 -59,6 +59,8 @@@ export interface GuestDrawerHistoryAlerinterface GuestDrawerHistoryProps {currentMetrics?: Record<string, number | null | undefined>;++ /** False when the owning resource no longer supplies current readings. */++ liveReadingsAvailable?: boolean;deferredMetrics?: Record<string, GuestDrawerHistoryDeferredMetric | undefined>;groups?: GuestDrawerHistoryGroupConfig[];range: HistoryTimeRange;@@@ -858,6 -858,6 +860,7 @@@ export const GuestDrawerHistory: Compon() => props.target,() => props.range,() => !locked() && props.enabled !== false,++ () => props.liveReadingsAvailable !== false,);const historyQuery = createGuestDrawerHistoryQuery(historySource);// A successful retry removes the focused Retry button with the error. Keep@@@ -1012,7 -1012,7 +1015,9 @@@group={group}loading={historyQuery.loading() && !historyQuery.resolvedOnce()}metrics={metrics()}-- currentMetrics={props.currentMetrics}++ currentMetrics={++ props.liveReadingsAvailable !== false ? props.currentMetrics : undefined++ }deferredMetrics={props.deferredMetrics}range={props.range}source={historyQuery.value().source}diff --cc frontend-modern/src/components/Workloads/__tests__/GuestDrawerHistory.refresh.test.tsxindex 91e44027ac,91e44027ac..bab700415c--- a/frontend-modern/src/components/Workloads/__tests__/GuestDrawerHistory.refresh.test.tsx+++ b/frontend-modern/src/components/Workloads/__tests__/GuestDrawerHistory.refresh.test.tsx@@@ -79,6 -79,6 +79,62 @@@ function deferred<T>()}describe('GuestDrawerHistory refresh recovery', () => {++ it('withdraws a cached live reply before refreshing when reporting stops, then resumes on a new report', async () => {++ const pending = deferred<AllMetricsHistoryResponse>();++ const fetch = vi++ .spyOn(ChartsAPI, 'getMetricsHistory')++ .mockResolvedValueOnce({ ...response(42), source: 'live' })++ .mockReturnValueOnce(pending.promise)++ .mockResolvedValueOnce({ ...response(77), source: 'live' });++ const [reporting, setReporting] = createSignal(true);++ const { container } = render(() => (++ <GuestDrawerHistory++ target={target}++ range="24h"++ liveReadingsAvailable={reporting()}++ currentMetrics={{ cpu: 99 }}++ />++ ));++ await waitFor(() => expect(container).toHaveTextContent('Current'));++ setReporting(false);++ await waitFor(() => expect(fetch).toHaveBeenCalledTimes(2));++ expect(container).not.toHaveTextContent('Current');++ expect(container.querySelector('[data-history-current]')).toBeNull();++ expect(paths(container)).toHaveLength(0);++ // An API response from a different delivery generation cannot restore a++ // current reading while the canonical source is still unreported.++ pending.resolve({ ...response(66), source: 'live' });++ await pending.promise;++ await Promise.resolve();++ expect(container).not.toHaveTextContent('Current');++ expect(paths(container)).toHaveLength(0);++ setReporting(true);++ await waitFor(() => expect(fetch).toHaveBeenCalledTimes(3));++ await waitFor(() => expect(container).toHaveTextContent('77.0%'));++ expect(container).toHaveTextContent('Current');++ });++++ it('retains real stored observations while a reporting transition refreshes history', async () => {++ const pending = deferred<AllMetricsHistoryResponse>();++ const fetch = vi++ .spyOn(ChartsAPI, 'getMetricsHistory')++ .mockResolvedValueOnce(response(37))++ .mockReturnValueOnce(pending.promise);++ const [reporting, setReporting] = createSignal(true);++ const { container } = render(() => (++ <GuestDrawerHistory target={target} range="24h" liveReadingsAvailable={reporting()} />++ ));++ await waitFor(() => expect(paths(container)).toHaveLength(1));++ setReporting(false);++ await waitFor(() => expect(fetch).toHaveBeenCalledTimes(2));++ expect(paths(container)).toHaveLength(1);++ expect(container).toHaveTextContent('Latest stored');++ expect(container).toHaveTextContent('37.0%');++ pending.resolve(response(38));++ await waitFor(() => expect(container).toHaveTextContent('38.0%'));++ expect(container).not.toHaveTextContent('Current');++ });++it('offers one scoped retry after initial failure without claiming collection or exposing errors', async () => {const pending = deferred<AllMetricsHistoryResponse>();const fetch = vidiff --cc frontend-modern/src/components/Workloads/guestDrawerHistoryQuery.tsindex dddfd10ede,dddfd10ede..661f91c40d--- a/frontend-modern/src/components/Workloads/guestDrawerHistoryQuery.ts+++ b/frontend-modern/src/components/Workloads/guestDrawerHistoryQuery.ts@@@ -21,6 -21,6 +21,7 @@@ export interface GuestDrawerHistoryQuerresourceType: ResourceType;resourceId: string;range: HistoryTimeRange;++ liveReadingsAvailable: boolean;}const EMPTY_HISTORY_RESPONSE: AllMetricsHistoryResponse = {@@@ -48,12 -48,12 +49,24 @@@ const normalizeHistoryResponse =};};++// A cached live reply describes the source before it stopped reporting. Keep++// real history, but withdraw that live snapshot as soon as its authority ends.++const filterUnavailableLiveReading = (++ response: AllMetricsHistoryResponse,++ available: boolean,++): AllMetricsHistoryResponse =>++ !available && response.source === 'live'++ ? { ...response, metrics: {}, source: 'store' }++ : response;++// Snapshot replacement often creates a new target object with unchanged--// coordinates. Only a real target/range/access change owns a new read.++// coordinates. A delivery transition also owns a new read, so an open drawer++// does not wait for its next poll after the source stops or resumes reporting.export const createGuestDrawerHistorySource = (target: Accessor<GuestDrawerHistoryTarget | null>,range: Accessor<HistoryTimeRange>,enabled: Accessor<boolean>,++ liveReadingsAvailable: Accessor<boolean> = () => true,): Accessor<GuestDrawerHistoryQueryKey | null> =>createMemo<GuestDrawerHistoryQueryKey | null>(() => {@@@ -63,6 -63,6 +76,7 @@@resourceType: current.resourceType,resourceId: current.resourceId,range: range(),++ liveReadingsAvailable: liveReadingsAvailable(),};},null,@@@ -74,7 -74,7 +88,8 @@@next &&previous.resourceType === next.resourceType &&previous.resourceId === next.resourceId &&-- previous.range === next.range,++ previous.range === next.range &&++ previous.liveReadingsAvailable === next.liveReadingsAvailable,),},);@@@ -84,18 -84,18 +99,21 @@@// History view reuse the Overview's read instead of starting another.export const createGuestDrawerHistoryQuery = (source: Accessor<GuestDrawerHistoryQueryKey | null>,--) =>-- createNonSuspendingQuery<AllMetricsHistoryResponse, GuestDrawerHistoryQueryKey>({++) => {++ const query = createNonSuspendingQuery<AllMetricsHistoryResponse, GuestDrawerHistoryQueryKey>({source,fetcher: async (key, signal) =>-- normalizeHistoryResponse(-- await ChartsAPI.getMetricsHistory({-- resourceType: key.resourceType,-- resourceId: key.resourceId,-- range: key.range,-- maxPoints: GUEST_DRAWER_HISTORY_MAX_POINTS,-- signal,-- }),++ filterUnavailableLiveReading(++ normalizeHistoryResponse(++ await ChartsAPI.getMetricsHistory({++ resourceType: key.resourceType,++ resourceId: key.resourceId,++ range: key.range,++ maxPoints: GUEST_DRAWER_HISTORY_MAX_POINTS,++ signal,++ }),++ ),++ key.liveReadingsAvailable,),initialValue: EMPTY_HISTORY_RESPONSE,cacheKey: (key) => `guest-drawer-history:${key.resourceType}:${key.resourceId}:${key.range}`,@@@ -104,3 -104,3 +122,12 @@@retainPreviousValueOnSourceChange: false,pollMs: GUEST_DRAWER_HISTORY_POLL_MS,});++ return {++ ...query,++ // Matching cached history stays visible during the fresh read. A live++ // cache entry loses authority synchronously, before that read can settle.++ value: createMemo(() =>++ filterUnavailableLiveReading(query.value(), source()?.liveReadingsAvailable !== false),++ ),++ };++};