Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Show a partial PMG read as degraded, not unreachable
rcourtman53990d55f025
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 552c227134..ad929610dc 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -9973,6 +9973,21 @@ than by a non-default org's detached config copy. Agent connection rows keeptheir own heartbeat cutoff. Agent registration, enrolment, install, update,removal and report identity are unchanged.+### Platform partial reads show as degraded rows++`frontend-modern/src/components/Settings/useConnectionsLedger.ts`,+`connectionsTableModel.ts` and `InfrastructureSourceManager.tsx` changed only+so a platform connection in the new `degraded` state (a poll that answered but+left one reading unread, #2960) renders an amber "Degraded" badge, counts as a+contributing source, and shows its error banner in warning tone naming the+failed reading ("Queue status could not be read: ...") through+`formatPartialReadErrorMessage`. Every other error banner keeps the critical+treatment. Agent connections never take the `degraded` state, so+`agentAttachmentProblem`, agent registration, enrolment, install, update,+removal and report identity are unchanged. The rendered rows are pinned in+`InfrastructureWorkspace.test.tsx` ("shows a partial read as a degraded+source..." and "keeps an outage error banner critical").+### Host snapshots report a linked agent's own heartbeat`internal/monitoring/monitor.go` changed only so a host produced from the readdiff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.mdindex 93db1538ba..3c56bc1cd4 100644--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@ -253,6 +253,20 @@ and exactly-once callbacks with shadow parity. `TestInterruptDiscreteRun` inacknowledgement, no fabricated resolution and idempotent interruption. Thesesynthetic source checks do not establish native outage or destination delivery.+### Partial platform reads alert as degraded warnings++A platform connection whose poll answered but left one reading unread (poll+error category `partial`) reaches `CheckConnection` as state `degraded`. It is+a degraded observation of the same `connection-degraded` discrete state, at+warning severity, never critical: it activates after the usual three+observations, keeps an open outage open (a missing reading is not recovery+evidence), and moves an already-firing critical outage down to warning once the+connection answers again. Only `active` runs the recovery confirmation. The+alert message and metadata keep the failed read (`stateReason`, `lastError`,+`lastErrorCategory: partial`). No new alert family, threshold, policy or route+is added. `TestConnectionDegradedPartialReadAlertsAtWarning` in+`operational_contract_test.go` pins this.+### Metric timing requires continuous observed evidenceUnknown metric evidence interrupts timing, not the occurrence. Missing ordiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 6e62140042..74581bf142 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -12586,6 +12586,29 @@ finding's resource reference through `Monitor.GetUnifiedStructureReadState`; inmock mode that is the view built for the current fixture structure, outside itthe read state it used before.+### A partial platform read is degraded, not unreachable (#2960)++`deriveConnectionState` (`internal/api/connections_aggregator.go`) maps an+outstanding poll error with category `monitoring.PollErrorCategoryPartial` to+the connection state `degraded`, with the failed read as `stateReason` and+`lastError` (whose `category` stays `partial` on the wire and in the+`frontend-modern/src/api/connections.ts` `ConnectionError` type). The poll+reached the connection, authenticated and read its version, so it is not+`unreachable`, and a 401/403 text from one optional endpoint does not make the+credential `unauthorized`. A partial connection whose last success has also+aged past the stale cutoff is still `stale`. `degraded` rolls up as fleet+adapter health `degraded`, keeps credential status `verified`, does not block+rollout or runtime workload inventory, and ranks between `pending` and `stale`+in the system-member and ledger severity orders. The Connections page shows it+as an amber "Degraded" badge with a "Degraded" problem line and the failed+read, and counts it as a contributing source. No route, request field or+status code changes; `degraded` is a new value of the existing `state` field.+`TestDeriveConnectionState_PartialReadIsDegraded`,+`TestDeriveConnectionState_PartialReadStillGoesStale` and+`TestBuildConnections_PMGPartialReadIsDegradedNotBlocked` in+`connections_aggregator_test.go`, and the degraded-row case in+`useConnectionsLedger.test.ts`, pin this.+### Loaded availability Settings honours evidence expiry (10 October 2026)The existing availability Settings client passes the shared current time todiff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex b5515d262c..7848744454 100644--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@ -10066,6 +10066,17 @@ transcript block moved to `frontend-modern/src/components/patrol/` with itsat `/settings/pulse-intelligence/agents`; the old `/assistant` path redirectsthere.+### Source-row error banners follow the row's severity++`InfrastructureSourceManager.tsx` renders a row's error banner, in both the+table and the narrow card layout, from one `rowErrorBannerClass` helper keyed+on `row.lastErrorTone`. `useConnectionsLedger.ts` sets that tone to `warning`+for a partial-read error on a `degraded` connection and to `critical` for every+other error, so a reachable source with one failed reading gets an amber+banner beside its amber "Degraded" badge, while outages keep the rose banner.+No new component, primitive or badge vocabulary is added. The rendered tones are+pinned in `InfrastructureWorkspace.test.tsx`.+### Configuration transfers retain their confirmed workspaceRecovery's overview identifies the selected-workspace configuration export asdiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 4e0cf6f55b..d0e779f1de 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -857,6 +857,11 @@ datastore and node-status reads out of the scheduler outcome.`TestRecordTaskOutcomeReadErrorSchedulesAsSuccess` in`monitor_additional_test.go` pins the scheduler bookkeeping, which records thesuccess and the outstanding read error in one update.+`poll_error_category.go` exports the category as `PollErrorCategoryPartial`,+the one value the connections ledger reads to present such a poll as a+degraded, still-reachable connection (see the API contract's partial-read+clause) rather than an outage; `TestRecordTaskOutcomeReadErrorSchedulesAsSuccess`+pins that the recorded category is that exported value.The PMG client rejects omitted/null object payloads for version, mail, queueand quarantine instead of materialising them as zero structs. Explicitdiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 979511f1b2..5637614157 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -2811,7 +2811,7 @@ recovery scope, or a storage/recovery-owned secret source.`internal/api/configapi/config_handlers.go` and `internal/api/configapi/config_node_handlers.go`may carry the new per-instance `Enabled`/`Disabled` round-trip, butstorage and recovery consumers must not reinterpret the derived- connection `state` (active/paused/unauthorized/unreachable/stale/pending)+ connection `state` (active/paused/unauthorized/unreachable/stale/pending/degraded)as storage health, backup-job posture, or recovery verification state;they also must not repurpose the shared probe route as a generic internalreachability scanner. Metadata, link-local, multicast, and unspecifieddiff --git a/frontend-modern/browser-verification/0ac5bcb6c59cbb8d.json b/frontend-modern/browser-verification/0ac5bcb6c59cbb8d.jsonnew file mode 100644index 0000000000..851f1feb1f--- /dev/null+++ b/frontend-modern/browser-verification/0ac5bcb6c59cbb8d.json@@ -0,0 +1,47 @@+{+ "version": 1,+ "base_sha": "0d7f2b7997dad75977d84ae35590f41d10374bd4",+ "verified_at": "2026-10-26T11:50:00Z",+ "result": "passed",+ "changed_paths": [+ "frontend-modern/src/api/connections.ts",+ "frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx",+ "frontend-modern/src/components/Settings/connectionsTableModel.ts",+ "frontend-modern/src/components/Settings/useConnectionsLedger.ts",+ "frontend-modern/src/utils/connectionErrorPresentation.ts"+ ],+ "content_sha256": {+ "frontend-modern/src/api/connections.ts": "ca3393144dc7949b40dafe2562105a1b99d613923c057a8a3be35aac11bdc182",+ "frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx": "7a9e0aa7b43febf84287dbd3b15d61aa859bc7f85655674f5301fa6f2caf3b44",+ "frontend-modern/src/components/Settings/connectionsTableModel.ts": "dacc4a4fa8c29483f03d036111d4d0583f3d5b4033362e184f6c359ccecfac6e",+ "frontend-modern/src/components/Settings/useConnectionsLedger.ts": "19ca6b9c1c98c90a83856f6526546e647d1eb4f390dbb42f34f02a38ee0f745c",+ "frontend-modern/src/utils/connectionErrorPresentation.ts": "c75d8dabf06db81f194c223b6b7e6bb44a3baf625ae20d930aeab14942b623b1"+ },+ "routes": [+ "/settings/infrastructure",+ "/alerts/overview"+ ],+ "viewports": [+ {+ "width": 1280,+ "height": 800+ },+ {+ "width": 375,+ "height": 812+ }+ ],+ "states": [+ "Before (pre-change binary, same fake PMG whose qshape read returns HTTP 500): mail-gw row shows a red Unreachable badge and a red 'Server error' banner; a critical connection-degraded alert fires",+ "After: the same row shows an amber Degraded badge, coverage Mail stats, Quarantine, Queues, and an amber banner 'Queue status could not be read: Server error. ...'",+ "After: /api/connections reports state degraded, adapterHealth degraded, lastError.category partial",+ "After: Alerts overview lists one connection-degraded alert at WARNING naming the pmg_queue_status read",+ "After at 375px: the card layout shows the Degraded badge and the amber banner without clipping"+ ],+ "interactions": [+ "Added the fake PMG through POST /api/config/nodes on each stack and waited for polls",+ "Opened the mail-gw Manage drawer (editor only, no health readout) and closed it",+ "Opened the Alerts overview and read the active alert",+ "Resized to phone width and reloaded Settings > Infrastructure"+ ]+}diff --git a/frontend-modern/src/api/__tests__/connections.test.ts b/frontend-modern/src/api/__tests__/connections.test.tsindex 8cbbb08e97..594cf66cea 100644--- a/frontend-modern/src/api/__tests__/connections.test.ts+++ b/frontend-modern/src/api/__tests__/connections.test.ts@@ -64,6 +64,38 @@ describe('ConnectionsAPI', () => {expect(result).toEqual({ connections, systems });});+ it('list() keeps a degraded partial-read row and its error category', async () => {+ const connections: Connection[] = [+ {+ id: 'pmg:mail',+ type: 'pmg',+ name: 'mail',+ address: 'https://pmg.lan:8006',+ state: 'degraded',+ stateReason: 'pmg_queue_status failed on mail: API error 500',+ enabled: true,+ surfaces: ['queues'],+ scope: { queues: true },+ lastSeen: '2026-10-26T12:00:00Z',+ lastError: {+ at: '2026-10-26T12:00:00Z',+ message: 'pmg_queue_status failed on mail: API error 500',+ category: 'partial',+ },+ source: 'manual',+ capabilities: { supportsPause: true, supportsScope: true, supportsTest: true },+ },+ ];+ mockedApiFetchJSON.mockResolvedValueOnce({ connections, systems: [] });++ const result = await ConnectionsAPI.list();++ expect(result.connections[0]).toMatchObject({+ state: 'degraded',+ lastError: { category: 'partial' },+ });+ });+it('list() normalizes missing fields to empty collections', async () => {mockedApiFetchJSON.mockResolvedValueOnce({} as { connections?: Connection[] });diff --git a/frontend-modern/src/api/connections.ts b/frontend-modern/src/api/connections.tsindex 747fe02767..eebef55d6d 100644--- a/frontend-modern/src/api/connections.ts+++ b/frontend-modern/src/api/connections.ts@@ -5,7 +5,7 @@ export type ConnectionType ='pve' | 'pbs' | 'pmg' | 'vmware' | 'truenas' | 'availability' | 'agent' | 'docker' | 'kubernetes';export type ConnectionState =- 'active' | 'paused' | 'unauthorized' | 'unreachable' | 'stale' | 'pending';+ 'active' | 'paused' | 'unauthorized' | 'unreachable' | 'stale' | 'pending' | 'degraded';export type ConnectionSource = 'manual' | 'agent' | 'script';@@ -86,6 +86,9 @@ export interface ConnectionCapabilities {export interface ConnectionError {message: string;at: string;+ // "partial" marks a downstream read that failed after the connection+ // itself answered; the connection state is then "degraded".+ category?: string;}export interface ConnectionFleetGovernance {diff --git a/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx b/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsxindex 81936b216b..6fc8a7fd6c 100644--- a/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx+++ b/frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx@@ -225,6 +225,13 @@ interface ConfiguredSourceGroup {rows: InfrastructureSystemRow[];}+// A partial read is a warning on a reachable source; any other error is an+// outage and keeps the critical treatment.+const rowErrorBannerClass = (row: InfrastructureSystemRow): string =>+ row.lastErrorTone === 'warning'+ ? 'rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-900/25 dark:bg-amber-950/25 dark:text-amber-200'+ : 'rounded-md border border-rose-300 bg-rose-50 px-3 py-2 text-xs text-rose-800 dark:border-rose-900/25 dark:bg-rose-950/25 dark:text-rose-200';+const formatCount = (count: number, noun: string): string =>`${count} ${noun}${count === 1 ? '' : 's'}`;@@ -1178,7 +1185,7 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP<Show when={row.lastErrorMessage}><divrole="alert"- class="mt-1 rounded-md border border-rose-300 bg-rose-50 px-3 py-2 text-xs text-rose-800 dark:border-rose-900/25 dark:bg-rose-950/25 dark:text-rose-200"+ class={`mt-1 ${rowErrorBannerClass(row)}`}>{row.lastErrorMessage}</div>@@ -1775,10 +1782,7 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP</footer><Show when={row.lastErrorMessage}>- <div- role="alert"- class="mt-2 rounded-md border border-rose-300 bg-rose-50 px-3 py-2 text-xs text-rose-800 dark:border-rose-900/25 dark:bg-rose-950/25 dark:text-rose-200"- >+ <div role="alert" class={`mt-2 ${rowErrorBannerClass(row)}`}>{row.lastErrorMessage}</div></Show>diff --git a/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx b/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsxindex 384951e125..3452aaedab 100644--- a/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx+++ b/frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx@@ -366,6 +366,79 @@ describe('InfrastructureWorkspace', () => {expect(screen.getByRole('button', { name: /^Manage$/i })).toBeInTheDocument();});+ it('shows a partial read as a degraded source with a warning banner naming the reading', async () => {+ const provider = connectionFixture({ state: 'degraded' });+ connectionState.rows = [+ {+ id: provider.id,+ ownerType: provider.type,+ name: provider.name,+ subtitle: 'via platform API',+ source: 'api',+ host: provider.address,+ coverageLabels: ['VMs'],+ statusLabel: 'Degraded',+ statusClassName: 'bg-yellow-100',+ agentUpdateCount: 0,+ lastActivityText: '1m ago',+ lastErrorMessage: 'Queue status could not be read: Server error.',+ lastErrorTone: 'warning',+ ...emptyFleetRow,+ enabled: true,+ canEdit: true,+ canPause: false,+ canRemove: true,+ isAgent: false,+ isCluster: false,+ attachedConnections: [],+ members: [],+ connection: provider,+ },+ ];+ renderWorkspace();++ const banner = await screen.findByText('Queue status could not be read: Server error.');+ expect(banner.getAttribute('class')).toContain('bg-amber-50');+ expect(banner.getAttribute('class')).not.toContain('bg-rose-50');+ expect(screen.getAllByText('Degraded').length).toBeGreaterThan(0);+ expect(screen.queryByText('Unreachable')).toBeNull();+ });++ it('keeps an outage error banner critical', async () => {+ const provider = connectionFixture({ state: 'unreachable' });+ connectionState.rows = [+ {+ id: provider.id,+ ownerType: provider.type,+ name: provider.name,+ subtitle: 'via platform API',+ source: 'api',+ host: provider.address,+ coverageLabels: ['VMs'],+ statusLabel: 'Unreachable',+ statusClassName: 'bg-rose-100',+ agentUpdateCount: 0,+ lastActivityText: '1m ago',+ lastErrorMessage: 'Connection refused.',+ lastErrorTone: 'critical',+ ...emptyFleetRow,+ enabled: true,+ canEdit: true,+ canPause: false,+ canRemove: true,+ isAgent: false,+ isCluster: false,+ attachedConnections: [],+ members: [],+ connection: provider,+ },+ ];+ renderWorkspace();++ const banner = await screen.findByText('Connection refused.');+ expect(banner.getAttribute('class')).toContain('bg-rose-50');+ });+it('renders the source manager landing without empty platform sections', async () => {renderWorkspace();diff --git a/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts b/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.tsindex c66e42d3ed..f4f257fdcc 100644--- a/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts+++ b/frontend-modern/src/components/Settings/__tests__/useConnectionsLedger.test.ts@@ -46,6 +46,52 @@ describe('useConnectionsLedger', () => {expect(result.rows().find((row) => row.ownerType === 'pve')?.statusLabel).toBe('Pending');});+ it('shows a partial read as a degraded, still-reporting connection', async () => {+ const connections: Connection[] = [+ {+ id: 'pmg:mail',+ type: 'pmg',+ name: 'mail',+ address: 'https://pmg.lan:8006',+ state: 'degraded',+ stateReason: 'pmg_queue_status failed on mail: active queue: API error 500: qshape failed',+ enabled: true,+ surfaces: ['mailStats', 'queues'],+ scope: { mailStats: true, queues: true },+ lastSeen: '2026-10-26T12:00:00Z',+ lastError: {+ at: '2026-10-26T12:00:00Z',+ message: 'pmg_queue_status failed on mail: active queue: API error 500: qshape failed',+ category: 'partial',+ },+ source: 'manual',+ capabilities: { supportsPause: true, supportsScope: true, supportsTest: true },+ fleet: {+ enrollmentState: 'configured',+ livenessState: 'degraded',+ versionDrift: 'not-applicable',+ adapterHealth: 'degraded',+ configRollout: 'configured',+ credentialStatus: 'verified',+ updateStatus: 'not-applicable',+ remoteControl: 'not-applicable',+ },+ },+ ];+ vi.spyOn(ConnectionsAPI, 'list').mockResolvedValue({ connections, systems: [] });++ const { result } = renderHook(() => useConnectionsLedger());++ await waitFor(() => expect(result.rows()).toHaveLength(1));+ const row = result.rows()[0];+ expect(row.statusLabel).toBe('Degraded');+ expect(row.statusClassName).toContain('bg-yellow-100');+ expect(row.source).toBe('api');+ // Name the missing reading, in warning tone: the source still answers.+ expect(row.lastErrorMessage).toMatch(/^Queue status could not be read: Server error\./);+ expect(row.lastErrorTone).toBe('warning');+ });+it('renders standalone agent rows with compact host identity and endpoint context', async () => {const connections: Connection[] = [{diff --git a/frontend-modern/src/components/Settings/connectionsTableModel.ts b/frontend-modern/src/components/Settings/connectionsTableModel.tsindex 6b9aad8e6e..04c6a99a8d 100644--- a/frontend-modern/src/components/Settings/connectionsTableModel.ts+++ b/frontend-modern/src/components/Settings/connectionsTableModel.ts@@ -443,6 +443,13 @@ const livenessSignal = (state: ConnectionFleetLivenessState): FleetGovernanceSigdetail: 'Pulse has not seen recent activity from this source.',tone: 'warning',};+ case 'degraded':+ return {+ key: 'liveness',+ label: 'Degraded',+ detail: 'Pulse reaches this source, but some of its readings failed.',+ tone: 'warning',+ };case 'pending':return {key: 'liveness',@@ -1030,6 +1037,9 @@ export interface InfrastructureSystemRow {agentUpdateCount: number;lastActivityText: string;lastErrorMessage?: string;+ // A partial read (some readings failed on a reachable source) is a warning,+ // not an outage.+ lastErrorTone?: 'warning' | 'critical';fleetSignals: FleetGovernanceSignal[];fleetHighlights: FleetGovernanceSignal[];problem?: InfrastructureRowProblem;diff --git a/frontend-modern/src/components/Settings/useConnectionsLedger.ts b/frontend-modern/src/components/Settings/useConnectionsLedger.tsindex a8b382aa61..45623ea45d 100644--- a/frontend-modern/src/components/Settings/useConnectionsLedger.ts+++ b/frontend-modern/src/components/Settings/useConnectionsLedger.ts@@ -1,6 +1,9 @@import { createMemo } from 'solid-js';import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery';-import { formatConnectionErrorMessage } from '@/utils/connectionErrorPresentation';+import {+ formatConnectionErrorMessage,+ formatPartialReadErrorMessage,+} from '@/utils/connectionErrorPresentation';import { AGENT_WAITING_FOR_FIRST_REPORT_LABEL } from '@/utils/agentResources';import {ConnectionsAPI,@@ -64,6 +67,10 @@ const STATE_PRESENTATION: Record<ConnectionState, { label: string; badgeClass: slabel: 'Stale',badgeClass: 'bg-yellow-100 text-yellow-800 dark:bg-yellow-900/25 dark:text-yellow-200',},+ degraded: {+ label: 'Degraded',+ badgeClass: 'bg-yellow-100 text-yellow-800 dark:bg-yellow-900/25 dark:text-yellow-200',+ },pending: {label: 'Pending',badgeClass: 'bg-surface-alt text-base-content',@@ -91,9 +98,10 @@ const CONNECTION_STATE_SEVERITY: Record<ConnectionState, number> = {active: 0,paused: 1,pending: 2,- stale: 3,- unauthorized: 4,- unreachable: 5,+ degraded: 3,+ stale: 4,+ unauthorized: 5,+ unreachable: 6,};const coverageLabelsFor = (connections: readonly Connection[]): string[] => {@@ -140,9 +148,10 @@ const subtitleFor = (connections: readonly Connection[], primaryConnection: Conn// A connection is "contributing" if it is currently delivering data — not// merely configured. Otherwise the source badge claims "API + Agent" when// the agent has been silent for hours, which directly contradicts the-// "Agent offline" chip on the same row.+// "Agent offline" chip on the same row. A degraded connection still+// delivers every reading except the one that failed.const isContributing = (connection: Connection): boolean =>- connection.state === 'active' || connection.state === 'paused';+ connection.state === 'active' || connection.state === 'paused' || connection.state === 'degraded';const sourceFor = (connections: readonly Connection[]): InfrastructureSourceKind => {const contributors = connections.filter(isContributing);@@ -364,10 +373,19 @@ const buildRow = (// Run raw backend error chains through the user-facing presenter so the// connection table doesn't surface implementation strings like 'context// deadline exceeded' or 'Client.Timeout exceeded while awaiting headers'.- const rawLastErrorMessage =- primaryConnection.lastError?.message ??- attachedConnections.find((connection) => connection.lastError?.message)?.lastError?.message;- const lastErrorMessage = formatConnectionErrorMessage(rawLastErrorMessage) ?? undefined;+ const lastError = primaryConnection.lastError?.message+ ? primaryConnection.lastError+ : attachedConnections.find((connection) => connection.lastError?.message)?.lastError;+ const partialRead = lastError?.category === 'partial';+ const lastErrorMessage =+ (partialRead+ ? formatPartialReadErrorMessage(lastError?.message)+ : formatConnectionErrorMessage(lastError?.message)) ?? undefined;+ const lastErrorTone: InfrastructureSystemRow['lastErrorTone'] = lastErrorMessage+ ? partialRead+ ? 'warning'+ : 'critical'+ : undefined;const hostTelemetryLabel = surfaceLabel('host');const coverageLabels = isCluster? coverageLabelsFor(componentConnections).filter((label) => label !== hostTelemetryLabel)@@ -426,6 +444,7 @@ const buildRow = (? lastActivityTextFromLastSeen(rollup.lastSeen): connectionLastActivityText(primaryConnection),lastErrorMessage,+ lastErrorTone,fleetSignals,fleetHighlights,problem:diff --git a/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts b/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.tsindex 46a2b78212..31fe332be7 100644--- a/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts+++ b/frontend-modern/src/utils/__tests__/connectionErrorPresentation.test.ts@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';import {formatConnectionErrorMessage,+ formatPartialReadErrorMessage,humanizeConnectionError,} from '../connectionErrorPresentation';@@ -170,3 +171,35 @@ describe('humanizeConnectionError', () => {expect(formatted).toBe('a totally unknown failure mode');});});++describe('formatPartialReadErrorMessage', () => {+ it('names the failed reading before the cause', () => {+ expect(+ formatPartialReadErrorMessage(+ 'pmg_queue_status failed on mail-gw: active queue: API error 500: {"errors":"qshape failed"}',+ ),+ ).toBe(+ 'Queue status could not be read: Server error. The platform returned a server error. Check the platform logs for the underlying cause.',+ );+ });++ it('counts the other failed readings', () => {+ expect(+ formatPartialReadErrorMessage(+ [+ 'pmg_spam_quarantine failed on mail-gw: API error 403: Permission check failed',+ 'pmg_virus_quarantine failed on mail-gw: API error 403: Permission check failed',+ 'pmg_queue_status failed on mail-gw: API error 500',+ ].join('\n'),+ ),+ ).toMatch(/^Spam quarantine and 2 other readings could not be read: Permission denied\./);+ });++ it('falls back to a generic subject without an operation prefix', () => {+ expect(formatPartialReadErrorMessage('context deadline exceeded')).toMatch(+ /^Some readings could not be read: Connection timed out\./,+ );+ expect(formatPartialReadErrorMessage('')).toBeNull();+ expect(formatPartialReadErrorMessage(undefined)).toBeNull();+ });+});diff --git a/frontend-modern/src/utils/connectionErrorPresentation.ts b/frontend-modern/src/utils/connectionErrorPresentation.tsindex 5746cb78b6..de202f1caa 100644--- a/frontend-modern/src/utils/connectionErrorPresentation.ts+++ b/frontend-modern/src/utils/connectionErrorPresentation.ts@@ -160,3 +160,38 @@ export const formatConnectionErrorMessage = (raw?: string | null): string | nullif (!humanized) return null;return humanized.hint ? `${humanized.headline}. ${humanized.hint}` : humanized.headline;};++// `pmg_queue_status failed on mail-gw: ...`: the monitor's read operation,+// the instance, then the cause.+const READ_OPERATION_PATTERN = /^([a-z]+(?:_[a-z]+)+)\s+failed\s+on\s+[^:]+:\s*/i;++const readOperationLabel = (operation: string): string => {+ const words = operation+ .toLowerCase()+ .replace(/^(?:pmg|pbs|pve)_/, '')+ .replace(/_/g, ' ');+ return words.charAt(0).toUpperCase() + words.slice(1);+};++/**+ * Render a partial-read error: the connection answered, but some readings+ * failed (one per line). The generic cause alone ('Server error') hides which+ * data is missing, so name the failed reading first.+ */+export const formatPartialReadErrorMessage = (raw?: string | null): string | null => {+ const failures = (raw ?? '')+ .split('\n')+ .map((line) => line.trim())+ .filter(Boolean);+ const first = failures[0];+ if (!first) return null;+ const match = first.match(READ_OPERATION_PATTERN);+ const label = match ? readOperationLabel(match[1]) : 'Some readings';+ const cause = formatConnectionErrorMessage(match ? first.slice(match[0].length) : first);+ const others = failures.length - 1;+ const subject =+ match && others > 0+ ? `${label} and ${others} other ${others === 1 ? 'reading' : 'readings'}`+ : label;+ return cause ? `${subject} could not be read: ${cause}` : `${subject} could not be read.`;+};diff --git a/internal/alerts/connection.go b/internal/alerts/connection.goindex 4d3fc7ccc7..17e6a8963d 100644--- a/internal/alerts/connection.go+++ b/internal/alerts/connection.go@@ -24,6 +24,7 @@ const (ConnectionStateUnauthorized ConnectionState = "unauthorized"ConnectionStatePaused ConnectionState = "paused"ConnectionStatePending ConnectionState = "pending"+ ConnectionStateDegraded ConnectionState = "degraded")// ConnectionType narrows the set of connections that participate in the@@ -181,8 +182,9 @@ func (m *Manager) clearConnectionDegradedIntentNoLock(resourceID string) {}// CheckConnection raises or clears the connection-degraded alert for one-// platform connection. Severity scales with observed state: stale → warning,-// unreachable / unauthorized → critical. State=active runs through the+// platform connection. Severity scales with observed state: stale and degraded+// (reachable, but some readings failed) → warning, unreachable / unauthorized+// → critical. State=active runs through the// recovery-confirmation gate before resolving. Paused, disabled, and// non-platform connections are skipped.func (m *Manager) CheckConnection(snap ConnectionSnapshot) {@@ -204,7 +206,7 @@ func (m *Manager) CheckConnection(snap ConnectionSnapshot) {case ConnectionStateActive:m.clearConnectionDegradedAlert(snap)return- case ConnectionStateStale, ConnectionStateUnreachable, ConnectionStateUnauthorized:+ case ConnectionStateStale, ConnectionStateDegraded, ConnectionStateUnreachable, ConnectionStateUnauthorized:// fall through and firedefault:// Pending/unknown breaks both confirmation runs without resolving an@@ -233,6 +235,7 @@ func (m *Manager) CheckConnection(snap ConnectionSnapshot) {connectionDegradedStateKey,[]string{string(ConnectionStateStale),+ string(ConnectionStateDegraded),string(ConnectionStateUnreachable),string(ConnectionStateUnauthorized),},diff --git a/internal/alerts/operational_contract_test.go b/internal/alerts/operational_contract_test.goindex abdef56a18..2d6bccd7bf 100644--- a/internal/alerts/operational_contract_test.go+++ b/internal/alerts/operational_contract_test.go@@ -949,6 +949,66 @@ func TestConnectionDegradedAlertNamesItsPlatform(t *testing.T) {}}+// A partial platform read (the connection answered, one reading failed) is a+// warning-level degraded observation, never an outage (#2960).+func TestConnectionDegradedPartialReadAlertsAtWarning(t *testing.T) {+ t.Run("degraded fires a warning, never critical", func(t *testing.T) {+ m := newTestManager(t)+ snap := platformConnectionSnapshot("conn-partial", "Mail gateway", ConnectionStateDegraded)+ snap.Type = ConnectionTypePMG+ snap.StateReason = "pmg_queue_status: api error 500"+ snap.LastError = &ConnectionErrorSnapshot{At: time.Now(), Message: snap.StateReason, Category: "partial"}+ alertID := canonicalDiscreteStateStateID(snap.ID, connectionDegradedStateKey)++ m.CheckConnection(snap)+ m.CheckConnection(snap)+ if testHasActiveAlert(t, m, alertID) {+ t.Fatal("expected no alert before three degraded observations")+ }+ m.CheckConnection(snap)++ alert := testRequireActiveAlert(t, m, alertID)+ if alert.Level != AlertLevelWarning {+ t.Errorf("expected warning level for degraded, got %s", alert.Level)+ }+ if got, _ := alert.Metadata["state"].(string); got != string(ConnectionStateDegraded) {+ t.Errorf("expected state=degraded in metadata, got %q", got)+ }+ if got, _ := alert.Metadata["lastErrorCategory"].(string); got != "partial" {+ t.Errorf("expected lastErrorCategory=partial in metadata, got %q", got)+ }+ })++ t.Run("degraded keeps an unreachable outage open at warning", func(t *testing.T) {+ m := newTestManager(t)+ snap := platformConnectionSnapshot("conn-partial-recover", "Mail gateway", ConnectionStateUnreachable)+ snap.Type = ConnectionTypePMG+ alertID := canonicalDiscreteStateStateID(snap.ID, connectionDegradedStateKey)++ m.CheckConnection(snap)+ m.CheckConnection(snap)+ m.CheckConnection(snap)+ if alert := testRequireActiveAlert(t, m, alertID); alert.Level != AlertLevelCritical {+ t.Fatalf("expected critical outage before the partial poll, got %s", alert.Level)+ }++ // The connection answers again but one reading still fails: a+ // missing reading is not recovery evidence.+ snap.State = ConnectionStateDegraded+ snap.StateReason = "pmg_queue_status: api error 500"+ for i := 0; i < 5; i++ {+ m.CheckConnection(snap)+ }+ alert := testRequireActiveAlert(t, m, alertID)+ if alert.Level != AlertLevelWarning {+ t.Errorf("expected the open alert to drop to warning, got %s", alert.Level)+ }+ if got, _ := alert.Metadata["state"].(string); got != string(ConnectionStateDegraded) {+ t.Errorf("expected state=degraded in metadata, got %q", got)+ }+ })+}+func TestCapacityForecastAlertNamesItsPlatform(t *testing.T) {m := newTestManager(t)m.ClearActiveAlerts()diff --git a/internal/api/connections_aggregator.go b/internal/api/connections_aggregator.goindex 8dd63aa70e..c821de8f70 100644--- a/internal/api/connections_aggregator.go+++ b/internal/api/connections_aggregator.go@@ -769,7 +769,7 @@ func connectionFleetAdapterHealth(conn Connection) string {switch conn.State {case ConnectionStateActive:return fleetStateHealthy- case ConnectionStateStale, ConnectionStatePending:+ case ConnectionStateStale, ConnectionStatePending, ConnectionStateDegraded:return fleetStateDegradedcase ConnectionStateUnauthorized, ConnectionStateUnreachable:return fleetStateBlocked@@ -1459,6 +1459,18 @@ func deriveConnectionState(enabled bool, h monitoring.InstanceHealth, now time.Treturn ConnectionStatePending, "awaiting first poll", nil, nil}+ // A partial error means the poll reached the connection, authenticated and+ // read its version; only a downstream read failed (#2960). The connection+ // is reachable and its other readings are fresh, so it is degraded, not+ // unreachable, and a 403 from one endpoint does not make the credential+ // unauthorized. A connection that also stopped polling is still stale.+ if lastError != nil && lastError.Category == monitoring.PollErrorCategoryPartial {+ if lastSeen != nil && now.Sub(*lastSeen) > connectionStaleThresholdFor(pollInterval) {+ return ConnectionStateStale, fmt.Sprintf("no successful poll in %s", now.Sub(*lastSeen).Round(time.Second)), lastSeen, lastError+ }+ return ConnectionStateDegraded, lastError.Message, lastSeen, lastError+ }+if lastError != nil && connectionAuthErrorPattern.MatchString(lastError.Message) {return ConnectionStateUnauthorized, lastError.Message, lastSeen, lastError}diff --git a/internal/api/connections_aggregator_test.go b/internal/api/connections_aggregator_test.goindex 4af4841114..ecb74087c1 100644--- a/internal/api/connections_aggregator_test.go+++ b/internal/api/connections_aggregator_test.go@@ -107,6 +107,78 @@ func TestDeriveConnectionState_Stale(t *testing.T) {}}+// A partial poll error (#2960) means the connection answered and only one+// downstream read failed: degraded, never unreachable or unauthorized.+func TestDeriveConnectionState_PartialReadIsDegraded(t *testing.T) {+ now := time.Now()+ for _, message := range []string{+ "pmg_queue_status: api error 500: queue status unavailable",+ "pmg_quarantine: api error 403: Permission check failed",+ } {+ h := healthEntry(ptrTime(now.Add(-10*time.Second)), message, monitoring.PollErrorCategoryPartial, "closed")+ state, reason, lastSeen, lastError := deriveConnectionState(true, h, now, 0)+ if state != ConnectionStateDegraded {+ t.Fatalf("%q: got %q, want %q", message, state, ConnectionStateDegraded)+ }+ if reason != message {+ t.Fatalf("%q: reason = %q, want the failed read", message, reason)+ }+ if lastSeen == nil {+ t.Fatalf("%q: expected lastSeen from the partial poll's success", message)+ }+ if lastError == nil || lastError.Category != monitoring.PollErrorCategoryPartial || lastError.Message != message {+ t.Fatalf("%q: lastError = %+v, want the partial read kept visible", message, lastError)+ }+ }+}++func TestDeriveConnectionState_PartialReadStillGoesStale(t *testing.T) {+ now := time.Now()+ h := healthEntry(ptrTime(now.Add(-5*time.Minute)), "pmg_queue_status: api error 500", monitoring.PollErrorCategoryPartial, "closed")+ state, _, _, lastError := deriveConnectionState(true, h, now, 0)+ if state != ConnectionStateStale {+ t.Fatalf("got %q, want %q", state, ConnectionStateStale)+ }+ if lastError == nil || lastError.Category != monitoring.PollErrorCategoryPartial {+ t.Fatalf("lastError = %+v, want the partial read kept visible", lastError)+ }+}++func TestBuildConnections_PMGPartialReadIsDegradedNotBlocked(t *testing.T) {+ now := time.Now()+ got := buildConnections(aggregatorInputs{+ pmgInstances: []config.PMGInstance{{Name: "mail-main", Host: "https://pmg.lan:8006", User: "root@pam", Password: "secret"}},+ instanceHealth: map[string]monitoring.InstanceHealth{+ "pmg::mail-main": healthEntry(ptrTime(now.Add(-10*time.Second)), "pmg_quarantine: api error 403: Permission check failed", monitoring.PollErrorCategoryPartial, "closed"),+ },+ now: now,+ })+ if len(got) != 1 {+ t.Fatalf("expected 1 connection, got %d", len(got))+ }+ conn := got[0]+ if conn.State != ConnectionStateDegraded {+ t.Fatalf("state = %q, want %q", conn.State, ConnectionStateDegraded)+ }+ if conn.Fleet.AdapterHealth != fleetStateDegraded {+ t.Fatalf("adapter health = %q, want %q", conn.Fleet.AdapterHealth, fleetStateDegraded)+ }+ if conn.Fleet.CredentialStatus != fleetStateVerified {+ t.Fatalf("credential status = %q, want %q: one endpoint's 403 is not a rejected credential", conn.Fleet.CredentialStatus, fleetStateVerified)+ }+ if conn.Fleet.Rollout != nil && conn.Fleet.Rollout.Status == fleetRolloutBlocked {+ t.Fatalf("rollout = %+v, a reachable connection must not be blocked", conn.Fleet.Rollout)+ }++ snaps := snapshotConnectionsForAlerts(got)+ if len(snaps) != 1 || snaps[0].State != alerts.ConnectionStateDegraded {+ t.Fatalf("alert snapshots = %+v, want one degraded snapshot", snaps)+ }+ if snaps[0].LastError == nil || snaps[0].LastError.Category != monitoring.PollErrorCategoryPartial {+ t.Fatalf("alert snapshot lastError = %+v, want the partial read", snaps[0].LastError)+ }+}+func TestDeriveConnectionState_Active(t *testing.T) {now := time.Now()h := healthEntry(ptrTime(now.Add(-10*time.Second)), "", "", "closed")diff --git a/internal/api/connections_grouping.go b/internal/api/connections_grouping.goindex 1597325242..b8e8d80b84 100644--- a/internal/api/connections_grouping.go+++ b/internal/api/connections_grouping.go@@ -415,12 +415,14 @@ func connectionSystemMemberStateSeverity(state ConnectionState) int {return 1case ConnectionStatePending:return 2- case ConnectionStateStale:+ case ConnectionStateDegraded:return 3- case ConnectionStateUnauthorized:+ case ConnectionStateStale:return 4- case ConnectionStateUnreachable:+ case ConnectionStateUnauthorized:return 5+ case ConnectionStateUnreachable:+ return 6default:return 2}diff --git a/internal/api/connections_types.go b/internal/api/connections_types.goindex a9d560514d..f7b31ca5aa 100644--- a/internal/api/connections_types.go+++ b/internal/api/connections_types.go@@ -14,6 +14,9 @@ const (ConnectionStateUnreachable ConnectionState = "unreachable"ConnectionStateStale ConnectionState = "stale"ConnectionStatePending ConnectionState = "pending"+ // ConnectionStateDegraded is a reachable connection whose latest poll+ // left some readings unread (a poll error with category "partial").+ ConnectionStateDegraded ConnectionState = "degraded")// ConnectionType is the product family that owns the connection. The value isdiff --git a/internal/api/contract_test.go b/internal/api/contract_test.goindex 3f88f18416..96f9c3982c 100644--- a/internal/api/contract_test.go+++ b/internal/api/contract_test.go@@ -18327,6 +18327,52 @@ func TestContract_ConnectionPayloadShapeStaysCanonical(t *testing.T) {assertJSONSnapshot(t, body, want)}+// TestContract_ConnectionPartialReadPayloadIsDegraded pins the wire shape of a+// platform connection whose poll answered but left one reading unread (#2960):+// state "degraded", the failed read as stateReason, and lastError.category+// "partial", with credentials still verified.+func TestContract_ConnectionPartialReadPayloadIsDegraded(t *testing.T) {+ now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)+ lastSuccess := now.Add(-10 * time.Second)+ message := "pmg_queue_status failed on mail: API error 500"+ conn := buildPMGConnection(config.PMGInstance{Name: "mail", Host: "https://pmg.lan:8006", User: "root@pam", Password: "secret"},+ map[string]monitoring.InstanceHealth{+ "pmg::mail": {+ PollStatus: monitoring.InstancePollStatus{+ LastSuccess: &lastSuccess,+ LastError: &monitoring.ErrorDetail{At: lastSuccess, Message: message, Category: monitoring.PollErrorCategoryPartial},+ },+ Breaker: monitoring.InstanceBreaker{State: "closed"},+ },+ }, now, 0)+ body, err := json.Marshal(conn)+ if err != nil {+ t.Fatalf("marshal Connection: %v", err)+ }+ var got struct {+ State string `json:"state"`+ StateReason string `json:"stateReason"`+ LastError struct {+ Message string `json:"message"`+ Category string `json:"category"`+ } `json:"lastError"`+ Fleet struct {+ LivenessState string `json:"livenessState"`+ AdapterHealth string `json:"adapterHealth"`+ CredentialStatus string `json:"credentialStatus"`+ } `json:"fleet"`+ }+ if err := json.Unmarshal(body, &got); err != nil {+ t.Fatalf("unmarshal Connection: %v", err)+ }+ if got.State != "degraded" || got.StateReason != message || got.LastError.Message != message || got.LastError.Category != "partial" {+ t.Fatalf("partial read payload = %s", body)+ }+ if got.Fleet.LivenessState != "degraded" || got.Fleet.AdapterHealth != "degraded" || got.Fleet.CredentialStatus != "verified" {+ t.Fatalf("partial read fleet payload = %s", body)+ }+}+func TestContract_ConnectionSystemMembersPayloadShapeStaysCanonical(t *testing.T) {system := ConnectionSystem{ID: "pve-lab",diff --git a/internal/monitoring/monitor_additional_test.go b/internal/monitoring/monitor_additional_test.goindex dd78c7cbb1..14dbaadbca 100644--- a/internal/monitoring/monitor_additional_test.go+++ b/internal/monitoring/monitor_additional_test.go@@ -1910,7 +1910,7 @@ func TestRecordTaskOutcomeReadErrorSchedulesAsSuccess(t *testing.T) {}status := m.pollStatusMap[key]if status.ConsecutiveFailures != 0 || status.LastSuccess.IsZero() || !status.FirstFailureAt.IsZero() ||- status.LastErrorCategory != pollReadErrorCategory || !strings.Contains(status.LastErrorMessage, "pmg_queue_status") ||+ status.LastErrorCategory != PollErrorCategoryPartial || !strings.Contains(status.LastErrorMessage, "pmg_queue_status") ||!status.LastErrorAt.Equal(status.LastSuccess) {t.Fatalf("read error not kept as the outstanding poll error: %+v", status)}diff --git a/internal/monitoring/poll_error_category.go b/internal/monitoring/poll_error_category.gonew file mode 100644index 0000000000..b762b8d783--- /dev/null+++ b/internal/monitoring/poll_error_category.go@@ -0,0 +1,7 @@+package monitoring++// PollErrorCategoryPartial is the ErrorDetail.Category recordTaskOutcome sets+// when a poll reached the connection but left some readings unread. The+// connections ledger reads it to present a degraded, still-reachable+// connection instead of an outage (#2960).+const PollErrorCategoryPartial = pollReadErrorCategory